Training method and apparatus for speech recognition model, and electronic device and storage medium

Abstract

A training method for a speech recognition model, and a speech recognition method. The training method comprises: acquiring training audio data (S202); performing encoding processing on the training audio data, so as to obtain first feature data (S204); performing feature extraction on key text data by means of an initial key text sub-model, so as to obtain second feature data (S206); performing feature fusion on the first feature data and the second feature data by means of an initial feature fusion sub-model, so as to obtain third feature data (S208); performing decoding processing on fourth feature data that corresponds to the first feature data and the third feature data, so as to obtain recognized text data corresponding to the training audio data (S210); and on the basis of the recognized text data and labeled text data, training the initial key text sub-model and the initial feature fusion sub-model, so as to obtain a speech recognition model on the basis of the trained key text sub-model and the trained feature fusion sub-model (S212).

Description

Speech recognition model training method, device, electronic device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202410155152.2, filed on February 2, 2024, entitled “Training method, device, electronic device and storage medium for speech recognition model”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to a method for training a speech recognition model, a device for training a speech recognition model, a speech recognition method, a speech recognition device, an electronic device, and a storage medium. Background Art

[0004] With the rapid development of artificial intelligence and deep learning technologies, automatic speech recognition (ASR) has become a mainstream field in speech recognition. Conformer-based ASR models employ an encoder-decoder network architecture and combine global and local audio feature modeling. They have proven superior to other ASR models in various ASR tasks.

[0005] In related technologies, the speech recognition performance of Conformer-based ASR models is highly dependent on the distribution of the training dataset. Therefore, when recognizing long-tail hot words (also known as keywords), such as names of people, places, and products, the accuracy and recall rates are low, resulting in poor recognition results.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0007] The embodiments of the present disclosure provide a method for training a speech recognition model, a device for training a speech recognition model, a speech recognition method, a speech recognition device, an electronic device, and a storage medium.

[0008] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0009] An embodiment of the present disclosure provides a method for training a speech recognition model, comprising: acquiring training audio data; encoding the training audio data to obtain first feature data; extracting features from the key text data through an initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined based on annotated text data corresponding to the training audio data; fusing the first feature data and the second feature data through an initial feature fusion sub-model to obtain third feature data; decoding fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; training the initial key text sub-model and the initial feature fusion sub-model based on the recognition text data and the annotated text data, so as to obtain the speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

[0010] In some exemplary embodiments of the present disclosure, the training of the initial key text sub-model and the initial feature fusion sub-model based on the recognized text data and the annotated text data includes: determining the loss value corresponding to the recognized text data and the annotated text data based on a preset loss function; and updating the first model parameter of the initial key text sub-model and the second model parameter of the initial feature fusion sub-model in response to the loss value failing to meet a preset convergence condition.

[0011] In some exemplary embodiments of the present disclosure, the key text data is first key text data excerpted from the annotated text data according to a preset excerpt window in response to a randomly generated value corresponding to the training audio data being less than a preset probability value.

[0012] In some exemplary embodiments of the present disclosure, the key text data is second key text data consisting of null characters generated in response to a randomly generated value corresponding to the training audio data being not less than a preset probability value.

[0013] In some exemplary embodiments of the present disclosure, the feature fusion of the first feature data and the second feature data through the initial feature fusion sub-model to obtain the third feature data includes: determining the similarity data corresponding to the first feature data and the second feature data through the initial feature fusion sub-model; and determining the third feature data based on the similarity data and the second feature data.

[0014] An embodiment of the present disclosure provides a speech recognition method, comprising: obtaining audio data to be recognized; inputting the audio data to be recognized into a speech recognition model to obtain recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to any of the above methods.

[0015] An embodiment of the present disclosure provides a training device for a speech recognition model, comprising: an acquisition module, configured to acquire training audio data; an encoding module, configured to encode the training audio data to obtain first feature data; a text module, configured to extract features of key text data through an initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined according to the annotated text data corresponding to the training audio data; a fusion module, configured to perform feature fusion on the first feature data and the second feature data through an initial feature fusion sub-model to obtain third feature data; a decoding module, configured to decode fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; a training module, configured to train the initial key text sub-model and the initial feature fusion sub-model based on the recognition text data and the annotated text data, so as to obtain the speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

[0016] An embodiment of the present disclosure provides a speech recognition device, comprising: an acquisition module, configured to acquire audio data to be recognized; a recognition module, configured to input the audio data to be recognized into a speech recognition model to obtain recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to any of the above methods.

[0017] An embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a storage device configured to store at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements a training method for a speech recognition model or a speech recognition method as described above.

[0018] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech recognition model training methods or speech recognition methods.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0021] FIG1 shows a schematic diagram of an exemplary system architecture to which a speech recognition model training method or a speech recognition method according to an embodiment of the present disclosure can be applied.

[0022] FIG2 is a flowchart of a method for training a speech recognition model in an exemplary embodiment of the present disclosure.

[0023] FIG3 is a schematic diagram of the structure of a speech recognition model in an exemplary embodiment of the present disclosure.

[0024] FIG4 is a flow chart of a speech recognition method in an exemplary embodiment of the present disclosure.

[0025] FIG5 is a block diagram of a training apparatus for a speech recognition model in an exemplary embodiment of the present disclosure.

[0026] FIG6 is a block diagram of a speech recognition apparatus in an exemplary embodiment of the present disclosure.

[0027] FIG7 is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0029] The features, structures or characteristics described in the present disclosure may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0030] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all content and steps, nor must they be executed in the order described. For example, some steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0032] In addition, in the description of the present disclosure, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element or component; the terms "comprising", "including" and "having" are used to express open-ended inclusion and mean that additional elements or components may exist in addition to the listed elements or components; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0033] FIG1 shows a schematic diagram of an exemplary system architecture to which a speech recognition model training method or a speech recognition method according to an embodiment of the present disclosure can be applied.

[0034] As shown in Figure 1, the system architecture may include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used as a medium for providing a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0035] Server 101 may be a server that provides various services, such as a background management server that supports devices operated by users using terminal device 103, terminal device 104, or terminal device 105. The background management server may analyze and process received data such as requests, and feed back the processing results to terminal device 103, terminal device 104, or terminal device 105.

[0036] Terminal device 103, terminal device 104 and terminal device 105 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, etc., but are not limited thereto.

[0037] In an embodiment of the present disclosure, the server 101 can: obtain training audio data; encode the training audio data to obtain first feature data; perform feature extraction on the key text data through an initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined based on the annotated text data corresponding to the training audio data; perform feature fusion on the first feature data and the second feature data through an initial feature fusion sub-model to obtain third feature data; decode the fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; train the initial key text sub-model and the initial feature fusion sub-model based on the recognition text data and the annotated text data, so as to obtain a speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

[0038] In an embodiment of the present disclosure, the server 101 may: obtain audio data to be recognized; input the audio data to be recognized into a speech recognition model to obtain recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to the above method.

[0039] It should be understood that the number of terminal devices 103, terminal devices 104, terminal devices 105, networks 102 and servers 101 in Figure 1 are merely schematic. Server 101 can be a physical server, a server cluster composed of multiple servers, or a cloud server. Depending on actual needs, it can have any number of terminal devices, networks and servers.

[0040] Below, each step of the method for training a speech recognition model in an exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments.

[0041] Figure 2 is a flow chart of a method for training a speech recognition model in an exemplary embodiment of the present disclosure. The method provided in the embodiment of the present disclosure can be executed by any electronic device with computing processing capabilities, such as the server or terminal device shown in Figure 1, but the present disclosure is not limited thereto.

[0042] As shown in FIG2 , the training method of the speech recognition model provided by the embodiment of the present disclosure may include the following steps.

[0043] In step S202, training audio data is obtained.

[0044] In the embodiment of the present disclosure, the training audio data can be a Chinese data set, and the audio sampling rate can be 16kHz. The training audio data obtained in each training process can be a training sample or a batch of training samples (batch): a total of N b training samples, and the training audio data in each batch of training samples is recorded as

[0045] In step S204, the training audio data is encoded to obtain first feature data.

[0046] In the embodiment of the present disclosure, referring to FIG3 , the speech recognition model 30 to be trained may include: a pre-trained Conformer-based ASR model (composed of an encoding network 301 and a decoding network 304, supporting general speech recognition), a key text sub-model 302, and a feature fusion sub-model 303. Among them, the encoding network 301 may be composed of 2 convolutional neural networks (CNN) and 12 standard Conformer modules, each of which may be 256-dimensional; the decoding network 304 may be composed of 6 layers of standard Transformer modules, each of which may also be 256-dimensional. During the training process of the speech recognition model 30, the above-mentioned training audio data may be Input into the encoding network 301 for encoding processing to obtain the first feature data Each of them may be an 80-dimensional Mel-cepstrum feature, the frame length may be 25ms, and the step length may be 10ms.

[0047] In step S206 , feature extraction is performed on the key text data using the initial key text sub-model to obtain second feature data.

[0048] In the embodiment of the present disclosure, the key text data may include text data determined according to the annotated text data corresponding to the training audio data.

[0049] In an exemplary embodiment, the key text data, as keyword data of the audio, can be the first key text data excerpted from the annotated text data according to a preset excerpt window in response to the randomly generated value corresponding to the training audio data being less than a preset probability value, or it can be the second key text data composed of empty characters generated in response to the randomly generated value corresponding to the training audio data being not less than a preset probability value.

[0050] Specifically, the above training audio data The corresponding annotation text data is Take the jth training sample x j For example, you can set a random generated value p between 0 and 1 j , when p j <Preset probability value (with random probability from the labeled text data y j When a keyword is selected from the preset probability value, for example, 60%, the preset selection window t j, from the labeled text data y j Randomly select multiple consecutive characters (ie, the first key text data) as keywords. Preset selection window t j The size of can be an integer between 2 and 6, which is used to simulate keyword scenarios of different lengths. j When the probability value is greater than or equal to the preset value, that is, a scenario in which there may not be any keywords in the audio of an actual application is simulated, second key text data consisting of a space character “\s” is generated as a keyword.

[0051] In the embodiment of the present disclosure, referring to FIG3 , the above training audio data can be trained by the (initial) key text sub-model 302 to be trained. The corresponding key text data is feature extracted to obtain the second feature data The key text sub-model 302 may adopt a network structure of an embedding layer and a 2-layer LSTM (Long Short-Term Memory) layer, and the LSTM may be 256-dimensional.

[0052] The disclosed embodiment adds a key text sub-model to the pre-trained Conformer-based ASR model. This model uses the full ASR training data and randomly selects keywords from it. Using the key text sub-model, the model generates word vectors for each keyword, enhancing the speech recognition model's ability to represent keywords and improving its recognition of keywords. Furthermore, by independently configuring different word vectors to enhance different keywords, the computational cost of repeatedly training the entire audio containing the keyword can be reduced.

[0053] In step S208, the first feature data and the second feature data are subjected to feature fusion by the initial feature fusion sub-model to obtain third feature data.

[0054] In the embodiment of the present disclosure, referring to FIG3 , the first feature data can be fused by the (initial) feature fusion sub-model 303 to be trained. and the second characteristic data Perform feature fusion to obtain the third feature data H = Attention(E,L,L). Here, Attention(q,k,v) can represent the standard attention mechanism function, and the three input variables represent query, key, and value respectively.

[0055] In an exemplary embodiment, the feature fusion sub-model 303 can fuse the key text sub-model 302 with the acoustic model encoding network 301, and obtain the audio features after the fused word vector based on the attention mechanism. Accordingly, the process of obtaining the third feature data in the above step S208 can further include: determining the similarity data corresponding to the first feature data E (as the input variable query) and the second feature data L (as the input variable key) through the above feature fusion sub-model 303, and then determining the third feature data H based on the above similarity data and the second feature data L (as the input variable value).

[0056] In step S210, the fourth feature data corresponding to the first feature data and the third feature data is decoded to obtain recognition text data corresponding to the training audio data.

[0057] In the disclosed embodiment, referring to FIG3 , during the training process of the speech recognition model 30, fourth feature data C corresponding to the first feature data E and the third feature data H can be input into the decoding network 304 for decoding processing to obtain recognized text data corresponding to the training audio data, i.e., the speech recognition result. The fourth feature data C can be obtained by summing the first feature data E and the third feature data H.

[0058] In step S212, based on the recognized text data and the annotated text data, an initial key text sub-model and an initial feature fusion sub-model are trained to obtain a speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

[0059] In the embodiment of the present disclosure, referring to FIG3 , during the training process of the speech recognition model 30, the encoding network 301 and the decoding network 304 are both pre-trained models, and their model parameters remain unchanged, while the key text sub-model 302 and the feature fusion sub-model 303 are the objects to be trained. In response to the completion of the training of the two, a complete speech recognition model 30 can be obtained.

[0060] In an exemplary embodiment, the model training process of the above-mentioned step S212 may include: determining the loss value corresponding to the recognized text data and the annotated text data based on a preset loss function; in response to the loss value not meeting the preset convergence condition, updating the first model parameters of the initial key text sub-model and the second model parameters of the initial feature fusion sub-model.

[0061] In an embodiment of the present disclosure, in response to the loss value satisfying the preset convergence condition, the first model parameters and the second model parameters currently updated are used as model parameters of the trained key text sub-model and the trained feature fusion sub-model respectively.

[0062] In the disclosed embodiment, the preset loss function can be a standard speech recognition loss function, such as a Connectionist Temporal Classification (CTC) loss function. The model parameters can be updated using a gradient backpropagation algorithm, such as an ADAM optimization algorithm based on a warm-up strategy, whose learning rate peak can be 0.001 and the warm-up strategy step size can be 1000 steps. The full amount of ASR training data can be used to iteratively train the key text sub-model and the feature fusion sub-model until the two models converge to obtain a keyword-enhanced speech recognition model.

[0063] The training method of the speech recognition model provided in the embodiment of the present disclosure fuses the key text sub-model with the acoustic model encoding network to train a keyword-enhanced speech recognition model, which solves the problem that the recognition results containing keywords are not among the recognition candidates. It can improve the recognition effect of keywords that conform to the long-tail distribution, for example, without affecting the performance of general speech recognition, thereby improving the overall user experience of the speech recognition system.

[0064] Figure 4 is a flow chart of a speech recognition method in an exemplary embodiment of the present disclosure. The method provided in the embodiment of the present disclosure can be executed by any electronic device with computing and processing capabilities, such as the server or terminal device shown in Figure 1, but the present disclosure is not limited thereto.

[0065] As shown in FIG4 , the speech recognition method provided by the embodiment of the present disclosure may include the following steps.

[0066] In step S402, audio data to be recognized is obtained.

[0067] In the embodiment of the present disclosure, the audio data to be recognized may be Chinese data, etc.

[0068] In step S404, the audio data to be recognized is input into the speech recognition model to obtain recognized text data corresponding to the audio data to be recognized.

[0069] In the embodiment of the present disclosure, the speech recognition model can be trained according to the speech recognition model training method provided in the above embodiment, with reference to the speech recognition model 30 shown in FIG3 .

[0070] The speech recognition method provided by the embodiments of the present disclosure uses a keyword-enhanced speech recognition model, which can accurately recognize keywords in audio and improve the user experience of the speech recognition system.

[0071] It should also be understood that the above is merely intended to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Based on the above examples, those skilled in the art can obviously make various equivalent modifications or variations. For example, certain steps in the above method may be unnecessary, or certain new steps may be added. Or any combination of any two or more of the above embodiments. Such modifications, variations, or combinations also fall within the scope of the embodiments of the present disclosure.

[0072] It should also be understood that the above description of the embodiments of the present disclosure focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0073] It should also be understood that the size of the sequence numbers of the above processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present disclosure.

[0074] It should also be understood that in the various embodiments of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments based on their internal logical relationships.

[0075] The above describes in detail the training method of the speech recognition model and the speech recognition method example provided by the present disclosure. It is understandable that, in order to implement the above functions, the computer device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0076] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0077] FIG5 is a block diagram of a training apparatus for a speech recognition model in an exemplary embodiment of the present disclosure.

[0078] As shown in FIG5 , the speech recognition model training device 500 may include: an acquisition module 502 , an encoding module 504 , a text module 506 , a fusion module 508 , a decoding module 510 and a training module 512 .

[0079] Among them, the acquisition module 502 is configured to acquire training audio data; the encoding module 504 is configured to encode the training audio data to obtain first feature data; the text module 506 is configured to extract features of the key text data through the initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined according to the annotated text data corresponding to the training audio data; the fusion module 508 is configured to perform feature fusion on the first feature data and the second feature data through the initial feature fusion sub-model to obtain third feature data; the decoding module 510 is configured to decode the fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; the training module 512 is configured to train the initial key text sub-model and the initial feature fusion sub-model based on the recognition text data and the annotated text data, so as to obtain a speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

[0080] In some exemplary embodiments of the present disclosure, the training module 512 is configured to: determine the loss value corresponding to the recognized text data and the annotated text data based on a preset loss function; and update the first model parameters of the initial key text sub-model and the second model parameters of the initial feature fusion sub-model in response to the loss value failing to meet the preset convergence condition.

[0081] In some exemplary embodiments of the present disclosure, the key text data is first key text data excerpted from the annotated text data according to a preset excerpt window in response to a randomly generated value corresponding to the training audio data being less than a preset probability value.

[0082] In some exemplary embodiments of the present disclosure, the key text data is second key text data consisting of null characters generated in response to a randomly generated value corresponding to the training audio data being not less than a preset probability value.

[0083] In some exemplary embodiments of the present disclosure, the fusion module 508 is configured to: determine similarity data corresponding to the first feature data and the second feature data through the initial feature fusion sub-model; and determine third feature data based on the similarity data and the second feature data.

[0084] FIG6 is a block diagram of a speech recognition apparatus in an exemplary embodiment of the present disclosure.

[0085] As shown in FIG. 6 , the speech recognition device 600 may include an acquisition module 602 and a recognition module 604 .

[0086] Among them, the acquisition module 602 is configured to obtain the audio data to be recognized; the recognition module 604 is configured to input the audio data to be recognized into the speech recognition model to obtain the recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to any of the above-mentioned speech recognition model training methods.

[0087] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor terminal devices and / or microcontroller terminal devices.

[0088] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0089] Figure 7 is a schematic diagram showing the structure of an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. It should be noted that the electronic device shown in Figure 7 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0090] As shown in Figure 7, electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in read-only memory (ROM) 702 or the program loaded from storage portion 708 into random access memory (RAM) 703. Various programs and data required for the operation of system 700 are also stored in RAM 703. CPU 701, ROM 702 and RAM 703 are connected to each other via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0091] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or a modem. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 710 as needed, so that computer programs read therefrom can be installed into the storage section 708 as needed.

[0092] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present disclosure are performed.

[0093] It should be noted that the computer-readable medium described in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor. For example, they may be described as follows: a processor includes a sending unit, an acquisition unit, a determination unit, and a first processing unit. The names of these units do not, in some cases, constitute a limitation on the units themselves. For example, the sending unit may also be described as a "unit that sends a picture acquisition request to the connected server."

[0096] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently and not be incorporated into the electronic device. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments. For example, the electronic device may implement the steps shown in Figures 2 and 4.

[0097] According to one aspect of the present disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the above-described embodiments.

[0098] It should be understood that any number of elements in the drawings of the present disclosure is for illustration only and not for limitation, and any naming is for distinction only and does not have any limiting meaning.

[0099] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing what is disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0100] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a speech recognition model, wherein: include: Get training audio data; Encoding the training audio data to obtain first feature data; Extracting features from key text data using the initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined based on the annotated text data corresponding to the training audio data; Performing feature fusion on the first feature data and the second feature data through an initial feature fusion sub-model to obtain third feature data; Decoding fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; Based on the recognized text data and the annotated text data, the initial key text sub-model and the initial feature fusion sub-model are trained to obtain the speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

2. The method according to claim 1, wherein The training of the initial key text sub-model and the initial feature fusion sub-model based on the recognized text data and the annotated text data includes: Determining the loss value corresponding to the recognized text data and the annotated text data based on a preset loss function; In response to the loss value failing to satisfy a preset convergence condition, the first model parameter of the initial key text sub-model and the second model parameter of the initial feature fusion sub-model are updated.

3. The method according to claim 1 or 2, wherein: The key text data is first key text data excerpted from the annotated text data according to a preset excerpt window in response to a randomly generated value corresponding to the training audio data being less than a preset probability value.

4. The method according to claim 1 or 2, wherein: The key text data is second key text data consisting of null characters generated in response to a randomly generated value corresponding to the training audio data being not less than a preset probability value.

5. The method according to claim 1 or 2, wherein: The step of fusing the first feature data and the second feature data by using the initial feature fusion sub-model to obtain the third feature data includes: using the initial feature fusion sub-model, determining similarity data corresponding to the first feature data and the second feature data; The third feature data is determined according to the similarity data and the second feature data.

6. A speech recognition method, wherein: include: Obtain audio data to be recognized; The audio data to be recognized is input into a speech recognition model to obtain recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to the method according to any one of claims 1 to 5.

7. A training device for a speech recognition model, wherein: include: an acquisition module, configured to acquire training audio data; an encoding module configured to encode the training audio data to obtain first feature data; a text module configured to perform feature extraction on key text data using an initial key text sub-model to obtain second feature data, wherein the key text data includes text data determined based on the annotated text data corresponding to the training audio data; a fusion module configured to perform feature fusion on the first feature data and the second feature data through an initial feature fusion sub-model to obtain third feature data; a decoding module configured to decode fourth feature data corresponding to the first feature data and the third feature data to obtain recognition text data corresponding to the training audio data; The training module is configured to train the initial key text sub-model and the initial feature fusion sub-model based on the recognition text data and the annotated text data, so as to obtain the speech recognition model based on the trained key text sub-model and the trained feature fusion sub-model.

8. A speech recognition device, wherein: include: An acquisition module, configured to acquire audio data to be recognized; The recognition module is configured to input the audio data to be recognized into a speech recognition model to obtain recognition text data corresponding to the audio data to be recognized, wherein the speech recognition model is trained according to the method according to any one of claims 1-5.

9. An electronic device, wherein: include: at least one processor; A storage device for storing at least one program, which, when executed by the at least one processor, enables the at least one processor to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images