Speech recognition model training method, device and system

Abstract

Embodiments of the present application relate to a speech recognition model training method, which comprises: a client obtaining text content in a target language, collecting a target language speech sample corresponding to the text content input by a user, storing the speech sample, and extracting speech features from the speech sample; in response to a user-triggered instruction to train a speech recognition model, obtaining first parameters of the speech recognition model from a server; the server sends the first parameters to the client in response to a request from the client; the client initializes the speech recognition model using the first parameters, takes the speech features as input and the text sequence as output, trains the speech recognition model, updates the parameters to obtain second parameters, and sends the second parameters to the server after completing the training; the server integrates the second parameters after receiving the second parameters and takes the integration result as the first parameters called by the client in the next round.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, in particular to a speech recognition model training method and device, and further relates to a speech recognition method and system, a computer readable storage medium, a federated speech recognition system and method. BACKGROUND

[0002] In speech recognition technology, it is difficult to collect speech data. In the existing technology, the main solution is to collect speech data in some real scenarios through a data provider, or to invite some speakers and simulate some dialogue scenarios, and then manually annotate the recordings. In these real or simulated scenarios (such as telephone scenarios, conference scenarios, etc.), the collected speech data involves content limited to these limited scenarios, and the process of inviting speakers to record is very costly. Due to the large number of languages, such as languages in various countries and local dialects, the number of people who can speak various languages is limited, which will limit the final recognition effect, and there is a risk of leaking data privacy. SUMMARY

[0003] In a first aspect, an embodiment of the present application provides a speech recognition model training method applied to a client, wherein the client stores a speech recognition model, and the method comprises: obtaining text content in a target language from a server; collecting a target language speech sample corresponding to the text content input by a user, storing the target language speech sample, and extracting speech features from the target language speech sample; in response to an instruction for training the speech recognition model triggered by the user, obtaining first parameters of the speech recognition model from the server, initializing the speech recognition model using the first parameters; training the speech recognition model by taking the speech features of the target language speech sample as input and taking a text sequence as output, and updating the first parameters to obtain second parameters; and after the training is completed, sending the second parameters to the server.

[0004] According to an embodiment of the present application, the first parameters and the second parameters comprise configuration parameters of the speech recognition model.

[0005] According to an embodiment of the present application, the speech features comprise Mel frequency spectrum features of the target language speech sample.

[0006] According to an embodiment of the present application, the method further comprises: after collecting the target language speech sample corresponding to the text content input by the user, checking whether the target language speech sample meets a preset condition; if the target language speech sample does not meet the preset condition, returning a prompt for re-entering the target language speech sample, collecting the target language speech sample re-entered by the user as the target language speech sample corresponding to the text content, and storing the target language speech sample.

[0007] According to an embodiment of the present application, the preset condition comprises: judging whether the number of words or the content in the target language speech sample and the text content is consistent; and / or judging whether the background noise in the target language speech sample exceeds a noise threshold.

[0008] According to an embodiment of the present application, the text content comprises a target language type and one or more text segments.

[0009] According to an embodiment of the present application, the number of words in the text segment does not exceed a preset word threshold.

[0010] According to an embodiment of the present application, the collecting of the speech sample corresponding to the text content input by the user comprises: collecting the target language speech sample corresponding to each text segment input by the user one by one, and only after the user completes the target language speech samples of all the text segments contained in the current text content, does the method respond to the instruction for training the speech recognition model on the client triggered by the user.

[0011] According to an embodiment of the present application, the speech recognition model is a Conformer model.

[0012] In a second aspect, an embodiment of the present application provides a method for training a speech recognition model, applied to a server, the method comprising: in response to a request of a client for calling a first parameter of a speech recognition model, sending the first parameter to the client; after receiving a second parameter returned by the client, integrating the second parameter and taking the integration result as the first parameter called by the next round of client request.

[0013] In a third aspect, an embodiment of the present application provides a speech recognition method, applied to a client, wherein the client stores a speech recognition model, and the method comprises:

[0014] The voice recognition model is trained on the client by: obtaining target language text content from the server; collecting a target language voice sample corresponding to the target language text content input by the user, storing the target language voice sample, and extracting voice features from the target language voice sample; in response to an instruction for training the voice recognition model triggered by the user, obtaining first parameters of the voice recognition model from the server, and initializing the voice recognition model using the first parameters; training the voice recognition model using the voice features of the target language voice sample as input and a text sequence as output, and updating the first parameters to obtain second parameters; and after the training is completed, sending the second parameters to the server.

[0015] The target language voice file to be recognized input by the user is received, and in response to an identification instruction triggered by the user, the voice recognition model after several rounds of training is used to identify the target language voice file to be recognized, and a voice recognition result corresponding to the target language voice file is obtained.

[0016] In a fourth aspect, an embodiment of the present application provides a voice recognition method applied to a server, the method comprising: in response to a request of a client for calling first parameters of a voice recognition model, sending the first parameters to the client; and after receiving second parameters returned by the client, integrating the second parameters and taking the integration result as the first parameters called by the client in the next round.

[0017] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method described above.

[0018] In a sixth aspect, an embodiment of the present application provides a device for training a speech recognition model, the device comprising a client and a server, the client storing a speech recognition model, the client being configured to: obtain text content in a target language from the server; collect a target language speech sample corresponding to the text content input by a user, store the target language speech sample, and extract speech features from the target language speech sample; in response to an instruction for training the speech recognition model triggered by the user, obtain first parameters of the speech recognition model from the server, initialize the speech recognition model using the first parameters; train the speech recognition model using the speech features of the target language speech sample as input and a text sequence as output, and update the first parameters to obtain second parameters; after the training is completed, send the second parameters to the server; and the server being configured to: in response to a request of the client for calling the first parameters of the speech recognition model, send the first parameters to the client; after receiving the second parameters returned by the client, integrate the second parameters and use the integration result as the first parameters called by the client in the next round.

[0019] In a seventh aspect, an embodiment of the present application provides a speech recognition system, the system comprising a server and a client, the server and the client interacting through a network, the server and the client being configured to train a speech recognition model, the speech recognition model being stored in the client; the client being configured to use the trained speech recognition model to recognize a speech file to be recognized, and obtain text corresponding to the speech file.

[0020] In an eighth aspect, an embodiment of the present application provides a federated speech recognition system, the system comprising a plurality of subsystems, the subsystems being the speech recognition system or the centralized training server as described above, the plurality of subsystems using the same speech recognition model and the same configuration, and the plurality of subsystems sharing the speech recognition model and parameters through the network.

[0021] In a ninth aspect, an embodiment of the present application provides a federated speech recognition method applied to the federated speech recognition system as described above, the federated speech recognition system comprising a plurality of subsystems and a cloud server, the subsystem being a speech recognition system or a centralized training server as described above, the federated speech recognition method comprising: the subsystem obtaining first parameters for training a speech recognition model from the cloud server, the first parameters being stored on the cloud server; the plurality of subsystems training the speech recognition model using the same speech recognition model and the same configuration; after the subsystem completes a round of training of the speech recognition model, sending second parameters to the cloud server; the cloud server integrating the second parameters and taking the integration result as the first parameters requested by the subsystem in the next round; using the speech recognition model after a plurality of rounds of training to recognize a speech file to be recognized to obtain a text corresponding to the speech file.

[0022] The above technical solution provided by the embodiments of the present application has at least some or all of the following advantages compared with the prior art:

[0023] In the manner of crowdsourcing, the requirement of speech input in the target language is published on an application program such as a mobile phone APP, a WeChat mini program and the like which is easy for users to obtain, the user extracts the crowdsourcing task in the manner of the user, the user reads the text in the target language according to the text displayed on the client, and the client collects the speech data in the target language corresponding to the text, a large amount of speech data is collected at a low cost by collecting the recordings from a large number of clients, and the training is directly performed on the client without manual annotation, thereby solving the problems of poor collection of speech data, high cost, limited number of speakers and the like affecting the training effect. In addition, by setting the text content to include various scenarios, the problem of too concentrated scenarios obtained from the data provider is avoided. The model is trained in the manner of federated learning to realize the iteration and update of the model, that is, the collected speech data is stored locally on the client, the training is performed on the client, and only the parameters are transmitted between the client and the server without transmitting the speech data, thereby protecting the data privacy, reducing the use of the server while realizing the iteration and update of the model, improving the training efficiency and reducing the training cost. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0026] Figure 1 The system architecture of the training method and device of the speech recognition model suitable for the embodiments of the present application is schematically shown.

[0027] Figure 2 The flowchart of the training method of the speech recognition model according to one embodiment of the present application is schematically shown.

[0028] Figure 3 The flowchart of the training method of the speech recognition model according to another embodiment of the present application is schematically shown.

[0029] Figure 4 The architecture of the training device of the speech recognition model according to still another embodiment of the present application is schematically shown.

[0030] Figure 5 The client application interface of the training method of the speech recognition model according to still another embodiment of the present application is schematically shown.

[0031] Figure 6 The architecture of the federated speech recognition system according to still another embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0032] In the related art, it is difficult to collect speech data, and usually a data provider collects real recordings and then manually labels them. The following two collection methods are mainly involved. The first collection method is to use the recordings collected by the data provider through its own channels. This collection method may cause the following problems: since the channels of the data provider are limited, the collected recordings are often concentrated in limited scenarios, such as collecting data in telephone scenarios, conference scenarios, and customer service scenarios of certain industries. Using this collection method, on the one hand, the content involved in the recorded speech data is concentrated in limited scenarios, and on the other hand, the speech data collected in these scenarios contains privacy-sensitive data. In addition to collecting speech and manual labeling, sensitive data needs to be deleted from the real data, that is, data desensitization operation. The second collection method is that the data provider invites some speakers who can speak the target language to simulate some dialogue scenarios, and then collects the speech data of the speakers. This method may cause the following problems: the process of inviting speakers who can speak the target language is very costly, and the number of speakers is often limited, which may limit the final recognition effect. In addition, the efficiency of manual labeling after collecting recording data is low, and the labor cost is high. Therefore, the above-mentioned methods of collecting recordings of the target language and then manually labeling them have the defects of low efficiency, high risk of privacy data leakage, few applicable scenarios of speech recognition content, high cost, and poor recognition effect.

[0033] In summary, how to collect speech data to avoid data privacy leakage and fully expand the applicable scenarios, and how to recognize speech content to improve the recognition efficiency and recognition effect of speech recognition and effectively reduce the cost have important practical value.

[0034] Therefore, embodiments of the present application provide a speech recognition model training method and device, and also provide a speech recognition method and system, a computer readable storage medium, a federated speech recognition system and method.

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0036] Figure 1 The system architecture of the speech recognition model training method and device suitable for the embodiments of the present application is schematically shown.

[0037] Reference Figure 1As shown, the system architecture 100 of the training method and device of the speech recognition model applicable to the embodiments of the present application comprises: terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can comprise various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0038] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can be installed with audio / video capture devices, picture / audio / video playing applications, etc. Other communication client applications can also be installed, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0039] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting voice capture, which can further comprise voice capture devices, such as electronic devices including but not limited to smart phones, tablet computers, notebook computers, desktop computers, etc.

[0040] The server 105 can be a server providing various services, such as a background management server providing service support for data processing of audio or video recorded by the user using the terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process received audio / video processing requests and other data, and feed back the processing results (such as web pages, information or data generated or obtained according to user requests, etc.) to the terminal devices.

[0041] It should be noted that the speech recognition method provided by the embodiments of the present application can generally be executed by the server 105 or a terminal device with certain computing power. Correspondingly, the speech recognition device provided by the embodiments of the present application can be generally arranged in the server 105 or the above-mentioned terminal device with certain computing power. The speech recognition method provided by the embodiments of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the speech recognition device provided by the embodiments of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the above-mentioned system architecture 100 is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0043] The first exemplary embodiment of the present application provides a speech recognition model training method.

[0044] Figure 2 The flowchart of the speech recognition model training method according to an embodiment of the present application is schematically shown.

[0045] Referring to Figure 2 The speech recognition model training method provided by the embodiments of the present application is applied to a client.

[0046] The client includes but is not limited to a mobile phone, a tablet computer, a desktop computer, etc. The embodiments of the present application can be applied to various clients. The speech recognition model can be stored on the client. The embodiments of the present application can adopt various types of speech recognition models, such as Conformer and other end-to-end structure models (Gulati A, Qin J, Chiu C C, et al. Conformer: Convolution-augmented transformer for speech recognition [J]. arXiv preprint arXiv:2005.08100, 2020). The client is installed with an application program that is easy for users to obtain to interact with the client, such as a mobile phone APP for collecting speech data and training a speech recognition model. Through the application program, the client can respond to the instructions triggered by the user to perform a series of operations.

[0047] The speech recognition model training method in this embodiment includes the following operations: S21, S22, S23, S24 and S25.

[0048] In operation S21, the client obtains text content in a target language from a server; the text content can include the type of the target language and one or more text segments, and the target language can be languages in the world and local dialects. For example, when the user opens the APP, the client obtains a plurality of text segments from the server, and the type of the target language in the text content can include multiple languages, for example, ten sentences are obtained, and the required language type for input is Chinese.

[0049] At operation S22, the client collects a target language speech sample corresponding to the text content input by the user, stores the target language speech sample, and extracts speech features from the target language speech sample; the user reads the text segment displayed by the client APP, and the client inputs the speech of the user reading the text segment, which is stored as a target language speech sample for training the speech recognition model in the local of the client, and extracts speech features from the target language speech sample, which can include Mel frequency spectrum features of the target language speech sample, for speech recognition.

[0050] At operation S23, the client responds to the instruction of training the speech recognition model triggered by the user, obtains the first parameter of the speech recognition model from the server, initializes the speech recognition model using the first parameter, which can be an initial parameter, inputs the first parameter and the target language speech sample into the speech recognition model for training; the first parameter and the second parameter include but are not limited to configuration parameters of the speech recognition model and / or gradient values for parameter updating.

[0051] When the user finishes the speech input of the text segment, an instruction for training the model can be issued on the APP, at which time the client starts to train the speech recognition model using the speech features extracted from the previously stored target language speech sample, requests the server to obtain the first parameter of the speech recognition model for initializing the model.

[0052] At operation S24, the speech features of the target language speech sample are used as input, for example, the extracted Mel frequency spectrum features of the speech sample are input into the speech recognition model, the text sequence is used as output, the speech recognition model is trained, and the first parameter is updated to obtain the second parameter; specifically, the first parameter can be updated to obtain the second parameter using various algorithms, such as gradient descent method (CTC loss function can also be used in the algorithm) and the like.

[0053] In operation S25, after the client completes a round of voice recognition model training, the first parameter of the voice recognition model is updated to the second parameter, and the second parameter is sent to the server. During the training process, the parameters of the model are updated, so after the training is completed, the second parameter output by the voice recognition model is the updated parameter, which is different from the first parameter. After the training is completed, the voice data is still stored in the client, and the client sends the second parameter to the server without including the voice data. Optionally, the voice data after the training is completed can be discarded, and the user can delete the voice data. Optionally, the instruction triggered by the user to train the voice recognition model can be set to trigger at least one round of training, for example, it can be set to use the currently collected voice data of the target language to train the model for two rounds, that is, after the user triggers the training, the client automatically completes two rounds of training, and the number of times of training performed by the client after the user triggers the training can be flexibly set according to the actual application scenario.

[0054] In this embodiment, the training of the voice recognition model is performed on the client, which belongs to on-device training. During the training process, the client only transmits parameters to the server without transmitting voice data, and the voice data is only stored in the client. The client obtains parameters from the server without including voice data. On the one hand, data privacy is protected, and on the other hand, the use of the server is reduced, the training efficiency is improved, and the training cost is reduced.

[0055] A second exemplary embodiment of the present application provides a voice recognition model training method.

[0056] Figure 3 A flowchart of a voice recognition model training method according to another embodiment of the present application is schematically shown.

[0057] Referring to Figure 3 The voice recognition model training method provided by the embodiments of the present application is applied to a server and includes the following operations: S31 and S32.

[0058] In operation S31, the server sends the first parameter to the client in response to a request of the client to call the first parameter of the voice recognition model. The parameters of the voice recognition model are stored on the server, and when the client requests to call the first parameter of the voice recognition model, the server sends the first parameter to the client in response to the request.

[0059] In operation S32, the server, after receiving the second parameter returned by the client, takes the second parameter as the first parameter of the next round of client request calling; when the client completes a round of training, the server receives the second parameter returned by the client, integrates the second parameter, and takes the integration result as the first parameter of the next round of client request calling. The server can receive second parameters returned by multiple clients, and the integration of the second parameters can be integration of multiple second parameters returned by multiple clients. The integration of the second parameters can be revision of data to ensure accuracy, for example, taking an average. When the client performs the next round of training request calling parameter, the integration result of the integrated multiple second parameters is taken as the first parameter of the next round of client request calling.

[0060] A third exemplary embodiment of the present application provides a speech recognition method. Applied to a client, the client stores a speech recognition model, and the method comprises:

[0061] The speech recognition model is trained on the client by the following operations: obtaining target language text content from a server; collecting a target language speech sample corresponding to the target language text content input by a user, storing the target language speech sample, and extracting speech features from the target language speech sample; in response to an instruction for training the speech recognition model triggered by the user, obtaining a first parameter of the speech recognition model from the server, initializing the speech recognition model using the first parameter; training the speech recognition model using the speech features of the target language speech sample as input and a text sequence as output, and updating the first parameter to obtain a second parameter; after training is completed, sending the second parameter to the server;

[0062] Receiving a target language speech file to be recognized input by a user, in response to a recognition instruction triggered by the user, using the speech recognition model after several rounds of training to recognize the target language speech file to be recognized, and obtaining a speech recognition result corresponding to the target language speech file.

[0063] A fourth exemplary embodiment of the present application provides a speech recognition method. Applied to a server, the method comprises: in response to a request of a client to call a first parameter of a speech recognition model, sending the first parameter to the client; after receiving a second parameter returned by the client, integrating the second parameter and taking the integration result as the first parameter of the next round of client request calling.

[0064] A fifth exemplary embodiment of the present application provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the method provided in the foregoing exemplary embodiments, which will not be described herein.

[0065] The sixth exemplary embodiment of the present application provides a training device of a speech recognition model.

[0066] Figure 4 The architecture diagram of the training device of the speech recognition model according to another embodiment of the present application is schematically shown. Figure 5 The application interface of the client of the training device of the speech recognition model according to another embodiment of the present application is schematically shown. The implementation process of the present embodiment is described in detail below in combination with Figure 4 and Figure 5 .

[0067] As shown in Figure 4 , in the present embodiment, the training device of the speech recognition model includes two types of participants, i.e. the client and the server, and the device can be a subsystem, such as enterprise A in the figure, which includes server A and multiple clients A1, A2,..., An (i.e. the number of clients is i, i = 1, 2,..., N). For example, in the application scenario of collecting target language speech data in the manner of crowdsourcing, the users participating in the crowdsourcing can collect and train the model through the mobile APP installed on the client. The speech recognition model can be trained simultaneously on the N clients, and the parameter updating and transmission can be performed between the N clients and the server A. The information interaction between the client and the server can adopt the method of horizontal federated learning, which belongs to the horizontal federated learning category. The combination of multiple clients can make the training sample size larger, thereby benefiting the improvement of the training efficiency and obtaining better training effect. N

[0068] In combination with the interface shown in Figure 5 , the user opens the APP on the client, and the client displays the interface. The client obtains the text content from the server, and the text content includes the target language type and multiple text segments. The interface of the APP can include multiple functional modules, such as the text content display module, the recording module, the feature extraction module, the training module, the text segment selection module, the speech data verification module, and the target language type selection module.

[0069] Taking the target language type as Chinese and various Chinese dialects as examples, Figure 5 ​The middle interface displays a text segment "Today is a good day", which is realized by the text content display module. In this embodiment, multiple text segments can be displayed one by one instead of simultaneously. Alternatively, the target language type selection module can be used to realize this. The user can input some additional information, such as the country or region to which the language to be entered belongs, and further select the dialect of the region. For example, the user can be prompted to select the target language type while a current text segment is displayed. After the user selects a target language (for example, Chinese, and further selects a dialect, such as Cantonese), the client starts recording after the user clicks the recording button. At this time, the user reads "Today is a good day" in Cantonese, and the client collects the recording of the user reading in Cantonese. During this process, the recording button triggers the recording module, which realizes the collection of the target language voice data entered by the user through the recording module. The client collects the voice of the user as the target language voice sample corresponding to the text segment, stores the target language voice sample in the client, and extracts the voice features, such as Mel frequency spectrum features, from the target language voice sample through the feature extraction module.

[0070] In this embodiment, the voice data verification module can also be provided. The voice data verification module can be set to automatically trigger the verification of the target language voice sample through the verification module after the recording module collects the target language voice sample to determine whether the target language voice sample meets the preset condition. If the target language voice sample does not meet the preset condition, the user can be prompted to re-enter the voice sample and overwrite the original target language voice sample that does not meet the preset condition (unqualified). The verification function can also be set according to the actual application scenario to realize a variety of other functions. For example, the voice data verification module can be set to allow the user to skip the current text segment if the user re-enters multiple times but still does not meet the preset condition. For another example, the voice data verification module can be set to allow the user to record the next text segment only after the target language voice sample of the current text segment meets the preset condition (qualified). When the target language is a multi-dialect of a country or region, for example, the target language is a dialect of a certain region in China instead of Mandarin, the voice data verification module can also be set to verify heavy dialect data, that is, dialects that differ greatly from Mandarin. This can be realized by setting the algorithm used by the module.

[0071] The above checking whether the voice sample meets the preset condition can include, for example, judging whether the number of words and the content in the target language voice sample and the current text segment are consistent, if consistent, the target language voice sample is qualified, if inconsistent, the target language voice sample is unqualified, for example, it can also be judged whether the background noise in the target language voice sample exceeds the threshold, a noise threshold is set, if the background noise in the current target language voice sample does not exceed the threshold, that is, the noise is not enough to affect the quality of the voice sample, the above target language voice sample is qualified, if it exceeds the threshold, that is, the background noise is large, which causes the sound quality of the target language voice sample to be unclear, which cannot be used for training the voice recognition model, the target language voice sample is unqualified. The dimensions of the target language voice sample checking include but are not limited to the above dimensions, and the dimensions, standards, etc. of the checking can also be set according to the actual application scenario.

[0072] Figure 5 The "previous" and "next" buttons displayed on the interface can trigger the text segment selection module to select multiple text segments in the current text content. Optionally, the buttons displayed on the interface can also be set in the form of a list or other forms, and any one text segment can be selected in the list of all text segments displayed. The text segment selection module is used to enable the user to freely select the text segment that is currently prepared to be recorded in voice, thereby facilitating the re-recording of voice, updating of voice data, etc.

[0073] In the embodiment, the length of the text segment of the text content can also be limited to not exceed a preset word threshold, which can be, for example, 30 words, 20 words, etc. The number limit can be adjusted to a suitable amount for the actual application scenario during implementation.

[0074] In the scenario of inviting users to record target language voice in a crowdsourcing manner, users can actively obtain text content to be read aloud for recording through an APP. For example, a user obtains a voice recording task, the text content in the task includes multiple text segments (for example, 20 text segments), after the user selects the language type (the language type of the target language) used by himself / herself, the user records the target language voice corresponding to the 20 text segments one by one, and after completing the voice recording of all the text segments included in the voice recording task, triggers a model training instruction on the APP. Figure 5As shown, the user can trigger the training module to train the model by clicking the "Start Training" button on the interface, that is, the client starts training the speech recognition model on the client in response to the user's instruction to train the speech recognition model. The training module can be set so that the user can start training only after clicking "Start Training", thereby enabling the user to control the use of his own device. The training module can also be set so that the client collects target language speech samples corresponding to each text segment entered by the user one by one, and only after the user has completed all target language speech samples for the text segments contained in the current text content, does the client respond to the user's instruction to train the speech recognition model on the client, thereby starting the training.

[0075] After starting the training, the client sends a request to the server to obtain the first parameters of the speech recognition model, the server responds to the client's request to call the first parameters of the speech recognition model, and sends the first parameters to the client. After the client obtains the first parameters of the speech recognition model, the client initializes the speech recognition model using the first parameters, and then trains the speech recognition model using the speech features extracted from the target language speech samples as input and the text sequence as output. The parameters of the speech recognition model change, and the second parameters are updated from the first parameters. The parameter update can be performed using the gradient descent method (which can also use the CTC loss function). The parameters include but are not limited to the configuration parameters of the speech recognition model. After the training is completed, the client sends the second parameters to the server. After the server receives the second parameters returned by the client, the server integrates the second parameters returned by other clients, and uses the integrated results as the first parameters called by the client in the next round. In this embodiment, the server end mainly serves as the server for training the speech recognition model, and updates and integrates the model parameters through algorithms, such as the commonly used parallel gradient descent method, Federated Averaging algorithm, etc.

[0076] The seventh exemplary embodiment of the present application provides a speech recognition system. The system includes a server and a client, the server and the client interact through a network, the server and the client are used to train a speech recognition model, and the speech recognition model is stored in the client; the client is used to recognize a speech file to be recognized using the trained speech recognition model to obtain a text corresponding to the speech file. For example, when the model is trained and optimized, an inference module can be set on the client to recognize the speech file using the model and infer the text corresponding to the speech file. In specific application scenarios, as the optimization degree of the speech recognition model improves, the accuracy of the inference result will be higher.

[0077] The eighth exemplary embodiment of the present application provides a federated speech recognition system.

[0078] Figure 6 The architecture of a federal speech recognition system according to yet another embodiment of the present invention is illustrated schematically.

[0079] The federal speech recognition system includes several subsystems, which are the speech recognition systems or centralized training servers described in the aforementioned embodiments. The subsystems use the same speech recognition model and the same configuration, and the speech recognition model is trained and parameters are shared among the subsystems via a network.

[0080] Figure 6 Company A and Company B in the text have similar structures and both belong to the category of... Figure 4 The training device for the speech recognition model described in the foregoing embodiments can be viewed as a traditional centralized model training structure, where subsystems A and B, and enterprise C, are not distributed structures, and their internal structures differ from those of enterprises A and B. The federated speech recognition system of this embodiment is compatible with both traditional centralized model training structures. When communicating between different enterprises, a federated learning model training architecture can be used to achieve model parameter sharing between enterprises. This inter-enterprise parameter sharing method also falls under the category of horizontal federated learning.

[0081] This embodiment employs hierarchical federated learning, by... Figure 6 It can be seen that federated learning frameworks (e.g., FATE, Yang Q, Liu Y, Cheng Y, et al. Federated learning[J]. Synthesis Lectureson Artificial Intelligence and Machine Learning, 2019, 13(3): 1-207.) are applied at both levels. Federated learning frameworks can be used for overall system communication, encryption, storage, client maintenance, etc. Figure 4 and Figure 6 These two levels are, respectively, the internal level of the enterprise and the inter-enterprise level. Figure 4 The federated learning framework is applied in each subsystem. Figure 6 This paper demonstrates the application of a federated learning framework among the subsystems in a federated speech recognition system comprising several subsystems, with parameter exchange between the servers of each subsystem. In this federated speech recognition system, parameter updates, integration, and encryption algorithms between the server and client can be reused in the training process at both levels.

[0082] The ninth exemplary embodiment of the present application provides a federated speech recognition system, which can include a plurality of speech recognition systems and / or a plurality of centralized training servers, the speech recognition systems and the centralized training servers adopt the same speech recognition model and the same configuration, and the plurality of speech recognition systems and the centralized training servers perform training of the speech recognition model and parameter sharing through a network.

[0083] In the federated speech recognition system of the present embodiment, by setting a plurality of subsystems to adopt the same speech recognition model and the same configuration, model parameter sharing among a plurality of enterprises can be achieved. The parameter sharing among enterprises belongs to the category of horizontal federated learning. For example, in a system including a plurality of enterprises, each subsystem can collect audio data displayed on a plurality of clients. When the plurality of subsystems participate in model training and speech recognition, the number of users participating in voice data entry is greatly increased, further expanding the number of samples, thereby more conducive to improving the accuracy and training efficiency of model training.

[0084] The tenth exemplary embodiment of the present application provides a federated speech recognition method applied to the above-mentioned federated speech recognition system, the federated speech recognition system includes a plurality of subsystems and a cloud server, and the subsystem is the speech recognition system or the centralized training server. The subsystem obtains first parameters for training the speech recognition model from the cloud server, and the first parameters are stored on the cloud server; the plurality of subsystems adopt the same speech recognition model and the same configuration to train the speech recognition model; after the subsystem completes a round of training of the speech recognition model, the subsystem sends second parameters to the cloud server;

[0085] The cloud server integrates the second parameters and uses the integration result as the first parameters requested by the subsystem in the next round; and the speech recognition model after a plurality of rounds of training is used to recognize a target language voice file to be recognized, and a text corresponding to the target language voice file is obtained.

[0086] It should be noted that in the above-mentioned embodiments of the present application, when using a mobile phone as a client for on-device training, compared with using a computer as a client, there is a problem that the computing resources of the mobile phone are not as powerful as the GPU of the computer. According to the calculation, on the GPU, it takes about 1 second to complete a training (for example, a step) for about 3 seconds of audio. On the CPU of the mobile phone client, the process may take 3-4 seconds. Although there is the above-mentioned problem of training on the mobile phone, in actual application scenarios, the time consumption of on-device training on the mobile phone client is still acceptable and has wide applicability. For example, training 30 minutes of audio on the mobile phone client, the total time for completing a round of training (for example, an epoch) is about 40 minutes. Therefore, even before the voice recognition model is trained and optimized, for the audio commonly used for training on the mobile phone client, which is about half an hour to an hour in total length, the time consumption of training on the mobile phone client is still acceptable. Further, after using the training method of the voice recognition model of the present application to continuously train, iterate, and optimize the model, the total time for completing a round of training (for example, an epoch) will be shortened accordingly. For example, using the optimized model to train 30 minutes of audio on the mobile phone client, the total time for completing a round of training is about 30 minutes, which improves the efficiency. In addition, the model training process can be set to run in the background, that is, after the client collects the user's input of voice data in the target language, the user triggers the client to train, and even if the user exits the application to use other functions, the model training process is still running in the background. Even if the time consumption of a round of training is about 30-40 minutes, it will not affect the user's use of other functions of the mobile phone. Therefore, when using the mobile phone to implement the technical solutions of the above-mentioned embodiments of the present application in actual application scenarios, the training efficiency can also be improved, the training cost can be reduced, the recognition efficiency and recognition effect of voice recognition can be improved, data privacy can be protected, and all the advantages and beneficial effects of the technical solutions provided by the embodiments of the present application are possessed.

[0087] The embodiment of the present application adopts a crowdsourcing manner, publishes the requirement of voice input of the target language on an application program such as a mobile phone APP, a WeChat applet and the like which is easy for users to obtain, extracts the crowdsourcing task by the user, the user reads the text in the target language which can be used by the user according to the text displayed on the client, and the voice data of the target language corresponding to the text is collected by the client, a large amount of voice data collection is completed at a low cost by collecting a large amount of voice from a large number of clients, and training is directly performed on the client without manual annotation, thereby solving the problems of poor voice data collection, high cost, limited number of speakers and influence on training effect. In addition, by setting the text content to include various scenes, the problem of too concentrated scenes obtained from a data provider is avoided. The model is trained in a federated learning manner to realize model iteration and update, that is, the collected voice data is stored in the local client, training is performed on the client, and only parameters are transmitted between the client and the server without transmitting voice data, thereby protecting data privacy, reducing the use of the server while realizing model iteration and update, improving training efficiency and reducing training cost

[0088] In this document, relational terms such as“first” and“second”, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms“comprises”,“comprising”, or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by“comprises a...” does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0089] The above description is merely that of the specific embodiments of the present application, making those skilled in the art understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a speech recognition model, applied to a client, wherein the client stores a speech recognition model, characterized in that, include: The text content of the target language is obtained from the server; wherein, the speech recognition system to which the client belongs is a subsystem of the federated speech recognition system, the subsystem includes the server and the client, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, the several subsystems train the speech recognition model and share parameters through the network, and the server and the client interact through the network; Collect target language speech samples corresponding to text content entered by users, store the target language speech samples, and extract speech features from the target language speech samples; In response to a user-triggered instruction to train a speech recognition model, the system obtains the first parameters of the speech recognition model from the server and initializes the speech recognition model using the first parameters. The speech recognition model is trained by taking the speech features of the target language speech sample as input and the text sequence as output, and the first parameter is updated to obtain the second parameter. After training is complete, the second parameter is sent to the server.

2. The method according to claim 1, characterized in that, The first parameter and the second parameter include the configuration parameters of the speech recognition model.

3. The method according to claim 1, characterized in that, The speech features include the Mel spectrum features of the target language speech samples.

4. The method according to claim 1, characterized in that, The method further includes: After collecting target language speech samples corresponding to the text content entered by the user, it is checked whether the target language speech samples meet preset conditions. If the target language speech sample does not meet the preset conditions, a prompt to re-enter the target language speech sample will be returned. The re-entered target language speech sample will be collected as the target language speech sample corresponding to the text content and stored.

5. The method according to claim 4, characterized in that, The preset conditions include: Determine whether the number of words or content in the target language speech sample and the text content are consistent; and / or Determine whether the background noise in the target language speech sample exceeds the noise threshold.

6. The method according to any one of claims 1-4, characterized in that, The text content includes the target language type and one or more text segments.

7. The method according to claim 6, characterized in that, The number of characters in the text segment does not exceed a preset character threshold.

8. The method according to claim 6, characterized in that, The collected target language speech samples corresponding to the text content entered by the user include: The system collects target language speech samples for each text segment entered by the user. Only after the user has completed the collection of target language speech samples for all text segments contained in the current text content will it respond to the user's instruction to train the speech recognition model on the client.

9. The method according to claim 1, characterized in that, The speech recognition model is the Conformer model.

10. A method for training a speech recognition model, applied to a server, characterized in that, include: In response to a client's request to call the first parameter of the speech recognition model, the first parameter is sent to the client; wherein, the speech recognition system to which the client belongs is a subsystem of a federated speech recognition system, the subsystem includes the server and the client, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, the several subsystems train the speech recognition model and share parameters through the network, and the server and the client interact through the network; After receiving the second parameter returned by the client, the second parameter is integrated and the integrated result is used as the first parameter for the next round of client request calls.

11. A speech recognition method, applied to a client, wherein the client stores a speech recognition model, characterized in that, include: The speech recognition model is trained on the client side using the following steps. The target language text content is obtained from the server; wherein, the speech recognition system to which the client belongs is a subsystem of the federated speech recognition system, the subsystem includes the server and the client, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, the several subsystems train the speech recognition model and share parameters through the network, and the server and the client interact through the network; Collect target language speech samples corresponding to target language text content entered by the user, store the target language speech samples, and extract speech features from the target language speech samples; In response to a user-triggered instruction to train a speech recognition model, the system obtains the first parameters of the speech recognition model from the server and initializes the speech recognition model using the first parameters. The speech recognition model is trained by taking the speech features of the target language speech sample as input and the text sequence as output, and the first parameter is updated to obtain the second parameter. After training is complete, the second parameter is sent to the server; The system receives a target language speech file input by the user, responds to the user-triggered recognition command, and uses a speech recognition model that has completed several rounds of training to recognize the target language speech file, thereby obtaining a speech recognition result corresponding to the target language speech file.

12. A speech recognition method applied to a server, characterized in that, include: In response to a client's request to call the first parameter of the speech recognition model, the first parameter is sent to the client; wherein, the speech recognition system to which the client belongs is a subsystem of a federated speech recognition system, the subsystem includes the server and the client, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, the several subsystems train the speech recognition model and share parameters through the network, and the server and the client interact through the network; After receiving the second parameter returned by the client, the second parameter is integrated and the integrated result is used as the first parameter for the next round of client request calls.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9, or claim 10, 11, or 12.

14. A training device for a speech recognition model, the device comprising a client and a server, wherein the client stores a speech recognition model, characterized in that: The client is used to perform the following operations. The text content of the target language is obtained from the server; wherein, the speech recognition system to which the client belongs is a subsystem of the federated speech recognition system, the subsystem includes the server and the client, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, the several subsystems train the speech recognition model and share parameters through the network, and the server and the client interact through the network; Collect target language speech samples corresponding to text content entered by users, store the target language speech samples, and extract speech features from the target language speech samples; In response to a user-triggered instruction to train a speech recognition model, the system obtains the first parameters of the speech recognition model from the server and initializes the speech recognition model using the first parameters. The speech recognition model is trained by taking the speech features of the target language speech sample as input and the text sequence as output, and the first parameter is updated to obtain the second parameter. After training is complete, the second parameter is sent to the server; The server is used to perform the following operations. In response to a client's request to access the first parameter of the speech recognition model, the first parameter is sent to the client; After receiving the second parameter returned by the client, the second parameter is integrated and the integrated result is used as the first parameter for the next round of client request calls.

15. A speech recognition system, the system comprising a server and a client, the server and client interacting via a network, characterized in that: The server and client are used to train a speech recognition model according to the training method of the speech recognition model according to any one of claims 1-10, and the speech recognition model is stored in the client; wherein, the speech recognition system is a subsystem of a federated speech recognition system, the federated speech recognition system includes several subsystems, the several subsystems use the same speech recognition model and the same configuration, and the several subsystems train the speech recognition model and share parameters through a network; The client is used to recognize the speech file to be recognized using the trained speech recognition model, and obtain the text corresponding to the speech file.

16. A federated speech recognition system, characterized in that: It includes several subsystems, which are the speech recognition system as described in claim 15. The several subsystems use the same speech recognition model and the same configuration, and the several subsystems train the speech recognition model and share parameters through a network.

17. A federated speech recognition method, applied to the federated speech recognition system as described in claim 16, wherein the federated speech recognition system comprises a plurality of subsystems and a cloud server, wherein the subsystems are the speech recognition system as described in claim 15, characterized in that: The subsystem obtains first parameters for training the speech recognition model from the cloud server, and the first parameters are stored on the cloud server. Several of the aforementioned subsystems train the speech recognition model using the same speech recognition model and the same configuration; After completing one round of training of the speech recognition model, the subsystem sends the second parameter to the cloud server. The cloud server integrates the second parameter and uses the integration result as the first parameter for the next round of subsystem request invocation; Using a speech recognition model that has completed several rounds of training, the speech file to be recognized is identified to obtain the text corresponding to the speech file.

Images