A speech recognition method, a model training method, and an electronic device

Abstract

Embodiments of the present application apply to the technical field of speech recognition, and provide a speech recognition method, a model training method and an electronic device. In the case that the electronic device collects original audio data, the electronic device can perform noise reduction processing on the original audio data to obtain noise-reduced audio data. The original audio data is obtained based on a speech instruction issued by a user. Then, the electronic device can merge the noise-reduced audio data and the original audio data according to a target compensation coefficient to obtain target audio data. Then, the electronic device can perform speech recognition on the target audio data to obtain a speech recognition result. Then, in response to the speech recognition result, the electronic device performs an operation corresponding to the speech instruction. In the present application, accurate recognition of the speech instruction is achieved.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method, a model training method, and an electronic device. Background Technology

[0002] With the continuous development of voice recognition technology, electronic devices (such as mobile phones) can perform corresponding actions according to the voice commands issued by users. For example, if a user issues the voice command "Hello Youyou, turn on the air conditioner", the mobile phone can control the air conditioner to turn on according to the voice command, so as to realize the interaction between the user and the electronic device.

[0003] However, if users are in noisy environments (such as shopping malls with loud noise), environmental noise may interfere with the voice recognition capabilities of electronic devices, making it impossible for the electronic devices to accurately recognize voice commands. This can result in the electronic devices not responding or responding incorrectly, thus affecting the user experience. Summary of the Invention

[0004] This application provides a speech recognition method, a model training method, and an electronic device for achieving accurate recognition of speech commands and improving the accuracy of speech recognition.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a speech recognition method is provided for use in an electronic device. In this method, the electronic device, upon acquiring raw audio data, performs noise reduction processing on the raw audio data to obtain noise-reduced audio data. The raw audio data is obtained based on a user's voice command. Then, the electronic device merges the noise-reduced audio data with the raw audio data according to a target compensation coefficient to obtain target audio data. Next, the electronic device performs speech recognition on the target audio data to obtain a speech recognition result. Finally, in response to the speech recognition result, the electronic device executes the operation corresponding to the voice command.

[0007] In this application, by merging the noise-reduced audio data with the original audio data, accurate compensation of the audio data can be achieved. This not only reduces the probability of nonlinear distortion during audio noise reduction, but also compensates for the audio data lost after noise reduction, making the target audio data essentially the same as the user audio data in the original audio data. This minimizes the loss of user audio data, thereby improving the accuracy of speech recognition, improving the accuracy of electronic device action execution, and ultimately enhancing the user's interactive experience.

[0008] In one possible implementation of the first aspect, the process of the electronic device performing noise reduction processing on the original audio data may specifically include: the electronic device performing noise reduction processing on the original audio data according to the target audio noise reduction model to obtain noise-reduced audio data;

[0009] The process by which the aforementioned electronic device performs speech recognition on target audio data can specifically include: the electronic device can perform speech recognition on the target audio data according to a target speech recognition model to obtain a speech recognition result. The target speech recognition model is obtained through joint training of a target audio noise reduction model.

[0010] In this application, considering that the mobile phone trains the aforementioned audio noise reduction model and the aforementioned speech recognition model separately, although it can perform audio noise reduction using the trained audio noise reduction model and speech recognition using the trained speech recognition model, if the original audio data still contains a small amount of noise after audio noise reduction, and the speech recognition model has not been trained for this noisy data, the accuracy of the mobile phone's audio noise reduction will affect the subsequent speech recognition results. This will cause the speech recognition performance of the speech recognition model to suddenly deteriorate, making it unable to accurately recognize voice commands and ultimately affecting the user's interactive experience. Therefore, to improve the mobile phone's speech recognition performance in noisy scenarios, electronic devices can jointly train the audio noise reduction model and the speech recognition model. In this way, by jointly training the two models, the recognition performance of the mobile phone can be ensured to the greatest extent, which can reduce the occurrence of inaccurate speech recognition results due to low audio quality after noise reduction, improve the accuracy of speech recognition, and thus improve the user's interactive experience.

[0011] In one possible implementation of the first aspect, before the electronic device jointly trains the audio denoising model and the speech recognition model, the method further includes: the electronic device can initialize the training of the pre-built audio denoising model and the pre-built speech recognition model, and obtain the audio denoising model to be trained and the speech recognition model to be trained when the audio denoising model reaches the first training standard and the speech recognition model reaches the second training standard.

[0012] The first training criterion is used to keep the first loss value stable. The second training criterion is used to keep the second loss value stable.

[0013] In this application, since the audio denoising model to be trained has reached the first training standard (i.e., it has achieved a certain denoising capability) and the speech recognition model to be trained has reached the second training standard (i.e., it has achieved a certain speech recognition capability), if the learning rate (or training capability) of the audio denoising model and the speech recognition model is not reduced, the two models may experience training collapse during subsequent joint training. This could lead to inaccurate audio denoising results and / or speech recognition results, ultimately affecting the training effect of the models. Therefore, to avoid training collapse, the electronic device can reduce the learning rate of the audio denoising model and the speech recognition model to be trained after obtaining them. Then, the electronic device performs joint training on these two models to obtain the target audio denoising model and the target speech recognition model. This improves the training effect while maintaining the generalization ability of the models, thereby enhancing the speech recognition performance of the electronic device in noisy scenarios and laying the foundation for accurate speech command recognition.

[0014] In another possible implementation of the first aspect, before the electronic device jointly trains the audio denoising model and the speech recognition model, the method further includes: the electronic device can initialize the training of the pre-built audio denoising model, and when the audio denoising model reaches the first training standard, obtain the audio denoising model to be trained, and initialize the weights of the pre-built speech recognition model to obtain the speech recognition model to be trained.

[0015] In this application, since the audio denoising model to be trained has already reached the first training standard, meaning it has achieved a certain denoising capability, not reducing its learning rate could lead to training failure during subsequent joint training, potentially resulting in inaccurate audio denoising results and affecting the training effect. Therefore, to avoid training failure, the electronic device can reduce the learning rate of the audio denoising model after obtaining it. Then, the electronic device performs joint training on both the audio denoising model and the speech recognition model to obtain the target audio denoising model and the target speech recognition model. This approach improves the model's training effect while maintaining its generalization ability, thereby enhancing the electronic device's speech recognition performance in noisy scenarios and laying the foundation for accurate speech command recognition.

[0016] In one possible implementation of the first aspect, the joint training process of the audio denoising model and the speech recognition model may specifically include: an electronic device acquiring an audio dataset. The audio dataset includes a first sub-audio dataset and a second sub-audio dataset. The first sub-audio dataset includes multiple noise-free frequency data points, and the second sub-audio dataset includes multiple noisy frequency data points, which are obtained by mixing noise from the noise-free frequency data. Then, for each noisy frequency data point in the second sub-audio dataset, the electronic device can input the noisy frequency data into the audio denoising model to be trained, obtaining denoised noisy frequency data. Next, the electronic device can train the audio denoising model to be trained based on the denoised noisy frequency data and the corresponding noise-free frequency data, obtaining a target audio denoising model. Then, the electronic device can input the denoised noisy frequency data into the speech recognition model to be trained, obtaining predicted text data. Finally, the electronic device can train the speech recognition model to be trained based on the predicted text data and the corresponding real text data from the noise-free frequency data, obtaining a target speech recognition model.

[0017] In this application, the audio noise reduction model and the speech recognition model are jointly trained by inputting the noise-reduced audio data output by the audio noise reduction model to be trained into the audio noise reduction model to be trained. This can ensure the recognition performance of the mobile phone to the greatest extent, that is, reduce the occurrence of inaccurate speech recognition results due to low audio quality after noise reduction, improve the accuracy of speech recognition, and thus improve the user's interactive experience.

[0018] In one possible implementation of the first aspect, the process of training the audio denoising model to be trained by the aforementioned electronic device may specifically include: the electronic device determining a first loss value based on the denoised noisy frequency data and the corresponding noiseless frequency data. Then, if the first loss value is less than a reference first loss value, the electronic device may perform gradient updates on the audio denoising model to be trained to obtain a first model gradient. The reference first loss value is determined based on one or more recent historical first loss values ​​prior to the current training of the audio denoising model. The electronic device may adjust the parameters of the audio denoising model to be trained based on the first model gradient until the first loss value is greater than or equal to the reference first loss value, and the number of iterations in which the first loss value is greater than or equal to the reference first loss value reaches a preset number of training iterations, thus obtaining the target audio denoising model.

[0019] In this application, the gradient of the audio denoising model is updated only when the first loss value is less than the reference first loss value. Based on the updated model gradient, the parameters of the audio denoising model are adjusted. This can improve the accuracy of model training, thereby providing a foundation for subsequent accurate training of the speech recognition model. Ultimately, this improves the recognition performance of electronic devices in noisy scenarios, reduces the occurrence of inaccurate speech recognition results due to low audio quality after denoising, and improves the accuracy of speech recognition.

[0020] In one possible implementation of the first aspect, the process of training the speech recognition model to be trained by the aforementioned electronic device may specifically include: the electronic device determining a second loss value based on the predicted text data and the real text data corresponding to the noise-free frequency data. Then, if the second loss value is less than a reference second loss value, the electronic device can perform gradient updates on the speech recognition model to be trained to obtain a second model gradient. The reference second loss value is determined based on one or more recent historical first loss values ​​prior to the current training of the speech recognition model to be trained. Subsequently, the electronic device can adjust the parameters of the speech recognition model to be trained according to the second model gradient until the second loss value is greater than or equal to the reference second loss value, and the number of iterations in which the second loss value is greater than or equal to the reference second loss value reaches a preset number of training iterations, thus obtaining the target speech recognition model.

[0021] In this application, the gradient of the speech recognition model is updated only when the second loss value is less than the reference second loss value. Based on the updated model gradient, the parameters of the speech recognition model are adjusted. This can improve the accuracy of model training, thereby improving the recognition performance of electronic devices in noisy scenarios, reducing the occurrence of inaccurate speech recognition results due to low audio quality after noise reduction, improving the accuracy of speech recognition, and ultimately improving the user's interactive experience.

[0022] In one possible implementation of the first aspect, after the electronic device obtains the second model gradient, the method further includes: if the angle between the direction of the first model gradient and the direction of the second model gradient is greater than a first preset angle, the electronic device can merge the first model gradient and the second model gradient to obtain a target second model gradient. Then, the electronic device can update the second model gradient to the target second model gradient.

[0023] In this application, considering that a large angle between the gradient of the audio denoising model and the gradient of the speech recognition model may lead to angular conflicts between the two models' gradients, thus degrading the speech recognition capability of the speech recognition model and affecting its performance in noisy scenarios, ultimately impacting the user experience, the electronic device can merge the gradients of the first and second models when the angle between their directions is greater than a first preset angle. This reduces the influence of the audio denoising model's gradient on the speech recognition model, thereby improving the convergence speed of the gradients and ultimately enhancing the model's training efficiency.

[0024] In one possible implementation of the first aspect, the process of merging the first model gradient and the second model gradient by the electronic device may specifically include: the electronic device, based on the direction of the second model gradient, selecting the direction where the angle between the second model gradient and the direction of the first model gradient is a second preset angle as the target merging direction. The second preset angle is less than the first preset angle. Then, the electronic device can map the first model gradient and the second model gradient onto the target merging direction to obtain mapped first model gradient and mapped second model gradient. Finally, the electronic device can merge the mapped first model gradient and the mapped second model gradient to obtain the target second model gradient.

[0025] In this application, the electronic device, based on the direction of the second model gradient, takes the direction where the angle between the second and first model gradients is a second preset angle as the target merging direction, and merges and maps the first and second model gradients onto the target merging direction to obtain the target second model gradient. This reduces the occurrence of angle conflicts in the model gradients caused by large angles between the two model gradients. This not only improves the noise reduction effect of the audio noise reduction model but also reduces the impact of the audio noise reduction model's gradient on the speech recognition model, thereby improving the speech recognition effect of the speech recognition model and ultimately enhancing the speech recognition performance of the electronic device.

[0026] In one possible implementation of the first aspect, after the electronic device obtains the second model gradient, the method further includes: if the numerical ratio between the first model gradient and the second model gradient is outside a preset numerical range, the electronic device can use the ratio between the numerical ratio and a second preset value within the preset numerical range as a gradient adjustment ratio. Then, the electronic device can adjust the first model gradient and / or the second model gradient according to the gradient adjustment ratio to obtain a target first model gradient and / or a target second model gradient. The preset numerical range includes a first preset value and a second preset value, where the first preset value is less than the second preset value. Then, the electronic device can update the first model gradient to the target first model gradient, and / or update the second model gradient to the target second model gradient.

[0027] In this application, considering that if the numerical ratio between the gradient of the first model and the gradient of the second model is too small or too large, it may lead to a difference in the magnitude of the model gradients, resulting in a deterioration in the speech recognition capability of the speech recognition model, thus affecting the speech recognition performance of the electronic device in noisy scenarios and ultimately impacting the user experience, this application addresses the issue. To prevent this, when the numerical ratio between the gradients of the first and second models falls outside a preset range, the electronic device can adjust the gradients of the first and / or second models to reduce the probability of a difference in their magnitudes. This reduces the likelihood of a deterioration in the noise reduction capability of the audio denoising model affecting subsequent speech recognition capabilities, improves the convergence speed of the model gradients during joint training, and ultimately enhances the training efficiency of the model.

[0028] In one possible implementation of the first aspect, the process of adjusting the target compensation coefficient by the electronic device may specifically include: for each noisy frequency data point in the second sub-audio dataset, the electronic device can input the noisy frequency data point into the target audio denoising model to obtain first audio data. Then, the electronic device can merge the first audio data with the noisy frequency data according to a learnable compensation coefficient to obtain second audio data. Next, the electronic device can input the second audio data into the target speech recognition model to obtain first text data. Then, the electronic device can adjust the learnable compensation coefficient based on the first text data and the real text data corresponding to the noisy frequency data, until the second loss value generated by the target speech recognition model is greater than or equal to a reference second loss value, thus obtaining the target compensation coefficient.

[0029] In this application, after obtaining the target audio denoising model and the target speech recognition model, the electronic device can adjust the learnable compensation coefficients. In this way, the probability of audio data loss can be reduced to the greatest extent, so that the denoised audio data is basically the same as the user audio data in the noisy audio data, thereby improving the accuracy of speech recognition.

[0030] In one possible implementation of the first aspect, the process of merging the original audio data and the noise-reduced frequency data by the aforementioned electronic device may specifically include: the electronic device may use the product between the target compensation coefficient and the noise-reduced frequency data as the compensated audio data. Then, the electronic device may merge the compensated audio data with the original audio data to obtain the target audio data.

[0031] In this application, the electronic device can use the product of the target compensation coefficient and the noise-reduced audio data as the compensated audio data, and then merge this compensated audio data with the original audio data. This achieves precise audio data compensation, not only reducing the probability of nonlinear distortion during audio noise reduction, but also compensating for audio data lost after noise reduction, ensuring that the target audio data is essentially identical to the user audio data in the original audio data. This minimizes the loss of user audio data and provides a foundation for the subsequent accurate recognition of voice commands by the electronic device.

[0032] Secondly, a model training method is provided for application in electronic devices. In this method, the electronic device acquires an audio dataset. The audio dataset includes a first sub-audio dataset and a second sub-audio dataset. The first sub-audio dataset includes multiple noise-free frequency data points, and the second sub-audio dataset includes multiple noisy frequency data points, which are obtained by mixing noise from the noise-free frequency data. Then, for each noisy frequency data point in the second sub-audio dataset, the electronic device can input the noisy frequency data into an audio denoising model to be trained, obtaining denoised noisy frequency data. Next, the electronic device can train the audio denoising model based on the denoised noisy frequency data and the corresponding noise-free frequency data, obtaining a target audio denoising model. Then, the electronic device can input the denoised noisy frequency data into a speech recognition model to be trained, obtaining predicted text data. Finally, the electronic device can train the speech recognition model based on the predicted text data and the corresponding real text data from the noise-free frequency data, obtaining a target speech recognition model.

[0033] In one possible implementation of the second aspect, the process of training the audio denoising model to be trained by the aforementioned electronic device may specifically include: the electronic device determining a first loss value based on the denoised noisy frequency data and the corresponding noiseless frequency data. Then, if the first loss value is less than a reference first loss value, the electronic device can perform gradient updates on the audio denoising model to be trained to obtain a first model gradient. The reference first loss value is determined based on one or more recent historical first loss values ​​prior to the training of the audio denoising model to be trained. The electronic device can adjust the parameters of the audio denoising model to be trained based on the first model gradient until the first loss value is greater than or equal to the reference first loss value, and the number of iterations in which the first loss value is greater than or equal to the reference first loss value reaches a preset number of training iterations, thus obtaining the target audio denoising model.

[0034] In one possible implementation of the second aspect, the process of training the speech recognition model to be trained by the aforementioned electronic device may specifically include: the electronic device determining a second loss value based on the predicted text data and the real text data corresponding to the noise-free frequency data. Then, if the second loss value is less than a reference second loss value, the electronic device can perform gradient updates on the speech recognition model to be trained to obtain a second model gradient. The reference second loss value is determined based on one or more recent historical first loss values ​​prior to the current training of the speech recognition model. Subsequently, the electronic device can adjust the parameters of the speech recognition model to be trained based on the second model gradient until the second loss value is greater than or equal to the reference second loss value, and the number of iterations in which the second loss value is greater than or equal to the reference second loss value reaches a preset number of training iterations, thus obtaining the target speech recognition model.

[0035] In one possible implementation of the second aspect, after the electronic device obtains the second model gradient, the method further includes: if the angle between the direction of the first model gradient and the direction of the second model gradient is greater than a first preset angle, the electronic device can merge the first model gradient and the second model gradient to obtain a target second model gradient. Then, the electronic device can update the second model gradient to the target second model gradient.

[0036] In one possible implementation of the second aspect, the process of merging the first model gradient and the second model gradient by the electronic device may specifically include: the electronic device, based on the direction of the second model gradient, taking the direction where the angle between the second model gradient and the direction of the first model gradient is a second preset angle as the target merging direction. The second preset angle is less than the first preset angle. Then, the electronic device can map the first model gradient and the second model gradient onto the target merging direction to obtain mapped first model gradient and mapped second model gradient. Finally, the electronic device can merge the mapped first model gradient and the mapped second model gradient to obtain the target second model gradient.

[0037] In one possible implementation of the second aspect, after the electronic device obtains the second model gradient, the method further includes: if the numerical ratio between the first model gradient and the second model gradient is outside a preset numerical range, the electronic device can use the ratio between the numerical ratio and a second preset value within the preset numerical range as a gradient adjustment ratio. Then, the electronic device can adjust the first model gradient and / or the second model gradient according to the gradient adjustment ratio to obtain a target first model gradient and / or a target second model gradient. The preset numerical range includes a first preset value and a second preset value, where the first preset value is less than the second preset value. Then, the electronic device can update the first model gradient to the target first model gradient, and / or update the second model gradient to the target second model gradient.

[0038] In one possible implementation of the second aspect, after the electronic device obtains the target speech recognition model, the method further includes: for each noisy frequency data point in the second sub-audio dataset, the electronic device can input the noisy frequency data into the target audio denoising model to obtain first audio data. Then, the electronic device can merge the first audio data with the noisy frequency data according to learnable compensation coefficients to obtain second audio data. Then, the electronic device can input the second audio data into the target speech recognition model to obtain first text data. Then, the electronic device can adjust the learnable compensation coefficients based on the first text data and the real text data corresponding to the noisy frequency data until the second loss value generated by the target speech recognition model is greater than or equal to a reference second loss value, thus obtaining the target compensation coefficients.

[0039] Thirdly, this application provides an electronic device, the electronic device including a display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the method described above.

[0040] Fourthly, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0041] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0042] In a sixth aspect, a chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to execute the method described above.

[0043] The beneficial effects that the model training method described in the second aspect, the electronic device described in the third aspect, the computer-readable storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the chip described in the sixth aspect can achieve can be referred to the beneficial effects of the first aspect and any of its possible design methods, and will not be repeated here. Attached Figure Description

[0044] Figure 1 A schematic diagram of a mobile phone interface for receiving voice commands issued by a user, provided as an embodiment of this application;

[0045] Figure 2 A schematic diagram illustrating the introduction of nonlinear distortion during noise reduction processing, as provided in an embodiment of this application;

[0046] Figure 3 A hardware structure diagram of an electronic device provided in an embodiment of this application;

[0047] Figure 4 A software structure diagram of an electronic device provided in an embodiment of this application;

[0048] Figure 5 A schematic diagram of a mobile phone's voice assistant activation interface provided in an embodiment of this application;

[0049] Figure 6 A flowchart illustrating a speech recognition method provided in an embodiment of this application;

[0050] Figure 7 This application provides a schematic diagram of an interface for setting up a voice wake-up function on a mobile phone.

[0051] Figure 8 This is a schematic diagram illustrating how target audio data is obtained, as provided in an embodiment of this application.

[0052] Figure 9 A schematic diagram illustrating a mobile phone performing a voice recognition operation, provided as an embodiment of this application;

[0053] Figure 10 A schematic diagram illustrating the joint training of an audio noise reduction model and a speech recognition model, as provided in an embodiment of this application;

[0054] Figure 11 This is a schematic diagram of a model gradient merging method provided in an embodiment of this application;

[0055] Figure 12 This is a schematic diagram of a model gradient adjustment provided in an embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0057] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0058] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] With the continuous development of electronic devices such as mobile phones and smart home devices, voice recognition technology has been widely applied to various usage scenarios of electronic devices. Specifically, electronic devices can perform corresponding actions by recognizing the voice commands (or voice signals) issued by users. For example, in a phone call scenario, if a user issues the voice command "Youyou, call Mom," it indicates that the user wants to talk to their mother. Therefore, the mobile phone can dial the mother's mobile phone number based on this voice command to realize the call between the user and their mother, that is, to realize the interaction between the user and the electronic device.

[0060] In one implementation, the electronic device can converse with the user through a voice assistant. For an example using a mobile phone as the electronic device, please refer to [link to relevant documentation]. Figure 1 When the phone displays interface 101, in response to inputting a wake-up voice command, such as "Hello Youyou," the phone can wake up the voice assistant. After waking up the voice assistant, the phone can display interface 102. Interface 102 includes a prompt card 1021 for the voice assistant, indicating that the voice assistant has been woken up. Subsequently, in response to the user inputting a commanding voice command, such as "Check the weather," the phone can recognize the user's command and display interface 103. Interface 103 includes a weather query result card 1031, which displays the weather query results.

[0061] In some embodiments, if there is noise in the electronic device and the user's environment, i.e., the user is in a noisy scene, the audio data received by the electronic device may involve environmental noise. This noise can interfere with the electronic device's speech recognition process, making it unable to accurately recognize the user's voice commands. Consequently, the electronic device may perform actions incorrectly (i.e., response errors) or fail to respond, affecting the user experience. For example, a noisy scene could be a shopping mall with various noises and voices, a car with windows open or air conditioning on, or a living room with the TV on, etc., without specific limitations. Environmental noise can be sounds emitted by devices (such as voices playing on a TV, the sound of a machine starting up), sounds emitted by people (such as people talking around, a child crying, etc.), or sounds emitted naturally (such as the sound of wind, the rustling of leaves, etc.), without specific limitations.

[0062] For example, taking an electronic device like a car (such as an in-vehicle terminal) as an example, if a user issues a voice command (such as "close the window") while the car is traveling on a construction site, the electronic device may not be able to accurately recognize the voice command if there is a lot of ambient noise in the environment. Therefore, the user needs to manually control the car to perform the corresponding action (such as the user controlling the window control device on the car to close it), or the user needs to issue the corresponding voice command after the car has passed the construction site to control the car to perform the corresponding window closing action.

[0063] In one implementation, to reduce the impact of environmental noise on speech recognition accuracy, the electronic device can perform noise reduction processing on the received raw audio data to obtain noise-reduced audio data. Then, the electronic device can perform speech recognition on this noise-reduced audio data to obtain a speech recognition result. This speech recognition result is used to instruct the electronic device to perform the action the user wants it to perform. Thus, by first performing noise reduction processing on the raw audio data received by the electronic device, noise data other than human voices can be removed from the raw audio data, making the noise-reduced audio data used for speech recognition noise-free audio data, improving the audio quality of speech recognition, thereby improving the accuracy of speech recognition, and ultimately enhancing the user's interactive experience.

[0064] For example, such as Figure 2As shown, if a user outputs a voice command in a noisy environment, the electronic device can receive the noisy audio data (or raw audio data) and perform noise reduction processing on it to obtain noise-reduced audio data. This noise-reduced audio data contains non-linear distortion, which refers to the distortion and deformation of the speech signal during transmission or processing. The electronic device can then perform speech recognition on this noise-reduced audio data to obtain the speech recognition result. In other words, the noise reduction process on the raw audio data may introduce non-linear distortion, causing loss and distortion of the user's audio data. This affects the accuracy of speech recognition, potentially leading to errors in voice command recognition by the electronic device, ultimately impacting the user's interactive experience.

[0065] Therefore, to reduce the impact of nonlinear distortion during the initial noise reduction process on speech recognition accuracy, this application provides a speech recognition method. In this method, when an electronic device receives raw audio data, it performs noise reduction processing on the raw audio data to obtain noise-reduced audio data. The raw audio data is obtained based on a user's voice command. Then, the electronic device merges the noise-reduced audio data with the raw audio data according to a target compensation coefficient to obtain target audio data. Next, the electronic device performs speech recognition on the target audio data to obtain a speech recognition result. Finally, in response to the speech recognition result, the electronic device executes the operation corresponding to the voice command.

[0066] In this embodiment, by merging the noise-reduced audio data with the original audio data, accurate compensation of the audio data can be achieved. This not only reduces the probability of nonlinear distortion during audio noise reduction but also compensates for audio data lost after noise reduction, making the target audio data essentially the same as the user audio data in the original audio data. This minimizes the loss of user audio data, thereby improving the accuracy of speech recognition, the accuracy of electronic device action execution, and ultimately enhancing the user's interactive experience.

[0067] For example, the aforementioned electronic devices can be mobile phones, tablets, desktop computers, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as voice assistant devices such as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices. This application does not impose special limitations on the specific form of the electronic device. The following description uses a mobile phone as an example.

[0068] See Figure 3 The mobile phone may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0069] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone. In other embodiments of this application, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0070] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, memory, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0071] In some embodiments, the mobile phone can perform the dialogue method provided in this application through the processor 110.

[0072] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0073] The mobile phone implements its display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0074] The display screen 194 is used to display images, videos, etc. In some embodiments, the mobile phone can display a continuous dialogue interface and multiple dialogue topics through the display screen 194.

[0075] Mobile phones can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0076] A mobile phone can implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor. Examples include music playback and recording. In some embodiments, the mobile phone can capture user-input voice through the microphone 170C.

[0077] Button 190 may include a power button, volume buttons, etc. Button 190 may be a mechanical button or a touch button. The mobile phone can receive button input and generate key signal inputs related to the phone's user settings and function control. Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. Indicator 192 may be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile phone.

[0078] For example, the software system of the aforementioned electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to illustrate the software structure of the electronic device 100.

[0079] Figure 4This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0080] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer (application layer), the application framework layer (framework layer), the Android runtime and system libraries, and the kernel layer (or driver layer). The application layer can include a series of application packages.

[0081] like Figure 4 As shown, the application package may include telephone, SMS, calendar, map, audio noise reduction module, speech recognition module, joint training module, and model deployment module, etc.

[0082] The aforementioned audio noise reduction module is used to perform noise reduction processing on noisy frequency data. In some embodiments, upon receiving noisy frequency data, the audio noise reduction module can perform noise reduction processing on the noisy frequency data to obtain noise-reduced frequency data. It is understood that the audio noise reduction module can support streaming audio noise reduction, meaning that the audio noise reduction module can continuously receive audio data and continuously process the received audio data. That is, as long as the audio noise reduction module receives noisy frequency data, it will perform noise reduction processing on the noisy frequency data. This can improve the audio noise reduction rate, thereby improving the operational efficiency of the electronic device and enhancing the user's interactive experience.

[0083] In one example, the audio noise reduction module described above can use an audio noise reduction model to process noisy frequency data, obtaining noise-reduced frequency data. In another example, the audio noise reduction module can also use a noise reduction filtering method to process noisy frequency data, obtaining noise-reduced frequency data. This noise reduction filtering method can be the Wiener filtering method.

[0084] In some embodiments, after obtaining the aforementioned noise-reduced audio data, the audio noise reduction module can further merge the noise-reduced audio data with the original audio data according to the target compensation coefficient to obtain the target audio data. This compensates for the audio data lost during noise reduction, ensuring that the target audio data is essentially identical to the user audio data in the original audio data, minimizing the loss of user audio data and thus improving the accuracy of speech recognition.

[0085] The aforementioned speech recognition module is used to perform speech recognition on the aforementioned target audio data. In some embodiments, after obtaining the target audio data, the speech recognition module can perform speech recognition on the target audio data to obtain a target voice command. This target voice command represents the operation that the user wants the electronic device to perform. It is understood that the speech recognition module also supports continuous speech recognition functionality; that is, the speech recognition module can continuously receive target audio data and perform speech recognition on the received target audio data in real time. In other words, as long as the speech recognition module receives target audio data, it will perform speech recognition on the target audio data. This improves the efficiency of audio data recognition, thereby improving the efficiency of the electronic device's action execution and enhancing the user's interactive experience.

[0086] For example, the above-mentioned speech recognition module can use a speech recognition model to perform speech recognition on the target audio data to obtain the target speech command.

[0087] The aforementioned joint training module is used to jointly train the audio denoising model in the denoising module and the speech recognition model in the speech recognition module to improve the performance of both models. In some embodiments, the joint training module can adjust the parameters of the audio denoising model and the speech recognition model to be trained based on the first loss value generated by the audio denoising model and the second loss value generated by the speech recognition model, respectively, to obtain the trained audio denoising model and the trained speech recognition model.

[0088] The model deployment module described above is used to deploy the audio denoising model and the speech recognition model. In some embodiments, after obtaining the trained audio denoising model and the trained speech recognition model, the model deployment module can deploy the trained audio denoising model and the trained speech recognition model in an electronic device so that noise reduction processing and speech recognition operations can be directly performed after receiving voice commands.

[0089] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 4 As shown, the application framework layer may include a window manager service (WMS), a content provider, a notification manager, a resource manager, and a view system.

[0090] The window manager manages the windowed applications. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc. The notification manager allows applications to display notifications in the status bar, conveying informative messages; these notifications can disappear automatically after a short pause without user interaction. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The view system includes visual controls, such as controls for displaying text and images. The view system is used to build applications. The display interface can consist of one or more views.

[0091] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0092] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0093] System libraries can include multiple functional modules. For example: surface manager, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), media libraries, etc.

[0094] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics. The media library supports playback and recording of various common audio and video formats, as well as still image files. The media library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0095] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0096] For example, the software architecture of the aforementioned electronic device 100 may also include a hardware abstraction layer. This hardware abstraction layer is a wrapper around Linux kernel drivers, providing interfaces to higher-level systems. It hides the hardware interface details of a specific platform, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include modules such as a camera HAL, audio HAL, GPS HAL, and Wi-Fi HAL.

[0097] Understandable, Figure 4 The layers in the illustrated structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100, i.e., the mobile phone. In other embodiments of this application, the structure may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0098] Based on the electronic devices described above, this application provides a speech recognition method. This method can be implemented in the aforementioned mobile phone. During the process of waking up the voice assistant on the mobile phone, the speech recognition method provided in this application can be used, enabling the voice assistant on the mobile phone to accurately recognize the user's voice commands, thereby improving the user's interactive experience.

[0099] The wake-up methods for the aforementioned voice assistant include, but are not limited to, at least one of the following: Wake-up method 1, keyword wake-up: the phone recognizes the user's input of a wake-up voice containing keywords, such as "Hello Youyou," and can then wake up the voice assistant; Wake-up method 2, the phone detects the user's long press of the power button and can then wake up the voice assistant; Wake-up method 3, breath wake-up: the phone detects the user raising the phone and bringing their mouth to the microphone and can then wake up the voice assistant. This application does not specifically limit the wake-up methods.

[0100] After the phone wakes up the voice assistant using the above method, the voice assistant will be in an active state. For example, in this active state, the phone can display... Figure 5 The interface 301 shown includes a prompt card 3011 for the voice assistant. It is understood that this article primarily uses the example of waking up the voice assistant when the phone's home screen is displayed; in practice, this is not a limitation. For example, the voice assistant can also be activated when the phone is displaying the lock screen, the interface of a video application, or the interface of a chat application.

[0101] Specifically, when the voice assistant is active, the user can input dialogue content 1. Dialogue content 1 can be in voice format, such as inputting commands like "check the weather" or "call mom," or question-and-answer format like "what hybrid cars under 300,000 yuan are available" or "what gift to give mom for Mother's Day." Alternatively, dialogue content 1 can also be in text format, such as... Figure 5The prompt card 3011 in the interface 301 shown may include a keyboard control 3012. In response to a user's trigger operation (such as a click) on the keyboard control 3012, the mobile phone can provide a keyboard for the user to input text-based dialogue content. Subsequently, in response to the user's voice input "Recommend a hybrid vehicle under 300,000 yuan", the mobile phone can display... Figure 5 The interface 302 shown includes a window 3021. Window 3021 includes a dialog option, "Dialogue" 3022, which is currently selected. Therefore, window 3021 also includes a dialog box 3023. Dialog box 3023 includes user-inputted dialogue content 1, such as "Recommend a hybrid car under 300,000" 3024; and the phone's response 1, such as "The following are recommendations for hybrid cars under 300,000: Brand A hybrid…; Brand B hybrid…" 3025. When the phone displays interface 302, in response to the user inputting "Brand A hybrid engine," the phone can display… Figure 5 The interface 303 shown also includes a window 3021, and the window 3021 also includes a dialog box 3023. However, unlike interface 302, the dialog box 3023 in interface 303 also includes dialogue content 2 and answer 2 for continuous dialogue, namely "Brand A hybrid engine" 3031 and "The characteristics of Brand A hybrid engine are as follows:..." 3032. That is, dialog box 3023 can be used for continuous dialogue.

[0102] Furthermore, window 3021 of interfaces 302 and 303 (using interface 302 as an example in the diagram) also includes multiple category options, such as "Smart Video Production" 3026 and "Schedule Service" 3027. "Smart Video Production" 3026 includes topic options for video creation; "Schedule Service" 3027 includes topic options for daily travel. It is understandable that window 3021 may also include more or fewer category options. See also... Figure 5 In response to the user's right-to-left swiping action on the option area (represented by the dashed rectangle in the figure) within window 3021 in interface 302, the phone can display more category options in window 3021, such as "Text Creation" 3041, "Knowledge Q&A" 3042, and "User Assistant" 3043 in window 3021 as shown in interface 304. Among them, "Text Creation" 3041 includes topic options for copywriting; "Knowledge Q&A" 3042 includes topic options for knowledge-related topics; and "User Assistant" 3043 includes topic options for convenient phone use.

[0103] While displaying multiple category options, the phone can also show recommended options, including topic options for trending topics, so that users can access various trending topics under one option. For example, the recommended options could be... Figure 5 The “Recommended” section 3028 is located in window 3021 of the interface 302 shown.

[0104] The method of this application embodiment will be described below based on the wake-up scenario of the voice assistant and the usage scenario of the voice assistant. Specifically, as shown... Figure 6 As shown, the speech recognition method may include S601 to S605.

[0105] S601, the mobile phone collects raw audio data. This raw audio data is obtained based on voice commands issued by the user.

[0106] It is understandable that during the process of a mobile phone receiving voice commands from a user, the phone may be subject to interference from ambient noise. In other words, the raw audio data collected by the phone may include both user audio data and noise data. The user audio data is obtained by converting the user's voice commands. Ambient noise refers to sounds other than the phone's owner's voice. For example, ambient noise could be the sound of keyboards clicking in an office, announcements in a shopping mall, or television sounds in a living room.

[0107] The aforementioned voice commands (or voice signals) include wake-up information (or wake-up words) and / or dialogue information (or dialogue commands).

[0108] The aforementioned wake-up information is used to indicate the device the user wants to control, that is, the device that needs to respond to the aforementioned voice command and perform the corresponding operation. In this embodiment, the wake-up information is the wake-up word corresponding to the mobile phone (such as Youyou). In other embodiments, the wake-up information can also be the wake-up word corresponding to other electronic devices with voice wake-up function (such as tablets, smart speakers, etc.), etc., and there is no specific limitation. It can be understood that the same wake-up word corresponds to only one electronic device, that is, the wake-up words corresponding to different electronic devices are different, so that the voice assistant can determine the device that the user wants to wake up based on the wake-up word. Furthermore, the wake-up word corresponding to the electronic device can be preset by the user according to their own habits, or it can be the default for each electronic device, as long as it can distinguish each electronic device through the wake-up word, there is no specific limitation.

[0109] The aforementioned dialogue information is used to instruct the mobile phone to perform the required operation. For example, the dialogue information can be a query command, which triggers the mobile phone to perform a query based on the content of the query command and display the corresponding query results. For example, the query command could be "Check the weather" or "Recommend a hybrid car under 300,000 yuan," etc. As another example, the dialogue information can also be an outgoing call command, which triggers the mobile phone to make an outgoing call based on the called number or contact information in the outgoing call command and display the corresponding outgoing call interface. The contact information can include the contact's name (e.g., mother) or contact's number.

[0110] For example, the action information can also be a control command (or device control command) from another device. This control command triggers the mobile phone to control the target device according to the target device specified in the control command. The target device can include a device name (such as an air conditioner, water heater, etc.). It should be noted that a smart voice sharing system is established between the controlled target device and the mobile phone. This means that only electronic devices within the smart voice sharing system can control each other. The smart voice sharing system is used to share voice commands received by all electronic devices. In other words, as long as one electronic device in the smart voice sharing system can receive a voice command from the user, that electronic device can control the target device to perform the corresponding operation according to the control command.

[0111] In some embodiments, electronic devices can establish a smart voice sharing system through an application (APP). In one example, if the electronic device is a device with a display screen (such as a mobile phone, computer, etc.), the user can download a smart voice sharing application from the electronic device (e.g., an app store within the electronic device). Then, the user can log in to the same account within the smart voice sharing APP to establish the smart voice sharing system.

[0112] In another example, if the electronic device is one without a display screen (such as a speaker, air conditioner, etc.), meaning it cannot install the aforementioned smart voice sharing app, the user can select an electronic device with the smart voice sharing app from the smart voice sharing system. Then, the user can manually add the electronic device without a display screen to the corresponding smart voice sharing app on the device with the app, thus establishing the smart voice sharing system.

[0113] Furthermore, it should be understood that electronic devices in an intelligent voice sharing system can communicate with each other through a preset wireless communication method. This preset wireless communication method may include one or more of Bluetooth, Wi-Fi, NFC, and mobile networks (such as 2G, 3G, 4G, 5G, etc.).

[0114] It should be noted that the aforementioned voice commands can be used to wake up the phone and control it to perform corresponding operations, thereby simplifying the user's workflow. However, to protect user privacy, the phone needs to determine whether the user has enabled the voice wake-up function (or voice wake-up service) before receiving the voice command. Only when the voice wake-up function is enabled, i.e., after the phone receives confirmation from the user that the voice wake-up function has been enabled, can the phone receive the user's voice command to further perform voice recognition operations.

[0115] For example, such as Figure 7 As shown, when the user clicks the settings control in the initial interface, the phone can display the settings interface, which includes settings for WLAN, Bluetooth, mobile network, desktop and wallpaper, smart voice, and smart sensing. Subsequently, when the user clicks the smart voice settings item (or smart voice control) in the settings interface, the phone can display the smart voice settings interface. This smart voice settings interface includes a voice wake-up control 701 and a wake-up word definition card 702. The wake-up word definition card 702 is used to define the wake-up word for the phone, so that the user can wake up the phone using the defined wake-up word. Afterwards, if the user clicks the voice wake-up control 701 in the smart voice settings interface, it means the user wants to enable the voice wake-up function; therefore, the phone can activate the voice wake-up function.

[0116] It should be noted that the above Figure 7 The interface shown for setting up the voice wake-up function is just an example. Users can also control the voice wake-up function to be turned on or off in other ways. For example, the phone can directly display the Smart Voice option so that the user can agree to control the Smart Voice function to be turned on or off. That is to say, if the Smart Voice setting item is clicked by the user, all the functions included in that Smart Voice setting item will be turned on or off.

[0117] S602, the mobile phone performs noise reduction processing on the above raw audio data to obtain noise-reduced audio data.

[0118] Specifically, after acquiring the raw audio data, the mobile phone can perform noise reduction processing on the raw audio data to obtain noise-reduced audio data.

[0119] In one implementation, the mobile phone can perform noise reduction processing on the aforementioned raw audio data using the Wiener filtering method. Wiener filtering is used to minimize the mean square value of the estimation error, which refers to the estimation of error during data processing, i.e., the estimation of noise data included in the raw audio data. Specifically, the mobile phone can collect historical audio data. This historical audio data refers to audio data collected by the mobile phone before collecting the raw audio data. For example, this historical audio data can be all audio data within a preset number of frames (e.g., 50 frames) preceding the raw audio data. Then, the mobile phone can perform noise estimation on this historical audio data to obtain candidate noise data, which represents noise data that the mobile phone may have collected during the audio data collection process. Finally, the mobile phone can perform noise reduction processing on the collected raw audio data according to this candidate noise data.

[0120] It's important to note that the Wiener filtering method used in mobile phones for audio noise reduction only works for steady-state noise and cannot handle non-steady-state noise. If the phone captures audio data including non-steady-state noise, it will not only fail to achieve accurate noise reduction but may also worsen the noise reduction effect, thus impacting the user experience. Steady-state noise refers to continuous noise generated in the phone's environment, such as the constant sound of keyboards in an office or the continuous sound of air conditioning in a car. Non-steady-state noise refers to sudden noise in the phone's environment, such as a sudden sound of someone slamming their hand on a table in an office or a sudden argument in a shopping mall.

[0121] In another implementation, to improve the phone's noise reduction performance, the phone can perform noise reduction processing on the original audio data based on a target audio noise reduction model. This audio noise reduction model is a deep learning model used to learn from a large amount of noisy data to achieve audio noise reduction. For example, this deep learning model can be a convolutional neural network. Specifically, the phone can input the original audio data into the audio noise reduction model to obtain the noise-reduced audio data.

[0122] In some embodiments, the training process of the aforementioned target audio noise reduction model may specifically include: the mobile phone acquiring an audio dataset. This audio dataset includes multiple audio data points, which are single-channel audio data. A single channel refers to the mobile phone acquiring audio data from only one audio channel (or channel). Thus, training the model using single-channel audio data allows for targeted model training, improving the training accuracy of the target audio noise reduction model and / or the target speech recognition model, thereby providing a foundation for accurate recognition of subsequent speech commands.

[0123] The aforementioned audio dataset may include sub-audio datasets for different scenarios. These sub-audio datasets could be a first sub-audio dataset for quiet scenarios and a second sub-audio dataset for noisy scenarios. A quiet scenario is a noise-free scenario, that is, a scenario containing only speech. A noisy scenario is any scenario with noise. This noisy scenario can be determined based on noisy scenarios that the user might encounter. For example, a noisy scenario could be inside a moving car, in a shopping mall with pedestrian traffic, etc. In other words, the audio dataset includes a first sub-audio dataset and a second sub-audio dataset. The first sub-audio dataset includes multiple noise-free audio frequencies. The second sub-audio dataset includes multiple noisy audio frequencies.

[0124] The aforementioned noisy frequency data can be obtained by mixing noise-free frequency data and pure noise data. Specifically, the mobile phone can use an audio simulation algorithm to mix noise-free frequency data and pure noise data to obtain noisy frequency data. This audio simulation algorithm is used to simulate noisy frequencies that are close to real noisy frequencies, thus achieving the function of noisy frequency simulation. The pure noise data can include environmental noise data and / or reverberation noise data. Environmental noise data refers to sound data that disturbs the surrounding living environment in daily life. Reverberation noise data refers to sound data reflected after sound waves encounter obstacles, such as walls, floors, and decorative objects.

[0125] It is understandable that, since the aforementioned noisy frequency data is obtained by mixing noise from noiseless frequency data, the number of noisy frequency data in the second sub-audio dataset is the same as the number of noiseless frequency data in the first sub-audio dataset, and there is a one-to-one correspondence between the noisy frequency data in the second sub-audio dataset and the noiseless frequency data in the first sub-audio dataset. This allows for accurate training of the audio denoising model, improves the model's training performance, and enhances its generalization ability, laying the foundation for the subsequent accurate generation of denoised frequency data.

[0126] Specifically, after obtaining the aforementioned audio dataset, the mobile phone can train a pre-built audio denoising model based on the audio dataset to obtain a trained audio denoising model, which is the target audio denoising model. For example, for each noisy frequency data point in the audio dataset, the mobile phone can input the noisy frequency data into the pre-built audio denoising model to obtain denoised noisy frequency data. Then, based on the denoised noisy frequency data and noiseless frequency data in the audio dataset that matches the noisy frequency data, the mobile phone can determine a first loss value. Afterwards, the mobile phone can adjust the parameters of the pre-built audio denoising model based on the first loss value until the first training result meets the first preset training requirement, thus obtaining the trained audio denoising model. The first training result meeting the first preset training requirement can include at least one of the following: the total number of training iterations (or first training iterations) of the audio denoising model reaches the first preset training iterations; the first loss value is less than the first preset loss value; and the second training iterations reach the second preset training iterations. The second training iterations are the number of times the loss value generated during the training of the audio denoising model remains continuously unchanged.

[0127] After obtaining the trained audio noise reduction model, the mobile phone can input the collected raw audio data into the trained audio noise reduction model to obtain noise-reduced frequency data. In this way, noise reduction processing of noisy frequency data can be achieved, providing a foundation for accurate recognition of voice commands in the subsequent process.

[0128] S603: The mobile phone merges the noise-reduced audio data with the original audio data according to the target compensation coefficient to obtain the target audio data.

[0129] Specifically, after obtaining the aforementioned noise-reduced audio data, the mobile phone can merge the noise-reduced audio data with the original audio data according to the target compensation coefficient to obtain the target audio data. This target audio data is the lossless audio data generated by the mobile phone after noise reduction processing. For example, the aforementioned target compensation coefficient can be pre-set by the mobile phone, or it can be obtained synchronously during the training of the aforementioned audio noise reduction model and speech recognition model; there is no specific limitation.

[0130] In some embodiments, such as Figure 8As shown, the mobile phone can first multiply the target compensation coefficient by the noise-reduced audio data to obtain the compensated audio data. Then, the phone can merge this compensated audio data with the original audio data to obtain the target audio data. This allows for precise audio data compensation, reducing the probability of non-linear distortion during audio noise reduction and compensating for audio data lost during noise reduction. This ensures that the target audio data is essentially identical to the user's audio data in the original audio data, minimizing loss of user audio data and providing a foundation for accurate voice command recognition by the phone.

[0131] In one implementation, the process by which the mobile phone obtains the aforementioned target audio data can be calculated using the following expression:

[0132] mix = noisy + alpha * enhancement (Expression 1);

[0133] Where mix is ​​the target audio data, noisy is the original audio data, alpha is the target compensation coefficient, and enhance is the noise-reduced audio data.

[0134] S604, the mobile phone performs speech recognition on the above target audio data and obtains the speech recognition result.

[0135] Specifically, after obtaining the target audio data, the mobile phone can perform speech recognition on the target audio data to obtain a speech recognition result (or target voice command). This speech recognition result is used to instruct the mobile phone to execute the operation corresponding to the aforementioned voice command; that is, the speech recognition result can include the mobile phone's wake word and dialogue information.

[0136] In one implementation, the mobile phone can perform speech recognition on the aforementioned target audio data based on a target speech recognition model. This target speech recognition model is an automatic speech recognition (ASR) model, which converts user-issued voice commands into text information that the mobile phone can understand, thereby achieving automatic recognition and processing of voice commands.

[0137] In some embodiments, the training process of the target speech recognition model may specifically include: the mobile phone acquiring the first sub-audio dataset. This first sub-audio dataset includes multiple noise-free frequency data points and the actual text data carried by each noise-free frequency data point. Then, the mobile phone can train a pre-built speech recognition model based on this first sub-audio dataset to obtain the trained speech recognition model, which is the target speech recognition model.

[0138] For example, for each noise-free frequency data point in the first sub-audio dataset, the mobile phone can input the noise-free frequency data into the pre-built speech recognition model to obtain predicted text data. Then, the mobile phone can determine a second loss value based on the predicted text data and the corresponding real text data. Afterwards, the mobile phone can adjust the parameters of the pre-built speech recognition model based on the second loss value until the second training result meets the second preset training requirements, thus obtaining a trained speech recognition model. The second training result meeting the second preset training requirements may include at least one of the following: the total number of training iterations (or the third number of training iterations) of the speech recognition model reaches the third preset number of training iterations; the second loss value is less than the second preset loss value; and the fourth number of training iterations reaches the fourth preset number of training iterations. The fourth number of training iterations is the number of times the loss value generated during the training of the speech recognition model remains continuously unchanged.

[0139] Specifically, after obtaining the trained speech recognition model, the mobile phone can input the target audio data into the trained speech recognition model to obtain the speech recognition result. This enables accurate recognition of audio data, improves the accuracy of speech recognition, and provides a foundation for subsequent precise execution of corresponding operations.

[0140] S605 responds to the voice recognition result, and the mobile phone executes the operation corresponding to the voice command.

[0141] Specifically, after obtaining the aforementioned voice recognition results, the mobile phone can execute the corresponding operation based on the voice command recognized in the results. This improves the accuracy of the phone's operations and thus enhances the user's interactive experience.

[0142] In one implementation, the mobile phone can execute the operation corresponding to the dialogue information in the voice command. For example, if the dialogue information is a query command, and the query command is "query the weather," then after receiving the query command "query the weather," the mobile phone can display the current weather query results. As another example, if the dialogue information is a device control command, and the device control command is "turn on the air conditioner," then after receiving the device control command "turn on the air conditioner," the mobile phone can control the air conditioner, which is connected to the same smart voice sharing system as the mobile phone, to start up.

[0143] The above described how a mobile phone performs voice recognition on user-issued voice commands. Below, we will continue to introduce one possible implementation process for a mobile phone to execute corresponding operations based on voice commands. For example... Figure 9As shown, the process can specifically include: the phone's microphone collecting raw audio data, which includes the user's voice command (such as "Hello Youyou, turn on the air conditioner") and noise data (i.e., audio unrelated to the voice command). The phone then sequentially inputs the raw audio data into an audio noise reduction model and a speech recognition model to obtain the speech recognition result, i.e., the correct voice command. The phone can then execute the operation corresponding to the correct voice command; that is, the phone can control the air conditioner, which is connected to the same smart voice sharing system as the phone, to start the air conditioner.

[0144] In some embodiments, considering that the mobile phone trains the above-mentioned audio noise reduction model and the above-mentioned speech recognition model separately, although it can perform audio noise reduction through the trained audio noise reduction model and speech recognition through the trained speech recognition model, if the original audio data still contains a small amount of noise data after audio noise reduction, and the speech recognition model has not been trained for the noisy frequency data containing a small amount of noise, that is to say, the accuracy of the mobile phone's audio noise reduction will affect the subsequent speech recognition results. This will cause the speech recognition performance of the speech recognition model to suddenly deteriorate, thus failing to accurately recognize the voice commands, and ultimately affecting the user's interactive experience.

[0145] Therefore, to improve the speech recognition performance of mobile phones in noisy scenarios, the phone can jointly train the aforementioned audio denoising model and speech recognition model. That is, the phone can input noisy audio data into a pre-built audio denoising model to obtain denoised noisy audio data. Then, the phone can input this denoised noisy audio data into a pre-built speech recognition model to obtain predicted text data. Afterwards, the phone can train the pre-built speech recognition model based on the predicted text data and real text data matching the noisy audio data, resulting in a trained speech recognition model. In this way, by jointly training the two models, the phone's recognition performance can be maximized, reducing the likelihood of inaccurate speech recognition results due to low-quality denoised audio, thus improving speech recognition accuracy and ultimately enhancing the user experience.

[0146] Specifically, the process of jointly training the audio noise reduction model and the speech recognition model on the aforementioned mobile phone is as follows: Figure 10As shown, specifically, this may include: the mobile phone acquiring the aforementioned second sub-audio dataset, wherein the second sub-audio dataset includes multiple noisy frequency data points and noise-free frequency data carried by each noisy frequency data point. It can be understood that the noisy frequency data is obtained by mixing noise from the noise-free frequency data. Then, for each noisy frequency data point in the second sub-audio dataset, the mobile phone can input the noisy frequency data into the audio denoising model to be trained, obtaining denoised noisy frequency data. Afterwards, the mobile phone can determine a first loss value based on the denoised noisy frequency data and the corresponding noise-free frequency data.

[0147] Having obtained the noise-reduced audio data, the mobile phone can input it into the speech recognition model to be trained to obtain predicted text data. Then, the mobile phone can determine a second loss value based on the predicted text data and the corresponding real text data from the noise-free audio data. Following this, the mobile phone can perform joint training on the audio noise reduction model and the speech recognition model according to a preset training strategy, using the second loss value and the first loss value, to obtain the trained audio noise reduction model and the trained speech recognition model, which are the target audio noise reduction model and the target speech recognition model.

[0148] The aforementioned preset training strategy is determined based on the relationship between the loss value (or reference loss value) generated during previous model training and the loss value generated during this model training. In other words, this preset training strategy is determined based on the relationship between the previously generated first loss value (or reference first loss value) and the currently generated first loss value, and the relationship between the previously generated second loss value (or reference second loss value) and the currently generated second loss value. For example, the historical loss value can be any loss value generated during previous model training, the loss value generated during the phone's last model training session, or the average or lowest value among multiple loss values ​​generated during previous model training sessions, etc., without specific limitations.

[0149] In some embodiments, if a reference first loss value is higher than the first loss value mentioned above, and a reference second loss value is higher than the second loss value mentioned above, it indicates that the loss generated by the audio denoising model is decreasing, and the loss generated by the speech recognition model is decreasing. This means that the audio denoising model and the speech recognition model are not yet fully trained. Therefore, the mobile phone can perform gradient updates on the audio denoising model and the speech recognition model separately until the losses generated by the audio denoising model and the speech recognition model no longer decrease, and the number of iterations at which the losses no longer decrease reaches a preset number of training iterations, thus obtaining a trained audio denoising model and a trained speech recognition model. The reference first loss value is determined based on the most recent one or more historical first loss values ​​before the training of the audio denoising model to be trained. The reference second loss value is determined based on the most recent one or more historical first loss values ​​before the training of the speech recognition model to be trained.

[0150] After the audio noise reduction model and speech recognition model are trained, that is, after obtaining the target audio noise reduction model and target speech recognition model, the mobile phone can adjust the learnable compensation coefficients to obtain the aforementioned target compensation coefficients. These learnable compensation coefficients can be pre-set by the mobile phone.

[0151] Specifically, after the audio denoising model and speech recognition model are trained, the mobile phone can input each noisy frequency data point in the second sub-audio dataset into the target audio denoising model to obtain the first audio data. Then, the mobile phone merges the first audio data with the noisy frequency data according to a learnable compensation coefficient to obtain the second audio data. Next, the mobile phone can input the second audio data into the target speech recognition model to obtain the first text data. Then, the mobile phone can adjust the learnable compensation coefficient based on the first text data and the corresponding real text data, until the second loss value generated by the target speech recognition model no longer decreases, thus obtaining the target compensation coefficient.

[0152] Alternatively, after obtaining the second audio data, the mobile phone can also adjust the learnable compensation coefficient based on the second audio data and the noiseless frequency data corresponding to the noisy frequency data, until the second audio data is the same as the noiseless frequency data, and obtain the target compensation coefficient.

[0153] In other embodiments, when the reference first loss value is higher than the aforementioned first loss value, but the reference second loss value is equal to the aforementioned second loss value, it indicates that the loss generated by the audio denoising model is still decreasing, but the loss generated by the speech recognition model is no longer decreasing. That is, the speech recognition model is basically trained, but the audio denoising model is not trained. Therefore, the mobile phone can only perform gradient updates on the audio denoising model until the loss generated by the audio denoising model no longer decreases, and the number of training times when the loss no longer decreases reaches the preset number of training times, thus obtaining the trained audio denoising model and the trained speech recognition model.

[0154] In some other embodiments, when the reference first loss value is equal to the first loss value, but the reference second loss value is higher than the second loss value, it indicates that the loss generated by the audio denoising model is no longer decreasing, but the loss generated by the speech recognition model is still decreasing. That is, the audio denoising model is basically trained, but the speech recognition model is not trained. Therefore, in order to reduce unnecessary training resources, the mobile phone can only perform gradient updates on the speech recognition model until the loss generated by the speech recognition model no longer decreases, and the number of iterations in which the loss no longer decreases reaches the preset number of training iterations, so as to obtain the trained audio denoising model and the trained speech recognition model.

[0155] The model gradient is used to characterize the direction in which the model's loss value decreases the fastest. In other words, the model gradient is determined based on the loss value of the loss function in the model, and the model parameters can be adjusted through this model gradient.

[0156] In other words, the phone only updates the gradient of the audio noise reduction model if the reference first loss value is higher than the aforementioned first loss value. Similarly, the phone only updates the gradient of the speech recognition model if the reference second loss value is higher than the aforementioned second loss value. This approach improves the utilization of training resources while ensuring model training accuracy.

[0157] In one implementation, before jointly training the aforementioned audio denoising model and speech recognition model, the mobile phone can initialize the pre-built audio denoising model and speech recognition model according to a preset parameter tuning strategy, resulting in the audio denoising model and speech recognition model to be trained. The initialization process refers to initializing the model's weights (or model parameters). These weights are parameters in the neural network used to adjust and learn the model's behavior, enabling it to effectively map the input data and extract useful features. In other words, by adjusting the model's weights, the model's loss function can be minimized, allowing the model to effectively map the input data.

[0158] In one example, the aforementioned preset parameter tuning strategy could involve first training a pre-built audio denoising model and a pre-built speech recognition model separately. Then, if the audio denoising model meets a first training criterion, a new audio denoising model to be trained is obtained. Similarly, if the speech recognition model meets a second training criterion, a new speech recognition model to be trained is obtained. In other words, the audio denoising model and the speech recognition model are trained separately first, and then combined for training. This approach improves the model's training effect while maintaining its generalization ability, thereby enhancing the phone's speech recognition performance in noisy scenarios and laying a foundation for accurate voice command recognition in the future.

[0159] The first training criterion is used to keep the loss value of the audio denoising model stable. This first training criterion may include at least one of the following: the denoising capability of the audio denoising model reaches a first preset proportion; the difference between the first loss value of the audio denoising model and a reference first loss value is less than a first preset difference; and the number of training iterations during model training where the difference is less than the first preset difference reaches a preset number of training iterations. The second training criterion is used to keep the loss value of the speech recognition model stable. This second training criterion may include at least one of the following: the denoising capability of the speech recognition model reaches a second preset proportion; the difference between the second loss value of the speech recognition model and a reference second loss value is less than a second preset difference; and the number of training iterations during model training where the difference is less than the second preset difference reaches a preset number of training iterations.

[0160] It's understandable that since the audio denoising model to be trained has already reached the first training standard (i.e., it has achieved a certain level of denoising capability), and the speech recognition model to be trained has already reached the second training standard (i.e., it has achieved a certain level of speech recognition capability), if the learning rate (or training capacity) of the audio denoising and speech recognition models is not reduced, the two models may experience training collapse during subsequent joint training. This could lead to inaccurate audio denoising and / or speech recognition results, ultimately affecting the training effect of the models. Therefore, to avoid training collapse, the phone can reduce the learning rate of the audio denoising and speech recognition models after obtaining them. Then, the phone can perform joint training on these models to obtain well-trained audio denoising and speech recognition models.

[0161] In another example, the aforementioned preset parameter tuning strategy can also involve first training a pre-built audio denoising model. Then, once the audio denoising model reaches a first training criterion, the audio denoising model to be trained is obtained, and the weights of the pre-built speech recognition model are initialized to obtain the speech recognition model to be trained. Here, weight initialization refers to randomly setting the model weights in advance.

[0162] It's understandable that since the audio denoising model to be trained has already reached the first training standard—meaning it has achieved a certain level of denoising capability—not reducing its learning rate could lead to training failure during subsequent joint training, potentially resulting in inaccurate denoising results and affecting the model's training effectiveness. Therefore, to avoid training failure, the phone can reduce the learning rate of the audio denoising model after obtaining it. Then, the phone can jointly train both the audio denoising model and the speech recognition model to obtain well-trained audio denoising and speech recognition models.

[0163] In another example, the aforementioned preset parameter tuning strategy can also involve directly initializing the weights of a pre-built audio denoising model and a pre-built speech recognition model to obtain the audio denoising model and the speech recognition model to be trained. Then, the mobile phone performs joint training on these two models to obtain the trained audio denoising model and the trained speech recognition model.

[0164] It should be noted that during the joint training of the audio denoising model and the speech recognition model, there may be angle conflicts and magnitude differences between the model gradients of the audio denoising model and the speech recognition model. This could lead to the speech recognition model being affected by the model gradient of the audio denoising model, reducing the convergence speed of the model gradient and ultimately affecting the training efficiency. Therefore, to reduce the impact of the model gradient of the audio denoising model on the speech recognition model, the mobile phone can perform separate correction processing on the model gradients of the two models to obtain the target model gradients for both models.

[0165] After obtaining the target model gradients of the two models mentioned above, the mobile phone can adjust the parameters of the audio denoising model based on the target model gradient (or the first target model gradient) to obtain a trained audio denoising model. Similarly, based on the target model gradient (or the second target model gradient) of the speech recognition model, the parameters of the speech recognition model can be adjusted to obtain a trained target speech recognition model. This reduces the impact of the audio denoising model's gradient on the speech recognition model, thereby improving the convergence speed of the model gradient and ultimately enhancing the model's training efficiency.

[0166] In one implementation, if the angle between the gradient of the audio denoising model and the gradient of the speech recognition model is large, it may lead to an angular conflict between the gradients of the two models, resulting in a deterioration in the speech recognition capability of the speech recognition model. This affects the speech recognition performance of the speech recognition model, which in turn affects the speech recognition performance of the mobile phone in noisy scenarios, ultimately impacting the user experience. Therefore, to avoid affecting the speech recognition performance of the speech recognition model, when the angle between the direction of the gradient of the audio denoising model (or the first model gradient) and the direction of the gradient of the speech recognition model (or the second model gradient) is greater than a first preset angle, the mobile phone can merge the gradients of the audio denoising model and the speech recognition model to obtain a synthesized gradient, which is the target second model gradient.

[0167] Specifically, the mobile phone can determine the target merging direction based on the direction of the gradient of the speech recognition model and a second preset angle. This target merging direction is the direction in which the gradients of the two models are merged. Then, the mobile phone can merge the gradients of the audio denoising model and the speech recognition model according to this target merging direction to obtain the synthesized model gradient. It can be understood that the angle between the target merging direction and the direction of the gradient of the speech recognition model is the second preset angle. This second preset angle is smaller than the first preset angle. In this embodiment, the first preset angle is 90 degrees, and the second preset angle is 45 degrees.

[0168] In some embodiments, the mobile phone can determine the target merging direction based on the direction of the gradient of the speech recognition model and the direction of the gradient of the audio denoising model, where the angle between the gradients is the second preset angle. The mobile phone can then map the gradients of the audio denoising model and the speech recognition model to this target merging direction, obtaining a first mapped gradient and a second mapped gradient. The mobile phone can then merge these two gradients to obtain a synthesized gradient. This reduces the occurrence of angle conflicts in the gradients due to large angles between the two models, improving not only the denoising effect of the audio denoising model but also reducing the impact of the audio denoising model's gradient on the speech recognition model, thus improving the speech recognition performance of the speech recognition model and ultimately enhancing the mobile phone's speech recognition performance.

[0169] For example, such as Figure 11 As shown, an example is given with the first preset angle being 90 degrees and the second preset angle being 45 degrees. Wherein, Figure 11 In the diagram, vector F1 represents the gradient of the audio denoising model, and vector F2 represents the gradient of the speech recognition model. It can be seen that the angle α between vectors F1 and F2 is greater than the first preset angle (90 degrees), indicating a gradient conflict between the audio denoising model's gradient and the speech recognition model's gradient. Therefore, to avoid this gradient conflict, the phone can use the direction of the speech recognition model's gradient—that is, the direction of vector F1—and the direction where the angle θ between it and vector F2 is the second preset angle (45 degrees) as the target merging direction. This target merging direction is vector F... 12 The corresponding direction. Then, the phone can map vector F1 onto this target merging direction to obtain vector F1', and map vector F2 onto this target merging direction to obtain vector F2'. Then, the phone can merge vectors F1' and F2' to obtain the synthesized model gradient, which is F. 12 .

[0170] In another implementation, if the numerical ratio between the gradient of the audio denoising model and the gradient of the speech recognition model is too small or too large, it may lead to a difference in the magnitude of the gradients of the two models, resulting in a decrease in the speech recognition capability of the speech recognition model. This affects the speech recognition performance of the speech recognition model, consequently impacting the phone's speech recognition performance in noisy scenarios and ultimately affecting the user experience. Therefore, to avoid affecting the speech recognition performance of the speech recognition model, when the numerical ratio between the gradients of the audio denoising model and the speech recognition model is outside a preset numerical range, the phone can process the gradients of the audio denoising model and / or the speech recognition model according to a second preset value within this preset numerical range to obtain the target gradients of the audio denoising model and / or the speech recognition model. The preset numerical range includes a first preset value and a second preset value, where the first preset value is less than the second preset value. In this embodiment, the first preset value is the opposite of the second preset value. That is, the preset numerical range is [-β, β], where -β is the first preset value and β is the second preset value.

[0171] In some embodiments, when the numerical ratio between the gradient of the audio denoising model and the gradient of the speech recognition model is outside a preset range, the mobile phone can use the ratio between this numerical ratio and a second preset value as the gradient adjustment ratio. Then, the mobile phone can adjust the gradients of the audio denoising model and / or the speech recognition model according to this gradient adjustment ratio to obtain the target gradients of the audio denoising model and / or the speech recognition model. In this way, by adjusting the model gradients, the probability of differences in the magnitude of model gradients can be reduced, thereby reducing the impact of a deterioration in the denoising capability of the audio denoising model on subsequent speech recognition capabilities, improving the convergence speed of the model gradients during joint training, and ultimately improving the training efficiency of the model.

[0172] In one example, the mobile phone can adjust the model gradient of the audio noise reduction model according to the gradient adjustment ratio described above to obtain the target model gradient of the audio noise reduction model. For example, please see Figure 12 For example, the gradient of the audio denoising model is set to 100, and the gradient of the speech recognition model is set to 10. Figure 12In this diagram, vector F3 represents the gradient of the audio noise reduction model, and vector F4 represents the gradient of the speech recognition model. Specifically, the phone can calculate the numerical ratio between vectors F3 and F4, obtaining a ratio of 100 ÷ 10 = 10. Then, the phone can use the ratio between this ratio and a second preset value (e.g., 1) as an adjustment ratio, which is 10 ÷ 1 = 10. Next, the phone can reduce vector F3 by a factor of 10 to obtain the target model gradient, i.e., vector F3', so that the numerical ratio between vector F3' and vector F4 is the second preset value. This solves the problem of differences in the magnitude of model gradients and improves the convergence speed of model gradients during joint training.

[0173] In another example, the mobile phone can adjust the gradient of the speech recognition model according to the gradient adjustment ratio described above to obtain the target model gradient of the speech recognition model. For example, let's continue with the explanation using an adjustment ratio of 10. The mobile phone can amplify the gradient of the speech recognition model by 10 times to obtain the target model gradient of the speech recognition model, so that the numerical ratio between the gradient of the audio denoising model and the target model gradient of the speech recognition model is a second preset value. This solves the problem of differences in the magnitude of model gradients and improves the convergence speed of the model gradient during joint training.

[0174] In another example, the phone can divide the gradient adjustment ratio according to a preset allocation ratio, obtaining a first adjustment ratio and a second adjustment ratio. This preset allocation ratio can be 0.5, 1.5, etc., and is not specifically limited. Then, the phone can adjust the model gradient of the audio denoising model according to the first adjustment ratio to obtain the target model gradient of the audio denoising model. Furthermore, the phone can adjust the target model gradient of the speech recognition model according to the second adjustment ratio to obtain the target model gradient of the speech recognition model. For example, using an adjustment ratio of 10 and a preset allocation ratio of 0.5 as an example, the phone can multiply the adjustment ratio of 10 by the preset allocation ratio of 0.5 to obtain a first adjustment ratio of 5 and a second adjustment ratio of 5. Then, the phone can reduce the model gradient of the audio denoising model by a factor of 5 to obtain the target model gradient of the audio denoising model, and amplify the model gradient of the speech recognition model by a factor of 5 to obtain the target model gradient of the speech recognition model. In this way, the numerical ratio between the target model gradient of the audio noise reduction model and the target model gradient of the speech recognition model can be the second preset value, thereby solving the problem of differences in the magnitude of model gradients and improving the convergence speed of model gradients during joint training.

[0175] It should be noted that the training process of the above-mentioned audio noise reduction model and speech recognition model does not have to be performed on a mobile phone. That is to say, the training process of the audio noise reduction model and speech recognition model can be performed on other devices and then the trained audio noise reduction model and speech recognition model can be sent to the mobile phone. Alternatively, the training process of the audio noise reduction model and speech recognition model can also be performed on a cloud server. There are no specific limitations.

[0176] It is understandable that, under the same noisy test scenario, the accuracy of speech recognition performed by a mobile phone varies depending on the model configuration. This model configuration characterizes the models involved and their training modes during the speech recognition process of the received raw audio data. For example, this model configuration may include at least one of the following: speech recognition only includes a speech recognition model; the speech recognition model and the audio noise reduction model are trained separately; the speech recognition model and the audio noise reduction model are jointly trained without audio compensation; and the speech recognition model and the audio noise reduction model are jointly trained with audio compensation. The following section, in conjunction with Table 1, describes the speech recognition accuracy of the mobile phone under different model configurations in a noisy scenario.

[0177] Table 1

[0178]

[0179] As shown in Table 1, in a noisy test scenario in a kitchen, with a distance of 80 cm between the phone and the user, the accuracy of the speech recognition result is 66.44% if the phone performs speech recognition solely based on the speech recognition model. If the phone trains both a pre-built speech recognition model and a pre-built audio noise reduction model separately, and then performs speech recognition on the original audio data using these two models, the accuracy is 78.35%. If the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, and then performs speech recognition on the original audio data using this jointly trained model, the accuracy is 81.05%. Finally, if the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, the accuracy is 76.44%. Next, the phone can use an audio noise reduction model to process the original audio data, obtaining noise-reduced audio data. Then, the phone performs audio compensation on the noise-reduced audio data to obtain the target audio data. Finally, a speech recognition model can be used to perform speech recognition on the target audio data, achieving an accuracy of 82.77%.

[0180] In the above noisy test scenario, a shopping mall, with a distance of 50 cm between the phone and the user, the accuracy of the speech recognition result is 47.33% if the phone performs speech recognition solely based on the speech recognition model. If the phone trains both a pre-built speech recognition model and a pre-built audio noise reduction model separately, resulting in separate speech recognition and noise reduction models, and then performs speech recognition on the original audio data using these separately trained models, the accuracy is 60.55%. If the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, resulting in separate speech recognition and noise reduction models, and then performs speech recognition on the original audio data using the jointly trained model, the accuracy is 67.03%. Finally, if the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, the accuracy is 67.03%. Next, the phone can perform noise reduction processing on the original audio data using an audio noise reduction model to obtain noise-reduced audio data. Then, the phone performs audio compensation on the noise-reduced audio data to obtain the target audio data. Finally, it can perform speech recognition on the target audio data using a speech recognition model, achieving an accuracy of 70.39%.

[0181] In the aforementioned noisy test scenario, inside a stationary vehicle with windows partially open and air conditioning set to high fan speed, the accuracy of speech recognition based solely on the phone's speech recognition model is 85.17%. If the phone trains both a pre-built speech recognition model and a pre-built audio noise reduction model separately, and then performs speech recognition on the original audio data using these two models, the accuracy is 94.49%. If the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, and then performs speech recognition on the original audio data using this jointly trained model, the accuracy is 94.71%. Finally, if the phone jointly trains both the pre-built speech recognition model and the pre-built audio noise reduction model, the accuracy is 94.71%. Next, the phone can use an audio noise reduction model to process the original audio data, obtaining noise-reduced audio data. Then, the phone performs audio compensation on the noise-reduced audio data to obtain the target audio data. Finally, a speech recognition model can be used to perform speech recognition on the target audio data, achieving an accuracy of 95.01%.

[0182] In the aforementioned noisy test scenario, inside a vehicle traveling on urban roads with windows closed and air conditioning set to high fan speed, the accuracy of speech recognition is 84.43% when the phone performs speech recognition solely based on the speech recognition model. If the phone trains both a pre-built speech recognition model and a pre-built audio denoising model separately, and then performs speech recognition on the original audio data using these two models, the accuracy is 93.13%. If the phone jointly trains both the pre-built speech recognition model and the pre-built audio denoising model, and then performs speech recognition on the original audio data using this jointly trained model, the accuracy is 93.62%. Finally, if the phone jointly trains both the pre-built speech recognition model and the pre-built audio denoising model, the accuracy is 93.62%. Next, the phone can perform noise reduction processing on the original audio data using an audio noise reduction model to obtain noise-reduced audio data. Then, the phone performs audio compensation on the noise-reduced audio data to obtain the target audio data. Finally, it can perform speech recognition on the target audio data using a speech recognition model, achieving an accuracy of 93.68%.

[0183] In the aforementioned noisy test scenario, inside a vehicle traveling on an elevated highway or expressway with windows closed and air conditioning set to high fan speed, the accuracy of speech recognition results is 81.65% when the phone performs speech recognition solely based on the speech recognition model. If the phone trains both a pre-built speech recognition model and a pre-built audio denoising model separately, and then performs speech recognition on the original audio data using these two models, the accuracy is 90.92%. If the phone jointly trains both the pre-built speech recognition model and the pre-built audio denoising model, and then performs speech recognition on the original audio data using this jointly trained model, the accuracy is 92.12%. Finally, if the phone jointly trains both the pre-built speech recognition model and the pre-built audio denoising model, the accuracy is 92.12%. Next, the phone can perform noise reduction processing on the original audio data using an audio noise reduction model to obtain noise-reduced audio data. Then, the phone performs audio compensation on the noise-reduced audio data to obtain the target audio data. Finally, it can perform speech recognition on the target audio data using a speech recognition model, achieving an accuracy of 92.53%.

[0184] It can be seen that, under the same noisy test scenario, the accuracy of speech recognition is lower when only the speech recognition model is trained, compared to the model configuration where the speech recognition model and audio denoising model are trained separately. Furthermore, the accuracy is lower when the speech recognition model and audio denoising model are trained jointly without audio compensation. Finally, the accuracy is lower when the speech recognition model and audio denoising model are trained jointly without audio compensation. In other words, under the same noisy test scenario, the highest accuracy for speech recognition is achieved when the speech recognition model and audio denoising model are trained jointly with audio compensation.

[0185] In some embodiments, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described above.

[0186] In some embodiments, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method described above.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech recognition method, characterized in that, Applied to electronic devices, the method includes: The electronic device acquires raw audio data; wherein the raw audio data is obtained based on voice commands issued by the user; The electronic device performs noise reduction processing on the original audio data according to the target audio noise reduction model to obtain noise-reduced frequency data; the target audio noise reduction model is trained based on a first sub-audio dataset and a second sub-audio dataset, the first sub-audio dataset includes multiple noise-free frequency data, and the second sub-audio dataset includes multiple noisy frequency data obtained by mixing the noise-free frequency data; The electronic device merges the original audio data and the noise-reduced frequency data according to the target compensation coefficient to obtain the target audio data. The target compensation coefficient is obtained by adjusting the learnable compensation coefficient based on the fourth sub-audio dataset. The fourth sub-audio dataset includes multiple pieces of second audio data obtained by merging the first audio data and the noisy frequency data in the second sub-audio dataset according to the learnable compensation coefficient. The first audio data is obtained by inputting the noisy frequency data into the target audio noise reduction model. The electronic device performs speech recognition on the target audio data according to the target speech recognition model to obtain the speech recognition result; In response to the speech recognition result, the electronic device performs an operation corresponding to the speech command; The target speech recognition model and the target audio denoising model are obtained by jointly training the audio denoising model and the speech recognition model to be trained based on a preset training strategy, according to a second loss value and a first loss value. The target audio denoising model is obtained by adjusting parameters based on a first model gradient, which is obtained by updating the gradient when the first loss value is less than a reference first loss value. The target speech recognition model is obtained by adjusting parameters based on a second model gradient, which is obtained by updating the gradient when the second loss value is less than a reference second loss value. The first loss value is obtained by inputting each noisy frequency data from the second sub-audio dataset into the audio denoising model to be trained, resulting in denoised noisy frequency data, and is determined based on the denoised noisy frequency data and the corresponding noiseless frequency data; the second loss value is obtained by inputting the denoised noisy frequency data into the speech recognition model to be trained, resulting in predicted text data, and is determined based on the predicted text data and the corresponding real text data of the noiseless frequency data.

2. The method according to claim 1, characterized in that, The target speech recognition model is trained based on a third sub-audio dataset and a text dataset. The third sub-audio dataset includes multiple noisy frequency data after noise reduction, and the text dataset includes multiple real text data, which are obtained based on noise-free frequency data corresponding to the noisy frequency data.

3. The method according to claim 2, characterized in that, The reference first loss value is determined based on the most recent one or more historical first loss values ​​before the training of the audio denoising model to be trained. The reference second loss value is determined based on the most recent one or more historical first loss values ​​prior to the training of the speech recognition model to be trained.

4. The method according to claim 3, characterized in that, The second model gradient is the target second model gradient, which is obtained by merging the first model gradient and the second model gradient when the angle between the direction of the first model gradient and the direction of the second model gradient is greater than a first preset angle.

5. The method according to claim 3, characterized in that, The second model gradient is the target second model gradient, which is obtained by adjusting the second model gradient according to the gradient adjustment ratio when the numerical ratio between the first model gradient and the second model gradient is outside a preset numerical range. And / or, The first model gradient is a target first model gradient, which is obtained by adjusting the first model gradient according to the gradient adjustment ratio when the numerical ratio between the first model gradient and the second model gradient is outside the preset numerical range; wherein, the gradient adjustment ratio is the ratio between the numerical ratio and a second preset value in the preset numerical range, the preset numerical range includes a first preset value and a second preset value, and the first preset value is less than the second preset value.

6. The method according to any one of claims 1-5, characterized in that, The electronic device merges the original audio data and the noise-reduced audio data according to the target compensation coefficient to obtain the target audio data, including: The electronic device uses the product of the target compensation coefficient and the noise-reduced audio data as the compensated audio data; The electronic device merges the compensated audio data with the original audio data to obtain the target audio data.

7. A model training method, characterized in that, Applied to electronic devices, the method includes: The electronic device acquires an audio dataset; wherein the audio dataset includes a first sub-audio dataset and a second sub-audio dataset, the first sub-audio dataset includes multiple noise-free frequency data, and the second sub-audio dataset includes multiple noisy frequency data, wherein the noisy frequency data is obtained by mixing noise based on the noise-free frequency data; For each noisy frequency data point in the second sub-audio dataset, the electronic device inputs the noisy frequency data into the audio denoising model to be trained to obtain the denoised noisy frequency data. The electronic device determines a first loss value based on the noise-reduced noisy frequency data and the noise-free frequency data corresponding to the noisy frequency data; The electronic device inputs the noise-reduced noisy frequency data into the speech recognition model to be trained to obtain predicted text data; and determines a second loss value based on the predicted text data and the real text data corresponding to the noise-free frequency data. Based on a preset training strategy, the audio denoising model and the speech recognition model to be trained are jointly trained according to the second loss value and the first loss value to obtain a target audio denoising model and a target speech recognition model. This includes: updating the gradient of the audio denoising model to be trained when the first loss value is less than a reference first loss value to obtain a first model gradient; adjusting the parameters of the audio denoising model to be trained according to the first model gradient until the first loss value is greater than or equal to the reference first loss value, and the number of iterations in which the first loss value is greater than or equal to the reference first loss value reaches a preset number of training iterations, thereby obtaining the target audio denoising model. If the second loss value is less than the reference second loss value, the gradient of the speech recognition model to be trained is updated to obtain the second model gradient; according to the second model gradient, the parameters of the speech recognition model to be trained are adjusted until the second loss value is greater than or equal to the reference second loss value, and the number of iterations in which the second loss value is greater than or equal to the reference second loss value reaches the preset number of training iterations, thereby obtaining the target speech recognition model; For each noisy frequency data point in the second sub-audio dataset, the noisy frequency data point is input into the target audio noise reduction model to obtain the first audio data; According to the learnable compensation coefficient, the first audio data and the noisy frequency data are merged to obtain the second audio data; the second audio data is input into the target speech recognition model to obtain the first text data; Based on the first text data and the real text data corresponding to the noisy frequency data, the learnable compensation coefficient is adjusted until the second loss value is greater than or equal to the reference second loss value to obtain the target compensation coefficient; the target compensation coefficient is used to merge the original audio data with the noise-reduced frequency data obtained by denoising the original audio data according to the target audio denoising model.

8. The method according to claim 7, characterized in that, in, The reference first loss value is determined based on the most recent one or more historical first loss values ​​prior to the training of the audio denoising model to be trained.

9. The method according to claim 7 or 8, characterized in that, The reference second loss value is determined based on the most recent one or more historical first loss values ​​prior to the training of the speech recognition model to be trained.

10. The method according to claim 9, characterized in that, After the electronic device updates the gradient of the speech recognition model to be trained to obtain the second model gradient, the method further includes: When the angle between the direction of the first model gradient and the direction of the second model gradient is greater than a first preset angle, the electronic device merges the first model gradient and the second model gradient to obtain the target second model gradient. The electronic device updates the second model gradient to the target second model gradient.

11. The method according to claim 10, characterized in that, The electronic device merges the first model gradient and the second model gradient to obtain the target second model gradient, including: The electronic device uses the direction where the gradient of the second model is located and the direction where the gradient of the first model is located are at a second preset angle as the target merging direction; wherein, the second preset angle is smaller than the first preset angle; The electronic device maps the first model gradient and the second model gradient to the target merging direction to obtain the mapped first model gradient and the mapped second model gradient. The electronic device merges the gradient of the first mapping model and the gradient of the second mapping model to obtain the gradient of the target second model.

12. The method according to claim 9, characterized in that, After the electronic device updates the gradient of the speech recognition model to be trained to obtain the second model gradient, the method further includes: When the numerical ratio between the first model gradient and the second model gradient is outside a preset numerical range, the electronic device uses the ratio between the numerical ratio and the second preset value within the preset numerical range as the gradient adjustment ratio. The electronic device adjusts the gradient of the first model and / or the gradient of the second model according to the gradient adjustment ratio to obtain the target gradient of the first model and / or the target gradient of the second model; wherein, the preset value range includes a first preset value and a second preset value, and the first preset value is less than the second preset value; The electronic device updates the first model gradient to the target first model gradient, and / or updates the second model gradient to the target second model gradient.

13. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display screen and the one or more memories; the one or more memories are used to store computer program code, the computer program code including computer instructions that, when the one or more processors execute the computer instructions, cause the electronic device to perform the method as claimed in any one of claims 1-6 or the method as claimed in any one of claims 7 to 12.

14. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6 or the method as described in any one of claims 7 to 12.

15. A computer program product, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6 or the method as described in any one of claims 7 to 12.

Images