Digital human speech recognition method based on multi-task learning

Through multi-task learning and multiple information fusion solutions, the problem of insufficient recognition of speakers, dialects and emotional information in speech interactions except for voice text content is solved, efficient and accurate multi-dimensional information recognition is achieved, and the speech recognition system is reduced delay and computing power overhead.

CN119028319APending Publication Date: 2024-11-26GIANT MOBILE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411247393.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art fails to effectively identify speakers, dialects and emotional information other than speech text content in voice interaction, resulting in insufficient comprehensive effects and increased delay.

Method used

Using a digital human speech recognition method based on multitasking learning, the speech features are extracted through a multitasking shared audio encoder and sent them to text, speakers, dialects and emotion decoder for processing, and multitasking learning is carried out according to task importance weighted loss Training, combining Transformer and convolutional structure, realizes the fusion of multi-dimensional information.

Benefits of technology

It realizes accurate and real-time recognition of user voice, improves recognition accuracy, reduces computing power and time overhead, and improves the working efficiency of the voice recognition system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028319A_ABST
    Figure CN119028319A_ABST
Patent Text Reader

Abstract

The invention relates to a digital human speech recognition method based on multi-task learning. The method comprises the following steps: S1, collecting and marking a plurality of pieces of data; s2, inputting a plurality of different tasks into an audio encoder by adopting a mode of sharing the audio encoder by multiple tasks; s3, after the audio encoder extracts voice features of each frame of each piece of data, the voice features are sent to different decoders to be processed, the decoders comprise a text decoder, a speaker decoder, a dialect decoder and an emotion decoder, and the different decoders output corresponding processing results; s4, multi-task learning training is carried out, and the mode is as follows: according to the weight of each task, weighted summation is carried out on the loss of each task, and an overall loss function is obtained; s5, completing multi-task network training according to the overall loss function; and S6, obtaining a recognition result according to multi-task network reasoning. According to the invention, the voice text content of the user and information such as the speaker, the dialect and the emotion can be accurately recognized in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a digital human speech recognition method based on multi-task learning. Background Art

[0002] For digital human and other application scenarios, information such as speakers, dialects, and emotions are very necessary for the subsequent dialogue response experience. However, existing technologies do not consider other recognition requirements in voice interaction besides voice recognition, or simply superimpose different functions, resulting in insufficient comprehensive effects and greatly increased latency.

[0003] Therefore, it is necessary to provide a digital human speech recognition method based on multi-task learning to accurately and real-time recognize the user's voice text content, as well as information such as the speaker, dialect, and emotion. Summary of the invention

[0004] The purpose of the present invention is to provide a digital human speech recognition method based on multi-task learning to accurately and in real time recognize the user's speech text content, as well as information such as the speaker, dialect and emotion.

[0005] In order to solve the problems existing in the prior art, the present invention provides a digital human speech recognition method based on multi-task learning, comprising the following steps:

[0006] S1: Collect and annotate multiple data, each of which covers a variety of different tasks, including text, speakers, dialects, and emotions;

[0007] S2: Using a multi-task shared audio encoder, multiple different tasks are input into the audio encoder;

[0008] S3: After the audio encoder extracts the speech features of each frame of each data, it sends them to different decoders for processing, including text decoder, speaker decoder, dialect decoder and emotion decoder. Each different decoder outputs the corresponding processing results.

[0009] S4: Perform multi-task learning training in the following way: determine the weight of each task according to the overall performance of the system and the importance of each task; based on the weight of each task, perform weighted summation of the losses of each task to obtain the overall loss function;

[0010] S5: According to the overall loss function, back-propagate and update all network parameters to complete multi-task network training;

[0011] S6: Based on the trained multi-task network, the recognition result is obtained by inference.

[0012] Optionally, in the digital human speech recognition method based on multi-task learning, the speakers include different genders and different age groups;

[0013] Emotions cover different emotions and their intensity.

[0014] Optionally, in the digital human speech recognition method based on multi-task learning, the text, speaker, dialect and emotion of each speech are marked in S1.

[0015] Optionally, in the digital human speech recognition method based on multi-task learning, the text decoder adopts a Transformer structure, and after being processed by the text decoder, the recognized text is output;

[0016] The speaker decoder adopts a convolutional structure, and after being processed by the speaker decoder, it outputs the speaker information;

[0017] The dialect decoder adopts a convolutional structure. After being processed by the dialect decoder, it outputs the dialect category or Mandarin.

[0018] The emotion decoder adopts a convolutional structure. After being processed by the emotion decoder, it outputs the emotion category and emotion intensity.

[0019] Optionally, in the digital human speech recognition method based on multi-task learning, after the text decoder processes and before the recognized text is output, the following steps are also included: customized post-processing.

[0020] Optionally, in the digital human speech recognition method based on multi-task learning, customized post-processing includes inverse text regularization, automatic punctuation, fusion language model decoding and hot word enhancement.

[0021] Optionally, in the digital human speech recognition method based on multi-task learning, pronunciation definition is performed inside the dialect decoder so that the same Chinese character corresponds to different pronunciations in different dialects.

[0022] Optionally, in the digital human speech recognition method based on multi-task learning, after the dialect decoder is processed and before the dialect category or Mandarin is output, the following steps are further included:

[0023] After the dialect decoder obtains the dialect category information, it performs dialect decoding enhancement processing according to the pronunciation definition to obtain the Chinese character information of the corresponding dialect.

[0024] In the digital human speech recognition method based on multi-task learning provided by the present invention, information such as text, speaker, dialect and emotion can be accurately and efficiently recognized through the user's voice input. Multi-task learning and multiple information fusion schemes are used to make full use of information of different dimensions, and the accuracy rate is higher than that of a single task model. At the same time, the use of multi-tasks effectively reduces the computing power and time overhead, ensuring the working efficiency of the speech recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flow chart of a method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The specific implementation of the present invention will be described in more detail below in conjunction with the schematic diagram. The advantages and features of the present invention will become clearer based on the following description. It should be noted that the drawings are all in a very simplified form and are not in exact proportions, and are only used to facilitate and clearly assist in explaining the purpose of the embodiments of the present invention.

[0027] Hereinafter, if the method described herein includes a series of steps, the order in which the steps are presented herein is not necessarily the only order in which the steps may be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0028] For digital human and other application scenarios, information such as speakers, dialects, and emotions are very necessary for the subsequent dialogue response experience. However, existing technologies do not consider other recognition requirements in voice interaction besides voice recognition, or simply superimpose different functions, resulting in insufficient comprehensive effects and greatly increased latency.

[0029] In order to solve the problems existing in the prior art, the present invention provides a digital human speech recognition method based on multi-task learning, comprising the following steps:

[0030] S1: Collect and annotate multiple data, each of which covers a variety of different tasks, including text, speakers, dialects, and emotions;

[0031] Specifically, the speakers include different genders and age groups; the emotions include different emotions and emotional intensities.

[0032] When annotating, it is necessary to objectively and accurately annotate the text, speaker, dialect and emotion of each speech.

[0033] Preferably, the method may further include the following steps: performing voice activity detection.

[0034] S2: Using a multi-task shared audio encoder, multiple different tasks are input into the audio encoder;

[0035] The encoder adopts the Transformer structure, which occupies most of the parameters and computational complexity of the entire system, so the operating efficiency of the entire system can be close to that of a single speech recognition model.

[0036] The overall architecture of Transformer can be divided into two parts: encoder and decoder, both of which are based on the self-attention mechanism. The encoder is responsible for converting the input sequence into an intermediate representation, while the decoder is responsible for converting the intermediate representation into the target sequence.

[0037] S3: After the audio encoder extracts the speech features of each frame of each data, it sends them to different decoders for processing, including text decoder, speaker decoder, dialect decoder and emotion decoder. Each different decoder outputs the corresponding processing results.

[0038] Specifically, the text decoder adopts a Transformer structure, and after being processed by the text decoder, it outputs the recognized text; the speaker decoder adopts a convolutional structure, and after being processed by the speaker decoder, it outputs the speaker information; the dialect decoder adopts a convolutional structure, and after being processed by the dialect decoder, it outputs the dialect category or Mandarin; the emotion decoder adopts a convolutional structure, and after being processed by the emotion decoder, it outputs the emotion category and emotion intensity.

[0039] S4: Perform multi-task learning training as follows:

[0040] Determine the weight of each task based on the overall performance of the system and the importance of each task;

[0041] According to the weight of each task, the loss of each task is weighted summed to obtain the overall loss function;

[0042] S5: According to the overall loss function, back-propagate and update all network parameters to complete multi-task network training;

[0043] S6: Based on the trained multi-task network, the recognition result is obtained by inference.

[0044] Preferably, after the text decoder is processed and before the recognized text is output, the following steps are further included: customized post-processing. The customized post-processing includes inverse text normalization (ITN), automatic punctuation, fusion language model decoding, and hot word enhancement.

[0045] In one embodiment: after the speaker decoder obtains the speaker information, the post-processing module may perform targeted customization, such as performing hot word enhancement for the speaker-specific hot word list.

[0046] Preferably, pronunciation is defined within the dialect decoder so that the same Chinese character corresponds to different pronunciations in different dialects.

[0047] After the dialect decoder is processed and before the dialect category or Mandarin is output, the following steps are also included: after the dialect decoder obtains the dialect category information, it performs dialect decoding enhancement processing according to the pronunciation definition to obtain the Chinese character information of the corresponding dialect, and sends the obtained Chinese character information to the text decoder, thereby achieving the purpose of distinguishing homophonic but different characters in different dialects.

[0048] In summary, in the digital human speech recognition method based on multi-task learning provided by the present invention, information such as text, speaker, dialect and emotion can be accurately and efficiently recognized through the user's voice input. Multi-task learning and multiple information fusion schemes are used to make full use of information of different dimensions, and the accuracy rate is higher than that of a single task model. At the same time, the use of multiple tasks effectively reduces the computing power and time overhead, ensuring the working efficiency of the speech recognition system.

[0049] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.

Claims

1. A digital human speech recognition method based on multi-task learning, It is characterized in that The following steps are involved: S1: Collect and annotate multiple data, each of which covers a variety of different tasks, including text, speakers, dialects, and emotions; S2: Using a multi-task shared audio encoder, multiple different tasks are input into the audio encoder; S3: After the audio encoder extracts the speech features of each frame of each data, it sends them to different decoders for processing, including text decoder, speaker decoder, dialect decoder and emotion decoder. Each different decoder outputs the corresponding processing results. S4: Perform multi-task learning training in the following way: determine the weight of each task according to the overall performance of the system and the importance of each task; based on the weight of each task, perform weighted summation of the losses of each task to obtain the overall loss function; S5: According to the overall loss function, back-propagate and update all network parameters to complete multi-task network training; S6: Based on the trained multi-task network, the recognition result is obtained by inference.

2. The digital human speech recognition method based on multi-task learning as claimed in claim 1, It is characterized in that The speakers cover different genders and age groups; Emotions cover different emotions and their intensity.

3. The digital human speech recognition method based on multi-task learning as claimed in claim 1, It is characterized in that S1 annotates the text, speaker, dialect, and emotion of each speech.

4. The digital human speech recognition method based on multi-task learning as claimed in claim 1, It is characterized in that The text decoder adopts the Transformer structure, and after being processed by the text decoder, the recognized text is output; The speaker decoder adopts a convolutional structure, and after being processed by the speaker decoder, it outputs the speaker information; The dialect decoder adopts a convolutional structure. After being processed by the dialect decoder, it outputs the dialect category or Mandarin. The emotion decoder adopts a convolutional structure. After being processed by the emotion decoder, it outputs the emotion category and emotion intensity.

5. The digital human speech recognition method based on multi-task learning as claimed in claim 4, It is characterized in that After the text decoder processes and before outputting the recognized text, the following steps are also included: customized post-processing.

6. The digital human speech recognition method based on multi-task learning as claimed in claim 5, It is characterized in that Customized post-processing includes inverse text regularization, automatic punctuation, fusion language model decoding, and hot word enhancement.

7. The digital human speech recognition method based on multi-task learning as claimed in claim 4, It is characterized in that The pronunciation is defined within the dialect decoder so that the same Chinese character has different pronunciations in different dialects.

8. The digital human speech recognition method based on multi-task learning as claimed in claim 7, It is characterized in that After the dialect decoder is processed and before the dialect category or Mandarin is output, the following steps are also included: After the dialect decoder obtains the dialect category information, it performs dialect decoding enhancement processing according to the pronunciation definition to obtain the Chinese character information of the corresponding dialect.