Speech recognition methods, speech recognition systems, computer equipment and storage media

Abstract

This application provides a speech recognition method, a speech recognition system, a computer device, and a storage medium, belonging to the field of financial technology. The method includes: inputting target speech with a preset emotion category into a pre-trained multi-task speech recognition model; encoding the target speech using a first speech coding sub-model to obtain initial speech features; performing speech attention processing on the initial speech features using a first attention sub-model to obtain first target attention features; encoding the initial speech features using a second speech coding sub-model to obtain hidden speech features; performing hidden attention processing on the first target attention features and the hidden speech features using a second attention sub-model to obtain second target attention features; and performing speech classification on the second target attention features using a multi-task classification sub-model to obtain a target speech label. This application embodiment can improve the recognition accuracy of multi-task speech recognition.

Description

Technical Field

[0001] This application relates to the field of financial technology, and in particular to a speech recognition method, a speech recognition system, a computer device, and a storage medium. Background Technology

[0002] Currently, as consumers have increasingly higher demands for the service levels of financial institutions and banks, banks have established dedicated systems to monitor the service quality of their telephone agents, including both human and AI-powered agents. Real-time monitoring and alerting of the emotions of both telephone agents and consumers during calls is an effective way to improve the consumer experience and enhance service levels. However, when performing multi-task speech recognition, related technologies often collect a large amount of irrelevant feature information, and the speech recognition of different sub-tasks is prone to interference, affecting the accuracy of multi-task speech recognition. In practical applications, these technologies cannot accurately and promptly identify the current emotions of consumers, thus failing to provide timely emotional reassurance. Therefore, improving the accuracy of multi-task speech recognition, enabling timely and accurate identification of changes in user voice emotions, and providing appropriate voice feedback has become a pressing technical problem that needs to be solved. Summary of the Invention

[0003] The main objective of this application is to propose a speech recognition method, a speech recognition system, a computer device, and a storage medium that can improve the recognition accuracy of multi-task speech recognition and can promptly and accurately identify changes in the user's emotional state in speech and provide speech feedback.

[0004] To achieve the above objectives, a first aspect of this application proposes a speech recognition method, the method comprising:

[0005] Acquire target speech with preset emotion categories;

[0006] The target speech is input into a pre-trained multi-task speech recognition model, which includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model.

[0007] The target speech is processed by speech coding using the first speech coding sub-model to obtain initial speech features;

[0008] The initial speech features are processed by the first attention sub-model to obtain the first target attention features;

[0009] The initial speech features are processed by the second speech coding sub-model to obtain hidden speech features;

[0010] The second target attention feature is obtained by performing hidden attention processing on the first target attention feature and the hidden speech feature using the second attention sub-model.

[0011] The second target attention features are processed by the multi-task classification sub-model to obtain the target speech label.

[0012] In some embodiments, the step of performing speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features includes:

[0013] The initial speech features are input into the first attention sub-model for feature parameterization to obtain initial speech index features, initial speech content features, and initial speech query features.

[0014] The initial speech index features, the initial speech content features, and the initial speech query features are processed using an attention mechanism to obtain the first target attention features.

[0015] In some embodiments, the step of performing hidden attention processing on the first target attention feature and the hidden speech feature using the second attention sub-model to obtain the second target attention feature includes:

[0016] The first target attention feature and the hidden speech feature are input into the second attention sub-model for feature parameterization processing to obtain the target speech index feature, the target speech content feature and the target speech query feature;

[0017] Update the target speech query features based on the hidden speech features;

[0018] The target speech index features, the target speech content features, and the target speech query features are processed using an attention mechanism to obtain the second target attention features.

[0019] In some embodiments, prior to inputting the target speech into a pre-trained multi-task speech recognition model, the method further includes:

[0020] Training the multi-task speech recognition model specifically includes:

[0021] Obtain sample speech with the preset emotion category and sample speech tags for the sample speech;

[0022] The sample speech is input into a pre-constructed initial speech recognition model, which includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and the multi-task classification sub-model.

[0023] The sample speech is processed by the first speech coding sub-model to obtain the initial sample speech features.

[0024] The initial sample speech features are processed by the first attention sub-model to obtain the first sample attention features;

[0025] The initial sample speech features are processed by the second speech coding sub-model to obtain hidden sample speech features;

[0026] The second attention feature is obtained by performing hidden attention processing on the first sample attention feature and the hidden sample speech feature through the second attention sub-model.

[0027] The second sample attention features are processed by the multi-task classification sub-model to perform speech classification, and the predicted speech labels are obtained.

[0028] The parameters of the initial speech recognition model are adjusted based on the sample speech labels and the predicted speech labels to obtain the multi-task speech recognition model.

[0029] In some embodiments, the predicted speech label includes a predicted emotion attribute label and a predicted speech cultural background label, and the sample speech label includes a sample emotion attribute label and a sample speech cultural background label. The step of adjusting the parameters of the initial speech recognition model based on the sample speech label and the predicted speech label to obtain the multi-task speech recognition model includes:

[0030] The sentiment attribute loss is calculated on the sample sentiment attribute labels and the predicted sentiment attribute labels to obtain sentiment attribute loss data.

[0031] Speech culture background loss is calculated on the sample speech culture background labels and the predicted speech culture background labels to obtain speech culture loss data.

[0032] The parameters of the initial speech recognition model are adjusted based on the emotional attribute loss data and the speech culture loss data to obtain the multi-task speech recognition model.

[0033] In some embodiments, the multi-task classification sub-model includes a fully connected layer and a normalization layer. The step of performing speech classification processing on the attention features of the second sample using the multi-task classification sub-model to obtain predicted speech labels includes:

[0034] Based on the fully connected layer, the attention features of the second sample are processed for feature classification to obtain the predicted speech classification features;

[0035] The predicted speech classification features are normalized according to the normalization layer to obtain predicted speech probability data;

[0036] The predicted speech label is obtained based on the predicted speech probability data.

[0037] In some embodiments, the step of performing speech encoding processing on the target speech using the first speech coding sub-model to obtain initial speech features includes:

[0038] The target speech is subjected to feature encoding processing to obtain initial speech encoding features;

[0039] The initial speech coding features are subjected to position coding to obtain the target speech coding features;

[0040] The target speech coding features are subjected to feature transformation processing to obtain initial speech features.

[0041] To achieve the above objectives, a second aspect of this application provides a speech recognition system, the system comprising:

[0042] The voice acquisition module is used to acquire target voices with preset emotion categories;

[0043] A voice input module is used to input the target voice into a pre-trained multi-task speech recognition model, the multi-task speech recognition model including a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model;

[0044] The first speech coding module is used to perform speech coding processing on the target speech through the first speech coding sub-model to obtain initial speech features;

[0045] The first attention processing module is used to perform speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features;

[0046] The second speech coding module is used to perform hidden feature coding on the initial speech features through the second speech coding sub-model to obtain hidden speech features;

[0047] The second attention processing module is used to perform hidden attention processing on the first target attention feature and the hidden speech feature through the second attention sub-model to obtain the second target attention feature.

[0048] The speech classification module is used to perform speech classification processing on the second target attention features through the multi-task classification sub-model to obtain the target speech label.

[0049] To achieve the above objectives, a third aspect of this application provides a computer device, comprising:

[0050] At least one memory;

[0051] At least one processor;

[0052] At least one computer program;

[0053] The computer program is stored in the memory, and the processor executes the at least one computer program to implement the method described in any of the first aspect embodiments above.

[0054] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program for causing a computer to perform the method described in any one of the embodiments of the first aspect above.

[0055] The speech recognition method, speech recognition system, computer device, and storage medium proposed in this application first acquire target speech with a preset emotion category and input the target speech into a pre-trained multi-task speech recognition model. This multi-task speech recognition model includes a first speech encoding sub-model, a first attention sub-model, a second speech encoding sub-model, a second attention sub-model, and a multi-task classification sub-model. The first speech encoding sub-model performs speech encoding processing on the target speech to obtain initial speech features. The first attention sub-model performs speech attention processing on the initial speech features to obtain first target attention features. Then, the second speech encoding sub-model performs hidden feature encoding processing on the initial speech features to obtain hidden speech features. The second attention sub-model then performs hidden attention processing on the first target attention features and the hidden speech features to obtain second target attention features. This application embodiment uses the output of the second speech encoding sub-model as part of the input of the second attention sub-model, enabling the improved multi-task speech recognition model to effectively extract deeper-dimensional emotional feature information from a speech segment. Finally, the multi-task classification sub-model performs speech classification processing on the second target attention features to obtain target speech labels. This application embodiment combines the output of attention processing and the output of feature encoding, which can avoid collecting a large amount of irrelevant feature information. Therefore, when the target speech is recognized by the multi-task speech recognition model provided in this application embodiment, the recognition accuracy of multi-task speech recognition can be effectively improved, so as to timely and accurately identify the user's voice emotion changes and make voice feedback. Attached Figure Description

[0056] Figure 1 This is a first flowchart of the speech recognition method provided in the embodiments of this application;

[0057] Figure 2 This is a flowchart of the training process of the multi-task speech recognition model provided in the embodiments of this application;

[0058] Figure 3 yes Figure 1 A flowchart illustrating the specific method of step S130;

[0059] Figure 4 yes Figure 1 A flowchart illustrating the specific method of step S140;

[0060] Figure 5 yes Figure 1 A flowchart illustrating the specific method of step S160;

[0061] Figure 6 This is a schematic diagram of a model structure of the initial speech recognition model provided in this application embodiment;

[0062] Figure 7 This is a schematic diagram of another model structure of the initial speech recognition model provided in the embodiments of this application;

[0063] Figure 8 yes Figure 2 A flowchart illustrating the specific method of step S270;

[0064] Figure 9 yes Figure 2 A flowchart illustrating the specific method of step S280;

[0065] Figure 10 This is a block diagram of the speech recognition system provided in the embodiments of this application;

[0066] Figure 11 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0070] First, let's analyze some of the terms used in this application:

[0071] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0072] Attention mechanism: The attention mechanism enables neural networks to focus on a subset of their input (or features), selecting specific inputs. It can be applied to any type of input regardless of its shape. In situations with limited computational power, the attention mechanism is a primary resource allocation scheme for addressing information overload problems, allocating computational resources to more important tasks.

[0073] Normalization refers to scaling data features proportionally to make them fall into a small, specific range, thereby removing the unit limitations of the data features and converting them into dimensionless pure numerical values, which facilitates comparison and weighting using indicators of different units or magnitudes.

[0074] Currently, as customers have increasingly higher demands for the service levels of financial institutions and banks, banks have established dedicated systems to monitor the service quality of their telephone agents, including both human and AI-powered agents. Real-time monitoring and alerting of the emotions of both the telephone agents and the customers during calls is an effective way to improve customer experience and service levels. However, when performing multi-task speech recognition, related technologies often collect a large amount of irrelevant feature information, and the speech recognition of different sub-tasks is prone to interference, affecting the accuracy of multi-task speech recognition. In practical applications, these technologies cannot accurately and promptly identify the current emotions of the customer, thus failing to provide timely emotional reassurance. Therefore, improving the accuracy of multi-task speech recognition to accurately and promptly identify changes in the user's emotional state and provide appropriate voice feedback has become a pressing technical problem that needs to be solved.

[0075] Based on this, embodiments of this application provide a speech recognition method, a speech recognition system, a computer device, and a storage medium, which can improve the recognition accuracy of multi-task speech recognition, thereby enabling timely and accurate recognition of changes in the user's voice emotion to provide voice feedback.

[0076] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0077] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0078] The speech recognition method provided in this application relates to the field of artificial intelligence technology. The speech recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the speech recognition method, but is not limited to the above forms.

[0079] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] Please refer to Figure 1 , Figure 1 This is an optional flowchart of the speech recognition method provided in the embodiments of this application. In some embodiments of this application, the speech recognition method specifically includes, but is not limited to, steps S110 to S170, which are described below in conjunction with... Figure 1 These seven steps will be explained in detail.

[0081] Step S110: Obtain target speech with a preset emotion category;

[0082] Step S120: Input the target speech into a pre-trained multi-task speech recognition model. The multi-task speech recognition model includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model.

[0083] Step S130: Perform speech coding processing on the target speech using the first speech coding sub-model to obtain initial speech features;

[0084] Step S140: Perform speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features;

[0085] Step S150: The initial speech features are processed by hidden feature encoding through the second speech coding sub-model to obtain hidden speech features;

[0086] Step S160: Perform hidden attention processing on the first target attention features and hidden speech features through the second attention sub-model to obtain the second target attention features;

[0087] Step S170: The second target attention features are processed for speech classification using a multi-task classification sub-model to obtain the target speech label.

[0088] It should be noted that in practical application environments, the speech recognition method provided in this application embodiment can be executed by a terminal or a server, or by a terminal and a server working together. Specifically, the terminal communicates with the server via a network. For example, taking the execution of this speech recognition method on a corresponding terminal as an example, the terminal acquires target speech with a preset emotion category from its local storage, performs speech recognition processing on the target speech according to a trained multi-task speech recognition model, and obtains the target speech label under the multi-task classification condition. This multi-task speech recognition model is trained by the terminal or server based on pre-acquired sample speech and corresponding sample speech labels, and is deployed on the terminal. This speech recognition method can also be deployed on a server so that the server can also implement the steps of the above speech recognition method, which will not be elaborated further here.

[0089] In step S110 of some embodiments, sudden vocalizations such as laughter, sobbing, and screaming in human emotional communication contain a great deal of hidden information that is effective for voice emotion recognition. This information is of certain importance in emotional expression and communication. The voice recognition method provided in this application embodiment is used to perform voice recognition on target voices with preset emotion categories, and can promptly and accurately identify changes in the user's voice emotion, thereby providing voice feedback. The preset emotion category is used to characterize the category of preset sudden vocalizations, such as laughter, sobbing, and screaming, which characterize vocalizations resulting from sudden emotional shifts in the target object.

[0090] It should be noted that, prior to step S110, this embodiment may further include: acquiring the initial speech of the target object, performing burst speech recognition on the initial speech, and obtaining target speech with a preset emotion category. Specifically, pitch distribution detection is performed on the initial speech to obtain speech pitch distribution data, which includes multiple pitch frame data. Pitch frame judgment is performed from the first pitch frame data of the speech pitch distribution data. If the pitch difference between the currently judged pitch frame data and its previous pitch frame data exceeds a preset pitch threshold, then the currently judged pitch frame data is determined to be the initial pitch frame data, and pitch frame judgment continues for subsequent pitch frame data. If the pitch difference between the currently judged pitch frame data and its previous pitch frame data is again detected to exceed the preset pitch threshold, then the currently judged pitch frame data is determined to be the end pitch frame data. Therefore, the initial speech is truncated using the speech position corresponding to the initial pitch frame data and the speech position corresponding to the end pitch frame data to obtain the target speech. Next, speech emotion recognition is performed on the target speech and the preset burst speech recognition model to obtain the preset emotion category of the target speech. The preset burst speech recognition model is trained based on a constructed burst sample set, which includes multiple burst speech instances and their corresponding burst emotion categories. The burst emotion categories are set in the same way as the preset emotion categories mentioned above, and will not be repeated here.

[0091] It should be noted that the model structure of the burst speech recognition model can be any of the following, including Dynamic Neural Network-Hidden Markov Model (DNN-HMM), N-Gram language model, or sentiment analysis model based on finite weighted state transition machine. This application does not limit the burst speech recognition model.

[0092] The speech recognition method of this application can be used to assist in speech recognition of the target's emotions in applications such as intelligent customer service, intelligent vehicles, and smart homes. For example, in the scenario of intelligent customer service in financial banks, when the intelligent customer service representative is communicating with the target, if it suddenly recognizes the target's emotion as sobbing or crying, it will prompt the intelligent customer service representative to invoke a script based on the change in the target's emotion, that is, to invoke a soothing speech script to comfort the target. When the target's emotion is recognized to be stable, business communication continues. Therefore, the speech recognition method provided by this application can identify sudden sounds in the target's speech, and then invoke the corresponding speech script based on the identified emotional category of the sudden sound, effectively improving the user experience.

[0093] In step S120 of some embodiments, in order to avoid interference from a large amount of redundant and irrelevant feature information, the target speech is input into a pre-trained multi-task speech recognition model. The multi-task speech recognition model includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model. The embodiments of this application construct a multi-task speech recognition model by combining self-supervised learning and attention mechanisms, which can better combine the various sub-tasks of speech recognition.

[0094] In step S130 of some embodiments, in order to extract the sound feature information of the target speech, firstly, the target speech is processed by speech coding through a first speech coding sub-model to obtain the initial speech features of the target speech, that is, to initially extract the sound feature information of the target speech.

[0095] In step S140 of some embodiments, in order to capture richer feature information, the initial speech features are subjected to speech attention processing through a first attention sub-model to determine the speech features of the most important emotional information expressed by the target object from the initial speech features.

[0096] In step S150 of some embodiments, the initial speech features are processed by hidden feature encoding through a second speech coding sub-model. The convolution kernel of the convolutional layer corresponding to the second speech coding sub-model is smaller than the convolution kernel of the convolutional layer corresponding to the first speech coding sub-model, so that the obtained hidden speech features can effectively extract speech detail feature information.

[0097] In step S160 of some embodiments, existing speech recognition models tend to collect a large number of redundant and irrelevant features, thus failing to effectively extract deep semantic information of the target speech. Furthermore, the existing model connection structure is prone to error accumulation. To avoid these issues, this application connects the output layer of the second speech coding sub-model and the output layer of the first attention sub-model to the input layer of the second attention sub-model. That is, the second attention sub-model performs hidden attention processing on the first target attention features and hidden speech features, thereby combining the output of the attention processing with the output of the feature encoding, enabling the extraction of higher-dimensional speech information.

[0098] In step S170 of some embodiments, since the speech signal in the real environment is a complex mixed signal, it contains not only rich semantic information, but also the relevant identity information, emotional information and environmental information of the target object. In order to enable speech recognition to recognize the multi-dimensional information contained in the speech signal at the same time, and to make it more in line with intelligent human-computer interaction speech recognition processing, the embodiments of this application use a multi-task classification sub-model to perform speech classification processing on the second target attention features to obtain target speech labels. This can realize multi-task speech recognition such as speech emotion attribute recognition and target object identity background recognition at the same time. Among them, speech emotion attribute is used to represent information such as speech emotion intensity and speech emotion type, and target object identity background recognition can be used to represent the gender information, environmental information, speech cultural background information, etc. of the target object.

[0099] It should be noted that the target speech labels obtained by this application based on the multi-task classification sub-model can be multi-dimensional labels, and when each dimension of the label is used for speech classification using its corresponding specific task classifier, the label state of each dimension can be in the form of one-hot, so as to ensure that the specific task classifier corresponding to each dimension can be executed normally.

[0100] It should be noted that, for example, the multi-task classification sub-model of this application includes four specific task classifiers, namely the first task classifier, the second task classifier, the third task classifier, and the fourth task classifier. The obtained target speech label can be represented as (a, b, c, d), where a represents the specific category of the target speech classified under the first task classifier, b represents the specific category of the target speech classified under the second task classifier, c represents the specific category of the target speech classified under the third task classifier, and d represents the specific category of the target speech classified under the fourth task classifier. Thus, the target speech can be more comprehensively and accurately identified and analyzed based on the obtained multi-dimensional target speech label, thereby effectively improving the recognition accuracy of multi-task speech recognition.

[0101] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the training process of the multi-task speech recognition model provided in the embodiments of this application. In some embodiments of this application, before step S120, the speech recognition method further includes: training the multi-task speech recognition model. This training process may include, but is not limited to, steps S210 to S280. The following describes the process in conjunction with... Figure 2 These eight steps will be explained in detail.

[0102] Step S210: Obtain sample speech with preset emotion categories and sample speech labels for the sample speech;

[0103] Step S220: Input the sample speech into the pre-constructed initial speech recognition model. The initial speech recognition model includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model.

[0104] Step S230: Perform speech coding processing on the sample speech using the first speech coding sub-model to obtain the initial sample speech features;

[0105] Step S240: Perform speech attention processing on the initial sample speech features through the first attention sub-model to obtain the first sample attention features;

[0106] Step S250: The initial sample speech features are processed by hidden feature encoding through the second speech coding sub-model to obtain hidden sample speech features;

[0107] Step S260: The attention features of the first sample and the speech features of the hidden sample are processed by the second attention sub-model to obtain the attention features of the second sample.

[0108] Step S270: The attention features of the second sample are processed for speech classification using a multi-task classification sub-model to obtain the predicted speech label;

[0109] Step S280: Adjust the parameters of the initial speech recognition model based on the sample speech labels and the predicted speech labels to obtain a multi-task speech recognition model.

[0110] In some embodiments, step S210 involves acquiring the sample speech in the same way as acquiring the target speech described above, and will not be repeated here. The sample speech label is a multi-dimensional label pre-labeled according to each classification task in the multi-task classification sub-model. This sample speech label is used as the target recognition result for speech recognition of the sample speech.

[0111] In step S220 of some embodiments, the sample speech is input into a pre-built initial speech recognition model, which includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model. In this embodiment, the multi-task speech recognition model is constructed by combining self-supervised learning and attention mechanisms, which can better combine the various sub-tasks of speech recognition.

[0112] It should be noted that this initial speech recognition model can be built based on the Wav2vec2.0 pre-trained speech model, and the structure of the model can be improved.

[0113] In step S230 of some embodiments, in order to extract the sound feature information of the target speech, the sample speech is processed by speech coding through the first speech coding sub-model to obtain the initial sample speech features.

[0114] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the specific method of step S130 provided in an embodiment of this application. In some embodiments of this application, step S130 specifically includes, but is not limited to, steps S310 to S330. The following is in conjunction with… Figure 3 These three will be explained in detail.

[0115] Step S310: Perform feature encoding processing on the target speech to obtain initial speech encoding features;

[0116] Step S320: Perform positional encoding processing on the initial speech coding features to obtain the target speech coding features;

[0117] Step S330: Perform feature transformation processing on the target speech coding features to obtain the initial speech features.

[0118] In steps S310 to S330 of some embodiments, the first speech coding sub-model includes a speech coding layer, a position coding layer, and a coding attention layer. The speech coding layer performs feature coding on the target speech to obtain initial speech coding features, which effectively extracts speech feature information. Then, the position coding layer performs position coding on the initial speech coding features to obtain target speech coding features. These target speech coding features embed positional information, enabling the initial speech recognition model to determine the position of each speech information in the sample speech within the feature sequence, thereby assigning a unique feature representation to each position. Finally, the coding attention layer performs feature transformation on the target speech coding features, i.e., feature transformation is performed according to the global attention mechanism in the Transform structure to obtain the initial speech features.

[0119] It should be noted that the speech coding layer can be set as a seven-layer convolutional network. The first convolutional layer is a group-normalized convolution, which divides the data channels of the target speech sequence into many groups, calculates the mean and variance of each group, and then normalizes each group to obtain normalized features. This speech group normalization layer can accelerate the model convergence speed and alleviate the gradient vanishing problem. The last convolutional layer of the speech coding layer is a layer-normalized convolution, which normalizes the channels within each batch of feature data in the encoded features obtained from the previous layer, thereby ensuring the stability of the data distribution of the corresponding convolutional layer.

[0120] In step S240 of some embodiments, in order to capture richer feature information, the initial sample speech features are processed by a first attention sub-model. Specifically, this application can use a first attention sub-model based on a multi-head attention structure to process the initial sample speech features, which can effectively avoid the deviation that may be caused by the same attention mechanism, and determine the most important emotional information of the target object when making speech expressions from the initial sample speech features.

[0121] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the specific method of step S140 provided in the embodiments of this application. In some embodiments, step S140 may specifically include, but is not limited to, steps S410 and S420, which are described below in conjunction with... Figure 4 These two steps will be explained in detail.

[0122] Step S410: Input the initial speech features into the first attention sub-model for feature parameterization to obtain the initial speech index features, initial speech content features, and initial speech query features;

[0123] Step S420: Apply attention mechanism processing to the initial speech index features, initial speech content features, and initial speech query features to obtain the first target attention features.

[0124] In steps S410 and S420 of some embodiments, the attention sub-model refers to a model that, when solving a problem in a specific scenario, assigns different weights (i.e., attention scores) to different information that needs to be considered in solving the problem. Information that is more helpful to the problem is given higher weights, while information that is less helpful is given lower weights, thereby better utilizing this information to solve the problem. In this embodiment, initial speech features are input into the first attention sub-model. Feature parameterization processing is performed on the initial speech features according to three preset parameter matrices to obtain initial speech index features K1, initial speech content features Q1, and initial speech query features V1. Attention mechanism processing is then applied to the initial speech index features K1, initial speech content features Q1, and initial speech query features V1. Specifically, an initial attention score is obtained based on the initial speech content features Q1 and the initial speech index features K1. The initial attention score is normalized to obtain a target attention score. A weighted calculation is then performed based on the target attention score and the initial speech query features V1 to obtain the first target attention feature.

[0125] In step S250 of some embodiments, the initial sample speech features are processed by hidden feature encoding using a second speech coding sub-model. The convolution kernel of the convolutional layer corresponding to the second speech coding sub-model is smaller than that of the convolutional layer corresponding to the first speech coding sub-model, thereby enabling the obtained hidden sample speech features to effectively extract speech detail feature information. The second speech coding sub-model includes a positional coding layer and a coding attention layer, which have the same structure as the corresponding layers in the first speech coding sub-model, and will not be described again here.

[0126] In step S260 of some embodiments, this application connects the output layer of the second speech coding sub-model and the output layer of the first attention sub-model to the input layer of the second attention sub-model. That is, the second attention sub-model performs hidden attention processing on the first target attention features and hidden speech features, thereby combining the output of attention processing and the output of feature encoding, and extracting higher-dimensional speech information from them.

[0127] Please refer to Figure 5 , Figure 5 This is a flowchart of step S160 provided in an embodiment of this application. In some embodiments, step S160 specifically includes, but is not limited to, steps S510 to S530, which are described below in conjunction with... Figure 5 These three steps will be explained in detail.

[0128] Step S510: Input the first target attention features and hidden speech features into the second attention sub-model for feature parameterization processing to obtain target speech index features, target speech content features and target speech query features;

[0129] Step S520: Update the target speech query features based on the hidden speech features;

[0130] Step S530: Apply attention mechanism processing to the target speech index features, target speech content features, and target speech query features to obtain the second target attention features.

[0131] In steps S510 to S530 of some embodiments, the first target attention feature and the hidden speech feature are input into the second attention sub-model for feature parameterization processing to obtain the target speech index feature K2, the target speech content feature V2, and the target speech query feature Q2. The target speech query feature Q2 is updated according to the hidden speech feature output by the second speech coding sub-model, and the target speech index feature K2, the target speech content feature V2, and the target speech query feature Q2 are then processed by an attention mechanism, meaning that the target speech query feature Q2 at this point is the hidden speech feature. This embodiment of the application combines the output of attention processing and the output of feature coding, enabling the extraction of higher-dimensional speech information.

[0132] For example, please refer to Figure 6 The initial speech recognition model includes a first speech coding sub-model 610, a first attention sub-model 620, a second speech coding sub-model 630, a second attention sub-model 640, and a multi-task classification sub-model 650. The output of the first speech coding sub-model 610 is connected to the first attention sub-model 620 and the second speech coding sub-model 630, respectively. The first attention sub-model 620 performs speech attention processing on the initial sample speech features output by the first speech coding sub-model 610 to obtain first sample attention features. The second speech coding sub-model 630 performs hidden feature encoding processing on the initial sample speech features to obtain hidden sample speech features. The second attention sub-model 640 performs hidden attention processing on the first sample attention features and the hidden sample speech features, and the resulting second sample attention features are input into the multi-task classification sub-model 650 for speech classification processing.

[0133] For example, please refer to Figure 7The initial speech recognition model further includes a third speech coding sub-model 660 and a third attention sub-model 670. The third speech coding sub-model 660 has the same structure as the second speech coding sub-model 630, and the third attention sub-model 670 has the same structure as the second attention sub-model 640. The output of the second speech coding sub-model 630 is connected to the input of the third speech coding sub-model 660, the output of the second attention sub-model 640 is connected to the input of the third attention sub-model 670, and the output of the third speech coding sub-model 660 is connected to the input of the third attention sub-model 670. This embodiment of the application can perform deep feature encoding processing on the hidden sample speech features through the third speech coding sub-model 660 to update the hidden sample speech features. The third attention sub-model 670 performs deep hidden attention processing on the second sample attention features and the updated hidden sample speech features, updates the second sample attention features based on the processing results, and inputs the updated second sample attention features into the multi-task classification sub-model 650 for speech classification processing to obtain predicted speech labels.

[0134] It should be noted that the multi-task speech recognition model obtained in this application embodiment is obtained by inputting the output of each speech coding sub-model and the output of the attention sub-model connected to the speech coding sub-model of the previous layer into a new attention sub-model, thereby extracting deeper speech information with higher attention weight.

[0135] In step S270 of some embodiments, in order to enable speech recognition to simultaneously recognize the multi-dimensional information contained in the speech signal, the embodiments of this application perform speech classification processing on the second target attention features through a multi-task classification sub-model to obtain target speech labels, which can realize multi-task speech recognition such as speech attribute recognition and target object identity background recognition at the same time.

[0136] Please refer to Figure 8 , Figure 8 This is a flowchart of step S270 provided in an embodiment of this application. In some embodiments, the multi-task classification sub-model includes a fully connected layer and a normalization layer. Step S270 may specifically include, but is not limited to, steps S810 to S830, as described below. Figure 8 These three steps will be explained in detail.

[0137] Step S810: Perform feature classification processing on the attention features of the second sample based on the fully connected layer to obtain the predicted speech classification features;

[0138] Step S820: Normalize the predicted speech classification features according to the normalization layer to obtain the predicted speech probability data.

[0139] Step S830: Obtain the predicted speech label based on the predicted speech probability data.

[0140] In steps S810 to S830 of some embodiments, the multi-task classification sub-model includes multiple classifiers, and each classifier includes a fully connected layer and a normalization layer. The attention features of the second sample are processed by the fully connected layer to obtain the predicted speech classification features, and the predicted speech classification features are normalized by the normalization layer to obtain the predicted speech probability data. The predicted speech probability data are numerically compared, and the label with the highest probability data is determined as the predicted speech label.

[0141] For example, a multi-task classification sub-model includes four classifiers: an emotion intensity classifier, a two-dimensional emotion attribute classifier, a speech cultural context classifier, and an emotion type classifier. The emotion intensity classifier represents the classification based on ten labeled high-dimensional emotion expression tags, such as awe, fear, surprise, and excitement, with each high-dimensional emotion expression associated with the intensity of each type of sound. The two-dimensional emotion attribute classifier represents the classification of speech emotion attributes based on emotional arousal and valence. Emotional valence is divided into positive and negative emotions, representing self-evaluation of emotional attributes; emotional arousal represents the activation level of emotion, for example, an arousal and valence of 1 indicates unpleasantness / suppression, 5 indicates neutral emotion, and 9 indicates pleasantness / stimulation. A speech cultural context classifier is used to classify emotions based on cross-cultural high-dimensional emotional context. This classifier is trained on a pre-constructed cultural dataset whose labels are built from 10 dimensions and 4 standard cultural contexts, containing 40 emotion intensities. Furthermore, based on the specific cultural structure, feature information of emotions within each cultural context is constructed. An emotion type classifier is used to classify specific sudden sound types, such as panting, laughing, screaming, and grumbling. It should be noted that there is no specific limit to the number of classifiers included in the multi-task classification sub-model, and the output of the last attention sub-model is connected to the input of each classifier.

[0142] In step S280 of some embodiments, the parameters of the initial speech recognition model are adjusted based on the predicted speech labels and corresponding sample speech labels obtained by each classifier in the multi-task classification sub-model to obtain a multi-task speech recognition model. The model structure of this multi-task speech recognition model is the same as that of the initial speech recognition model after parameter adjustment. Therefore, this application, through the classification and recognition of multi-dimensional attribute information using a multi-task classification sub-model, can improve the recognition accuracy of multi-task speech recognition, i.e., obtain more accurate target speech labels, and can promptly and accurately identify changes in the user's emotional state in their speech.

[0143] Please refer to Figure 9 , Figure 9 This is a flowchart of step S280 provided in some embodiments of this application. In some embodiments, the predicted speech label includes a predicted emotion attribute label and a predicted speech cultural background label, and the sample speech label includes a sample emotion attribute label and a sample speech cultural background label. Step S280 may specifically include, but is not limited to, steps S910 to S930.

[0144] Step S910: Calculate the sentiment attribute loss for the sample sentiment attribute labels and the predicted sentiment attribute labels to obtain the sentiment attribute loss data.

[0145] Step S920: Calculate the speech culture background loss for the sample speech culture background labels and the predicted speech culture background labels to obtain speech culture loss data;

[0146] Step S930: Adjust the parameters of the initial speech recognition model based on the emotional attribute loss data and the speech culture loss data to obtain a multi-task speech recognition model.

[0147] In steps S910 to S930 of some embodiments, when the multi-task classification sub-model includes a two-dimensional sentiment attribute classifier and a speech cultural background classifier, sentiment attribute loss is calculated for the sample sentiment attribute labels and predicted sentiment attribute labels to obtain sentiment attribute loss data; speech cultural background loss is calculated for the sample speech cultural background labels and predicted speech cultural background labels to obtain speech cultural loss data. A weighted calculation is performed based on the sentiment attribute loss data and the speech cultural loss data to obtain a total loss value, and the parameters of the initial speech recognition model are adjusted based on the total loss value to obtain a multi-task speech recognition model.

[0148] It should be noted that when the multi-task classification sub-model includes four classifiers: an emotion intensity classifier, a two-dimensional emotion attribute classifier, a speech cultural background classifier, and an emotion type classifier, the predicted speech labels also include predicted emotion intensity labels and predicted emotion type labels, and the sample speech labels also include sample emotion intensity labels and sample emotion type labels. Therefore, emotion intensity loss is calculated on the predicted emotion intensity labels and sample emotion intensity labels to obtain emotion intensity loss data, and emotion type loss is calculated on the predicted emotion type labels and sample emotion type labels to obtain emotion type loss data. A weighted calculation is performed based on the emotion intensity loss data, emotion type loss data, and the total loss value to update the total loss value. The parameters of the initial speech recognition model are then adjusted based on the updated total loss value to obtain the multi-task speech recognition model.

[0149] For example, in the scenario of intelligent or human customer service in financial banks, when intelligent or human customer service representatives communicate with a target user via voice, they collect and identify the target user's voice in real time. This collected voice is then input into a pre-trained multi-task speech recognition model to obtain a more accurate voice label through the identification of multi-dimensional attribute information. For instance, if a sudden sound, such as a scream, is detected in the target voice, it indicates a significant change in the target user's emotional state, potentially hindering normal communication. Therefore, the intelligent or human customer service representative is prompted via page display or notification to invoke pre-stored voice prompts for dealing with screaming emotions, such as "What happened to you?" or "Do you need any help?" During the communication, pre-stored reassurance phrases, such as "I understand how you feel, I'll handle it for you immediately," are also invoked to soothe the target user. Subsequently, in the real-time monitoring and recognition of the target voice, when no sudden sound is detected in the target voice, and the emotional category of the target voice is the same as the voice label before the emotional change, meaning the target user's emotional state has returned to normal, business communication continues.

[0150] Please refer to Figure 10 , Figure 10 This is a block diagram of a speech recognition system provided in some embodiments of this application. In some embodiments, the speech recognition system may specifically include a speech acquisition module 1010, a speech input module 1020, a first speech encoding module 1030, a first attention processing module 1040, a second speech encoding module 1050, a second attention processing module 1060, and a speech classification module 1070.

[0151] The voice acquisition module 1010 is used to acquire target voice with a preset emotion category;

[0152] The voice input module 1020 is used to input the target voice into a pre-trained multi-task speech recognition model. The multi-task speech recognition model includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model.

[0153] The first speech coding module 1030 is used to perform speech coding processing on the target speech through the first speech coding sub-model to obtain initial speech features;

[0154] The first attention processing module 1040 is used to perform speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features;

[0155] The second speech coding module 1050 is used to perform hidden feature coding on the initial speech features through the second speech coding sub-model to obtain hidden speech features.

[0156] The second attention processing module 1060 is used to perform hidden attention processing on the first target attention features and hidden speech features through the second attention sub-model to obtain the second target attention features.

[0157] The speech classification module 1070 is used to perform speech classification processing on the second target attention features through a multi-task classification sub-model to obtain the target speech label.

[0158] It should be noted that the speech recognition system in this application embodiment is used to implement the above speech recognition method. The speech recognition system in this application embodiment corresponds to the aforementioned speech recognition method. For the specific processing procedure, please refer to the aforementioned speech recognition method, which will not be described in detail here.

[0159] This application also provides a computer device, including:

[0160] At least one memory;

[0161] At least one processor;

[0162] At least one computer program;

[0163] The program is stored in memory, and the processor executes at least one computer program to implement the speech recognition method described in the embodiments of this application. The computer device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0164] The following is combined with Figure 11 The computer devices described in the embodiments of this application will be described in detail.

[0165] like Figure 11 , Figure 11 The hardware structure of a computer device according to another embodiment is illustrated, the computer device including:

[0166] The processor 1110 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0167] The memory 1120 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1120 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1120 and called and executed by the processor 1110 using the speech recognition method of the embodiments of this application.

[0168] The input / output interface 1130 is used to implement information input and output;

[0169] The communication interface 1140 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0170] Bus 1150 transmits information between various components of the device (e.g., processor 1110, memory 1120, input / output interface 1130, and communication interface 1140);

[0171] The processor 1110, memory 1120, input / output interface 1130 and communication interface 1140 are connected to each other within the device via bus 1150.

[0172] This application also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the speech recognition method described above.

[0173] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0174] This application provides a speech recognition method, speech recognition system, computer device, and storage medium. The method involves acquiring target speech with a preset emotion category and inputting it into a pre-trained multi-task speech recognition model. This multi-task speech recognition model includes a first speech encoding sub-model, a first attention sub-model, a second speech encoding sub-model, a second attention sub-model, and a multi-task classification sub-model. The first speech encoding sub-model encodes the target speech to obtain initial speech features. These initial speech features are then input into the first attention sub-model for feature parameterization, resulting in initial speech index features, initial speech content features, and initial speech query features. An attention mechanism is then applied to these initial speech index features, initial speech content features, and initial speech query features to obtain first target attention features. Subsequently, the first target attention features and hidden speech features are input into the second attention sub-model for feature parameterization, resulting in target speech index features, target speech content features, and target speech query features. The target speech query features are updated based on the hidden speech features, and an attention mechanism is then applied to these target speech index features, target speech content features, and target speech query features to obtain second target attention features. This embodiment of the application uses the output of the second speech coding sub-model as part of the input of the second attention sub-model, enabling the effective extraction of deeper-dimensional emotional features from a speech segment through an improved multi-task speech recognition model. The second target attention features are then processed for speech classification using a multi-task classification sub-model to obtain target speech labels. This embodiment uses hidden activation vectors from a multi-layer Transformer as input to multiple attention sub-models constructed based on a multi-head attention network, fully utilizing the correlation between features from different tasks. Furthermore, by combining the output of attention processing and the output of feature encoding, this embodiment avoids collecting a large amount of irrelevant feature information. Therefore, when performing speech recognition on target speech using the multi-task speech recognition model provided in this embodiment, the recognition accuracy of multi-task speech recognition can be effectively improved.

[0175] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0176] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0177] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0179] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0180] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speech recognition method, characterized in that, The method includes: Acquire target speech with preset emotion categories; The target speech is input into a pre-trained multi-task speech recognition model, which includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model. The target speech is processed by speech coding using the first speech coding sub-model to obtain initial speech features; The initial speech features are processed by the first attention sub-model to obtain the first target attention features; The initial speech features are processed by the second speech coding sub-model to obtain hidden speech features; The first target attention feature and the hidden speech feature are subjected to hidden attention processing by the second attention sub-model to obtain the second target attention feature; wherein, the step of performing hidden attention processing on the first target attention feature and the hidden speech feature by the second attention sub-model to obtain the second target attention feature includes: inputting the first target attention feature and the hidden speech feature into the second attention sub-model for feature parameterization processing to obtain target speech index feature, target speech content feature and target speech query feature; updating the target speech query feature according to the hidden speech feature; and performing attention mechanism processing on the target speech index feature, target speech content feature and target speech query feature to obtain the second target attention feature; The second target attention features are processed by the multi-task classification sub-model to obtain the target speech label.

2. The method according to claim 1, characterized in that, The step of performing speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features includes: The initial speech features are input into the first attention sub-model for feature parameterization to obtain initial speech index features, initial speech content features, and initial speech query features. The initial speech index features, the initial speech content features, and the initial speech query features are processed using an attention mechanism to obtain the first target attention features.

3. The method according to claim 1, characterized in that, Before inputting the target speech into a pre-trained multi-task speech recognition model, the method further includes: Training the multi-task speech recognition model specifically includes: Obtain sample speech with the preset emotion category and sample speech tags for the sample speech; The sample speech is input into a pre-constructed initial speech recognition model, which includes a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model. The sample speech is processed by the first speech coding sub-model to obtain the initial sample speech features. The initial sample speech features are processed by the first attention sub-model to obtain the first sample attention features; The initial sample speech features are processed by the second speech coding sub-model to obtain hidden sample speech features; The second attention feature is obtained by performing hidden attention processing on the first sample attention feature and the hidden sample speech feature through the second attention sub-model. The second sample attention features are processed by the multi-task classification sub-model to perform speech classification, and the predicted speech labels are obtained. The parameters of the initial speech recognition model are adjusted based on the sample speech labels and the predicted speech labels to obtain the multi-task speech recognition model.

4. The method according to claim 3, characterized in that, The predicted speech labels include predicted emotion attribute labels and predicted speech cultural background labels, and the sample speech labels include sample emotion attribute labels and sample speech cultural background labels. The step of adjusting the parameters of the initial speech recognition model based on the sample speech labels and the predicted speech labels to obtain the multi-task speech recognition model includes: The sentiment attribute loss is calculated on the sample sentiment attribute labels and the predicted sentiment attribute labels to obtain sentiment attribute loss data. Speech culture background loss is calculated on the sample speech culture background labels and the predicted speech culture background labels to obtain speech culture loss data. The parameters of the initial speech recognition model are adjusted based on the emotional attribute loss data and the speech culture loss data to obtain the multi-task speech recognition model.

5. The method according to claim 3, characterized in that, The multi-task classification sub-model includes a fully connected layer and a normalization layer. The step of performing speech classification processing on the attention features of the second sample using the multi-task classification sub-model to obtain predicted speech labels includes: Based on the fully connected layer, the attention features of the second sample are processed for feature classification to obtain the predicted speech classification features; The predicted speech classification features are normalized according to the normalization layer to obtain predicted speech probability data; The predicted speech label is obtained based on the predicted speech probability data.

6. The method according to any one of claims 1 to 5, characterized in that, The step of performing speech encoding processing on the target speech through the first speech encoding sub-model to obtain initial speech features includes: The target speech is subjected to feature encoding processing to obtain initial speech encoding features; The initial speech coding features are subjected to position coding to obtain the target speech coding features; The target speech coding features are subjected to feature transformation processing to obtain initial speech features.

7. A speech recognition system, characterized in that, The system includes: The voice acquisition module is used to acquire target voices with preset emotion categories; A voice input module is used to input the target voice into a pre-trained multi-task speech recognition model, the multi-task speech recognition model including a first speech coding sub-model, a first attention sub-model, a second speech coding sub-model, a second attention sub-model, and a multi-task classification sub-model; The first speech coding module is used to perform speech coding processing on the target speech through the first speech coding sub-model to obtain initial speech features; The first attention processing module is used to perform speech attention processing on the initial speech features through the first attention sub-model to obtain the first target attention features; The second speech coding module is used to perform hidden feature coding on the initial speech features through the second speech coding sub-model to obtain hidden speech features; The second attention processing module is used to perform hidden attention processing on the first target attention feature and the hidden speech feature through the second attention sub-model to obtain the second target attention feature; wherein, the step of performing hidden attention processing on the first target attention feature and the hidden speech feature through the second attention sub-model to obtain the second target attention feature includes: inputting the first target attention feature and the hidden speech feature into the second attention sub-model for feature parameterization processing to obtain target speech index feature, target speech content feature and target speech query feature; updating the target speech query feature according to the hidden speech feature; and performing attention mechanism processing on the target speech index feature, target speech content feature and target speech query feature to obtain the second target attention feature; The speech classification module is used to perform speech classification processing on the second target attention features through the multi-task classification sub-model to obtain the target speech label.

8. A computer device, characterized in that, include: At least one memory; At least one processor; At least one computer program; The computer program is stored in the memory, and the processor executes the at least one computer program to achieve: The method as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is used to cause the computer to execute: The method as described in any one of claims 1 to 6.

Images