Speech processing method and apparatus, computer device, readable storage medium, and program product

Abstract

The application relates to a speech processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining to-be-recognized speech; pre-processing the to-be-recognized speech; inputting the pre-processed to-be-recognized speech into a speech recognition model for processing to obtain to-be-recognized text; inputting the to-be-recognized text into a semantic understanding model for processing to obtain semantic data; inputting the semantic data into an emotion classification model for processing to obtain an emotion category corresponding to the to-be-recognized speech; and generating a reply speech of the to-be-recognized speech based on the semantic data and the emotion category, and playing the reply speech. The method can improve speech recognition accuracy and user experience.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology and the accelerated advancement of automotive intelligence, in-vehicle voice recognition and dialogue systems have gradually become an important means of human-machine interaction in vehicles. This system can provide corresponding voice responses to the driver's voice, greatly improving driving convenience and safety.

[0003] Current in-vehicle voice recognition systems collect user voice data, convert it into corresponding acoustic features, and then convert those features into a text sequence. The system then provides a corresponding voice response based on the recognized text sequence. However, this method can only perform simple recognition of user voice data and cannot understand the deeper meaning of the user's voice. It also suffers from low voice recognition accuracy and issues such as mechanical and stiff responses that fail to establish a natural communication with the driver, resulting in a poor user experience. Summary of the Invention

[0004] Therefore, it is necessary to provide a speech processing method, device, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of speech recognition and enhance the user experience, in order to address the aforementioned technical problems.

[0005] In a first aspect, this application provides a speech processing method, the method comprising:

[0006] Acquire the speech to be recognized;

[0007] The speech to be recognized is preprocessed;

[0008] The preprocessed speech to be recognized is input into the speech recognition model for processing to obtain the text to be recognized;

[0009] The text to be recognized is input into the semantic understanding model for processing to obtain semantic data;

[0010] The semantic data is input into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified.

[0011] Based on the semantic data and the emotion category, a response voice for the speech to be identified is generated and played.

[0012] In one embodiment, generating the response speech based on the semantic data and the emotion category includes:

[0013] If the speech to be identified is a question-type speech, a response speech is generated based on the semantic data and the emotion category.

[0014] If the speech to be recognized is a command-type speech, the command corresponding to the command-type speech is executed, and a response speech is generated based on the semantic data and the emotion category.

[0015] In one embodiment, the step of inputting the semantic data into an emotion classification model for processing to obtain the emotion category corresponding to the speech to be identified includes:

[0016] The semantic data is preprocessed and then converted into a vector to be identified.

[0017] The vector to be identified is input into the sentiment classification model, and the target features, context features and question-and-answer interaction features of the vector to be identified are extracted by the sentiment classification model. The question-and-answer interaction features represent the mutual influence between the question-type speech and the corresponding response speech.

[0018] The target features and context features are fused to obtain fused features;

[0019] The emotion category is determined based on the fusion features and the question-and-answer interaction features.

[0020] In one embodiment, the step of processing the preprocessed speech to be recognized into a speech recognition model to obtain the text to be recognized includes:

[0021] The preprocessed speech to be recognized is input into the speech recognition model;

[0022] The preprocessed speech to be recognized is processed by the speech recognition model to obtain speech sequence features; and based on the speech sequence features and preset normalization parameters, a normalization result is obtained, and the text to be recognized is obtained based on the normalization result.

[0023] The normalization parameter is obtained by training on a batch of speech samples, and the normalization parameter is related to the mean of the sample vectors corresponding to the batch of speech samples.

[0024] In one embodiment, the step of processing the text to be recognized into a semantic understanding model to obtain semantic data includes:

[0025] The text to be identified is input into the semantic understanding model, and the text to be identified is processed through the embedding layer in the semantic understanding model to obtain the processed text to be identified.

[0026] The semantic understanding model processes the processed text to be recognized through an encoding layer to obtain semantic understanding features; wherein, the encoding layer includes a feedforward neural network and a multi-head attention mechanism;

[0027] The semantic understanding features are processed by the fully connected layer in the semantic understanding model to obtain the semantic data.

[0028] In one embodiment, the method further includes:

[0029] Based on the speech to be identified, the text to be identified, the semantic data, and the emotion category, a file to be encrypted is generated;

[0030] The file to be encrypted is encrypted to obtain an encrypted file, and the encrypted file is stored.

[0031] Secondly, this application also provides a voice processing apparatus, the apparatus comprising:

[0032] The acquisition module is used to acquire the speech to be recognized;

[0033] The preprocessing module is used to preprocess the speech to be recognized;

[0034] The first processing module is used to input the preprocessed speech to be recognized into the speech recognition model for processing, so as to obtain the text to be recognized.

[0035] The second processing module is used to process the text to be recognized into the semantic understanding model to obtain semantic data;

[0036] The classification module is used to input the semantic data into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified.

[0037] The generation module is used to generate a response voice for the speech to be identified based on the semantic data and the emotion category, and to play the response voice.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the methods of various embodiments.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods of the various embodiments.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods of the various embodiments.

[0041] The aforementioned speech processing methods, apparatus, computer devices, computer-readable storage media, and computer program products, after obtaining the speech to be recognized, can preprocess the speech, then input the preprocessed speech into a speech recognition model for processing to obtain the text to be recognized. By inputting the text to be recognized into a semantic understanding model for processing, semantic data is obtained, which can understand deeper semantics and recognize complex user speech, thus improving the accuracy of speech recognition. At the same time, the semantic data is input into an emotion classification model for processing to obtain the emotion category corresponding to the speech to be recognized. Based on the semantic data and the emotion category, a response speech is generated for the speech to be recognized and played. The generation of the response speech takes into account the emotion category, realizing a human-like response and avoiding problems such as mechanical and stiff responses that fail to establish natural communication with the driver, thereby improving the user experience. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a speech processing method in one embodiment;

[0044] Figure 2 This is a flowchart illustrating the speech processing method in another embodiment;

[0045] Figure 3 Here is a flowchart of the Kalman filtering algorithm;

[0046] Figure 4 Here is a diagram of the DeepSpeech2 model structure;

[0047] Figure 5 Here is a diagram of the BERT-CNN model structure;

[0048] Figure 6 This is a structural diagram of the sentiment analysis model;

[0049] Figure 7 This is a flowchart illustrating the speech processing method in another embodiment;

[0050] Figure 8 This is a schematic diagram of the speech processing device.

[0051] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] Existing in-vehicle voice recognition technology has some limitations in voice recognition and dialogue, specifically as follows: (1) In the in-vehicle environment, due to various noise interferences (such as wind noise, road noise, and conversations of other passengers), the accuracy of existing in-vehicle voice recognition technology in recognizing user voice is often affected. In addition, differences in accents, speaking speeds, and pronunciation habits among different drivers can also lead to a decrease in recognition accuracy; (2) Existing in-vehicle voice recognition systems can often only recognize simple instructions or keywords, and have limited ability to understand complex sentences or semantics; (3) Existing in-vehicle voice responses are mechanical and stiff, and cannot establish natural communication with drivers. This makes drivers feel unnatural and uncomfortable when using it, reducing the user experience; (4) In-vehicle voice systems involve drivers' voice data, and how to protect the privacy and security of this data has become an important issue. Existing systems still need to be strengthened in terms of data security and privacy protection, and cannot prevent data leakage or abuse. (5) At present, drivers have increasingly diverse requirements for in-vehicle voice response. Some in-vehicle voice recognition systems may consider noise interference, learn to understand complex sentences, or establish natural communication with drivers. However, this can only meet the single needs of drivers. Therefore, how to design an in-vehicle voice recognition system to meet the diverse needs of users is an urgent technical problem to be solved.

[0054] In one embodiment, such as Figure 1 As shown, a voice processing method is provided. This embodiment uses the application of this method to a vehicle as an example for illustration. The method includes the following steps:

[0055] Step 102: Obtain the speech to be recognized;

[0056] The voice to be recognized can be a question or a command uttered by the driver through the vehicle's microphone.

[0057] Step 104: Preprocess the speech to be recognized;

[0058] In this embodiment, the preprocessing includes noise reduction, which can improve the clarity and signal-to-noise ratio of the speech signal.

[0059] Step 106: Input the preprocessed speech to be recognized into the speech recognition model for processing to obtain the text to be recognized;

[0060] In this embodiment, the speech recognition model uses the deepSpeech2 algorithm to parse and recognize speech, convert it into text, and then forms corresponding text based on the converted text as the text to be recognized.

[0061] Step 108: Input the text to be recognized into the semantic understanding model for processing to obtain semantic data;

[0062] Semantic understanding models primarily perform semantic analysis on the text output by speech recognition models to understand the user's intent and needs;

[0063] The semantic understanding model employs the bidirectional encoder representations from tnsformers-cnvolutional neural networks (BERT-CNN) method. This algorithm is a general language understanding model trained on massive corpora, which can not only describe the semantic and syntactic relationships between words, but also describe the relational features within sentences and even between sentences.

[0064] Step 110: Input the semantic data into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be recognized;

[0065] Semantic data can be input into an emotion classification model for processing. The emotion classification model can analyze the emotions in speech, identify the user's emotional state, and then classify it into the corresponding emotion category.

[0066] The emotion classification model utilizes feature fusion and bidirectional attention mechanisms to analyze the emotions in a user's speech, determine the user's emotional state, and provide an emotional basis for subsequent anthropomorphic dialogue.

[0067] Step 112: Based on semantic data and sentiment category, generate a response voice for the speech to be recognized and play the response voice.

[0068] Based on semantic data and emotion categories, it generates human-like response voices or executes commands indicated by command-type voices.

[0069] The anthropomorphic response voice is played to the user through the car's audio system.

[0070] As can be seen, after obtaining the speech to be recognized, this application can preprocess the speech, and then input the preprocessed speech into a speech recognition model for processing to obtain the text to be recognized. By inputting the text to be recognized into a semantic understanding model for processing, semantic data is obtained, which can understand deeper semantics and recognize complex user speech, thus improving the accuracy of speech recognition. At the same time, the semantic data is input into a sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be recognized. Based on the semantic data and sentiment category, a response speech is generated and played. The generation of the response speech takes into account the sentiment category, realizing a human-like response and avoiding problems such as mechanical and stiff responses that fail to establish a natural communication with the driver, thereby improving the user experience.

[0071] In one embodiment, generating a response voice for the speech to be identified based on semantic data and sentiment category includes: if the speech to be identified is a question-type speech, generating a response voice for the speech to be identified based on semantic data and sentiment category; if the speech to be identified is a command-type speech, executing the command corresponding to the command-type speech, and generating a response voice for the speech to be identified based on semantic data and sentiment category.

[0072] In this embodiment, the speech to be recognized can be a question-type speech. When a driver asks a question, he or she may have different emotions. At this time, it is necessary to consider the semantic data and emotional category of the question-type speech, and then generate a corresponding response speech for the question-type speech and play it to the driver.

[0073] The speech to be recognized can be a command, and drivers may issue commands with different emotions. Therefore, it is necessary to consider the semantic data and emotional category of the command speech, generate a corresponding response speech, and play it to the driver. At the same time, the vehicle also needs to execute the command corresponding to the command speech.

[0074] In this embodiment, the speech to be recognized is processed sequentially through a preprocessing module, a speech recognition module, a semantic understanding and intent recognition module, a sentiment analysis model, and a privacy protection module. The preprocessing module preprocesses the speech signal, including noise reduction and echo cancellation. The speech recognition module recognizes the preprocessed speech signal and converts it into text. The semantic understanding and intent recognition module performs semantic analysis on the text output by the speech recognition module to understand the user's intent and needs. The sentiment analysis module analyzes the emotions in the user's speech based on the output of the semantic understanding and intent recognition module, identifies the user's emotional state, and generates a human-like response. The privacy protection module encrypts and stores the speech signal and the process data during the processing of the speech signal by each model. The following is a detailed description of each module:

[0075] 1. Preprocessing module.

[0076] The preprocessing module primarily performs preprocessing on the input speech to be recognized. Preprocessing includes noise reduction, echo cancellation, and volume normalization to improve the accuracy of subsequent processing. The preprocessing module mainly uses the Kalman filter algorithm to perform the preprocessing of the speech to be recognized.

[0077] The flowchart of the Kalman filtering algorithm is as follows: Figure 3 As shown, the process includes the following steps: setting initial values ​​for the algorithm, calculating the predicted state value, calculating the state estimation error covariance matrix, calculating the Kalman gain, combining the initial value of the estimation error covariance, calculating the state estimate, calculating the prediction error covariance matrix, and incrementing the time step by 1 when the current time step k is not greater than the sampling time, entering the next loop, until it is greater than the sampling time.

[0078] This application processes the speech to be recognized through a preprocessing module, which can remove or reduce interference factors such as background noise, wind noise, and road noise, thereby improving the clarity and signal-to-noise ratio of the speech signal.

[0079] 2. Speech recognition module.

[0080] As mentioned above, the conversion of the speech to be recognized into text to be recognized is performed by the speech recognition model. Figure 4 As shown in the figure, recurrent represents RNN, bidrection represents BRNN, invariant convolution is invariant convolution, and spectrogram is spectrogram. The specific processing procedure of the speech recognition model is as follows:

[0081] In one embodiment, the step of inputting the preprocessed speech to be recognized into a speech recognition model for processing to obtain the text to be recognized includes: inputting the preprocessed speech to be recognized into the speech recognition model; processing the preprocessed speech to be recognized through the speech recognition model to obtain speech sequence features; and obtaining a normalization result based on the speech sequence features and a preset normalization parameter; wherein the normalization parameter is obtained by training on a batch of speech samples, and the normalization parameter is related to the mean of the sample vectors corresponding to the batch of speech samples.

[0082] The speech recognition module is mainly based on the DeepSpeech2 algorithm, which uses CNN and recurrent neural network (RNN) to extract features of the speech to be recognized and uses connectionist temporal classification (CTC) algorithm for speech recognition.

[0083] After the speech to be recognized enters the speech recognition model, it first undergoes convolution and pooling operations through the CNN layer of the speech recognition model to reduce the dimensionality and complexity of the input data.

[0084] Next, the features output by the CNN are further processed through the RNN layer of the speech recognition model to obtain higher-level speech features. Since the RNN layer has memory function and can process time series data, it can capture the sequence information in the speech signal and obtain speech sequence features.

[0085] Then, the output of each RNN layer is input into the batch normalization (BN) model of the speech recognition model. After processing by the BN model, the text to be recognized is obtained, which accelerates network convergence and improves recognition accuracy.

[0086] The training process of the BN model is as follows:

[0087] The normalization parameters in the Batch Normalization (BN) model are obtained through training with a batch of speech samples. The batch of speech samples is input into the initial BN model, and by calculating the mean and variance of the corresponding sample vectors, the value of each speech sample is standardized to a distribution with zero mean and unit variance. This results in the normalized prediction of the initial BN model. The algorithm formula for the initial BN model is shown below:

[0088] ;

[0089] ;

[0090] This represents the k-th speech sample in a batch of speech samples. Hehe Let represent the mean and variance of this batch of speech samples, respectively. This is a constant added to avoid division by zero. It is the standardized k-th speech sample. and These are learnable normalization parameters used to scale and shift the standardized data. It is the normalized result of the BN model output.

[0091] Each speech sample is labeled with an actual normalized result. The normalized result of the label is compared with the normalized result predicted by the initial BN model to determine the loss function. Based on the loss function, the normalization parameters are iteratively optimized to obtain the trained BN model.

[0092] During training, DeepSpeech2 uses the CTC loss function, which can directly calculate the difference between the model output and the label sequence without needing to align the labels, thus avoiding a complex alignment process and improving the training efficiency of the model.

[0093] The application process of the BN model is as follows:

[0094] The speech sequence features are input into the trained BN model for processing to obtain the normalized result of the speech to be recognized. The normalization parameters corresponding to the trained BN model are the preset normalization parameters.

[0095] Finally, a fully connected layer maps the normalized parameters output by the BN model to the probability distribution of the text. The beam search algorithm is used to obtain the most likely words, and the results are post-processed to obtain the final text to be recognized.

[0096] 3. Semantic understanding and intent recognition module.

[0097] As mentioned above, understanding user intent and needs is achieved through a semantic understanding model, referring to... Figure 5 The specific processing steps of the semantic understanding model are as follows:

[0098] In one embodiment, the step of inputting the text to be recognized into a semantic understanding model for processing to obtain semantic data includes: inputting the text to be recognized into the semantic understanding model; processing the text to be recognized through an embedding layer in the semantic understanding model to obtain processed text to be recognized; processing the processed text to be recognized through an encoding layer in the semantic understanding model to obtain semantic understanding features; wherein the encoding layer includes a feedforward neural network and a multi-head attention mechanism; and processing the semantic understanding features through a fully connected layer in the semantic understanding model to obtain the semantic data.

[0099] The embedding layer transforms the text to be recognized into a vector representation, uses wordpiece segmentation technology to segment the text sequence to be recognized, and assigns a unique identifier to each word piece;

[0100] The transformer encoding layer includes a multi-head attention mechanism and a feedforward neural network, which can effectively handle the contextual dependencies and feature representations in the text to be recognized;

[0101] The text to be identified is processed by the encoding layer in the semantic understanding model. Specifically, the text to be identified after being processed by the embedding layer is processed by the multi-head attention mechanism in the encoding layer to obtain the attention mechanism processing result. The attention mechanism processing result is then input into the feedforward neural network in the encoding layer and processed by the feedforward neural network.

[0102] If there are multiple coding layers, the multiple coding layers process the data sequentially, and the final output layer outputs semantic understanding features.

[0103] The voice system based on AI perception and interaction technology can adapt to various scenarios, understand and respond to complex voice commands and inquiries, accurately understand the voice to be recognized, and thus generate more accurate response voices, making the interaction more natural and smooth, thereby greatly optimizing the user experience.

[0104] 4. Sentiment Analysis Module.

[0105] The sentiment analysis module includes a sentiment classification model.

[0106] In one embodiment, semantic data is input into an emotion classification model for processing to obtain the emotion category corresponding to the speech to be identified. This includes: preprocessing the semantic data and converting the preprocessed semantic data into a vector to be identified; inputting the vector to be identified into the emotion classification model, and extracting target features, context features, and question-and-answer interaction features from the emotion classification model, wherein the question-and-answer interaction features characterize the mutual influence between the question-type speech and the corresponding response speech; fusing the target features and context features to obtain fused features; and determining the emotion category based on the fused features and the question-and-answer interaction features.

[0107] Reference Figure 6 In this application, the sentiment analysis model primarily utilizes sentiment computing technology to analyze the emotions in user speech, identify the user's emotional state, and generate human-like responses or dialogues based on the results of semantic understanding and sentiment analysis. The sentiment analysis model employs a neural network model based on feature fusion and a bidirectional attention mechanism. This model combines bidirectional encoder representations from TNSformers (BERT), convolutional neural networks (CNNs), bidirectional gated recurrent units (BiGRUs), and attention mechanisms.

[0108] This sentiment classification model mainly consists of the following parts:

[0109] (1) Text pre-training layer: used to preprocess semantic data and convert the preprocessed semantic data into vectors to be recognized. Specifically:

[0110] After preprocessing the semantic data, the BERT model is used for pre-training, converting it into an input format that the neural network can process. Since neural networks can only process numerical data, it is necessary to convert the semantic data into a processable vector form, that is, into a vector to be recognized, so that it can be input into the neural network for processing.

[0111] In the figure, q1, q2 and q3 represent three semantic data, and a1, a2 and a3 represent the vectors to be identified obtained by converting the three semantic data respectively.

[0112] (2) Feature Extraction Layer: This stage employs parallel CNN and BiGRU neural network structures to extract features from the input vector to be identified. The CNN channel extracts local feature vectors of the vector to be identified through convolution and pooling, serving as target features; the BiGRU channel utilizes a gated recurrent unit (GRU) network to learn global context information and extract context feature vectors. The target features and context features extracted by these two channels are concatenated to obtain a comprehensive feature vector sequence, serving as the fusion feature.

[0113] (3) Question-answering interaction layer: By using a bidirectional attention mechanism to extract question-answering interaction features from the vector to be identified, the question-answering interaction features represent the mutual influence between the question-type speech and the corresponding response speech. It can simultaneously consider the influence of the question on the answer and the answer on the question, thereby obtaining a more comprehensive and accurate text feature representation.

[0114] The bidirectional attention mechanism allows neural networks to consider the influence of other words while calculating the feature representation of each word, thus enabling them to better capture interactive information in question-and-answer text.

[0115] (4) Sentiment Classification Layer: The sentiment classification layer uses the softmax function for classification. It receives the final feature vector representation obtained after feature extraction and question-and-answer interaction, and maps it to an N-dimensional vector probability distribution, where N represents the number of sentiment categories. The softmax function normalizes these probabilities so that their sum is 1, and each probability represents the likelihood that a sample belongs to the corresponding category. By comparing the probability magnitudes of each sentiment category, the speech to be recognized is classified into the corresponding sentiment category.

[0116] For example, there are four emotion categories: happiness, sadness, anger, and surprise, and the output of the softmax layer is [0.1, 0.7, 0.15, 0.05], which means that the probability of the speech to be recognized belonging to the four emotion categories of happiness, sadness, anger, and surprise is 0.1, 0.7, 0.15, and 0.05, respectively. Then the model will predict that the speech to be recognized has the highest probability of belonging to the "sad" category, so the emotion category corresponding to the speech to be recognized is "sad".

[0117] This application's AI-based perception and interaction voice system analyzes the emotions in the user's voice by introducing an emotion analysis module, judges the user's emotional state, and provides personalized responses and suggestions, making the driving process more comfortable and enjoyable.

[0118] 5. Privacy protection module.

[0119] In one embodiment, the method further includes: generating a file to be encrypted based on the speech to be recognized, the text to be recognized, the semantic data, and the emotion category; encrypting the file to be encrypted to obtain an encrypted file, and storing the encrypted file.

[0120] The preprocessing module, speech recognition module, semantic understanding and intent recognition module, and sentiment analysis model generate process data during the processing of the speech to be recognized. The privacy protection module of this application can package the speech to be recognized and the process data to generate a file to be encrypted, then encrypt the file to be encrypted, obtain the encrypted file, and store it in the cloud.

[0121] Specifically, the process data includes the speech to be recognized after preprocessing by the preprocessing module, the text to be recognized after processing by the speech recognition model, the semantic data after processing by the semantic understanding model, and the sentiment category after processing by the sentiment classification model.

[0122] The privacy protection module is mainly designed to ensure the security of user voice data and privacy by taking security measures such as encryption and anonymization. The privacy protection module mainly uses the MD5 encryption algorithm, the core idea of ​​which is to generate a 128-bit hash value from input data of arbitrary length through a series of complex transformations.

[0123] This application's AI-based voice system for perception and interaction employs advanced encryption technologies and privacy protection algorithms to ensure the security and privacy of user data.

[0124] In summary, the overall scheme of the speech processing method in this application is as follows:

[0125] Step 1: Voice input. The user issues voice commands or asks questions through the vehicle's microphone.

[0126] Step 2: Noise reduction processing. The system processes the input speech signal through the noise reduction module to remove or reduce interference factors such as background noise, wind noise, and road noise, so as to improve the clarity and signal-to-noise ratio of the speech signal.

[0127] Step 3: Speech recognition. The noise-reduced speech signal is sent to the speech recognition module, which uses the DeepSpeech2 algorithm to parse and recognize the speech and convert it into text information.

[0128] Step 4: Semantic understanding. The identified text enters the semantic understanding stage, where the BERT-CNN algorithm is used to understand the user's intent and needs.

[0129] Step 5: Sentiment Analysis. The system uses a sentiment analysis model based on feature fusion and bidirectional attention to further process the semantic understanding results, analyze the emotions in the user's speech, determine the user's emotional state, and provide emotional basis for subsequent anthropomorphic dialogue.

[0130] Step 6: Privacy Protection. The system uses the MD5 encryption algorithm to encrypt the user's voice data to ensure the security and privacy of the user's data.

[0131] Step 7: Humanized dialogue generation. Based on the results of semantic understanding and sentiment analysis, generate humanized responses or perform corresponding operations.

[0132] Step 8: Output a response or perform an operation. The anthropomorphic response is played to the user through the car audio system, or an operation such as navigation or music playback is performed.

[0133] Step 9: Collect feedback. The system continuously collects user feedback data, including voice input, recognition results, user satisfaction, etc., and uses the collected feedback data to optimize and iterate the system.

[0134] One application scenario of this application is as follows: In a highway driving scenario, the driver requests the system to play music via voice command. The preprocessing module performs noise reduction, echo cancellation, and other processing on the voice signal to more accurately recognize the driver's voice command. The voice recognition module recognizes the driver's voice command and converts it into text. The semantic understanding and intent recognition module performs semantic analysis on the output text and identifies that the current scenario is a highway driving scenario. The sentiment analysis module analyzes that the driver's current emotional state is pleasant and generates a human-like dialogue: "Okay, relaxing and pleasant music has been played for you. We hope you enjoy your drive." The privacy protection module encrypts the user's voice data to ensure the security and privacy of the user data.

[0135] As can be seen, this application inputs the text to be recognized into a preprocessing module to remove noise, then inputs the preprocessed speech into a speech recognition model for processing to obtain the text to be recognized, then inputs the text into a semantic understanding model for processing to obtain semantic data, then inputs the semantic data into a sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be recognized, and finally generates a human-like response. At the same time, the privacy protection module also encrypts the speech to be recognized and the process data during the speech processing. In this way, the interaction and collaborative data processing between the preprocessing module, the speech recognition model, the semantic understanding model, the sentiment classification model, and the privacy protection module are realized, thereby achieving a series of functions such as removing speech noise, understanding deep semantics, considering the driver's sentiment category when responding to speech, and improving the safety of driver's speech, thus meeting the diverse needs of users for response speech.

[0136] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0137] Based on the same inventive concept, this application also provides a speech processing apparatus for implementing the speech processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more speech processing apparatus embodiments provided below can be found in the limitations of the speech processing method described above, and will not be repeated here.

[0138] In one exemplary embodiment, such as Figure 8 As shown, a voice processing device 800 is provided, comprising:

[0139] The acquisition module 801 is used to acquire the speech to be recognized;

[0140] Preprocessing module 802 is used to preprocess the speech to be recognized;

[0141] The first processing module 803 is used to input the preprocessed speech to be recognized into the speech recognition model for processing to obtain the text to be recognized.

[0142] The second processing module 804 is used to process the text to be recognized into the semantic understanding model to obtain semantic data;

[0143] The classification module 805 is used to input the semantic data into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified.

[0144] The generation module 806 is used to generate a response voice for the speech to be identified based on the semantic data and the emotion category, and to play the response voice.

[0145] In one embodiment, the generation module 806 is specifically used for:

[0146] If the speech to be identified is a question-type speech, a response speech is generated based on the semantic data and the emotion category.

[0147] If the speech to be recognized is a command-type speech, the command corresponding to the command-type speech is executed, and a response speech is generated based on the semantic data and the emotion category.

[0148] In one embodiment, the classification module 805 is specifically used for:

[0149] The semantic data is preprocessed and then converted into a vector to be identified.

[0150] The vector to be identified is input into the sentiment classification model, and the target features, context features and question-and-answer interaction features of the vector to be identified are extracted by the sentiment classification model. The question-and-answer interaction features represent the mutual influence between the question-type speech and the corresponding response speech.

[0151] The target features and context features are fused to obtain fused features;

[0152] The emotion category is determined based on the fusion features and the question-and-answer interaction features.

[0153] In one embodiment, the first processing module 803 is specifically used for:

[0154] The preprocessed speech to be recognized is input into the speech recognition model;

[0155] The preprocessed speech to be recognized is processed by the speech recognition model to obtain speech sequence features; and based on the speech sequence features and preset normalization parameters, a normalization result is obtained, and the text to be recognized is obtained based on the normalization result.

[0156] The normalization parameter is obtained by training on a batch of speech samples, and the normalization parameter is related to the mean of the sample vectors corresponding to the batch of speech samples.

[0157] In one embodiment, the second processing module 804 is specifically used for:

[0158] The text to be identified is input into the semantic understanding model, and the text to be identified is processed through the embedding layer in the semantic understanding model to obtain the processed text to be identified.

[0159] The semantic understanding model processes the processed text to be recognized through an encoding layer to obtain semantic understanding features; wherein, the encoding layer includes a feedforward neural network and a multi-head attention mechanism;

[0160] The semantic understanding features are processed by the fully connected layer in the semantic understanding model to obtain the semantic data.

[0161] In one embodiment, the device further includes:

[0162] The file generation module is used to generate a file to be encrypted based on the speech to be recognized, the text to be recognized, the semantic data, and the sentiment category.

[0163] An encryption module is used to encrypt the file to be encrypted, obtain an encrypted file, and store the encrypted file.

[0164] Each module in the aforementioned voice processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0165] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a voice processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0166] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0168] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0169] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0171] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0172] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0173] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A speech processing method, characterized in that, The method includes: Acquire the speech to be recognized; The speech to be recognized is preprocessed; The preprocessed speech to be recognized is input into the speech recognition model for processing to obtain the text to be recognized; The text to be recognized is input into the semantic understanding model for processing to obtain semantic data; The semantic data is input into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified. Based on the semantic data and the emotion category, a response voice for the speech to be identified is generated and played. The step of generating a response voice based on the semantic data and the emotion category includes: If the speech to be identified is a question-type speech, a response speech is generated based on the semantic data and the emotion category. If the speech to be recognized is a command-type speech, execute the command corresponding to the command-type speech, and generate a response speech based on the semantic data and the emotion category. The step of inputting the semantic data into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified includes: The semantic data is preprocessed and then converted into a vector to be identified. The vector to be identified is input into the sentiment classification model, and the target features, context features and question-and-answer interaction features of the vector to be identified are extracted by the sentiment classification model. The question-and-answer interaction features represent the mutual influence between the question-type speech and the corresponding response speech. The target features and context features are fused to obtain fused features; The emotion category is determined based on the fusion features and the question-and-answer interaction features.

2. The method according to claim 1, characterized in that, The step of inputting the preprocessed speech to be recognized into the speech recognition model for processing to obtain the text to be recognized includes: The preprocessed speech to be recognized is input into the speech recognition model; The preprocessed speech to be recognized is processed by the speech recognition model to obtain speech sequence features; and based on the speech sequence features and preset normalization parameters, a normalization result is obtained, and the text to be recognized is obtained based on the normalization result. The normalization parameter is obtained by training on a batch of speech samples, and the normalization parameter is related to the mean of the sample vectors corresponding to the batch of speech samples.

3. The method according to claim 1, characterized in that, The step of inputting the text to be recognized into the semantic understanding model for processing to obtain semantic data includes: The text to be identified is input into the semantic understanding model, and the text to be identified is processed through the embedding layer in the semantic understanding model to obtain the processed text to be identified. The semantic understanding model processes the processed text to be recognized through an encoding layer to obtain semantic understanding features; wherein, the encoding layer includes a feedforward neural network and a multi-head attention mechanism; The semantic understanding features are processed by the fully connected layer in the semantic understanding model to obtain the semantic data.

4. The method according to claim 1, characterized in that, The method further includes: Based on the speech to be identified, the text to be identified, the semantic data, and the emotion category, a file to be encrypted is generated; The file to be encrypted is encrypted to obtain an encrypted file, and the encrypted file is stored.

5. A voice processing device, characterized in that, The device includes: The acquisition module is used to acquire the speech to be recognized; The preprocessing module is used to preprocess the speech to be recognized; The first processing module is used to input the preprocessed speech to be recognized into the speech recognition model for processing, so as to obtain the text to be recognized. The second processing module is used to process the text to be recognized into the semantic understanding model to obtain semantic data; The classification module is used to input the semantic data into the sentiment classification model for processing to obtain the sentiment category corresponding to the speech to be identified. The generation module is used to generate a response voice for the speech to be recognized based on the semantic data and the emotion category, and to play the response voice. The generation module is configured to generate a response voice for the speech to be recognized based on the semantic data and the emotion category when the speech to be recognized is a question-type speech; and to execute the command corresponding to the command-type speech when the speech to be recognized is a command-type speech, and generate a response voice for the speech to be recognized based on the semantic data and the emotion category. The classification module is used to preprocess the semantic data and convert the preprocessed semantic data into a vector to be identified; the vector to be identified is input into the sentiment classification model, and the sentiment classification model extracts the target features, context features, and question-and-answer interaction features of the vector to be identified, wherein the question-and-answer interaction features represent the mutual influence between the question-type speech and the corresponding response speech; the target features and context features are fused to obtain fused features; and the sentiment category is determined based on the fused features and the question-and-answer interaction features.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images