Multilingual speech recognition method, device and equipment based on emotion perception AI model
Through a multilingual speech recognition method based on the emotion-perception AI model, combined with time series segmentation and deep neural network, the problems of insufficient attention to emotional information and waste of resources in traditional speech recognition are solved, and high-accurate recognition of multilingual speech is achieved.
Patent Information
- Application Number
- CN202411147566.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Traditional speech recognition technology has problems of insufficient attention to emotional information and waste of resources in multilingual speech processing, and the recognition of small languages or dialects is inaccurate.
Using a multilingual speech recognition method based on emotion perception AI model, through time series segmentation, voiceprint multidimensional vector transformation, emotion correlation analysis, spectrum feature coding and data fusion, speech recognition is combined with deep neural network to generate emotional color annotations. and the recognition results of language information.
It improves the accuracy of multilingual speech recognition, overcomes the problems of waste of resources and inaccurate recognition in traditional methods, and can handle emotional and language information more accurately.
Smart Images

Figure CN118675501B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech recognition, and in particular to a multilingual speech recognition method, device and equipment based on an emotion perception AI model. Background Art
[0002] Traditional speech recognition technology faces many challenges when processing multilingual speech. On the one hand, most speech recognition systems focus mainly on the recognition of speech content, but pay little attention to the emotional information contained in speech. However, emotions play a vital role in communication, and different emotional states can lead to very different meanings and messages conveyed by the same speech content. For example, a sentence with anger and the same sentence with joy have completely different actual meanings and subsequent responses.
[0003] On the other hand, for multi-language recognition, traditional methods often build models for each language separately, or rely on a general large and comprehensive model, but both methods have defects. Building models separately will lead to waste of resources and increase maintenance costs, and the process of switching between different language models in actual applications is cumbersome and delays. When faced with the complex characteristics of multiple languages, general models are difficult to accurately recognize each language, especially some small languages or dialects with unique pronunciation characteristics and grammatical structures.
[0004] Therefore, current speech recognition has the defect of inaccurate speech recognition because it is not combined with sentiment analysis and multi-language recognition. Summary of the invention
[0005] The main purpose of the present invention is to provide a multilingual speech recognition method, device and equipment based on an emotion perception AI model, aiming to overcome the defect of inaccurate multilingual speech recognition.
[0006] To achieve the above object, the present invention provides a multilingual speech recognition method based on an emotion perception AI model, comprising the following steps:
[0007] Perform time series segmentation processing on the collected multilingual voice stream to obtain multiple basic voice units of equal duration, and convert each basic voice unit into a voiceprint multidimensional vector;
[0008] Input the voiceprint multidimensional vector into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value;
[0009] Performing audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; performing feature coding processing on each spectrum interval to generate spectrum feature codes; performing language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags;
[0010] Performing data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value;
[0011] The comprehensive feature values are input into the emotion perception AI model trained by deep neural network for processing, and multilingual speech recognition results with emotional color annotations and language information annotations are obtained.
[0012] Furthermore, the step of converting each basic speech unit into a voiceprint multi-dimensional vector includes:
[0013] Extract voiceprint features from each basic speech unit to obtain basic voiceprint feature data;
[0014] The basic voiceprint feature data is converted into a multi-dimensional vector to obtain a voiceprint multi-dimensional vector.
[0015] Furthermore, the performing feature coding processing on each spectrum interval to generate spectrum feature codes includes:
[0016] Calculate the frequency energy value of each spectrum interval to obtain the frequency energy value data corresponding to each spectrum interval;
[0017] Perform feature coding processing on all frequency energy value data to generate spectrum feature codes.
[0018] Furthermore, the voiceprint multi-dimensional vector is input into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value, including:
[0019] Perform principal component analysis on the voiceprint multidimensional vector to obtain the key voiceprint multidimensional vector;
[0020] Input the key voiceprint multidimensional vector into the first hidden layer of the voiceprint emotion association model pre-trained based on a deep neural network; the first hidden layer is composed of multiple neurons, and each neuron performs linear weighting and nonlinear activation function processing on the input vector;
[0021] The output from the first hidden layer is passed to the second hidden layer. The neurons in the second hidden layer are fully connected with the neurons in the first hidden layer, and the output of the first hidden layer is further feature extracted and combined.
[0022] After being processed by the second hidden layer, a preliminary emotional tendency value in the range of [0, 1] is output; among them, 0 represents an extremely negative emotional tendency, 1 represents an extremely positive emotional tendency, and the intermediate values represent different degrees of emotional states.
[0023] Furthermore, the preliminary sentiment tendency value and the language feature mark are subjected to data fusion processing to obtain a comprehensive feature value, including:
[0024] The preliminary sentiment tendency value is normalized and mapped to the interval [0, 1] to obtain the normalized sentiment tendency value; the language feature mark is converted into a corresponding digital code to form a language code value;
[0025] Constructing a fusion weight matrix, wherein the fusion weight matrix includes weight parameters for different languages under different emotional tendencies, wherein the weight parameters are obtained in advance by training and statistically analyzing multi-language and multi-emotion data;
[0026] Selecting corresponding weight parameters from the fusion weight matrix according to the language encoding value and the normalized sentiment tendency value;
[0027] The normalized sentiment tendency value is multiplied by the selected weight parameter to obtain a weighted sentiment tendency value component; the language code value is multiplied by the corresponding weight parameter to obtain a weighted language code value component;
[0028] The weighted sentiment tendency value component and the weighted language coding value component are vector-added to obtain an initial vector of the comprehensive feature value; the initial vector of the comprehensive feature value is normalized to make its value within a preset range to obtain the comprehensive feature value.
[0029] Furthermore, the comprehensive feature values are input into the emotion perception AI model trained by the deep neural network for processing to obtain multilingual speech recognition results with emotion color annotations and language information annotations, including:
[0030] The comprehensive feature value is input into the input layer of the emotion perception AI model; the emotion perception AI model includes an input layer, multiple hidden layers and an output layer; the hidden layer includes multiple neuron nodes, and each node performs weighted summation and nonlinear activation function processing on the input data;
[0031] When the hidden layer receives the comprehensive feature value transmitted by the input layer, based on the attention mechanism, key features related to emotional color and language information are focused on and strengthened;
[0032] After gradual processing and feature extraction by multiple hidden layers, the output layer generates multilingual speech recognition results with emotional color annotations and language information annotations based on the data passed from the hidden layers; among them, the emotional color annotation is expressed by combining emotional intensity values and emotional type classification labels, and the language information annotation is expressed by international standard language codes.
[0033] Furthermore, the attention mechanism automatically adjusts the attention paid to different parts of the comprehensive feature value according to the weight parameters learned during the training process;
[0034] The hidden layer is a recursive neural network or a long short-term memory network.
[0035] The present invention also provides a multilingual speech recognition device based on an emotion perception AI model, comprising:
[0036] The segmentation module is used to perform time series segmentation processing on the collected multilingual voice streams to obtain multiple basic voice units of equal duration, and convert each basic voice unit into a voiceprint multidimensional vector;
[0037] The association analysis module is used to input the voiceprint multi-dimensional vector into the voiceprint emotion association model to perform emotion association analysis and obtain a preliminary emotion tendency value;
[0038] The matching module is used to perform audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; perform feature coding processing on each spectrum interval to generate spectrum feature codes; perform language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags;
[0039] A fusion module is used to perform data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value;
[0040] The recognition module is used to input the comprehensive feature values into the emotion perception AI model trained by the deep neural network for processing, and obtain multilingual speech recognition results with emotional color annotations and language information annotations.
[0041] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.
[0042] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.
[0043] The present invention provides a multilingual speech recognition method, device and equipment based on an emotion perception AI model, including: performing time series segmentation processing on the collected multilingual speech stream to obtain multiple basic speech units of equal duration, converting each basic speech unit into a voiceprint multidimensional vector; inputting the voiceprint multidimensional vector into a voiceprint emotion association model for emotion correlation analysis processing to obtain a preliminary emotion tendency value; performing audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; performing feature encoding processing on each spectrum interval to generate spectrum feature coding; performing language feature search and matching processing on the spectrum feature coding to obtain a matching language feature tag; performing data fusion processing on the preliminary emotion tendency value and the language feature tag to obtain a comprehensive feature value; inputting the comprehensive feature value into an emotion perception AI model trained by a deep neural network for processing to obtain a multilingual speech recognition result with emotion color annotation and language information annotation. In the present invention, by combining emotion analysis and multilingual recognition, the defect of inaccurate multilingual speech recognition is overcome. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 1 is a schematic diagram of the workflow of a multilingual speech recognition method based on an emotion perception AI model in one embodiment of the present invention;
[0045] Figure 2 is a schematic diagram of the steps of a multilingual speech recognition method based on an emotion perception AI model in one embodiment of the present invention;
[0046] Figure 3 is a schematic diagram of the workflow of a multilingual speech recognition method based on an emotion perception AI model in another embodiment of the present invention;
[0047] Figure 4 This is a structural block diagram of a multilingual speech recognition device based on an emotion perception AI model in one embodiment of the present invention;
[0048] Figure 5 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.
[0049] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0051] Reference Figure 1 as well as Figure 2In one embodiment of the present invention, a multilingual speech recognition method based on an emotion-aware AI model is provided, comprising the following steps:
[0052] Step S1, performing time series segmentation processing on the collected multilingual voice stream to obtain multiple basic voice units of equal duration, and converting each basic voice unit into a voiceprint multidimensional vector;
[0053] Step S2, inputting the voiceprint multidimensional vector into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value;
[0054] Step S3, performing audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; performing feature coding processing on each spectrum interval to generate spectrum feature codes; performing language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags;
[0055] Step S4, performing data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value;
[0056] Step S5, input the comprehensive feature value into the emotion perception AI model trained by the deep neural network for processing, and obtain the multilingual speech recognition results with emotion color annotation and language information annotation.
[0057] In this embodiment, by combining sentiment analysis and multi-language recognition, the defect of inaccurate multi-language speech recognition is overcome.
[0058] As described in step S1 above, the multilingual voice stream is a continuous voice signal, and it is difficult to directly analyze and process it. Therefore, by segmenting the time series, it is divided into multiple basic voice units of equal duration, so that the complex voice stream can be simplified into a series of relatively independent and easy-to-process small segments. Among them, the equal-duration segmentation method ensures the consistency and standardization of subsequent processing, and facilitates the use of the same processing flow and algorithm for different basic voice units. For example, if a one-minute voice stream is segmented into a basic voice unit of 0.1 seconds, 600 basic voice units can be obtained.
[0059] Each of the above basic speech units contains rich information, such as the speaker's timbre, pitch, speaking speed and other characteristics. The purpose of converting it into a voiceprint multidimensional vector is to express these complex speech features in mathematical form for computer processing and analysis. Voiceprint multidimensional vectors can contain information in multiple dimensions, such as frequency features, amplitude features, Mel frequency cepstral coefficients, etc. Through preset algorithms and models, the sound signals in the basic speech units are converted into these numerical vector representations. For example, a voiceprint multidimensional vector can contain 128 dimensional values, each of which represents the quantized value of the basic speech unit on a specific feature.
[0060] As described in step S2 above, the voiceprint emotion association model is a model specifically used to analyze the association between voiceprints and emotions. It is trained with a large amount of speech data with emotion annotations. By continuously learning the voiceprint feature patterns under different emotional states, the model can establish a mapping relationship between voiceprint features and emotional tendencies. The model can be a neural network model based on deep learning, such as a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN). CNN is used to extract local features of voiceprints, and RNN is used to process the time series information of voiceprints and the dynamic changes of emotions.
[0061] When the voiceprint multidimensional vector is input into the voiceprint emotion association model, the model will analyze and interpret the features in the voiceprint multidimensional vector based on the previously learned knowledge and patterns. The model determines the emotional tendency corresponding to the voiceprint by calculating and comparing the similarity or difference between the voiceprint multidimensional vector and the feature patterns corresponding to different emotion categories. For example, the model determines that the emotional tendency represented by a voiceprint multidimensional vector is "happy" and gives a corresponding preliminary emotional tendency value, such as 0.8 (where 0 can represent completely negative emotions and 1 represents completely positive emotions).
[0062] As described in step S3 above, the purpose of dividing the audio spectrum for each basic speech unit is to analyze the frequency composition and characteristics of the speech in more detail. The audio spectrum reflects the energy distribution of the sound at different frequencies. By converting the audio signal of the basic speech unit to the frequency domain and dividing it into multiple continuous spectrum intervals, the characteristics of the speech in different frequency ranges can be better understood. For example, the audio spectrum can be divided into 100 continuous spectrum intervals from 0Hz to 20kHz, and each interval represents the sound characteristics of a certain frequency range. The purpose of feature encoding for each spectrum interval is to convert the complex information in the spectrum interval into a concise, computer-processed and recognized coding form. Feature encoding can be performed in a variety of ways, such as hash encoding, vector quantization, etc. Through feature encoding, the key features of the spectrum interval can be extracted and represented as a fixed-length coding value. For example, after feature encoding a spectrum interval, a 16-bit binary code is obtained, which represents the frequency characteristics, energy characteristics and other information of the spectrum interval.
[0063] Different languages have differences in speech spectral features, pronunciation rules, etc. By searching and matching the language features of the spectral feature codes, the language to which the basic speech unit belongs can be determined. First, a database or model containing various language features needs to be established. This database stores typical spectral feature coding patterns and feature parameters of different languages. Then, the generated spectral feature codes are compared and matched with the language features in the database. Similarity calculation algorithms, such as cosine similarity and Euclidean distance, can be used to measure the similarity between spectral feature codes and features of different languages. For example, if a spectral feature code has the highest similarity to the feature pattern of English in the database, then the language of the basic speech unit can be determined to be English, and the corresponding language feature label can be obtained.
[0064] As described in step S4 above, the preliminary emotional tendency value and the language feature mark represent the emotional information and language information of the speech respectively. Fusion of their data can obtain a comprehensive feature representation so that the emotional and language factors can be considered simultaneously for more accurate speech recognition. Data fusion can be performed in a variety of ways, such as vector concatenation, weighted summation, etc. Vector concatenation is to directly concatenate the vectors corresponding to the preliminary emotional tendency value and the language feature mark together to form a longer comprehensive feature vector. Weighted summation is to linearly combine the two according to a certain weight coefficient. For example, assuming that the preliminary emotional tendency value is a value between 0 and 1, and the language feature mark is a specific coding value, they can be concatenated into a new vector [preliminary emotional tendency value, language feature mark] as a comprehensive feature value.
[0065] As described in step S5 above, the emotion perception AI model trained by a deep neural network is a deep neural network model specifically used for multilingual speech recognition and capable of processing emotion and language information simultaneously. It is trained with a large amount of multilingual speech data with emotion color annotations and language information annotations. The model can have a complex network structure, such as multiple layers of convolutional layers, loop layers, and fully connected layers. Through the combination of these layers, the model can learn complex patterns and features in speech, including emotion features, language features, and speech content features.
[0066] When the comprehensive feature value is input into the model, the model will analyze and interpret the comprehensive feature value based on the previously learned knowledge and patterns. The model calculates and transforms the comprehensive feature value, and finally outputs the multilingual speech recognition results with emotional color annotation and language information annotation. Emotional color annotation can be represented by emotional categories (such as happy, sad, angry, etc.) and emotional intensity values. Language information annotation is represented by specific language codes or names. At the same time, the model will also recognize and translate the content of the speech (if necessary), and output the corresponding text or other forms of results. For example, the model outputs the result "The emotional color of this sentence is happy, the language is English, and the content is 'Hello, howare you?'".
[0067] Combined with reference Figure 3 In one embodiment, the step of converting each basic speech unit into a voiceprint multi-dimensional vector includes:
[0068] Extract voiceprint features from each basic speech unit to obtain basic voiceprint feature data;
[0069] The basic voiceprint feature data is converted into a multi-dimensional vector to obtain a voiceprint multi-dimensional vector.
[0070] In this embodiment, the voiceprint feature is the key information that can uniquely identify a speaker's identity and his pronunciation characteristics. In multilingual speech recognition, even if different people speak the same language, their voices will show different characteristics due to the differences in their voiceprint features; and when the same person speaks different languages, his unique voiceprint essence still exists. Therefore, accurate extraction of voiceprint features is crucial for subsequent sentiment analysis and language recognition. The above voiceprint feature extraction includes rhythm feature extraction, timbre feature extraction, and frequency domain feature extraction.
[0071] Specifically, the audio signal of the basic speech unit is first converted from the time domain to the frequency domain to analyze its energy distribution at different frequencies. For example, the Mel Frequency Cepstral Coefficient (MFCC) is calculated, which is a feature parameter widely used in the field of speech processing. MFCC can reflect the human ear's perception of sounds of different frequencies. By performing operations such as fast Fourier transform on the basic speech unit, it is converted to the frequency domain and a series of MFCC coefficients are calculated as part of the spectrum feature.
[0072] Furthermore, prosodic features are extracted, including features of speech rate, intonation, and stress. The speech rate can be measured by calculating the duration of the speech signal in the basic speech unit and the number of speech frames contained. The intonation can be obtained by analyzing the fundamental frequency change trend of the audio signal. For example, in a sentence, the pronunciation of certain words will have a higher fundamental frequency, representing an emphasis or questioning tone. The extraction of stress can be determined by detecting the energy peak and duration change in the audio signal.
[0073] Tone feature extraction, the tone is mainly determined by the harmonic structure of the sound. By analyzing the harmonic components of the audio signal of the basic speech unit, such as calculating the frequency, amplitude, and phase of the harmonics, the tone features can be obtained. For example, some people's voices are rich in harmonics, with relatively more high-frequency components, showing a bright tone; while others' voices are relatively simple in harmonics, with a large proportion of low-frequency components, showing a low tone.
[0074] The various voiceprint features extracted above are combined and sorted according to the preset format and structure to form basic voiceprint feature data. These data can be in the form of a vector or matrix containing multiple feature parameters. For example, a basic voiceprint feature data vector contains 10 MFCC coefficients, 3 prosodic feature parameters (speech rate, intonation, stress) and 5 timbre feature parameters, a total of 18 feature values, forming a 1×18 vector.
[0075] Although the basic voiceprint feature data already contains rich voiceprint information, these data have different dimensions and numerical ranges. Directly using them for subsequent analysis and processing will increase the complexity of calculations and instability of model performance. Through multidimensional vector conversion processing, the basic voiceprint feature data can be converted into a more standardized form suitable for model processing. Multidimensional vector conversion can also further extract and fuse the potential information in the voiceprint features and map it to a high-dimensional space, so that the differences and connections between different voiceprint features can be more clearly presented, which is conducive to subsequent emotional correlation analysis and other speech processing tasks.
[0076] The multi-dimensional vector conversion process includes:
[0077] Standardization is the process of standardizing each feature parameter in the basic voiceprint feature data so that its mean is zero and its variance is unit. This can eliminate the impact of different dimensions between features, so that each feature has the same importance in subsequent calculations. For example, for a basic voiceprint feature data set containing multiple samples, the mean and variance of each feature parameter are calculated, and then each feature value of each sample is standardized by subtracting the mean and dividing by the variance.
[0078] Principal component analysis (PCA), PCA is a commonly used dimensionality reduction and feature extraction method. The standardized basic voiceprint feature data is used as input, and the main components (i.e., principal components) in the data are found by calculating the covariance matrix of the data. The above principal components are linear combinations of the original features, which can retain the variance information of the data to the greatest extent. Through PCA, the high-dimensional basic voiceprint feature data can be converted into a low-dimensional multidimensional vector, while removing redundant information and noise in the data.
[0079] Autoencoder processing, which is an unsupervised learning neural network model consisting of an encoder and a decoder. The basic voiceprint feature data is input into the encoder, which compresses it into a low-dimensional potential representation (i.e., a multi-dimensional vector), and then the decoder reconstructs the original basic voiceprint feature data from this potential representation. During the training process, the reconstruction error is minimized by adjusting the parameters of the autoencoder, thereby learning a multi-dimensional vector that can effectively represent the basic voiceprint feature data.
[0080] In one embodiment, performing feature coding processing on each spectrum interval to generate spectrum feature codes includes:
[0081] Calculate the frequency energy value of each spectrum interval to obtain the frequency energy value data corresponding to each spectrum interval;
[0082] Perform feature coding processing on all frequency energy value data to generate spectrum feature codes.
[0083] In this embodiment, the frequency energy value is a key indicator to describe the energy distribution of the audio signal in different frequency spectrum intervals. In the speech signal, different frequency components carry different information, for example, the pitch, timbre and some specific speech features of the speech are all related to the frequency. By calculating the frequency energy value, the signal strength in each frequency spectrum interval can be quantified, providing basic data for subsequent feature coding and speech analysis.
[0084] The calculation of frequency energy value is usually based on the principle of Fourier transform. After the audio signal of the basic speech unit is converted from the time domain to the frequency domain, the frequency components in each spectrum interval are integrated or summed according to the division of the spectrum interval to obtain the energy value of the interval. For example, for a discrete audio signal, its frequency domain representation is obtained after fast Fourier transform (FFT), and then the amplitude values corresponding to the frequency points in each spectrum interval are squared and summed, and then divided by the number of frequency points in the interval to obtain the average frequency energy value of the spectrum interval.
[0085] First, determine how to divide the spectrum interval. You can use equal bandwidth division, for example, divide the audio spectrum from 0Hz to half of the sampling frequency (according to the Nyquist sampling theorem) into several spectrum intervals. You can also use non-uniform division methods such as Mel scale according to the auditory characteristics of the human ear to better simulate the difference in the human ear's perception of sounds of different frequencies.
[0086] Then, the signal amplitude values corresponding to the frequency points in each spectrum interval are processed. The square of the amplitude value is used as a measure of energy, and the square amplitude values of all frequency points in the interval are accumulated. If the audio signal is a sampled discrete signal, suppose there are N frequency points in a certain spectrum interval, and the corresponding amplitude values are A1, A2,..., AN, then the frequency energy value calculation formula of the spectrum interval can be expressed as E = (A1^2 + A2^2 +... + AN^2) / N.
[0087] After calculating each spectrum interval, the frequency energy value corresponding to each spectrum interval is obtained. These values can form a data set, namely, frequency energy value data. For example, if the audio spectrum is divided into 100 spectrum intervals, 100 frequency energy values will be obtained, forming a frequency energy value data vector containing 100 elements.
[0088] The original frequency energy value data has high dimensionality and complexity. Directly using this data for subsequent language feature search and matching operations will face problems such as large amount of calculation, low efficiency, and interference from noise and redundant information. Through feature coding processing, the frequency energy value data can be converted into a more concise and representative coding form, highlighting key information, reducing data dimensionality, and retaining important information related to speech features and language information.
[0089] Feature coding can also enhance the distinguishability of data, so that different languages or different speech features have obvious differences in the coding space, which is convenient for subsequent pattern recognition and classification processing. The above feature coding includes hash coding, vector quantization, deep learning feature coding, etc., which will not be described in detail here.
[0090] After feature encoding, the obtained encoding value is the spectrum feature encoding. This encoding is an abstract and simplified representation of the original frequency energy value data, which contains the key feature information of the speech signal in the spectrum interval. For example, a 32-bit binary code obtained by hash encoding, or an integer index value obtained by vector quantization, can be used as a spectrum feature encoding. The spectrum feature encoding will be used in subsequent speech processing tasks such as language feature search and matching as key data to describe the speech spectrum features.
[0091] In one embodiment, the step of inputting the voiceprint multi-dimensional vector into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value includes:
[0092] Perform principal component analysis on the voiceprint multidimensional vector to obtain the key voiceprint multidimensional vector;
[0093] Input the key voiceprint multidimensional vector into the first hidden layer of the voiceprint emotion association model pre-trained based on a deep neural network; the first hidden layer is composed of multiple neurons, and each neuron performs linear weighting and nonlinear activation function processing on the input vector;
[0094] The output from the first hidden layer is passed to the second hidden layer. The neurons in the second hidden layer are fully connected with the neurons in the first hidden layer, and the output of the first hidden layer is further feature extracted and combined.
[0095] After being processed by the second hidden layer, a preliminary emotional tendency value in the range of [0, 1] is output; among them, 0 represents an extremely negative emotional tendency, 1 represents an extremely positive emotional tendency, and the intermediate values represent different degrees of emotional states.
[0096] In this embodiment, the voiceprint multidimensional vector is first subjected to principal component analysis to obtain a key voiceprint multidimensional vector. The voiceprint multidimensional vector contains a large amount of original feature information, but there are some redundant or highly correlated features, which will not only increase the complexity of subsequent calculations, but also affect the training efficiency and accuracy of the model. Principal component analysis (PCA) can effectively reduce the dimension of the data and extract the most representative and distinguishing key features, namely the key voiceprint multidimensional vector, thereby simplifying the data and highlighting important information. For example, the original voiceprint multidimensional vector contains 100 feature dimensions, but after PCA processing, only the most critical 30 dimensions are retained. These dimensions can retain the variance information of the original data to the greatest extent, while removing noise and redundant features.
[0097] The process of principal component analysis includes:
[0098] First, the covariance matrix of the voiceprint multidimensional vector is calculated. The covariance matrix reflects the correlation between each feature dimension. For example, for a voiceprint multidimensional vector dataset containing N samples, each sample has D feature dimensions, then the covariance matrix is a D×D matrix, in which the elements represent the covariance between different feature dimensions.
[0099] Then, the covariance matrix is decomposed by eigenvalue. By calculating the eigenvalue and eigenvector, the main component direction of the data can be obtained. The larger the eigenvalue, the more important the main component represented by the corresponding eigenvector.
[0100] Finally, select the eigenvectors corresponding to the first K largest eigenvalues, project the original voiceprint multidimensional vector into the subspace formed by these K eigenvectors, and obtain the key voiceprint multidimensional vector. This process can be achieved through matrix multiplication, that is, multiplying the original voiceprint multidimensional vector with the matrix composed of these K eigenvectors.
[0101] In the step of inputting the key voiceprint multidimensional vector into the first hidden layer of the voiceprint emotion association model pre-trained based on the deep neural network, the deep neural network has a strong nonlinear mapping ability and feature learning ability, and can automatically learn complex patterns and relationships from the data. In voiceprint emotion association analysis, the deep neural network can learn the implicit association between voiceprint features and emotional tendencies through a large amount of training data with emotion annotations, thereby accurately predicting emotional tendencies. Compared with traditional machine learning methods, deep neural networks can process high-dimensional voiceprint data and can automatically extract more advanced and abstract features without the need to manually design complex feature extractors.
[0102] The first hidden layer consists of multiple neurons, each of which is fully connected to the input key voiceprint multidimensional vector. This means that each neuron has a weight vector and a bias term, which is used to linearly weight the input vector. For a neuron, its input is the key voiceprint multidimensional vector X = [x1, x2,..., xK] (K is the dimension of the key voiceprint multidimensional vector), the weight vector is W = [w1, w2,..., wK], and the bias term is b. The linear weighted result of the neuron is z =∑(wi * xi) + b.
[0103] Then, a nonlinear activation function is applied to the linear weighted result. Common activation functions include ReLU (Rectified Linear Unit) function, Sigmoid function, etc. The role of the activation function is to introduce nonlinear factors so that the neural network can learn complex nonlinear relationships. For example, when using the ReLU function, if the linear weighted result z is less than 0, the output is 0; if z is greater than or equal to 0, the output is z itself. This can enhance the expressive power of the neural network and enable it to fit various complex functions.
[0104] Furthermore, the output from the first hidden layer is passed to the second hidden layer for further processing. The neurons in the second hidden layer are fully connected to the neurons in the first hidden layer, which means that each neuron in the second hidden layer can receive the output of all neurons in the first hidden layer as input. This fully connected structure allows the second hidden layer to make full use of the feature information extracted by the first hidden layer to perform more advanced feature combination and abstraction. For example, the first hidden layer learns some basic voiceprint feature patterns, and the second hidden layer can extract more complex features that are more related to emotional tendencies by combining and further processing these basic patterns.
[0105] The neurons in the second hidden layer also perform linear weighting and nonlinear activation function processing on the input, but because they receive the output of the first hidden layer, they can extract and combine features at a higher level. For example, assuming that the output of the first hidden layer is A = [a1, a2,..., aM] (M is the number of neurons in the first hidden layer), the weight vector of a neuron in the second hidden layer is W' = [w1', w2',..., wM'], and the bias term is b', then the linear weighted result of the neuron is z' = ∑(wi' * ai) + b'. Then, the output of the neuron is obtained through nonlinear activation function processing.
[0106] By continuously adjusting the weights and biases of the second hidden layer and optimizing the network parameters during training, the second hidden layer can learn more representative and discriminative emotional features.
[0107] After being processed by the second hidden layer, the network outputs a preliminary emotional tendency value in the range of [0, 1]. This value represents the degree of prediction of the emotional tendency corresponding to the input voiceprint.
[0108] 0 represents an extremely negative emotional tendency, such as strong negative emotions such as anger and sadness; 1 represents an extremely positive emotional tendency, such as strong positive emotions such as joy and excitement; the middle value represents different degrees of emotional state, for example, 0.3 can represent mild negative emotions, and 0.7 can represent moderate positive emotions.
[0109] Usually, the last layer (output layer) of the network uses a specific activation function to map the output of the second hidden layer to between [0, 1]. For example, the Sigmoid activation function can be used, whose output value is between 0 and 1 and has the property of continuous and differentiable, which is convenient for using optimization algorithms such as gradient descent during training.
[0110] When training the network, a large number of voiceprint data samples with emotion annotations are used to continuously adjust the network's weights and biases so that the network's output can be as close to the real emotional tendency label as possible. For example, for a voiceprint sample labeled as positive emotion, the network should output a value close to 1 after training; for a voiceprint sample labeled as negative emotion, the network should output a value close to 0. By continuously optimizing the network's parameters, the prediction error on the training set is minimized, thereby improving the network's prediction accuracy for emotional tendency.
[0111] In one embodiment, the data fusion processing of the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value includes:
[0112] The preliminary sentiment tendency value is normalized and mapped to the interval [0, 1] to obtain the normalized sentiment tendency value; the language feature mark is converted into a corresponding digital code to form a language code value;
[0113] Constructing a fusion weight matrix, wherein the fusion weight matrix includes weight parameters for different languages under different emotional tendencies, wherein the weight parameters are obtained in advance by training and statistically analyzing multi-language and multi-emotion data;
[0114] Selecting corresponding weight parameters from the fusion weight matrix according to the language encoding value and the normalized sentiment tendency value;
[0115] The normalized sentiment tendency value is multiplied by the selected weight parameter to obtain a weighted sentiment tendency value component; the language code value is multiplied by the corresponding weight parameter to obtain a weighted language code value component;
[0116] The weighted sentiment tendency value component and the weighted language coding value component are vector-added to obtain an initial vector of the comprehensive feature value; the initial vector of the comprehensive feature value is normalized to make its value within a preset range to obtain the comprehensive feature value.
[0117] In this embodiment, the numerical range and distribution of the preliminary sentiment tendency value are inconsistent. In order to facilitate subsequent data fusion and processing, it is necessary to normalize it to the interval [0, 1]. The normalization method can adopt linear mapping. For example, if the original preliminary sentiment tendency value range is [-10, 10], it can be mapped to the interval [0, 1] by the formula: normalized sentiment tendency value = (original value + 10) / 20. This allows different sentiment tendency values to be compared and fused on the same scale, avoiding the problem of weight imbalance caused by differences in numerical ranges. After normalization, the normalized sentiment tendency value obtained is more comparable and operational, which is convenient for fusion with other features in subsequent calculations.
[0118] Language feature tags originally exist in text form or other complex representations. In order to be conveniently used in mathematical calculations, they need to be converted into corresponding digital codes to form language code values. Code conversion can be done in the form of integer codes. For example, if the languages to be processed are English, Chinese, French, etc., they can be encoded as integers such as 1, 2, and 3 respectively. In this way, each language has a unique digital identifier, which is convenient for operation in subsequent calculations. The advantage of digital codes is that they can directly participate in mathematical operations, such as multiplication and addition, and can save storage space and improve calculation efficiency.
[0119] Different languages have different characteristics and expressions under different emotional tendencies. The role of the fusion weight matrix is to provide appropriate weight parameters for data fusion based on the combination of language and emotional tendencies. For example, when some languages express positive emotions, certain speech features will be more prominent, while when expressing negative emotions, other speech features will be more representative. The fusion weight matrix contains weight parameters for different languages under different emotional tendencies. It can perform targeted weighting processing on the preliminary emotional tendency value and language encoding value according to the specific language and emotional situation, so as to better integrate the information of the two.
[0120] Construction method: The fusion weight matrix is constructed in advance by training and statistically analyzing a large amount of multilingual and multi-emotional data. First, collect speech data samples containing various languages and various emotional tendencies, and annotate these samples, including language labels and emotional tendency labels. Then, use machine learning algorithms or statistical methods to analyze the feature distribution and importance of different languages under different emotional tendencies. For example, a neural network model can be used, with language encoding values and emotional tendency values as inputs, and the network can be trained to learn the optimal weight parameters under different combinations. During the training process, the weight parameters are continuously adjusted so that the model can output the most accurate comprehensive feature values based on the input language and emotional information. Through learning and optimization of a large amount of data, a fusion weight matrix containing reasonable weight parameters is finally obtained.
[0121] The language coding value and the normalized emotional tendency value determine the combination of the language and emotional state of the current speech data. According to this combination, the corresponding weight parameter is selected from the fusion weight matrix to provide a suitable weight value for the subsequent weighting process. For example, if the language coding value represents English and the normalized emotional tendency value represents a medium positive emotion, then the corresponding weight parameter is found in the fusion weight matrix based on these two pieces of information to weight the preliminary emotional tendency value and language coding value of English under a medium positive emotion.
[0122] The language encoding value and the normalized sentiment tendency value can be used as indexes to search in the fusion weight matrix. For example, if the fusion weight matrix is a two-dimensional matrix, where the rows represent the language encoding values and the columns represent the different intervals of the sentiment tendency values, then the corresponding row can be found according to the current language encoding value, and then the corresponding column can be found according to the interval where the sentiment tendency value is located, so as to determine the corresponding weight parameter.
[0123] The normalized sentiment tendency value is multiplied by the selected weight parameter to obtain the weighted sentiment tendency value component. The purpose of this is to adjust and strengthen the normalized sentiment tendency value according to the current language and emotional situation. For example, if the selected weight parameter is large, it means that the importance of the sentiment tendency value is higher under the current language and emotional combination, and a larger weight needs to be given. Similarly, the language coding value is multiplied by the corresponding weight parameter to obtain the weighted language coding value component. This is to highlight the importance of the language coding value in the current situation and make corresponding adjustments based on different language and emotional combinations.
[0124] The weighted sentiment tendency value component and the weighted language coding value component are vector-added to obtain the initial vector of the comprehensive feature value. The process of vector addition is to combine the two weighted feature components together to form a comprehensive vector containing sentiment and language information.
[0125] The purpose of normalizing the initial vector of the comprehensive eigenvalue is to make the final comprehensive eigenvalue within a preset reasonable value range to facilitate subsequent model processing and analysis. Normalization can avoid the impact of excessively large or small values on subsequent calculations and model performance, and can also make different data comparable. For example, if normalization is not performed, some data will dominate the model calculation due to their large values, resulting in the influence of other data being ignored.
[0126] The normalization method can adopt common normalization methods, such as minimum-maximum normalization or Z-score normalization. After normalization, the final comprehensive feature value is obtained, which contains both emotional tendency information and language feature information, and is within a suitable value range, which is convenient for subsequent multilingual speech recognition and other related processing tasks.
[0127] In one embodiment, the comprehensive feature value is input into an emotion perception AI model trained by a deep neural network for processing to obtain a multilingual speech recognition result with emotion color annotation and language information annotation, including:
[0128] The comprehensive feature value is input into the input layer of the emotion perception AI model; the emotion perception AI model includes an input layer, multiple hidden layers and an output layer; the hidden layer includes multiple neuron nodes, and each node performs weighted summation and nonlinear activation function processing on the input data;
[0129] When the hidden layer receives the comprehensive feature value transmitted by the input layer, based on the attention mechanism, key features related to emotional color and language information are focused on and strengthened;
[0130] After gradual processing and feature extraction by multiple hidden layers, the output layer generates multilingual speech recognition results with emotional color annotations and language information annotations based on the data passed from the hidden layers; among them, the emotional color annotation is expressed by combining emotional intensity values and emotional type classification labels, and the language information annotation is expressed by international standard language codes.
[0131] In this embodiment, the input layer is the entry point for the emotion perception AI model to receive external data. Its main task is to preliminarily organize and transmit the comprehensive feature values in a form suitable for model processing. The comprehensive feature values contain key information such as emotional tendencies and language characteristics after preliminary processing and fusion. The input layer converts it into a numerical format that can be calculated within the model and passes it to the next layer (hidden layer).
[0132] The hidden layer contains multiple neuron nodes, which are the core units of the model for feature learning and information processing. Each neuron has the ability to perform weighted summation on the input data and apply nonlinear activation functions. Each neuron has a set of weight parameters, which are continuously learned and adjusted during the model training process. When the input data (comprehensive eigenvalue) enters the neuron, the neuron multiplies each feature component of the input by the corresponding weight, then adds these products, plus a bias term, to obtain a weighted summation result. This process can be expressed as: z = ∑(wi * xi) + b, where wi is the weight, xi is the input eigenvalue, and b is the bias term.
[0133] After the weighted summation, the neuron applies a nonlinear activation function to the result. Common activation functions include ReLU (Rectified Linear Unit), Sigmoid, Tanh, etc. The role of the activation function is to introduce nonlinear characteristics to the model so that it can learn and simulate complex real-world data relationships. For example, the ReLU function maps values less than 0 to 0 and values greater than 0 remain unchanged, which can increase the sparsity and expressiveness of the model.
[0134] Emotion-aware AI models usually contain multiple hidden layers, each of which can further extract and abstract features from the input data. The lower hidden layers learn some basic feature patterns, such as the spectral features of speech and the local features of voiceprints; while the higher hidden layers can learn more advanced and abstract feature combinations related to emotional color and language information based on the features extracted by the lower layers. The output of each hidden layer serves as the input to the next hidden layer. In this way, the model can gradually learn more and more complex and abstract feature representations. For example, the first hidden layer extracts some simple speech features, and the second hidden layer can combine these simple features into more complex feature patterns, and so on.
[0135] In multilingual speech recognition, the feature distribution of emotional color and language information is very complex, and different features have different importance for the final recognition results. The introduction of the attention mechanism is to enable the model to automatically focus on the key features related to emotional color and language information, while relatively ignoring some unimportant features or noise. For example, in a speech, the pronunciation features of certain words are very important for the expression of emotions, while other background noise or irrelevant speech fragments can be ignored. Through the attention mechanism, the model can use limited computing resources more efficiently and focus on processing key features, thereby improving the accuracy and efficiency of recognition.
[0136] First, the model calculates an attention weight distribution based on the current hidden layer state and the comprehensive feature value of the input. This weight distribution represents the degree of attention the model pays to different feature positions or feature dimensions. For example, a neural network-based attention model can be used, taking the hidden layer state and comprehensive feature value as input, and outputting an attention weight vector through a series of calculations and transformations.
[0137] Then, the attention weights are weighted and summed with the input comprehensive feature values. In this way, features with higher attention weights will make a greater contribution after weighted summation, while features with lower attention weights will be relatively weakened. In this way, the model achieves focused attention and strengthening of key features. For example, in emotional color analysis, the attention mechanism will pay more attention to features closely related to emotions, such as changes in intonation in speech and the pronunciation intensity of specific words; in language information recognition, it will focus on key features related to language, such as certain specific frequency intervals in the spectral features of speech or stress patterns in the rhythmic features of speech.
[0138] The output layer is the last layer of the emotion perception AI model. Its main task is to generate multilingual speech recognition results with emotional color annotations and language information annotations based on the information passed from the hidden layer after multiple processing and feature extraction.
[0139] For emotional color labeling, a combination of emotional intensity values and emotional type classification labels is used. The emotional intensity value can indicate the intensity of the emotion, for example, a value range from 0 to 1, where 0 indicates no emotional tendency at all and 1 indicates a very strong emotion. The emotional type classification label divides emotions into different categories, such as happiness, sadness, anger, calmness, etc. The output layer further calculates and classifies the information transmitted by the hidden layer to determine the emotional intensity value and the classification label to which it belongs.
[0140] For language information annotation, international standard language codes are used. The output layer recognizes the language of the input speech based on the language feature patterns learned by the model and outputs the corresponding language code, such as "en" for English and "zh" for Chinese.
[0141] The output layer usually contains some fully connected layers and classifiers. The fully connected layer performs the final integration and transformation of the features passed from the hidden layer and maps them to the output space. The classifier classifies the emotion type and identifies the language based on the output of the fully connected layer. During the training process, the model learns from a large amount of multilingual speech data with emotional color annotations and language information annotations, and continuously adjusts the parameters of the output layer so that the output results can be as close to the actual annotation information as possible. In practical applications, when a new comprehensive feature value is input, the model can quickly generate accurate multilingual speech recognition results based on the knowledge and parameters that have been learned.
[0142] In one embodiment, the attention mechanism automatically adjusts the attention paid to different parts of the comprehensive feature value according to the weight parameters learned during the training process;
[0143] The hidden layer is a recursive neural network or a long short-term memory network.
[0144] In this embodiment, in multilingual speech processing, the comprehensive feature value contains rich information, but not all parts contribute equally to the emotional color and language information. The key role of the attention mechanism is to be able to intelligently distinguish which parts of the comprehensive feature value are more critical and need to be given more attention, and which parts are relatively less important and can be appropriately reduced in attention according to the experience accumulated by the model during the training process and the weight parameters learned. During the training process, the model learns through a large amount of multilingual speech data with annotations. In each training iteration, the model calculates an attention distribution based on the current parameter state and the input comprehensive feature value. This distribution is actually a set of weight values, corresponding to different parts of the comprehensive feature value. For the learning of weight parameters, an optimization algorithm based on gradient descent is usually used. The model will back-propagate and calculate the gradient according to the difference (loss function value) between the predicted result and the actual annotation, and then update the weight parameters. As the training continues, the weight parameters are gradually adjusted to a state that enables the model to better pay attention to the key features related to emotional color and language information. For example, when the attention mechanism is applied in the long short-term memory network (LSTM), the hidden state and cell state of the LSTM will participate in the calculation of the attention weight. According to the current task requirements and historical information, the model automatically adjusts the attention to different time steps or feature dimensions of the comprehensive feature value. If the feature of a certain time step is very important for the current emotion or language judgment, then the attention weight corresponding to the time step will increase, otherwise it will decrease.
[0145] Multilingual speech data has the characteristics of time series, that is, speech is pronounced in chronological order. Recurrent neural networks (RNN) and long short-term memory networks (LSTM) are very suitable for processing such time series data. They can remember previous information and combine it with current input for analysis, so as to better understand the contextual information and dynamic changes of speech.
[0146] In one embodiment, the multilingual speech recognition result also includes speech recognition text content;
[0147] After obtaining the multilingual speech recognition results with emotional color annotations and language information annotations, the method further includes:
[0148] Obtaining the emotion level corresponding to each of the emotion color labels; wherein the database stores a mapping relationship between the emotion color labels and the emotion levels; the emotion levels are represented by digital codes;
[0149] Obtaining identification characters corresponding to each of the language information labels; wherein each of the language information labels is represented by an international standard language code in the form of English characters;
[0150] Obtain a standard character matrix table; each cell of the character matrix table is filled with characters;
[0151] Based on the identification character, the character matrix table is changed to obtain a changed character matrix table;
[0152] Based on the numerical codes in each of the emotion levels, a virtual curve is generated;
[0153] According to a preset superposition rule, the virtual curve and the changed character matrix table are superimposed; in the changed character matrix table, all characters passed by the virtual curve are searched as target characters;
[0154] The target characters are combined in sequence, and the obtained character combination is used as a key to encrypt and save the speech recognition text content.
[0155] In this embodiment, the above multilingual speech recognition results include speech recognition text content. The above AI model can not only analyze the emotional color and language information in the speech, but also convert the speech into corresponding text content. Through speech recognition technology, such as a speech-to-text algorithm based on deep learning, the input multilingual speech stream is converted into a readable text form. This is a basic and key function of the multilingual speech recognition system, which provides a text basis for subsequent information processing and application.
[0156] A mapping relationship between emotion color labeling and emotion level is pre-established in the database. Emotion color labeling is usually a descriptive label for speech emotion, such as "happy", "sad", "angry", etc. The emotion level uses numerical codes to quantify the intensity or degree of these emotions. For example, the emotion color label of "happy" can be mapped to the emotion level "3", indicating a medium-intensity happy emotion.
[0157] When the multilingual speech recognition results with emotional color annotations are obtained, the corresponding emotional level is searched in the database according to the annotations. Through fast data retrieval and matching algorithms, the emotional level in the form of digital codes corresponding to the current emotional color annotations is accurately found. This step enables the emotional information to be subsequently processed and operated in a more concise digital form.
[0158] Language information annotation is represented by international standard language codes, usually in the form of English characters, such as "en" for English and "fr" for French. These codes are internationally recognized identifiers that facilitate communication and understanding between different systems and applications. After the language information of the speech is recognized, the corresponding language code is directly read as the language information annotation. Then, this language code is used as an identification character for subsequent processing. For example, if the recognized language is English, the identification character obtained is "en".
[0159] The above character matrix table is a predefined table structure containing various characters. Each cell is filled with at least one character, which can be letters, numbers, symbols, etc. The role of the character matrix table is to serve as a basic data structure to provide character resources in the subsequent encryption process.
[0160] Based on the obtained identification character, a specific change operation is performed on the character matrix table. This change can be to adjust the order of rows and columns of the character matrix table according to certain rules, or to replace or rearrange the characters in the character matrix table according to the value of the identification character. For example, if the identification character is "en", the characters at specific positions in the first row and second column of the character matrix table can be exchanged.
[0161] The emotional level is represented by a digital code. Through a specific curve generation algorithm, the digital code of the emotional level is used as an input parameter to generate a virtual curve with a corresponding shape and characteristics. This algorithm can draw a virtual curve in a two-dimensional or three-dimensional space according to the size and range of the digital code and the preset curve generation rules. For example, a mathematical function is used, the digital code is used as the input variable of the function, and the coordinates of each point on the curve are determined according to the output value of the function, thereby generating a continuous virtual curve. Alternatively, the emotional level with a smaller digital code can generate a relatively gentle curve, while the emotional level with a larger digital code can generate a curve with greater fluctuations. Alternatively, a curve can be generated together according to multiple digital codes, for example, multiple digital codes are generated as several coordinate points according to preset rules, and then multiple coordinate points are connected by straight lines or curves, and a virtual curve is obtained after smoothing. There are many ways as mentioned above, which will not be repeated here.
[0162] The preset superposition rules define how to combine and interact with the virtual curve and the character matrix table. For example, you can specify the position relationship of the virtual curve with the character matrix table in a specific coordinate system, and how to determine which characters are considered as target characters when the virtual curve passes through the cells of the character matrix table.
[0163] According to the preset superposition rule, the virtual curve is placed at the spatial position corresponding to the changed character matrix table. Then, by calculating the intersection of the virtual curve and the character matrix table or the position of the cell passed through, it is determined which characters are located on the path of the virtual curve. These characters passed through by the virtual curve are the target characters. For example, if the virtual curve is represented as a continuous line in a two-dimensional coordinate system, when it passes through a cell of the character matrix table, the character in the cell is determined as the target character.
[0164] The determined target characters are combined in order according to their appearance order in the character matrix table to form a character combination. This character combination will be used as a key to encrypt the speech recognition text content. The key generation process is based on the comprehensive processing results of the emotional level and language information, and has a certain degree of randomness and uniqueness, which increases the security of encryption. The generated key is used to encrypt the speech recognition text content. Common encryption algorithms, such as symmetric encryption algorithms or asymmetric encryption algorithms, can be used to convert the text content into a ciphertext. Then, the encrypted ciphertext is saved in a storage device, such as a hard disk, a database, or a cloud storage. In this way, even if others obtain the saved information, they cannot directly understand the content. Only with the correct decryption method and key can it be restored to the original speech recognition text content. This step ensures the security and confidentiality of the text content in the multilingual speech recognition results.
[0165] Reference Figure 4 In another embodiment of the present invention, a multilingual speech recognition device based on an emotion perception AI model is provided, comprising:
[0166] The segmentation module is used to perform time series segmentation processing on the collected multilingual voice streams to obtain multiple basic voice units of equal duration, and convert each basic voice unit into a voiceprint multidimensional vector;
[0167] The association analysis module is used to input the voiceprint multi-dimensional vector into the voiceprint emotion association model to perform emotion association analysis and obtain a preliminary emotion tendency value;
[0168] The matching module is used to perform audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; perform feature coding processing on each spectrum interval to generate spectrum feature codes; perform language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags;
[0169] A fusion module is used to perform data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value;
[0170] The recognition module is used to input the comprehensive feature values into the emotion perception AI model trained by the deep neural network for processing, and obtain multilingual speech recognition results with emotional color annotations and language information annotations.
[0171] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.
[0172] Reference Figure 5 In an embodiment of the present invention, a computer device is also provided. The computer device may be a server, and its internal structure may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.
[0173] Those skilled in the art will understand that Figure 5The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.
[0174] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0175] In summary, the multilingual speech recognition method, device and equipment based on the emotion perception AI model provided in the embodiment of the present invention include: performing time series segmentation processing on the collected multilingual speech stream to obtain multiple basic speech units of equal duration, and converting each basic speech unit into a voiceprint multidimensional vector; inputting the voiceprint multidimensional vector into the voiceprint emotion association model for emotion correlation analysis processing to obtain a preliminary emotion tendency value; performing audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; performing feature encoding processing on each spectrum interval to generate spectrum feature coding; performing language feature search and matching processing on the spectrum feature coding to obtain matching language feature tags; performing data fusion processing on the preliminary emotion tendency value and the language feature tag to obtain a comprehensive feature value; inputting the comprehensive feature value into the emotion perception AI model trained by a deep neural network for processing to obtain a multilingual speech recognition result with emotion color annotation and language information annotation. In the present invention, by combining emotion analysis and multilingual recognition, the defect of inaccurate multilingual speech recognition is overcome.
[0176] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.
[0177] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.
[0178] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A multilingual speech recognition method based on emotion-aware AI model, It is characterized in that The following steps are involved: Perform time series segmentation processing on the collected multilingual voice stream to obtain multiple basic voice units of equal duration, and convert each basic voice unit into a voiceprint multidimensional vector; Input the voiceprint multidimensional vector into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value; Performing audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; performing feature coding processing on each spectrum interval to generate spectrum feature codes; performing language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags; Performing data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value; The comprehensive feature values are input into the emotion perception AI model trained by deep neural network for processing, and multilingual speech recognition results with emotion color annotation and language information annotation are obtained; The data fusion processing of the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value includes: The preliminary sentiment tendency value is normalized and mapped to the interval [0, 1] to obtain the normalized sentiment tendency value; the language feature mark is converted into a corresponding digital code to form a language code value; Constructing a fusion weight matrix, wherein the fusion weight matrix includes weight parameters for different languages under different emotional tendencies, wherein the weight parameters are obtained in advance by training and statistically analyzing multi-language and multi-emotion data; Selecting corresponding weight parameters from the fusion weight matrix according to the language encoding value and the normalized sentiment tendency value; The normalized sentiment tendency value is multiplied by the selected weight parameter to obtain a weighted sentiment tendency value component; the language code value is multiplied by the corresponding weight parameter to obtain a weighted language code value component; The weighted sentiment tendency value component and the weighted language coding value component are vector-added to obtain an initial vector of the comprehensive feature value; the initial vector of the comprehensive feature value is normalized to make its value within a preset range to obtain the comprehensive feature value.
2. The multilingual speech recognition method based on the emotion perception AI model according to claim 1, It is characterized in that The step of converting each basic speech unit into a voiceprint multi-dimensional vector comprises: Extract voiceprint features from each basic speech unit to obtain basic voiceprint feature data; The basic voiceprint feature data is converted into a multi-dimensional vector to obtain a voiceprint multi-dimensional vector.
3. The multilingual speech recognition method based on the emotion perception AI model according to claim 1, It is characterized in that The step of performing feature coding processing on each spectrum interval to generate spectrum feature codes includes: Calculate the frequency energy value of each spectrum interval to obtain the frequency energy value data corresponding to each spectrum interval; Perform feature coding processing on all frequency energy value data to generate spectrum feature codes.
4. The multilingual speech recognition method based on the emotion perception AI model according to claim 1, It is characterized in that The method of inputting the voiceprint multi-dimensional vector into the voiceprint emotion association model to perform emotion association analysis and processing to obtain a preliminary emotion tendency value includes: Perform principal component analysis on the voiceprint multidimensional vector to obtain the key voiceprint multidimensional vector; Input the key voiceprint multidimensional vector into the first hidden layer of the voiceprint emotion association model pre-trained based on a deep neural network; the first hidden layer is composed of multiple neurons, and each neuron performs linear weighting and nonlinear activation function processing on the input vector; The output from the first hidden layer is passed to the second hidden layer. The neurons in the second hidden layer are fully connected with the neurons in the first hidden layer, and the output of the first hidden layer is further feature extracted and combined. After being processed by the second hidden layer, a preliminary emotional tendency value in the range of [0, 1] is output; among them, 0 represents an extremely negative emotional tendency, 1 represents an extremely positive emotional tendency, and the intermediate values represent different degrees of emotional states.
5. The multilingual speech recognition method based on the emotion perception AI model according to claim 1, It is characterized in that The comprehensive feature values are input into the emotion perception AI model trained by the deep neural network for processing, and multilingual speech recognition results with emotion color annotations and language information annotations are obtained, including: The comprehensive feature value is input into the input layer of the emotion perception AI model; the emotion perception AI model includes an input layer, multiple hidden layers and an output layer; the hidden layer includes multiple neuron nodes, and each node performs weighted summation and nonlinear activation function processing on the input data; When the hidden layer receives the comprehensive feature value transmitted by the input layer, based on the attention mechanism, the key features related to the emotional color and language information are paid attention to; After gradual processing and feature extraction by multiple hidden layers, the output layer generates multilingual speech recognition results with emotional color annotations and language information annotations based on the data passed from the hidden layers; among them, the emotional color annotation is expressed by combining emotional intensity values and emotional type classification labels, and the language information annotation is expressed by international standard language codes.
6. The multilingual speech recognition method based on the emotion perception AI model according to claim 5, It is characterized in that The attention mechanism automatically adjusts the attention to different parts of the comprehensive feature value according to the weight parameters learned during the training process; The hidden layer is a recursive neural network or a long short-term memory network.
7. A multilingual speech recognition device based on emotion perception AI model, It is characterized in that include: The segmentation module is used to perform time series segmentation processing on the collected multilingual voice streams to obtain multiple basic voice units of equal duration, and convert each basic voice unit into a voiceprint multidimensional vector; The association analysis module is used to input the voiceprint multi-dimensional vector into the voiceprint emotion association model to perform emotion association analysis and obtain a preliminary emotion tendency value; The matching module is used to perform audio spectrum division processing on each basic speech unit to obtain multiple continuous spectrum intervals; perform feature coding processing on each spectrum interval to generate spectrum feature codes; perform language feature search and matching processing on the spectrum feature codes to obtain matching language feature tags; A fusion module is used to perform data fusion processing on the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value; The recognition module is used to input the comprehensive feature values into the emotion perception AI model trained by the deep neural network for processing, and obtain multilingual speech recognition results with emotion color annotations and language information annotations; The data fusion processing of the preliminary sentiment tendency value and the language feature mark to obtain a comprehensive feature value includes: The preliminary sentiment tendency value is normalized and mapped to the interval [0, 1] to obtain the normalized sentiment tendency value; the language feature mark is converted into a corresponding digital code to form a language code value; Constructing a fusion weight matrix, wherein the fusion weight matrix includes weight parameters for different languages under different emotional tendencies, wherein the weight parameters are obtained in advance by training and statistically analyzing multi-language and multi-emotion data; Selecting corresponding weight parameters from the fusion weight matrix according to the language encoding value and the normalized sentiment tendency value; The normalized sentiment tendency value is multiplied by the selected weight parameter to obtain a weighted sentiment tendency value component; the language code value is multiplied by the corresponding weight parameter to obtain a weighted language code value component; The weighted sentiment tendency value component and the weighted language coding value component are vector-added to obtain an initial vector of the comprehensive feature value; the initial vector of the comprehensive feature value is normalized to make its value within a preset range to obtain the comprehensive feature value.
8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
