Audio analysis method based on user characteristics

Abstract

The invention relates to the technical field of audio analysis, in particular to an audio analysis method based on user characteristics, which comprises the following steps: acquiring an audio signal of a target user, and preprocessing the audio signal to obtain a standard audio signal; and feature extraction is carried out on the standard audio signal to obtain an audio feature corresponding to the standard audio signal, and the audio feature comprises a spectrogram and a Mel-frequency cepstrum coefficient. According to the method, the audio features and the text features are extracted at the same time and fused, so that information of different modes can be fully utilized, the audio features such as the spectrogram and the Mel-frequency cepstrum coefficient can provide voice features, tones, rhythms, speed and other information of sound, the text features can provide semantic content of language, and therefore, the method has the advantages of being high in practicability and the like. Through multi-modal fusion of the audio signal and the text, the emotion color in the voice and the language expression thereof can be fully considered, so that the accuracy of emotion classification is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio analysis, and in particular to an audio analysis method based on user characteristics. Background Art

[0002] Audio analysis refers to the processing, extraction and interpretation of audio signals through various technical means to obtain meaningful information. Audio analysis can be used in a variety of applications, including speech recognition, sentiment analysis, audio content classification, sound separation, noise elimination, music recommendation and other fields. The core purpose of audio analysis is to extract useful data that can reflect the characteristics and content of the original audio signal to achieve more efficient recognition and processing.

[0003] Traditional methods lack multimodal fusion and usually rely on only one feature in audio or text for sentiment classification. For example, using audio features (such as tone, speaking speed, pitch, etc.) or text features (such as semantics, syntactic structure, etc.) alone may not be able to fully reflect the emotional state, which limits the accuracy and robustness of sentiment classification, especially in complex situations. For example, when emotional expressions have multiple meanings or the emotions are more complex, the information of a single modality cannot be fully captured; and traditional methods that use audio features alone are often affected by noise and environmental changes; and traditional methods usually do not have a deep enough understanding of the context, especially in text sentiment analysis. Many times, they rely on the emotional judgment of a single sentence or a single paragraph of text, while ignoring the contextual information behind the emotional expression. Emotional expression is dynamic and is greatly affected by the context before and after. Therefore, it is difficult to fully understand and analyze emotions with text or audio information alone. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide an audio analysis method based on user characteristics.

[0005] The technical solution adopted to solve the above technical problems is: an audio analysis method based on user characteristics, including: Acquire an audio signal of a target user, and preprocess the audio signal to obtain a standard audio signal; Performing feature extraction on the standard audio signal to obtain audio features corresponding to the standard audio signal, wherein the audio features include a spectrogram and a Mel-frequency cepstral coefficient; Performing speech recognition on the audio signal based on a pre-trained speech recognition model to obtain an audio text, and pre-processing the audio text to obtain a standard audio text; Performing feature extraction on the standard audio text to obtain text features corresponding to the standard audio text, and performing multimodal fusion of the text features and the audio features to obtain fusion features; The target user is sentimentally classified based on the fusion feature to obtain the sentiment label corresponding to the target user.

[0006] Preferably, preprocessing the audio signal to obtain a standard audio signal includes: The audio signal is pre-emphasized to obtain a pre-emphasized audio signal, wherein the pre-emphasis formula is as follows: ; in, Indicates the pre-emphasized audio signal, Indicates The audio signal of sampling points, Indicates The audio signal of sampling points, represents a pre-emphasis coefficient; performing windowing processing on the audio signal to obtain a standard audio signal, wherein the formula for windowing processing is as follows: ; in, Represents a standard audio signal, represents the windowing function, and , Indicates the length of the windowed audio signal.

[0007] Preferably, performing feature extraction on the standard audio signal to obtain an audio feature corresponding to the standard audio signal includes: Perform a fast Fourier transform on the standard audio signal to obtain frequency domain information corresponding to the standard audio signal, wherein the formula of the fast Fourier transform is as follows: ; in, Indicates the frequency domain information corresponding to the standard audio signal. Indicates the length of the standard audio signal, represents the frequency index, represents the complex exponential function.

[0008] Preferably, performing feature extraction on the standard audio signal to obtain an audio feature corresponding to the standard audio signal further includes: Arranging the frequency domain information corresponding to the standard audio signal of each frame in time sequence to obtain a frequency spectrum corresponding to the standard audio signal; Calculating the spectral line energy of the frequency domain information, and calculating the logarithmic spectrum of the frequency domain information based on the spectral line energy; Performing discrete cosine transformation on the logarithmic spectrum of the frequency domain data to obtain Mel-cepstral coefficients of the standard audio signal, wherein the expression of the Mel-cepstral coefficients is as follows: ; in, Represents the Mel cepstral coefficients of the standard audio signal, Indicates frequency index Logarithmic spectrum of the complex spectral components at .

[0009] Preferably, feature extraction is performed on the standard audio text to obtain text features corresponding to the standard audio text, including: Segmenting the standard audio text to obtain an audio segmentation set; Performing vector encoding on the audio segmentations in the audio segmentation set to obtain segmentation vectors; The word segmentation vector is input into the BERT pre-training model to obtain the output vector of each layer of the BERT pre-training model, and the CLS vector is taken from the output vector. The output vector expression is as follows: ; in, Represents the output vector of each layer of the BERT pre-trained model, Indicates the vector encoding of audio words in the audio word set. represents the word segmentation vector, represents the BERT pre-trained model, Represents the number of layers in the BERT pre-trained model.

[0010] Preferably, feature extraction is performed on the standard audio text to obtain text features corresponding to the standard audio text, and further includes: The CLS vector of the first layer in the BERT pre-trained model is deleted, and the CLS vectors of the remaining layers in the BERT pre-trained model are concatenated to obtain a coding vector, and the coding vector expression is as follows: ; in, represents the encoding vector, Represents the CLS vectors of the remaining layers in the BERT pre-trained model; Inputting the encoding vector into the Bi-LSTM model to obtain feature information; The feature information is fused with the output vector of the BERT pre-trained model, and a vector representation is obtained through a fully connected layer network, that is, the text features corresponding to the standard audio text are obtained.

[0011] Preferably, the text feature and the audio feature are multimodally fused to obtain a fused feature, including: Based on a pre-trained feature fusion module, multimodal fusion is performed on the text features and the audio features to obtain fused features.

[0012] Preferably, the feature fusion module combines the text features and the audio features multiple times based on the feature branches to perform an attention mechanism. After passing through multiple mutual attention modules, the text features and the audio features are fused, and then the two are spliced ​​to obtain the final fusion features. The mutual attention module first performs self-attention calculation on the audio features, and both the query and key-value pairs use the audio features, perform mutual attention calculation on the audio features and the text features, and use the feedforward layer and the GELU activation function to enhance the audio features.

[0013] Preferably, the emotion classification of the target user is performed based on the fusion feature to obtain the emotion label corresponding to the target user, including: The target user's emotions are classified based on the fusion features by using a pre-trained emotion classification module to obtain an emotion label corresponding to the target user.

[0014] Preferably, the sentiment classification module first extracts abstract features through the fusion features of the time-delay neural network, uses SE-Res2Blocks to model the correlation between global channels, and concatenates and aggregates the output features of multiple layers of SE-Res2Blocks through the TDNN model, then uses a dense layer to generate features for attention statistical pooling, and finally obtains the sentiment label corresponding to the target user through a fully connected layer and Softmax.

[0015] The beneficial effects of the present invention are as follows: (1) The present invention can fully utilize information of different modalities by extracting both audio and text features at the same time and fusing them. Audio features such as spectrogram and Mel-frequency cepstral coefficients (MFCC) can provide information such as voice characteristics, pitch, rhythm, speaking speed, etc., while text features can provide semantic content of language. Through multimodal fusion of audio signals and text, the emotional color and language expression in speech can be fully considered, thereby improving the accuracy of emotion classification; (2) The present invention can more comprehensively identify emotions by fusing features of different modalities, improve the robustness of the system in different environments and different contexts, and make emotion classification more accurate and stable; (3) The present invention uses a pre-trained speech recognition model to convert audio signals into text, which not only saves time and computing costs in the training process, but also uses a speech recognition model trained with a large amount of data to provide efficient and accurate text conversion capabilities. High-quality text information can provide more accurate input for subsequent emotion analysis, further improving the performance of emotion classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic flow chart of the steps of an overall method in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Embodiment 1, as Figure 1 As shown, the present invention proposes an audio analysis method based on user characteristics, comprising: S1. Acquire an audio signal of a target user and pre-process the audio signal to obtain a standard audio signal; S2. Extracting features of the standard audio signal to obtain audio features corresponding to the standard audio signal, wherein the audio features include a spectrogram and a Mel-frequency cepstrum coefficient; S3, performing speech recognition on the audio signal based on a pre-trained speech recognition model to obtain an audio text, and preprocessing the audio text to obtain a standard audio text; S4, extracting features from the standard audio text to obtain text features corresponding to the standard audio text, and performing multimodal fusion of the text features and the audio features to obtain fusion features; S5. Perform sentiment classification on the target user based on the fusion features to obtain the sentiment label corresponding to the target user.

[0018] In the present invention, the audio signal is converted into an electrical signal through the propagation of sound waves, which is usually collected by a microphone or other device. It can be a representation of speech, music or any other sound form; the preprocessing of the audio signal refers to a series of operations on the original audio signal, the purpose of which is to improve the effect of subsequent analysis or processing; the spectrum diagram shows the energy distribution of the audio signal at different frequencies, which is obtained by methods such as Fourier transform, and reflects the change of the frequency components of the signal over time; Mel-frequency cepstral coefficients (MFCC), MFCC is a feature widely used in speech recognition, which describes the audio characteristics of the audio signal. By mapping the spectrum diagram to the Mel scale (close to the frequency perception method of human hearing), MFCC can effectively represent the main features of the audio signal; the speech recognition model is a model based on machine learning, which is used to convert the audio signal into corresponding text or text. It analyzes the audio features (such as MFCC, etc.) to identify the language information contained therein. Common speech recognition models include deep neural networks (DNN), convolutional neural networks (CNN) and recursive neural networks (RNN), etc.; multimodal fusion refers to combining features from different modes (such as audio and text) for more accurate analysis. In the method, the audio features of the audio signal (such as spectrogram and MFCC) and the text features obtained through speech recognition are combined to form a fusion feature. This fusion feature contains information from both audio and text, and is used to more comprehensively analyze the user's emotions. Sentiment classification is a natural language processing task, whose goal is to identify and classify the emotional information (such as happiness, anger, sadness, etc.) in the input text or speech content. In this method, based on the fusion features (i.e., the features of audio and text), the model classifies the target user's emotions and finally obtains the user's emotion label. The emotion label is an annotation of the emotion classification result, which is used to represent the user's emotional state. For example, the emotion label may be "happy", "sad", "angry" or "neutral".

[0019] Embodiment 2, an audio analysis method based on user characteristics proposed by the present invention, compared with embodiment 1, this embodiment further includes: preprocessing the audio signal to obtain a standard audio signal, including: The audio signal is pre-emphasized to obtain a pre-emphasized audio signal, wherein the pre-emphasis formula is as follows: ; in, Indicates the pre-emphasized audio signal, Indicates The audio signal of sampling points, Indicates The audio signal of sampling points, Indicates the pre-emphasis coefficient; The audio signal is subjected to windowing processing to obtain a standard audio signal, wherein the formula for windowing processing is as follows: ; in, Represents a standard audio signal, represents the windowing function, and , Indicates the length of the windowed audio signal.

[0020] In this embodiment, pre-emphasis is a common signal processing technology, which is used to enhance the high-frequency part of the audio signal. In the speech signal, the high-frequency part usually contains more speech information, while the low-frequency part is easily affected by noise. Pre-emphasis performs differential processing on the original audio signal to enhance the high-frequency component, thereby improving the clarity and recognizability of the signal; windowing is a commonly used technology in audio signal processing, which is used to decompose a long time series signal into short time windows for further analysis. Since the signal changes continuously in time, the purpose of windowing is to localize the signal so that the signal can be considered to be stable in each time window, thereby performing effective frequency domain analysis (such as Fourier transform); in windowing, a fixed-length time window is usually selected to segment the signal, and the signal in each window can be considered to be stable. Then, Fourier transform (or other spectrum analysis methods) is performed on each window to extract frequency features. The role of the window function is to reduce the sudden change of the signal at both ends of the window and avoid spectrum leakage.

[0021] In an optional embodiment, feature extraction is performed on the standard audio signal to obtain audio features corresponding to the standard audio signal, including: Perform fast Fourier transform on the standard audio signal to obtain the frequency domain information corresponding to the standard audio signal, wherein the formula of fast Fourier transform is as follows: ; in, Indicates the frequency domain information corresponding to the standard audio signal. Indicates the length of the standard audio signal, represents the frequency index, represents the complex exponential function.

[0022] It should be noted that frequency domain information is a description of the frequency components of the signal. After Fourier transform, the signal is converted from time domain to frequency domain, which means that we can obtain the components of different frequencies in the signal as well as their amplitude and phase. Through frequency domain analysis, we can understand which frequency components are contained in the signal and their strengths.

[0023] In an optional embodiment, extracting features from the standard audio signal to obtain audio features corresponding to the standard audio signal further includes: Arrange the frequency domain information corresponding to each frame of the standard audio signal in time order to obtain a frequency spectrum corresponding to the standard audio signal; Calculate the spectral line energy of the frequency domain information, and calculate the logarithmic spectrum of the frequency domain information based on the spectral line energy; The logarithmic spectrum of the frequency domain data is subjected to discrete cosine transformation to obtain the Mel cepstral coefficient of the standard audio signal, where the expression of the Mel cepstral coefficient is as follows: ; in, Represents the Mel cepstral coefficients of the standard audio signal, Indicates frequency index Logarithmic spectrum of the complex spectral components at .

[0024] It should be noted that spectral line energy refers to the energy of each frequency component in the spectrum, which reflects the importance of each frequency component in the signal; the logarithmic spectrum is the logarithmic transformation of the spectrum amplitude, which helps to compress large-amplitude spectral information while enhancing low-amplitude frequency components; the Mel frequency scale is a frequency scale that simulates human auditory perception. The human ear's perception of frequency is nonlinear, with a higher resolution of frequency in the low frequency band and a lower resolution of frequency in the high frequency band.

[0025] In an optional embodiment, feature extraction is performed on the standard audio text to obtain text features corresponding to the standard audio text, including: Segment the standard audio text to obtain an audio segmentation set; Vectorize the audio words in the audio word set to obtain word vectors; The word segmentation vector is input into the BERT pre-trained model to obtain the output vector of each layer of the BERT pre-trained model, and the CLS vector is taken from the output vector. The output vector expression is as follows: ; in, Represents the output vector of each layer of the BERT pre-trained model, Indicates the vector encoding of audio words in the audio word set. represents the word segmentation vector, represents the BERT pre-trained model, Represents the number of layers in the BERT pre-trained model.

[0026] It should be noted that audio segmentation is the process of converting audio into meaningful units (such as words, syllables, or audio clips); audio segmentation is based on the text after speech recognition, so it is also similar to text segmentation in natural language processing; the purpose is to decompose the speech signal in the audio into operable audio "segmentation" units; the output of audio segmentation can be words, phrases, or other audio-based units; the audio segmentation set refers to the result set after segmentation from the standard audio text; the audio segmentation set is a collection of individual audio "segmentation" units extracted from the audio signal, usually containing text information after the speech recognition results of the audio signal are segmented; the segmentation vector refers to the process of converting the segmentation (i.e., audio segmentation) into a high-dimensional vector representation; this process is usually completed using a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.); vector encoding is the process of converting audio segmentation into a vector representation; this is usually achieved through a trained model (such as an embedding model); BERT (Bidirectional Encoder Representations from Transformers)** is a pre-trained language model based on the Transformer architecture, which can be used for a variety of natural language processing tasks, such as text classification, named entity recognition, question answering, etc.; BERT is characterized by bidirectional encoding, which can consider contextual information at the same time, unlike traditional language models that only consider the order from left to right; in the BERT model, the CLS token is a special token that is usually at the beginning of the input sequence; the BERT model uses the CLS vector to represent the contextual information of the entire input sequence; usually, when the input text is processed, the vector of the CLS token is used for sequence-level tasks (such as classification tasks); it serves as an aggregate representation of the entire sentence or text; the BERT model usually consists of multiple Transformer layers; each layer outputs a vector representation to capture more complex language features; the number of layers determines the depth of the model, and usually, the BERT model has 12 or 24 layers (for example, BERT-base and BERT-large models).

[0027] In an optional embodiment, feature extraction is performed on the standard audio text to obtain text features corresponding to the standard audio text, and further includes: The CLS vector of the first layer in the BERT pre-trained model is deleted, and the CLS vectors of the remaining layers in the BERT pre-trained model are concatenated to obtain the encoding vector. The encoding vector expression is as follows: ; in represents the encoding vector, Represents the CLS vectors of the remaining layers in the BERT pre-trained model; Input the encoded vector into the Bi-LSTM model to obtain feature information; The feature information is fused with the output vector of the BERT pre-trained model, and the vector representation is obtained through a fully connected layer network, that is, the text features corresponding to the standard audio text are obtained.

[0028] It should be noted that feature fusion refers to combining feature information from different models or different sources in order to improve the performance of the model from multiple angles. Feature fusion fuses the feature information obtained from Bi-LSTM with the output vector of the original BERT pre-trained model (that is, the last layer output of BERT). In this way, the feature information of two different models can be combined to enhance the model's understanding of the text; the fully connected layer is a type of layer in a neural network, in which each input node is connected to each output node. The function of the fully connected layer is to linearly transform the input features (through a weight matrix) and then process them through nonlinear activation functions (such as ReLU, sigmoid, etc.) to learn more complex feature representations; text features refer to numerical vectors obtained after processing by models (such as BERT, Bi-LSTM, etc.) that can represent text content, semantics, and contextual relationships.

[0029] In an optional embodiment, the text features and the audio features are multimodally fused to obtain fused features, including: Based on the pre-trained feature fusion module, multimodal fusion of text features and audio features is performed to obtain fused features.

[0030] In an optional embodiment, the feature fusion module combines text features and audio features multiple times based on feature branches to perform an attention mechanism. After passing through multiple mutual attention modules, the text features and audio features are fused, and then the two are spliced ​​to obtain the final fusion features. The mutual attention module first performs self-attention calculation on the audio features. Both the query and key-value pairs use audio features, perform mutual attention calculation on the audio features and text features, and use the feedforward layer and GELU activation function to enhance the audio features.

[0031] It should be noted that Feature Branches refers to the fact that in the model structure, the input data (such as audio, text, etc.) may be split into multiple different processing paths (branches), each branch is responsible for processing a specific feature type, and the audio features and text features are processed through different "branches" respectively, and finally merged in the feature fusion stage; Attention Mechanism is a commonly used technology in neural networks, which aims to enable the model to "focus" on the key parts of the input data, so as to effectively process long sequence data. In natural language processing (NLP) and computer vision (CV), the attention mechanism is particularly important because it can help the model automatically select the most relevant information without processing all inputs; Cross-Attention Module is a commonly used technology in multimodal learning. In this module, audio features and text features influence each other through the cross-attention mechanism. The model can weight and adjust according to the correlation between the two features, which helps to capture the relationship between text and audio information and improve the comprehensive understanding ability of the model; Feedforward Layer Layer) refers to a standard layer in a neural network, which maps the input signal to a higher dimension through a linear transformation, and then performs a nonlinear transformation through an activation function (such as ReLU, GELU, etc.).

[0032] In an optional embodiment, emotion classification of the target user is performed based on the fusion features to obtain an emotion label corresponding to the target user, including: The target user's emotions are classified based on the fusion features through the pre-trained sentiment classification module to obtain the emotion label corresponding to the target user.

[0033] In an optional embodiment, the sentiment classification module first extracts abstract features through time-delay neural network fusion features, uses SE-Res2Blocks to model the correlation between global channels, and concatenates and aggregates the output features of multiple layers of SE-Res2Blocks through the TDNN model, then uses a dense layer to generate features for attention statistical pooling, and finally obtains the sentiment label corresponding to the target user through a fully connected layer and Softmax.

[0034] It should be noted that the time-delay neural network is a neural network architecture that specializes in processing sequence data. TDNN can capture the time dependency in the data by introducing a "time delay" mechanism between the input layer and the hidden layer. It is particularly suitable for processing time series data such as speech and audio. Through the "sliding window" method, TDNN can extract local features between time steps in the sequence and then learn time correlation; SE-Res2Blocks is based on the structure of ResNet (residual network) and SE (Squeeze-and-Excitation) modules. It is an improved version of the convolutional neural network module that combines residual learning and channel attention mechanism; global channel correlation modeling refers to the SE-Res2Blocks. The SE module in 2Blocks models the correlation between different channels. In convolutional neural networks, features are usually extracted from multiple channels, and each channel represents a different feature of the input data. Global channel correlation modeling adjusts the importance of each channel so that the model can focus on the feature channels that are useful for sentiment classification tasks and suppress unimportant channels. The TDNN model refers to the use of a time-delay neural network (TDNN) to process input features. The above description mentions the use of TDNN to concatenate and aggregate the output features of multiple layers of SE-Res2Blocks. The TDNN model integrates and processes the features of multiple layers to capture the temporal characteristics of the data, thereby obtaining a richer feature representation. The dense layer (Dense Layer, also known as the fully connected layer, is a common layer type in neural networks, which means that each input node is connected to each output node. In this model, the dense layer passes the concatenated and aggregated features into the network, performs linear transformation and nonlinear activation to extract more advanced features, and finally generates features for attention statistical pooling; Attention pooling is a pooling method that introduces an attention mechanism to perform weighted averaging on features. In sentiment classification, attention pooling can help the model weight the importance of features at each moment in the input sequence to extract the most representative features. This pooling method allows the model to automatically learn how to focus on key information in the input sequence; Statistical pooling refers to performing statistical operations on features (such as mean, variance, etc.) to extract global features. In this model, the pooling operation combined with the attention mechanism can generate a more effective global representation; Fully Connected Layer (FC) Layer is a classic layer type of neural network. Each neuron is connected to all neurons in the previous layer. In the sentiment classification task, the last fully connected layer is used to generate the final sentiment prediction result based on the pooled features. Softmax is an activation function for multi-classification problems, which converts the real value output by the network into a probability distribution.

[0035] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An audio analysis method based on user characteristics, characterized in that: include: Acquire an audio signal of a target user, and preprocess the audio signal to obtain a standard audio signal; Performing feature extraction on the standard audio signal to obtain audio features corresponding to the standard audio signal, wherein the audio features include a spectrogram and a Mel-frequency cepstral coefficient; Performing speech recognition on the audio signal based on a pre-trained speech recognition model to obtain an audio text, and pre-processing the audio text to obtain a standard audio text; Performing feature extraction on the standard audio text to obtain text features corresponding to the standard audio text, and performing multimodal fusion of the text features and the audio features to obtain fusion features; The target user is sentimentally classified based on the fusion feature to obtain the sentiment label corresponding to the target user.

2. The method for audio analysis based on user characteristics according to claim 1, characterized in that: Preprocessing the audio signal to obtain a standard audio signal includes: The audio signal is pre-emphasized to obtain a pre-emphasized audio signal, wherein the pre-emphasis formula is as follows: ； in, Indicates the pre-emphasized audio signal, Indicates The audio signal of sampling points, Indicates The audio signal of sampling points, Indicates the pre-emphasis coefficient; The audio signal is subjected to windowing processing to obtain a standard audio signal, wherein the formula for windowing processing is as follows: ； in, Represents a standard audio signal, represents the windowing function, and , Indicates the length of the windowed audio signal.

3. The audio analysis method based on user characteristics according to claim 2, characterized in that: Extracting features from the standard audio signal to obtain audio features corresponding to the standard audio signal includes: Perform a fast Fourier transform on the standard audio signal to obtain frequency domain information corresponding to the standard audio signal, wherein the formula of the fast Fourier transform is as follows: ； in, Indicates the frequency domain information corresponding to the standard audio signal. Indicates the length of the standard audio signal, represents the frequency index, represents the complex exponential function.

4. The method for audio analysis based on user characteristics according to claim 3, characterized in that: Extracting features from the standard audio signal to obtain audio features corresponding to the standard audio signal also includes: Arranging the frequency domain information corresponding to the standard audio signal of each frame in time sequence to obtain a frequency spectrum corresponding to the standard audio signal; Calculating the spectral line energy of the frequency domain information, and calculating the logarithmic spectrum of the frequency domain information based on the spectral line energy; Performing discrete cosine transformation on the logarithmic spectrum of the frequency domain data to obtain Mel-cepstral coefficients of the standard audio signal, wherein the expression of the Mel-cepstral coefficients is as follows: ； in, Represents the Mel cepstral coefficients of the standard audio signal, Indicates frequency index Logarithmic spectrum of the complex spectral components at .

5. The method for audio analysis based on user characteristics according to claim 4, characterized in that: Performing feature extraction on the standard audio text to obtain text features corresponding to the standard audio text includes: Segmenting the standard audio text to obtain an audio segmentation set; Performing vector encoding on the audio segmentations in the audio segmentation set to obtain segmentation vectors; The word segmentation vector is input into the BERT pre-training model to obtain the output vector of each layer of the BERT pre-training model, and the CLS vector is taken from the output vector. The output vector expression is as follows: ； in, Represents the output vector of each layer of the BERT pre-trained model, Indicates the vector encoding of audio words in the audio word set. represents the word segmentation vector, represents the BERT pre-trained model, Represents the number of layers in the BERT pre-trained model.

6. The method for audio analysis based on user characteristics according to claim 5, characterized in that: Extracting features from the standard audio text to obtain text features corresponding to the standard audio text also includes: The CLS vector of the first layer in the BERT pre-trained model is deleted, and the CLS vectors of the remaining layers in the BERT pre-trained model are concatenated to obtain a coding vector, and the coding vector expression is as follows: ； in, represents the encoding vector, Represents the CLS vectors of the remaining layers in the BERT pre-trained model; Inputting the encoding vector into the Bi-LSTM model to obtain feature information; The feature information is fused with the output vector of the BERT pre-trained model, and a vector representation is obtained through a fully connected layer network, that is, the text features corresponding to the standard audio text are obtained.

7. The method for audio analysis based on user characteristics according to claim 6, characterized in that: The text feature and the audio feature are multimodally fused to obtain a fused feature, including: Based on a pre-trained feature fusion module, multimodal fusion is performed on the text features and the audio features to obtain fused features.

8. The method for audio analysis based on user characteristics according to claim 7, characterized in that: The feature fusion module combines the text features and the audio features multiple times based on the feature branches to perform an attention mechanism. After passing through multiple mutual attention modules, the text features and the audio features are fused, and then the two are spliced ​​to obtain the final fusion features. The mutual attention module first performs self-attention calculation on the audio features, and both the query and key-value pairs use the audio features, perform mutual attention calculation on the audio features and the text features, and use the feedforward layer and GELU activation function to enhance the audio features.

9. The method for audio analysis based on user characteristics according to claim 8, characterized in that: Performing emotion classification on the target user based on the fusion feature to obtain an emotion label corresponding to the target user includes: The target user's emotions are classified based on the fusion features by using a pre-trained emotion classification module to obtain an emotion label corresponding to the target user.

10. The method for audio analysis based on user characteristics according to claim 9, characterized in that: The sentiment classification module first extracts abstract features through the fusion features of the time-delay neural network, uses SE-Res2Blocks to model the correlation between global channels, and concatenates and aggregates the output features of multiple layers of SE-Res2Blocks through the TDNN model. Then, a dense layer is used to generate features for attention statistical pooling, and finally the sentiment label corresponding to the target user is obtained through a fully connected layer and Softmax.

Images