Chinese acoustic text depression detection system based on channel attention mechanism fusion

Abstract

The invention provides a Chinese acoustic text depression detection system based on channel attention mechanism fusion, and the system obtains an original audio signal of a target object through each unit in the system, carries out the preprocessing, obtains audio data, carries out the text transcription, obtains the corresponding text data, carries out the fragment division of the audio data, and carries out the segmentation of the audio data. Corresponding audio clips are obtained, and audio-text data are formed; performing feature extraction on the audio clips and the text data to obtain low-level descriptor features, high-level time-frequency features, audio waveform features and text features; fusing the low-level descriptor features, the high-level time-frequency features and the audio waveform features step by step to determine acoustic features, performing cross-modal fusion on the acoustic features and the text features to obtain fusion features, performing depression detection according to the fusion features, and determining a depression detection result of the target object. According to the scheme, more comprehensive acoustic expression and text features are used, voice text modes are effectively fused, depression detection is achieved, and clinical doctors are assisted in diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of application of artificial intelligence in depression, in particular to a Chinese acoustic text depression detection system based on channel attention mechanism fusion. BACKGROUND

[0002] Depression, as a common mental health problem, has three main symptoms including persistent low mood, loss of interest or pleasure, and decreased energy. In extreme cases, depressed patients may have suicidal thoughts or even take action. According to the latest data from the World Health Organization, more than 300 million people worldwide are affected by depression, nearly half of whom are distributed in Southeast Asia and the western Pacific region, including China. Currently, the diagnosis of depression mainly relies on questionnaire surveys and assessments by medical professionals. However, the treatment rate of depression patients worldwide is still not optimistic, mainly due to two reasons.

[0003] Firstly, traditional depression treatment methods are not only time-consuming and costly, but sometimes ineffective. For economically disadvantaged individuals, the cost of diagnosis and treatment constitutes a heavy financial burden, which leads them to be reluctant to seek professional medical help. Secondly, in clinical interviews, patients may conceal their true psychological state due to fear of prejudice or discrimination, making it difficult for clinicians to make accurate diagnoses.

[0004] In this case, it is particularly important to develop an objective and accurate technology for early diagnosis and re-evaluation of depression. Such technology not only helps individuals privately assess their depression status, increasing their willingness to seek psychologist consultation, but also provides psychologists with a powerful diagnostic tool when patients hide their true psychological state. In recent years, clinical observations and research have found that the language behavior of depressed patients exhibits some specific characteristics, such as slow speech, monotonous tone, low volume, frequent pauses, and narrow range of pitch, which are significantly different from non-depressed individuals. Therefore, depression recognition technology based on speech signals has become a new focus of research due to its low cost, easy collection, and non-invasive nature. Depression detection methods based on audio combined with deep learning have become a research hotspot in this field due to their efficient feature extraction and powerful pattern recognition capabilities. By analyzing features such as speech rate, tone, rhythm, and intensity in speech signals, deep learning models can identify unique patterns of depressed patients. This method not only improves the accuracy of depression diagnosis, but also helps to achieve early intervention and treatment of depression, thereby reducing the impact of depression on individuals and society.

[0005] In the field of automatic diagnosis of depression based on speech, some methods involve extracting low-level descriptor (LLD) features, Mel spectrograms, and other single features as inputs to recurrent neural networks (RNNs), and finally fusing text features. LLD features refer to low-level time and frequency domain characteristics designed for the original audio signal. These features are usually calculated on a single frame of speech signal to represent the characteristics of that frame. Specifically, these features are directly extracted from the waveform of the audio signal, covering time and frequency domain attributes such as short-time energy, zero-crossing rate, spectral centroid, etc. Recurrent neural networks (RNNs) are a neural network structure specifically designed to handle sequential data, capable of efficiently capturing dependencies within time series. Common variants of RNNs include long short-term memory networks (LSTMs) and gated recurrent units (GRUs), which can more effectively handle long-term dependencies and avoid the gradient vanishing problem in traditional RNNs. RNN models are trained through supervised learning, during which the model predicts the target output based on the input LLD feature sequence, i.e., whether the sample has depression. The training goal of the model is to minimize the loss function between the predicted output and the true label, and the model parameters are updated through the backpropagation algorithm.

[0006] In the field of depression diagnosis, the fusion of multi-modal data can provide more comprehensive and complementary information compared to single data modalities, thereby enhancing the accuracy and reliability of diagnosis. Multi-modal data includes but is not limited to speech, text, physiological signals, and other types, each of which can reflect individual psychological state and emotional changes from different angles. For example, the pitch, rhythm, and intensity of speech can reveal emotional state, while text content can provide individual self-expression and cognitive patterns, and physiological signals such as heart rate and skin electrical activity can reflect the body's stress response. In recent years, depression detection technology based on multi-modal data has made some progress, especially with the support of deep learning technology, which can automatically extract and learn complex patterns in data, providing a new perspective for depression recognition. For example, in the text part, BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture, which introduces the Transformer architecture that uses self-attention mechanisms to solve the long dependency problem in traditional recurrent neural networks (RNNs). The self-attention mechanism allows the model to dynamically focus on different parts of the input sequence when processing each input, thereby better capturing contextual information.

[0007] However, due to the difficulty of data acquisition and cross-cultural differences, current research mainly focuses on western languages such as English and Spanish, while the research on depression diagnosis models for non-western languages such as Chinese is relatively lagging behind. This limits the application of multi-modal depression detection technology on a global scale, especially among Chinese user groups. In order to fill this gap, it is of great significance to develop a depression diagnosis model specifically for Chinese. SUMMARY

[0008] The purpose of the embodiments of the present application is to provide a Chinese acoustic text depression detection system based on channel attention mechanism fusion, which extracts multi-level audio representations in time and spectral feature space to provide more complete audio modal representations containing more extensive information, designs a multi-level audio feature interaction module that can effectively fuse information of different levels of features together, realizes the fusion of three-level speech features and text features, uses more comprehensive acoustic expressions and localized text features, effectively fuses speech and text modalities, realizes depression detection, and assists clinicians in diagnosis.

[0009] In order to achieve the above purpose, the embodiments of the present application are realized by the following way: In a first aspect, the embodiments of the present application provide a Chinese acoustic text depression detection system based on channel attention mechanism fusion, comprising: a data acquisition unit configured to acquire an original audio signal of a target object; a data preprocessing unit configured to preprocess the original audio signal to obtain audio data; a data transcription unit configured to transcribe the audio data to text data corresponding to the audio data, wherein each sentence text in the text data contains its start and end time stamp in the audio data; an audio segmentation unit configured to segment the audio data based on the text data to obtain a plurality of audio segments corresponding to the sentence texts in the text data, each audio segment and the corresponding sentence text being a group of audio-text data; a feature extraction unit configured to extract multi-level features from the audio segments in each group of audio-text data to determine low-level descriptor features, high-level time-frequency features and audio waveform features corresponding to the audio segments, and to extract features from the sentence texts in each group of audio-text data to determine text features corresponding to the sentence texts; a feature fusion unit configured to fuse the low-level descriptor features, the high-level time-frequency features and the audio waveform features in each group of audio-text data step by step to determine acoustic features, and then to fuse the acoustic features and the text features in each group of audio-text data to obtain fusion features; a depression detection unit configured to detect depression based on the fusion features to determine a depression detection result of the target object.

[0010] In a first possible implementation manner of the first aspect, the feature extraction unit comprises an acoustic feature extraction subunit, and the acoustic feature extraction subunit is configured to: for each audio-text data in each group of audio-text data: extract low-level descriptor features from the audio segment by using an OpenSmile toolkit; decompose the audio segment into a plurality of window data by using a sliding window, calculate a mel-scale spectrum of each window data, and input the mel-scale spectrum of each window data into a convolutional neural network to obtain high-level time-frequency features; and determine audio waveform features of the audio segment by using a pre-trained wav2vec model, wherein the wav2vec model comprises a plurality of layers of convolutional neural networks.

[0011] In a second possible implementation manner of the first aspect, the feature extraction unit further comprises a text feature extraction subunit, and the text feature extraction subunit is configured to: for each sentence text in each group of audio-text data: extract text features of the sentence text by using a pre-trained BERT-china-base model.

[0012] In a third possible implementation manner of the first aspect, the feature fusion unit comprises an acoustic feature fusion subunit, and the acoustic feature fusion subunit comprises a first fusion module and a second fusion module, and the acoustic feature fusion subunit is configured to: for each group of audio-text data: perform first-level fusion on the low-level descriptor features and the high-level time-frequency features by using the first fusion module to determine first-level fusion features ; and perform second-level fusion on the first-level fusion features and the audio waveform features by using the second fusion module to determine acoustic features.

[0013] In the fourth possible implementation manner of the first aspect, the first fusion module comprises a first cross-attention sub-module, a second cross-attention sub-module, a first gating signal sub-module, a second gating signal sub-module, a first fusion sub-module, a second fusion sub-module, and a first feature splicing sub-module, and the acoustic feature fusion subunit is specifically configured to: input the high-level time-frequency features and the low-level descriptor features into the first cross-attention sub-module to generate interaction features ; input the and the in the first cross-attention sub-module into the first gating signal sub-module to generate a gating signal ; input the gating signal and the interaction features into the first fusion sub-module to generate component features ; input the low-level descriptor features and the high-level time-frequency features input the second cross-attention sub-module to generate interaction features ; input the interaction features and input the second gating signal sub-module to generate a gating signal ; input the gating signal and the interaction features input the second fusion sub-module to generate component features ; input the component features and the component features input the first feature splicing sub-module to splice to obtain first-level fusion features .

[0014] With reference to the third possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the second fusion module has a third cross-attention sub-module, a fourth cross-attention sub-module, a third gating signal sub-module, a fourth gating signal sub-module, and a third fusion sub-module, a fourth fusion sub-module, and a second feature splicing sub-module, and the acoustic feature fusion subunit is specifically configured to: input the audio waveform features and the first-level fusion features input the third cross-attention sub-module to generate interaction features ; input the interaction features and input the third gating signal sub-module to generate a gating signal ; input the gating signal and the interaction features input the third fusion sub-module to generate component features ; input the first-level fusion features and the audio waveform features input the fourth cross-attention sub-module to generate interaction features ; input the interaction features and input the fourth gating signal sub-module to generate a gating signal ; input the gating signal and the interaction features input the fourth fusion sub-module to generate component features ; input the component features and the component features input the second feature splicing sub-module to splice to obtain acoustic features .

[0015] With reference to the third possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the loss function term of the first fusion module is: , wherein, is a loss function term of the first fusion module, represents a difference between a distribution of and a distribution of , represents a difference between a distribution of and a distribution of ; a loss function term of the second fusion module is: , wherein, is a loss function term of the second fusion module, represents a difference between a distribution of and a distribution of , represents a difference between a distribution of and a distribution of .

[0016] With reference to the first aspect, in a seventh possible implementation manner of the first aspect, the feature fusion unit comprises a cross-modal fusion subunit, the cross-modal fusion subunit having a cross-modal attention module, and the cross-modal fusion subunit is configured to: for each group of audio-text data: input the acoustic feature and the text feature of the audio-text data into the cross-modal attention module to perform cross-modal feature stacking: , , , wherein, represents a stacking operation, , , respectively represent a query vector, a key vector and a value vector of the cross-modal attention module, and respectively represent the corresponding query vectors of the acoustic feature and the text feature , and respectively represent the corresponding key vectors of the acoustic feature and the text feature , and respectively represent the corresponding value vectors of the acoustic feature and the text feature ; calculate the fusion feature of the audio-text data: , wherein, is a transpose of the fusion feature, is a transpose of the fusion feature, is a dimension of the fusion feature. is a dimension of the fusion feature.

[0017] With reference to the seventh possible implementation of the first aspect, in an eighth possible implementation of the first aspect, the loss function term of the cross-modal fusion subunit is characterized in that: , wherein, is the loss function term of the cross-modal fusion subunit, is a transpose of the fusion feature, is a transpose of the fusion feature, is a dimension of the fusion feature. is a dimension of the fusion feature. represents a square of a Frobenius norm of the fusion feature, represents a square of a Frobenius norm of the fusion feature, represents a square of a Frobenius norm of the fusion feature, represents a square of a Frobenius norm of the fusion feature, represents splicing.

[0018] With reference to the first aspect, in a ninth possible implementation of the first aspect, the depression detection unit is specifically configured to: perform feature aggregation on the fusion feature corresponding to each group of audio-text data pairs to form a sample-level feature of the target object; and input the sample-level feature into a fully connected network to output a depression detection result of the target object, wherein a loss function term of the fully connected network is: , wherein, is the loss function term of the fully connected network, is a true label of the sample, is a model prediction value.

[0019] Advantages: ​The system provided in the scheme fuses Chinese acoustic text depression detection system based on channel attention mechanism, obtains the original audio signal of the target object through the data acquisition unit, preprocesses the original audio signal through the data preprocessing unit, and obtains the audio data; the audio data is transcribed by the data transcription unit to obtain the text data corresponding to the audio data (each sentence text contains its start and end timestamp in the audio data); the audio segmentation unit divides the audio data based on the text data to obtain a plurality of audio segments corresponding to the sentence text in the text data, and each audio segment and the corresponding sentence text are taken as a group of audio-text data; the feature extraction unit extracts multi-level features from the audio segment in each group of audio-text data, determines the low-level descriptor feature, high-level time-frequency feature and audio waveform feature corresponding to the audio segment, and extracts features from the sentence text in each group of audio-text data to determine the text feature corresponding to the sentence text; the feature fusion unit fuses the low-level descriptor feature, high-level time-frequency feature and audio waveform feature in each group of audio-text data step by step to determine the acoustic feature, and then fuses the acoustic feature and the text feature in each group of audio-text data to obtain the fusion feature; the depression detection unit detects depression based on the fusion feature to determine the depression detection result of the target object. In the aspect of acoustic feature extraction, the system constructs a three-level speech feature structure, extracts low-level descriptor features (LLD), high-level time-frequency features based on Mel spectrum graph, and deep speech embedding obtained by pre-training wav2vec model. In view of the modal complementarity contained in different levels of features, the system uses an optimized fusion strategy, combines an improved feature interaction mechanism and a calculation formula, realizes effective information integration under the premise of preserving the independence of the information of each level of features, avoids feature confusion and redundant superposition, and thus constructs a more discriminative comprehensive acoustic representation. The text part uses the BERT Chinese version pre-trained on a large amount of Chinese text, can process the characteristics of Chinese text such as multi-sound character and word order, and fuses the text and speech modalities based on the channel attention mechanism, which can enable the model to allocate different weights according to the different contributions of the modalities to the diagnosis. This framework specially designed for the Chinese context can promote better localized cross-modal feature representation and improve the accuracy of depression detection. In the cross-modal fusion of speech and text, considering the high heterogeneity of speech and text in statistical properties and expression methods, the system designs a fusion module based on the channel attention mechanism, which can adaptively learn and allocate the contribution weight of each modality to the task. This mechanism not only improves the ability of the model to utilize complementary information between modalities, but also strengthens the cross-modal semantic alignment effect, thereby generating high-quality, unified multi-modal fusion embedding representation, significantly improving the robustness and accuracy of depression recognition.

[0020] In the three-level fusion process of acoustic features, the acoustic feature fusion subunit, the first cross-attention sub-module, the second cross-attention sub-module, the first gating signal sub-module, the second gating signal sub-module, and the first fusion sub-module, the second fusion sub-module, the first feature splicing sub-module, the third cross-attention sub-module, the fourth cross-attention sub-module, the third gating signal sub-module, the fourth gating signal sub-module, and the third fusion sub-module, the fourth fusion sub-module, and the second feature splicing sub-module are involved in the speech feature fusion process at different levels, respectively. The multi-level feature fusion process avoids the strong wav2vec feature directly "flooding" other features, ensuring that unique information in the LLD and Mel features can be fully retained and utilized. The bidirectional complementary attention mechanism ensures that complementary information from two perspectives is fully captured in each level of the fusion process. The gating mechanism introduced in the fusion process can completely close the information flow judged as redundant or useless, allowing valuable information to pass through, greatly improving the purity and effectiveness of the fused features. The splicing operation ensures that all useful information extracted in the two-direction attention process is completely transmitted to the next stage, reducing any loss. In the cross-modal fusion process of acoustic features and text features, a unified joint attention space is constructed through stacking operation, so that the model can dynamically and bidirectionally capture the complex correlation within and between audio and text modalities. Residual connection is used to ensure that the original feature information is not lost, and finally a powerful representation that is deeply fused and retains the original information is generated, greatly enhancing the discriminability of multi-modal representation.

[0021] Through hierarchical loss function design, the fusion process is optimized from two dimensions of distribution alignment and feature orthogonality: and KL divergence is used to force the distribution alignment of input features at the same level in the representation space, reducing the difficulty of fusion. Then through the orthogonal constraint, the cross-modal features have both complementarity (audio and text features are independent of each other) and information increment, finally realizing the deep fusion of redundancy minimization and information maximization. Feature-level aggregation strategy is adopted to integrate the deep fusion features of all dialogue segments of the target object into a unified sample-level representation, and then through the full connection network and binary cross-entropy loss for end-to-end optimization, so as to realize the accurate discrimination of individual overall depression state, avoiding the instability of local decision based on segments, and significantly improving the comprehensive judgment ability and robustness of the model.

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0024] Figure 1 The architecture diagram of the Chinese acoustic text depression detection system based on channel attention mechanism fusion provided in the embodiments of the present application.

[0025] Icon: 10-Chinese acoustic text depression detection system based on channel attention mechanism fusion; 11-data acquisition unit; 12-data preprocessing unit; 13-data transcription unit; 14-audio segmentation unit; 15-feature extraction unit; 151-acoustic feature extraction subunit; 152-text feature extraction subunit; 16-feature fusion unit; 161-acoustic feature fusion subunit; 162-cross-modal fusion subunit; 17-depression detection unit. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below in combination with the drawings in the embodiments of the present application.

[0027] Please refer to Figure 1 , Figure 1 The architecture diagram of the Chinese acoustic text depression detection system 10 based on channel attention mechanism fusion provided in the embodiments of the present application.

[0028] In the present embodiment, the Chinese acoustic text depression detection system 10 based on channel attention mechanism fusion can include a data acquisition unit 11, a data preprocessing unit 12, a data transcription unit 13, an audio segmentation unit 14, a feature extraction unit 15, a feature fusion unit 16, and a depression detection unit 17.

[0029] In the present embodiment, the data acquisition unit 11 is mainly used to acquire the original audio signal of the target object. For example, the system can provide an interface for a user (such as a clinician or other person) to upload the original audio signal of the target object (the object to be analyzed and detected), and the system can acquire the uploaded original audio signal of the target object. It should be noted that for the original audio signal of non-single-person conversation, the original audio signal of the target object needs to be separated from it, which can be realized by voiceprint comparison technology, which is not the focus of the present scheme, and will not be described here.

[0030] After obtaining the original audio signal of the target object, the data preprocessing unit 12 can preprocess the original audio signal (for example, denoising, identifying and cutting off the silent part, in other schemes, the silence duration can also be introduced as an indicator of depression analysis, but this embodiment does not adopt this way) to obtain audio data.

[0031] Then, the data transcription unit 13 can perform text transcription on the audio data to obtain text data corresponding to the audio data, wherein each sentence text in the text data contains its start and end time stamp in the audio data. For example, OpenAI Whisper can be used, which can realize transcription and output the time stamp of the sentence-level text. GoogleCloud Speech-to-Text can also be used, by setting enable_word_time_offsets=True in the request, the returned result will contain the start and end time of each word, and the start and end time stamp of each sentence can be generated accordingly.

[0032] After obtaining the text data corresponding to the audio data, the audio segmentation unit 14 can segment the audio data based on the text data, that is, segment the audio data according to the start and end time stamp of the sentence text in the audio data, thereby obtaining a plurality of audio segments corresponding to the sentence text in the text data, and each audio segment and the corresponding sentence text as a set of audio-text data.

[0033] The feature extraction unit 15 includes an acoustic feature extraction subunit 151 and a text feature extraction subunit 152.

[0034] The acoustic feature extraction subunit 151 can perform multi-level feature extraction on the audio segment in each set of audio-text data to determine the low-level descriptor feature, high-level time-frequency feature and audio waveform feature corresponding to the audio segment.

[0035] Specifically, for the audio segment in each set of audio-text data: the acoustic feature extraction subunit 151 can extract low-level descriptor features from the audio segment by the OpenSmile toolkit. The feature extraction unit 15 can use a sliding window to decompose the audio segment into several window data (generally a short time window of 20-40 ms, which can be set as needed), calculate the mel-scale spectrum of each window data, and input the mel-scale spectrum of each window data into a convolutional neural network (which is already trained and does not need to participate in the end-to-end training process of feature fusion and depression detection of the system), to obtain high-level time-frequency features. In addition, the feature extraction unit 15 can use a pre-trained wav2vec model to extract features from the audio segment to determine the audio waveform features of the audio segment. The wav2vec model contains a multi-layer convolutional neural network for extracting audio waveform features at different frequency and time scales. Similarly, the wav2vec model is also already trained and does not need to participate in the end-to-end training process of feature fusion and depression detection of the system.

[0036] In addition, the text feature extraction subunit 152 can extract features from the sentence text in each set of audio-text data to determine the text features corresponding to the sentence text. For example, a pre-trained BERT-china-base model is used to extract text features of the sentence text. The BERT-china-base model is also already trained and does not need to participate in the end-to-end training process of feature fusion and depression detection of the system.

[0037] After feature extraction, the feature fusion unit 16 can gradually fuse the low-level descriptor features, high-level time-frequency features, and audio waveform features in each set of audio-text data to determine acoustic features.

[0038] In this embodiment, the feature fusion unit 16 includes an acoustic feature fusion subunit 161, which has a first fusion module and a second fusion module. The first fusion module has a first cross-attention sub-module, a second cross-attention sub-module, a first gate signal sub-module, a second gate signal sub-module, and a first fusion sub-module, a second fusion sub-module, and a first feature splicing sub-module. The second fusion module has a third cross-attention sub-module, a fourth cross-attention sub-module, a third gate signal sub-module, a fourth gate signal sub-module, and a third fusion sub-module, a fourth fusion sub-module, and a second feature splicing sub-module.

[0039] For each set of audio-text data: the acoustic feature fusion subunit 161 can first perform one-level fusion of the low-level descriptor features and the high-level time-frequency features through the first fusion module to determine one-level fusion features .

[0040] Exemplarily, the acoustic feature fusion subunit 161 can input the high-level time-frequency feature and the low-level descriptor feature into a first cross-attention sub-module to generate interaction features , for example: , (1) wherein, denotes the low-level descriptor feature corresponding to the query vector, denotes the high-level time-frequency feature corresponding to the key vector, denotes the high-level time-frequency feature corresponding to the value vector, is the dimension of the key vector , and denotes normalizing the calculated similarity scores into a probability distribution.

[0041] Accordingly, the acoustic feature fusion subunit 161 can input the and in the first cross-attention sub-module into a first gating signal sub-module to generate a gating signal . For example: , (2) , (3) , (4) , (5) wherein, is the new query vector after linear transformation, is a learnable weight matrix, is the new key vector after linear transformation, is a learnable weight matrix, denotes the intermediate gating signal obtained after sum-pooling, denotes sum-pooling, denotes the window size of the sum-pooling operation, denotes the normalized gating signal, denotes L2 normalization on . Subsequently, the acoustic feature fusion subunit 161 can input the gating signal and the interaction features into a first fusion sub-module to generate component features . For example: , (6) where, denotes the high-level time-frequency feature , is a component feature of the low-level descriptor feature , is a learnable weight matrix.

[0042] Similarly, the acoustic feature fusion subunit 161 can input the low-level descriptor feature and the high-level time-frequency feature into the second cross-attention sub-module to generate the interaction feature . For example: where, denotes the high-level time-frequency feature corresponding to the query vector, denotes the low-level descriptor feature corresponding to the key vector, denotes the low-level descriptor feature corresponding to the value vector, where is the dimension of the key vector , denotes the normalization of the calculated similarity score into a probability distribution.

[0043] Accordingly, the acoustic feature fusion subunit 161 can input and in the second cross-attention sub-module into the second gating signal sub-module to generate the gating signal . For example: (8) (9) (10) (11) where, is the new query vector after linear transformation, is a learnable weight matrix, is the new key vector after linear transformation, is a learnable weight matrix, denotes the intermediate gating signal obtained after sum-pooling, denotes the sum-pooling, denotes the window size of the sum-pooling operation, denotes the normalized gating signal, denotes the L2 normalization of . Then, the acoustic feature fusion subunit 161 can input the gating signal and the interaction feature to a second fusion sub-module to generate component features . For example: , (12) wherein represents the low-level descriptor feature and the component feature of the high-level time-frequency feature , and is a learnable weight matrix.

[0044] Accordingly, the acoustic feature fusion subunit 161 can input the component feature and the component feature to a first feature concatenation sub-module to concatenate to obtain a first-level fusion feature .

[0045] For each group of audio-text data: the acoustic feature fusion subunit 161 can perform second-level fusion on the first-level fusion feature and the audio waveform feature through a second fusion module to determine the acoustic feature.

[0046] Similarly to the foregoing, the acoustic feature fusion subunit 161 can input the audio waveform feature and the first-level fusion feature to a third cross-attention sub-module to generate an interaction feature , input and in the third cross-attention sub-module to a third gating signal sub-module to generate a gating signal , and input the gating signal and the interaction feature to a third fusion sub-module to generate component features .

[0047] In addition, the acoustic feature fusion subunit 161 can input the first-level fusion feature and the audio waveform feature to a fourth cross-attention sub-module to generate an interaction feature , input and in the fourth cross-attention sub-module to a fourth gating signal sub-module to generate a gating signal , and input the gating signal and the interaction feature to a fourth fusion sub-module to generate component features .

[0048] Accordingly, the acoustic feature fusion subunit 161 can input the component features and the component features to a second feature splicing sub-module to splice the acoustic feature .

[0049] Since the processing procedure of the acoustic feature obtained through the secondary fusion is similar to the processing procedure of the primary fusion feature , no further description is given herein.

[0050] The loss function term of the first fusion module is designed as: , (13) wherein, is the loss function term of the first fusion module, denotes the difference between the distribution of and the distribution of , and denotes the difference between the distribution of and the distribution of .

[0051] The loss function term of the second fusion module is designed as: , (14) wherein, is the loss function term of the second fusion module, denotes the difference between the distribution of and the distribution of , and denotes the difference between the distribution of and the distribution of .

[0052] After obtaining the acoustic feature , the feature fusion unit 16 can perform cross-modal fusion on the acoustic feature and the text feature in each set of audio-text data to obtain a fusion feature.

[0053] In this embodiment, the feature fusion unit 16 further includes a cross-modal fusion subunit 162, which has a cross-modal attention module.

[0054] For each set of audio-text data: the cross-modal fusion subunit 162 can input the acoustic feature and the text feature of the audio-text data to the cross-modal attention module to perform cross-modal feature stacking: , (15) , (16) , (17) in, Indicates a stacking operation. , , These represent the query vector, key vector, and value vector of the cross-modal attention module, respectively. and Representing acoustic features and text features The corresponding query vector, and Representing acoustic features and text features The corresponding key vector, and Representing acoustic features respectively and text features The corresponding value vector.

[0055] Accordingly, the cross-modal fusion subunit 162 can calculate the fusion features of audio-text data: , (18) in, As a feature of fusion, for transpose, for Dimensions.

[0056] The loss function term for the cross-modal fusion subunit 162 is designed as follows: , (19) in, For the loss function term of cross-modal fusion subunit 162, for transpose, for transpose, express The square of the Frobenius norm, express The square of the Frobenius norm, This indicates a splicing operation.

[0057] Accordingly, the feature fusion unit 16 can achieve cross-modal feature fusion for each group of audio-text data to obtain fused features of multiple audio-text data.

[0058] In this embodiment, the depression detection unit 17 can perform depression detection based on fusion features to determine the depression detection result of the target object.

[0059] To ensure the detection effect, we consider the sample-level feature aggregation (i.e., integrating the fusion features of multiple audio-text data, and then performing depression detection) and decision aggregation (i.e., first performing depression detection based on the fusion features of each audio-text data, then integrating the detection results corresponding to the fusion features of each audio-text data to determine the final depression detection result), and finally choose the feature aggregation scheme because the feature aggregation scheme can consider more rich feature information and is beneficial to improve the stability of the detection result.

[0060] Therefore, the depression detection unit 17 can first perform feature aggregation on the fusion features corresponding to each group of audio-text data to form the sample-level features of the target object. For example, the mean value of the fusion features corresponding to all audio-text data is calculated as the sample-level feature of the target object; or the mean value of the fusion features corresponding to all audio-text data is calculated, and then the fusion feature with the maximum length and the fusion feature with the minimum length among the fusion features corresponding to all audio-text data are determined, and are spliced to form a feature as the sample-level feature of the target object.

[0061] Then, the depression detection unit 17 can input the sample-level features into the fully connected network to output the depression detection result of the target object.

[0062] The loss function term (using binary cross-entropy loss) of the fully connected network is designed as: , (20) wherein, is the loss function term of the fully connected network, is the true label of the sample, is the model prediction value.

[0063] The entire system needs to perform end-to-end unified training on the feature fusion unit 16 and the depression detection unit 17, and the loss function of the entire model is: , (21) wherein, is the loss function of the entire model covering the feature fusion unit 16 and the depression detection unit 17 in the system, , , and are the weights of each loss, of course, equal weight can be used, but other ways of allocating weights can also be used, for example, , , , , the present embodiment is not limited.

[0064] In summary, the embodiment of the present application provides a Chinese acoustic text depression detection system 10 based on channel attention mechanism fusion. The original audio signal of the target object is obtained through the data acquisition unit 11, the original audio signal is preprocessed through the data preprocessing unit 12, and the audio data is obtained; the audio data is transcribed into text data through the data transcription unit 13 (each sentence text contains its start and end timestamp in the audio data); the audio data is divided into segments based on the text data through the audio segmentation unit 14, and a plurality of audio segments corresponding to the sentence text in the text data are obtained, each audio segment and the corresponding sentence text are taken as a group of audio-text data; the multi-level feature extraction is performed on the audio segment in each group of audio-text data through the feature extraction unit 15, the low-level descriptor feature, the high-level time-frequency feature and the audio waveform feature corresponding to the audio segment are determined, and the feature extraction is performed on the sentence text in each group of audio-text data, and the text feature corresponding to the sentence text is determined; the low-level descriptor feature, the high-level time-frequency feature and the audio waveform feature in each group of audio-text data are fused level by level through the feature fusion unit 16, the acoustic feature is determined, and the acoustic feature and the text feature in each group of audio-text data are cross-modally fused to obtain the fusion feature; the depression detection unit 17 detects depression based on the fusion feature, and determines the depression detection result of the target object. In the aspect of acoustic feature extraction, the system constructs a three-level speech feature structure, extracts low-level descriptor features (LLD), high-level time-frequency features based on Mel spectrograms, and deep speech embeddings obtained through a pre-trained wav2vec model. In view of the modal complementarity contained in different levels of features, the system uses an optimized fusion strategy, combines an improved feature interaction mechanism and a calculation formula, realizes effective information integration on the premise of preserving the independence of the information of each level of features, avoids feature confusion and redundant superposition, and thus constructs a more discriminative comprehensive acoustic representation. The text part uses the BERT Chinese version pre-trained on a large amount of Chinese text, can process the characteristics of Chinese text such as multi-sound characters and word order, and fuses the text and speech modalities based on the channel attention mechanism, which can enable the model to allocate different weights according to the different contributions of the modalities to the diagnosis. This framework specially designed for the Chinese context can promote better localized cross-modal feature representation and improve the accuracy of depression detection. In the aspect of cross-modal fusion of speech and text, considering the high heterogeneity of speech and text in statistical properties and expression methods, the system designs a fusion module based on the channel attention mechanism, which can adaptively learn and allocate the contribution weights of each modality to the task. This mechanism not only improves the ability of the model to utilize complementary information between modalities, but also strengthens the cross-modal semantic alignment effect, thereby generating high-quality, unified multi-modal fusion embedding representation, significantly improving the robustness and accuracy of depression recognition.

[0065] In the three-level fusion process of acoustic features, the acoustic feature fusion subunit 161, the first cross-attention sub-module, the second cross-attention sub-module, the first gating signal sub-module, the second gating signal sub-module, and the first fusion sub-module, the second fusion sub-module, the first feature splicing sub-module, the third cross-attention sub-module, the fourth cross-attention sub-module, the third gating signal sub-module, the fourth gating signal sub-module, and the third fusion sub-module, the fourth fusion sub-module, the second feature splicing sub-module are involved in the speech feature fusion process at different levels respectively. The multi-level feature fusion process avoids the strong wav2vec feature directly “flooding” other features, ensuring that unique information in the LLD and Mel features can be fully retained and utilized. The bidirectional complementary attention mechanism ensures that complementary information from two perspectives is fully captured in each level of the fusion process. The introduction of the gating mechanism in the fusion process can completely close the information flow judged as redundant or useless, allowing valuable information to pass through, greatly improving the purity and effectiveness of the fused features. The splicing operation ensures that all useful information extracted in the two-direction attention process is completely transmitted to the next stage, reducing any loss. In the cross-modal fusion process of acoustic features and text features, a unified joint attention space is constructed through stacking operation, enabling the model to dynamically and bidirectionally capture the complex correlations within and between audio and text modalities. Residual connection is used to ensure that the original feature information is not lost, and finally a powerful representation is generated that is deeply fused and retains the original information, greatly enhancing the discriminability of multi-modal representation.

[0066] Through hierarchical loss function design, the fusion process is optimized from two dimensions of distribution alignment and feature orthogonality: and KL divergence is used to force the distribution alignment of input features at the same level in the representation space, reducing the difficulty of fusion. Then through the orthogonal constraint, the cross-modal features are ensured to have complementarity (audio and text features are independent of each other) and information increment, finally realizing the deep fusion of redundancy minimization and information maximization. Feature-level aggregation strategy is adopted to integrate the deep fusion features of all dialogue segments of the target object into a unified sample-level representation, and then through the full connection network and binary cross-entropy loss for end-to-end optimization, the precise discrimination of individual overall depression state is realized, avoiding the instability of local decision based on segments, and significantly improving the comprehensive judgment ability and robustness of the model.

[0067] In this article, relational terms such as first and second are used only to distinguish one entity or action from another, and do not necessarily require or imply that these entities or actions exist in any such actual relationship or order.

[0068] The above merely provides an example of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A Chinese acoustic text depression detection system based on channel attention mechanism fusion, characterized by, The method comprises the following steps: a data acquisition unit for acquiring the original audio signal of the target object; a data preprocessing unit for preprocessing the original audio signal to obtain audio data; a data transcription unit for transcribing the audio data into text data corresponding to the audio data, wherein each sentence text in the text data contains its start and end time stamp in the audio data; an audio segmentation unit for segmenting the audio data based on the text data to obtain a plurality of audio segments corresponding to the sentence texts in the text data, each audio segment and the corresponding sentence text being a group of audio-text data; a feature extraction unit for multi-level feature extraction of the audio segments in each group of audio-text data, determining the low-level descriptor features, high-level time-frequency features and audio waveform features corresponding to the audio segments, and for feature extraction of the sentence texts in each group of audio-text data, determining the text features corresponding to the sentence texts; a feature fusion unit for step-by-step fusion of the low-level descriptor features, high-level time-frequency features and audio waveform features in each group of audio-text data to determine acoustic features, and for cross-modal fusion of the acoustic features and text features in each group of audio-text data to obtain fusion features; a depression detection unit for depression detection based on the fusion features to determine the depression detection result of the target object.

2. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 1, characterized in that, The feature extraction unit includes an acoustic feature extraction subunit, which is configured to: for each audio segment in each group of audio-text data: extract low-level descriptor features from the audio segment using the OpenSmile toolkit; divide the audio segment into a plurality of window data using a sliding window, calculate the mel-scale spectrum of each window data, and input the mel-scale spectrum of each window data into a convolutional neural network to obtain high-level time-frequency features; use a pre-trained wav2vec model to extract features from the audio segment to determine the audio waveform features of the audio segment, wherein the wav2vec model includes a multi-layer convolutional neural network.

3. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 1, characterized in that, The feature extraction unit further includes a text feature extraction subunit, which is configured to: for each sentence text in each group of audio-text data: use a pre-trained BERT-china-base model to extract text features of the sentence text.

4. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 1, characterized in that, The feature fusion unit includes an acoustic feature fusion subunit having a first fusion module and a second fusion module, which is configured to: for each group of audio-text data: The first fusion module is used for fusing the low-level descriptor features and the high-level time-frequency features to determine first-level fusion features ; The second fusion module analyzes the first-level fusion features. The acoustic features are determined by performing a two-stage fusion with the audio waveform features.

5. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 4, characterized in that, the first fusion module has a first cross-attention submodule, a second cross-attention submodule, a first gate signal submodule, a second gate signal submodule, and a first fusion submodule, a second fusion submodule, and a first feature concatenation submodule, and the acoustic feature fusion subunit is specifically configured to: inputting the high-level time-frequency feature and the low-level descriptor feature into a first cross-attention sub-module to generate interaction features ; The first cross-attention sub-module is inputted into and The first gate signal sub-module is inputted into ; Gating signal And interaction features Input the first fusion sub-module to generate component features ; low-level descriptor features and high-level time-frequency features input a second cross-attention sub-module to generate interaction features ; The second cross-attention sub-module is inputted into and The second gate signal sub-module is inputted into ; gate signal and interaction features inputting a second fusion sub-module to generate component features ; The component features and the component features input the first feature splicing sub-module, and splice to obtain a first-level fusion feature .

6. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 4, characterized in that, the second fusion module has a third cross-attention submodule, a fourth cross-attention submodule, a third gate signal submodule, a fourth gate signal submodule, and a third fusion submodule, a fourth fusion submodule, and a second feature concatenation submodule, and the acoustic feature fusion subunit is specifically configured to: audio waveform features and primary fusion features input a third cross-attention sub-module to generate interaction features ; The third cross-attention sub-module is inputted into and The third gate signal sub-module is inputted into ; gate signal and interaction features inputting a third fusion sub-module to generate component features ; The first-level fusion feature And the audio waveform feature Input the fourth cross-attention submodule to generate interaction features ; The fourth cross-attention sub-module is used for and The fourth gate signal sub-module is used for generating a gate signal ; gate signal and interaction features input fourth fusion sub-module, generate component features ; concatenate the component features and the component features input a second feature concatenation submodule, and concatenate to obtain acoustic features .

7. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 4, characterized in that, the loss function term of the first fusion module is: ， wherein, is a loss function term of the first fusion module, represents a difference between a distribution of and a distribution of represents a difference between a distribution of and a distribution of ​​ The loss function term of the second fusion module is: ， wherein, is a loss function term of the second fusion module, denotes the difference between the distribution of and the distribution of denotes the difference between the distribution of and the distribution of 8. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 1, characterized in that, The feature fusion unit includes a cross-modal fusion subunit, the cross-modal fusion subunit has a cross-modal attention module, and the cross-modal fusion subunit is configured to: For each set of audio-text data: acoustic features of audio-text data and text features input cross-modal attention module, cross-modal feature stacking: ， ， ， wherein, denotes a stacking operation, , , denote query, key and value vectors of the cross-modal attention module, respectively, and denote acoustic features and text features corresponding to the query vector, and denote acoustic features and text features corresponding to the key vector, and denote acoustic features and text features corresponding to the value vector. Calculate the fusion features of the audio-text data: ， wherein, is a fused feature, is the transpose of is the dimension of 9. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 8, characterized in that, The loss function term of the cross-modal fusion subunit is: ， wherein, is a loss function term for the cross-modal fusion subunit, is is the transpose of is is the transpose of denotes denotes the square of the Frobenius norm of denotes denotes the square of the Frobenius norm of denotes concatenation.

10. The Chinese acoustic text depression detection system based on channel attention mechanism fusion according to claim 1, characterized in that, The depression detection unit is specifically configured to: Aggregate the fusion features corresponding to each set of audio-text data to form sample-level features of the target object; Input the sample-level features into a fully connected network to output a depression detection result of the target object, wherein the loss function term of the fully connected network is: ， wherein, is a loss function term for the fully connected network, is a true label of the sample, is a model predicted value.

Images