Depression audio acquisition and analysis method based on mobile terminal

Abstract

The invention belongs to the technical field of computers, and particularly discloses a depression audio acquisition and analysis method based on a mobile terminal, which comprises the following steps: acquiring audio information of a tested person, the audio information comprising single-tone audio and / or interactive continuous audio; determining audio features according to the audio information, wherein the audio features comprise single-tone audio features and / or interactive continuous audio features; transmitting the single-tone audio features to a machine learning model in a cloud, and determining a depression degree classification probability; or transmitting the interactive continuous audio features to a large language model in the cloud, and determining a depression degree classification result; or jointly transmitting the depression degree classification probability and the interactive continuous audio features to a large language model in the cloud, and determining a depression degree classification result. According to the method, the depression degree of the tested person is determined by adopting the single-tone audio and / or the interactive continuous audio, so that the initial screening cost of the depression is greatly reduced while the accuracy is improved, and early warning of the depression is facilitated.

Description

Technical Field

[0001] The invention discloses a depression audio collection and analysis method based on a mobile terminal, and belongs to the technical field of computers. Background Art

[0002] Currently, the diagnosis of depression in psychiatric outpatient clinics is mainly completed through clinical symptoms and questionnaires, and is often supplemented by biomarker testing. Clinical symptom diagnosis is the most traditional method for diagnosing depression. Specifically, the doctor interviews the patient to understand his symptoms and medical history, and evaluates whether he meets the diagnostic criteria for depression based on international or national diagnostic standards. This method relies on the experience of psychiatrists and the subjective description of patients, so there is a certain degree of subjectivity. Questionnaire scale evaluation specifically refers to: during the diagnosis process, clinicians or mental health professionals use some standardized questionnaires or scales to assist in the diagnosis of depression. These questionnaires are usually filled out by patients themselves. Common scales include the Beck Depression Inventory (BDI), the Hamilton Depression Scale (HAMD), and the Patient Health Questionnaire-9 items (PHQ-9). Biomarker testing specifically refers to: during the diagnosis process, a certain biomarker of the patient, such as chemicals in the blood or cerebrospinal fluid, gene mutations, or activity patterns in specific brain regions, is used to assist in the diagnosis of depression. Blood and cerebrospinal fluid tests are used to detect the levels of biochemical substances related to depression, such as cortisol (stress hormone), brain-derived neurotrophic factor (BDNF), etc. The activity patterns of specific brain regions are often manifested by imaging techniques such as functional magnetic resonance imaging (fMRI) and positron emission computed tomography (PET), etc., to detect abnormal activity or structural changes in brain regions related to depression.

[0003] There are problems with clinical symptom diagnosis and questionnaire scale assessment: first, there is a lack of quantitative detection methods, the diagnostic process is complex and strongly dependent on the doctor's subjective judgment, the accuracy of the judgment of the course of depression still depends on the doctor's experience, and the diagnostic criteria are not clear enough, which makes it easier to miss and misdiagnose, which is not conducive to the accurate diagnosis and early detection of depression. Secondly, depression is closely related to patient privacy. Some patients will be passive in seeking medical treatment, actively conceal their true thoughts in scales, questionnaires, and questions and answers, and even have problems such as distrust of doctors and resistance to medical treatment, which will delay treatment and allow the disease to develop to the next stage, resulting in more serious consequences.

[0004] The problem with biomarker testing is that both sampling and imaging will cause harm to the patient's body and are expensive. Whether from the perspective of the patient's medical experience or considering the burden of the cost on the patient's family, it is not conducive to large-scale screening for depression.

[0005] The diagnosis of diseases through body surface information is a technology that has emerged in recent years. Because of its non-invasive and harmless characteristics, it is widely expected by the next generation of disease diagnosis technology. The body surface information technology currently available for depression diagnosis includes but is not limited to oral exhalation, pulse wave, tongue image, face image, voice, gait, pupil, electroencephalogram, etc. Among them, the diagnosis of depression based on voice analysis often relies on microphone collection. The collected audio signal is the audio signal of the subject reading the task text. The audio signal is then preprocessed and aligned for classification diagnosis. For voice hardware, most of them currently directly use professional microphones on the market for recording, or use telephone hotlines to collect depression voices. For voice software, there is currently no ready-made depression disease voice collection software on the market. Most of them directly use functions such as the recorder that comes with the computer to collect data, and the subsequent analysis is based on the study of the collected data.

[0006] For existing voice collection hardware, professional microphones lack anti-interference capabilities, and noise has a strong impact on audio analysis. If a telephone hotline is used, various audio parameters, such as sampling frequency and bit depth, will be limited by the transmission method. At the same time, there may be problems such as signal interference caused by transmission and the sampled person's lack of clarity about the task. A dedicated operator is required to communicate with him, which is time-consuming and laborious.

[0007] As for the existing voice collection software, there is a lack of dedicated collection software. Most of them directly use the system's own applications such as recorders to complete the recording, which makes it difficult to unify the various indicators of the collected audio, such as fixed audio length. At the same time, collection and analysis are often completely separated. Collection exists as a special task, while analysis is a subsequent means. It is impossible to directly feed back the analysis results to the sampled person, and there is no complete application. The existing applications for diagnosing depression through voice analysis can only be used as a possible verification. It is impossible to directly deploy application services and provide adjustment suggestions, which makes it difficult for users to conduct early warning and self-examination of depression at home. Summary of the invention

[0008] The purpose of the present invention is to provide a method for audio collection and analysis of depression based on a mobile terminal, so as to solve the problem that the audio collected for depression in the prior art is a task text read by the subject, which is difficult to truly reflect the depression condition of the subject. At the same time, the existing method separates collection and analysis, which makes it difficult for users to conduct self-examination and early warning of depression at home.

[0009] The present invention provides a method for collecting and analyzing audio of depression based on a mobile terminal, comprising: Acquiring audio information of the subject, wherein the audio information includes single-tone audio and / or interactive continuous audio; Determine an audio feature according to the audio information, wherein the audio feature includes a single-tone audio feature and / or an interactive continuous audio feature; Transmitting the single tone audio feature to a machine learning model in the cloud to determine the probability of depression level classification; or transmitting the interactive continuous audio feature to a large language model in the cloud to determine the depression level classification result; Alternatively, the depression level classification probability and the interactive continuous audio feature are transmitted together to a large language model in the cloud to determine the depression level classification result.

[0010] Preferably, the machine learning model is a deep random configuration network model based on manifold regularization.

[0011] Preferably, obtaining the audio information of the subject specifically includes: Receive audio tasks and corresponding guidance information sent by the cloud; The audio information sent by the subject with reference to the guide information and based on the audio task is obtained.

[0012] Preferably, the audio feature includes at least one of a basic audio feature, an audio spectrum feature, and a signal processing feature.

[0013] Preferably, the basic audio features include at least one of turbulent noise, glottal quotient, harmonic-to-noise ratio, noise-to-harmonic ratio, jitter, and flicker.

[0014] Preferably, the audio spectrum feature is Mel-frequency cepstral coefficients.

[0015] Preferably, the signal processing features include wavelet transform features and / or stability features; The stability characteristics include at least one of sliding window statistical characteristics, dynamic window characteristics, trend characteristics, standard deviations of derivatives of various orders, derivative change characteristics, local-global characteristics, drift average characteristics, and range evolution characteristics.

[0016] Preferably, determining the audio feature according to the audio information includes: Extracting common features from the audio information; A Q-Learning algorithm is used to extract core features from the common features as the audio features.

[0017] Preferably, the large language model is a Llama large language model.

[0018] Preferably, after determining the depression level classification probability or determining the depression level classification result, the method further includes: Obtain medical advice corresponding to the depression level classification probability or the depression level classification result from the cloud.

[0019] Compared with the prior art, the depression audio collection and analysis method based on a mobile terminal of the present invention has the following beneficial effects: The present invention uses single-tone audio and / or interactive continuous audio to determine the depression level of the subject, which greatly reduces the cost of preliminary screening for depression while improving the accuracy, and facilitates early warning of depression.

[0020] The present invention facilitates early warning of depression by being deployed on mobile terminals (mobile phones, tablet computers, etc.), so that users can easily have a preliminary understanding and numerical standards of their own mental state. At the same time, the present invention adopts an audio diagnosis method, and does not require patients to go to the hospital for various blood tests, which greatly reduces the cost of preliminary screening for depression. In addition, the present invention can help users interpret the analysis results, give users specific advice to help users make subsequent psychological adjustments, and effectively alleviate the shortage of psychiatrists in China and the high fees for psychological consultation, which makes it difficult for patients to obtain effective support in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The present invention is a flowchart of a method for collecting and analyzing audio signals for depression based on a mobile terminal according to an embodiment of the present invention.

[0022] Figure 2 Schematic diagram of single-tone audio guidance information in an embodiment of the present invention.

[0023] Figure 3 Flow chart of the Q-Learning algorithm in an embodiment of the present invention.

[0024] Figure 4 Schematic diagram of the deep random configuration network structure in an embodiment of the present invention.

[0025] Figure 5 The figure is a schematic diagram of the working process of the depression audio collection and analysis system based on a mobile terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0027] The embodiment of the present invention provides a Figure 1 The depression audio collection and analysis method based on a mobile terminal shown includes: Step 1: Obtain the audio information of the subject, which includes single-tone audio and / or interactive continuous audio.

[0028] The above step 1 specifically includes: Step 1.1: Receive the audio task and corresponding guidance information sent by the cloud.

[0029] The mobile terminal (mobile phone, tablet computer, etc.) of the embodiment of the present invention receives the audio task and corresponding guidance information sent by the cloud, wherein the audio task includes a single-tone audio task and / or an interactive continuous audio task, and the guidance information includes a single-tone audio guidance information and / or an interactive continuous audio guidance information.

[0030] Exemplarily, the above-mentioned monophonic audio guidance information is an animation composed of pinyin and Chinese characters, such as Figure 2 As shown, the sample pronunciation is played with headphones (if external playback will affect the recording quality when the audio information is subsequently collected, the user is reminded to use headphones or turn off the sample audio guidance).

[0031] The above-mentioned interactive continuous audio guidance information is an animation consisting of a question-and-answer dialogue, accompanied by sample pronunciation played through headphones (if playing the audio player externally will affect the recording quality when subsequently collecting audio information, the user will be reminded to use headphones, or turn off the sample audio guidance).

[0032] Step 1.2, obtaining the subject's reference guidance information and audio information based on the audio task.

[0033] Exemplarily, the method for collecting audio information in an embodiment of the present invention is to directly call the recording API of a mobile terminal (mobile phone, tablet computer, etc.), and use the noise reduction unit of the mobile terminal (mobile phone, tablet computer, etc.) to reduce environmental noise. The audio parameters collected as a whole in an embodiment of the present invention are selected from a sampling frequency of 16KHz-912KHz and a bit depth of 16bit to 96bit. Exemplarily, the above sampling frequency may be 16 KHz, 22.05KHz, 24 KHz, 32 KHz, 44.1KHz, 48KHz, 64 KHz, 96 KHz, 192KHz, etc., preferably 48KHz; the above bit depth may be 16bit, 24bit, 32bit, etc., preferably 32bit.

[0034] When the embodiment of the present invention collects single-tone audio, the duration of each single tone is 2 seconds to 3 seconds, preferably 3 seconds.

[0035] When the embodiment of the present invention collects interactive continuous audio, the continuous voice recording time is 50 seconds to 70 seconds, preferably 60 seconds. The interactive continuous audio collection content is a question-and-answer dialogue, which is played by a mobile terminal (mobile phone, tablet computer, etc.) or headphones, and the corresponding text is displayed on the screen simultaneously. After the user clearly receives the question-and-answer intention, he clicks the answer to start recording.

[0036] The present invention uses single-tone audio and / or interactive continuous audio to determine the depression level of the subject. Since the above two types of audio can more realistically reflect the voice conditions of the subject under different emotions, the accuracy of the determined depression condition can be greatly improved.

[0037] Step 2: Determine audio features based on the audio information, where the audio features include single-tone audio features and / or interactive continuous audio features.

[0038] In an embodiment of the present invention, determining audio features based on audio information specifically involves extracting audio features from the audio information, wherein features extracted from single-tone audio are single-tone audio features, and features extracted from interactive continuous audio are interactive continuous audio features.

[0039] Before extracting audio features, the embodiment of the present invention needs to perform software noise reduction (including filter noise reduction and deep learning model noise reduction methods) on the collected audio information, and perform corresponding audio alignment according to the tuning style of each mobile terminal manufacturer to ensure the consistency of parameters of audio collected by different devices as much as possible, and then perform feature extraction on the audio information that completes these steps.

[0040] The above-mentioned single-tone audio features and interactive continuous audio features both include at least one of audio basic features, audio spectrum features, and signal processing features.

[0041] The basic audio features include at least one of turbulent noise, glottal quotient, harmonic noise ratio, noise harmonic ratio, jitter, and flicker. The audio spectrum feature is the Mel frequency cepstrum coefficient. The signal processing features include wavelet transform features and / or stability features; the stability features include at least one of sliding window statistical features, dynamic window features, trend features, standard deviation of derivatives of various orders, derivative change features, local-global features, drift average features, and range evolution features.

[0042] The above turbulence noise is obtained using Detrended Fluctuation Analysis (DFA), which can characterize the degree of turbulence noise and quantify its random self-similarity. The specific calculation steps and formula of DFA are as follows: 1. Accumulation process: the original time series Accumulate as cumulative sequence : (1) in, is the mean of the time series, The original time series The data.

[0043] 2. Segmentation: Accumulate the sequence Divided into length non-overlapping segments.

[0044] 3. Detrending: Calculate the local trend for each segment and subtract it from the cumulative series.

[0045] 4. Volatility function: Calculate the root mean square volatility function of the residual after detrending in each segment : (2) in, is the number of segments, It is The local trend of the segment.

[0046] 5. Scaling index: for different segment lengths ,calculate and plotted on a double logarithmic scale right The scaling exponent is obtained by linear fitting. : (3) Finally, through right and Perform a linear fit and get the slope , which is the scaling exponent.

[0047] The above-mentioned Glottis Quotient (Glottis Quotient, referred to as ) characterizes the degree to which the duration of the glottis in the open state in each cycle deviates from its mean value, and its calculation steps are as follows: 1. Input parameters: VF_close and VF_open: The moments when the glottis closes and opens.

[0048] fs: sampling frequency.

[0049] f0min and f0max: minimum and maximum fundamental frequencies.

[0050] flag: A flag used to control whether to remove the error cycle.

[0051] 2. Cycle calculation: Calculate the difference between the glottal opening and closing periods.

[0052] (4) (5) 3. Remove the error cycle: If flag is true, remove periods that do not meet the criteria according to the fundamental frequency range.

[0053] 4. Statistical calculations: Calculates the statistical value of the period in the range of 5 to 95 percentiles.

[0054] (6) (7) 5. Calculation of glottal quotient: (1) Calculate the Glottis Quotient.

[0055] (8) (2) Calculate the standard deviation of the opening and closing cycles.

[0056] (9) (10) 6. Handling NaN values: if If it is NaN, set it to -1.

[0057] The above harmonic-to-noise ratio (HNR) provides some non-periodic information of the speech signal, showing hoarseness and wheezing. The HNR of a diseased voice is lower, while the HNR of a healthy voice is higher.

[0058] The above-mentioned noise harmonic ratio NHR can perceive dysphonia and structural lesions of the vocal cords. The disease NHR is higher and basically consistent with the HNR.

[0059] The steps to calculate HNR and NHR are as follows: 1. Input parameters: data: input audio signal.

[0060] fs: sampling frequency.

[0061] 2. Initialization parameters: f0max and f0min: maximum and minimum values ​​of the fundamental frequency.

[0062] tstep: The time interval between each step.

[0063] : The length of each window.

[0064] 3. Segment processing: The input signal is divided into multiple windows, each with a length of , the step size is tstep.

[0065] 4. Calculation of HNR and NHR for each window: (1) Remove the mean and add a Hanning window.

[0066] (11) (12) In the formula, For the step (2) Calculate the autocorrelation function.

[0067] (13) (3) Calculate the autocorrelation function of the Hanning window.

[0068] (14) (4) Normalized autocorrelation function.

[0069] (15) (5) Select the maximum value within the fundamental frequency range.

[0070] (16) (17) (18) In the formula, It represents the lag index, which can also be understood as the length of the candidate period, in sample points. Its actual physical meaning is the candidate fundamental frequency period, that is, the time interval of signal repetition, in sample points. By searching for the maximum autocorrelation value, the delay corresponding to the fundamental frequency can be determined. and The delay range is limited to ensure that the search is performed within a reasonable base frequency range.

[0071] Calculate HNR and NHR.

[0072] (19) (20) The jitter in the embodiment of the present invention is characterized by a timing error (Jitter), which is a physical quantity that describes the change in the basic frequency of the sound wave between adjacent cycles. It is a measure of the irregularity of the vocal cord vibration frequency, the so-called sound stability, including absolute jitter (Jitter, Absolute), relative jitter (Jitter, Relative) and jitter percentage (Jitter, Percent).

[0073] The formula for absolute jitter is: (twenty one) The steps to obtain the absolute jitter are: (1) Extract The period length of the acoustic cycle , is the total number of acoustic cycles.

[0074] (2) Calculate the absolute value of the difference between adjacent periods.

[0075] (3) Calculate the average of these differences.

[0076] The formula for the above relative jitter is: (twenty two) The steps to obtain relative jitter are: (1) Extract The period length of the acoustic cycle .

[0077] (2) Calculate the absolute value of the difference between adjacent periods and divide it by the length of the current period.

[0078] (3) Calculate the average of these relative differences.

[0079] The formula for the above jitter percentage is: (twenty three) in, is the average value of all cycles.

[0080] The steps to obtain the jitter percentage are: (1) Extract The period length of the acoustic cycle .

[0081] (2) Calculate the average value of all cycles .

[0082] (3) Calculate the absolute value of the difference between adjacent cycles and divide it by the average cycle length .

[0083] (4) Calculate the average of these relative differences and multiply it by 100% to get the percentage.

[0084] The flicker in the embodiment of the present invention is characterized by amplitude perturbation (Amplitude Perturbation), also known as shimmer, which describes the change in the amplitude of the sound wave between adjacent cycles and mainly reflects the degree of hoarseness. Jitter and shimmer together reflect the stability of the vocal cord vibration. The smaller the value, the fewer small changes in the acoustic signal during the phonation process. The flicker in the embodiment of the present invention includes absolute flicker Shimmer, Absolute, relative flicker (Shimmer, dB) and flicker percentage (Shimmer, Percent).

[0085] The formula for absolute flicker is: (twenty four) The steps to get the absolute flicker are: (1) Extract The amplitude of the acoustic cycle .

[0086] (2) Calculate the absolute value of the amplitude difference between adjacent periods.

[0087] (3) Calculate the average of these differences.

[0088] The formula for the relative flicker above is: (25) The steps to get the relative flicker are: (1) Extract The amplitude of the acoustic cycle .

[0089] (2) Calculate the amplitude ratio between adjacent periods and take its logarithm.

[0090] (3) Calculate the average of the logarithms of these ratios and multiply by 20 to convert .

[0091] The formula for the above flicker percentage is: (26) in, is the average amplitude of all cycles.

[0092] The steps to obtain the flicker percentage are: (1) Extract The amplitude of the acoustic cycle .

[0093] (2) Calculate the average amplitude of all cycles .

[0094] (3) Calculate the absolute value of the amplitude difference between adjacent periods and divide it by the average amplitude .

[0095] (4) Calculate the average of these relative differences and multiply it by 100% to get the percentage.

[0096] The audio spectrum feature of the embodiment of the present invention is the Mel-frequency cepstral coefficient. Mel-frequency cepstral coefficients (MFCCs) are cepstral parameters extracted in the Mel-scale frequency domain. The Mel-scale describes the nonlinear characteristics of the human ear frequency and can roughly describe the energy level of sounds of different frequencies. In addition, MFCCs can also roughly reflect the resonance peak of the sound.

[0097] The steps to determine the Mel frequency cepstral coefficients are as follows: 1. Pre-emphasis The purpose of pre-emphasis is to enhance the high-frequency components and make the signal spectrum flatter. The corresponding formula is: (27) in, is the input signal, is the pre-emphasized signal, is the pre-emphasis factor, usually set to 0.97.

[0098] 2. Framing Split the signal into overlapping frames, each of which is usually 20-40 milliseconds. Specifically: select the frame length ( ) and frame shift ( ), and then divide the signal into frames, each frame contains samples, overlapping between adjacent frames samples.

[0099] 3. Windowing In order to reduce spectrum leakage, a window function is usually applied to each frame, such as a Hamming Window, specifically: (28) in, is the window function, is the length of the window.

[0100] 4. Fast Fourier Transform (FFT) Each frame is converted to the frequency domain by performing a Fast Fourier Transform (FFT), specifically: applying FFT to each frame, calculating its spectrum, then taking the real and imaginary parts of the spectrum, and calculating the power spectrum.

[0101] 5. Mel Filterbank The spectrum is converted to the Mel frequency scale and filtered through a set of triangular filters. The corresponding formula is: (29) in, is the frequency, is the mel frequency.

[0102] 6. Logarithmic Compression The energy after filtering is logarithmically transformed, and the specific formula is: (30) in, It is The output energy of the filter.

[0103] 7. Discrete Cosine Transform (DCT) Perform discrete cosine transform (DCT) on the logarithmic Mel spectrum to obtain MFCC, and the corresponding formula is: (31) in, is the number of filters, is the coefficient index of MFCC.

[0104] The embodiment of the present invention extracts the mean value of the frequency domain under the Mel scale and the standard deviation of the frequency domain under the Mel scale.

[0105] The steps of the wavelet transform (WT) method used in the above wavelet transform feature are: 1. Parameter initialization Check whether the wavelet basis wname and decomposition level dec_levels are provided. If not provided, the default values ​​are used, including the default wavelet basis: 'db8' (Daubechies 8 wavelet); the default decomposition level: 10.

[0106] 2. Wavelet decomposition Perform discrete wavelet transform (DWT) on the input signal data and decompose the signal into approximate coefficients and detail coefficients.

[0107] The discrete wavelet transform (DWT) process is as follows: (1) Initial signal: Assume the input signal is , the length is .

[0108] (2) Analysis filter: Low-pass filter: ; High pass filter: .

[0109] (3) First-level discrete wavelet transform: The approximate coefficients (low-frequency components) are: (32) In the formula, For input signal The data.

[0110] The detail coefficient (high frequency component) is: (33) (4) Multi-level decomposition: Recursively calculate the approximate coefficients of each level Perform the above transformation to obtain higher-level approximate coefficients and detail factor .

[0111] (34) (35) 3. Feature extraction (1) Calculation of energy The energy formula used to calculate the energy of the approximation coefficient and detail coefficient of each layer is: (36) in, It is coefficient value.

[0112] (2) Extracting detail coefficient features Feature extraction is performed on the detail coefficients of each layer.

[0113] The Shannon entropy is: (37) in, It is the signal Partial probability distribution.

[0114] The logarithmic energy entropy is: (38) The Teager-Kaiser energy operator ( )for: (39) Pick The mean and standard deviation of .

[0115] (3) Extracting approximate coefficient features The feature extraction is performed on the approximate coefficients of each layer. The extraction method is the same as that of the detail coefficients, including Shannon entropy, logarithmic energy entropy, TKEO The mean and standard deviation of .

[0116] 4. Wavelet decomposition after logarithmic transformation The signal is logarithmized and the wavelet decomposition and feature extraction are repeated.

[0117] The logarithmic transformation is: (40) 5. Feature vector construction All extracted features are stored in a structure and converted into vector form for subsequent classification or regression tasks.

[0118] The stability feature in the embodiment of the present invention is an analysis of the stability of the sound in the time dimension, and the feature dimension is also much higher than other feature extraction details.

[0119] The formula for the sliding window statistical feature in the above stability feature is: (41) in: is the window length; is the window step size (usually a fraction of the window length), The window number.

[0120] The calculation steps of the dynamic window feature in the above stability feature are: Divide the time series into segments, each segment is : (42) Calculate features (such as mean, standard deviation, etc.) for each segment: (43) (44) Compute the standard deviation of these features: (45) The trend characteristics in the above stability characteristics are specifically: 1. Standard Deviation Ratio ( ), the corresponding formula is: (46) in, For time series The standard deviation of For time series The cumulative sum of is the standard deviation of the cumulative sum.

[0121] Explanation: This ratio measures the relationship between the magnitude of the original signal change and its cumulative and varying magnitude.

[0122] 2. Linear Fit (LinearFit): Fitting Linear Models .

[0123] Fitting time series using least squares method Time relationship.

[0124] Calculate the parameters of the linear fit and .

[0125] Explanation: Linear fitting is used to capture the direction and strength of linear trends in time series. Parameters represents the slope, parameter represents the intercept.

[0126] 3. Cumulative and statistics (CumulativeStats), specifically: Calculate the cumulative sum: (47) Statistical cumulative sum mean and standard deviation: (48) (49) Explanation: By analyzing the mean and standard deviation of the cumulative sum, we can assess the stability and changes in the overall trend of the time series.

[0127] The calculation steps of the standard deviation of each order derivative in the stability characteristic in the embodiment of the present invention are: 1. Calculate the Derivative of order.

[0128] For a given time series , No. The order derivatives are approximated using differences: (50) in, Indicates Specifically: First-order difference (i.e. first-order derivative): (51) Second-order difference (i.e. second-order derivative): (52) The order difference is derived in a similar way.

[0129] 2. Calculate the The standard deviation of the derivative.

[0130] For The standard deviation is calculated by taking derivatives: (53) in: is the length of the time series; It is n The average value of the derivative is as follows: (54) The derivative change characteristics in the stability characteristics in the embodiment of the present invention are specifically: 1. Mean absolute deviation of the derivative.

[0131] Compute the mean absolute deviation of the first derivative: (55) 2. Skewness of the derivative.

[0132] Compute the skewness of the first derivative, which describes the asymmetry of the distribution: (56) 3. Kurtosis of the derivative Compute the kurtosis of the first derivative, which describes the peakedness of the distribution: (57) Where StdFirstDer is the standard deviation of the first-order derivative.

[0133] The local-global features in the stability features in the embodiment of the present invention are specifically: 1. Local-global mean difference.

[0134] This feature measures the difference in the mean of the time series at the local and global scales.

[0135] (58) in: is the length of the time series; is the window size; It is The mean of windows: (59) is the mean of the entire time series: (60) 2. Local-global standard deviation difference.

[0136] This feature measures the difference in the standard deviation of the time series at the local and global scales.

[0137] (61) in: is the length of the time series; is the window size.

[0138] It is The standard deviation of the window: (62) is the standard deviation of the entire time series: (63) 3. Local-global autocorrelation differences.

[0139] This feature measures the difference in autocorrelation of the time series at local and global scales.

[0140] (64) in: is the length of the time series; is the window size; is the lag order.

[0141] It is Window hysteresis The autocorrelation coefficient of is: (65) is the lag of the entire time series The autocorrelation coefficient of is: (66) 4. Local-global skewness difference.

[0142] This feature measures the difference in skewness of the time series at the local and global scales.

[0143] (67) in: It is The skewness of the window: (68) is the skewness of the entire time series: (69) The drift average feature in the stability feature in the embodiment of the present invention is specifically: 1. Simple Moving Average (SMA) For a given window size , the simple moving average is defined as: (70) in: is the value of the time series; is the time index, starting from arrive .

[0144] 2. Exponential Moving Average (EMA) EMA assigns an exponentially decaying weight to each data point. For a given smoothing factor : (71) in: (72) Initial Value .

[0145] 3. Weighted Moving Average (WMA) The weighted moving average applies different weights to each data point within the window.

[0146] (73) in: It is The weight of the data point.

[0147] 4. Cumulative Moving Average (CMA) The cumulative moving average is the average of all data points up to a given point in time.

[0148] (74) 5. Double Exponential Moving Average (Double EMA), which is used to reduce the lag of the simple exponential moving average.

[0149] (75) 6. Triple EMA is used to further reduce lag.

[0150] (76) The range evolution characteristics in the stability characteristics in the embodiment of the present invention are specifically: 1. Local range. This feature calculates the range (difference between maximum and minimum values) of the time series within each local window.

[0151] (77) in: Indicates that from Data points to A subsequence of data points. is the window size.

[0152] 2. The mean of the local range, calculate the mean of all local window ranges.

[0153] (78) in: is the length of the time series. is the window size.

[0154] 3. Standard deviation of local range, calculate the standard deviation of all local window ranges.

[0155] (79) 4. Skewness of local range, calculate the skewness of all local window ranges to describe the asymmetry of range distribution.

[0156] (80) 5. Local range kurtosis, calculate the kurtosis of all local window ranges and describe the peak degree of range distribution.

[0157] (81) 6. Maximum and minimum values ​​of local ranges, calculate the maximum and minimum values ​​of all local window ranges.

[0158] (82) (83) 7. The rate of change of the local range, calculate the rate of change of the adjacent window range.

[0159] (84) The embodiment of the present invention achieves the goal of ensuring the accuracy of collection and analysis while reducing the amount of calculation, and further improves the step of determining audio features based on audio information, specifically including: A. Extracting common features from audio information; the common features include at least one of basic audio features, audio spectrum features, and signal processing features.

[0160] B. Use the Q-Learning algorithm to extract core features from common features as audio features.

[0161] The Q-Learning algorithm is a method that applies reinforcement learning to feature selection problems, especially for situations where the data dimension is high. In this algorithm, the feature selection process is regarded as a reinforcement learning problem, in which the agent needs to learn how to select important features to improve the performance of the model or simplify the complexity of the model.

[0162] The algorithm flow is as follows Figure 3 As shown, the formula for updating the Q value is as follows: (85) in, is the current feature combination, is the learning rate, which is controlled by parameter settings The update speed of the value, is the currently selected feature, is the reward after selecting the feature, is the discount factor, The meaning of is the feature selected in the next step. The meaning is the feature combination for the next step.

[0163] The corresponding steps of the algorithm are shown in Table 1.

[0164] Table 1 Q-Learning algorithm steps

[0165] Step 3: Transmit the single-tone audio features to the machine learning model in the cloud to determine the probability of depression level classification; or transmit the interactive continuous audio features to the large language model in the cloud to determine the depression level classification result; or transmit the depression level classification probability and the interactive continuous audio features to the large language model in the cloud to determine the depression level classification result.

[0166] In the embodiment of the present invention, the extracted and encrypted audio features are uploaded to the cloud for processing through the network API interface of the mobile terminal (mobile phone, tablet computer, etc.). Uploading the encrypted feature information can protect the user's privacy as much as possible. If the user wants to use the analysis of interactive continuous audio simultaneously and is willing to upload his own audio sample, he can upload the interactive continuous audio and the single-tone audio to the cloud together if he agrees to the terms.

[0167] During the cloud analysis process, the embodiment of the present invention integrates the single-tone audio features, inputs them into the trained machine learning model for disease identification, and then outputs the result data and feeds it back to the user. For sliding continuous speech, the present invention extracts semantic information and pause information through natural language processing methods, inputs it into the fine-tuned Llama large language model, uses the large language model to analyze the user's continuous speech, and returns the possible degree of illness and corresponding suggestions. At the same time, the large language model of the present invention can also interact with users in the cloud, chat and provide medical advice.

[0168] The machine learning model in the embodiment of the present invention is a deep random configuration network model based on manifold regularization.

[0169] Deep Stochastic Configuration Network (DeepSCN). Deep models can learn higher-level features. DeepSCN was proposed in 2018. The network structure is as follows: Figure 4 As shown in Figure 1, its configuration process is similar to that of SCN. The network structure starts with one hidden layer and one hidden layer node, and gradually increases the number of hidden layer nodes and the number of hidden layer layers. The hidden layer node parameters are configured through the supervision mechanism, and all hidden layers are directly connected to the output layer. The output weight is calculated using the least squares method.

[0170] The embodiment of the present invention introduces manifold regularization into deep SCN to improve the generalization ability of the model. After introducing manifold regularization, the objective function of deep SCN is as follows: (86) in, Output for the target; represents the hidden layer output matrix.

[0171] Let the derivative of the above formula be zero, as follows: (87) Then the hidden layer output matrix is ​​calculated by formula (87). for: (88) In the embodiment of the present invention, the large language model is the Llama large language model.

[0172] After determining the depression level classification probability or determining the depression level classification result, the embodiment of the present invention further includes: obtaining medical advice corresponding to the depression level classification probability or the depression level classification result from the cloud.

[0173] The embodiment of the present invention uses the sampled depression consultation corpus and the doctor-patient question-and-answer corpus collected from the Internet for depression to fine-tune the Llama large language model, so that it has the ability to communicate with users and provide medical advice. At the same time, the embodiment of the present invention also uses the prepared corpus based on the results of the previous analysis for fine-tuning, so that it has a certain ability to understand the results of the analysis system, so that it can answer questions for users based on the analysis results.

[0174] The present invention also discloses a depression audio collection and analysis system based on a mobile terminal, the system includes a mobile terminal and a cloud, wherein the mobile terminal uses the above depression audio collection and analysis method based on a mobile terminal, the overall system framework complies with the Android APP development model, completes simple local and cloud setup, completes the user database setup and registration and login logic. The working process of the system is as follows Figure 5 shown.

[0175] The goal of the embodiment of the present invention is to build a depression audio collection and analysis system based on mobile terminals (mobile phones, tablet computers, etc.) that integrates collection, analysis and suggestions, so as to facilitate users to achieve early warning and preliminary diagnosis of depression at home, and at the same time provide users with certain adjustments and treatment suggestions based on the system analysis results.

[0176] The present invention will provide a unified, standardized and automated sampling process during the sampling process, supporting users to complete audio collection by themselves following the system guidance after installation on mobile terminals (mobile phones, tablet computers, etc.). The sampling process is divided into single-voice audio sampling and interactive continuous audio sampling. The system will specify the required sampling parameters and collect multiple single-voice audios and a segment of interactive continuous speech. Specific tasks can be updated locally through the cloud.

[0177] The present invention provides an analysis process integrated with sampling during the analysis process. After the user completes the sampling, the data is pre-processed locally, and the encrypted features, rather than the audio information itself, are distributedly uploaded to the cloud for subsequent analysis, and the specific results of the analysis are fed back to the user. Since the encrypted transmission is the audio features, it does not involve private data, which can protect user privacy as much as possible and reduce the risk of information leakage.

[0178] After completing the analysis, the present invention can provide the user with specific life suggestions and treatment suggestions that meet the user's current situation based on the analysis results and the data used in the analysis process, so that the user can have a clear understanding of his or her specific situation and adjustment methods.

[0179] Table 2 provides the performance of the machine learning classical classifier under the highest performance pronunciation, proving that the monophonic audio and interactive continuous audio of the present invention can effectively diagnose depression patients.

[0180] Table 2 Performance of classic machine learning classifiers

[0181] In Table 2, Random Forest is a random forest model; SVM is a support vector machine model; LogisticRegression is a logistic regression analysis model; Decision Tree is a decision tree model; AdaBoost is an iterative algorithm model, the core idea of ​​which is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier); XGBoost is an optimized distributed gradient boosting library model, which implements machine learning algorithms under the Gradient Boosting framework; GBM is a gradient boosting machine, which is an iterative decision tree integration method; LightGBM is a machine learning model based on gradient boosting decision trees.

[0182] The present invention facilitates early warning of depression by being deployed on mobile terminals (mobile phones, tablet computers, etc.), so that users can easily have a preliminary understanding and numerical standards of their own mental state. At the same time, the present invention adopts an audio diagnosis method, and does not require patients to go to the hospital for various blood tests, which greatly reduces the cost of preliminary screening for depression. In addition, the present invention can help users interpret the analysis results, give users specific advice to help users make subsequent psychological adjustments, and effectively alleviate the shortage of psychiatrists in China and the high fees for psychological consultation, which makes it difficult for patients to obtain effective support in a timely manner.

[0183] The above are only several embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, using the above disclosed technical content to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A depression audio collection and analysis method based on a mobile terminal, characterized in that: include: Acquiring audio information of the subject, wherein the audio information includes single-tone audio and / or interactive continuous audio; Determine an audio feature according to the audio information, wherein the audio feature includes a single-tone audio feature and / or an interactive continuous audio feature; Transmitting the single tone audio feature to a machine learning model in the cloud to determine the probability of depression level classification; or transmitting the interactive continuous audio feature to a large language model in the cloud to determine the depression level classification result; Alternatively, the depression level classification probability and the interactive continuous audio feature are transmitted together to a large language model in the cloud to determine the depression level classification result.

2. The method for collecting and analyzing depression audio based on a mobile terminal according to claim 1 is characterized in that: The machine learning model is a deep random configuration network model based on manifold regularization.

3. The method for collecting and analyzing depression audio based on a mobile terminal according to claim 1 is characterized in that: Obtain the audio information of the subject, including: Receive audio tasks and corresponding guidance information sent by the cloud; The audio information sent by the subject with reference to the guide information and based on the audio task is obtained.

4. The method for collecting and analyzing depression audio based on a mobile terminal according to claim 1 is characterized in that: The audio feature includes at least one of a basic audio feature, an audio spectrum feature, and a signal processing feature.

5. The method for collecting and analyzing audio signals of depression based on a mobile terminal according to claim 4 is characterized in that: The basic audio features include at least one of turbulent noise, glottal quotient, harmonic-to-noise ratio, noise-to-harmonic ratio, jitter, and flicker.

6. The method for collecting and analyzing audio signals of depression based on a mobile terminal according to claim 4 is characterized in that: The audio spectrum feature is Mel-frequency cepstral coefficients.

7. The method for collecting and analyzing audio signals of depression based on a mobile terminal according to claim 4, characterized in that: The signal processing features include wavelet transform features and / or stability features; The stability characteristics include at least one of sliding window statistical characteristics, dynamic window characteristics, trend characteristics, standard deviations of derivatives of various orders, derivative change characteristics, local-global characteristics, drift average characteristics, and range evolution characteristics.

8. The method for collecting and analyzing depression audio based on a mobile terminal according to claim 1, characterized in that: Determining audio features according to the audio information includes: Extracting common features from the audio information; A Q-Learning algorithm is used to extract core features from the common features as the audio features.

9. The method for collecting and analyzing audio signals of depression based on a mobile terminal according to claim 1, characterized in that: The large language model is the Llama large language model.

10. The method for collecting and analyzing audio signals of depression based on a mobile terminal according to claim 1, characterized in that: After determining the probability of depression degree classification or determining the depression degree classification result, it also includes: Obtain medical advice corresponding to the depression level classification probability or the depression level classification result from the cloud.

Images