A method, device and humanoid robot for recognizing depression

Abstract

The application discloses a kind of depression identification method, device and humanoid robot, method includes the multi-modal time series data of depression patient that acquisition equipment is gathered;According to the multi-modal time series data, depression identification preprocessing operation is carried out, and first characteristic vector matrix is obtained;The spatial proximity matrix and pixel value similarity matrix are extracted to the first characteristic vector matrix;The spatial proximity matrix and the pixel value similarity matrix are input into pre-configured bilateral filter and are carried out noise reduction and smoothing, and second characteristic vector matrix is obtained;The priority of recognition is set to the characteristic vector in the second characteristic vector matrix, and third characteristic vector matrix is obtained;The third characteristic vector matrix is input into pre-configured LSTM identification model, and recognition result is obtained;The application carries out depression identification by using bilateral filter and priority setting, and improves the automatic identification accuracy of depression.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of depression recognition, and in particular to a depression recognition method, device and humanoid robot. BACKGROUND

[0002] Depression is a common mental disorder that affects millions of people worldwide. It not only has a significant impact on the quality of life of patients, but also brings serious social and economic burdens. Symptoms of depression include persistent sadness, loss of interest in daily activities, weight changes, sleep disturbances, fatigue, feelings of worthlessness or excessive guilt, difficulty concentrating, and recurring thoughts of suicide. Due to the complexity and diversity of symptoms of depression, and the fact that different individuals may exhibit different symptoms, early recognition and diagnosis of depression is a significant challenge.

[0003] Currently, the recognition of depression mainly relies on the clinical diagnosis of professional doctors, including face-to-face interviews, psychological tests, and observations of patients' behavior and emotions. In addition, with the development of technology, some auxiliary tools and methods have been introduced into the process of recognizing depression. For example, by analyzing patients' language patterns, facial expressions, physiological signals and other non-verbal information, machine learning algorithms are used to assist in diagnosis. These technologies have improved the accuracy and efficiency of recognition to some extent.

[0004] Although existing recognition technologies have made some progress, there are still some obvious defects and limitations. The current recognition methods are highly dependent on the experience and judgment of professional doctors, which not only increases the burden of medical resources, but also makes it difficult to achieve widespread application in the case of uneven resource allocation. In addition, many methods that use machine learning algorithms to recognize and diagnose depression only use partial data, and the generalization ability and accuracy of the algorithm still need to be improved. SUMMARY

[0005] The present application provides a depression recognition method, device and humanoid robot to recognize depression through multiple data, thereby improving the accuracy of automatic recognition of depression.

[0006] In a first aspect, to solve the above technical problems, the present application provides a depression recognition method, comprising:

[0007] acquiring multi-modal time series data of a depression patient collected by a collection device; wherein the multi-modal time series data includes audio data, video data, heart rate data, blood pressure data and body temperature data;

[0008] performing a depression recognition preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix;

[0009] The spatial proximity matrix and pixel value similarity matrix are extracted from the first feature vector matrix;

[0010] The spatial proximity matrix and the pixel value similarity matrix are input into a pre-configured bilateral filter for noise reduction and smoothing to obtain a second feature vector matrix; wherein, the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function;

[0011] The recognition priority is set for the feature vectors in the second feature vector matrix to obtain the third feature vector matrix;

[0012] The third feature vector matrix is ​​input into a pre-configured LSTM recognition model to obtain the recognition result.

[0013] Preferably, the formula for calculating the priority of the eigenvectors in the second eigenvector matrix is ​​as follows:

[0014]

[0015] in, It is the eigenvector in the second eigenvector matrix. Priority; It is the eigenvector in the second eigenvector matrix. The weights are used to reflect the eigenvectors in the second eigenvector matrix. Its fundamental importance in the identification of depression; It is the eigenvector in the second eigenvector matrix. The information gain is used to represent the eigenvectors in the second eigenvector matrix. Contribution to the classification; It is a balancing coefficient used to balance the effects of information gain and feature covariance; It is the eigenvector in the second eigenvector matrix. and the eigenvectors in the second eigenvector matrix The covariance is used to reflect the interaction between two features.

[0016] Preferably, the configuration process of the LSTM recognition model includes:

[0017] Acquire historical multimodal time series data;

[0018] Based on the historical multimodal time series data, a model training preprocessing operation is performed to obtain the first historical feature vector matrix;

[0019] Based on the first historical feature vector matrix, data annotation operations are performed to obtain an annotation vector matrix;

[0020] inputting the first historical feature vector matrix into a pre-configured bilateral filter for noise reduction and smoothing to obtain a second historical feature vector matrix;

[0021] inputting the second historical feature vector matrix and the labeled vector matrix into the LSTM recognition model for training, judging that the training is completed after a loss function of the LSTM recognition model meets a convergence condition, and obtaining a trained LSTM recognition model.

[0022] Preferably, the depression recognition preprocessing operation is performed according to the multi-modal time series data to obtain a first feature vector matrix, specifically:

[0023] According to the multi-modal time series data, a feature extraction operation is performed to obtain multi-modal time series features;

[0024] According to the multi-modal time series features, a feature fusion operation is performed to obtain a first feature vector matrix.

[0025] Preferably, the bilateral filter is specifically configured as:

[0026]

[0027]

[0028] wherein, and are positions of sample i and sample j of the first feature vector matrix, respectively; and are feature vectors of sample i and sample j of the first feature vector matrix, respectively; is a spatial Gaussian function for processing spatial proximity of sample i and sample j; is an intensity Gaussian function for processing pixel value similarity of sample i and sample j; is a normalization factor to ensure that the filtered result is a weighted average value;

[0029] wherein, the spatial Gaussian function is specifically:

[0030]

[0031] wherein, is a spatial distance between sample i and sample j of the first feature vector matrix; is a spatial standard deviation;

[0032] wherein, the intensity Gaussian function is specifically:

[0033]

[0034] wherein, is the feature vector distance between sample i and sample j of the first feature vector matrix, is the intensity standard deviation.

[0035] Preferably, the label vectors in the label vector matrix include depression symptoms, depression severity and emotional state.

[0036] wherein, the depression symptom is whether there is depression; the depression severity is the severity of depression, including mild, moderate and severe; and the emotional state includes happiness, sadness, fear, anger, surprise and disgust.

[0037] Preferably, after inputting the third feature vector matrix into the pre-configured LSTM recognition model to obtain a recognition result, the method further comprises:

[0038] inputting the recognition result into a pre-configured depression database to obtain diagnosis suggestion information, and outputting the diagnosis suggestion information to a user;

[0039] wherein, the configuration process of the depression database comprises:

[0040] obtaining a depression knowledge dataset;

[0041] performing knowledge graph construction preprocessing operation according to the depression knowledge dataset to obtain a preprocessed depression knowledge dataset;

[0042] inputting the preprocessed depression knowledge dataset into a database system to perform knowledge graph construction operation to obtain a depression database.

[0043] Preferably, the performing knowledge graph construction preprocessing operation according to the depression knowledge dataset to obtain a preprocessed depression knowledge dataset comprises: performing text cleaning operation according to the depression knowledge dataset to obtain a cleaned depression knowledge dataset.

[0044] performing entity recognition and relationship extraction operation according to the cleaned depression knowledge dataset to obtain a preprocessed depression knowledge dataset.

[0045] In a second aspect, the present application provides a depression recognition device, comprising:

[0046] an acquisition module configured to acquire multi-modal time series data of a depression patient collected by an acquisition device; wherein the multi-modal time series data comprises audio data, video data, heart rate data, blood pressure data and body temperature data;

[0047] The preprocessing module is used to perform depression identification preprocessing operations based on the multimodal time series data to obtain a first feature vector matrix;

[0048] Extraction module: used to extract the spatial proximity matrix and pixel value similarity matrix from the first feature vector matrix;

[0049] A filtering module is used to input the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter for noise reduction and smoothing to obtain a second feature vector matrix; wherein, the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function;

[0050] The priority setting module is used to set the recognition priority of the feature vectors in the second feature vector matrix to obtain the third feature vector matrix;

[0051] The recognition module inputs the third feature vector matrix into a pre-configured LSTM recognition model to obtain the recognition result.

[0052] The configuration process of the LSTM recognition model includes:

[0053] Acquire historical multimodal time series data;

[0054] Based on the historical multimodal time series data, a model training preprocessing operation is performed to obtain the first historical feature vector matrix;

[0055] Based on the first historical feature vector matrix, data annotation operations are performed to obtain an annotation vector matrix;

[0056] The first historical feature vector matrix is ​​input into a pre-configured bilateral filter for noise reduction and smoothing to obtain the second historical feature vector matrix;

[0057] The second historical feature vector matrix and the labeled vector matrix are input into the LSTM recognition model for training. The training is considered complete after the loss function of the LSTM recognition model satisfies the convergence condition, and the trained LSTM recognition model is obtained.

[0058] Thirdly, the present invention also provides a humanoid robot, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the depression recognition method described in any one of the above.

[0059] In a fourth aspect, the present application further provides a computer readable storage medium comprising a stored computer program, wherein the computer readable storage medium controls a device in which the computer readable storage medium is located to perform the depression identification method according to any one of the preceding aspects when the computer program is executed.

[0060] Compared with the prior art, the present application has the following beneficial effects: the embodiments of the present application provide a depression identification method, device and humanoid robot, comprising: acquiring multi-modal time series data of a depression patient collected by a collection device; wherein the multi-modal time series data comprises audio data, video data, heart rate data, blood pressure data and body temperature data; performing a depression identification preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix; extracting a spatial proximity matrix and a pixel value similarity matrix from the first feature vector matrix; inputting the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second feature vector matrix; wherein the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function; inputting the third feature vector matrix into a pre-configured LSTM identification model to obtain an identification result. The method uses a bilateral filter and priority setting to identify depression, thereby improving the automatic identification accuracy of depression BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of a depression identification method provided by an embodiment of the present application;

[0062] Figure 2 is a flowchart of a configuration process of an identification model provided by an embodiment of the present application;

[0063] Figure 3 is a flowchart of a configuration process of a depression database provided by an embodiment of the present application;

[0064] Figure 4 is a structural diagram of a depression identification device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0066] With reference to Figure 1 , the embodiments of the present application provide a depression identification method, comprising the following steps:

[0067] S01, obtaining multi-modal time series data of a depression patient collected by a collection device; wherein the multi-modal time series data includes audio data, video data, heart rate data, blood pressure data and body temperature data.

[0068] S02, performing depression recognition preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix.

[0069] S03, extracting a spatial proximity matrix and a pixel value similarity matrix from the first feature vector matrix.

[0070] S04, inputting the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second feature vector matrix; wherein the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function.

[0071] S05, setting a priority of recognition for a feature vector in the second feature vector matrix to obtain a third feature vector matrix.

[0072] S06, inputting the third feature vector matrix into a pre-configured LSTM recognition model to obtain a recognition result.

[0073] In step S01, it should be noted that the multi-modal time series data of a depression patient collected by a collection device; wherein the multi-modal time series data includes audio data, video data, heart rate data, blood pressure data and body temperature data.

[0074] Specifically, in an optional embodiment, the collection device includes a microphone, a camera integrated on a humanoid robot, and an external collection device including a heart rate detection device, a blood pressure measuring device and a body temperature collection sensor. Among them, the microphone and the camera are responsible for collecting the audio data and video data of the patient, ensuring that the patient's voice, facial expression and other related behavior can be captured. The audio data is transmitted in the form of raw audio stream, and the optional embodiment includes uncompressed WAV format and compressed MP3 format. The video data is transmitted in the form of frame sequence, and the optional embodiment includes that each frame can be a static image in JPEG or PNG format. Among them, the external collection device is used to monitor the physiological data of the patient, including heart rate data, blood pressure data and body temperature data. The heart rate data, the blood pressure data and the body temperature data are all recorded as measurement values at a series of time points, and are transmitted in the form of time series.

[0075] In order to transmit the physiological data collected by the external collection device, including heart rate data, blood pressure data and body temperature data, to the humanoid robot, two implementation modes are provided as follows:

[0076] In one implementation mode, the external collection device and the humanoid robot are both equipped with wireless communication modules, and the external collection device and the humanoid robot communicate through Bluetooth protocol. The external collection device transmits the collected physiological data to the humanoid robot. The receiving module equipped inside the humanoid robot decodes the received signal and stores it in the internal database.

[0077] In another implementation mode, data transmission is realized between the external collection device and the humanoid robot through wired connection. The external collection device is connected to the humanoid robot through USB interface, and transmits the collected physiological data to the humanoid robot. The receiving module equipped inside the humanoid robot decodes the received signal and stores it in the internal database.

[0078] In step S02, it is necessary to note that the depression recognition preprocessing operation is performed according to the multi-modal time series data to obtain the first feature vector matrix.

[0079] Specifically, the first step is to synchronize the multi-modal time series data to eliminate the time difference between different data sources. The synchronization processing refers to aligning the time series data from different sensors and data sources to ensure their consistency in time. Since different devices may have different sampling frequencies and start times, in order to ensure that the data can accurately reflect the patient's state at the same time, the following steps need to be taken: add accurate time stamp to each data point, if the device itself has no time stamp function, then manually add time stamp according to the order and frequency of data collection. In some cases where the data sampling frequency is low, interpolation methods can be used to supplement the data so that the time stamps of all data sources are consistent. Common interpolation methods include linear interpolation and spline interpolation. Select a unified time window to clip all data within the same time range, so that the time stamps of different data sources have consistency within the same time range. If the sampling rates of different devices are different, all data can be unified to the same sampling rate through resampling.

[0080] In the second step, feature extraction is performed on the multi-modal time series data including audio data, video data, heart rate data, blood pressure data, and body temperature data to extract key features that can reflect the characteristics of depression. The heart rate data, blood pressure data, and body temperature data can be directly used as features without additional processing. The features extracted from the audio data include MFCCs, fundamental frequency, and volume. The features extracted from the video data include facial landmarks, facial action units, facial expressions, and facial skin color.

[0081] Specifically, the MFCCs (Mel Frequency Cepstral Coefficients) capture the speech characteristics and timbre of the sound, which can help analyze the emotional state and speech patterns of the voice. The fundamental frequency reflects the pitch of the sound. The fundamental frequency of the voice of a person with depression may be lower or less stable than normal. The volume reflects the intensity and emotional expression of the sound. People with depression may exhibit lower volume, and changes in volume can reveal emotional states.

[0082] In the process of calculating the Mel Frequency Cepstral Coefficients (MFCCs), the audio signal is first preprocessed. The audio signal is divided into multiple small segments, usually lasting 20 to 40 milliseconds, and there is a certain overlap between adjacent segments to ensure that the smooth transition of the signal can be captured. Next, a window function (such as the Hamming window) is applied to each frame to reduce edge effects and smooth the signal. Then, a Fast Fourier Transform (FFT) is performed on each frame of signal to convert the time domain signal to the frequency domain signal. This step generates a frequency spectrum, which represents the intensity of each frequency component. To further analyze the frequency spectrum, it is processed through a set of Mel filters that are uniformly distributed on the Mel scale. These filters simulate the perceptual characteristics of the human ear for different frequencies. The output power spectrum of each Mel filter is processed by logarithmic compression to obtain a log power spectrum. This step helps to reduce the dynamic range of the power spectrum, making the features more suitable for subsequent analysis. Finally, a Discrete Cosine Transform (DCT) is performed on the log power spectrum to obtain the Mel Frequency Cepstral Coefficients (MFCCs). Usually, only the first 13 MFCC coefficients are retained as features, as these coefficients can effectively represent the main speech characteristics of the audio signal.

[0083] Specifically, the facial landmarks include 68 facial landmarks (such as eye corners, nose tip, mouth corners, etc.), through which the shape and pose of the face can be captured. The facial action units are features that describe facial muscle movements, such as eyebrow raising, mouth corner lifting, etc., which can be used to describe specific expression changes and are usually used for expression analysis. The facial expressions include happiness, sadness, anger, surprise, disgust, and fear. The facial skin color is used to capture skin color changes caused by emotions, such as flushing, pallor, etc.

[0084] In the third step, the extracted multi-modal features are fused to obtain a high-dimensional feature vector matrix. The feature fusion uses early fusion, that is, directly splicing the features of all modalities at the feature level to form a high-dimensional feature vector.

[0085] In step S03, it should be noted that after obtaining the high-dimensional feature vector matrix after preprocessing in step S02, the spatial proximity matrix and the pixel value similarity matrix of the high-dimensional feature vector matrix need to be further extracted. The calculation of the spatial proximity matrix involves determining the relative position of the feature vectors in the high-dimensional feature vector matrix in space. This process is mainly achieved by calculating the distance between each feature vector in the feature vector matrix and its adjacent feature vectors. Specifically, for the pixel features in the video data, in an optional implementation, the Euclidean distance is used to calculate the relative position between the feature vectors. The calculation formula of the Euclidean distance is as follows:

[0086]

[0087] where D is the Euclidean distance, and represent the coordinates of the feature vectors in the high-dimensional feature vector matrix in the ith dimension, and n is the dimension of the feature vectors. The Euclidean distance calculation is performed on all the feature vectors in the high-dimensional feature vector matrix to obtain a complete proximity matrix. In the calculation process, all pairs of feature vectors in the feature vector matrix are first traversed, and then the above Euclidean distance formula is applied to calculate the distance between each pair of feature vectors, and finally a proximity matrix is generated, where each element represents the spatial distance between two feature vectors. This matrix helps to identify the distribution relationship of features in space, for example, which feature vectors are close to or far from each other, so that the layout of features in space can be understood.

[0088] where the calculation of the pixel value similarity involves measuring the similarity between the pixel values of the feature vectors in the high-dimensional feature vector matrix. For the pixel features in the video data, optional implementations include using cosine similarity or Pearson correlation coefficient and other measurement methods to calculate the similarity between different feature values.

[0089] In an implementation, cosine similarity is used to calculate the similarity between different feature values. The calculation formula of the cosine similarity is as follows:

[0090]

[0091] where S is the cosine similarity, and These represent the coordinates of the eigenvectors in the i-th dimension of the high-dimensional eigenvector matrix. The cosine similarity is calculated for all eigenvectors in the high-dimensional eigenvector matrix to obtain a complete similarity matrix, where each element represents the degree of similarity between the pixel values ​​of two eigenvectors.

[0092] In another implementation, the Pearson correlation coefficient is used to calculate the similarity between different feature values. The formula for calculating the Pearson correlation coefficient is as follows:

[0093]

[0094] Where S is the Pearson correlation coefficient. and and represent the coordinates of the eigenvectors in the i-th dimension of the high-dimensional eigenvector matrix, respectively. and These represent the mean of the eigenvectors in the i-th dimension of the high-dimensional eigenvector matrix. The Pearson correlation coefficient is calculated for all eigenvectors in the high-dimensional eigenvector matrix to obtain a complete similarity matrix, where each element represents the degree of similarity between the pixel values ​​of two eigenvectors.

[0095] In step S04, it should be noted that after obtaining the spatial proximity matrix and the pixel value similarity matrix, the spatial proximity matrix and the pixel value similarity matrix are input into a pre-configured bilateral filter for noise reduction and smoothing to obtain a second feature vector matrix; wherein, the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function.

[0096] Specifically, the bilateral filter is a nonlinear filtering technique used to smooth and denoise feature data while preserving edge details. The bilateral filter is composed of two Gaussian functions: a spatial Gaussian function and an intensity Gaussian function. The spatial Gaussian function is used to measure the positional relationship of feature vectors in space, calculating the spatial distance weight between feature vectors, thus giving higher weight to feature vectors with similar spatial positions. The intensity Gaussian function is used to calculate the similarity weight between feature vector values, giving higher weight to points with similar feature values. The combination of these two functions ensures that important details are not blurred during the smoothing process. When applying the bilateral filter, the weighted average value between each feature vector and its neighboring feature vectors is first calculated. The weight is determined by the spatial Gaussian function and the intensity Gaussian function, where the spatial weight focuses on the positional relationship of the feature vectors, and the intensity weight focuses on the similarity of the feature values. Through this weighted average method, the filter can effectively reduce the noise in the feature data while preserving important edge and detail information. Finally, the data processed by the bilateral filter generates a second feature vector matrix, which preserves key feature information while reducing noise and smoothing, providing a clearer and more reliable data basis for subsequent LSTM model input.

[0097] The bilateral filter is specifically configured as:

[0098]

[0099]

[0100] wherein, and are the positions of sample i and sample j of the first feature vector matrix, respectively; and are the feature vectors of sample i and sample j of the first feature vector matrix, respectively; is a spatial Gaussian function used to process the spatial proximity of sample i and sample j; is an intensity Gaussian function used to process the pixel value similarity of sample i and sample j; is a normalization factor to ensure that the filtered result is a weighted average value;

[0101] wherein the spatial Gaussian function is specifically:

[0102]

[0103] wherein, is the spatial distance between sample i and sample j of the first feature vector matrix; is the spatial standard deviation;

[0104] wherein the intensity Gaussian function Specifically,

[0105]

[0106] wherein, is the feature vector distance between sample i and sample j of the first feature vector matrix, is the intensity standard deviation.

[0107] In step S05, it should be noted that the priority of the feature vectors in the second feature vector matrix is set to obtain a third feature vector matrix, including: the feature vectors in the second feature vector matrix are sorted according to their contribution and importance in identification. Specifically, first, the priority score of each feature vector is calculated , which comprehensively considers the information gain and the covariance with other feature vectors, and the priority score is calculated by the following formula:

[0108] wherein, is the priority of the feature vector in the second feature vector matrix; is the weight of the feature vector in the second feature vector matrix, used to reflect the basic importance of the feature vector in the second feature vector matrix in depression identification; is the information gain of the feature vector in the second feature vector matrix, used to represent the contribution of the feature vector in classification; is a balance coefficient, used to balance the influence of information gain and inter-feature covariance; is the covariance of the feature vector in the second feature vector matrix and the feature vector in the second feature vector matrix, used to reflect the interaction between the two features.

[0109] In step S06, it should be noted that the third feature vector matrix is input into a pre-configured LSTM identification model to obtain an identification result.

[0110] wherein, referring to Figure 2 , the configuration process of the LSTM identification model includes:

[0111] S11, obtaining historical multi-modal time series data;

[0112] S12, performing a model training preprocessing operation according to the historical multi-modal time series data, to obtain a first historical feature vector matrix;

[0113] S13, performing a data labeling operation according to the first historical feature vector matrix, to obtain a labeled vector matrix;

[0114] S14, inputting the first historical feature vector matrix into a pre-configured bilateral filter to perform noise reduction and smoothing, to obtain a second historical feature vector matrix;

[0115] S15, inputting the second historical feature vector matrix and the labeled vector matrix into the LSTM recognition model for training, and determining that the training is completed after a loss function of the LSTM recognition model meets a convergence condition, to obtain a trained LSTM recognition model.

[0116] Specifically, the historical multi-modal time series data is obtained from a clinical trial or a medical research project, or a public depression patient data set, and then data preprocessing is performed on the historical multi-modal time series data, specifically including:

[0117] The collected multi-modal time series data is processed synchronously to eliminate the time difference between different data sources. The purpose of synchronous processing is to align the time series data from different sensors and data sources to ensure their consistency in time. Since different devices may have different sampling frequencies and start times, in order to ensure that the data can accurately reflect the patient's state at the same time, some processing must be done. Precise timestamps are added to each data point. If the device itself does not have timestamp function, the timestamps need to be added manually according to the order and frequency of data collection. For the case of low data sampling frequency, interpolation methods can be used to supplement the data to make the timestamps of all data sources consistent. Common interpolation methods include linear interpolation and spline interpolation. Next, a unified time window is selected to crop all data to the same time range to ensure that the timestamps of different data sources are consistent within this time range. If there is a difference in sampling rate between different devices, all data can be unified to the same sampling rate through resampling.

[0118] After completing the synchronous processing, feature extraction is performed on the multi-modal time series data to extract key parameters that can reflect the characteristics of depression. The multi-modal time series data includes audio data, video data, heart rate data, blood pressure data, and body temperature data. Heart rate data, blood pressure data, and body temperature data can be used as features directly without additional processing.

[0119] In the feature extraction of audio data, the main features include Mel Frequency Cepstral Coefficients (MFCCs), fundamental frequency, and volume. MFCCs capture the speech characteristics and timbre of the sound, which helps to analyze the emotional state and speech patterns. The fundamental frequency reflects the pitch of the sound, and the fundamental frequency of the voice of a patient with depression may be lower or more unstable than normal. The volume reflects the intensity of the sound and emotional expression, and patients with depression may exhibit lower volume, and changes in volume can reveal emotional states.

[0120] In the process of calculating Mel Frequency Cepstral Coefficients (MFCCs), the audio signal is first preprocessed. The audio signal is divided into multiple small segments, each usually lasting 20 to 40 milliseconds, and there is a certain overlap between adjacent segments to ensure that the smooth transition of the signal can be captured. Then, a window function (such as the Hamming window) is applied to each frame to reduce edge effects and smooth the signal. Subsequently, a Fast Fourier Transform (FFT) is performed on each frame of signal to convert the time domain signal to the frequency domain signal, generating a frequency spectrum representing the intensity of each frequency component. To further analyze the frequency spectrum, the frequency spectrum is processed through a set of Mel filters uniformly distributed in the Mel scale. These filters simulate the perceptual characteristics of the human ear for different frequencies. The output power spectrum of each Mel filter is processed by logarithmic compression to obtain a log power spectrum, which helps to reduce the dynamic range of the power spectrum, making the features more suitable for subsequent analysis. Finally, a Discrete Cosine Transform (DCT) is performed on the log power spectrum to obtain Mel Frequency Cepstral Coefficients (MFCCs). Usually, only the first 13 MFCC coefficients are retained as features, as these coefficients can effectively represent the main speech characteristics of the audio signal.

[0121] For video data, the extracted features include facial key points, facial action units, facial expressions, and facial skin color. Facial key points include 68 landmark points (such as eye corners, nose tip, mouth corners, etc.), which are used to capture the shape and pose of the face. Facial action units describe the movement characteristics of facial muscles, such as eyebrow lifting and mouth corner lifting, which are used for expression analysis. Facial expressions include happiness, sadness, anger, surprise, disgust, and fear, etc. Facial skin color features are used to capture skin color changes caused by emotions, such as flushing or pallor, etc.

[0122] These multi-modal features are fused and integrated into a high-dimensional feature vector matrix for comprehensive analysis and modeling.

[0123] Then, the feature vector matrix is input into a pre-configured bilateral filter for noise reduction and smoothing to obtain a noise-reduced and smoothed feature vector matrix.

[0124] Then, the feature vector matrix is data-labeled to obtain a labeled vector matrix, and the labeled vectors in the labeled vector matrix include depression symptoms, depression levels, and emotional states; wherein the depression symptoms are whether there is depression; the depression levels are the severity of depression, including mild, moderate, and severe; and the emotional states include happiness, sadness, fear, anger, surprise, and disgust. In an optional embodiment, in order to ensure the accuracy of data labeling, experts in the field are invited to review the labeling results. Experts can check whether the labels are consistent with the feature data to ensure the professionalism and accuracy of the labeling. Different labelers label the same data, and then the results are compared. Through the consistency of the comparison, errors in the labeling can be identified and corrected.

[0125] The feature vector matrix and the labeled vector matrix are input into the LSTM recognition model for training. In this process, the LSTM model continuously adjusts its internal parameters through the backpropagation algorithm to learn the relationship between multi-modal features and depression states. Specifically, the feature vector matrix and the labeled vector matrix are divided into a training set and a validation set to ensure that the model can effectively evaluate its performance during the training process. The training set is used to optimize the model parameters, and the validation set is used to evaluate the generalization ability of the model to prevent overfitting.

[0126] During the training process, the LSTM model processes the input time series data. Since LSTM has the ability to process time series data, it can capture the time dependence in the feature vector matrix, which is crucial for identifying the behavior patterns and physiological state changes of depression patients. The input of the LSTM includes the feature vectors at each time step in the sequence, which can include audio features (such as MFCCs, fundamental frequency, volume), video features (such as facial key points, facial action units), and physiological features (such as heart rate, blood pressure, body temperature). At each time step, the LSTM model updates its internal state based on the current input features and the state in its memory cells, and outputs a new hidden state. These hidden states contain the cumulative feature information in the time series, which can effectively capture the behavior and physiological feature changes of depression patients at different time points. The output of the LSTM model is compared with the labeled vector through a fully connected layer to calculate the loss function. Common loss functions include cross-entropy loss and mean square error, which measure the difference between the model's prediction and the true label.

[0127] Through the backpropagation algorithm, the LSTM model continuously adjusts its weights and biases to minimize the loss function. Backpropagation passes the gradient of the loss with respect to the model parameters back through the network using the chain rule, updating the weight matrices and bias terms to optimize the model performance. This training process usually requires multiple iterations, called training epochs. At the end of each training epoch, the model's performance on the validation set is evaluated to monitor the training progress. The training is considered complete when the loss function of the LSTM recognition model meets the convergence condition, and the trained LSTM recognition model is obtained.

[0128] After inputting the third feature vector matrix into the pre-configured LSTM recognition model to obtain the recognition result, the method further comprises:

[0129] inputting the recognition result into the pre-configured depression database to obtain diagnosis suggestion information, and outputting the diagnosis suggestion information to the user;

[0130] wherein, referring to Figure 3 The configuration process of the depression database comprises:

[0131] S21, acquiring a depression knowledge dataset;

[0132] S22, performing a knowledge graph construction preprocessing operation according to the depression knowledge dataset to obtain a preprocessed depression knowledge dataset;

[0133] S23, inputting the preprocessed depression knowledge dataset into a database system to perform a knowledge graph construction operation to obtain a depression database.

[0134] Specifically, a depression knowledge dataset is acquired. The dataset includes various information related to depression, such as symptoms, treatment methods, drugs, expert opinions, research literature, patient cases, etc. These data can be collected from medical databases, research papers, hospital records, patient reports, and public medical information resources.

[0135] Next, according to the depression knowledge dataset, a knowledge graph construction preprocessing operation is performed. Preprocessing includes data cleaning, deduplication, format conversion and standardization. For example, noise and error information in the data will be removed during the cleaning process; the deduplication operation ensures that there is no duplicate information in the dataset; format conversion unifies the data into a format suitable for knowledge graph construction, such as JSON or CSV; standardization operations ensure that all data fields are consistent, such as uniform date format, standardized drug names and symptom descriptions, etc. The preprocessed data forms a preprocessed depression knowledge dataset.

[0136] Then, in an optional implementation, the preprocessed depression knowledge dataset is input into a Neo4j database system for knowledge graph construction. First, the schema of the knowledge graph is defined, including the types of nodes and relationships. For example, node types can include "symptoms", "treatment methods", "drugs", "experts", "research literature", "patient cases", etc., and relationship types can include "association", "treatment", "recommendation", "reference", etc. Then, according to the defined schema, the preprocessed dataset is imported into Neo4j to create corresponding nodes and relationships. The Neo4j is a graph-based NoSQL database system for storing and managing interrelated data. After the construction of the knowledge graph is completed, the depression knowledge base can be accessed and analyzed through the query function of Neo4j. For example, using Cypher to query the treatment methods of a specific symptom, find the side effects of a certain drug, or analyze the commonalities in patient cases, etc. The Cypher is a declarative query language for querying and manipulating graph databases, mainly used with Neo4j graph databases. It is similar to SQL, but targets the graph data model rather than the relational data model. Cypher allows users to create, read, update, and delete graph data in a simple and intuitive way.

[0137] In summary, the first embodiment of the present application provides a depression recognition method, which includes obtaining multi-modal time series data of depression patients using a collection device, including audio, video, heart rate, blood pressure, and body temperature data. Then, these data are preprocessed to generate a first feature vector matrix. Next, the spatial proximity and pixel value similarity matrices of this matrix are extracted, and a bilateral filter is used to denoise and smooth these matrices to obtain a second feature vector matrix. Finally, this processed matrix is input into an LSTM model to obtain the recognition result of depression. In the specific implementation process, the collection device can include a microphone and a camera integrated on a robot, as well as an external physiological data detection device. Data synchronization processing ensures the consistency of all data in time. Feature extraction includes MFCCs, fundamental frequency, and volume of audio, facial key points and expressions of video, etc. Spatial proximity and pixel value similarity are calculated by Euclidean distance and cosine similarity, and processed by a bilateral filter. The LSTM model is trained to learn the relationship between multi-modal features and depression status. Finally, the recognition result is compared with the information in the depression knowledge database to provide diagnostic recommendations.

[0138] Reference Figure 4 The second embodiment of the present application provides a depression device, which includes:

[0139] The acquisition module 101 is configured to acquire multi-modal time series data of a depressive patient collected by a collection device, wherein the multi-modal time series data comprises audio data, video data, heart rate data, blood pressure data and body temperature data.

[0140] The preprocessing module 102 is configured to perform a depressive recognition preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix.

[0141] The extraction module 103 is configured to extract a spatial proximity matrix and a pixel value similarity matrix from the first feature vector matrix.

[0142] The filtering module 104 is configured to input the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second feature vector matrix, wherein the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function.

[0143] The priority setting module 105 is configured to set a recognition priority of a feature vector in the second feature vector matrix to obtain a third feature vector matrix.

[0144] The recognition module 106 is configured to input the third feature vector matrix into a pre-configured LSTM recognition model to obtain a recognition result.

[0145] The configuration process of the LSTM recognition model comprises the following steps.

[0146] Acquire historical multi-modal time series data.

[0147] Perform a model training preprocessing operation according to the historical multi-modal time series data to obtain a first historical feature vector matrix.

[0148] Perform a data labeling operation according to the first historical feature vector matrix to obtain a labeled vector matrix.

[0149] Input the first historical feature vector matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second historical feature vector matrix.

[0150] Input the second historical feature vector matrix and the labeled vector matrix into the LSTM recognition model for training, and determine that the training is completed when a loss function of the LSTM recognition model meets a convergence condition to obtain a trained LSTM recognition model.

[0151] It should be noted that the depressive recognition device provided by the embodiment of the present application is used to execute all process steps of the depressive recognition method provided by the above-mentioned embodiment, and the working principles and beneficial effects of the two are one-to-one correspondence, thus it is not repeated here.

[0152] The embodiments of the present application further provide an electronic device. The electronic device comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor. The processor implements the steps in each of the above-mentioned depression identification method embodiments when running the computer program. Alternatively, the processor implements the functions of each module / unit in each of the above-mentioned apparatus embodiments when running the computer program.

[0153] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device.

[0154] The electronic device can be a desktop computer, a notebook computer, a palm computer, a smart tablet and the like. The electronic device can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above components are only examples of the electronic device and do not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the above, or combine certain components, or different components, for example, the electronic device can further include an input / output device, a network access device, a bus, etc.

[0155] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The processor is the control center of the electronic device, which connects all parts of the electronic device through various interfaces and lines.

[0156] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the electronic device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a nonvolatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0157] The modules / units integrated in the electronic device, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of the above-mentioned various method embodiments can be realized. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0158] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate units can or can not be physically separate, and the units displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0159] The above specific embodiments further illustrate the purpose, technical scheme and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method of identifying depression, characterized by, The method comprises: acquiring multi-modal time series data of a depressive patient collected by a collection device; wherein the multi-modal time series data comprises audio data, video data, heart rate data, blood pressure data and body temperature data; performing depressive disorder recognition preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix; extracting a spatial proximity matrix and a pixel value similarity matrix from the first feature vector matrix; inputting the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second feature vector matrix; wherein the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function; setting a priority of a feature vector in the second feature vector matrix for recognition to obtain a third feature vector matrix; inputting the third feature vector matrix into a pre-configured LSTM recognition model to obtain a recognition result; The configuration process of the LSTM recognition model comprises: acquiring historical multi-modal time series data; performing model training preprocessing operation according to the historical multi-modal time series data to obtain a first historical feature vector matrix; performing data labeling operation according to the first historical feature vector matrix to obtain a labeled vector matrix; inputting the first historical feature vector matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second historical feature vector matrix; setting a priority of a feature vector in the second historical feature vector matrix for recognition to obtain a third historical feature vector matrix, inputting the third historical feature vector matrix and the labeled vector matrix into the LSTM recognition model for training, and determining that the training is completed after the loss function of the LSTM recognition model meets a convergence condition to obtain a trained LSTM recognition model; wherein the input of the LSTM comprises feature vectors at each time step in a sequence, and the vectors comprise audio features, video features and physiological features; at each time step, the LSTM model updates its internal state according to the current input feature and the state in its memory unit, and outputs a new hidden state; the hidden state contains the cumulative feature information in the time series, and captures the behavior and physiological feature changes of the depressive patient at different time points; wherein the priority calculation formula of the feature vector in the second feature vector matrix is specifically: wherein, is a priority of the eigenvector in the second eigenvector matrix; is a weight of the eigenvector in the second eigenvector matrix, for reflecting a basic importance of the eigenvector in the second eigenvector matrix in depression recognition; is an information gain of the eigenvector in the second eigenvector matrix, for representing a contribution of the eigenvector in the second eigenvector matrix in classification; is a balance coefficient, for balancing the influence of the information gain and the inter-feature covariance; is a covariance of the eigenvector and the eigenvector in the second eigenvector matrix, for reflecting the interaction between the two features.

2. The depression recognition method according to claim 1, characterized by, The depressive disorder recognition preprocessing operation according to the multi-modal time series data to obtain the first feature vector matrix specifically comprises: performing feature extraction operation according to the multi-modal time series data to obtain multi-modal time series features; performing feature fusion operation according to the multi-modal time series features to obtain the first feature vector matrix. 3.The depression recognition method of claim 1, wherein, The bilateral filter is specifically configured as: wherein, and are the positions of sample i and sample j of the first feature vector matrix, respectively; and are the feature vectors of sample i and sample j of the first feature vector matrix, respectively; is a spatial Gaussian function for handling the spatial proximity of sample i and sample j; is an intensity Gaussian function for handling the pixel value similarity of sample i and sample j; is a normalization factor to ensure that the filtered result is a weighted average. where the spatial Gaussian function Specifically: wherein, is the spatial distance between sample i and sample j of the first feature vector matrix; is the spatial standard deviation; where the intensity Gaussian function Specifically: wherein, is the feature vector distance between sample i and sample j of the first feature vector matrix, is the intensity standard deviation. 4.The depression recognition method of claim 1, wherein, the labeled vectors in the labeled vector matrix comprise depressive symptoms, depressive degrees and emotional states; wherein the depressive symptoms are whether there is depression; the depressive degrees are the severity of depression, including mild, moderate and severe; and the emotional states comprise happiness, sadness, fear, anger, surprise and disgust.

5. The depression recognition method of claim 1, wherein, After inputting the third feature vector matrix into a pre-configured LSTM recognition model to obtain a recognition result, the method further comprises: inputting the recognition result into a pre-configured depression database to obtain diagnosis suggestion information, and outputting the diagnosis suggestion information to a user; wherein the configuration process of the depression database comprises: obtaining a depression knowledge dataset; performing knowledge graph construction preprocessing operation according to the depression knowledge dataset to obtain a preprocessed depression knowledge dataset; inputting the preprocessed depression knowledge dataset into a database system to perform knowledge graph construction operation to obtain a depression database. 6.The depression recognition method according to claim 5, characterized in that, The method according to any one of claims 1 to 6, comprising: an acquisition module configured to acquire multi-modal time series data of a depression patient collected by a collection device; wherein the multi-modal time series data comprises audio data, video data, heart rate data, blood pressure data and body temperature data; a preprocessing module configured to perform depression recognition preprocessing operation according to the multi-modal time series data to obtain a first feature vector matrix; 7. A depression recognition apparatus characterized by comprising: an extraction module configured to extract a spatial proximity matrix and a pixel value similarity matrix from the first feature vector matrix; a filtering module configured to input the spatial proximity matrix and the pixel value similarity matrix into a pre-configured bilateral filter to perform noise reduction and smoothing to obtain a second feature vector matrix; wherein the bilateral filter is composed of a spatial Gaussian function and an intensity Gaussian function; a priority setting module configured to set a recognition priority for a feature vector in the second feature vector matrix to obtain a third feature vector matrix; an identification module configured to input the third feature vector matrix into a pre-configured LSTM recognition model to obtain a recognition result. The depression recognition device according to claim 7. The depression recognition device according to claim 7. ​ 8. A humanoid robot, characterized by, ​

Images