A method, device and equipment for emotion orbit identification based on voice biomarker

Abstract

The application discloses a speech biomarker-based emotion track identification method, device and equipment, relates to the fields of speech signal processing and medical health data analysis, and comprises the following steps: acquiring speech data, performing framing, short-term feature extraction, difference calculation and second-level statistics, and constructing a speech feature matrix; acquiring a three-dimensional emotion track label matrix, wherein the three-dimensional emotion track comprises emotion valence, emotion arousal and emotion stability; training a multi-output regression model based on the speech feature matrix and the emotion track label matrix to obtain an emotion track identification model; inputting to-be-identified speech data into the model to output a three-dimensional emotion track prediction result; and generating a speech digital phenotype through explainability analysis. The application can objectively and low-burdenly capture the dynamic change process of the instantaneous emotion of a subject, realize multi-modal data fusion, provide a digital phenotype with clinical explanatory force, and is suitable for emotion care evaluation and intervention of sensitive groups.

Description

Technical Field

[0001] This application relates to the fields of speech signal processing and medical and health data analysis, and in particular to a method, apparatus and device for emotion trajectory recognition based on speech biomarkers. Background Technology

[0002] With the development of antiviral therapy, HIV / AIDS (human immunodeficiency virus infection / acquired immunodeficiency syndrome) has gradually become a chronic disease requiring long-term management, and the number of elderly patients is constantly increasing. Under the influence of factors such as long-term disease management, increasing age, comorbidity burden, and insufficient social support, elderly HIV / AIDS patients are prone to persistent or fluctuating emotional distress. Therefore, identifying their emotional state has significant value in nursing assessment and intervention. Voice signals, as a non-invasive, easily collected, and low-cost type of biobehavioral data, contain rich acoustic, linguistic, and paralinguistic information and have shown potential for application in mental health assessment.

[0003] However, existing technical solutions have the following drawbacks: First, there is a lack of a mechanism for the synchronous acquisition and time alignment of speech data and continuous emotional trajectory data. Existing speech models mostly output emotion categories or psychological states (such as scale scores), which makes it difficult to reflect the dynamic trajectory of emotional valence, arousal, and stability over time, and to reflect instantaneous emotional changes. They also fail to establish a precise correspondence between "speech segment - task time point - emotional state".

[0004] Second, existing emotion recognition results struggle to describe instantaneous emotional changes. Existing models typically output discrete emotion categories or single predicted scores, failing to characterize the dynamic changes in an individual's emotions at different points in time.

[0005] Third, existing emotion recognition methods suffer from limited data modalities. They often use single acoustic or text features as input, making it difficult to simultaneously utilize numerical acoustic features, time-frequency image features, and linguistic text features. This results in models that do not adequately represent the dynamic changes in emotions.

[0006] Fourth, there is a lack of quality control and data security management mechanisms for the collection of voice emotion data from sensitive populations. The disease identity, voice data, and emotional state of patients with HIV / AIDS are all sensitive information, and ordinary models lack anonymous encoding, access control, and encrypted storage procedures for such sensitive data. Summary of the Invention

[0007] The purpose of this application is to provide a method, device, and equipment for emotion trajectory recognition based on speech biomarkers, in order to solve the problems of existing emotion assessment methods relying on subjective reports, difficulty in describing instantaneous emotional dynamic changes, and insufficient utilization of multimodal information.

[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for emotion trajectory recognition based on speech biomarkers, including: Acquire speech data of subjects performing standardized speech tasks; The speech data is subjected to feature extraction to obtain a speech feature matrix; Obtain an emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability. Using the speech feature matrix as input variables and the emotion track label matrix as supervised learning labels, a multi-output regression model is trained to obtain an emotion track recognition model. Acquire the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

[0009] Optionally, feature extraction is performed on the speech data to obtain a speech feature matrix, specifically including: The voice data is subjected to frame-segmentation and windowing processing to obtain multiple short time frames; Extract the short-term acoustic features of each short frame to obtain the first feature vector; Calculate the first-order difference features of the first feature vectors of adjacent short-time frames to obtain the second feature vector; The first feature vector and the second feature vector are concatenated to obtain the frame-level feature vector corresponding to each short frame; Using a preset time length as the intermediate window, the mean and standard deviation of multiple frame-level feature vectors within the intermediate window are calculated dimension by dimension, and the mean and standard deviation are concatenated to obtain the second-level feature vector corresponding to each intermediate window. Arrange the second-level feature vectors of all intermediate windows in chronological order to form the speech feature matrix.

[0010] Optionally, the short-term acoustic features include 34-dimensional features, with the first feature vector being 34-dimensional, the second feature vector being 34-dimensional, and the frame-level feature vector being 68-dimensional; the second-level feature vector formed by concatenating the mean and the standard deviation is 136-dimensional.

[0011] Optionally, obtaining an emotion track label matrix aligned with the voice data on the time axis includes: Obtain labeled data that continuously describes the emotional valence and emotional arousal when the subject listens back to the voice data; The emotional stability is calculated based on the changes in emotional valence and emotional arousal between adjacent time points; The labeled data and the emotional stability are summarized according to the same time window scale as the speech feature matrix to form a three-dimensional emotion label corresponding to each time window, and arranged in chronological order to form the emotion track label matrix.

[0012] Optionally, calculating the emotional stability based on the changes in emotional valence and emotional arousal between adjacent time points specifically includes: The following formula can be used to calculate the degree of emotional fluctuation between adjacent time points. : ; in, Indicates the first Emotional valence at a given point in time Indicates the first Emotional arousal at a given point in time; Based on the degree of emotional fluctuation The moving average or cumulative trend of change within the time series is used to calculate the [number]. Emotional stability at various time points , .

[0013] Optionally, the multi-output regression model includes at least one of gradient boosting regression model, extremely random tree regression model, random forest regression model, support vector regression model, convolutional neural network regression model, or long short-term memory network regression model; the output layer of the multi-output regression model is set to three consecutive numerical output nodes, corresponding to the emotional valence, emotional arousal and emotional stability, respectively.

[0014] Optionally, when training the multi-output regression model, multimodal data is also introduced for joint modeling, wherein the multimodal data includes: The speech data is converted into a time-frequency spectrum or a Mel spectrum. A point projection map is formed by projecting emotion markers along the time axis onto a two-dimensional emotion plane. Text features extracted from the language content generated in the standardized speech task.

[0015] Optionally, a speech-digit phenotype is generated based on an interpretability analysis method, including: The SHAP interpretability analysis method is used to calculate the contribution of each speech biomarker feature in the speech feature matrix to the three-dimensional emotion trajectory prediction result. Key speech features whose contribution exceeds a preset threshold are sorted and visualized; By combining the acoustic meaning and contribution direction of the key speech features with the three-dimensional emotion trajectory prediction results, reverse reasoning is performed to generate the speech digit phenotype of the subject.

[0016] Secondly, this application provides an emotion trajectory recognition device based on voice biomarkers, comprising: The speech data acquisition module is used to acquire speech data of subjects performing standardized speech tasks; The feature extraction module is used to extract features from the speech data to obtain a speech feature matrix; The emotion track label matrix acquisition module is used to acquire an emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability; The emotion trajectory recognition model determination module is used to train a multi-output regression model using the speech feature matrix as input variables and the emotion trajectory label matrix as supervised learning labels to obtain the emotion trajectory recognition model. The three-dimensional emotion trajectory prediction module is used to acquire the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

[0017] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the emotion trajectory recognition method based on speech biomarkers as described above.

[0018] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, device, and equipment for emotion trajectory recognition based on speech biomarkers, with the following advantages: (1) The output results can reflect the dynamic changes in emotions. This invention takes emotional valence, emotional arousal, and emotional stability as core dimensions and outputs a three-dimensional emotion trajectory that changes over time, which is more suitable for describing the instantaneous dynamic changes in individual emotions. (2) Multimodal fusion improves recognition stability. By jointly modeling numerical features, spectrograms, dot projection maps, and text features, emotional changes are characterized from multiple perspectives such as acoustics, time frequency, language, and trajectory images, improving the comprehensiveness of the recognition results. (3) It has clinical interpretability. By clarifying the contribution of key speech features through SHAP analysis, speech digital phenotypes are generated through reverse reasoning, providing intuitive technical support for nursing assessment and subsequent emotional intervention. (4) It is more suitable for sensitive populations and long-term chronic disease management. This application sets up standardized speech tasks, emotion marking methods, and data quality control processes for special populations such as the elderly with HIV / AIDS. At the same time, this method can also be extended to the general population or patients with other diseases for emotion monitoring. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an emotion trajectory recognition method based on voice biomarkers provided in an embodiment of this application; Figure 2(a) is a schematic diagram of emotion recording at 0 minutes and 0 seconds according to an embodiment of this application; Figure 2(b) is a schematic diagram of emotion recording at 0 minutes and 31 seconds according to an embodiment of this application. Figure 3(a) is a schematic diagram of an emotion trajectory depiction provided in an embodiment of this application. Figure 1 ; Figure 3(b) is a schematic diagram of emotional trajectory tracing provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] In one exemplary embodiment, such as Figure 1 As shown, an emotion trajectory recognition method based on voice biomarkers is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, it includes the following steps: Step 101: Obtain speech data of the subjects performing standardized speech tasks.

[0024] The subject is placed in a relatively quiet environment with sound insulation or noise less than 100dB. The voice input terminal (such as a microphone) is placed 5-10cm in front of the subject's mouth. The voice data acquisition frequency is set to 20-16000Hz. The operator gives the subject pronunciation guidance and signals the start of the test.

[0025] Subjects completed standardized speech tasks during speech acquisition. These tasks included semantic fluency, phonemic fluency, reading aloud, and emotion expression.

[0026] 1) Semantic fluency and phonological fluency test: Subjects are required to say as many words as possible that belong to the same category or begin with the same letter within 1 minute.

[0027] 2) Reading task: Select a text of about 500 words (such as literary works, news excerpts, etc.) and require the test takers to read it aloud without time limit, and mark the pinyin of the uncommon characters in the text.

[0028] 3) Emotional expression task: Participants describe a scene in turn with neutral emotions, positive emotions, neutral emotions (emotional cleansing), and negative emotions. Each scene lasts at least 10 minutes.

[0029] After the acquisition is complete, the audio is preprocessed: The acquired raw speech data undergoes preprocessing to improve audio data quality and ensure consistency in subsequent feature extraction. The specific steps are as follows: (1) Audio quality check Perform integrity and readability checks on the original audio files, and remove unreadable, silent, and invalid audio files with a duration of less than 1 second.

[0030] (2) Unified audio parameters The audio files included in the analysis were uniformly converted to mono and resampled to 16000Hz to reduce the impact of audio parameter differences on subsequent feature extraction.

[0031] (3) Background noise processing Spectral noise reduction is performed based on the audio quality to reduce the interference of environmental noise on speech feature extraction. The preprocessed audio data is then used for subsequent speech feature extraction.

[0032] Step 102: Extract features from the speech data to obtain a speech feature matrix.

[0033] Specifically, it includes: (1) Frame division and windowing First, the preprocessed audio is divided into frames and windowed. The frame length is set to 50 ms, the frame shift to 25 ms, and the overlap between adjacent frames is 50%. To reduce spectral leakage, each frame of audio signal is multiplied by a Hamming window, the window function of which is defined as: ; in, It is the total number of samples in the window. It is a sample index Window coefficient at the location, It represents the total number of samples in the window.

[0034] (2) Frame-level feature extraction: Using the pyAudioAnalysis speech analysis toolkit, 34 short-term acoustic features were extracted from each short-time frame to describe the instantaneous acoustic state of the speech signal within the short-time frame, as shown in Table 1, denoted as... To further extract the dynamic information of the speech signal over time, the first-order difference feature between adjacent short-time frames is calculated, denoted as... The calculation formula is as follows: ; in, Indicates the time index of a short frame. Indicates the first 34-dimensional raw speech features of a short time frame Indicates the first 34-dimensional raw speech features of a short time frame Indicates the first First-order difference features of a short time frame.

[0035] The 34-dimensional original short-term features are concatenated with the 34-dimensional first-order difference features to form a 68-dimensional frame-level feature vector. The calculation formula is as follows: ; Indicates the first This 68-dimensional frame-level speech feature vector is formed by concatenating several short time frames. This vector simultaneously contains the static acoustic information of the short time frames and the dynamic change information between adjacent frames, and serves as the basis for subsequent intermediate feature extraction.

[0036] Table 1. 34-dimensional short-term acoustic characteristics

[0037] (3) Second-level feature extraction: To further characterize the overall acoustic features of speech signals on a second-level timescale, this application extends the analysis time window from the frame level to the second level, setting 1 second as the intermediate window. Each 1-second intermediate window consists of a sequence of consecutive short-time frames, and each short-time frame within the window corresponds to a 68-dimensional frame-level feature vector.

[0038] For the j-th intermediate window, the mean and standard deviation of each of the 68-dimensional frame-level features within the window are calculated dimension by dimension. The mean reflects the average level of each acoustic feature within this time window, and the standard deviation reflects its fluctuation. The 68-dimensional mean vector and the 68-dimensional standard deviation vector are concatenated to form a 136-dimensional intermediate feature vector, calculated as follows: ; in, Let represent the mean vector of the 68-dimensional frame-level features within the j-th intermediate window. This represents the standard deviation vector of the 68-dimensional frame-level features within the j-th intermediate window. This represents the 136-dimensional mid-term feature vector formed by concatenating the two.

[0039] (4) Formation of the characteristic matrix: An audio segment is divided into multiple time windows of 1 second each. Each time window generates a 136-dimensional speech feature vector. All time windows are arranged in rows to form a speech feature matrix X, as shown in Table 2.

[0040] Table 2 136-dimensional speech feature matrix

[0041] Step 103: Obtain the emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability.

[0042] Emotional trajectory mapping is performed after the speech acquisition step. The dimensions of instantaneous emotional change selected in this application include emotional valence, emotional arousal, and emotional stability. Among them, emotional valence is used to represent the positive or negative direction of emotional experience, and its value range can be set to -10 to +10; emotional arousal is used to represent the activation level or intensity of emotion, and its value range can be set to 0 to +10, where 0 represents a calm or low arousal state, and 10 represents a highly activated or strongly aroused state; emotional stability is used to represent the degree of fluctuation of an individual's emotions during the task and the ability to return to a stable state, and is evaluated by the fluctuation amplitude and change trend within the time series.

[0043] After each audio task was completed, the recording file was imported into a self-made emotion tracking app (emotion trajectory tracking software). While listening back to their own recordings, the subjects continuously tracked their emotional valence and arousal at different points in time during the recording playback by dragging tracking points. Each point in the figure corresponds to the subject's emotional state at a specific time point, as shown in Figure 2(a) and Figure 2(b).

[0044] The recorded audio is uploaded to the emotion tracking software, which records the emotion every second. In Figures 2(a) and 2(b), the horizontal axis represents emotional valence, or positive or negative value, ranging from 0 to 10. The higher the score, the more positive the emotion; from -10 to 0, the higher the absolute value, the more negative the emotion. The vertical axis represents emotional arousal, ranging from 0 to 10. The higher the score, the stronger the emotional arousal. The emotion at each moment can be represented by a point in two-dimensional space, and as time passes, the user can move the point on the plane to represent the fluctuation of the emotion.

[0045] Existing technologies often use scales and questionnaires to label and assess the types of emotions, such as distinguishing between emotions like sadness, anxiety, depression, joy, and excitement. However, this application directly classifies emotions based on their essence, including three dimensions. This approach omits the cognitive processing process, making it more direct and closer to the real experience of the moment.

[0046] Once an audio recording is fully transcribed, a three-dimensional emotion track is generated. This track can be analyzed not only as digital data but also as multimodal data such as graph data. The changes in the emotion track in three-dimensional space represent the third dimension of "stability" in emotion evaluation. That is, after time-lapse transcription, each recording can form a three-dimensional emotion track corresponding to the speech time axis, as shown in Figures 3(a) and 3(b). When incorporating emotion track data into the model, researchers can extract the necessary data according to the scenario and research needs. For example, they can mine the data association between speech features per millisecond and X (emotional valence) and Y (emotional arousal); use the degree of change of X and Y per unit time as an indicator of Z (stability); and incorporate graph data such as the top-to-bottom projection area and position of the emotion track into the model analysis.

[0047] Emotional valence and emotional arousal were manually and continuously labeled by the subjects, while emotional stability was automatically calculated by the app based on a preset algorithm. Specifically, the app used the magnitude of change in emotional valence and emotional arousal between adjacent time points as an indicator of emotional fluctuation, and the calculation formula is as follows: ; ; in, Indicates the first Emotional valence at a given point in time Indicates the first Emotional arousal at a given point in time This indicates the degree of emotional fluctuation between adjacent time points. Indicates the first Emotional stability at a given point in time.

[0048] After the recording is completed, the app saves the correspondence between the recording playback time and the recording results, and summarizes the emotion recording results according to 1-second time windows to match the extraction scale of speech biomarker features. For each 1-second time window, the average values ​​of emotional valence, emotional arousal, and emotional stability within that time window are calculated to form the corresponding three-dimensional emotion label, denoted as (Y1, Y2, Y3). Here, Y1 represents emotional valence, Y2 represents emotional arousal, and Y3 represents emotional stability. The three-dimensional emotion track labels of all time windows are arranged in rows to form a label matrix Y, which serves as the supervised learning labels for the subsequent model training stage, as shown in Table 3.

[0049] Table 3 Emotional trajectory data records

[0050] Step 104: Using the speech feature matrix as the input variable and the emotion track label matrix as the supervised learning label, train a multi-output regression model to obtain an emotion track recognition model.

[0051] Model training and parameter optimization: During the model training phase, the speech biomarker feature matrix X (within a 1-second time window) was used as the model input variable, and the three-dimensional emotion track label matrix Y was used as the supervised learning label to construct an emotion track recognition model. The purpose of model training is to learn the correspondence between speech biomarker features X and three-dimensional emotion track labels Y. The correspondence between speech features and emotion track labels is shown in Table 4. Table 4. Correspondence between speech features and emotion track labels

[0052] Since Y1 emotional valence, Y2 emotional arousal, and Y3 emotional stability are all continuous numerical labels, this application employs a multi-output regression model for training and parameter optimization. Six regression models are used for training and parameter optimization, including gradient boosting regression, extremely random tree regression, and random forest regression—common ensemble learning methods for regression problems; support vector regression (SVR) that finds the optimal regression function to fit the relationship between input features and continuous target variables; convolutional neural network regression (CNN), a deep learning model adept at image data processing; and long short-term memory network regression (LSTM), adept at natural language processing and time series data analysis. In the three decision tree-based regression models, the initial number of trees was set to 100, and the learning rate was set to 0.15. The SVR model used a Gaussian kernel. The CNN model used a 2×2 kernel with a stride of 2, an RMSProp optimizer, ELU as the activation function, Xavier for initializing the weight matrices in all layers, Dropout regularization with a probability of 0.5, and three consecutive numerical output nodes corresponding to Y1 emotional valence, Y2 emotional arousal, and Y3 emotional stability. The LSTM model had two layers, a Dropout layer with a probability of 0.5, an Adam optimizer, a learning rate of 0.05, and three consecutive numerical output nodes. By comparing the prediction performance of different models and parameter combinations, the optimal emotion trajectory recognition model and its optimal parameter combination were determined.

[0053] Furthermore, this application allows for joint modeling of different types of data. Specifically, this includes: extracting features from speech signals to form a 136-dimensional numerical speech biomarker feature vector; performing time-frequency conversion on audio signals to generate spectrograms or Mel spectrograms, forming image modal data; projecting the emotion markers corresponding to each audio segment from right to left along the time axis onto an initial two-dimensional emotion plane to form a point projection map, which serves as image modal data; and for language content generated in semantic fluency, reading aloud, or image description tasks, text features can also be extracted to form text modal data. The aforementioned numerical feature vectors, image modal data, time-series data, and text modal data can be input into the adaptation model respectively, and used together to construct a multimodal emotion trajectory recognition model.

[0054] Model Validation and Model Selection During the model validation phase, the Bootstrap 1000-times resampling method was used to evaluate the recognition performance of different models and different parameter combinations. After each resampling, the model was trained according to the method described in the "Model Training and Parameter Optimization" section, and the 3D emotion trajectory results predicted by the model were compared with the actual 3D emotion trajectory labels.

[0055] Evaluation metrics include accuracy, precision, recall, and F1 score, used to evaluate the model's performance in recognizing emotion valence, emotional arousal, and emotional stability. After Bootstrap resampling, the evaluation results under different models and parameter combinations are compared to determine the best emotion trajectory recognition model and its optimal parameter combination.

[0056] Step 105: Obtain the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

[0057] After the model is trained, the speech to be recognized is input into the emotion track recognition model. The model outputs the corresponding 136-dimensional speech features X, as well as the predicted values ​​of emotional valence Y1, emotional arousal Y2, and emotional stability Y3 in 1-second time windows. The prediction results of each time window are connected in chronological order to obtain the three-dimensional emotion track of the audio, which is used to describe the dynamic changes in emotional state.

[0058] After determining the final emotion trajectory recognition model, the SHAP interpretability analysis method was used to calculate the contribution of each speech biomarker feature to the model recognition results, and the key features were ranked and visualized.

[0059] Through the above process, this application can convert the speech signals of elderly HIV / AIDS patients in standardized tasks into results that can be used for emotion trajectory recognition and digital phenotypic analysis, providing technical support for nursing assessment, follow-up management and subsequent emotion intervention.

[0060] Implementing the method described in this application has the following beneficial effects: (1) The output results can reflect the dynamic changes in emotions.

[0061] Existing solutions often output scale results, psychological symptom classifications, or single emotion categories, making it difficult to reflect the process of emotion change. This application uses emotional valence, emotional arousal, and emotional stability as core dimensions, which can output emotional trajectories that change over time, making it more suitable for describing the dynamic characteristics of individual emotions.

[0062] (2) More suitable for elderly HIV / AIDS patients.

[0063] Existing speech emotion recognition solutions are mostly based on publicly available speech databases or constructed from general populations, lacking task design and quality control specifically for elderly HIV / AIDS patients. This application sets up standardized speech tasks, emotion labeling methods, and data quality control processes for this specific population, improving the model's applicability in clinical nursing scenarios.

[0064] (3) It is more in line with the data security requirements of sensitive groups.

[0065] This application addresses the sensitive nature of HIV / AIDS patients' information, including their disease identity, emotional state, and voice data. It establishes a process for encoding association, encrypted storage, access control, and management of raw voice data, which helps reduce the risk of sensitive information leakage and meets the privacy protection needs in clinical research.

[0066] Based on the same inventive concept, this application also provides a speech biomarker-based emotion trajectory recognition device for implementing the speech biomarker-based emotion trajectory recognition method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more speech biomarker-based emotion trajectory recognition device embodiments provided below can be found in the limitations of the speech biomarker-based emotion trajectory recognition method described above, and will not be repeated here.

[0067] In one exemplary embodiment, an emotion trajectory recognition device based on voice biomarkers is provided, comprising: The speech data acquisition module is used to acquire speech data of subjects performing standardized speech tasks; The feature extraction module is used to extract features from the speech data to obtain a speech feature matrix; The emotion track label matrix acquisition module is used to acquire an emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability; The emotion trajectory recognition model determination module is used to train a multi-output regression model using the speech feature matrix as input variables and the emotion trajectory label matrix as supervised learning labels to obtain the emotion trajectory recognition model. The three-dimensional emotion trajectory prediction module is used to acquire the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

[0068] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 4 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores emotion trajectory recognition data based on speech biomarkers. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an emotion trajectory recognition method based on speech biomarkers.

[0069] Those skilled in the art will understand that Figure 4 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0070] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0071] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0073] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0074] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for emotion trajectory recognition based on speech biomarkers, characterized in that, The emotion trajectory recognition method based on voice biomarkers includes: Acquire speech data of subjects performing standardized speech tasks; The speech data is subjected to feature extraction to obtain a speech feature matrix; Obtain an emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability. Using the speech feature matrix as input variables and the emotion track label matrix as supervised learning labels, a multi-output regression model is trained to obtain an emotion track recognition model. Acquire the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

2. The emotion trajectory recognition method based on speech biomarkers according to claim 1, characterized in that, Feature extraction is performed on the speech data to obtain a speech feature matrix, specifically including: The voice data is subjected to frame-segmentation and windowing processing to obtain multiple short time frames; Extract the short-term acoustic features of each short frame to obtain the first feature vector; Calculate the first-order difference features of the first feature vectors of adjacent short-time frames to obtain the second feature vector; The first feature vector and the second feature vector are concatenated to obtain the frame-level feature vector corresponding to each short frame; Using a preset time length as the intermediate window, the mean and standard deviation of multiple frame-level feature vectors within the intermediate window are calculated dimension by dimension, and the mean and standard deviation are concatenated to obtain the second-level feature vector corresponding to each intermediate window. Arrange the second-level feature vectors of all intermediate windows in chronological order to form the speech feature matrix.

3. The emotion trajectory recognition method based on speech biomarkers according to claim 2, characterized in that, The short-term acoustic features include 34-dimensional features, with the first feature vector being 34-dimensional, the second feature vector being 34-dimensional, and the frame-level feature vector being 68-dimensional; the second-level feature vector formed by concatenating the mean and the standard deviation is 136-dimensional.

4. The emotion trajectory recognition method based on speech biomarkers according to claim 1, characterized in that, Obtaining an emotion track label matrix aligned with the speech data on the time axis includes: Obtain labeled data that continuously describes the emotional valence and emotional arousal when the subject listens back to the voice data; The emotional stability is calculated based on the changes in emotional valence and emotional arousal between adjacent time points; The labeled data and the emotional stability are summarized according to the same time window scale as the speech feature matrix to form a three-dimensional emotion label corresponding to each time window, and arranged in chronological order to form the emotion track label matrix.

5. The emotion trajectory recognition method based on speech biomarkers according to claim 4, characterized in that, The calculation of emotional stability based on the changes in emotional valence and emotional arousal between adjacent time points specifically includes: The following formula can be used to calculate the degree of emotional fluctuation between adjacent time points. : ； in, Indicates the first Emotional valence at a given point in time Indicates the first Emotional arousal at a given point in time; Based on the degree of emotional fluctuation The moving average or cumulative trend of change within the time series is used to calculate the [number]. Emotional stability at various time points , .

6. The emotion trajectory recognition method based on speech biomarkers according to claim 1, characterized in that, The multi-output regression model includes at least one of gradient boosting regression model, extremely random tree regression model, random forest regression model, support vector regression model, convolutional neural network regression model, or long short-term memory network regression model; the output layer of the multi-output regression model is set to three consecutive numerical output nodes, corresponding to the emotional valence, emotional arousal and emotional stability, respectively.

7. The emotion trajectory recognition method based on speech biomarkers according to claim 1, characterized in that, When training the multi-output regression model, multimodal data is also introduced for joint modeling. This multimodal data includes: The speech data is converted into a time-frequency spectrum or a Mel spectrum. A point projection map is formed by projecting emotion markers along the time axis onto a two-dimensional emotion plane. Text features extracted from the language content generated in the standardized speech task.

8. The emotion trajectory recognition method based on speech biomarkers according to claim 1, characterized in that, Speech-digit phenotypes are generated based on interpretability analysis methods, including: The SHAP interpretability analysis method is used to calculate the contribution of each speech biomarker feature in the speech feature matrix to the three-dimensional emotion trajectory prediction result. Key speech features whose contribution exceeds a preset threshold are sorted and visualized; By combining the acoustic meaning and contribution direction of the key speech features with the three-dimensional emotion trajectory prediction results, reverse reasoning is performed to generate the speech digit phenotype of the subject.

9. An emotion trajectory recognition device based on voice biomarkers, characterized in that, The emotion trajectory recognition device based on voice biomarkers includes: The speech data acquisition module is used to acquire speech data of subjects performing standardized speech tasks; The feature extraction module is used to extract features from the speech data to obtain a speech feature matrix; The emotion track label matrix acquisition module is used to acquire an emotion track label matrix aligned with the speech data on the time axis; the emotion track label matrix includes three-dimensional emotion labels corresponding to multiple time windows, and the three-dimensional emotion labels include emotion valence, emotion arousal and emotion stability; The emotion trajectory recognition model determination module is used to train a multi-output regression model using the speech feature matrix as input variables and the emotion trajectory label matrix as supervised learning labels to obtain the emotion trajectory recognition model. The three-dimensional emotion trajectory prediction module is used to acquire the speech data to be recognized, input it into the trained emotion trajectory recognition model, output the corresponding three-dimensional emotion trajectory prediction result, and generate a speech digit phenotype based on the interpretability analysis method.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the emotion trajectory recognition method based on speech biomarkers according to any one of claims 1-8.

Images