Voice-based psychological health assessment multi-mode system and method with emotion stimulation

Abstract

The psychological health assessment multi-mode system with emotion stimulation and based on voice comprises a task construction module which is used for constructing a task for capturing acoustics, language and emotion characteristics of user voice; the stimulus output module comprises stimulus based on the constructed task, the stimulus is presented to the user to trigger one or more types of user behaviors, and the triggering adopts an input response form; the response receiving module presents the stimuli to a user and receives corresponding responses in one or more formats in response; the automatic encoder is used for defining the relation between an audio mode and a text mode by using the fusion features, the audio mode outputs extracted advanced text features, and the text mode outputs the extracted advanced audio features; the autoencoder receives the extracted advanced textual features and advanced audio features in parallel to output a shared performance feature dataset for sentiment classification, the sentiment classification relating to the mental health assessment.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, machine learning, computing networks and computer engineering.

[0002] Specifically, the present invention relates to a multimodal system and method for mental health assessment based on speech using induced emotions and emotion recognition technology. Background Art

[0003] It is reported that the incidence of mental illness has increased significantly worldwide due to the impact of the COVID-19 pandemic, with 10-15% of the population facing mental illness. The current growth rate has a significant impact on the economy compared to cancer, cardiovascular disease, diabetes and respiratory diseases. Suicide due to mental health problems is currently the second leading cause of death among people aged 15 to 29, causing huge social disruption and productivity losses. In response to these worrying situations, the World Health Organization announced in 2016 that depression is the leading cause of disability worldwide.

[0004] However, as the severity of the disease continues to increase, the traditional healthcare system can no longer cope with the influx of patients with mental illness. Early detection and early intervention techniques can greatly improve the cure rate of patients, thereby reducing the economic burden and improving patients' productivity and quality of life. Automated screening and continuous monitoring have been proven to be an effective screening alternative.

[0005] Voice-based artificial intelligence (AI) screening technology is gaining popularity among users for detecting various mental health disorders, including depression and anxiety. Because it can be easily accessed through digital devices (such as mobile phones), it is seen as an inexpensive, scalable solution. Using voice as a "biomarker," these technologies can also be more effective in resisting dishonesty or cheating during assessment or monitoring than traditional outpatient screening mechanisms such as PHQ-9 (Patient Health Questionnaire 9) or GAD-7 (Generalized Anxiety Disorder 7).

[0006] Voice-based artificial intelligence (AI) screening systems and methods typically rely on scripted conversations and collect audio-based answers (and possibly other signals as well). Based on the answers, it can use a model built on acoustic features or acoustic plus NLP (natural language processing) features to produce a final classification, which is well documented in the prior art. To avoid using useless answers as input, certain statistical-based techniques can be used to test each answer before moving on to the next step. A simple solution is to measure the total time to record the answer, while more complex solutions involve voice activation or using ASR (Automatic Speech Recognition) to check if there are sentences that include enough words. If the preset threshold is not met, the user may be re-prompted with the same question (or a different question) and the user must answer again.

[0007] This method of the prior art has the following defects:

[0008] First, longer answers have a natural bias that is not always correct. A relatively short answer may truly reflect the user's mental health, but on the other hand, a longer answer (containing more words) may lead to a wrong assessment when the user is dishonest.

[0009] Secondly, the ASR (automatic speech recognition) technology used in such systems and methods usually ignores certain speech behaviors, such as pauses, hesitations, fundamental frequency perturbations, and amplitude perturbations, and only generates sentences / words that it can recognize. Filtering based on the output of such ASR technology may miss some answers that are meaningful to the screening model.

[0010] Finally, the user's emotional state when answering questions may affect the mental health assessment results. However, existing answer verification techniques ignore emotions as input, which is also insufficient.

[0011] Therefore, a better mechanism is needed to help ensure relatively higher input quality and thus improve model performance.

[0012] In addition, in recent years, the use of smartphone technology and wearable devices to monitor physiological and psychological data non-invasively and continuously has attracted the interest of researchers. Advances in acoustic and speech processing technology have opened up new areas for behavioral health diagnosis using machine learning. Studies have found that patients with depression show reduced stress, monotonous pitch, and loudness attenuation in language, which is consistent with clinical observations of patients with depression. These patients speak slower, with more pauses and lower volume than ordinary people. Studies have shown that speech biomarkers identified by machine learning models are positively correlated and have greater potential in detecting mental disorders such as depression. Researchers have devoted a lot of effort to studying depression and its correlation with acoustic and semantic aspects of speech.

[0013] AI technologies in this field can be divided into three main categories:

[0014] 1. Semantic-based: The speech data is converted into transcribed text by applying automatic speech recognition (ASR), and then natural language processing (NLP) is performed on it to build a natural language-based classification model.

[0015] 2. Based on acoustics: Extract acoustic features directly from speech, and then build a classification model based on these features. These features can be manually designed features (such as rhythm features / spectrum-based related features) or potential features embedded by pre-trained models.

[0016] 3. Multimodal based: Some people have also tried to combine these two modes to create multimodal artificial intelligence models to improve the accuracy of the evaluation.

[0017] These technologies and related studies often use users' voice recordings on a certain topic as model inputs, such as answers to certain fixed questions, such as sleep and rest time, or descriptions of health conditions. On the other hand, it is worth noting that major depressive disorder (MDD) often involves emotional numbness, especially numbness to sadness, fear, anger, and shame. Many reviews in the past have shown that patients with depression differ from non-depressed patients, especially in their emotional preferences and the use of emotional regulation strategies. Unfortunately, existing voice-based AI technologies, from training data collection to model training to reasoning when the model is applied in practice, do not pay enough attention to users' emotional preferences or emotional regulation abilities.

[0018] According to previous studies, it has become a well-known fact that human voices convey a lot of emotional information. For example, some existing technical literature found that people can detect not only basic emotional tones from voices (e.g., positive emotions vs. negative emotions, or excitement vs. calmness), but also subtle emotional nuances. Actively covering more emotional effects or better utilizing emotional information in voices (especially valence) may improve the accuracy of such AI models.

[0019] This application discusses systems and methods for building such technology.

[0020] Purpose of the Invention

[0021] The purpose of the present invention is to assess mental health based on voice biomarkers combined with emotion judgment / recognition.

[0022] Another object of the present invention is to utilize induced emotions and emotion recognition to eliminate bias in assessing mental health based on voice biomarkers.

[0023] Another object of the present invention is to provide an enhanced filtering technique while assessing mental health based on speech-based biomarkers.

[0024] Another object of the present invention is to ensure that speech signals have relatively high input quality while assessing mental health based on speech-based biomarkers.

[0025] Another object of the present invention is to enhance the understanding of language behavior while assessing mental health based on speech-based biomarkers. Summary of the invention

[0026] According to the present invention, a multimodal system and method for mental health assessment based on speech with emotional stimulation is provided.

[0027] According to the present invention, a multimodal system for mental health assessment based on speech with emotional stimulation is provided, the system comprising:

[0028] A task building module for building tasks that capture acoustic, linguistic, and emotional features in user speech;

[0029] a stimulus output module, for receiving data from the task construction module, the stimulus output module including one or more stimuli based on the constructed task, presenting the stimuli to the user to trigger one or more types of user behavior, the trigger taking the form of an input response;

[0030] a response receiving module, for presenting the one or more stimuli from the stimulus output module to the user according to the constructed task, and receiving corresponding responses in one or more formats as responses;

[0031] Feature modules, including:

[0032] A feature constructor, for defining features for each of the constructed tasks, wherein the features are defined according to a learnable heuristic weighted task;

[0033] a feature extractor for extracting one or more defined features from the received corresponding responses, the corresponding responses being related to the constructed tasks, wherein the learnable heuristic weighted model considers at least one of the ranked constructed tasks;

[0034] A feature fusion module, used to fuse two or more defined features to obtain a fused feature;

[0035] The autoencoder uses the fused features to define the relationship between:

[0036] an audio modality of the feature fusion module, the audio modality working in conjunction with the response receiving module to extract high-level features in the response to output extracted high-level text features; and

[0037] a text modality of the feature fusion module, the text modality working in conjunction with the response receiving module to extract high-level features in the response to output extracted high-level audio features;

[0038] The autoencoder is used to receive high-level text features and high-level audio features extracted from the audio modality and the text modality in parallel to output a shared expressive feature dataset for emotion classification, wherein the emotion classification is related to the mental health assessment.

[0039] In at least one embodiment of the system, the constructed task is a pronunciation task and / or a writing task.

[0040] In at least one embodiment of the system, the task construction module includes a first-order sorting module for sorting the constructed tasks in order of difficulty; thereby assigning a first-order weight to each constructed task.

[0041] In at least one embodiment of the system, the task construction module includes a first-order sorting module for sorting the constructed tasks in order of difficulty; thereby assigning a first-order weight to each constructed task, wherein the constructed tasks are valence levels ordered by stimulus markers, the valence levels are related to the analyzed responses, and the valence levels are selected from the group consisting of positive valence, negative valence, and neutral valence.

[0042] In at least one embodiment of the system, the task construction module includes a second-order sorting module for sorting the complexity of the constructed tasks in order of complexity; thereby assigning a second-order weight to each constructed task.

[0043] In at least one embodiment of the system, the task construction module includes a second order sorting module for sorting the complexity of the constructed tasks and emotional expectations in the form of response vectors, and sorting the complexity of the constructed tasks to create a data collection pool.

[0044] In at least one embodiment of the system, the constructed task is selected from the group consisting of: a cognitive task of counting within a predetermined duration, a task related to the pronunciation of vowels within a predetermined duration, a task of saying words with voiced and unvoiced components within a predetermined duration, a task of reading words within a predetermined duration, a task of reading paragraphs within a predetermined duration, a task related to reading paragraphs with phonemic and emotional complexity, a task of open-ended questions with emotional changes, and a predetermined open task within a predetermined duration.

[0045] In at least one embodiment of the system, the constructed task includes one or more questions as stimuli, assigning a question embedding of 0-N vectors to each question, training a question-specific feature extractor by determining the extraction of the embeddings, and forcing alignment of the extracted embeddings from the questions for mid-level feature fusion, wherein the extracted embeddings are related to word embeddings, phoneme embeddings, and syllable-level embeddings.

[0046] In at least one embodiment of the system, the one or more stimuli are selected from the group of stimuli consisting of audio stimuli, video stimuli, a combination of audio and video stimuli, textual stimuli, multimedia stimuli, physiological stimuli, and combinations thereof.

[0047] In at least one embodiment of the system, the one or more stimuli include a stimulus vector calibrated to elicit a text response vector and / or an audio response vector and / or a video response vector and / or a multimedia response vector and / or a physiological response vector in response to the stimulus vector.

[0048] In at least one embodiment of the system, the one or more stimuli are parsed by a first vector engine configured to determine component vectors of the stimuli in order to determine weighted basis states associated with the stimulus vectors.

[0049] In at least one embodiment of the system, the one or more responses are selected from a group of responses consisting of an audio response, a video response, a combination of audio and video responses, a textual response, a multimedia response, a physiological response, and combinations thereof.

[0050] In at least one embodiment of the system, the one or more stimuli include a response vector, which is associated with an induced audio response vector and / or an induced video response vector, and the induced audio response vector and / or the induced video response vector are responsive to the stimulus vector of the stimulus output module.

[0051] In at least one embodiment of the system, the one or more stimuli are parsed by a first vector engine configured to determine component vectors of the stimuli in order to determine weighted basis states associated with the stimulus vectors.

[0052] In at least one embodiment of the system, the response receiving module includes a paragraph reading module, and the paragraph reading module is used to allow the user to perform tasks related to reading paragraphs within a predetermined time.

[0053] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for analyzing audio responses.

[0054] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalist Acoustic Parameter Set (GeMAPS), the feature constructor:

[0055] A set of 62 parameters is used to analyze speech;

[0056] providing a 3-frame long symmetrical moving average filter for smoothing over time, wherein the smoothing is performed within the voiced region of the response and is used to process pitch, fundamental frequency perturbations and amplitude perturbations;

[0057] The arithmetic mean and coefficient of variation are applied as functions to 18 low-level descriptors (LLDs), resulting in 36 parameters;

[0058] Apply 8 functions to loudness;

[0059] Apply 8 functions to pitch;

[0060] Determine the arithmetic mean of the Alpha Ratio;

[0061] Determine the Hammarberg index;

[0062] The spectral characteristics are determined by the spectral slopes of all unvoiced segments in the range of 0-500 Hz and 500-1500 Hz;

[0063] determining temporal characteristics of continuous voiced regions and unvoiced regions based on the responses; and

[0064] Determines Viterbi-based F0 contour smoothing to prevent single tone frame loss due to errors.

[0065] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalist Acoustic Parameter Set (GeMAPS), the feature constructor is configured with a set of low-level descriptors (LLDs) for analyzing spectral, pitch and temporal characteristics of the response, the response being an audio response, the features being selected from the group consisting of:

[0066] Mel-Frequency Cepstral Coefficients (MFCCs) and their first and second order derivatives;

[0067] pitch and pitch changes;

[0068] Energy and energy entropy;

[0069] Spectral centroid, spectral extension and spectral flatness;

[0070] Spectral slope;

[0071] Spectrum roll-off;

[0072] Spectrum changes;

[0073] Zero crossing rate;

[0074] Amplitude perturbation, fundamental frequency perturbation and harmonic noise ratio;

[0075] Speech probability (based on pitch); and

[0076] Temporal characteristics, such as the rate of loudness peaks and the mean length and standard deviation of continuous voiced and unvoiced regions.

[0077] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalist Acoustic Parameter Set (GeMAPS), the feature constructor being configured with a set of frequency-dependent parameters selected from the following group of frequency-dependent parameters:

[0078] Pitch, logarithmic F0 on a semitone frequency scale, starting at 27.5 Hz (semitone 0);

[0079] Fundamental frequency perturbation, deviation in the length of each consecutive F0 period;

[0080] Formant 1, 2, 3 frequencies, the center frequencies of the first, second, and third formants;

[0081] Formant 1, bandwidth of the first formant;

[0082] Energy related parameters;

[0083] Amplitude related parameters;

[0084] Amplitude perturbation, the difference in peak amplitude between consecutive F0 cycles;

[0085] Loudness, estimating perceived signal strength from the auditory spectrum;

[0086] Harmonic-to-Noise Ratio (HNR), the relationship between the energy of the harmonic component and the energy of the noise-like component;

[0087] Spectral (balance) parameters;

[0088] Alpha ratio, the ratio of the total energy of 50-1000Hz to 1-5kHz;

[0089] Hammarberg Index, the ratio of the strongest energy peak in the 0-2kHz region to the strongest energy peak in the 2-5kHz region;

[0090] Spectral slope 0-500Hz and 500-1500Hz, the linear regression slope of the logarithmic power spectrum in two given frequency bands;

[0091] The relative energies of formants 1, 2, and 3, and the ratio of the spectral harmonic peak energies at the center frequencies of the first, second, and third formants to the spectral peak energy at F0;

[0092] Harmonic difference H1-H2, the ratio of the energy of the first F0 harmonic (H1) to the energy of the second F0 harmonic (H2);

[0093] as well as

[0094] Harmonic difference H1-A3, the energy of the first F0 harmonic (H1) and the energy of the highest harmonic in the third formant range (A3)

[0095] Ratio.

[0096] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalist Acoustic Parameter Set (GeMAPS), and the feature constructor uses a higher order spectra (HOSA) function, which is a function of two or more component frequencies that realizes a dual spectrum frequency.

[0097] In at least one embodiment of the system, the feature constructor is a feature constructor based on the Geneva Minimalist Acoustic Parameter Set (GeMAPS), which uses a higher-order spectrum (HOSA) function, which is a function of two or more component frequencies that implements a bi-spectrum frequency, wherein the bi-spectrum uses a third-order cumulant to analyze the relationship between frequency components in a signal to check for nonlinear signals, which are associated with the response.

[0098] In at least one embodiment of the system, the feature extractor uses mid-level fusion to fuse higher-level feature embeddings.

[0099] In at least one embodiment of the system, the feature extractor comprises a dedicated language feature extractor, and each stimulus has a learnable weight associated with the language task.

[0100] In at least one embodiment of the system, the feature extractor comprises a dedicated emotion feature extractor, with each stimulus having a learnable weight associated with an emotion task.

[0101] In at least one embodiment of the system, the feature fusion module includes an audio module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses.

[0102] In at least one embodiment of the system, the feature fusion module includes an audio module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses, wherein the audio modality uses a local question-specific feature extractor to extract high-level features from time-frequency domain relationships to output extracted high-level audio features, and the time-frequency domain relationships are time-frequency domain relationships in the responses.

[0103] In at least one embodiment of the system, the feature fusion module includes a text module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify sentiment from one or more responses.

[0104] In at least one embodiment of the system, the feature fusion module includes a text module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses, wherein the text modality uses a bidirectional long short-term memory network with an attention mechanism to simulate intra-modality dynamics to output extracted high-level text features.

[0105] In at least one embodiment of the system, the feature fusion module includes an audio module:

[0106] Use the extracted features in the acoustic feature embedding of the pre-trained model;

[0107] comparing the extracted features in a spectral domain;

[0108] Determine vocal tract coordination characteristics;

[0109] Determine the characteristics of the cycle quantitative analysis;

[0110] bigram count features and bigram duration features relative to speech markers; and

[0111] The features are fused in an autoencoder.

[0112] In at least one embodiment of the system, the feature extractor includes a speech landmark extractor for determining an event marker associated with the response, wherein the determination of the event marker is related to the temporal position of a sound event from the response, wherein the determination includes determining a timestamp boundary representing an abrupt change in the sound response, regardless of the frame.

[0113] In at least one embodiment of the system, the feature extractor includes a speech marker extractor for determining event markers associated with the response, each of the event markers having a start value and an offset value, the event markers being selected from a marker group, the marker group consisting of a glottal-based marker (g), a periodicity-based marker (p), a sonorant-based marker (s), a fricative-based marker (f), a voiced fricative-based marker (v) and an explosive-based marker (v), each of the markers being used to determine a point in time at which a different sudden articulation event occurs, associated with rapid changes in power across multiple frequency ranges and multiple time scales.

[0114] In at least one embodiment of the system, the autoencoder is configured with a multi-modal multi-question input fusion architecture, comprising:

[0115] at least one encoder for mapping one or more of the features having task types to a low-dimensional representation, each task being multiplied by the features and a learnable weight encoding matrix based on the task type, the weights being associated with a mental health assessment;

[0116] at least one decoder for mapping said low-dimensional representation to said one or more said features to output a mental health assessment; and

[0117] The autoencoder is trained with a loss function to minimize the reconstruction error between the input task and the decoder output.

[0118] The present invention provides a multimodal method for mental health assessment based on speech with emotional stimulation, the method comprising the following steps:

[0119] Build tasks that capture the acoustic, linguistic, and emotional features of user speech;

[0120] receiving data of a constructed task, the constructed task comprising one or more stimuli, presenting the stimuli to a user to trigger one or more types of user behavior based on the constructed task, the trigger being in the form of an input response;

[0121] presenting one or more stimuli to a user according to the constructed task, and receiving in response corresponding responses in one or more formats;

[0122] For each constructed task, defining features, wherein the features are defined according to a learnable heuristic weighted task;

[0123] extracting one or more defined features from the received corresponding responses, the corresponding responses being associated with the constructed tasks, the learnable heuristic weighted model taking into account at least one of the ranked constructed tasks;

[0124] fusing two or more defined features to obtain a fused feature;

[0125] The fused features are used to define the relationship between:

[0126] The audio modality of the feature fusion module works in conjunction with the response to extract high-level features in the response to output extracted high-level text features; and

[0127] The text modality of the feature fusion module cooperates with the response receiving module to extract high-level features in the response to output extracted high-level audio features;

[0128] The step of defining a relationship includes receiving extracted high-level text features and extracted high audio text features from the audio modality and the text modality in parallel to output a shared representation feature dataset of emotion classification, wherein the emotion classification is related to the mental health assessment.

[0129] In at least one embodiment of the method, the constructed task includes one or more questions as stimuli, a question embedding with a 0-N vector is assigned to each question, a question-specific feature extractor is trained by determining the extraction of the embedding, the extracted embeddings from the questions are forced to be aligned for mid-level feature fusion, and the extracted embeddings are related to word embeddings, phoneme embeddings, and syllable-level embeddings.

[0130] In at least one embodiment of the method, the step of defining the relationship includes configuring a multi-modal multi-question input fusion architecture, including:

[0131] Mapping one or more of the features with task types to a low-dimensional representation via an encoder, each task being multiplied by the features and a learnable weight encoding matrix based on the task type, the weights being associated with mental health assessment;

[0132] mapping the low-dimensional representation to the one or more features via a decoder to output a mental health assessment;

[0133] The training is performed through a loss function to minimize the reconstruction error between the input task and the decoder output.

[0134] In at least one embodiment of the method, the step of extracting high-level audio features includes extracting high-level features from time-frequency domain relationships in the response to output extracted high-level audio features.

[0135] In at least one embodiment of the method, the step of extracting high-level text features includes simulating intra-modal dynamics using a bidirectional long short-term memory network with an attention mechanism to output extracted high-level text features. BRIEF DESCRIPTION OF THE DRAWINGS

[0136] The present invention is now disclosed in conjunction with the accompanying drawings, wherein:

[0137] Figure 1 A schematic block diagram of a computing environment is shown;

[0138] Figure 2 A training data collection system for inducing emotions is shown;

[0139] Figure 3 A sample set of emotion-inducing questions is shown, showing a number of tasks / questions designed to elicit emotion-based responses;

[0140] Figure 4 shows the known interactions between speech components, forming useful speech standards;

[0141] Figure 5 A fixed reading paragraph is shown;

[0142] Figure 6 A phoneme graph is shown, based on a passage read by a user, without intonation;

[0143] Figure 7 Another fixed reading paragraph is shown;

[0144] Figure 8 A graph showing the relationship between phonemes and the number of occurrences is the basis for users to read articles;

[0145] Fig. 9 It shows the performance of HOSA function for different psychological states of users;

[0146] Fig.10 Shows the autocorrelation and cross-correlation between the first and second delta MFCCs extracted from a 10-second speech file, and the framework for extracting delayed correlations from an acoustic file reflecting psychomotor retardation;

[0147] Fig.11 shows a schematic block diagram of an autoencoder;

[0148] Figures 12A to 12H A plurality of graphs are shown, each graph for at least one type of corresponding task (question) with an original label selected from "health" or "depression" for different users at different start times, each graph showing phonemes and segments with correlation with model predictions, the regions highlighting portions of the speech file with high activation features, according to a non-limiting exemplary embodiment;

[0149] Fig.13 A flow chart is shown; and

[0150] Fig.14 A high-level flow chart of speech-based emotional stimulation mental health assessment is shown. DETAILED DESCRIPTION

[0151] The present invention provides a multimodal mental health assessment system and method based on speech with emotional stimulation. Mental health issues are closely related to emotions. In determining whether the user input is useful to the system, the user's emotions are as important as the length of the answer, or even more important.

[0152] The present invention may be a system, method and / or computer program product and / or mobile device program / product. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0153] Aspects of the disclosed embodiments may include tangible computer-readable media storing software instructions that, when executed by one or more processors, are capable of performing one or more methods, operations, etc. consistent with the disclosed embodiments. Additionally, the disclosed embodiments may be performed by one or more processors that are special-purpose processors based on software instructions that are programmed with logic and instructions that, when executed, perform one or more operations consistent with the disclosed embodiments.

[0154] In describing the present invention, the following definitions apply throughout (including above).

[0155] "Computer" may refer to one or more devices or one or more systems that can accept structured input, process the structured input according to specified rules, and produce processed results as output. Examples of computers may include: computers; fixed or portable computers; computers with a single processor, multiple processors, or multi-core processors, which can run in parallel or non-parallel; general-purpose computers; supercomputers; mainframes; superminicomputers; minicomputers; workstations; microcomputers; servers; clients; interactive televisions; network equipment; telecommunications equipment with Internet access; hybrid combinations of computers and interactive televisions; portable computers; tablet personal computers (PCs); personal digital assistants (PDAs); portable phones; special-purpose hardware for emulating computers or software, such as digital signal processors (DSPs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), chips, multiple chips, or chipsets; systems on chips (SoCs) or multi-processor systems on chips (MPSoCs); optical computers; quantum computers; biological computers; and devices that can accept data, can process data according to one or more stored software programs, can generate results, and generally can include input, output, storage, arithmetic, logic, and control units.

[0156] "Software" may refer to the specified rules for operating a computer or part of a computer. Examples of software may include: code segments; instructions; applets; precompiled code; compiled code; interpreted code; computer programs; and programming logic.

[0157] "Computer-readable media" may refer to any storage device for storing data that is accessible to a computer. Examples of computer-readable media may include: magnetic hard disks; floppy disks; optical disks, such as CD-ROMs and DVDs; magnetic tapes; memory chips; or other types of media that can store machine-readable instructions.

[0158] A "computer system" may refer to a system having one or more computers, each of which may include a computer-readable medium containing software for operating the computer. Examples of computer systems may include: a distributed computer system for processing information through networked computer systems; two or more computer systems connected together through a network to transmit or receive information between computer systems; and one or more devices or one or more systems that can accept data, can process the data according to one or more stored software programs, can generate results, and generally may include input, output, storage, arithmetic, logic, and control units.

[0159] A "network" may refer to a plurality of computers and related devices that may be connected by communication facilities. A network may involve permanent connections (e.g., cables) or temporary connections (e.g., connections that may be established by telephone or other communication links). A network may also include wired connections (e.g., coaxial cables, twisted pairs, optical fibers, waveguides, etc.) or wireless connections (e.g., radio frequency waveforms, free space optical waveforms, acoustic waveforms, satellite transmissions, etc.). Examples of networks may include: Internet, such as the Internet; intranets; local area networks (LANs); wide area networks (WANs); and combinations of networks, such as the Internet and intranets. Exemplary networks may operate using any of a variety of protocols, such as Internet Protocol (IP), asynchronous transfer mode (ATM) or synchronous optical network (SONET), user datagram protocol (UDP), IEEE 802.x, etc.

[0160] As used herein, the terms "data" and "data item" refer to sequences of bits. Thus, a data item may be the contents of a file, a portion of a file, a page in memory, an object in an object-oriented program, a digital message, a digital scanned image, a portion of a video or audio signal, or any other entity that can be represented by a sequence of bits. The term "data processing" as used herein refers to the processing of a data item, which sometimes depends on the type of data item being processed. For example, a data processor for a digital image may be different from a data processor for an audio signal.

[0161] The words "first", "second", etc. used in this document do not indicate any order, priority, or importance, but are merely used to distinguish one element from another. The words "a" and "an" used below do not indicate a limitation of quantity, but merely indicate the presence of at least one of the mentioned items.

[0162] Figure 1 A networked server (100) and one or more networked client devices (112, 114, 116, 118) interfaced with the networked server (100) via a network, and one or more databases (122, 124, 126, 128) interfaced with the networked server (100) are shown.

[0163] Speaking may seem effortless, but it is actually a complex process that involves the coordination of cognitive and physiological activities in the brain. It requires a complex series of actions to be executed effectively. Normal speech originates from the body's physiological processes, which first generate potential energy in the lungs and then change the air pressure in the vocal tract. When speaking to produce voiced sounds, the lungs release air, and the speed of the air affects the regularity of the vocal cords. As the harmonically rich sound from the glottis travels through the vocal tract and laryngeal flow, the amplitude of the harmonics is affected by the movements of the pharynx, oral and nasal cavities, and the organs of articulation (such as the tongue, teeth, lips, jaw, and soft palate) that act as filters.

[0164] In some people with depression, depression and psychomotor retardation are associated with impaired vocal function, reduced harmonic formant amplitudes, and other abnormalities. This can also cause them to feel "breathy" when speaking due to lack of laryngeal control caused by psychomotor retardation. This is in stark contrast to the speech of healthy people. Intensity has also been shown to be a strong indicator of depression severity. People with depression often have weaker voice intensity and seem to speak in a monotone.

[0165] The researchers studied how people with depression use language in two ways: (a) by listening to recordings of their speech and (b) by analyzing written texts.

[0166] People with depression often have poor verbal skills, such as using inappropriate or unclear words, speaking in incomplete sentences, and excessive repetition.

[0167] Acoustic prosodic elements that are critical for identifying depression may be abnormal, and these elements also interact with linguistic information (such as word and phrase meaning) to express an individual's emotional state.

[0168] In the past, subjective assessments of depressed patients often focused on their speech patterns and related behaviors, based on preconceived notions of how depressed patients display emotions.

[0169] Figure 2 A training data collection system for inducing emotions is described.

[0170] In at least one embodiment, a client device (112, 114, 116, 118) is communicatively coupled to at least one stimulus output module (202) for presenting one or more stimuli to a user of such client device.

[0171] In at least one embodiment, the stimulus output module (202) is used to provide output stimuli to the user to elicit a user response in the form of an input response. The output stimuli may include stimuli such as audio stimuli, video stimuli, a combination of audio and video stimuli, text stimuli, multimedia stimuli, physiological stimuli, and combinations thereof. The stimuli may include stimulus vectors, and the stimulus vectors are calibrated to elicit a text response vector and / or an audio response vector and / or a video response vector and / or a multimedia response vector and / or a physiological response vector in response to the stimulus vector. One or more output stimuli are parsed by a first vector engine (232), which is used to determine component vectors of the output stimuli in order to determine a weighted basis state associated with such stimulus vectors.

[0172] In some embodiments, the stimulus output module (202) is used to send speech-based tasks to target subjects (users) with specific mental illnesses (based on certain criteria, such as DSM-V diagnosis) as well as healthy subjects (users) without mental illnesses. This is accomplished through the networked server (100) and through the user / subject's client device (112, 114, 116, 118). In a preferred embodiment, these stimuli are vector-configured stimuli to induce certain emotional responses in the subject, including but not limited to:

[0173] happiness emotional response;

[0174] Sad emotional response;

[0175] Neutral emotional response.

[0176] In some embodiments, the stimulus output module (202) includes stimuli that trigger certain user / subject behaviors; for example, imperative questions that force the user to repeat a fixed number of words or vowels.

[0177] Embodiments of the present invention may include a task building module for building tasks that capture acoustic, linguistic, and emotional features of a user's speech. Data from the task building module is provided to a stimulus output module (202). Preferably, these tasks are pronunciation tasks, since the present invention deals with speech-based biomarkers.

[0178] In at least one embodiment, the task construction module includes a first sorting module for sorting the constructed tasks by difficulty, thereby assigning a weight to each task. In a preferred embodiment, in order to better capture the user's emotional preferences and the use of psychologically relevant emotion regulation strategies, it is necessary to actively stimulate different valence (sorting) levels (positive, negative and neutral) of the user (speaker) when recording the user's (speaker's) response.

[0179] In at least one embodiment, the task building module includes a second ranking module for ranking the complexity of the questions and the complexity of the emotional expectations of the ranked questions to create a data collection pool.

[0180] In multi-class classification, depression levels have an ordinal relationship; therefore, the expression tasks from the task building module for different depression levels also have learnable weights to optimize the loss for each problem and each depression level.

[0181] Figure 3An exemplary set of induced emotion tasks (question sets) is presented, in which some voice tasks are presented to elicit emotion-based responses. The following is the emotion elicitation pattern for each voice task in at least one embodiment. Well-designed tasks can not only cover different emotions well, but also have good coverage at the phoneme level of the target language.

[0182]

[0183]

[0184] In at least one embodiment, the client device (112, 114, 116, 118) is communicatively coupled to at least one response receiving module (204) for presenting one or more stimuli from the stimulus output module (202) to a user and, in response, receiving corresponding input in one or more formats.

[0185] At least one embodiment of the response receiving module (204) includes a receiving module (204a) for capturing an input response of a user in response to an output stimulus. The input response may include an audio input, a video input, a combination of audio and video inputs, a multimedia input, and the like. The input response may include a response vector, the response vector being associated with an induced audio response vector and / or an induced video response vector in response to a stimulus vector of the stimulus output module (202).

[0186] Embodiments of the present invention may include a data preprocessing module, and in some embodiments of the task construction module, question construction may include constructing questions from simple cognitive tasks (such as counting, pronouncing vowels, pronouncing words with voiced and unvoiced components, reading paragraphs with phonemic and emotional complexity) to open questions with emotional changes, and the generation and response of cognitive sentences can be collected by the response receiving module (204).

[0187] Embodiments of the receiving module (204a) may also include constructing and evaluating responses, recording answers, and processing response vectors until boundary conditions are met as qualified data points for training. The response vectors may also include flash cards for data collection specific tasks, metadata for collecting user records, and task completion and qualification details.

[0188] An embodiment of the response receiving module (204) may include a first measurement module for analyzing, measuring and outputting at least the following as a set of first measurement data:

[0189] measured content comprehension relative to the response vector;

[0190] the correctness of the measured response relative to the response vector;

[0191] The measured acoustic signal quality relative to the response vector.

[0192] An embodiment of the response receiving module (204) may include a second measurement module for analyzing, measuring and outputting at least the following as a set of second measurement data:

[0193] the measured silent portion from the response vector;

[0194] the measured signal-to-noise ratio from the response vector;

[0195] measured speech clarity from the response vector;

[0196] The measured vitality quotient from the response vector.

[0197] According to a preset threshold, and the second measurement data.

[0198] In a preferred embodiment, the system and method of the present invention uses segments of response vectors having a measured signal-to-noise ratio greater than 15 as qualified speech samples.

[0199] In at least one embodiment of the receiving module (204a), at least one sound receiving module (204a.1) is provided for capturing an acoustic input response of a user, wherein the acoustic input response is a response to one or more output stimuli in the form of a response acoustic signal. The response acoustic signal includes a response acoustic vector, and the response acoustic vector is correlated with a stimulus vector of a stimulus output module (202) through a second vector engine (234) to determine a first state of the user, wherein the first state is correlated with a stimulus vector of a stimulus output of the stimulus output module (202).

[0200] In at least one embodiment of the receiving module (204a), at least one text receiving module (204a.2) is provided, which is communicatively coupled to the voice receiving module (204a.1) so as to transcribe the user voice input response through the transcription engine to provide a response text signal, wherein the user voice input response is obtained from the voice receiving module (204a.1). The response text signal includes a response text vector, which is correlated with the stimulus vector of the stimulus output module (202) through the third vector engine (236) to determine the component response text vector, thereby determining the second state of the user, wherein the second state is correlated with the stimulus vector of the stimulus output module (202).

[0201] In at least one embodiment of the response receiving module (204), a physiological receiving module (204b) is provided for sensing one or more physiological response signals of the user through one or more physiological sensors, wherein the physiological response signals are generated in response to the output stimulation of the stimulation output module (202). The physiological response signals may include physiological vectors related to the physiological signals output in response to the stimulation output of the stimulation output module (202). These vectors are parsed by a fourth vector engine (238) to determine a third state of the user, wherein the third state is related to the stimulation vector output by the stimulation output module (202).

[0202] In at least one embodiment of the response receiving module (204), a neural receiving module is provided for sensing one or more neural response signals of a user through one or more neural sensors, the neural response signals being generated in response to the output stimulation of the stimulation output module (202). The neural response signals may include neural vectors, the neural vectors being associated with the neural signals, the neural signals being responsive to the stimulation output of the stimulation output module (202). These vectors are parsed by a fifth vector engine to determine a fourth state of the user, the fourth state being associated with the stimulation vector of the stimulation output of the stimulation output module (202).

[0203] In an embodiment, the vector engine forms a sound biomarker recognition engine. Generally, the sound biomarker recognition engine adopts a 3-step approach.

[0204] Lead knowledge to construct voice data collection protocols to enhance the presence and stability of vocal biomarkers associated with depression;

[0205] Identify a set of features that best capture subtle auditory differences between depressed and non-depressed individuals; these features need to have good detection and robustness to noise — so they can be used to detect depression in natural settings;

[0206] The presence of vocal biomarkers in speech systems is a non-binary situation; therefore, the use of automated detection systems for people with depression requires the correct task and a predetermined time to perform the task in order for the vocal biomarker to be visible or present.

[0207] The detection time or Task Based Detection Threshold (TBDT) may vary by gender, depression severity, and age. Therefore, it is necessary to perform a procedure to identify the detection threshold and the detection confidence metric score (DCMS) by arranging the speech tasks in an effective order so that the system and method of the present invention can have a stable symptom detection range.

[0208] Embodiments of the present invention may include a protocol construction engine.

[0209] Automatic speech processing using machine learning is increasingly being used in digital health and holds promise as a non-invasive telemedicine screening tool. However, we need to understand more about the protocols used to process speech and take measurements to help create new protocols with specific standards.

[0210] Figure 4 The known interactions between speech components are described to form useful speech standards.

[0211] Healthcare clinicians use speech-language assessments to screen, diagnose, and monitor a variety of conditions. During the assessment, clinicians observe the patient's speech skills, including articulation, breathing, phonation, and voice quality, as well as the patient's language abilities, such as grammar, pragmatics, memory, and expression. Abnormal speech and language symptoms are often early indicators of different conditions.

[0212] In at least one embodiment of the response receiving module (204), a paragraph reading module is provided for allowing a user to read a paragraph according to a predefined task / prompt. Advantages of reading voice protocols include ease of use, repeatability, and the ability to provide clear reference points, as well as the limited range of voices used and controlled variation. In addition, they are relatively easy to integrate into digital smart device applications. When selecting a reading voice protocol to analyze a medical condition, factors such as the background of the speaker, the specific disease of concern, and the time and number of samples required need to be considered.

[0213] In at least one embodiment of the paragraph reading module, fixed reading paragraph examples based on emotions are provided.

[0214] Figure 5 A fixed reading paragraph is shown.

[0215] Figure 6 A phoneme graph is shown, based on a passage read by a user, without intonation.

[0216] Figure 7 Another fixed reading passage is illustrated.

[0217] Figure 8 A graph showing the relationship between phonemes and the number of occurrences is the basis for user reading.

[0218] Embodiments of the present invention may include a feature module having a feature constructor for defining features for each of the constructed tasks, wherein the features are defined according to a learnable heuristic weighted task.

[0219] In at least one embodiment of the feature constructor, the Geneva Minimalist Acoustic Parameter Set (GeMAPS) is used. The Geneva Minimalist Standard Parameter Set is a set of 62 parameters for analyzing speech. It uses a 3-frame long symmetric moving average filter to smooth over time, and only smooths pitch, fundamental frequency perturbations, and amplitude perturbations in the voiced region. The arithmetic mean and coefficient of variation are applied as functions to all 18 LLDs, resulting in 36 parameters. In addition, 8 functions are applied to loudness and pitch, and include the arithmetic mean of the Alpha ratio, Hammarberg index, and spectral slopes of 0-500Hz and 500-1500Hz for all unvoiced segments. Temporal features such as the rate of loudness peaks, the average length and standard deviation of continuous voiced and unvoiced regions, and the number of continuous voiced regions per second are also included. No minimum length is imposed on voiced or unvoiced regions, and Viterbi-based F0 contour smoothing prevents single voiced frames from being lost due to errors.

[0220] eGeMAPS (Geneva Minimal Acoustic Parameter Set) is a new member of openSMILE (The Munich Open-Source

[0221] eGeMAPS is a set of features for analyzing speech, audio, and music in the Munich Multimodal Interface for Language and Emotion Recognition (Munich Multimodal Interface for Language and Emotion Recognition Open Source). eGeMAPS is a subset of the larger Geneva Minimalist Acoustic Parameter Set (GeMAPS) designed specifically for emotion recognition tasks. It includes a set of low-level descriptors (LLDs) for analyzing the spectral, pitch, and temporal properties of audio signals. These features include:

[0222] Mel-frequency cepstral coefficients (MFCCs) and their first and second derivatives;

[0223] changes in pitch and pitch;

[0224] Energy and energy entropy;

[0225] Spectral centroid, spectral extension and spectral flatness;

[0226] Spectral slope;

[0227] Spectrum roll-off;

[0228] Spectrum changes;

[0229] Zero crossing rate;

[0230] Amplitude perturbation, fundamental frequency perturbation and harmonic noise ratio;

[0231] Speech probability (based on pitch);

[0232] Temporal characteristics, such as the rate of loudness peaks and the mean length and standard deviation of continuous voiced and unvoiced regions.

[0233] A total of 87 features are included.

[0234] eGeMAPS is designed as a minimal set of features that can adapt to different recording conditions and speaker characteristics and is effective in a variety of emotion recognition tasks.

[0235] Frequency related parameters:

[0236] Pitch, logarithm F0 on the semitone frequency scale, starting at 27.5 Hz (semitone 0);

[0237] Fundamental frequency perturbation, deviation in the length of each consecutive F0 period;

[0238] Formant 1, 2 and 3 frequencies, the center frequencies of the first, second and third formants;

[0239] Formant 1, bandwidth of the first formant;

[0240] Energy / amplitude related parameters;

[0241] Amplitude perturbation, the difference in peak amplitude between consecutive F0 cycles;

[0242] Loudness, estimating perceived signal strength from the auditory spectrum;

[0243] Harmonic-to-Noise Ratio (HNR), the energy of the harmonic components relative to the energy of the noise-like components;

[0244] Spectral (balance) parameters;

[0245] Alpha ratio, the ratio of the total energy of 50-1000Hz and 1-5kHz;

[0246] Hammarberg index, the ratio of the strongest energy peak in the 0-2kHz region to the strongest energy peak in the 2-5kHz region;

[0247] Spectral slope 0-500Hz and 500-1500Hz, the linear regression slope of the logarithmic power spectrum in two given frequency bands;

[0248] The relative energies of formants 1, 2, and 3, and the ratio of the spectral harmonic peak energies at the center frequencies of the first, second, and third formants to the spectral peak energy at F0;

[0249] Harmonic difference H1-H2, the ratio of the energy of the first F0 harmonic (H1) to the energy of the second F0 harmonic (H2);

[0250] Harmonic difference H1-A3, the ratio of the energy of the first F0 harmonic (H1) to the energy of the highest harmonic in the third formant range (A3).

[0251] In at least one embodiment of the feature constructor, a high-order spectrum (HOSA) function is used. The high-order spectrum (HOSA) is a function of two or more component frequencies, in contrast to the power spectrum, which is a function of a single frequency. These spectra can be used to identify phase coupling between Fourier components, and they are particularly useful in detecting and characterizing nonlinearities in systems. To achieve this, the amplitude of the high-order spectrum is normalized by the power of the component frequencies. The normalized high-order spectrum, also known as the n-th order coherence index, is a function that combines the n-th order cumulant spectrum with the power spectrum.

[0252] The bispectrum is a method that uses third-order cumulants to analyze the relationship between frequency components in a signal. It is particularly useful for examining nonlinear signals. The bispectrum is more informative than the power spectrum because it provides information about the phase relationship between frequency components that is not available in the spectral domain. Higher-order statistics are an effective method for studying nonlinear signals because they include the relationship between phase components. The bispectrum is one of the best methods for this purpose because it shows information that is not available in the spectral domain.

[0253] Fig. 9 The performance of the HOSA function for various psychological states of the user is shown.

[0254] Embodiments of the present invention may include a feature module having a feature extractor having a variable / heuristic weighting model that considers selection from the group consisting of a ranking task, a ranking problem, etc. In a preferred embodiment, the weights for each problem / task are different and inherently learnable. Preferably, the feature extractor uses mid-level fusion to fuse higher-level feature embeddings.

[0255] Embodiments of Feature Extractors. When training a task (eg, acoustic questions), each stimulus (eg, question) may be assigned a dedicated feature extractor with learnable weights.

[0256] In a preferred embodiment, a dedicated language feature extractor may be assigned to a language task / stimulus (question), with learnable weights for each stimulus (eg, question).

[0257] In a preferred embodiment, a dedicated emotion feature extractor may be assigned to an emotion task / stimulus (question), with learnable weights for each stimulus (eg, question).

[0258] In a preferred embodiment, the selector is used to select a random sample of response vectors based on preset thresholds of signal-to-noise ratio, response (speech) clarity, or the like.

[0259] Embodiments of the present invention may include a data loader for randomly upsampling or downsampling the response vector until a balanced data batch for training is achieved. When constructing the data loader, the upsampling or downsampling is performed randomly accordingly until a balanced data batch for training is achieved. Embodiments may also include using soft labels rather than hard labels in the systems and methods to solve classification problems due to the ordinal nature of the data.

[0260] Despite attempts to exploit unimodal emotional feature representations, prior art has proven inadequate for recognition due to lack of uniqueness and inability to effectively capture the dynamic interactions between emotions in speech recognition tasks. Therefore, embodiments of the present invention may include a feature module having a feature fusion module, wherein the feature fusion module is used to fuse the various output stimuli and / or user states described above. Typically, the feature fusion module works in conjunction with a response receiving module (204).

[0261] In a preferred embodiment, the feature fusion module includes an audio modality and at least one autoencoder-based feature fusion to classify mental health states from multiple types of response vectors, the audio modality being configured with a high-level feature extractor. In a preferred embodiment, the audio modality extracts high-level features from the time-frequency domain relationships in the response vector using a local question-specific feature extractor. The output here is the extracted high-level audio features.

[0262] In a preferred embodiment, the feature fusion module includes a text modality and at least one autoencoder-based feature fusion to classify emotions from multiple types of response vectors, the text modality being configured with a high-level feature extractor. In a preferred embodiment, the text modality uses a bidirectional long short-term memory network with an attention mechanism to model intra-modality dynamics. The output here is the extracted high-level text features.

[0263] In a preferred embodiment of the audio modality, feature fusion module, the system and method use features from acoustic feature embeddings from pre-trained models (e.g., huBert, Wav2vec, Whisper), where raw audio input is used to extract high-level embeddings from the pre-trained models, which are high-dimensional vector embeddings. Similarly, the system and method of the present invention extract acoustic feature embeddings (from GeMAPS, emobaseegemap), which are compared in the spectral domain. The system and method extract logarithmic Mel spectrograms and MFCCs as well as higher order spectral features (HOSA). In addition, vocal tract coordination features and cyclic quantization analysis features are used to extract psychomotor retardation and neuromuscular activation features. Bigram counts and bigram durations are calculated using speech markers to calculate pronunciation efficacy. All features are fused with an autoencoder, and the nonlinearity of the features remains unchanged when fusing features in the latent feature space.

[0264] Previous studies have shown that the use of lexical information can improve valence estimation performance. Lexical information can be obtained from pre-trained acoustic models, where the learned representations can improve valence estimation of speech. The systems and methods of the present invention study the representations of pre-trained models and the use of task-specific feature extractors (e.g., neuromuscular coordination features for estimating psychomotor retardation) to improve depression biomarker estimation on acoustic speech signals. The systems and methods of the present invention also explore the fusion of representations to improve depression biomarker estimation. Human speech communication is roughly composed of two layers: the linguistic layer, which conveys information in the form of words and their meanings; and the paralinguistic layer, which conveys information through the way these words are expressed, including vocal expression or emotional tone.

[0265] We hypothesize that, given their self-supervised learning architecture and the large speech datasets they are exposed to, the representations generated by these pre-trained models may contain lexical information, thereby facilitating better valence estimation.

[0266] In some embodiments, the systems and methods of the present invention explore a multimodal granularity framework to extract speech embeddings from multiple levels of sub-words. It is known that the extracted embedding-related models are typically frame-level embeddings, and have been shown to be effective in acquiring rich frame-level information. However, it lacks the ability to capture segment-level information that can be used to identify depression biomarkers. Therefore, in addition to frame-level embeddings, the systems and methods of the present invention also introduce segment-level embeddings that are closely related to prosody, including word, phoneme, and syllable-level embeddings. Prosody can convey characteristics of discourse, such as depressive states, because it contains information about the rhythm of speech signals. Therefore, segment-level embeddings may be helpful in identifying multimodal depression biomarkers.

[0267] Using the force alignment method, the temporal boundaries of phonemes can be obtained, which can then be grouped to obtain the boundaries of syllables. Given the force alignment information, the relevant speech segments corresponding to these units can then be extracted.

[0268] Embodiments of the present invention may include a speech landmark extractor.

[0269] Speech landmarks are event markers associated with speech articulation. They rely solely on the position of acoustic events in time, such as consonant closure / release, nasal closure / release, glide minima, and vowel maxima, and can provide information about articulatory events (e.g., vocal fold vibrations). Unlike frame-based processing frameworks, landmark methods are independent of frames and are able to detect timestamp boundaries that represent abrupt changes in speech articulation. This approach offers an alternative to frame-based processing and has the potential to circumvent its shortcomings by focusing on acoustically measurable changes in speech. VTH employs six landmarks, each with a start and end state. They are “g(glottis)”, “p(periodicity)”, “s(sonorant)”, “f(fricative)”, “v(voiced fricative)”, and “b(bursts)”, which are used to specify the time points at which different burst events occur. They are detected by observing evidence of rapid changes (i.e., rises or falls) in power across multiple frequency ranges and multiple time scales. Among these landmarks, “s” and “v” are associated with voiced sounds, while “f” and “b” are associated with unvoiced sounds.

[0270] Embodiments of the present invention may include a soundtrack coordination engine.

[0271] VTC features have been shown to capture psychomotor activity associated with depression, with the most success in two A VEC challenges on depression severity prediction, based on the observation that vocal tract parameters of depressed patients are less “coordinated” (correlated) than those of healthy subjects.

[0272]

[0273] Fig.10 Shown are the autocorrelations and cross-correlations between the 1st and 2nd delta MFCCs extracted from a 10-second speech file, and the framework of delayed correlations extracted from an acoustic file reflecting psychomotor disorders.

[0274] Fig.11 A schematic block diagram of an autoencoder is shown.

[0275] Embodiments of the present invention may include an autoencoder to define the relationship between an audio modality (belonging to a feature fusion module working in conjunction with a response receiving module (204)) and a text modality (belonging to a feature fusion module working in conjunction with a response receiving module (204)), the audio modality and the text modality being in conjunction with a stimulus output module (202). The configuration of the autoencoder is such that once the video audio modality and the text modality complete the above procedure, the extracted high-level text features and the extracted high-level audio features can be input into the autoencoder in parallel to output a shared performance feature dataset for emotion classification. This is unique because the system and method of the present invention not only measures the accuracy of the autoencoder in reconstructing the shared performance feature dataset and minimizing the reconstruction error, but also evaluates the performance of depression detection in the emotion recognition module (300).

[0276] This is different from previous systems and methods, which only focus on learning high-level features from input data using autoencoders. Autoencoders are good at capturing low-level shared representations in a non-linear way; the system and method of the present invention takes advantage of this advantage, so that closed fixed tasks (such as reading paragraphs, counting or phoneme pronunciation tasks) are evaluated with the best local feature extractor, while open tasks (questions) are evaluated with both acoustic and textual modalities while optimizing the loss.

[0277] An autoencoder is a neural network that can be used for feature fusion of audio data. It can achieve multimodal feature fusion by combining data from multiple modalities (such as an audio representation and a textual representation) into a single joint representation. This can be achieved by training an autoencoder to encode the input data into a low-dimensional space and then decode it back to the original space. The encoded representation is a bottleneck or compact feature representation that captures the most important information from the input data.

[0278] A high-level overview of the autoencoder architecture for multimodal feature fusion on audio data:

[0279] 1. Collect and preprocess audio data and other modal data.

[0280] 2. Build the autoencoder architecture using encoder and decoder.

[0281] 3. The encoder takes input data from audio and other modalities and compresses it into a low-dimensional representation (bottleneck or latent representation).

[0282] 4. The decoder uses the bottleneck representation and reconstructs the original input data.

[0283] 5. Train the autoencoder by taking the input data of both modalities as input to the encoder and the original input data as the target output of the decoder.

[0284] 6. After training, the bottleneck representation is used as the fusion feature representation of the two modalities.

[0285] 7. The fused feature representation can be used for further analysis, such as classification or clustering.

[0286] Intermediate fusion strategies exploit prior knowledge to learn the marginal representations of each modality, discover correlations within the modalities, and then use these to learn a joint representation or make predictions directly.

[0287] The intermediate fusion strategy can alleviate the dimensionality imbalance between modalities by forcing edge representations to have similar sizes, but if the imbalance is very large, reducing the dimensionality of the larger modality too much may lead to a large loss of information. However, if the input features of the lower dimensional modality are selected based on prior knowledge, imbalance does not necessarily lead to poor performance.

[0288] Each speech task in our dataset contributes to very specific emotional derivatives, for example, the words in paragraph 1 are mainly composed of positive emotional words, which are combined with neutral emotions, leading the reader to generate positive intentions in the utterances of healthy individuals, while the words in the paragraph or the paragraph itself contains positive, negative and neutral emotions. The utterances capture emotional information associated with acoustic information, which is an indicator for detecting differences in phoneme / emotional information between healthy and patients.

[0289] In order to use all different questions in the same model, the inventors assign question embeddings to the questions, which are 0-N vectors where the question-specific feature extractor has prior knowledge of which word, phoneme, and syllable-level embeddings to extract from the data and enforces alignment for mid-level feature fusion.

[0290] The prior knowledge comes from linguists and speech experts, and their knowledge about specific tasks and task sequences is validated through a model evaluation process that develops integrated gradients on the raw audio signal using SHAP values.

[0291] Fig. 12A A graph is shown for at least one type of task (question) with an original label of “healthy”, showing phonemes and segments that have positive correlations with model predictions, with these areas highlighting portions of the speech file with high activation features, according to a non-limiting exemplary embodiment.

[0292] Fig. 12BA graph showing at least one type of task (problem) with an original label of “frustration” in accordance with a non-limiting exemplary embodiment, the graph showing phonemes and segments that have positive correlations with model predictions, with these areas highlighting portions of the speech file that have high activation characteristics for the second user (at start time 0 seconds).

[0293] Fig. 12C A graph showing at least one type of task (question) with an original label of “health” according to a non-limiting exemplary embodiment shows phonemes and segments that have positive correlations with model predictions, with these areas highlighting portions of the speech file that have high activation features for a third user (at start time 0 seconds).

[0294] Fig.12D A graph showing at least one type of task (problem) with an original label of “frustration” in accordance with a non-limiting exemplary embodiment, the graph displays phonemes and segments that have positive correlations with model predictions, with the regions highlighting portions of the speech file having high activation features for a fourth user (at start time 1 second).

[0295] Fig.12E A graph showing at least one type of task (question) with an original label of “health” according to a non-limiting exemplary embodiment shows phonemes and segments that have positive correlations with model predictions, with these areas highlighting portions of the speech file having high activation features for the fifth user (at start time 0 seconds).

[0296] Fig.12F A graph showing at least one type of task (problem) with an original label of “frustration” in accordance with a non-limiting exemplary embodiment, the graph showing phonemes and segments that have positive correlations with model predictions, with the regions highlighting portions of the speech file having high activation features for the sixth user (at start time 0 seconds).

[0297] Figure 12G A graph showing at least one type of task (question) according to a non-limiting exemplary embodiment, with the original label "health", the graph displays phonemes and segments that have positive correlations with model predictions, with these areas highlighting portions of the speech file having high activation features for the seventh user (at the start time 10 seconds).

[0298] Fig.12H A graph showing at least one type of task (problem) with an original label of “frustration” according to a non-limiting exemplary embodiment, the graph displays phonemes and segments that have positive correlations with model predictions, with the regions highlighting portions of the speech file with high activation features for the eighth user (at start time 0 seconds).

[0299] The mathematical equation of the autoencoder-based multi-modal multi-question input fusion architecture can be expressed as:

[0300] Let X1,X2,X3,...,Xn be n modalities (e.g., audio, text), Q1,Q2,Q3,...,Qm be m questions. Each question will be multiplied by a learnable weight that is an encoding matrix based on the question type.

[0301] The encoder function E maps n modalities and m questions to a low-dimensional representation Z:

[0302] Z=E(X1,X2,X3,…,Xn,Q1,Q2,Q3,…,Qm).

[0303] The decoder function D maps the low-dimensional representation Z back to the original input space:

[0304] X ′ 1,X ′ ,X ′ 3,…,X ′ n=D(Z).

[0305] The autoencoder is trained to minimize the reconstruction error between the original input and the decoder output, typically using a loss function such as mean squared error (MSE) or cross entropy:

[0306] L=1 / nm∑(X ′ i-Xi)^2or L=-1 / nm∑Xi*log(X ′ i).

[0307] The low-dimensional representation Z is a fusion representation of n modalities and m problems, which can be used for further analysis, such as classification or clustering, and the encoder weights will be used as a feature extractor for the downstream task of depression classification.

[0308] The autoencoder-based feature fusion is as follows. First, the hidden representations of the audio input and the corresponding text input are obtained using the HuBERT and BERT models and psychomotor hysteresis features respectively. Since the dataset contains similar answers to a large number of questions, the weights representing the question id are also input into the overall model. The question id weights are used to multiply the text hidden features element by element. The resulting text and audio embeddings are then concatenated with forced alignment, and these concatenated features are used to train the autoencoder model to obtain the fused representation from the bottleneck layer. The learned autoencoder weights until the bottleneck layer are then loaded into the model, and the classification head is connected to the bottleneck layer to train the classification model.

[0309] In some embodiments, subjects (users) respond to data from the stimulus output module (202) by interacting with the response receiving module (204) through their client devices (112, 114, 116, 118) to record their responses.

[0310] In at least one embodiment, the various vector engines (232, 234, 236, 238) are provided as part of a networked server (100) and are further communicatively coupled to an emotion recognition module (300) and an automatic speech recognition module (400).

[0311] In a preferred embodiment, the response at the client device (112, 114, 116, 118) is sampled at a sampling rate of 16KHz, and then sent back to the networked server (100), and then sent from the networked server to the emotion recognition module (300). The emotion recognition module (300) generates multiple categories of emotions through one or more emotion vectors, and interweaves them with one or more stimuli and their confidence. If the emotion is inconsistent with the target induced emotion with a predetermined confidence, the user will be prompted to answer again. In some embodiments, a first predefined stimulus with a first predefined emotion vector is presented to the user through the stimulus output module (202) so that the response and the related response vector are recorded through the response receiving module (204). In some embodiments, a first predefined stimulus with a second predefined emotion vector is presented to the user through the stimulus output module (202) so that the response and the related response vector are recorded through the response receiving module (204). Thus, a particular predefined stimulus may be configured with one or more predefined emotion vectors, and various combinations of stimuli and emotion vectors may be presented to the user via the stimulus output module (202) so that responses and associated response vectors are recorded via the response receiving module (204).

[0312] At the client device (112, 114, 116, 118), the response is sampled at a sampling rate of 16KHz and then sent back to the networked server (100), from which it is sent to the automatic speech recognition module (400). The response data is also transcribed using automatic speech recognition technology.

[0313] In at least one embodiment, the response data, response data transcription, and identified emotions are saved to a server training repository / server training database (122).

[0314] In at least one embodiment, the iteration and prioritization module is used to prioritize the stimuli according to predefined rules applied to the incoming corresponding responses. For example, separate different stimuli from neutral stimuli, put happy before sad, etc. If this occurs, the saved data should reflect the order of the stimulus questions.

[0315] In at least one embodiment, the networked server (100) is communicatively coupled to a multimodal mental health classifier (500) (i.e., the autoencoder), which is an artificial intelligence (AI) module that is trained to classify mental health issues based on voice recordings and text responses.

[0316] Artificial Intelligence (AI) modules can be trained on structured and / or unstructured stimulus and response datasets. Training can be supervised, unsupervised, or a combination of both. Machine Learning (ML) and AI algorithms can be used to learn from various modules. The AI ​​module can query the vector engine to identify responses to stimuli, the order of stimuli, the delay of responses, etc., in order to learn rules about the user's mental health state (defined by confidence scores). Deep learning models, neural networks, deep belief networks, decision trees, genetic algorithms, and other ML and AI models can be used alone or in combination to learn from the solution grid.

[0317] The artificial intelligence (AI) module may be a tandem combination of CNN and LSTM.

[0318] The high-level representation of raw waveforms and short-term temporal variations as well as long-term temporal variations is captured to accurately establish the relationship between the vocal biomarkers and language model of the user (subject) with a specific (depressive) mental health condition. Considering the comparison of deep features with shallow features, the model encodes depression-related temporal cues that occur when the motor cortex is affected.

[0319] As described in the data collection section, data were collected with different stimulus-question orders. The above training process was also repeated on this specialized dataset to ensure that the performance gain brought by inducing emotions was not affected by the leading questions.

[0320] Given the current size of the data, nothing has changed significantly. As more data is collected using different sequences and combinations, the systems and methods of the present invention may one day yield new discoveries. When this happens, it will be easy to compare the results of these different "experiments" and select the best stimulation sequence.

[0321] Fig.13 A flow chart is illustrated.

[0322] Step 1: Provide a stimulus to the user to elicit an acoustic response.

[0323] Step 2: Store the emotion vectors associated with various stimuli.

[0324] Step 3: Record the user's vocal response.

[0325] Step 4: Transcribe and record a text version of the user's recorded voice response.

[0326] Step 4a: Optionally, record the user's physiological response.

[0327] Step 5: Extract and analyze vectors from the user's voice response for emotion recognition (obtained through an emotion detection model that generates multiple categories of emotions) and classify based on the emotion recognition information.

[0328] Step 6: Extract and analyze vectors from the user's text response to perform emotion recognition based on the emotion recognition information.

[0329] Step 6a: Optionally, extract and analyze vectors from the user's physiological response to perform emotion recognition based on the emotion recognition information (obtained by generating an emotion detection model for multiple categories of emotions).

[0330] Step 7: Compare the emotion recognition information from the text response with the emotion signal vector from the training data set to obtain a first space vector distance and a first weighted vector difference, wherein the training data set is related to vectors, emotions, and various stimuli.

[0331] Step 7a.1: Optionally, first analyze the training dataset for certain stimuli and obtain the main emotions associated with the patient's response to that stimulus, and the confidence score threshold associated using a pre-configured percentile.

[0332] Step 7a.2: Record the emotion and confidence score threshold associated with a given stimulus into a database.

[0333] Step 7a.3: At runtime, for each input, get the sentiment category / confidence score and then compare with the data recorded in step 7a.2.

[0334] Step 8: Compare the emotion recognition information from the acoustic response with the emotion signal vector from the training data set to obtain a second space vector distance and a second weighted vector difference, wherein the training data set is related to the vector, related to the emotion, and related to various stimuli.

[0335] Step 8a: Optionally, compare the emotion recognition information from the physiological response with the emotion signal vector from the training data set to obtain a third space vector distance and a third weighted vector difference, wherein the training data set is related to the vector, related to the emotion, and related to various stimuli.

[0336] Step 8b: Optionally, as an alternative to step 7 and step 8, the emotion output is first obtained using a fusion layer using emotion recognition information from different modalities (acoustic, and / or text, and / or physical), and then compared with the emotion signal vector.

[0337] Step 9: Intelligently combine the first space vector distance, the first weighted vector difference, the second space vector distance, and the second weighted vector difference to obtain a first confidence score.

[0338] Step 10: Optionally, intelligently merge the first space vector distance, the first weighted vector difference, the second space vector distance, the second weighted vector difference, the third space vector distance, and the third weighted vector difference to obtain a second confidence score.

[0339] Step 11: Iterate the stimulation (step 1) to obtain further vocal responses and associated transcribed text responses based on the identified emotion, the first confidence score and / or the second confidence score.

[0340] Fig.14 A high-level flow chart of a speech-based mental health assessment with emotional stimuli is shown.

[0341] According to a non-limiting exemplary embodiment, in order to capture the impact of induced emotions (stimuli stimulated by multimedia output) on the performance of the system of the present invention, a model is trained for each induced emotion using a given data set. The performance of various such models is determined by comparison (which can be performed using n-fold cross validation or on a dedicated validation set), and the top three questions (stimuli) with the best performance are selected. A possible alternative is to train a model for all combined data and then run the model against a validation set consisting of answers to a single stimulus. A single measure (e.g., AUROC) is then used to compare the model performance for each stimulus, and the top three stimuli are selected.

[0342] The users respond to these stimuli one by one in their own environment using their respective client devices. For each response received, the system and method of the present invention performs some pre-processing and validation (such as voice activation, loose length check, etc.) and then runs it through the emotion recognition model (200). If the recognized emotion and its confidence score are consistent with the induced emotion, the system and method of the present invention will move on to the next question. Otherwise, the system and method of the present invention will induce a stimulus to prompt the user to respond to the same stimulus again. The process continues. If the user's response attempts exceed certain criteria for the same stimulus, but emotion recognition still fails, a similar alternative stimulus will be requested under the same induced emotion. And if the total number of attempts exceeds a certain threshold, the system and method of the present invention can continue to evaluate and warn the user that the collected data has low confidence.

[0343] The collected responses and records of the corresponding stimuli are sent to a multimodal mental health classifier (500), which evaluates the mental health problems and related risk indexes of the current user. In a preferred embodiment, the classification results of emotions caused by open sadness are prioritized and given higher weights in the final result calculation, and then an assessment report is constructed and sent back to the user and the supplier via real-time voice or text (via a client device) or non-real time (e.g., via email, message, etc.).

[0344] All results of induced emotions (reactions to stimuli) are recorded in the evaluation result memory / evaluation result database (124).

[0345] Table 1 shows the performance metrics of the model on the main induced emotion questions (10-fold average). Using induced emotions and emotion recognition, the system and method of the present invention outperform other multimodal techniques, and improves by at least 10% compared to models with artificially designed features.

[0346] Table 1

[0347]

[0348] The technical advantage of the present invention is to provide a speech-based mental health assessment with an emotion stimulation and monitoring system that utilizes induced emotions as well as emotion recognition to provide better sensitivity / specificity when processing audio input from a user.

[0349] The technical advancement of the present invention is to provide a multimodal architecture of a speech-based mental health assessment system and method with emotional stimulation, involving acoustic features and speech / language features.

[0350] Although this specification has disclosed certain specific embodiments for illustrative purposes, it is obvious to those skilled in the art that all modifications that do not depart from the spirit and scope of the present invention as defined in the following claims are obvious. It should be clearly understood that the foregoing descriptive content should only be interpreted as an illustration of the present invention, not a limitation.

Claims

1. A multimodal system for mental health assessment based on speech with emotional stimulation, the system comprising: A task building module for building tasks that capture the acoustic, linguistic, and emotional features of user speech; a stimulus output module (202) for receiving data from the task construction module, the stimulus output module (202) including one or more stimuli based on the constructed task, presenting the stimuli to the user to trigger one or more types of user behavior, the triggering being in the form of an input response; a response receiving module (204) for presenting the one or more stimuli from the stimulus output module (202) to the user according to the constructed task, and receiving corresponding responses in one or more formats as responses; Feature modules, including: A feature constructor, for defining features for each of the constructed tasks, wherein the features are defined according to a learnable heuristic weighted task; a feature extractor for extracting one or more defined features from the received corresponding responses, the corresponding responses being related to the constructed tasks, wherein the learnable heuristic weighted model considers at least one of the ranked constructed tasks; A feature fusion module, used to fuse two or more defined features to obtain a fused feature; The autoencoder uses the fused features to define the relationship between: an audio modality of the feature fusion module, the audio modality working in conjunction with the response receiving module (204) to extract high-level features in the response to output extracted high-level text features; and a text modality of the feature fusion module, the text modality working in conjunction with the response receiving module (204) to extract high-level features from the response to output extracted high-level audio features; The autoencoder is used to receive high-level text features and high-level audio features extracted from the audio modality and the text modality in parallel to output a shared expressive feature dataset for emotion classification, wherein the emotion classification is related to the mental health assessment.

2. The system according to claim 1, characterized in that The constructed task is a pronunciation task and / or a writing task.

3. The system according to claim 1, characterized in that The task construction module includes a first-order sorting module for sorting the constructed tasks in order of difficulty; thereby assigning a first-order weight to each constructed task.

4. The system according to claim 1, characterized in that The task construction module includes a first-order sorting module for sorting the constructed tasks in order of difficulty; thereby assigning a first-order weight to each constructed task, wherein the constructed tasks are marked by valence levels in order of stimulus, the valence levels are related to the analyzed responses, and the valence levels are selected from the group consisting of positive valence, negative valence and neutral valence.

5. The system according to claim 1, characterized in that The task construction module includes a second-order sorting module for sorting the complexity of the constructed tasks in order of complexity; thereby assigning a second-order weight to each constructed task.

6. The system according to claim 1, characterized in that The task construction module includes a second-order sorting module for sorting the complexity of the constructed tasks and emotional expectations in the form of response vectors, and sorting the complexity of the constructed tasks to create a data collection pool.

7. The system according to claim 1, characterized in that The constructed task is selected from the group consisting of: a cognitive task of counting within a predetermined duration, a task related to the pronunciation of vowels within a predetermined duration, a task of saying words with voiced and unvoiced components within a predetermined duration, a task of reading words within a predetermined duration, a task of reading paragraphs within a predetermined duration, a task related to reading paragraphs with phonemic and emotional complexity, a task of open-ended questions with emotional changes, and a predetermined open task within a predetermined duration.

8. The system according to claim 1, characterized in that The constructed task includes one or more questions as stimuli, assigns a 0-N vector question embedding to each question, trains a question-specific feature extractor by determining the extraction of the embedding, and enforces alignment of the extracted embeddings from the questions for mid-level feature fusion, wherein the extracted embeddings are related to word embeddings, phoneme embeddings, and syllable-level embeddings.

9. The system according to claim 1, characterized in that The one or more stimuli are selected from the group consisting of audio stimuli, video stimuli, a combination of audio and video stimuli, textual stimuli, multimedia stimuli, physiological stimuli, and combinations thereof.

10. The system according to claim 1, characterized in that The one or more stimuli include a stimulus vector calibrated to elicit a text response vector and / or an audio response vector and / or a video response vector and / or a multimedia response vector and / or a physiological response vector in response to the stimulus vector.

11. The system according to claim 1, characterized in that The one or more stimuli are parsed by a first vector engine (232) for determining component vectors of the stimuli in order to determine weighted basis states associated with the stimulus vectors.

12. The system according to claim 1, characterized in that The one or more responses are selected from a group of responses consisting of an audio response, a video response, a combination of an audio and video response, a textual response, a multimedia response, a physiological response, and combinations thereof.

13. The system according to claim 1, characterized in that The one or more stimuli include a response vector associated with an evoked audio response vector and / or an evoked video response vector responsive to the stimulus vector of the stimulus output module (202).

14. The system according to claim 1, characterized in that The one or more stimuli are parsed by a first vector engine (232) for determining component vectors of the stimuli in order to determine weighted basis states associated with the stimulus vectors.

15. The system according to claim 1, characterized in that The response receiving module (204) includes a paragraph reading module for allowing the user to perform tasks related to reading paragraphs within a predetermined time.

16. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva minimalist acoustic parameter set and is used to analyze audio responses.

17. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva minimalist acoustic parameter set. The feature constructor: A set of 62 parameters is used to analyze speech; providing a 3-frame long symmetrical moving average filter for smoothing over time, wherein the smoothing is performed within the voiced region of the response for processing pitch, fundamental frequency perturbations, and amplitude perturbations; The arithmetic mean and coefficient of variation were applied as functions to the 18 low-level descriptors, resulting in 36 parameters; Apply 8 functions to loudness; Apply 8 functions to pitch; Determine the arithmetic mean of the alpha ratios; Determine the Hammarberg Index; The spectral characteristics are determined by the spectral slopes of all unvoiced segments in the range of 0-500 Hz and 500-1500 Hz; determining temporal characteristics of continuous voiced regions and unvoiced regions based on the responses; and Determines Viterbi-based F0 contour smoothing to prevent single-tone frame loss due to errors.

18. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva minimalist acoustic parameter set, the feature constructor is configured with a set of low-level descriptors for analyzing the spectrum, pitch and time characteristics of the response, the response is an audio response, and the features are selected from the group consisting of: Mel-frequency cepstral coefficients and their first and second derivatives; pitch and pitch changes; Energy and energy entropy; Spectral centroid, spectral extension and spectral flatness; Spectral slope; Spectrum roll-off; Spectrum changes; Zero crossing rate; Amplitude perturbation, fundamental frequency perturbation and harmonic noise ratio; Pitch-based speech probability; as well as Temporal characteristics, including the rate of loudness peaks and the mean length and standard deviation of continuous voiced and unvoiced regions.

19. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva minimalist acoustic parameter set, and the feature constructor is configured with a set of frequency-dependent parameters, and the frequency-dependent parameters are selected from the following frequency-dependent parameter group: Pitch, logarithmic F0 on a semitone frequency scale, starting at 27.5 Hz, semitone 0; Fundamental frequency perturbation, deviation in the length of each consecutive F0 period; Formant 1, 2, 3 frequencies, the center frequencies of the first, second, and third formants; Formant 1, bandwidth of the first formant; Energy related parameters; Amplitude related parameters; Amplitude perturbation, the difference in peak amplitude between consecutive F0 cycles; Loudness, estimating perceived signal strength from the auditory spectrum; Harmonic-to-noise ratio, the relationship between the energy of the harmonic components and the energy of the noise-like components; Spectral balance parameters; Alpha ratio, the ratio of the total energy of 50-1000Hz and 1-5kHz; Hammarberg index, the ratio of the strongest energy peak in the 0-2kHz region to the strongest energy peak in the 2-5kHz region; Spectral slope 0-500Hz and 500-1500Hz, the linear regression slope of the logarithmic power spectrum in two given frequency bands; The relative energies of formants 1, 2, and 3, and the ratio of the spectral harmonic peak energies at the center frequencies of the first, second, and third formants to the spectral peak energy at F0; Harmonic difference H1-H2, the ratio of the energy of the first F0 harmonic to the energy of the second F0 harmonic; as well as Harmonic difference H1-A3, the ratio of the energy of the first F0 harmonic to the energy of the highest harmonic in the range of the third formant.

20. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva parsimonious acoustic parameter set, and the feature constructor uses a high-order spectral function, which is a function of two or more component frequencies that realizes a bi-spectral frequency.

21. The system according to claim 1, characterized in that The feature constructor is a feature constructor based on the Geneva minimalist acoustic parameter set, and the feature constructor uses a high-order spectral function, which is a function of two or more component frequencies that realizes a bispectral frequency, wherein the bispectrum uses a third-order cumulant to analyze the relationship between frequency components in the signal to check for nonlinear signals, which are related to the response.

22. The system according to claim 1, characterized in that The feature extractor uses mid-level fusion to fuse higher-level feature embeddings.

23. The system according to claim 1, characterized in that The feature extractor includes a dedicated language feature extractor with learnable weights for each stimulus, associated with the language task.

24. The system according to claim 1, characterized in that The feature extractor includes a dedicated emotion feature extractor with learnable weights for each stimulus, associated with the emotion task.

25. The system according to claim 1, characterized in that The feature fusion module includes an audio module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses.

26. The system according to claim 1, characterized in that The feature fusion module includes an audio module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses, wherein the audio modality uses a local question-specific feature extractor to extract high-level features from time-frequency domain relationships to output extracted high-level audio features, and the time-frequency domain relationships are time-frequency domain relationships in the responses.

27. The system according to claim 1, characterized in that The feature fusion module includes a text module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses.

28. The system according to claim 1, characterized in that The feature fusion module includes a text module configured with a high-level feature extractor and at least one autoencoder-based feature fusion to classify emotions from one or more responses, wherein the text modality uses a bidirectional long short-term memory network with an attention mechanism to simulate intra-modality dynamics to output extracted high-level text features.

29. The system according to claim 1, characterized in that The feature fusion module includes an audio module: Use the extracted features in the acoustic feature embedding of the pre-trained model; comparing the extracted features in a spectral domain; Determine vocal tract coordination characteristics; Determine the cycle quantitative analysis characteristics; bigram count features and bigram duration features relative to speech markers; and The features are fused in an autoencoder.

30. The system according to claim 1, characterized in that The feature extractor comprises a speech landmark extractor for determining event markers associated with the response, wherein the determination of the event markers is related to the temporal position of a sound event originating from the response, wherein the determination comprises determining timestamp boundaries representing abrupt changes in the sound response, regardless of the frame.

31. The system according to claim 1, characterized in that The feature extractor comprises a speech marker extractor for determining event markers associated with the response, each of the event markers having a start value and an offset value, the event markers being selected from a marker group consisting of a glottal-based marker, a periodicity-based marker, a sonorant-based marker, a fricative-based marker, a voiced fricative-based marker and an explosive-based marker, each marker being used to determine a point in time at which a different sudden articulatory event occurs, associated with rapid changes in power across multiple frequency ranges and multiple time scales.

32. The system according to claim 1, characterized in that The autoencoder configuration has a multi-modal multi-question input fusion architecture, including: at least one encoder for mapping one or more of the features having task types to a low-dimensional representation, each task being multiplied by the features and a learnable weight encoding matrix based on the task type, the weights being associated with a mental health assessment; at least one decoder for mapping said low-dimensional representation to said one or more said features to output a mental health assessment; and The autoencoder is trained with a loss function to minimize the reconstruction error between the input task and the decoder output.

33. A multimodal method for mental health assessment based on speech with emotional stimulation, the method comprising the following steps: Build tasks that capture the acoustic, linguistic, and emotional features of user speech; receiving data of a constructed task, the constructed task comprising one or more stimuli, presenting the stimuli to a user to trigger one or more types of user behavior based on the constructed task, the trigger being in the form of an input response; presenting one or more stimuli to a user according to the constructed task, and receiving in response corresponding responses in one or more formats; For each constructed task, defining features, wherein the features are defined according to a learnable heuristic weighted task; extracting one or more defined features from the received corresponding responses, the corresponding responses being associated with the constructed tasks, the learnable heuristic weighted model taking into account at least one of the ranked constructed tasks; fusing two or more defined features to obtain a fused feature; The fused features are used to define the relationship between: The audio modality of the feature fusion module works in conjunction with the response to extract high-level features in the response to output extracted high-level text features; and The text modality of the feature fusion module cooperates with the response receiving module to extract high-level features in the response to output extracted high-level audio features; The step of defining a relationship includes receiving extracted high-level text features and extracted high audio text features from the audio modality and the text modality in parallel to output a shared representation feature dataset of emotion classification, wherein the emotion classification is related to the mental health assessment.

34. The method according to claim 33, characterized in that The constructed task includes one or more questions as stimuli, assigns a question embedding with a 0-N vector to each question, trains a question-specific feature extractor by determining the extraction of the embedding, and enforces alignment of the extracted embeddings from the question for mid-level feature fusion, wherein the extracted embeddings are related to word embeddings, phoneme embeddings, and syllable-level embeddings.

35. The method according to claim 33, characterized in that The step of defining the relationship includes configuring a multi-modal multi-question input fusion architecture, including: Mapping one or more of the features with task types to a low-dimensional representation via an encoder, each task being multiplied by the features and a learnable weight encoding matrix based on the task type, the weights being associated with mental health assessment; mapping the low-dimensional representation to the one or more features via a decoder to output a mental health assessment; The training is performed through a loss function to minimize the reconstruction error between the input task and the decoder output.

36. The method according to claim 33, characterized in that The step of extracting high-level audio features includes extracting high-level features from the time-frequency domain relationship in the response to output extracted high-level audio features.

37. The method according to claim 33, characterized in that The step of extracting high-level text features includes simulating intra-modal dynamics using a bidirectional long short-term memory network with an attention mechanism to output extracted high-level text features.

Images