A multimodal system for voice-based mental health assessment with emotional stimuli and its uses

Abstract

A multimodal system for speech-based mental health assessment with emotional stimuli, comprising: a task construction module for constructing a task to capture acoustic, linguistic, and emotional characteristics of a user's speech; a stimulus output module including stimuli based on said constructed task that are presented to a user to elicit triggers for one or more types of user actions, said triggers being in the form of input answers; an answer receiving module for presenting the stimuli to the user and, in response, receiving corresponding answers in one or more forms; and an autoencoder for defining a relationship between a speech modality that outputs extracted high-level text features and a text modality that outputs extracted high-level speech features using said fused features, said autoencoder receiving the extracted high-level text features and speech features in parallel and outputting a shared expression feature dataset for emotion classification that correlates with said mental health assessment.

Description

FIELD OF THE PRESENT APPLICATION

[0001] The invention relates to the fields of artificial intelligence, machine learning, computational networks, and computer engineering. In particular, the present invention relates to a multimodal system for voice-based mental health assessment utilizing induced emotion and emotion recognition techniques, and methods of using the same.

[0002] Due to the impact of the COVID-19 (new coronavirus) pandemic on the global economy, the prevalence of mental illness is accelerating, affecting 10-15% of the world's population. Given the current rate of increase, it has a significant negative economic impact, even compared to cancer, cardiovascular disease, diabetes, and respiratory diseases. Suicide due to mental health issues is now the second leading cause of death among people aged 15-29, resulting in significant social disruption and loss of productivity. In response to this worrying situation, the World Health Organization (WHO) declared in 2016 that depression is the leading cause of mental illness worldwide. However, in the traditional medical system, the number of patients suffering from mental illness continues to increase significantly, and the more severe the number of patients, the more difficult it becomes to provide appropriate treatment. However, early detection and early intervention techniques can dramatically improve the cure rate of mental illness, which can reduce the economic burden on patients and even improve their daily productivity and quality of life. Therefore, automatic screening and continuous monitoring have proven to be an effective alternative that can smoothly realize screening. Voice-based and artificial intelligence (AI)-based screening technologies are gaining popularity due to their proven track record in detecting a range of mental illnesses, including depression and anxiety, across a range of users. The technology is easily accessible via digital devices such as mobile phones, making it an inexpensive and scalable solution. By leveraging vocalizations as "biomarkers," these technologies can be more robust in addressing inappropriate or misconduct during assessment or monitoring sessions than traditional outpatient screening mechanisms such as the Patient Health Questionnaire-9 (PHQ-9) or GAD-7 (Generalized Anxiety Disorder-7). Voice-based artificial intelligence (AI) screening systems and their uses typically rely on scripted dialogues and collect voice-based responses (along with other signals). Based on the collected responses, models built on acoustic features or models built on acoustic and NLP features may be used to generate a final classification, as is well documented in the prior art. To avoid accepting non-useful responses as inputs, certain statistically based techniques may be used to validate each response before proceeding to the next step. A simple solution may measure the total time spent on the recorded response, while more complex solutions may use voice activation or ASR (automatic speech recognition) to check if a sentence with a sufficient number of words is present. If the pre-configured threshold is not met, the user may be presented with the same question (or a different question) again and asked to answer it again. This type of approach with traditional technology has unintended consequences. First, long answers are inherently biased and may not always be appropriate. While shorter answers tend to be more candid about a user's mental health, longer answers tend to be more wordy, which can lead to incorrect ratings if the user is dishonest. Secondly, the ASR (Automatic Speech Recognition) techniques used in such systems and methods typically generate only recognizable sentences / words and may ignore certain speech behaviors such as pauses, hesitations, trembling, wavering, etc. Filtering based on the output from such ASR techniques may miss meaningful answers for the screening model. Another issue is that the emotional state of the user when answering the questions may have some effect on the outcome of the mental health assessment, but the answer validation technology of the prior art is also not a sufficient solution in this respect, as it does not take into account the emotions at the time of input. Therefore, to ensure a relatively high level of input quality, better mechanisms are needed to improve model performance. In recent years, smart phone technology and wearable devices for non-invasive and continuous monitoring of physiological and psychological data have attracted deep interest from many researchers. Advances in acoustic and speech processing have opened up new areas of behavioral health diagnostics using machine learning, demonstrating that patients with depression exhibit reduced stress, monophonia, and attenuation in speech, which are consistent with clinical observations of patients with depression. These patients also speak at a slower rate, with longer pauses, and at a lower volume compared to the general population. Furthermore, studies have shown that vocal biomarkers recognized by machine learning models are positively correlated and have a high potential for detecting mental illnesses such as depression. Researchers have devoted significant efforts to studying the acoustic and semantic-based aspects and correlations of depression and its voice. AI technologies in this field can be divided into three main categories: 1. Semantic-based: This involves applying Automatic Speech Recognition (ASR) to convert speech data into text strings and then running Natural Language Processing (NLP) on top of that to build a natural language-based classification model. 2. Acoustic-based: Extract acoustic features directly from the audio and build a classification model based on them. These features can be either manually designed features, such as rhythmic features / spectral-based correlation features, or latent feature embeddings through pre-trained models. 3. Multimodal-based: There have been some attempts to combine these two modalities to create multimodal AI models and improve the accuracy of the evaluation. These technologies and related research often use user voice recordings related to specific topics as model inputs. For example, answers to certain fixed questions about rest schedules or health conditions. However, it is notable that major depressive disorder (MDD) can cause a person to be numb to emotions, especially sadness, fear, anger, and shame. Many previous studies have suggested that depressed patients, unlike non-depressed people, are particularly important in utilizing emotional preferences and emotion regulation strategies. Unfortunately, existing voice-based AI technologies have not paid enough attention to users' emotional preferences and emotion regulation abilities, from collecting training data to training models and inference when applying models to real applications. Previous research has shown that the human voice conveys a lot of emotional information. For example, some prior art literature has shown that it is possible to detect not only basic emotional tendencies in a voice (e.g., positive vs. negative emotions, excitement vs. calmness, etc.), but also finer emotional nuances. This could improve the accuracy level of such AI models by proactively covering more emotional effects or better leveraging emotional information from voice (especially emotional valence). This patent describes a system for constructing such technology and methods for its use.

[0003] The aim of the present invention is to assess mental health status based on vocal-based biomarkers combined with emotion determination / recognition. Another objective of the present invention is to leverage induced emotions and emotion recognition to eliminate bias in mental health assessment based on vocal-based biomarkers. It is yet another object of the present invention to provide an enhanced filtering technique in mental health assessment based on speech-based biomarkers. It is yet another object of the present invention to ensure a relatively high level of input quality in the speech signal while conducting mental health assessments based on speech-based biomarkers. Yet another object of the present invention is to enhance understanding of vocal behavior in conjunction with mental health assessments based on speech-based biomarkers.

[0004] The present invention provides a multimodal system for audio-based mental health assessment with emotional stimuli and methods for its use. According to the present invention, there is provided a multimodal system for audio-based mental health assessment with emotional stimuli, said system comprising: - A task building module, which is configured to build tasks to capture the acoustic, linguistic, and emotional features of the user's voice; - a stimulus output module configured to receive data from said task construction module, said stimulus output module including one or more stimuli to be presented to the user based on the constructed task, for inducing one or more types of triggers for the user's actions, said triggers being in the form of input answers; - an answer receiving module (configured to present one or more stimuli to a user based on a task constructed from the aforementioned stimulus output module and to receive answers corresponding to one or more formats thereto); - Functional modules include: o a feature construction module (configured to define, for each of the aforementioned constructed tasks, features defined in terms of a learnable heuristic-based evaluation task); o a feature extraction module configured to extract one or more defined features related to said constructed tasks from said received corresponding answers, using a learnable heuristic evaluation model taking into account at least one selected from the ranked constructed tasks; o a feature fusion module, configured to fuse two or more defined features to obtain a fused feature; - An autoencoder leverages the fused features mentioned above to define the relationship between: o The speech modality with respect to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level text features. o The text modality relating to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level speech features. The autoencoder is configured to receive extracted high-level text features and extracted high-level audio features in parallel from the audio modality and the text modality, and to output a shared expression feature dataset for emotion classification that correlates with the mental health assessment. In at least one embodiment of the system, the constructed task is a pronunciation task or a writing task. In at least one embodiment of the system, the task construction module includes a first ranking module for ranking the constructed tasks in order of difficulty, thereby assigning a first value to each constructed task. In at least one embodiment of the system, the task construction module includes a first ranking module configured to rank the constructed tasks in order of difficulty, thereby assigning a first value to each constructed task, where the constructed tasks are stimuli corresponding to analyzed answers and classified into one of the assessed valence levels (positive valence, negative valence, neutral valence). In at least one embodiment of the system, the task construction module includes a second ranking module configured to rank the complexity of the constructed tasks in order of complexity, thereby assigning a second value to each constructed task. In at least one embodiment of the system, the task construction module includes a second ranking module configured to create a data collection pool based on the ranked constructed tasks by ranking the emotional expectations in terms of the complexity and answer vectors of the constructed tasks. In at least one embodiment of the system, the constructed task is selected from a group of tasks including a cognitive task of counting numbers within a predetermined time, a task of pronouncing vowels within a predetermined time, a task of pronouncing words containing voiced and unvoiced sounds within a predetermined time, a task of reading words within a predetermined time, a task of reading a paragraph within a predetermined time, a task of reading a paragraph with phonemic and emotional complexity, a task related to open-ended questions with emotional variations, and a task related to open tasks to be performed within a predetermined time. In at least one embodiment of the system, the constructed task includes one or more questions as stimuli, each of which is assigned a question embedding represented by a vector from 0 to N. Based on these question embeddings, a question-specific feature extractor is trained to extract word embeddings, phoneme embeddings, and syllable-level embeddings from the questions, and these extracted embeddings are forced to be aligned and mid-level feature fusion is performed. In at least one embodiment of the system, the one or more stimuli are selected from a group of stimuli consisting of audio stimuli, video stimuli, combined audio and video stimuli, text stimuli, multimedia stimuli, physiological stimuli, and combinations thereof. In at least one embodiment of the system, the one or more stimuli include a stimulus vector tailored to elicit a text response vector, an audio response vector, a video response vector, a multimedia response vector, a physiological response vector, or any combination thereof, that responds to the stimulus vector. In at least one embodiment of the system, one or more of the aforementioned stimuli are configured to be analyzed through a first vector engine to identify a configuration vector and determine a ranked base state that correlates with the stimulus vector. In at least one embodiment of the system, one or more of the responses are selected, the responses being selected from a group of responses consisting of audio responses, video responses, combined audio and video responses, text responses, multimedia responses, physiological responses, and combinations thereof. In at least one embodiment of the system, the one or more stimuli include an answer vector that correlates to an audio answer vector and / or a video answer vector elicited in response to a stimulus vector from the stimulus output module. In at least one embodiment of the system, one or more of the aforementioned stimuli are configured to be analyzed through a first vector engine to identify a configuration vector and determine a ranked base state that correlates with the stimulus vector. In at least one embodiment of the system, the answer receiving module includes a sentence reading module configured to perform a sentence reading task within a time period preconfigured by a user. In at least one embodiment of the system, the feature construction module is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction module. In at least one embodiment of the system, the feature construction module is a Geneva Minimum Acoustic Parameter Set (GeMAPS) based feature construction module, the feature construction module comprising: - Analyzes the voice using a set of 62 parameters. - Provides a 3 frame long symmetric moving average filter to smooth over time (this smoothing is performed in the voiced regions of the previous answer for pitch, jitter, and shimmer). - The arithmetic mean and coefficient of variation are applied as functions to 18 low-level descriptors (LLDs) to generate 36 parameters. - Apply 8 functions to the volume. - Apply 8 functions to the pitch. - Apply 8 functions to the pitch. - Determine the Hammerberg Index. - Determine the spectral features (for all unvoiced segments, this is done by looking at the spectral tilts between 0-500Hz and 500-1500Hz). - From the answers given above, determine the temporal features in continuous voiced and unvoiced regions. - Determines Viterbi-based smoothing of the F0 contour (prevents errors from causing a single voiced frame to be dropped). In at least one embodiment of the system, the feature construction module is a Geneva Minimum Acoustic Parameter Set (GeMAPS) based feature construction module comprising a set of low level descriptors (LLDs) for analyzing spectral, pitch and temporal features in the speech response, the features being selected from a group of features consisting of: Mel-Frequency Cepstral Coefficients (MFCCs) and their first and second derivatives. · Pitch and pitch variability. Energy and energy entropy. · Spectral centroid, broadness, and flatness. Spectral tilt. Spectral roll-off. Spectral variability. Zero crossing rate. Shimmer, jitter and harmonic-to-noise ratio. Voicing probability (based on pitch) Temporal features such as loudness peak ratio, mean duration and standard deviation in continuous voiced and unvoiced regions In at least one embodiment of the system, the feature construction module is a Geneva Minimum Acoustic Parameter Set (GeMAPS) based feature construction module, the feature construction module being configured with a set of frequency-related parameters selected from the group consisting of: Pitch (measured as the logarithmic fundamental frequency (F0) on a chromatic frequency scale, starting at 27.5 Hz (semitone 0)) Jitter (deviation in the length of each successive F0 period) Formant 1, 2, 3 Frequencies (center frequencies for the first, second and third formants) Formant 1 (bandwidth related to the first formant) Energy related parameters. Amplitude related parameters. Shimmer (the difference in peak amplitude between successive F0 intervals) Loudness (an estimate of perceived signal strength from the auditory spectrum) Harmonic-to-noise ratio (the ratio of the energy containing harmonic components to the energy containing noise-like components) Spectral (balance) parameters Alpha ratio (ratio of the total energy between 50-1000 Hz and 1-5 kHz) Hammerberg Index (ratio of the strongest energy peak in the 0-2 kHz range to the strongest peak in the 2-5 kHz range) Spectral tilt (the slope of the linear regression on the logarithmic power spectrum in two specified bands: 0-500 Hz and 500-1500 Hz) Relative energy of formants 1, 2, 3 (ratio of the energy associated with the spectral harmonic peaks at the center frequencies of the first, second, and third formants to the energy associated with the spectral peak at the fundamental frequency (F0)) Harmonic difference H1‐H2 (ratio of the energy associated with the first harmonic (H1) of the fundamental frequency (F0) to the energy associated with the second harmonic (H2)) Harmonic difference H1‐A3 (ratio of the energy associated with the first harmonic (H1) of the fundamental frequency (F0) to the energy associated with the highest harmonic in the range of the third formant (A3)) In at least one embodiment of the system, the feature construction module is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction module that utilizes Higher Order Spectral Analysis (HOSA) functions that utilize two or more component frequencies to achieve bispectral frequencies. In at least one embodiment of the system, the feature construction module is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction module that utilizes Higher Order Spectral Analysis (HOSA) functions that utilize two or more component frequencies to achieve a bispectrum frequency that utilizes a third accumulator to analyze the relationship between frequency components in a signal and examine nonlinear signals related to the answer. In at least one embodiment of the system, the feature extraction module fuses high-level feature embeddings by leveraging mid-level fusion. In at least one embodiment of the system, the feature extraction module includes the ability to extract specialized linguistic features, the extent to which which can be learned for each stimulus, making it highly suitable for a variety of linguistic tasks. In at least one embodiment of the system, the feature extraction module includes the ability to extract specialized emotional features and can learn degrees of each stimulus, making it highly suitable for a variety of emotional tasks. In at least one embodiment of the system, the feature fusion module includes a speech module configured to provide high-level feature fusion capabilities and to classify sentiment from one or more answers by leveraging at least autoencoder-based feature fusion. In at least one embodiment of the system, the feature fusion module includes a speech module configured with high level feature fusion capabilities and configured to leverage at least autoencoder based feature fusion to classify sentiment from one or more answers, and the speech modality leverages question specific feature extraction capabilities to extract high level features from time frequency domain relationships in the answers and output the extracted high level speech features. In at least one embodiment of the system, the feature fusion module includes a text module configured to provide high-level feature fusion capabilities and leverage at least autoencoder-based feature fusion to classify sentiment from one or more answers. In at least one embodiment of the system, the feature fusion module includes a text module configured with high level feature extraction and configured to utilize at least autoencoder based feature fusion to classify sentiment from one or more answers, and the text modality outputs extracted high level speech features utilizing a bidirectional long short term memory network and an attention mechanism. In at least one embodiment of the system, the feature fusion module includes a speech module: - A module that uses extracted features from the pre-trained model in acoustic feature embedding. - A module that compares the previously extracted features over the spectral domain. - A module for determining the characteristics of vocal tract coordination. - A module for determining features in replicate quantification analysis. - A module for determining features relating to the number of bigrams and the bigram duration corresponding to the vocal landmarks. - A module that fuses the aforementioned features in an autoencoder. In at least one embodiment of the system, the feature extraction module includes an audio landmark extractor configured to determine event markers associated with the answer, the determination of the event markers being correlated with positions on a timeline of acoustic events from the answer, including determining timestamp boundaries that indicate abrupt changes in the acoustic answer, in a frame-independent manner. In at least one embodiment of the system, the feature extraction module includes a speech landmark extractor configured to identify event markers associated with the response, each having a start value and an end value, selected from a group of glottal-based landmarks (g), periodicity-based landmarks (p), sonorant-based landmarks (s), fricative-based landmarks (f), voiced fricative-based landmarks (v), and burst-based landmarks (v), each of which is used to identify a point in time at which a distinct abrupt articulatory event occurs, which correlates with an abrupt change in power across multiple frequency ranges and multiple time scales. In at least one embodiment of the system, the autoencoder comprises a multi-modal, multi-question input fusion architecture that includes: - One or more encoders that map one or more of the aforementioned specific features, in combination with a task type, to a low-dimensional representation (each task is multiplied by a learnable degree encoding matrix based on the task type, and these degrees are correlated with mental health assessments). - one or more decoders that map said one or more particular features to said lower dimensional representation (configured to output a mental health assessment); - The aforementioned autoencoder is trained to minimize the reconstruction error between the input task and the decoder's output by leveraging a loss function. According to the present invention, a multimodal use method for audio-based mental health assessment with emotional stimuli is provided, said use method including: - Building tasks to capture the acoustic, linguistic, and emotional characteristics of a user's voice. - receiving data containing one or more stimuli based on said constructed task and generating triggers that induce one or more types of user actions by presenting said task to the user (wherein said triggers are in the form of input answers); - presenting one or more stimuli to the user based on said constructed task and receiving corresponding responses in one or more formats. - Defining, for each of the aforementioned constructed tasks, features defined in terms of a learnable heuristic-type evaluation task. - extracting one or more defined features related to said constructed tasks from said received corresponding answers by utilizing a learnable heuristic evaluation model that considers at least one selected from the ranked constructed tasks. - Fusing two or more defined features to obtain a fused feature. - Leveraging the aforementioned fused characteristics, define the relationships between: o The speech modality with respect to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level text features. o The text modality relating to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level speech features. The step of defining the relationship is configured to receive extracted high level text features and extracted high level audio features in parallel from the audio modality and the text modality, and output a shared expression feature dataset for emotion classification correlated with the mental health assessment. In at least one embodiment of the method of use, the constructed task includes one or more questions as stimuli, each of which is assigned a question embedding represented by a vector from 0 to N. Based on these question embeddings, a question-specific feature extraction module is trained to extract word embeddings, phoneme embeddings, and syllable level embeddings from the questions, and these extracted embeddings are forced to be aligned, as well as perform mid-level feature fusion. In at least one embodiment of the method of use, the step of defining the relationships comprises a multi-modal, multi-question input fusion architecture including: - Through an encoder, one or more of said specific features are combined with a task type and mapped to said low-dimensional representation (each task is multiplied by a learnable degree encoding matrix based on the task type, and these degrees are correlated with mental health assessments). - via a decoder, mapping said one or more particular features to said low dimensional representation and outputting a mental health assessment. - By utilizing a loss function, it is trained to minimize the reconstruction error between the input task and the decoder's output. In at least one embodiment of the method of use, the step of extracting high level speech features includes extracting high level features from time-frequency domain relationships in the answer, thereby outputting the extracted high level speech features. In at least one embodiment of the method of use, the step of extracting high level text features includes simulating intra-modal dynamics using a bidirectional long short-term memory network and an attention mechanism to output the extracted high level text features. [Brief description of the drawings]

[0005] The present invention is now disclosed with reference to the accompanying drawings, in which: Figure 1 shows a high-level block diagram of the computing environment. Figure 2 shows the system for collecting training data with induced emotions. Figure 3 shows a sample emotion-elicitation question set that presents several tasks and questions to elicit emotion-based answers. Figure 4 shows the known interactions between phonetic components that form useful phonetic criteria. Figure 5 shows one example of a fixed read-aloud sentence. Figure 6 shows a tone-free phoneme map based on a user reading the sentence. Figure 7 shows another example of a fixed read-aloud sentence. Figure 8 shows a graph of phonemes versus occurrences based on a user reading a sentence. Figure 9 shows the content regarding the representation of HOSA (Higher Order Spectral Analysis) functions for different mental states of users. Figure 10 shows the auto- and cross-correlations between the first and second delta MFCCs (Mel-frequency cepstral coefficients) extracted from a 10-second audio file, illustrating our framework for extracting delayed correlations from acoustic files reflecting psychomotor retardation. Figure 11 shows a schematic block diagram of an autoencoder. 12A-12H show various graphs for at least one type of corresponding task (question), where the original labels are selected from "health" or "depression", and the graphs show phonemes and segments that correlate with the model predictions, in a non-limiting exemplary embodiment. Areas highlight linguistic utterances with high activation characteristics for different users at different start times. Figure 13 shows the flow chart. Figure 14 shows a high-level flowchart of a voice-based mental health assessment with emotional stimuli.

[0006] The present invention provides a multimodal system for speech-based mental health assessment with emotional stimuli and its use. Mental health issues are closely linked to the emotions of patients. The emotions of a user are at least as important as the length of the response when determining whether the user's input is useful to the system, and in some circumstances may be more important. The present disclosure may be embodied as a system, method of use, computer program product, or program / product related to a mobile device. The computer program product may include a computer readable storage medium (or media) that includes computer readable program instructions for a processor to execute aspects of the present disclosure. Aspects of the disclosed embodiments may include a tangible computer readable medium storing software instructions that, when executed by one or more processors, are configured and capable of performing one or more methods or operations in accordance with the disclosed embodiments. Aspects of the disclosed embodiments may also include logic and instructions executed by one or more processors configured as special purpose processors based on programmed software instructions that, when executed, perform one or more operations in accordance with the disclosed embodiments. When describing this invention, the following definitions apply throughout (including above): "Computer" may mean one or more devices or one or more systems that can accept structured input, process the input in accordance with defined rules, and produce the results of the processing as output. Examples of computers may include computers, fixed or portable computers, computers with single processors, multiple processors, or multi-core processors (which may operate in parallel or non-concurrently), general purpose computers, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, client servers, interactive televisions, web appliances, communications devices with Internet access, hybrid computer interactive televisions, tablets (PCs), personal digital assistants (PDAs), mobile phones, application specific hardware for emulating a computer or software (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), chips, chipsets), systems-on-chips (SoCs), multiprocessor systems-on-chips (MPSoCs), optical computers, quantum computers, biological computers, and devices (which typically include input, output, memory, arithmetic, logic and control units) that accept data, process the data according to one or more stored software programs, and produce a result. "Software" can refer to the rules set forth for operating a computer or part of a computer. Examples of software may include code segments, instructions, applets, pre-compiled code, compiled code, interpreted code, computer programs, and programmed logic. "Computer-readable medium" may refer to any storage device used to store data that can be accessed by a computer. Examples of computer-readable medium may include magnetic hard disks, floppy disks, optical disks such as CD-ROMs and DVDs, magnetic tapes, memory chips, or any other type of media capable of storing machine-readable instructions. "Computer system" may refer to a system containing one or more computers, each of which may contain a computer-readable medium that implements software that operates the computer. Examples of computer systems include a distributed computer system that processes information across a network of connected computer systems, a system in which two or more computer systems are connected over a network to send information to and receive information from each other, and one or more devices or systems (typically including input, output, memory, arithmetic, logic and control units) that may accept data and process the data in accordance with one or more stored software programs to produce a result. A "network" may refer to multiple computers and related devices that may be connected by communications facilities. A network may include permanent connections such as cables as well as temporary connections performed through telephone or other communications links. A network may also include wired connections (e.g., coaxial cable, twisted pair, fiber optics, waveguide, etc.) or wireless connections (e.g., radio frequency waveforms, free space optical waveforms, acoustic waveforms, satellite communications, etc.). Examples of networks include the Internet, intranets, local area networks (LANs), wide area networks (WANs), and combinations of networks (e.g., the Internet and an intranet). Examples of typical networks may operate over a variety of protocols, including Internet Protocol (IP), Asynchronous Transfer Mode (ATM), Synchronous Optical Network (SONET), User Datagram Protocol (UDP), and IEEE 802.x. The terms "data" and "data item" as used in this entry refer to a sequence of bits. Thus, a data item could be the contents of a file, part of a file, a page in memory, an object in an object-oriented program, a digital message, a digitally scanned image, part of a video or audio signal, or any other entity that can be represented by a sequence of bits. The term "data processing" as used in this entry refers to the processing of data items and may depend on the type of data item being processed. For example, a data processor for digital images may be different from a data processor for audio signals. The terms "first", "second", etc., as used herein do not denote any order, priority, quantity or importance, but are used merely to distinguish one element from another. Additionally, the words "1" and "a" as used hereinafter do not denote a limitation of quantity, but rather denote the presence of one or more of the referenced item. FIG. 1 illustrates a conceptual block diagram of a computing environment including one or more network client devices (112, 114, 116, 118) connected to a network server (100) through a network, and one or more databases (122, 124, 126, 128) connected to the network server (100). Although speech production may seem simple, it is actually a very complex process involving the coordination of cognitive and physiological behaviors in the brain, as it requires a complex set of actions to be performed effectively. Usually, speech production originates from physiological processes in the body, starting with the generation of potential energy in the lungs and the change in air pressure in the vocal tract. When speaking with vocal sounds, the lungs expel air, and the speed of the air also affects the regularity of the vocal cords. As the harmonic and rich sound energy from the glottis passes through the vocal tract and larynx, the movements of the pharynx, oral cavity, nasal cavity, and speech organs (tongue, teeth, lips, jaw, soft palate, etc.) change the amplitude of the harmonic sounds of the voice, which act as sound filters. Depression and psychomotor retardation are associated with speech dysfunction and reduced harmonic formant amplitude as well as other physical abnormalities in some depressed individuals. This can result in poor laryngeal control due to psychomotor retardation, leading to a "breathiness" in the speech of the patient, in contrast to the speech of healthy individuals. Voice intensity has been shown to be a strong indicator of the severity of depression. Depressed patients often speak with a low voice intensity and may appear to speak in a monotone voice. Many researchers have studied how people with depression use language in two ways: (a) by listening to recordings of the patients speaking, and (b) by analyzing the texts that the patients have actually written. People with depression often have some problems with language skills and may use inappropriate or slurred language, leave things unfinished, or repeat the same words or phrases over and over again. To identify depression, it is important to detect possible abnormalities in phonological elements, which also interact with linguistic information (e.g. the meanings of words and phrases) to describe an individual's emotional state. Previous subjective assessments of depressed individuals have often focused on the patient's speech patterns and associated behaviors, based on preconceived notions of how depressed people "behave emotionally." Figure 2 shows the system for collecting training data with induced emotions. In at least one embodiment, the client devices (112, 114, 116, 118) are communicatively coupled to a stimulus output module (202) configured to provide one or more stimuli to a user corresponding to the client device. In at least one embodiment, the stimulus output module (202) is configured to provide output stimuli to the user to elicit a user response in the form of an input response. The output stimuli may include audio stimuli, video stimuli, combined audio and video stimuli, text stimuli, multimedia stimuli, physiological stimuli, combinations thereof, and the like. The stimuli may include stimulus vectors tailored to elicit a text response vector, an audio response vector, a video response vector, a multimedia response vector, a physiological response vector, or combinations thereof that respond to the stimulus vector. One or more stimuli are analyzed through a first vector engine (232) to identify constituent vectors and determine ranked baseline states that correlate with the stimulus vectors. In some embodiments, the stimuli output module (202) is used to send audio-based tasks to subjects (users) with certain mental disorders based on certain criteria (e.g., DSM-V diagnoses) and healthy subjects (users) without mental disorders through a network server (100) or through client devices (112, 114, 116, 118) corresponding to the users or subjects. In a preferred embodiment, these stimuli are vector-structured to elicit specific emotional responses in the subjects, including, but not limited to: - Emotional response due to happiness - Emotional reactions due to sadness - Neutral emotional response In some embodiments, the stimulus output module (202) includes stimuli that elicit specific user or subject actions, such as imperative questions that force the user to repeat a certain number of words or vowel utterances. An embodiment of the present disclosure may include a task construction module configured to construct a task to capture acoustic, linguistic, and emotional characteristics of a user's linguistic vocalizations. Data from the task construction module is provided to a stimulus output module (202). The present invention, which deals with vocal-based biomarkers, may be preferably utilized for tasks involving such vocalization tasks. In at least one embodiment, the task construction module includes a first ranking module for ranking the constructed tasks in order of difficulty, thereby assigning a value to each constructed task. In a preferred embodiment, it is necessary to actively stimulate the user (speaker) according to different emotional valences (ranked levels: positive, negative, neutral) when recording the patient's answers in order to better capture the utilization of emotion regulation strategies related to the user's emotional preferences and mental state. In at least one embodiment, the task construction module includes a second ranking module configured to evaluate question complexity and emotional expectation of the ranked questions to create a data collection pool. In multi-class classification, depression levels have an ordinal relationship, so that in the vocalization tasks for different depression levels from the task construction module, a learnable “degree” is set so that the loss can be optimized for each question and each depression level. Figure 3 shows a sample emotion elicitation task set (question set) that presents several speech tasks to elicit emotion-based answers. Below, we show the emotion elicitation patterns for each speech task in at least one embodiment. Well-designed tasks not only adequately cover different emotions, but also provide good coverage at the phoneme level in the target language. [Table 2] In at least one embodiment, the client devices (112, 114, 116, 118) are communicatively coupled to at least one response receiving module (204) configured to present one or more stimuli from the stimulus output module (202) to a user of the corresponding client device and to receive response input in one or more forms. At least one embodiment of the answer receiving module (204) includes a receiving module (204a) configured to capture a user's input answer to an output stimulus. The input answer may include audio input, video input, combined audio and video input, multimedia input, or other similar input modalities. The input answer may include an answer vector related to an audio answer vector or a video answer vector elicited in response to a stimulus vector associated with the stimulus output module (202). Embodiments of the present disclosure may also include a data pre-processing module. In some embodiments of the task construction module, question construction may include simple cognitive tasks consisting of open-ended questions such as number counting, vowel pronunciation, vocalization of words with voiced and unvoiced components, paragraph reading with phonemic and emotional complexity, emotional variation and cognitive sentence generation, and answers may be collected through an answer receiving module (204). Embodiments of the receiving module (204a) may include constructing and evaluating answers, recording answers, and processing answer vectors until boundary conditions are met to make them eligible data points for training. The answer vectors may also include flashcards for specific data collection tasks, and may also collect metadata about the user's recordings, including details about task completion and qualifications. An embodiment of the response receiving module (204) may also include a first measurement module configured to analyze, measure, and output as a first measurement data set including at least the following: - Content comprehension measured in terms of answer vectors. - The accuracy of the answer content measured in terms of the answer vector. - The quality of the acoustic signal measured with respect to the answer vector. Embodiments of the response receiving module (204) may also include a second measurement module configured to analyze, measure, and output as a second measurement data set including at least the following: - Silence measured from the answer vector. - Signal to noise ratio measured from the response vector. - Articulation clarity measured from response vectors. - Activity index measured from the response vector. It is based on a threshold that is pre-set on the second measurement data mentioned above. In a preferred embodiment, segments of the response vector that are measured to have a signal-to-noise ratio of 15 or greater are used as qualifying speech samples by the system and method of the present invention. In at least one embodiment of the receiving module (204a), at least one acoustic receiving module (204a.1) is provided, which is configured to capture an acoustic input response of a user in response to one or more output stimuli in the form of an answer acoustic signal, which includes an answer acoustic vector, which is correlated with a stimulus vector of the stimulus output module (202) by a second vector engine (234), which determines a constituent answer acoustic vector and determines a first state of the user in relation to the stimulus vector of said stimulus output module (202). In at least one embodiment of the receiving module (204a), at least one text receiving module (204a.2) is provided, which is communicatively connected to the acoustic receiving module (204a.1) and configured to obtain the user's acoustic input answer from the acoustic receiving module (204a.1) and transcribe it into text through a transcription engine to provide an answer text signal. The answer text signal includes answer text vectors, which are correlated with the stimulus vectors of the stimulus output module (202) by a third vector engine (236). This second vector engine identifies the constituent answer text vectors and identifies a second state of the user in relation to the stimulus vectors of said stimulus output module (202). At least one embodiment of the response receiving module (204) includes a physiological response receiving module (204b) configured to sense one or more physiological response signals of a user to the output stimuli of the stimulation output module (202) through one or more physiological sensors. The physiological response signals may include physiological vectors that correlate to the physiological signals in response to the output stimuli of the stimulation output module (202). These vectors are analyzed through a fourth vector engine (238) configured to determine a third state of the user in relation to the stimulation vectors of the stimulation output module (202). In at least one embodiment of the response receiving module (204), a neurological response receiving module is provided that is configured to sense one or more neurological response signals of a user to the output stimuli of the stimulation output module (202) through one or more neurological sensors. The neurological response signals may include neurological vectors that correlate to the neurological signals in response to the output stimuli of the stimulation output module (202). These vectors are analyzed through a fifth vector engine that is configured to determine a fourth state of the user in relation to the stimulation vectors of the stimulation output module (202). Embodiments relating to the vector engine form an engine for identifying vocal biomarkers. Typically, a vocal biomarker identification engine uses a three-stage approach. - Use knowledge of the vocal domain to develop speech data collection protocols to increase the presence and robustness of vocal biomarkers associated with depression. - Identifying the optimal set of features that capture subtle acoustic differences between depressed and non-depressed individuals (these features should be relevant for good detection and robust to noise, making them useful for detecting depression in natural environments). - Require multi-dimensional analysis, as the status of vocal biomarkers in speech systems is not limited to binary "yes" or "no" (this will require pre-determining appropriate tasks and time to perform those tasks so that vocal biomarkers remain visible or present when using automated detection systems for people with depression). The time required for detection, or the Task-Based Detection Threshold (TBDT), may vary with gender, severity of depression, and age. Therefore, a procedure must be implemented that allows for the efficient ordering of speech tasks to identify the Detection Threshold and Detection Confidence Index Score (DCMS) so that the system and its uses can have a stable detection range for symptoms. Embodiments of the present disclosure may also include a protocol construction engine. Automatic voice processing using machine learning is becoming more and more used in the digital healthcare sector and has great potential as a non-invasive and remote medical screening tool, but there is a need to better understand the protocols used for voice processing and perform measurements that will help to create new protocols with specific criteria. Figure 4 shows the known interactions between phonetic components that form useful phonetic criteria. Healthcare professionals use speech and language assessments to screen, diagnose, and monitor patients for a variety of physical disorders. During the assessment, clinicians observe the patient's speech production, including articulation, breathing, phonation, and voice quality, as well as the patient's own language skills, including grammar, pragmatics, memory, and expression. Abnormal speech and language symptoms are often early indicators of a variety of physical disorders and illnesses. At least one embodiment of the answer receiving module (204) comprises a text-to-speech module that allows the user to read text based on predefined tasks or prompts. Advantages of utilizing text-to-speech protocols include ease of use, reproducibility, smooth provision of clear reference points, limited range of sounds used, and controlled sound variations. Furthermore, they are relatively easy to integrate into digital smart device applications. When selecting a text-to-speech protocol for analyzing a patient's health condition, it is very important to carefully consider factors such as the speaker's background, the specific disease to be focused on, the time required for the task, and the number of samples. In at least one embodiment of the sentence reading module, emotion-based, anchored sentence reading samples are provided. Figure 5 shows one example of a fixed read-aloud sentence. Figure 6 shows a tone-free phoneme map based on a user reading the sentence. Figure 7 shows another example of a fixed read-aloud sentence. Figure 8 shows a graph of phonemes versus occurrences based on a user reading a sentence. An embodiment of the present disclosure includes a feature module with feature construction capabilities for each constructed task, where the features are defined in terms of a learnable heuristic-based evaluation task. At least one embodiment of the feature building function utilizes the Geneva Minimal Acoustic Parameter Set (GeMAPS). The Geneva Minimal Acoustic Parameter Set is a set of 62 parameters utilized for speech analysis. A symmetric moving average filter with a length of 3 frames is utilized to smooth over time, and this smoothing is performed only within the voiced regions of pitch, jitter, and shimmer. Arithmetic mean and coefficient of variation are applied as functions across all 18 LLDs (low level descriptions) to generate 36 parameters. In addition, eight functions are applied to loudness and pitch, including the arithmetic mean of alpha ratio, Hammerberg exponent, and spectral slope (0-500Hz and 500-1500Hz) across all unvoiced segments. Temporal features such as the occurrence rate of loudness peaks, the mean length and standard deviation of consecutive voiced and unvoiced regions, and the number of consecutive voiced regions per second are also included. It imposes no minimum length on voiced or unvoiced regions and performs Viterbi-based smoothing of the F0 (fundamental frequency) contour to prevent errors from causing a single voiced frame to be dropped. eGeMAPS (Geneva Minimal Acoustic Parameter Set) is a feature set used in speech, audio and music analysis and is used in openSMILE (Open Source Multimodal Interface for Language and Emotion Recognition) eGeMAPS is a subset of the larger Geneva Minimal Acoustic Parameter Set (GeMAPS) specifically designed for the task of emotion recognition. It contains a set of low-level descriptors (LLDs) for analyzing the spectral, pitch and temporal characteristics of speech signals. Features include: Mel-frequency cepstral coefficients (MFCCs) and their first and second derivatives Pitch and pitch variability Energy and Energy Entropy Spectral centroid, broadness, and flatness Spectral Tilt Spectral Rolloff Spectral variability Zero crossing rate Shimmer, jitter, and harmonic-to-noise ratio Voicing probability (based on pitch) Temporal features such as loudness peak ratio, mean duration and standard deviation in continuous voiced and unvoiced regions In total, 87 features are included. eGeMAPS is designed as a minimal feature set that can provide robust analytical performance across a range of situations. It performs robustly regardless of recording conditions and speaker characteristics, and has also been shown to be effective in multiple emotion recognition tasks. Amplitude-related parameters Pitch (measured on a chromatic frequency scale as the logarithmic fundamental frequency (F0), starting at 27.5 Hz (semitone 0)) Jitter (deviation in the length of individual successive F0 periods) Formant 1, 2, 3 Frequencies (center frequencies for the first, second and third formants) Formant 1 (bandwidth related to the first formant) Energy / amplitude related parameters Shimmer (difference in peak amplitude between successive F0 periods) Loudness (an estimate of perceived signal strength from the auditory spectrum) Harmonic-to-Noise (HNR-) ratio (the ratio of the energy containing harmonic components to the energy containing noise-like components) Spectral (balance) parameters Alpha ratio (ratio of 50-1000 Hz and 1-5 kHz with respect to the total energy) Hammerberg Index (ratio of the strongest energy peak in the 0-2 kHz range to the strongest peak in the 2-5 kHz range) Spectral tilt (the slope of the linear regression on the logarithmic power spectrum in two specified bands: 0-500 Hz and 500-1500 Hz) Relative energy of formants 1, 2, 3 (ratio of the energy associated with the spectral harmonic peaks at the center frequencies of the first, second, and third formants to the energy associated with the spectral peak at the fundamental frequency (F0)) Harmonic difference H1‐H2 (ratio of the energy associated with the first harmonic (H1) of the fundamental frequency (F0) to the energy associated with the second harmonic (H2)) Harmonic difference H1‐A3 (ratio of the energy associated with the first harmonic (H1) of the fundamental frequency (F0) to the energy associated with the highest harmonic in the range of the third formant (A3)) At least one embodiment of the feature building function uses higher order spectral analysis (HOSA) functions, which are functions of two or more component frequencies, as opposed to a power spectrum, which is a function of a single frequency. These spectral analysis functions can be used to identify phase coupling between Fourier components, and are particularly useful for detecting nonlinearities in a system to facilitate characterization. To achieve this, the amplitudes in the higher order spectral analysis are normalized by the power at the component frequencies. The normalized spectral analysis function, also known as the nth order coherence index, is also a function of the nth order cumulant spectrum and the power spectrum combined. The bispectrum is a method that uses third-order cumulants to analyze the relationships between frequency components in a signal and is particularly useful in investigating nonlinear signals. The bispectrum is unique in that it provides information about the phase relationships between frequency components, and therefore contains more information than the power spectrum, as it contains information not shown in the spectral domain. Higher order statistics are a very effective method for studying nonlinear signals, as they can capture the relationships between phase components. The bispectrum reveals information not shown in the spectral domain, making it one of the best methods for this purpose. Figure 9 shows the content regarding the representation of HOSA (Higher Order Spectral Analysis) functions for different mental states of users. An embodiment of the present disclosure may include a feature module including a feature extractor with a variable / heuristic-based evaluation model that considers at least one selected from the ranked tasks, ranked questions, in a preferred embodiment, to a different extent per question or task, with learnable properties. In a preferred embodiment, the feature extractor uses fusion at mid-levels and fuses feature embeddings at higher levels. In this paragraph, an embodiment of the feature extractor is described. When training a task (e.g., acoustic questions), each stimulus (e.g., question) may be assigned a dedicated feature extractor with a degree of learnability. In a preferred embodiment, linguistic tasks / stimuli (questions) may be assigned a dedicated linguistic feature extractor with a degree of learnability for each stimulus (e.g., question). In a preferred embodiment, emotion tasks / stimuli (questions) may be assigned a dedicated emotion feature extractor with a learnable degree for each stimulus (e.g., question). In a preferred embodiment, the selector is configured to randomly select a sample of answer vectors based on pre-set thresholds such as signal to noise ratio, clarity of the answer (vocalization), etc. Embodiments of the present disclosure may include a data loader configured to randomly upsample or downsample answer vectors until the training data batches are balanced. The data loader is constructed to randomly upsample or downsample as appropriate until the training data batches are balanced. Embodiments may also include systems and methods configured to use soft labels instead of hard labels in classification problems due to data ordering. Despite attempts to leverage unitary emotion feature representations, existing techniques have proven insufficient to achieve recognition due to poor feature differentiation and lack of ability to effectively capture dynamic interactions between emotions in speech recognition tasks. Accordingly, embodiments of the present disclosure may include a feature module comprising a feature fusion module configured to fuse the various output stimuli and / or user states described above. Typically, the feature fusion module works in conjunction with an answer receiving module (204). In a preferred embodiment, the feature fusion module is configured to classify mental health conditions from multiple types of answer vectors by leveraging an audio modality with high level feature extraction and at least autoencoder based feature fusion. In a preferred embodiment, the audio modality uses question specific feature extraction to extract high level features from time frequency domain relationships in the answer vectors, where the output is the extracted high level audio features. In a preferred embodiment, the feature fusion module is configured to classify sentiment from multiple types of answer vectors by leveraging a text modality with high-level feature extraction and at least autoencoder-based feature fusion. In a preferred embodiment, the text modality simulates intra-modal dynamics by using a bidirectional long short-term memory network and an attention mechanism, where the output is extracted high-level text features. In a preferred embodiment of the speech modality, feature fusion module, the system and its use extracts high-level embeddings, i.e. high-dimensional vector embeddings, from pre-trained models by utilizing acoustic feature embedding features from pre-trained models such as huBert, Wav2vec, Whisper, etc., where raw speech input is used. Similarly, the system and its use extracts acoustic feature embeddings (from GeMAPS and emobase egemap) and compares them in the spectral domain. The system and its use also extracts high-order spectral features (HOSA) as well as log-mel spectrograms and MFCCs (Mel Frequency Cepstral Coefficients), and psychomotor delay and neuromuscular activity features by utilizing vocal tract tuning characteristics and iterative quantitative analysis characteristics. Bigram count and bigram duration are calculated by utilizing vocal landmarks that indicate the effectiveness of the pronunciation. All features are fused using an autoencoder, and feature-related nonlinearities are preserved during feature fusion in the latent feature space. Previous studies have shown that leveraging lexical information improves the estimation performance of emotional valence. Lexical information is obtained from a pre-trained acoustic model, and the learned representations can improve the estimation performance of emotional valence from speech. The system and its usage also explores the utilization of the representations of the pre-trained model and improves the estimation performance of depression biomarkers from speech signals while evaluating psychomotor retardation through task-specific feature extraction functions such as neuromuscular coordination features. The system and its usage also explores the fusion of representations to improve the estimation performance of depression biomarkers. Human vocal communication is roughly composed of two layers: the linguistic layer, which conveys the message in the form of words and their meanings, and the paralinguistic layer, which conveys how the words were spoken, such as the expressiveness and emotional tone of the voice. Given the self-supervised learning architecture of pre-trained models and the existence of large-scale vocalization datasets made publicly available, we can speculate that the representations produced by these pre-trained models may contain lexical information that can help make the estimation of emotional valence smoother. In some embodiments of the present invention, a multimodal granularity framework is explored so that the system and its usage can extract speech embeddings from various subword levels. In the figure, it is shown that the embeddings extracted from the relevant models are usually frame-level embeddings and are also effective in capturing rich frame-level information. However, it lacks the ability to capture segment-level information that is useful for identifying depression biomarkers. Thus, the system and usage of the present invention introduces segment-level embeddings that include word, phoneme, and syllable level embeddings that are closely related to phonology as well as frame-level embeddings. Phonology contains information about the rhythm (cadence) of a language's speech and can convey features in the speech (e.g., depression state). As a result, segment-level embeddings can be very useful in multimodal depression biomarker recognition. By using the forced alignment method, we can obtain the temporal boundaries of phonemes, which can then be grouped to obtain syllable boundaries. After the forced alignment information is provided, we can extract the relevant speech segments corresponding to those linguistic units. Embodiments of the present disclosure may also include speech landmark extraction functionality. Articulation landmarks are event markers related to the pronunciation of a language utterance. They provide information about articulation events such as vocal fold vibrations, relying only on the location of acoustic events in time, such as consonant closure and release, nasal closure and release, spontaneous minimum, and vowel maximum. Unlike frame-based processing frameworks, the landmark method is able to detect timestamp boundaries that indicate abrupt changes in speech intelligibility without relying on frames. Our approach provides an alternative to frame-based processing and avoids its shortcomings by focusing on acoustically measurable changes in a language utterance. VTH (Vocal Tract Adjustment) employs six landmarks, each with a start and end state. These landmarks are 'g' (larynx), 'p' (periodicity), 's' (rumble), 'f' (fricative), 'v' (voiced fricative), and 'b' (plosive), and are used to identify the time points of different sudden phonation events. These are detected by observing significant evidence of abrupt changes in energy intensity (i.e., rise or fall) associated with voicing across multiple frequency ranges and multiple time scales. In landmarks, "s" and "v" are associated with voiced sounds, while "f" and "b" are associated with unvoiced sounds. Embodiments of the present disclosure may also include a vocal tract adjustment engine. VTC features have been shown to have the ability to capture psychomotor activity associated with depression, and have been the most successful in two A VEC challenges for predicting depression severity, based on the observation that vocal tract parameters in depressed patients are less "tunable" (correlated) compared to healthy speakers. [Table 3] Figure 10 shows the auto- and cross-correlations between the first and second delta MFCCs (Mel-frequency cepstral coefficients) extracted from a 10-second audio file, illustrating our framework for extracting delayed correlations from acoustic files reflecting psychomotor retardation. Figure 11 shows a schematic block diagram of an autoencoder. An embodiment of the present disclosure may include an autoencoder that defines a relationship between the speech modality of the feature fusion module working in conjunction with the answer receiving module (204) and the text modality (of the feature fusion module working in conjunction with the answer receiving module (204)) by coordinating with the stimulus output module (202). The autoencoder is configured to output a shared expressive feature dataset for emotion classification by feeding the extracted high-level text features and the extracted high-level speech features in parallel to the autoencoder after the aforementioned processing, i.e., processing of the speech modality and the text modality, is completed. What is unique about the present system and its usage is that it can measure the accuracy of the autoencoder in reconstructing the shared expressive feature dataset and minimize the reconstruction error, as well as evaluate the performance of depression detection in the emotion recognition module (300). This is a major departure from previous systems and usages that focus solely on learning high-level features from input data using autoencoders. Autoencoders excel at capturing low-level shared representations in a non-linear way. This major strength is exploited in the current inventive system and usage, where fixed, closed tasks (e.g., paragraph reading, counting, or phoneme pronunciation tasks) are evaluated using feature extractors with optimal specific questions, while open-ended tasks (questions) are evaluated in both audio and text modalities while optimizing losses. Autoencoders are a type of neural network that can be used for feature fusion of speech data. Autoencoders can be used for multimodal feature fusion by combining data from multiple modalities (e.g., audio and textual representations) into a single joint representation. This can be done by training an autoencoder to encode the input data into a low-dimensional space and then decoding it back to the original space. The representation encoded here is a compact feature representation that captures the bottleneck, or most important information, from the input data. A detailed overview of the autoencoder architecture used for multimodal feature fusion of speech data 1. Collect and pre-process speech and other modality data. 2. Build an autoencoder architecture with an encoder and decoder. 3. The encoder takes input data from speech or other modalities and compresses it into a lower dimensional representation (the bottleneck or latent representation). 4. The decoder takes the bottleneck representation and reconstructs the original input data. 5. Train the autoencoder by providing the input data from both modalities as input to the encoder and using the original input data as the target output for the decoder. 6. After training, the bottleneck representation is used as the fused feature representation for both modalities. 7. This fused feature representation can then be used for more advanced analysis such as classification and clustering. Intermediate fusion strategies leverage prior knowledge to learn marginal representations for each modality, discover intra-modality correlations, and then either learn joint representations depending on their content or perform predictions directly. Although intermediate fusion strategies can mitigate dimensional imbalance between modalities by forcing marginal representations to be similar in scale, excessive reduction of the dimensionality of larger modalities can lead to significant loss of important information when the imbalance is very large. However, if the input features of the lower-dimensional modalities are selected based on prior knowledge, the imbalance does not necessarily lead to poor performance. Each speech task in the current invention dataset contributes very specific emotional derivatives. For example, the words in paragraph 1 are composed of mostly positive emotion words combined with neutral emotion, which leads healthy individuals to have positive intentions when speaking. That is, the words in the paragraph are spoken combined with different emotions (positive, negative, neutral), which allows us to capture the emotional information associated with the acoustic information, which is an important indicator to detect the differences in phonemes and emotional information between healthy and patients. To use different questions in the same model, we assign each question a question embedding, which is a 0-N vector. This question-specific feature extractor knows in advance which word, phoneme, and syllable level embeddings need to be extracted from the data, and are forced to align for mid-level feature fusion. Prior knowledge can come from linguists and phonetics experts about the specific tasks and the order of tasks validated from a model evaluation process, which is performed by developing an integrated gradient leveraging SHAP values ​​over the raw speech signal. FIG. 12A shows, in a non-limiting exemplary embodiment, a graph for at least one type of corresponding task (question), where the original label is “healthy”, and the graph shows phonemes and segments that are positively correlated with the model predictions. Areas highlight linguistic utterances that are characterized by high activation in the audio file. FIG. 12B shows, in a non-limiting exemplary embodiment, a graph for at least one type of task (question) with an original label of “depression” and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the second user (start time 0 seconds). FIG. 12C shows, in a non-limiting exemplary embodiment, a graph for at least one type of task (question) where the original label is “healthy” and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for a third user (start time 0 seconds). FIG. 12D shows, in a non-limiting exemplary embodiment, a graph for at least one type of task (question) with an original label of “depression” and a graph showing phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the fourth user (start time 1 second). FIG. 12E shows a graph for at least one type of task (question), in a non-limiting exemplary embodiment, where the original label is “healthy” and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the fifth user (start time 0 seconds). FIG. 12F shows, in a non-limiting exemplary embodiment, a graph for at least one type of task (question) with an original label of “depression” and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the sixth user (start time 0 seconds). FIG. 12G shows, in a non-limiting exemplary embodiment, a graph for at least one type of task (question), where the original label is “healthy”, and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the seventh user (start time 10 seconds). FIG. 12H shows a graph for at least one type of task (question) in a non-limiting exemplary embodiment, where the original label is “depression”, and the graph shows phonemes and segments that are positively correlated with the model prediction. The area highlights verbal utterances with high activation characteristics for the eighth user (start time 0 seconds). The mathematical formulation of the autoencoder-based multi-modal multi-question-input fusion architecture can be expressed as follows: Let X1, X2, X3, ..., Xn be the n modalities (speech, text, etc.) and Q1, Q2, Q3, ..., Qm be the m questions. Each question is multiplied by the learnable degree of the encoded matrix based on the question type. The encoder function E maps the n modalities and m questions into a low-dimensional representation Z. Z = E(X1, X2, X3, ..., Xn, Q1, Q2, Q3, ..., Qm) The decoder function D is Map the reduced dimensional representation Z back to the original input space. X'1, X'2, X'3, ..., X'n = D(Z) Autoencoders are typically trained to minimize the reconstruction error between the original input and the decoder output by leveraging a loss function such as mean squared error (MSE) or cross-entropy. L = 1 / nm Σ (X'i - Xi)^2 or L = - 1 / nm Σ Xi * log(X'i) The low-dimensional representation Z is a fused feature representation of the n modalities and m questions, which can be used for further analysis such as classification and clustering. Additionally, the encoder degree is used as a feature extractor for the downstream task of depression classification. Autoencoder-based feature fusion is obtained in the following way: First, hidden representations for the audio and text inputs are obtained using HuBERT and BERT models, respectively, and psychomotor delay features. Since the dataset contains similar answers to a large number of questions, the degree representing the question ID is also fed into the overall model. The question ID degree is element-wise multiplied to the hidden text feature. The resulting text and audio embeddings are concatenated by forced alignment, and this concatenated feature is used to train an autoencoder model with the goal of obtaining the fused representation from the bottleneck layer. The learned autoencoder degrees (up to the bottleneck layer) are loaded into one model and a classification feature head is connected to the bottleneck layer to train the classification model. In some embodiments, the subject (user) accesses the data from the stimulus output module (202) through a client device (112, 114, 116, 118) in response to the response receiving module (204) and records their response. In at least one embodiment, the various vector engines (232, 234, 236, 238) are configured as part of a network server (100) and are further communicatively connected to an emotion recognition module (300) and an automatic speech recognition module (400). In a preferred embodiment, the answers at the client devices (112, 114, 116, 118) are sampled at a sampling rate of 16 kHz and then transmitted back to the network server (100), which transmits them to the emotion recognition module (300). The emotion recognition module (300) generates multiple emotion classes through one or more emotion vectors, alternately combines them with one or more stimuli, and outputs them with their respective confidence scores. If the emotion does not match the target elicited emotion at a pre-defined confidence score, the user is asked to respond again. In some embodiments, the first pre-defined stimulus and the first pre-defined emotion vector are presented to the user through the stimulus output module (202), and the answer and / or associated answer vector are recorded through the answer receiving module (204). In some embodiments, the second pre-defined stimulus and the first pre-defined emotion vector are presented to the user through the stimulus output module (202), and the answer and / or associated answer vector are recorded through the answer receiving module (204). Thereby, certain predefined stimuli are composed of one or more predefined emotion vectors, different combinations of stimuli and emotion vectors are presented to the user through the stimulus output module (202), and answers and associated answer vectors are recorded through the answer receiving module (204). In a preferred embodiment, responses at the client devices (112, 114, 116, 118) are sampled at a sampling rate of 16 kHz and transmitted back to the network server (100), which transmits them to the automatic speech recognition module (400). The response data is also transcribed using automatic speech recognition technology. In at least one embodiment, the response data, associated transcripts, and recognized emotions are stored on the server in a training storage or training database (122). In at least one embodiment, the iteration and prioritization module is configured to prioritize the stimuli based on predefined rules applied to the received answers, such as distinguishing between different emotional stimuli and neutral stimuli, presenting happiness before sadness, etc. In such case, the stored data should reflect the order of questions asked with respect to the stimuli. In at least one embodiment, the networked server (100) is communicatively connected to a multimodal mental health classifier (500) (the autoencoder described above), which is an artificial intelligence (AI) module configured to be trained to classify mental health issues based on speech recordings and text responses. Artificial Intelligence ("AI") modules may be trained on structured and unstructured datasets of stimuli and answers. Training may be performed using supervised learning, unsupervised learning, or a combination of both. Machine Learning (ML) and AI algorithms may be utilized to learn from the various modules. The AI ​​module may learn rules (defined by confidence scores) about the user's mental health status by querying the vector engine to identify responses to stimuli, order of stimuli, response delays, etc. Deep learning models, neural networks, deep belief networks, decision trees, genetic algorithms, and other ML and AI models may be used alone or in combination to learn from the solution grid. The Artificial Intelligence ("AI") module may consist of a sequential combination of CNNs (Convolutional Neural Networks) and LSTMs (Long Short-Term Memory). By capturing high-level representations of the raw waveforms and short-term and long-term temporal variations, it is used to accurately build relationships between vocal biomarkers and language models specific to users (subjects) suffering from a specific mental health condition (depression). The model encodes temporal cues associated with depression manifested by motor cortex involvement, taking into account deep versus shallow features. As mentioned in the data collection section, data was collected based on a different order of questions and stimuli presented. The training process described above was run again on this specialized dataset to ensure that performance gains from induced emotions were not influenced by the leading questions. Given the current size of the data, no clear results or trends were observed. As more data is collected on different sequences and combinations of the system and its use, new findings may one day emerge. When this happens, comparing the results from these different "experiments" will allow for efficient selection of optimal stimulation sequences. Figure 13 shows the flowchart. Step 1: Present stimuli to the user to elicit an acoustic response. Step 2: Store emotion-related vectors associated with different stimuli. Step 3: Record the user's acoustic response. Step 4: Transcribe and record the text versions of the users’ recorded acoustic responses. Step 4a: Optionally, record the user's physiological responses. Step 5: Extract vectors from the user’s acoustic responses and further analyze them to classify the emotion recognition (obtained through an emotion detection model that produces multi-class emotions) in terms of emotion recognition information. Step 6: Emotion Recognition To perform emotion recognition in terms of information, we proceed with the analysis by extracting vectors from the user's text answers. Step 6a: If necessary, extract vectors from the user’s physiological answers to further analyze and perform emotion recognition (obtained through an emotion detection model that produces multi-class emotions) in terms of emotion recognition information. Step 7: Compare the emotion recognition information from the text responses and the emotion signal vectors from the training dataset by correlation with the vectors related to emotions associated with different stimuli to obtain the first spatial vector distance and the first graded vector difference. Step 7a.1: Optionally, first proceed with the analysis of the training dataset for a given stimulus and extract the most relevant emotion from the patient's responses to that stimulus using pre-defined percentiles and a confidence score threshold for that emotion. Step 7a.2: For a given stimulus, record in a database the emotion resulting from the user's answer and a confidence score threshold for that emotion. STEP 7a.3: At runtime, for each input, obtain a sentiment class and / or confidence score and compare it with the data recorded in STEP 7a.2. Step 8: Compare the emotion recognition information from the voice responses and the emotion signal vectors from the training dataset by correlation with the emotion vectors associated with different stimuli to obtain a second spatial vector distance and a second graded vector difference. Step 8a: Optionally, compare the emotion recognition information from the physiological responses and the emotion signal vectors from the training dataset in correlation with vectors relating to emotions associated with the different stimuli to obtain a third spatial vector distance and a third graded vector difference. Step 8b: Optionally, as an alternative to steps 7 and 8, use emotion recognition information from different modalities (acoustic, text, physical answers) to obtain an emotion output and then compare it with the emotion signal vector. Step 9: Intelligently integrate the first space vector distance, the first degree vector difference, the second space vector distance, and the second degree vector difference to obtain a first confidence score. Step 10: Optionally, intelligently combine the first spatial vector distance, the first degree vector difference, the second spatial vector distance, the second degree vector difference, the third spatial vector distance, and the third degree vector difference to obtain a second confidence score. Step 11: Repeatedly present the stimuli (step 1) and obtain a transcribed text answer from the further acoustic answer with respect to a first confidence score and / or a second confidence score based on the recognized emotion. Figure 14 shows a high-level flowchart of a voice-based mental health assessment with emotional stimuli. In a non-limiting exemplary embodiment, to capture the impact of induced emotions (e.g., stimuli presented through multimedia output stimuli) on the performance of the invented system, a model is trained for each induced emotion leveraging a given dataset. A comparison (which can be performed using either n-fold cross-validation or a dedicated validation set) is then conducted to compare the performance of different models, and the top three questions (stimuli) that performed best are selected. An alternative approach would be to train a model on all data combined, and then run the model on a validation set consisting of answers to only one stimulus. A single measure (e.g., AUROC) is then used to compare the performance of the models for each stimulus, and the top three stimuli are selected. The user answers the stimuli one by one in his / her favorite environment by using each client device. The system and its usage of the present invention performs some pre-processing and validation (e.g., vocal activation, loose length check, etc.) for each received answer, and then passes it through the emotion recognition model (200) to the emotion recognition process. If the recognized emotion and its confidence score match the induced emotion, the system and its usage of the present invention proceeds to the next stage of questions. If the emotion does not match, the system and its usage of the present invention executes a process to prompt the user to answer appropriately by presenting the same stimulus again. This process continues. If the user attempts to answer the same stimulus but exceeds a certain number of times, and emotion recognition still fails, a similar alternative stimulus under the same induced emotion is utilized and presented. Also, if the number of answer attempts exceeds a certain threshold, the system and its usage of the present invention can proceed with the evaluation while warning the user that the collected data is unreliable. The collected responses and corresponding stimulus records are sent to a multimodal mental health classifier (500) which evaluates the current user's mental health status and associated risk index. In a preferred embodiment, classification results with a higher degree of latitude for sadness-inducing emotions are prioritized and weighted higher in the calculation of the final result, and an evaluation report is generated along with a transmission back to the user and provider in real-time voice or text (through the client device) or non-real-time (e.g., email, message, etc.). The results for all induced emotions (responses to stimuli) are recorded in a memory storage or database of evaluation results (124). Table 1 below shows the model performance metrics (averaged over 10 runs) for the top induced emotion questions. By leveraging induced emotions and emotion recognition, our system and its usage outperform other multimodal techniques, improving performance by over 10% when compared to models with hand-designed features. [Table 1] A technical advantage of the present invention is that the combined use of emotion elicitation and emotion recognition in processing vocal input from a user provides a quality voice-based mental health assessment and monitoring system with much greater sensitivity and specificity than has previously been possible. The technical advancement of the present invention is the provision of an optimal multimodal architecture for a speech-based mental health assessment system and its uses, which not only includes emotional cues but also combines linguistic speech and linguistic characteristics along with acoustic characteristics. Although certain specific examples have been disclosed in this detailed illustrative description, it will be apparent to those skilled in the art that various modifications may be made thereto without departing from the spirit of the invention and the scope of the following claims. It should further be clearly understood that the foregoing description is to be interpreted as purely illustrative of the present invention and not as limiting thereof.

Claims

1. A multimodal system for voice-based mental health assessment with emotional stimuli. The aforementioned system includes: - A task building module, which is configured to build tasks to capture the acoustic, linguistic, and emotional features of the user's voice; - a stimulus output module (202) configured to receive data from the aforementioned task construction module, said stimulus output module (202) including one or more stimuli to be presented to the user based on the constructed task, for inducing one or more types of triggers regarding the user's actions, said triggers being in the form of input answers, - an answer receiving module (204) (configured to present one or more stimuli to the user based on the task constructed from the aforementioned stimulus output module (202) and to receive answers corresponding to one or more formats thereto); - Functional modules include: o a feature construction module (configured to define, for each of the aforementioned constructed tasks, features defined in terms of a learnable heuristic-based evaluation task); o a feature extraction module configured to extract one or more defined features related to said constructed tasks from said received corresponding answers, using a learnable heuristic evaluation model taking into account at least one selected from the ranked constructed tasks; o a feature fusion module, configured to fuse two or more defined features to obtain a fused feature; - An autoencoder leverages the fused features mentioned above to define the relationship between: o The speech modality with respect to said feature fusion module works in conjunction with said answer receiving module (204) to extract high level features from said answer and output extracted high level text features. o The text modality on said feature fusion module works in conjunction with said answer receiving module (204) to extract high level features from said answer and output extracted high level speech features. The autoencoder is configured to receive extracted high-level text features and extracted high-level audio features in parallel from the audio modality and the text modality, and to output a shared expression feature dataset for emotion classification that correlates with the mental health assessment.

2. The system according to claim 1, wherein the constructed task is a pronunciation task or a writing task.

3. The system according to claim 1, wherein said task construction module includes a first ranking module for ranking said constructed tasks in order of difficulty, thereby assigning a first value (degree) to each constructed task.

4. The system according to claim 1, wherein the task construction module includes a first ranking module configured to rank the constructed tasks in order of difficulty, thereby assigning a first value to each constructed task, whereby the constructed tasks are stimuli corresponding to the analyzed answers and classified into one of the assessed valence levels (positive valence, negative valence, neutral valence).

5. 2. The system as recited in claim 1, wherein said task construction module includes a second ranking module configured for ranking the complexity of said constructed tasks in order of complexity, thereby assigning a second value to each constructed task.

6. The system according to claim 1, wherein said task construction module comprises a second ranking module, which is configured to create a data collection pool based on the ranked construction tasks by ranking the emotional expectations in terms of the complexity of the construction tasks and the answer vectors.

7. The system according to claim 1, wherein said constructed tasks are selected from a group of tasks (cognitive task of counting numbers within a predetermined time, task of pronouncing vowels within a predetermined time, task of pronouncing words containing voiced and unvoiced sounds within a predetermined time, task of reading words within a predetermined time, task of reading a paragraph within a predetermined time, task of reading a paragraph with phonemic and emotional complexity, task related to open-ended questions with emotional variations, task related to open tasks to be performed within a predetermined time).

8. The system of claim 1, wherein the constructed task includes one or more questions as stimuli, each of which is assigned a question embedding represented by a vector from 0 to N. Based on these question embeddings, a question-specific feature extractor is trained to extract word embeddings, phoneme embeddings, and syllable-level embeddings from the questions, and these extracted embeddings are forced to be aligned and mid-level feature fusion is performed.

9. The system of claim 1, wherein one or more of the stimuli are selected, the stimuli being selected from a group of stimuli consisting of audio stimuli, video stimuli, combined audio and video stimuli, text stimuli, multimedia stimuli, physiological stimuli, and combinations thereof.

10. 2. The system of claim 1, wherein the one or more stimuli include a stimulus vector tailored to elicit a text response vector, an audio response vector, a video response vector, a multimedia response vector, a physiological response vector, or any combination thereof, that responds to the stimulus vector.

11. 2. The system of claim 1, configured to analyze the one or more stimuli through a first vector engine (232) to identify constituent vectors and determine ranked base states that correlate with the stimulus vectors.

12. 2. The system of claim 1, wherein one or more of the aforementioned answers are selected from the group of answers consisting of audio answers, video answers, combined audio and video answers, text answers, multimedia answers, physiological answers, and combinations thereof.

13. The system of claim 1, wherein the one or more stimuli include response vectors that correlate to audio response vectors and / or video response vectors elicited in response to the stimulus vectors of the stimulus output module (202).

14. 2. The system of claim 1, configured to analyze the one or more stimuli through a first vector engine (232) to identify constituent vectors and determine ranked base states that correlate with the stimulus vectors.

15. The system as recited in claim 1, wherein said answer receiving module (204) includes a text reading module configured to perform a text reading task within a time period preconfigured by a user.

16. The system as claimed in claim 1, wherein said feature construction module is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction module for analysis of spoken responses.

17. 2. The system of claim 1, wherein the feature construction function is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction function, the feature construction function comprising: - Analyzes the voice using a set of 62 parameters. - Provides a 3 frame long symmetric moving average filter to smooth over time (this smoothing is performed in the voiced regions of the previous answer for pitch, jitter, and shimmer). - The arithmetic mean and coefficient of variation are applied as functions to the 18 low-level descriptors (LLDs) to generate 36 parameters. - Apply 8 functions to the volume. - Apply 8 functions to the pitch. - Apply 8 functions to the pitch. - Determine the Hammerberg Index. - Determine the spectral features (for all unvoiced segments, this is done by looking at the spectral tilts between 0-500Hz and 500-1500Hz). - From the answers given above, determine the temporal features in continuous voiced and unvoiced regions. - Determines Viterbi-based smoothing of the F0 contour (prevents errors from causing a single voiced frame to be dropped).

18. The system of claim 1, wherein the feature construction function is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction function comprising a set of low level descriptors (LLDs) for analyzing spectral, pitch and temporal features in the speech response, the features being selected from a group of features consisting of: - Mel-frequency cepstral coefficients (MFCCs) and their first and second derivatives. - Pitch and pitch variability. - Energy and energy entropy. - Spectral centroid, broadness, and flatness. Spectral tilt. Spectral roll-off. Spectral variability. - Zero crossing rate. - Shimmer, jitter and harmonic-to-noise ratio. Voicing probability (based on pitch) Temporal features such as loudness peak ratio, mean duration and standard deviation of continuous voiced and unvoiced regions

19. The system of claim 1, wherein the feature construction function is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction function, the feature construction function being comprised of a set of frequency-related parameters selected from the group consisting of: Pitch (measures the logarithmic fundamental frequency (F0) on a chromatic frequency scale, starting at 27.5 Hz (semitone 0)) Jitter (deviation in the length of each successive F0 interval) Formant 1, 2, 3 Frequencies (center frequencies for the first, second, and third formants) Formant 1 (bandwidth related to the first formant) - Energy related parameters. Amplitude-related parameters. Shimmer (the difference in peak amplitude between successive F0 intervals) Loudness (an estimate of perceived signal strength from the auditory spectrum) Harmonic-to-noise ratio (the ratio of the energy that contains harmonic components to the energy that contains noise-like components) - Spectral (balance) parameters Alpha ratio (ratio of the total energy between 50-1000 Hz and 1-5 kHz) Hammerberg Index (the ratio of the strongest energy peak in the 0-2 kHz region to the strongest peak in the 2-5 kHz region) Spectral tilt (the slope of the linear regression on the logarithmic power spectrum in two specified bands: 0 to 500 Hz and 500 to 1500 Hz) Relative energy of formants 1, 2, and 3 (the ratio of the energy associated with the spectral harmonic peaks at the center frequencies of the first, second, and third formants to the energy associated with the spectral peak at the fundamental frequency (F0)) - Harmonic difference H1-H2 (the ratio of the energy related to the first harmonic (H1) of the fundamental frequency (F0) to the energy related to the second harmonic (H2)) Harmonic difference H1‐A3 (the ratio of the energy associated with the first harmonic (H1) of the fundamental frequency (F0) to the energy associated with the highest harmonic in the range of the third formant (A3))

20. The system of claim 1, wherein the feature construction function is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction function utilizing Higher Order Spectral Analysis (HOSA) functions, which utilize two or more component frequencies to achieve bispectral frequencies.

21. The system of claim 1, wherein the feature construction function is a Geneva Minimal Acoustic Parameter Set (GeMAPS) based feature construction function utilizing Higher Order Spectral Analysis (HOSA) functions utilizing two or more component frequencies to achieve a bispectrum frequency, the bispectrum utilizing a third accumulator to analyze the relationship between frequency components in the signal and examine nonlinear signals related to the answer.

22. The system of claim 1, wherein said feature extraction function fuses high level feature embeddings by leveraging mid-level fusion.

23. The system according to claim 1, wherein the aforementioned feature extraction function includes a function capable of extracting dedicated linguistic features and can also learn the degree of each stimulus, making it highly suitable for various linguistic tasks.

24. The system according to claim 1, wherein said feature extraction function includes a function capable of extracting dedicated emotional features and can also learn the degree of each stimulus, making it highly suitable for various emotional tasks.

25. The system as recited in claim 1, wherein said feature fusion module comprises a speech module configured to perform high-level feature fusion and to classify sentiment from one or more answers by utilizing at least autoencoder based feature fusion.

26. The system of claim 1, wherein the feature fusion module includes a speech module configured to utilize at least autoencoder based feature fusion with high level feature fusion capabilities to classify emotions from one or more answers, and the speech modality is adapted to extract high level features from time-frequency domain relationships in the answers by utilizing question specific feature extraction capabilities and output the extracted high level speech features.

27. The system as recited in claim 1, wherein said feature fusion module includes a text module configured to provide high-level feature fusion capabilities and leverage at least autoencoder based feature fusion to classify sentiment from one or more answers.

28. The system of claim 1, wherein said feature fusion module includes a text module configured to extract high-level features and utilize at least autoencoder based feature fusion to classify sentiment from one or more answers, and wherein the text modality utilizes a bidirectional long short-term memory network and an attention mechanism to simulate intra-modality dynamics and output extracted high-level text features.

29. The system according to claim 1, wherein said feature fusion module comprises a speech module. - A module that uses extracted features from the pre-trained model in acoustic feature embedding. - A module that compares the previously extracted features over the spectral domain. - A module for determining the characteristics of vocal tract coordination. - A module for determining features in replicate quantification analysis. - A module for determining features relating to the number of bigrams and the bigram duration corresponding to the vocal landmarks. - A module that fuses the aforementioned features in an autoencoder.

30. 2. The system of claim 1, wherein the feature extraction functionality includes an audio landmark extraction functionality configured to determine event markers associated with the answer, the determination of the event markers being correlated with positions on a timeline relative to acoustic events from the answer, including determining timestamp boundaries indicative of abrupt changes in the acoustic answer, and is performed in a frame-independent manner.

31. The system of claim 1, wherein the feature extraction function includes a speech landmark extraction function configured to identify event markers associated with the response, each event marker having a start value and an end value, selected from a group of landmarks consisting of glottal-based landmarks (g), periodicity-based landmarks (p), sonorant-based landmarks (s), fricative-based landmarks (f), voiced fricative-based landmarks (v), and burst-based landmarks (v), each of which is used to identify points in time at which different abrupt articulatory events occur, which correlate with abrupt changes in power in multiple frequency ranges and multiple time scales.

32. The system of claim 1, wherein the autoencoder comprises a multimodal, multiquestion input fusion architecture including: - One or more encoders that map one or more of the aforementioned specific features, in combination with a task type, to a low-dimensional representation (each task is multiplied by a learnable degree encoding matrix based on the task type, and these degrees are correlated with mental health ratings). - one or more decoders that map said one or more particular features to said lower dimensional representation (configured to output a mental health rating); - The aforementioned autoencoder is trained to minimize the reconstruction error between the input task and the decoder's output by leveraging a loss function.

33. Multimodal uses for voice-based mental health assessment with emotional stimuli are provided, including: - Building tasks to capture the acoustic, linguistic, and emotional characteristics of a user's voice. - receiving data containing one or more stimuli based on the constructed task and generating triggers that induce one or more types of user actions by presenting the task to the user (the triggers being in the form of input answers); - presenting one or more stimuli to the user based on said constructed task and receiving corresponding responses in one or more formats. - Defining, for each of the aforementioned constructed tasks, features defined in terms of a learnable heuristic-type evaluation task. - extracting one or more defined features related to said constructed tasks from said received corresponding answers by utilizing a learnable heuristic evaluation model that considers at least one selected from the ranked constructed tasks. - Fusing two or more defined features to obtain a fused feature. - Leveraging the aforementioned fused characteristics, define the relationships between: o The speech modality with respect to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level text features. o The text modality relating to said feature fusion module works in conjunction with said answer receiving module to extract high level features from said answer and also output extracted high level speech features. The step of defining the relationship is configured to receive extracted high level text features and extracted high level audio features in parallel from the audio modality and the text modality, and output a shared expression feature dataset for emotion classification correlated with the mental health assessment.

34. 34. The method of claim 33, wherein the constructed task includes one or more questions as stimuli, each of which is assigned a question embedding represented by a vector from 0 to N. Based on the question embeddings, a question-specific feature extractor is trained to extract word embeddings, phoneme embeddings and syllable level embeddings from the questions, and the extracted embeddings are forced to be aligned and mid-level feature fusion is performed.

35. 34. The method of claim 33, wherein the step of defining the relationships comprises a multi-modal and multi-question input fusion architecture including: - Through an encoder, one or more of said specific features are combined with a task type and mapped to said low-dimensional representation (each task is multiplied by a learnable degree encoding matrix based on the task type, and these degrees are correlated with mental health assessments). - via a decoder, mapping said one or more particular features to said low dimensional representation and outputting a mental health assessment. - By utilizing a loss function, it is trained to minimize the reconstruction error between the input task and the decoder's output.

36. 34. The method of claim 33, wherein the step of extracting high level speech features includes the step of extracting high level features from time-frequency domain relationships in the response, thereby outputting the extracted high level speech features.

37. 34. The method of claim 33, wherein said extracting high-level text features comprises simulating intra-modal dynamics by using a bidirectional long short-term memory network and an attention mechanism to output the extracted high-level text features.

Images