Multi-model-based user emotion monitoring and intervention methods, devices, and computer program products

Abstract

This invention provides a multi-model-based user emotion monitoring and intervention method, device, and computer program product. By training multiple models to simulate the communication style of the target object, it simultaneously generates a virtual avatar matching lip movements, facial expressions, and gestures, providing users with more emotionally supportive interactions. Simultaneously, it introduces an emotion analysis model for emotion recognition, early warning, and risk level analysis. When abnormal emotions are identified, it uses a second model learned from a psychological intervention strategy library and a psychological knowledge graph to conduct reassuring dialogue. It can detect and respond to sudden emotional abnormalities in real time and intervene with professional reassuring dialogue.

Description

Technical Field

[0001] This invention relates to the field of emotion management technology, and in particular to a multi-model-based user emotion monitoring and intervention method, device, and computer program product. Background Technology

[0002] Currently, emotion detection and intervention technologies can be mainly categorized as follows: Simple dialogue systems based on natural language processing (NLP) and deep learning, such as chatbots and virtual assistants, can provide basic emotional support, but lack depth in emotional interaction, cannot simulate the dialogue style of specific individuals, and lack psychological intervention functions; Human psychological intervention systems rely on human review of dialogue content and psychological assessment and intervention through the intervention of psychology experts, but suffer from problems such as high response latency, inability to process massive amounts of user data in real time, and difficulty in dealing with acute psychological crisis scenarios; Single-model systems rely on a single model to complete all functions (such as emotional companionship or psychological monitoring), which are functionally limited and prone to conflict, unable to balance the needs of "emotional companionship" and "rational intervention," and have high model complexity, making them difficult to optimize. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a user emotion monitoring and intervention method, device and computer program product based on a multi-model approach to eliminate or improve one or more defects in the prior art, and solve the problem that the prior art lacks emotion support in the interaction process and cannot efficiently implement emotion monitoring and intervention.

[0004] One aspect of the present invention provides a multi-model-based user sentiment monitoring and intervention method, the method comprising the following steps:

[0005] Based on the first model obtained by simulating the dialogue style of the target object, a multimodal dialogue interaction is conducted with the specified user. The dialogue information generated by the first model is converted into speech information according to the voiceprint and style characteristics of the target object through a preset text-to-speech model. A preset image generation model is used to generate a virtual image with synchronized lip movements, facial expressions and / or actions according to the image of the target object based on the speech information.

[0006] Based on a pre-trained sentiment analysis model, the system performs sentiment recognition on the information input by the specified user in the dialogue interaction, and combines keyword detection to issue early warnings of abnormal sentiments and identify risk levels.

[0007] When abnormal emotions are identified, the corresponding intervention mode is selected according to the risk level, and a reassuring dialogue is conducted based on the second major model obtained from the psychological intervention strategy library and the psychological knowledge graph.

[0008] In some embodiments, the pre-training step of the first large model includes:

[0009] Obtain sample data containing texts expressed by the target object during social media, private communications, interview transcripts, or speeches, and perform noise reduction, segmentation, and privacy anonymization.

[0010] The sample data is labeled with sentiment and style as tags to construct the first training sample set;

[0011] The first training sample set is used to train the Transformer-based dialogue generation model. The parameters of the dialogue generation model are updated by using the cross-entropy loss, sentiment consistency loss and style consistency loss of the generated text. A regularization strategy based on dropout and weight decay is added to obtain the first large model.

[0012] In some embodiments, the training steps of the preset text-to-speech model include:

[0013] Obtain sample data containing the speech of the target object in the social media, the private communication, the interview record or the speech, perform noise reduction, text transcription, speech segmentation, phoneme alignment and voiceprint feature extraction on the speech, add prosodic annotation, and construct a second training sample set;

[0014] A generator based on a speech generation model and a discriminator based on a convolutional neural network are constructed. The generator is used to generate simulated speech from text transcription, and the discriminator is used to identify whether the input speech belongs to the simulated speech. The generator and the discriminator are trained using the second training sample set based on adversarial learning. The generator is updated by combining the simulated speech reconstruction loss, the voiceprint similarity loss, the prosodic annotation loss, and the discriminator's discrimination loss. The discriminator is also updated based on the discrimination loss. The generator obtained through adversarial learning is used as the preset text-to-speech model.

[0015] In some embodiments, the speech generation model is a Tacotron 2 or FastSpeech 2 model; the prosodic annotation includes fundamental frequency trajectory, speech rate, pause frequency, and accent patterns based on RMS value mutation markers.

[0016] In some embodiments, the preset image generation model adopts a lip-sync generation model based on FaceFormer or Wav2Lip; or an expression and motion generation model based on AD-NeRF or GeneFace; or a full-body motion synchronization generation model based on VOCA.

[0017] In some embodiments, the training steps of the sentiment analysis model include:

[0018] Obtain a third training sample set, in which each sample contains user dialogue text, synchronized speech waveforms and facial expression videos, and add real emotion labels;

[0019] An initial neural network is obtained, including text branch, speech branch, vision branch, cross-modal fusion layer, and fully connected layer. The text branch extracts semantic features of the dialogue text based on the RoBERTa model; the speech branch extracts acoustic temporal features of the synchronized speech waveform based on Wav2Vec 2; the vision branch extracts facial expression space features of the facial expression video based on the VisionTransformer model; the cross-modal fusion layer dynamically weights and fuses the semantic features, acoustic temporal features, and facial expression space features through a gated attention mechanism to obtain a joint emotion representation vector; the fully connected layer outputs an emotion prediction result for the user based on the joint emotion representation vector; and the initial neural network is updated with parameters based on the deviation between the emotion prediction result and the actual emotion to obtain the emotion analysis model.

[0020] In some embodiments, the training steps of the second large model include:

[0021] A psychological intervention database is constructed, which includes psychotherapy dialogue records, crisis intervention scripts and reassuring dialogue templates annotated by experts, as well as structured intervention paths constructed based on a psychological knowledge graph. The structured intervention paths are marked with strategy priorities corresponding to different risk levels.

[0022] The second large model is obtained by adapting and training the psychological intervention database based on the large language model.

[0023] On the other hand, the present invention provides a user emotion monitoring and intervention device based on a multi-model, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0024] On the other hand, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0025] On the other hand, the present invention provides a computer program product, including a computer program / instruction, characterized in that the computer program / instruction implements the steps of the above-described method when executed by a processor.

[0026] The beneficial effects of the present invention are at least as follows:

[0027] The multi-model-based user emotion monitoring and intervention method, device, and computer program product described in this invention simulates the communication style of the target object by training multiple models, simultaneously generating virtual avatars that match lip movements, facial expressions, and actions, providing users with more emotionally supportive interactions. It also introduces an emotion analysis model for emotion recognition, early warning, and risk level analysis. When abnormal emotions are identified, a second major model learned from a psychological intervention strategy library and a psychological knowledge graph is used to conduct reassuring dialogue. This allows for real-time detection and response to sudden emotional abnormalities, and intervention with professional reassuring dialogue.

[0028] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0029] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0031] Figure 1 This is a flowchart illustrating a multi-model-based user emotion monitoring and intervention method according to an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of the workflow of an intelligent psychological intervention system combining multi-model collaboration, as described in another embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0034] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0035] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0036] Users need a real-time, automated psychological monitoring and intervention system that can trigger alerts and provide intervention the moment a user's emotions spiral out of control, compensating for the limitations of human services. Existing emotional interaction systems, such as chatbots and virtual assistants, are typically based on natural language processing (NLP) and deep learning technologies, enabling simple conversations and providing a certain level of emotional support. However, these systems have significant functional limitations, such as insufficient depth of emotional interaction, lack of psychological intervention, and inability to meet the emotional needs of family and friends. Traditional psychological intervention systems rely on manual review, which cannot meet the needs of acute psychological crisis scenarios.

[0037] Therefore, this invention provides a multi-model-based user emotion monitoring and intervention method, such as... Figure 1 As shown, the method includes the following steps S101~S103:

[0038] Step S101: Based on the first model obtained by simulating the dialogue style of the target object, conduct multimodal dialogue interaction with the specified user. The dialogue information generated by the first model is converted into speech information according to the voiceprint and style characteristics of the target object through a preset text-to-speech model. A preset image generation model is used to generate a virtual image with synchronized lip movements, facial expressions and / or actions according to the image of the target object based on the speech information.

[0039] Step S102: Based on the pre-trained sentiment analysis model, perform sentiment recognition on the information input by the specified user in the dialogue interaction, and combine keyword detection to issue an early warning of abnormal sentiment and identify the risk level.

[0040] Step S103: When abnormal emotions are identified, select the corresponding intervention mode according to the risk level, and conduct a reassuring dialogue based on the second major model obtained from the psychological intervention strategy library and the psychological knowledge graph.

[0041] In step S101, a multimodal dialogue interaction is conducted with a designated user by simulating the dialogue style of the target object. First, the dialogue style of the target object is accurately simulated using the first major model, enabling it to communicate with the user like the target object. During the practice, the target object can be an object that the designated user can emotionally rely on; for example, a close friend or relative can be used, or a preset public figure can be used. During the dialogue, to convert text information into more vivid speech information, a preset text-to-speech model is introduced. This model not only synthesizes the speech content generated by the first major model but also optimizes it based on the voiceprint and style characteristics of the target object, ensuring that the generated speech highly matches the target object in terms of timbre and intonation. Simultaneously, to enhance the visual effect of the dialogue interaction, a preset image generation model is used. Based on the generated speech information and combined with the image characteristics of the target object, a virtual image synchronized with the speech is generated in real time, including accurate lip movements, expressive facial expressions, and natural movements, providing the user with a richer and more three-dimensional interactive experience.

[0042] In some embodiments, the pre-training step of the first large model includes steps S201 to S203:

[0043] Step S201: Obtain sample data containing texts expressed by the target subject in social media, private communications, interview transcripts, or speeches, and perform noise reduction, segmentation, and privacy anonymization.

[0044] Step S202: Label the sample data with sentiment and style as tags to construct the first training sample set.

[0045] Step S203: Train the Transformer-based dialogue generation model using the first training sample set, update the parameters of the dialogue generation model using the cross-entropy loss, sentiment consistency loss and style consistency loss of the generated text, add a regularization strategy based on dropout and weight decay, and obtain the first large model.

[0046] In step S201, social media includes platforms such as Weibo, WeChat Moments, and Douyin, used to capture everyday communication styles. Private communications include WeChat chat logs, emails, and letters. Interview transcripts include transcripts of interviews in books, magazines, and podcasts to understand how in-depth viewpoints are expressed. Texts from speeches include speech drafts to learn the wording and logic used in formal settings. After acquiring the data, noise reduction processing is required to remove irrelevant information, errors, and noise interference, such as error correction and cleaning up irrelevant symbols, to ensure the validity and accuracy of the data. Next, the data is segmented, dividing long texts into appropriately sized segments according to semantics or logic so that the model can better understand and learn. Privacy desensitization is also a crucial step, requiring the desensitization of personal information and sensitive content in the data to protect privacy while ensuring the legality and security of the data, providing a reliable data foundation for subsequent model training.

[0047] Denoising of sample data includes removing irrelevant symbols, correcting typos, and standardizing formatting. Segmented processing preserves the original sentence structure and avoids disrupting the rhythm of the language. Privacy desensitization includes anonymizing sensitive names, locations, or events.

[0048] In step S202, the main task is to label the sample data after denoising, segmentation, and privacy anonymization to construct the first training sample set. The labeling work includes two dimensions: sentiment and style. For sentiment labeling, based on the text content, the emotions expressed by the target object are determined, such as happiness, sadness, and anger, and corresponding sentiment labels are assigned. Style labeling focuses on the target object's language style, such as humor, seriousness, and gentleness, labeling its stylistic features. More specific quantification can be achieved through word frequency analysis and sentence structure annotation. A combination of manual and automated labeling tools improves labeling efficiency and accuracy. The completed first training sample set not only contains the target object's text data but also its sentiment and style information, providing rich and accurate data support for the first major model to learn the target object's language features.

[0049] In step S203, the Transformer-based dialogue generation model is trained using the first training sample set to generate the first main model. The dialogue generation model can use GPT-3 / 4 or PaLM, while lightweight options include GPT-2 Medium or DistilGPT-2. During training, cross-entropy loss, sentiment consistency loss, and style consistency loss of the generated text are used as loss functions. Cross-entropy loss measures the difference between the generated text and the real text, enabling the model to generate text that more closely resembles the expression of the target object. Sentiment consistency loss ensures that the sentiment of the generated text is consistent with the sentiment label of the target object, while style consistency loss ensures that the style of the generated text matches the style characteristics of the target object. Simultaneously, a regularization strategy based on dropout and weight decay is added to prevent overfitting and improve its generalization ability. Through continuous iterative optimization and updating of model parameters, the first main model that can accurately simulate the dialogue style of the target object is finally obtained, providing core support for subsequent multimodal dialogue interactions.

[0050] In some embodiments, the training steps of the preset text-to-speech model include steps S301-S302:

[0051] Step S301: Obtain sample data containing the target object's speech in social media, private communications, interview transcripts, or speeches; perform noise reduction, text transcription, speech segmentation, phoneme alignment, and voiceprint feature extraction on the speech; add prosodic annotations; and construct a second training sample set.

[0052] Step S302: Construct a generator based on a speech generation model and a discriminator based on a convolutional neural network. The generator is used to generate simulated speech from text transcription, and the discriminator is used to identify whether the input speech belongs to the simulated speech. The generator and discriminator are trained using the second training sample set based on adversarial learning. The generator is updated by combining the simulated speech reconstruction loss, voiceprint similarity loss, prosodic annotation loss and the discriminator's discriminant loss. The discriminator is updated based on the discriminant loss. The generator obtained through adversarial learning is used as the preset text-to-speech model.

[0053] In step S301, a basic audio sample is obtained, requiring at least 3-5 hours of clean speech (sampling rate ≥16kHz), covering scenarios such as daily conversation, reading aloud, and impromptu speeches, including different emotional states. Background noise is removed using RNNoise or Demucus, and the audio is segmented into 2-10 second segments based on VAD (Voice Activity Detection), with phoneme boundaries marked using Montreal ForcedAligner. Mel-frequency cepstral coefficients (MFCC) are used to extract voiceprint features. First, the speech signal is preprocessed, including framing and windowing; then, a Fast Fourier Transform is performed to convert the signal from the time domain to the frequency domain; next, the spectrum is filtered using a Mel filter bank, and the output energy of each filter is calculated; finally, the logarithm of the filtered energy is taken, and a Discrete Cosine Transform is performed to obtain the MFCC coefficients. Alternatively, perceptual linear prediction (PLP) or linear discriminant analysis (LDA) algorithms can be used to extract voiceprint features. The Praat script was used to extract intonation fluctuation patterns, calculate the number of syllables per second, pause frequency, and energy changes of emphasized words using RMS value mutations to obtain prosodic annotation.

[0054] In step S302, VITS (Variational Inference Text-to-Speech) or FastSpeech 2 + HiFi-GAN can be used as the generator and discriminator architecture in this embodiment. The VITS model is an end-to-end speech synthesis model based on conditional variational autoencoders (cVAE) and adversarial learning. It mainly consists of a posterior encoder, a prior encoder, a decoder, a discriminator, and a random duration predictor. By combining variational autoencoders, adversarial learning, and normalized flow techniques, it can generate natural, diverse, and expressive speech from text. It also proposes a random duration predictor to synthesize speech with different rhythms from the input text, expressing natural one-to-many relationships. The FastSpeech2+HiFi-GAN model is an end-to-end speech synthesis model that combines FastSpeech 2 and HiFi-GAN. FastSpeech 2 includes a text encoder, variance adapter, and speech decoder, which can convert input text into Mel spectra and model and adjust features such as duration, pitch, and energy of speech. The HiFi-GAN generator and discriminator convert Mel spectra into high-quality speech waveforms. The model as a whole achieves efficient and high-quality synthesis from text to speech.

[0055] In some embodiments, the speech generation model may be a Tacotron 2 or FastSpeech 2 model; prosodic annotation includes fundamental frequency trajectory, speech rate, pause frequency, and accent patterns based on RMS value mutation markers.

[0056] In some embodiments, the preset image generation model adopts a lip-sync generation model based on FaceFormer or Wav2Lip; or an expression and motion generation model based on AD-NeRF or GeneFace; or a full-body motion synchronization generation model based on VOCA.

[0057] Step S102 focuses on monitoring the emotional state of designated users during dialogue interactions and issuing early warnings for abnormal emotions. First, a pre-trained emotion analysis model is used to identify the emotions in the user's input. This model, trained on extensive data, accurately captures emotional elements contained in text, speech, and other information, determining the user's current emotional category, such as happiness, sadness, anger, or anxiety. Simultaneously, keyword detection technology is used to further analyze the text content in depth, filtering out keywords with potential risks, such as words expressing extreme emotions or potentially implying dangerous behavior. Once key information related to abnormal emotions is detected, an early warning mechanism is quickly activated. Based on preset rules and standards, the risk level of the abnormal emotion is accurately identified, providing crucial evidence for subsequent intervention measures and ensuring timely and effective response to potential emotional problems of the user.

[0058] In some embodiments, the training steps of the sentiment analysis model include steps S401-S402:

[0059] Step S401: Obtain the third training sample set, in which each sample contains user dialogue text, synchronized speech waveforms and facial expression videos, and add real emotion labels.

[0060] Step S402: Obtain the initial neural network, including text branch, speech branch, vision branch, cross-modal fusion layer, and fully connected layer. The text branch extracts semantic features of the dialogue text based on the RoBERTa model; the speech branch extracts acoustic temporal features of the synchronized speech waveform based on Wav2Vec 2; the vision branch extracts facial expression space features of the facial expression video based on the Vision Transformer model; the cross-modal fusion layer dynamically weights and fuses semantic features, acoustic temporal features, and facial expression space features through a gating attention mechanism to obtain a joint emotion representation vector; the fully connected layer outputs the emotion prediction result for the user based on the joint emotion representation vector; the initial neural network is updated with parameters based on the deviation between the emotion prediction result and the actual emotion, resulting in the emotion analysis model.

[0061] In step S103, after identifying abnormal emotions in the user through the above steps, the intervention phase begins. Based on a pre-determined risk level, the most suitable intervention model is selected from multiple options. This intervention model is a second major model built upon in-depth learning of a psychological intervention strategy library and a psychological knowledge graph. The psychological intervention strategy library encompasses a rich variety of psychological intervention methods applicable to different emotional problems and risk levels; while the psychological knowledge graph systematically integrates professional knowledge and theories in the field of psychology, providing solid theoretical support for the development of intervention strategies. This second major model can flexibly utilize these psychological intervention strategies according to the user's specific situation and emotional state, generating targeted and soothing dialogue content to gently and professionally guide the user to alleviate abnormal emotions, help them adjust their mindset, and return to a more positive and stable emotional state, thereby achieving effective intervention in the user's emotions.

[0062] In some embodiments, the training steps of the second large model include steps S501-S502:

[0063] Step S501: Construct a psychological intervention database, which includes psychotherapy dialogue records, crisis intervention scripts and reassuring dialogue templates annotated by experts, as well as structured intervention paths built based on a psychology knowledge graph. The structured intervention paths are marked with the strategy priorities corresponding to different risk levels.

[0064] Step S502: Adapt and train the psychological intervention database based on the large language model to obtain the second large model.

[0065] The psychotherapy dialogue records can utilize CBT (Cognitive Behavioral Therapy) case studies; the psychological knowledge graph can employ the DSM-5 diagnostic criteria or emotion regulation theory; and the large language model can be based on models such as GPT-3.5 or LLaMA. Furthermore, the risk-level intervention model can be structured with a three-tiered response: low-risk interventions use open-ended questions to guide emotional catharsis; medium-risk interventions combine cognitive restructuring techniques with breathing and relaxation guidance; and high-risk interventions trigger an emergency protocol and transfer the patient to human intervention.

[0066] On the other hand, the present invention provides a user emotion monitoring and intervention device based on a multi-model, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0067] On the other hand, the present invention provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0068] On the other hand, the present invention provides a computer program product, including a computer program / instruction, characterized in that the computer program / instruction implements the steps of the above-described method when executed by a processor.

[0069] The present invention will be described in detail below with reference to a specific embodiment:

[0070] This embodiment provides an intelligent psychological intervention system that combines multi-model collaboration. The first model simulates the conversational style of family and friends (multimodal fusion of sound, image, and language) to provide emotional support to users. The second model monitors the user's emotions in real time, automatically triggering an intervention mechanism when potential psychological problems or even extreme risks are detected, taking over the conversation and alerting the administrator. The system is suitable for improving the real-time nature and humanistic care of mental health services. Figure 2 As shown, the system includes the following modules:

[0071] 1. Multimodal data acquisition module

[0072] Image acquisition and OCR recognition: Visual information such as facial features and body language of relatives and friends is extracted from uploaded image files (mp4 files with a sampling rate of 16kHz). For multimodal unification, facial and body features are mapped to a unified embedding space (e.g., a 1024-dimensional vector) by aligning temporal and spatial coordinates, and metadata annotation (emotion and scene labels) is used to enhance correlation. OCR technology is then used to recognize text content in the images (e.g., diary entries, scanned letters) to build a database of personal language styles.

[0073] Voice recording and RVC timbre analysis: Collect voice data from relatives and friends (such as audio recordings and phone records), extract timbre features through RVC (Real-time Voice Cloning) technology, and generate a high-fidelity voice cloning model. Supports real-time speech synthesis, simulating intonation, pauses, and emotional expression in natural conversation.

[0074] Video recording and digital avatar creation: Extract dynamic features such as facial expressions and movements of people through video analysis technology, and generate digital avatars by combining them with 3D modeling.

[0075] Text data collection and construction of a character's language style database: Text data is extracted from various channels, including WeChat chat history (exporting data from WeChat or exporting it as a large model training set via third-party methods), social media (through platform APIs, web scraping, or image recognition), and emails (email APIs), to analyze language style, common vocabulary, and topic preferences. Data is labeled using annotation tools to ensure consistency, combining automated initial screening (e.g., BERT sentiment prediction) with manual review. The focus is primarily on text annotation; therefore, speech is converted to text first. Annotated data is delivered in JSON or CSV format. Structured data is stored in a database, indexed by keywords and user IDs to improve retrieval efficiency. Relevant data is collected from public corpora and user-specific scenario data, categorized by sentiment and keywords (e.g., specific scenarios, high-frequency words), and expression accuracy is improved through hierarchical labeling and adversarial training. The data is then stored in JSON format or in a database.

[0076] 2. Data Processing and Vectorization Module

[0077] Data cleaning and format standardization: Remove noisy data (such as duplicate content and irrelevant information) and standardize text encoding formats (such as UTF-8). Denoise the speech data and use a VAD-like method to detect and remove silent segments, ensuring data quality. Video data should be as high-resolution as possible (e.g., 2K and above) to avoid jitter and distortion caused by compression. Select segments with normal lighting, clear facial features and limbs, and rich facial expressions. After generating the digital human avatar, delete the original data segments.

[0078] Privacy protection technologies include: anonymization to remove user identity information (such as name and contact information), retaining only emotional and behavioral characteristics; and data vectorization to transform raw data into vector representations (such as word vectors and voiceprint vectors) through feature extraction, avoiding the direct storage of sensitive content.

[0079] 3. Introduce the first major model to build an emotional companionship module.

[0080] Long-term memory database: Through the long-term memory database, the dialogue content is labeled and recorded to record user interaction feature data (such as names of people, places and their relationships), forming an interaction history and a personalized emotional profile, which is stored in vector format.

[0081] Dialogue Simulation: Based on an NLP model with a Transformer architecture, it learns the text, speech, and visual features of friends and family to generate responses that match their style. Combining multimodal output (such as simultaneous speech generation and digital avatar actions) enhances the realism of the dialogue.

[0082] Early warning monitoring: Sentiment analysis models (such as BERT + sentiment dictionary) are used to identify users' emotional states (such as depression, anxiety). When high-risk keywords (such as "despair") or abnormal behavioral patterns are detected, an early warning signal is triggered.

[0083] 4. Introduce the second major model to construct a psychological intervention module.

[0084] Training the model using a dataset of mental health professionals: Input historical dialogue data from professional mental health professionals and a database of psychological intervention strategies to train the model to identify crisis scenarios. Integrate psychological knowledge graphs (such as the DSM-5 diagnostic criteria and cognitive behavioral therapy steps).

[0085] Dynamic intervention strategy: Based on the user's emotional intensity and risk level, select the intervention mode (such as reassuring dialogue, distraction, or emergency contact with human customer service). Pre-set ethical constraints prevent the generation of inappropriate content (such as responses that escalate conflict).

[0086] Real-time intervention mechanism: Takes over the dialogue in the first major model, intervening as a "psychologist" to provide professional guidance (such as questioning guidance and mindfulness training suggestions). Simultaneously sends an alert to the administrator, triggering offline rescue procedures (such as contacting family members or medical institutions).

[0087] The training and deployment methods for the first major model are as follows:

[0088] We collected a large amount of historical communication data between users and their relatives and friends, denoised the data to remove noise and irrelevant information (especially special symbols), standardized the format to text data, and annotated it with sentiment and events.

[0089] The model is based on a Transformer-based dialogue generation model (such as the GPT-3 architecture) and includes an encoder-decoder structure.

[0090] During training, the data is split: 70% for the training set, 15% for the validation set, and 15% for the test set. Loss functions are constructed and their parameters are updated and optimized, including cross-entropy loss (to improve the accuracy of text generation), sentiment consistency loss (to penalize mismatches between generated responses and labeled sentiment tags), and style consistency loss (to ensure response styles are similar to historical data through comparative learning).

[0091] During the content generation and optimization process, user queries are input and processed in text or speech-to-text format; the sentiment analysis module predicts the sentiment tags of the current dialogue scenario, such as nostalgia; the model generates candidate responses (Top-K sampling, K=5), filters responses that do not conform to the style (such as those with similarity to historical data <0.6), and outputs the optimal response.

[0092] The model is encapsulated as a REST API to facilitate conversational services. The initial process involved repeated evaluation and optimization by internal technical testing.

[0093] The training and deployment methods for the second major model are as follows:

[0094] Model Building and Training: Data sources include social media comments, psychological counseling records, and relevant datasets, with a focus on cognitive-behavioral and psychodynamic approaches to ensure evidence-based support for intervention methods. Data sets containing various emotional states and psychological problems are collected, including but not limited to images, text, and audio. The data is split (e.g., training set: 70%, validation set: 15%, test set: 15%), and adversarial examples (e.g., noisy data generated by GANs) are considered. The labeled data is used to train an emotion monitoring model based on sentiment analysis and psychological knowledge. During training, the model's hyperparameters are adjusted to improve its accuracy in identifying users' emotional states and psychological problems.

[0095] Specific monitoring procedures and optimizations:

[0096] Based on keyword monitoring: such as the frequency of negative emotion keywords like sadness and grief, or the direct appearance of negative behavior keywords.

[0097] User profile analysis: The intervention strategy is dynamically adjusted according to the patient's age, cultural background, and personality traits, taking into account the main psychological characteristics of the current user's group and the patient's cultural background and values ​​in relation to the intervention.

[0098] Divide the intervention process into sections:

[0099] Primary prevention, or health promotion, targets the general population by enhancing psychological resilience through mental health education and stress management.

[0100] Secondary prevention, or preventive intervention, involves selective or guiding interventions for high-risk groups to reduce risk and reporting warnings.

[0101] The third level of prevention, namely psychotherapy, involves alleviating symptoms for individuals who have developed mental disorders through cognitive behavioral therapy (CBT), psychoanalysis, etc., and reporting and warning them to request timely medical care.

[0102] The ethical rules for setting up responses include: 1) The principle of harmlessness: Do not generate misleading advice or discriminatory content, especially if it generates negative thoughts or behaviors in users; these should be promptly stopped rather than supported. 2) The principle of autonomy: Users are required to retain their independent critical thinking abilities; do not make choices for them. 3) The principle of benefit: Responses must be based on scientific evidence and psychological theories.

[0103] Ethical rule implementation: RLHF (Reinforcement Learning Based on Human Feedback) is used, with human annotators scoring responses ethically, penalizing harmful responses and rewarding effective ones. When negative intent is detected, the model should automatically trigger human intervention or provide emergency contact information. An interpretability module explains the model's reasoning logic, ensuring users understand the source of the suggestions. A hybrid generation and validation architecture is used to verify the accuracy of responses by retrieving data from authoritative databases (such as PubMed).

[0104] The model is encapsulated as a REST API to provide dialogue services. At the same time, it monitors and selects secondary and tertiary prevention measures to promptly interrupt the dialogue with the first model and report the dialogue content.

[0105] Furthermore, a multi-model-based user emotion monitoring and intervention system is provided, including a multi-modal data acquisition module, a data processing and vectorization module, an emotional support module, and a psychological intervention module.

[0106] The multimodal data acquisition module is used to collect users' image, voice, video and text data, and extracts the visual features, timbre features and language style of relatives and friends through optical character recognition (OCR) and voice cloning (RVC) technologies; it also uses word vectors, voiceprint vectors and other technologies to transform the raw data into non-sensitive feature vectors.

[0107] The data processing and vectorization module cleans, unifies, and reduces noise in the multimodal data, and generates standardized feature vectors through privacy protection technologies (including anonymization and data vectorization).

[0108] The emotional companionship module is equipped with the first major model, which simulates the conversational style of relatives and friends based on deep learning models (such as the Transformer architecture). It interacts with users through multimodal output (voice, digital avatar actions, and text replies) and records the user's interaction history to form a personalized emotional profile.

[0109] The psychological intervention module, equipped with the second major model, monitors the user's emotional state in real time, detects potential psychological risks, and automatically takes over the conversation when the risk level reaches a threshold, providing professional psychological intervention strategies (such as cognitive behavioral therapy guidance or emergency contact with human customer service), while triggering system alerts.

[0110] The multimodal data acquisition module includes:

[0111] Image acquisition unit, used to extract facial features and body language of relatives and friends;

[0112] The voice recording unit clones the vocal characteristics of relatives and friends and generates a high-fidelity voice model through RVC technology;

[0113] The video analysis unit extracts the dynamic expressions and actions of relatives and friends to construct digital avatars;

[0114] The text acquisition unit extracts language style features from channels such as chat logs and social media.

[0115] The first major model's dialogue simulation function includes: a long-term memory database based on memory networks, dynamically updating user interaction history to form personalized emotional profiles;

[0116] The psychological intervention module also includes an emotion analysis module, which uses the BERT model or other relevant models in conjunction with an emotion dictionary to identify the user's emotional state (such as depression or anxiety).

[0117] The intervention strategies of the second major model include: selecting intervention modes (such as reassuring dialogue, attention diversion, or emergency warning) based on the intensity of the user's emotions and the level of risk; integrating psychological knowledge graphs (such as DSM-5 diagnostic criteria or cognitive behavioral therapy steps) to generate intervention scripts; and filtering inappropriate content through ethical constraints to avoid escalating conflicts.

[0118] This embodiment also provides a multi-model-based method for user emotion monitoring and intervention, including the following steps:

[0119] (a) Collect users’ multimodal data (images, voice, video, text).

[0120] (b) Clean the data, standardize the format and reduce noise, and generate standardized feature vectors through privacy protection technology;

[0121] (c) Use the first model to simulate the conversational style of relatives and friends to provide emotional companionship; learn the text, voice and visual features of relatives and friends based on the Transformer architecture; and generate voice, digital image actions and text replies in sync through OCR and RVC technologies to enhance the realism of the conversation.

[0122] (d) Use the second model to monitor the user's emotional state in real time. If high-risk keywords or abnormal behavior patterns are detected, the intervention mechanism is triggered: automatically take over the conversation, intervene as a psychologist and provide professional guidance; and simultaneously send an alarm to the administrator to trigger offline rescue procedures (such as contacting medical institutions or family members).

[0123] Risk monitoring includes: identifying users' emotional states by combining sentiment analysis models (such as BERT) and a pre-set keyword library; and selecting intervention strategies based on risk levels (low, medium, and high), for example: low risk: continuous emotional support; high risk: immediately triggering dialogue takeover and emergency alerts.

[0124] In summary, the multi-model-based user emotion monitoring and intervention method, device, and computer program product described in this invention simulates the communication style of the target object by training multiple models, simultaneously generating virtual avatars that match lip movements, facial expressions, and actions, thus providing users with more emotionally supportive interactions. Simultaneously, an emotion analysis model is introduced for emotion recognition, early warning, and risk level analysis. When abnormal emotions are identified, a second major model learned from a psychological intervention strategy library and a psychological knowledge graph is used to conduct reassuring dialogue. This allows for real-time detection and response to sudden emotional abnormalities, and intervention with professional reassuring dialogue.

[0125] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0126] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0127] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A user emotion monitoring and intervention method based on multiple models, characterized in that, The method includes the following steps: Based on a first model obtained by simulating the dialogue style of the target object, a multimodal dialogue interaction is conducted with a specified user. A preset text-to-speech model converts the dialogue information generated by the first model into speech information according to the voiceprint and style characteristics of the target object. A preset avatar generation model then generates a virtual avatar with synchronized lip movements, facial expressions, and / or actions based on the speech information and the image of the target object. The preset avatar generation model may employ a lip-syncing model based on FaceFormer or Wav2Lip; or an facial expression and action generation model based on AD-NeRF or GeneFace; or a full-body motion synchronization generation model based on VOCA. Based on a pre-trained sentiment analysis model, the system performs sentiment recognition on the information input by the specified user in the dialogue interaction, and combines keyword detection to issue early warnings of abnormal sentiments and identify risk levels. When abnormal emotions are identified, the corresponding intervention mode is selected according to the risk level, and a soothing dialogue is conducted based on the second major model learned from the psychological intervention strategy library and the psychological knowledge graph. The training steps of the preset text-to-speech model include: acquiring sample data containing the speech of the target object in social media, private communications, interview transcripts, or speeches; performing noise reduction, text transcription, speech segmentation, phoneme alignment, and voiceprint feature extraction on the speech; adding prosodic annotations; and constructing a second training sample set. The model also includes: constructing a generator based on a speech generation model and a discriminator based on a convolutional neural network. The generator is used to generate simulated speech from text transcription, and the discriminator is used to identify whether the input speech belongs to simulated speech. The speech generation model is either a Tacotron 2 or FastSpeech 2 model. The prosodic annotations include fundamental frequency trajectory, speech rate, pause frequency, and stress patterns based on RMS value mutation markers. The generator and the discriminator are trained using the second training sample set based on adversarial learning. The generator is updated by combining the simulated speech reconstruction loss, voiceprint similarity loss, prosody annotation loss and the discriminator's discrimination loss. The discriminator is updated based on the discrimination loss. The generator obtained through adversarial learning is used as the preset text-to-speech model.

2. The user emotion monitoring and intervention method based on multiple models according to claim 1, characterized in that, The pre-training steps of the first large model include: Obtain sample data containing texts expressed by the target object during social media, private communications, interview transcripts, or speeches, and perform noise reduction, segmentation, and privacy anonymization. The sample data is labeled with sentiment and style as tags to construct the first training sample set; The first training sample set is used to train the Transformer-based dialogue generation model. The parameters of the dialogue generation model are updated by using the cross-entropy loss, sentiment consistency loss and style consistency loss of the generated text. A regularization strategy based on dropout and weight decay is added to obtain the first large model.

3. The user emotion monitoring and intervention method based on multiple models according to claim 1, characterized in that, The training steps of the sentiment analysis model include: Obtain a third training sample set, in which each sample contains user dialogue text, synchronized speech waveforms and facial expression videos, and add real emotion labels; An initial neural network is obtained, including text branch, speech branch, vision branch, cross-modal fusion layer, and fully connected layer. The text branch extracts semantic features of the dialogue text based on the RoBERTa model; the speech branch extracts acoustic temporal features of the synchronized speech waveform based on Wav2Vec 2; the vision branch extracts facial expression space features of the facial expression video based on the VisionTransformer model; the cross-modal fusion layer dynamically weights and fuses the semantic features, acoustic temporal features, and facial expression space features through a gated attention mechanism to obtain a joint emotion representation vector; the fully connected layer outputs an emotion prediction result for the user based on the joint emotion representation vector; and the initial neural network is updated with parameters based on the deviation between the emotion prediction result and the actual emotion to obtain the emotion analysis model.

4. The user emotion monitoring and intervention method based on multiple models according to claim 1, characterized in that, The training steps for the second major model include: A psychological intervention database is constructed, which includes psychotherapy dialogue records, crisis intervention scripts and reassuring dialogue templates annotated by experts, as well as structured intervention paths constructed based on a psychological knowledge graph. The structured intervention paths are marked with strategy priorities corresponding to different risk levels. The second large model is obtained by adapting and training the psychological intervention database based on the large language model.

5. A user emotion monitoring and intervention device based on a multi-model framework, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the method as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 4.

Images