User emotion monitoring intervention method and device based on multiple models and computer program product

Abstract

According to the multi-model-based user emotion monitoring intervention method and device and the computer program product provided by the invention, the communication style of the target object is simulated by training multiple types of models, a virtual image is synchronously generated to match mouth shapes, expressions and actions, and interaction with more emotion support can be provided for the user. Meanwhile, an emotion analysis model is introduced to carry out emotion recognition, early warning and risk level analysis, and when emotion abnormity is recognized, a pacific dialogue is carried out based on a second model obtained by learning a psychological intervention strategy library and a psychological knowledge graph. And sudden emotional abnormal conditions can be detected and responded in real time, and professional pacifying dialogues can be intervened.

Description

Technical Field

[0001] The present invention relates to the field of emotion management technology, and in particular to a user emotion monitoring and intervention method, device, and computer program product based on multiple models. Background Art

[0002] At present, emotion detection and intervention technologies are mainly divided into the following categories: simple dialogue systems based on natural language processing (NLP) and deep learning, such as chatbots and virtual assistants, can provide basic emotional support, but the emotional interaction is not deep enough, and they cannot simulate the conversation style of specific characters, and they lack psychological intervention functions; artificial psychological intervention systems rely on manual review of conversation content and psychological assessment and intervention through the intervention of psychology experts, but there are problems such as high response delay, inability to process massive user data in real time, and difficulty in dealing with acute psychological crisis scenarios; single model systems rely on a single model to complete all functions (such as emotional companionship or psychological monitoring), which has a single function and is prone to conflicts. It cannot balance the needs of "emotional companionship" and "rational intervention", and the model is highly complex and difficult to optimize. Summary of the Invention

[0003] In view of this, the embodiments of the present invention provide a user emotion monitoring and intervention method, device and computer program product based on multiple models to eliminate or improve one or more defects existing in the prior art, and solve the problem that the interaction process of the prior art lacks emotional support and cannot efficiently implement emotion monitoring and intervention.

[0004] One aspect of the present invention provides a user emotion monitoring and intervention method based on multiple models, the method comprising the following steps:

[0005] Conducting a multimodal conversational interaction with a designated user based on a first large model obtained by simulating the target subject's conversational style, converting conversation information generated by the first large model into speech information based on the target subject's voiceprint and style characteristics using a preset text-to-speech model, and generating a virtual avatar with synchronized lip movements, facial expressions, and / or movements based on the speech information and the target subject's image using a preset image generation model;

[0006] Based on the pre-trained sentiment analysis model, the user's input information in the dialogue interaction is subjected to emotion recognition, and abnormal emotion warning and risk level identification are performed in combination with keyword detection;

[0007] When abnormal emotions are identified, the corresponding intervention mode is selected according to the risk level, and a soothing dialogue is conducted based on the second largest model obtained by learning the psychological intervention strategy library and the psychological knowledge graph.

[0008] In some embodiments, the pre-training step of the first large model includes:

[0009] Obtain sample data containing text expressed by the target subject on social media, in private communications, in interviews, or during speeches, and perform denoising, segmentation, and privacy masking;

[0010] Labeling the sample data with emotions and styles as labels to construct a first training sample set;

[0011] The first training sample set is used to train a Transformer-based dialogue generation model to generate text cross-entropy loss, sentiment consistency loss, and style consistency loss to update the parameters of the dialogue generation model. A regularization strategy based on dropout and weight decay is added to obtain the first large model.

[0012] In some embodiments, the step of training the preset text-to-speech model includes:

[0013] Obtaining sample data containing the target subject's speech in the social media, private communication, interview record, or speech, performing noise reduction processing, text transcription, speech segmentation, phoneme alignment, and voiceprint feature extraction on the speech, adding prosody annotation, and constructing a second training sample set;

[0014] A generator based on a speech generation model and a discriminator based on a convolutional neural network are constructed, wherein the generator is used to generate simulated speech based on text transcribed from text, and the discriminator is used to identify whether the input speech belongs to simulated speech; the generator and the discriminator are trained based on adversarial learning using the second training sample set, the generator is updated by jointly using the simulated speech reconstruction loss, the voiceprint similarity loss, the prosody annotation loss and the discriminant loss of the discriminator, and the discriminator is updated based on the discriminant loss; the generator obtained by adversarial learning is used as the preset text-to-speech model.

[0015] In some embodiments, the speech generation model is a Tacotron 2 or FastSpeech 2 model; and the prosodic annotation includes fundamental frequency trajectory, speech rate, pause frequency, and stress pattern based on RMS value mutation markers.

[0016] In some embodiments, the preset image generation model adopts a lip-sync generation model based on FaceFormer or Wav2Lip; or an expression and action generation model based on AD-NeRF or GeneFace; or a whole-body action synchronization generation model based on VOCA.

[0017] In some embodiments, the sentiment analysis model training step includes:

[0018] Obtain a third training sample set, where each sample contains user conversation text, synchronized speech waveform, and facial expression video, and adds a true emotion label;

[0019] An initial neural network is obtained, including a text branch, a speech branch, a vision branch, a cross-modal fusion layer, and a fully connected layer. The text branch extracts semantic features of the conversation text based on the RoBERTa model, the speech branch extracts acoustic temporal features of the synchronized speech waveform based on Wav2Vec 2, and the vision branch extracts expression space features of the facial expression video based on the VisionTransformer model. The cross-modal fusion layer dynamically weighted fuses the semantic features, the acoustic temporal features, and the expression space features through a gated attention mechanism to obtain a joint emotion representation vector. The fully connected layer outputs an emotion prediction result for the user based on the joint emotion representation vector. A loss is constructed based on the deviation between the emotion prediction result and the true emotion to update the parameters of the initial neural network to obtain the emotion analysis model.

[0020] In some embodiments, the step of training the second large model includes:

[0021] Constructing a psychological intervention database, which includes psychotherapy conversation records, crisis intervention techniques, and expert-annotated comforting conversation templates, as well as a structured intervention path built based on a psychological knowledge graph. The structured intervention path annotates the strategic priorities corresponding to different risk levels.

[0022] Adaptive training is performed on the psychological intervention database based on the large language model to obtain the second large model.

[0023] On the other hand, the present invention provides a user emotion monitoring and intervention device based on multiple models, including a processor, a memory and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0024] In another aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0025] On the other hand, the present invention provides a computer program product, comprising a computer program / instruction, characterized in that the computer program / instruction implements the steps of the above method when executed by a processor.

[0026] The beneficial effects of the present invention are at least:

[0027] The multi-model-based user emotion monitoring and intervention method, device, and computer program product described in this invention simulate the target subject's communication style by training multiple models, and simultaneously generate a virtual avatar that matches the target subject's lip movements, expressions, and movements, providing users with more emotionally supportive interactions. Simultaneously, an emotion analysis model is introduced for emotion recognition, early warning, and risk level analysis. When abnormal emotions are identified, soothing conversations are initiated based on a second model learned from a psychological intervention strategy library and a psychological knowledge graph. This system is capable of detecting and responding to sudden emotional abnormalities in real time, and intervening in professional soothing conversations.

[0028] Additional advantages, objects, and features of the present invention will be set forth in part in the following description and will become apparent to those skilled in the art upon examination of the following or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained by the structures particularly pointed out in the description and drawings.

[0029] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. In the drawings:

[0031] Figure 1 This is a flow chart of a multi-model based user emotion monitoring and intervention method according to an embodiment of the present invention.

[0032] Figure 2 This is a schematic diagram of the workflow of an intelligent psychological intervention system combining multi-model collaboration according to another embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0034] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0035] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0036] Users need a real-time, automated psychological monitoring and intervention system that can trigger warnings and provide intervention as soon as a user's emotions get out of control, thus overcoming the limitations of manual services. For example, in the case of "childless families," users may fall into long-term depression due to the loss of relatives and friends. The system needs to intervene immediately when users express extreme emotions to prevent tragedy. Existing emotional interaction systems, such as chatbots and virtual assistants, are typically based on natural language processing (NLP) and deep learning technologies. They can engage in simple conversations with users and provide a certain degree of emotional support. However, these systems have significant functional limitations, such as insufficient depth of emotional interaction, lack of psychological intervention, and inability to meet the emotional support needs of relatives and friends. Traditional psychological intervention systems rely on manual review and cannot meet the needs of acute psychological crisis scenarios.

[0037] In view of this, the present invention provides a user emotion monitoring intervention method based on multiple models, such as Figure 1 As shown, the method includes the following steps S101 to S103:

[0038] Step S101: Based on the first large model obtained by simulating the conversation style of the target object, a multimodal conversation interaction is conducted with a designated user. The conversation information generated by the first large model is converted into speech information according to the voiceprint and style characteristics of the target object through a preset text-to-speech model, and a preset image generation model is used to generate a virtual image with synchronized lip movements, expressions and / or movements according to the image of the target object based on the voice information.

[0039] Step S102: Based on the pre-trained emotion analysis model, the user's input information in the dialogue interaction is subjected to emotion recognition, and abnormal emotion warning and risk level identification are performed in combination with keyword detection.

[0040] Step S103: When abnormal emotions are identified, the corresponding intervention mode is selected according to the risk level, and a soothing dialogue is conducted based on the second largest model obtained by learning the psychological intervention strategy library and the psychological knowledge graph.

[0041] In step S101, a multimodal conversational interaction is conducted with a designated user by simulating the target user's conversational style. First, a first large model is used to accurately simulate the target user's conversational style, enabling the user to communicate with the target user in the same way as the target user. In practice, the target user can select someone with whom the designated user has developed an emotional connection. For example, this can be a friend or relative, or a pre-set public figure. During the conversation, a pre-set text-to-speech model is introduced to convert text messages into more vivid voice messages. This model not only synthesizes the conversational content generated by the first large model but also optimizes it based on the target user's voiceprint and style characteristics, ensuring that the generated voice is highly compatible with the target user in terms of timbre and intonation. Furthermore, to enhance the visual quality of the conversational interaction, a pre-set image generation model is used. Based on the generated voice information and the target user's image characteristics, a virtual avatar synchronized with the voice is generated in real time, including precise lip movements, expressive expressions, and natural movements, providing the user with a richer and more comprehensive interactive experience.

[0042] In some embodiments, the pre-training step of the first large model includes steps S201 to S203:

[0043] Step S201: Obtain sample data containing text expressed by the target subject in social media, private communications, interview records or speeches, and perform denoising, segmentation processing and privacy masking.

[0044] Step S202: label the sample data with emotions and styles as labels to construct a first training sample set.

[0045] Step S203: The Transformer-based dialogue generation model is trained using the first training sample set to generate text cross-entropy loss, sentiment consistency loss, and style consistency loss to update the parameters of the dialogue generation model, and a regularization strategy based on dropout and weight decay is added to obtain the first large model.

[0046] In step S201, social media includes social platforms such as Weibo, WeChat Moments, or Tik Tok, which are used to capture daily communication styles. Private communications include WeChat chat records, emails, and letter messages. Interview records include interview transcripts in books, magazines, and podcasts to understand how their in-depth views are expressed. The text during the speech includes the speech manuscript, which is used to learn the wording and logic of its formal occasions. After obtaining the data, denoising processing is required to remove irrelevant information, erroneous content, and noise interference in the data, such as error correction, cleaning irrelevant symbols, etc., to ensure the validity and accuracy of the data. Next, the data is segmented, and the long text is divided into segments of appropriate length according to semantics or logic, so that the model can better understand and learn. Privacy desensitization is also a key step. Personal information and sensitive content in the data need to be desensitized to ensure the legitimacy and security of the data while protecting privacy, providing a reliable data foundation for subsequent model training.

[0047] Denoising of sample data includes removing irrelevant symbols, correcting typos and unifying the format. Segment processing retains the original sentence structure to avoid disrupting the language rhythm. Privacy desensitization includes anonymizing sensitive names, places or events.

[0048] In step S202, the main task is to label the sample data after denoising, segmentation and privacy masking, and construct the first training sample set. The labeling work includes two dimensions: emotion and style. In terms of emotion labeling, the emotions expressed by the target object, such as happiness, sadness, anger, etc., are judged based on the text content, and the corresponding emotion labels are labeled. Style labeling focuses on the language style of the target object, such as humor, seriousness, gentleness, etc., and its style characteristics are labeled. More specifically, it can be quantified by doing word frequency analysis and sentence structure labeling. The efficiency and accuracy of labeling are improved by combining manual labeling and automatic labeling tools. The constructed first training sample set not only contains the text data of the target object, but also contains its emotion and style information, providing rich and accurate data support for the first large model to learn the language characteristics of the target object.

[0049] In step S203, the Transformer-based dialogue generation model is trained using the first training sample set to generate the first large model. The dialogue generation model can use GPT-3 / 4 or PaLM, and a lightweight solution can use GPT-2Medium or DistilGPT-2. During the training process, the cross-entropy loss, sentiment consistency loss, and style consistency loss of the generated text are used as loss functions. The cross-entropy loss is used to measure the difference between the generated text and the real text, so that the model generates text that is closer to the expression of the target object. The sentiment consistency loss ensures that the sentiment of the generated text is consistent with the sentiment label of the target object, and the style consistency loss ensures that the style of the generated text conforms to the stylistic characteristics of the target object. At the same time, a regularization strategy based on dropout and weight decay is added to prevent model overfitting and improve its generalization ability. Through continuous iterative optimization and updating of model parameters, the first large model that can accurately simulate the target object's conversational style is ultimately obtained, providing core support for subsequent multimodal dialogue interactions.

[0050] In some embodiments, the training step of the preset text-to-speech model includes steps S301 to S302:

[0051] Step S301: Obtain sample data containing the target subject's speech in social media, private communications, interview records or speeches, perform noise reduction processing, text transcription, speech segmentation, phoneme alignment and voiceprint feature extraction on the speech, add prosody annotation, and construct a second training sample set.

[0052] Step S302: Construct a generator based on a speech generation model and a discriminator based on a convolutional neural network, the generator is used to generate simulated speech based on text transcription, and the discriminator is used to identify whether the input speech belongs to simulated speech; use the second training sample set to train the generator and the discriminator based on adversarial learning, jointly update the simulated speech reconstruction loss, voiceprint similarity loss, prosody annotation loss and the discriminator's discriminant loss, and update the discriminator based on the discriminant loss; use the generator obtained by adversarial learning as the preset text-to-speech model.

[0053] 1. In step S301, a baseline prediction is obtained, requiring at least 3 to 5 hours of clean speech (sampling rate ≥ 16kHz), covering scenarios such as daily conversations, reading aloud, and impromptu speeches, encompassing different emotional states. Background noise is removed using RNNoise or Demucs, segmented into 2-10 second segments based on VAD (Voice Activity Detection), and phoneme boundaries are annotated using Montreal Forced Aligner. Voiceprint features are extracted using the Mel-Frequency Cepstral Coefficient (MFCC) algorithm. The speech signal is first preprocessed, including framing and windowing. A Fast Fourier Transform (FFT) is then performed to convert the signal from the time domain to the frequency domain. The spectrum is then filtered using a Mel filter bank, and the output energy of each filter is calculated. Finally, the logarithm of the filtered energy is taken and a discrete cosine transform is performed to obtain the MFCC coefficients. Alternatively, perceptual linear prediction (PLP) or linear discriminant analysis (LDA) can be used to extract voiceprint features. The Praat script was used to extract the intonation fluctuation pattern, calculate the number of syllables per second, pause frequency and energy change of emphasized words using RMS value mutation to obtain the prosodic annotation.

[0054] In step S302, VITS (Variational Inference Text-to-Speech) or FastSpeech 2+HiFi-GAN can be used as the generator and discriminator architecture in this embodiment. The VITS model is an end-to-end speech synthesis model based on conditional variational autoencoder (cVAE) and adversarial learning. It mainly consists of a posterior encoder, a priori encoder, a decoder, a discriminator and a random duration predictor. By combining technologies such as variational autoencoder, adversarial learning and normalized flow, it can generate natural, diverse and expressive speech from text. A random duration predictor is also proposed for synthesizing speech with different rhythms from input text, expressing a natural one-to-many relationship. The FastSpeech2+HiFi-GAN model is an end-to-end speech synthesis model that combines FastSpeech 2 and HiFi-GAN. FastSpeech 2 includes a text encoder, a variance adapter, and a speech decoder, which can convert input text into Mel spectrograms and model and adjust speech features such as duration, pitch, and energy; the generator and discriminator of HiFi-GAN convert the Mel spectrograms into high-quality speech waveforms. The model as a whole achieves efficient and high-quality synthesis from text to speech.

[0055] In some embodiments, the speech generation model may be a Tacotron 2 or FastSpeech 2 model; and the prosodic annotation includes fundamental frequency trajectory, speech rate, pause frequency, and stress pattern based on RMS value mutation markers.

[0056] In some embodiments, the preset image generation model adopts a lip-sync generation model based on FaceFormer or Wav2Lip; or adopts an expression and action generation model based on AD-NeRF or GeneFace; or adopts a whole-body action synchronization generation model based on VOCA.

[0057] Step S102 focuses on monitoring the emotional state of the designated user during the conversation interaction and warning of abnormal emotions. First, the information input by the user is subjected to emotion recognition based on the pre-trained emotion analysis model. The emotion analysis model has been trained with a large amount of data and can accurately capture the emotional elements contained in text, voice and other information, and determine the user's current emotion category, such as happiness, sadness, anger, anxiety, etc. At the same time, combined with keyword detection technology, the text content is further analyzed in depth to screen out keywords with potential risks, such as some words that express extreme emotions and may imply dangerous behaviors. Once key information related to abnormal emotions is detected, the early warning mechanism is quickly activated, and the risk level of abnormal emotions is accurately identified based on preset rules and standards, providing a key basis for subsequent intervention measures, ensuring that emotional problems that may arise in users can be addressed in a timely and effective manner.

[0058] In some embodiments, the sentiment analysis model training step includes steps S401 to S402:

[0059] Step S401: Obtain a third training sample set, where each sample contains user conversation text, synchronized speech waveform and facial expression video, and adds a real emotion label.

[0060] Step S402: Obtain an initial neural network, including a text branch, a speech branch, a vision branch, a cross-modal fusion layer, and a fully connected layer. The text branch extracts semantic features of the conversation text based on the RoBERTa model, the speech branch extracts acoustic timing features of the synchronized speech waveform based on Wav2Vec 2, and the vision branch extracts expression space features of the facial expression video based on the Vision Transformer model. The cross-modal fusion layer dynamically weights and fuses the semantic features, acoustic timing features, and expression space features through a gated attention mechanism to obtain a joint emotion representation vector. The fully connected layer outputs an emotion prediction result for the user based on the joint emotion representation vector. A loss is constructed based on the deviation between the emotion prediction result and the actual emotion to update the parameters of the initial neural network to obtain an emotion analysis model.

[0061] In step S103, when the user is identified to have abnormal emotions through the above steps, the intervention phase is entered. According to the predetermined risk level, the most appropriate mode is selected from a variety of intervention modes. The intervention mode here is the second largest model constructed based on in-depth study of the psychological intervention strategy library and the psychology knowledge map. The psychological intervention strategy library covers a wide variety of psychological intervention methods, which are suitable for situations with different emotional problems and risk levels; and the psychology knowledge map systematically integrates professional knowledge and theories in the field of psychology, providing solid theoretical support for the formulation of intervention strategies. The second largest model can flexibly apply these psychological intervention strategies according to the user's specific situation and emotional state, generate targeted and soothing dialogue content, and guide users to alleviate abnormal emotions in a gentle and professional manner, help them adjust their mentality, and return to a more positive and stable emotional state, thereby achieving effective intervention in the user's emotions.

[0062] In some embodiments, the training step of the second largest model includes steps S501 to S502:

[0063] Step S501: Construct a psychological intervention database, which includes psychotherapy dialogue records, crisis intervention scripts, soothing dialogue templates annotated by experts, and a structured intervention path constructed based on a psychological knowledge graph. The structured intervention path annotates the strategy priorities corresponding to different risk levels.

[0064] Step S502: Adapting and training the psychological intervention database based on the large language model to obtain a second large model.

[0065] Psychotherapy conversation records can use CBT cognitive behavioral therapy cases, psychology knowledge graphs can use DSM-5 diagnostic criteria or emotion regulation theory, and large language models can be based on large models such as GPT-3.5 or LLaMA. Furthermore, the risk-based intervention model can have a three-level response: low-risk uses open-ended questions to guide emotional expression, medium-risk combines cognitive restructuring techniques with breathing and relaxation guidance, and high-risk triggers an emergency protocol and transfers to manual intervention.

[0066] On the other hand, the present invention provides a user emotion monitoring and intervention device based on multiple models, including a processor, a memory and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the above method.

[0067] In another aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when executed by a processor.

[0068] On the other hand, the present invention provides a computer program product, comprising a computer program / instruction, characterized in that the computer program / instruction implements the steps of the above method when executed by a processor.

[0069] The present invention is described in detail with reference to a specific embodiment below:

[0070] This embodiment provides an intelligent psychological intervention system that combines multi-model collaboration. The first model simulates the conversation style of relatives and friends (multi-modal fusion of sound, image and language) to provide emotional companionship for users; the second model monitors the user's emotions in real time, and automatically triggers the intervention mechanism when potential psychological problems or even extreme risks are detected, taking over the conversation and alerting the administrator. The system is suitable for improving the real-time nature and humanistic care of mental health services. Figure 2 As shown, the system includes the following modules:

[0071] 1. Multimodal data acquisition module

[0072] Image acquisition and OCR recognition: Uploaded image files (mp4 files with a 16kHz sampling rate) are used to extract visual information such as facial features and body language of friends and family. Multimodal unification requires aligning timelines with spatial coordinates, mapping facial and body language features to a unified embedding space (e.g., a 1024-dimensional vector), and combining metadata annotation (emotion and scene tags) to enhance relevance. OCR technology is used to identify textual content in images (e.g., scanned diaries and letters) to build a library of person language styles.

[0073] Voice Recording and RVC Voice Analysis: We collect voice data from friends and family (such as audio recordings and phone logs) and use RVC (Real-time Voice Cloning) technology to extract voice characteristics and generate a high-fidelity voice cloning model. This supports real-time speech synthesis, simulating the intonation, pauses, and emotional expressions of natural conversation.

[0074] Video recording and digital image construction: Use video analysis technology to extract dynamic features such as facial expressions and movements of characters, and combine them with 3D modeling to generate digital images.

[0075] Text data collection and construction of character language style library: Extract text data through channels such as WeChat chat history export (WeChat exports data or exports to large model training sets through third parties), social media (through platform API, crawler scraping or image recognition, etc.), and email (mailbox API), and analyze language style, common vocabulary and topic preferences. Use annotation tools to label the data to ensure annotation consistency, combining automated initial screening (such as BERT sentiment prediction) with manual review. The focus is on text annotation, so the voice will be converted into text first, and the annotated data will be delivered in JSON or CSV format. Store structured data in the database, index keywords and user IDs, and improve retrieval efficiency for retrieval optimization. Collect relevant data through public corpora and user demand-specific scenario data, classify styles based on emotions and key words and phrases (such as specific scenarios, high-frequency words, etc.), improve expression accuracy through hierarchical labeling and adversarial training, and store data in JSON format or in a database accordingly.

[0076] 2. Data processing and vectorization module

[0077] Data cleaning and format unification: Remove noisy data (such as duplicate content and irrelevant information) and standardize text encoding formats (such as UTF-8). Perform noise reduction on voice data and use a VAD-like method to detect and remove silent segments to ensure data quality. Video data should be as high-resolution as possible (such as 2K or higher) to avoid jitter and distortion caused by compression. Select clips with normal lighting, clear facial features, and rich facial expressions. After generating the digital human, delete the original data clips.

[0078] Privacy protection technology: Anonymization removes user identity information (such as name and contact information), retaining only sentiment and behavioral characteristics. Data vectorization converts raw data into vector representations (such as word vectors and voiceprint vectors) through feature extraction, avoiding the direct storage of sensitive content.

[0079] 3. Introducing the first model to build the emotional companionship module

[0080] Long-term memory database: Through the long-term memory database, the conversation content is annotated and recorded to record user interaction feature data (such as names of people, places and their relationships), forming an interaction history and a personalized emotional file, which is stored in vector form.

[0081] Conversation Simulation: A Transformer-based NLP model learns the text, speech, and visual features of friends and family members to generate responses that match their style. This combined with multimodal output (such as simultaneous speech generation and digital avatar motion) enhances the realism of conversations.

[0082] Early warning monitoring: Use sentiment analysis models (such as BERT + sentiment dictionary) to identify user emotional states (such as depression, anxiety). When high-risk keywords (such as "despair") or abnormal behavior patterns are detected, early warning signals are triggered.

[0083] 4. Introducing the second model to build a psychological intervention module

[0084] Psychologist dataset training: This training model uses historical conversation data from professional psychologists and a library of psychological intervention strategies to identify crisis scenarios. This training also integrates psychological knowledge graphs (e.g., DSM-5 diagnostic criteria and cognitive behavioral therapy steps).

[0085] Dynamic intervention strategies: Based on the intensity of user emotions and risk levels, we select intervention modes (such as soothing conversations, distraction, and emergency contact with customer service). Pre-set ethical constraints prevent the generation of inappropriate content (such as replies that escalate conflicts).

[0086] Real-time intervention mechanism: Takes over the conversation in the first model and intervenes as a "psychologist," providing professional guidance (such as asking guided questions and mindfulness training suggestions). Simultaneously sends an alert to the administrator, triggering offline rescue processes (such as contacting family members or medical institutions).

[0087] Among them, the training and deployment methods of the first large model are as follows:

[0088] Collect a large amount of historical communication data between users and their relatives and friends, denoise the data to remove noise and irrelevant information (especially special symbols), unify the format into text data, and annotate emotions and events.

[0089] The model is based on the Transformer dialogue generation model (such as the GPT-3 architecture) and includes an encoder-decoder structure.

[0090] During training, the data is split into 70% for the training set, 15% for the validation set, and 15% for the test set. A loss function is constructed to optimize parameter updates, namely the cross-entropy loss (to optimize the accuracy of text generation), the sentiment consistency loss (to penalize mismatches between generated responses and annotated sentiment labels), and the style consistency loss (to ensure that the response style is similar to historical data through contrastive learning).

[0091] During the content generation and optimization process, user queries are input in text or speech-to-text format; the sentiment analysis module predicts the emotional label of the current conversation scenario, such as "nostalgia"; the model generates candidate responses (Top-K sampling, K=5), filters out responses that do not conform to the style (such as similarity with historical data <0.6), and outputs the optimal response.

[0092] The model is encapsulated as a REST API to facilitate conversational services. In the early stages, internal technical testing was conducted to repeatedly evaluate and optimize the process.

[0093] The second largest model was trained and deployed as follows:

[0094] Model construction and training: Data sources include social media comments, psychological counseling records, and related data sets, with a focus on cognitive behavior and psychodynamics to ensure that intervention methods are evidence-based. Collect data sets containing various emotional states and psychological problem cases, including but not limited to images, text, audio, etc. Split the data (such as training set: 70%, validation set: 15%, test set: 15%) and consider adding adversarial samples (such as noise data generated by GAN). Use the labeled data to train the emotion monitoring model based on sentiment analysis and psychological knowledge. During the training process, the model's hyperparameters are adjusted to improve the model's recognition accuracy of users' emotional states and psychological problems.

[0095] Specific monitoring process and optimization:

[0096] Based on keyword monitoring: the frequency of negative emotion keywords such as sadness and grief, or the direct appearance of negative behavior keywords.

[0097] Based on user profile analysis: Dynamically adjust the intervention strategy according to the patient's age, cultural background, and personality traits, consider the main psychological characteristics of the current user's group, and consider the patient's cultural background and value system's acceptance of the intervention.

[0098] Divide the intervention process:

[0099] Primary prevention, namely health promotion: for the general population, improving psychological resilience through mental health education, stress management, etc.

[0100] Secondary prevention, namely preventive intervention: targeting high-risk groups, using selective or guided interventions to reduce risks and reporting warnings.

[0101] The third level of prevention, namely psychotherapy, is to alleviate the symptoms of individuals who have developed psychological disorders through cognitive behavioral therapy (CBT) and psychoanalysis, and report warnings and require timely medical care.

[0102] Ethical rules for reply content include: 1) The principle of non-harm: Avoid misleading suggestions or discriminatory content. In particular, negative user behavior should be promptly prevented rather than supported. 2) The principle of autonomy: Require users to retain their independent judgment and not make choices for them. 3) The principle of benefit: Responses must be based on scientific evidence and psychological theory.

[0103] Ethical Rules Implementation: Using RLHF (Reinforcement Learning with Human Feedback), human annotators assign ethical ratings to responses, penalizing harmful responses and rewarding effective ones. When negative user intent is detected, the model should automatically trigger human intervention or provide emergency contact information. An explainability module explains the model's reasoning logic to ensure users understand the source of the recommendations. A hybrid generation-verification architecture is used to verify the accuracy of responses by searching authoritative databases (such as PubMed).

[0104] The model is encapsulated as a REST API to facilitate the provision of conversation services. At the same time, it monitors and selects secondary and tertiary prevention measures to promptly interrupt the conversation with the first model and report the conversation content.

[0105] Furthermore, a multi-model-based user emotion monitoring and intervention system is provided, which includes a multimodal data acquisition module, a data processing and vectorization module, an emotional companionship module and a psychological intervention module.

[0106] The multimodal data acquisition module is used to collect users' images, voice, video and text data, and extract the visual features, timbre features and language style of relatives and friends through optical character recognition (OCR) and voice cloning (RVC) technology; and uses word vectors, voiceprint vectors and other technologies to convert raw data into non-sensitive feature vectors.

[0107] The data processing and vectorization module cleans, unifies the format, and reduces noise of the multimodal data, and generates standardized feature vectors through privacy protection technology (including anonymization and data vectorization).

[0108] The emotional companionship module is equipped with the first large model, which simulates the conversation style of relatives and friends based on deep learning models (such as the Transformer architecture), interacts with users through multimodal output (voice, digital image actions and text replies), and records user interaction history to form a personalized emotional profile.

[0109] The psychological intervention module is equipped with the second largest model, which monitors the user's emotional state in real time, detects potential psychological risks, and automatically takes over the conversation when the risk level reaches the threshold. It provides professional psychological intervention strategies (such as cognitive behavioral therapy guidance or emergency contact with manual customer service) and triggers system warnings.

[0110] Among them, the multimodal data acquisition module includes:

[0111] Image acquisition unit, used to extract facial features and body language of relatives and friends;

[0112] The voice recording unit uses RVC technology to clone the voice characteristics of relatives and friends and generate a high-fidelity voice model;

[0113] A video analysis unit extracts the dynamic expressions and movements of friends and family members to construct digital images;

[0114] The text collection unit extracts language style features from chat records, social media and other channels.

[0115] The first model’s conversation simulation capabilities include: a long-term memory database based on a memory network that dynamically updates user interaction history to form a personalized emotional profile;

[0116] The psychological intervention module also includes a sentiment analysis module, which uses the BERT model or other related models combined with the sentiment dictionary to identify the user's emotional state (such as depression and anxiety).

[0117] The intervention strategies of the second model include: selecting intervention modes (such as soothing conversations, attention diversion, or emergency warnings) based on the user's emotional intensity and risk level; integrating psychological knowledge graphs (such as DSM-5 diagnostic criteria or cognitive behavioral therapy steps) to generate intervention language; and filtering inappropriate content through ethical constraints to avoid exacerbating conflicts.

[0118] This embodiment also provides a user emotion monitoring and intervention method based on multiple models, including the following steps:

[0119] (a) Collect multimodal data of users (images, voice, video, text).

[0120] (b) Clean the data, unify its format, and reduce noise, and generate standardized feature vectors using privacy protection technology;

[0121] (c) Use the first model to simulate the conversation style of relatives and friends and provide emotional companionship; the NLP model based on the Transformer architecture learns the text, voice and visual features of relatives and friends; and use OCR and RVC technology to simultaneously generate voice, digital image actions and text replies to enhance the realism of conversations.

[0122] (d) The second largest model is used to monitor the user's emotional state in real time. If high-risk keywords or abnormal behavior patterns are detected, the intervention mechanism is triggered: the system automatically takes over the conversation, intervenes in the role of a psychologist, and provides professional counseling; and simultaneously sends an alert to the administrator, triggering the offline rescue process (such as contacting medical institutions or family members).

[0123] Risk monitoring includes: combining sentiment analysis models (such as BERT) and preset keyword libraries to identify the user's emotional state; selecting intervention strategies based on risk levels (low, medium, and high), for example: low risk: continuous emotional companionship; high risk: immediate triggering of conversation takeover and emergency warning.

[0124] In summary, the multi-model-based user emotion monitoring intervention method, device, and computer program product described in the present invention simulate the communication style of the target subject by training multiple models, and simultaneously generate a virtual image to match the lip shape, expression, and action, thereby providing users with more emotionally supportive interactions. At the same time, an emotion analysis model is introduced to perform emotion recognition, early warning, and risk level analysis. When an emotional anomaly is identified, a soothing conversation is conducted based on a second model learned from a psychological intervention strategy library and a psychological knowledge graph. It is capable of detecting and responding to sudden emotional anomalies in real time and intervening in professional soothing conversations.

[0125] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0126] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0127] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0128] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A user emotion monitoring and intervention method based on multiple models, characterized in that: The method comprises the following steps: Conducting a multimodal conversational interaction with a designated user based on a first large model obtained by simulating the target subject's conversational style, converting conversation information generated by the first large model into speech information based on the target subject's voiceprint and style characteristics using a preset text-to-speech model, and generating a virtual avatar with synchronized lip movements, facial expressions, and / or movements based on the speech information and the target subject's image using a preset image generation model; Based on the pre-trained sentiment analysis model, the user's input information in the dialogue interaction is subjected to emotion recognition, and abnormal emotion warning and risk level identification are performed in combination with keyword detection; When abnormal emotions are identified, the corresponding intervention mode is selected according to the risk level, and a soothing dialogue is conducted based on the second largest model obtained by learning the psychological intervention strategy library and the psychological knowledge graph.

2. The user emotion monitoring and intervention method based on multiple models according to claim 1 is characterized in that: The pre-training steps of the first model include: Obtain sample data containing text expressed by the target subject on social media, in private communications, in interviews, or during speeches, and perform denoising, segmentation, and privacy masking; Labeling the sample data with emotions and styles as labels to construct a first training sample set; The first training sample set is used to train a Transformer-based dialogue generation model to generate text cross-entropy loss, sentiment consistency loss, and style consistency loss to update the parameters of the dialogue generation model. A regularization strategy based on dropout and weight decay is added to obtain the first large model.

3. The user emotion monitoring and intervention method based on multiple models according to claim 1 is characterized in that: The training steps of the preset text-to-speech model include: Obtaining sample data containing the target subject's speech in the social media, private communication, interview record, or speech, performing noise reduction processing, text transcription, speech segmentation, phoneme alignment, and voiceprint feature extraction on the speech, adding prosody annotation, and constructing a second training sample set; A generator based on a speech generation model and a discriminator based on a convolutional neural network are constructed, wherein the generator is used to generate simulated speech based on text transcribed from text, and the discriminator is used to identify whether the input speech belongs to simulated speech; the generator and the discriminator are trained based on adversarial learning using the second training sample set, the generator is updated by jointly using the simulated speech reconstruction loss, the voiceprint similarity loss, the prosody annotation loss and the discriminant loss of the discriminator, and the discriminator is updated based on the discriminant loss; the generator obtained by adversarial learning is used as the preset text-to-speech model.

4. The user emotion monitoring and intervention method based on multiple models according to claim 3 is characterized in that: The speech generation model is Tacotron 2 or FastSpeech 2 model; The prosodic annotations include fundamental frequency trajectory, speaking rate, pause frequency, and stress pattern based on RMS value mutation marks.

5. The user emotion monitoring and intervention method based on multiple models according to claim 3 is characterized in that: The preset image generation model adopts a lip-sync generation model based on FaceFormer or Wav2Lip; or an expression and action generation model based on AD-NeRF or GeneFace; or a whole-body action synchronization generation model based on VOCA.

6. The user emotion monitoring and intervention method based on multiple models according to claim 1 is characterized in that: The training steps of the sentiment analysis model include: Obtain a third training sample set, where each sample contains user conversation text, synchronized speech waveform, and facial expression video, and adds a true emotion label; An initial neural network is obtained, including a text branch, a speech branch, a vision branch, a cross-modal fusion layer, and a fully connected layer. The text branch extracts semantic features of the conversation text based on the RoBERTa model, the speech branch extracts acoustic temporal features of the synchronized speech waveform based on Wav2Vec 2, and the vision branch extracts expression space features of the facial expression video based on the VisionTransformer model. The cross-modal fusion layer dynamically weighted fuses the semantic features, the acoustic temporal features, and the expression space features through a gated attention mechanism to obtain a joint emotion representation vector. The fully connected layer outputs an emotion prediction result for the user based on the joint emotion representation vector. A loss is constructed based on the deviation between the emotion prediction result and the true emotion to update the parameters of the initial neural network to obtain the emotion analysis model.

7. The user emotion monitoring and intervention method based on multiple models according to claim 1 is characterized in that: The training steps of the second largest model include: Constructing a psychological intervention database, which includes psychotherapy conversation records, crisis intervention techniques, and expert-annotated comforting conversation templates, as well as a structured intervention path built based on a psychological knowledge graph. The structured intervention path annotates the strategic priorities corresponding to different risk levels. Adaptive training is performed on the psychological intervention database based on the large language model to obtain the second large model.

8. A user emotion monitoring and intervention device based on multiple models, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is configured to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images