A family conflict perception and proactive intervention method and system

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a family conflict perception and active intervention method and system, dialogue audio of at least two family members in a family environment is continuously received by a microphone array, speech streams corresponding to each family member are obtained through speaker separation processing, acoustic features and text content of each speech stream are identified, emotional states of each speaker and whether a preset conflict mode exists in the dialogue are determined based on multi-modal emotion analysis, the emotional state and conflict mode identification result are input into a risk prediction model to predict a risk value of the dialogue escalating into a fierce conflict within a predetermined time window in the future, when the risk value exceeds a preset threshold, a non-judgmental mitigation intervention sentence is automatically generated based on the dialogue context without receiving an active request from a user, and the intervention sentence is output in voice through a loudspeaker. The present application realizes active perception and immediate intervention of family conflicts.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for sensing and proactively intervening in family conflicts. Background Technology

[0002] In recent years, with the deep integration of artificial intelligence and the Internet of Things (IoT) technologies, smart homes have evolved from simple device linkage and voice command execution to a relationship-based smart era that places greater emphasis on human emotional needs and family well-being. As the basic unit of society, the harmony of the family's internal relationships directly affects the mental health of its members and social stability. However, the fast pace of modern life exacerbates communication pressures within families, with arguments and conflicts becoming common factors that disrupt the family atmosphere. Traditional conflict resolution methods, such as family counseling, often suffer from delays, high costs, and low accessibility.

[0003] On the technological front, significant progress has been made in areas such as speech recognition, affective computing, natural language processing, and generative AI. Methods based on convolutional neural networks to process speech spectrograms have proven effective in identifying basic emotions such as happiness, sadness, and anger. Multimodal emotion recognition databases like IEMOCAP provide a standardized foundation for model training in this field. Regarding emotion recognition patents, Amazon's patent, filed in 2017 and granted in 2018, describes how Alexa devices receive user speech through microphones and analyze information such as volume, pitch, and fluctuations to detect whether the user is in an emotional state such as happiness, anger, or sadness. Google filed a patent application in 2023 concerning its voice assistant's ability to respond with emotional intelligence. Apple filed a patent application in 2019 to help Siri interpret user emotional states through facial analysis.

[0004] However, the aforementioned existing technologies have the following fundamental flaws: First, they all adopt a passive response mode, relying on the user to actively wake them up or make a clear request before they will react, and cannot proactively intervene when no one requests it; second, the purpose of emotion recognition is to optimize device control or personalized recommendations, rather than to intervene in language with the goal of easing interpersonal conflicts; third, they lack the ability to recognize complex conflict patterns. Existing technologies often remain at the level of classifying the emotional state of a single speaker and cannot handle complex tasks such as dynamic conflict pattern recognition, conflict escalation prediction, and situation-adaptive easing speech generation in multi-person dialogues.

[0005] In the field of companion AI, several related patent applications have emerged in recent years, involving maintaining the personality consistency of companion AI in multi-agent collaborative scenarios, recognizing users' emotional states through multimodal perception and outputting anthropomorphic feedback, and assisting parent-child communication based on interaction conflict data between guardians and children. However, none of the above technologies have achieved the organic integration of four aspects: continuous monitoring without wake words, speaker separation in multi-person dialogues, conflict pattern recognition and real-time escalation prediction based on psychology, and proactive, situation-adaptive, non-judgmental intervention. In the field of conflict resolution, some patents have also disclosed intelligent agent methods for real-time conflict mediation based on multimodal large language models. However, this technology is mainly aimed at relatively formal and structured scenarios such as contract disputes and civil mediation. The AI ​​plays the role of a tool to assist human mediators, rather than an intelligent system that independently and proactively intervenes in interpersonal dialogue in the private environment of the family. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a method and system for family conflict perception and proactive intervention, so as to solve the problem that the existing technology lacks proactive conflict intervention solutions in the family environment.

[0007] To achieve the above objectives, the present invention provides a method for family conflict perception and proactive intervention, comprising the following steps:

[0008] Acquire audio recordings of conversations between at least two family members in a home environment;

[0009] The dialogue audio is processed by speaker separation to obtain speech streams corresponding to each family member, and the acoustic features and text content of each speech stream are identified.

[0010] Multimodal sentiment analysis is performed based on the acoustic features and the text content to determine the current emotional state of each speaker in the dialogue and whether there is a preset conflict pattern in the dialogue.

[0011] The emotional state and conflict pattern recognition results of each speaker are input into a pre-trained risk prediction model to predict the risk value of the dialogue escalating into a violent conflict within a preset time window in the future.

[0012] When the risk value exceeds the preset trigger threshold, a mitigation intervention statement is automatically generated based on the current dialogue context without receiving an active request from a family member.

[0013] The mitigation intervention statement is output in voice form.

[0014] Preferably, the dialogue audio is continuously received via a multi-microphone array, and the speaker separation processing includes:

[0015] Extracting speech regions from a continuous audio stream through speech activity detection;

[0016] Extract the speaker embedding vector for each speech region;

[0017] Speech intervals are clustered based on the similarity of voiceprint embedding vectors to separate speech segments corresponding to different speakers;

[0018] The clustered speaker clusters are compared with pre-registered family member voiceprint templates to identify the family member identities corresponding to each speech segment.

[0019] Preferably, the acoustic features include one or more of the following: Mel frequency cepstral coefficients, pitch, volume, speech rate, and speech energy; the text content is obtained by automatically performing speech recognition on the speech streams of each speaker.

[0020] Preferably, multimodal sentiment analysis includes:

[0021] The speech segments of each speaker are converted into Mel spectrograms, and acoustic emotion features are extracted using a convolutional neural network.

[0022] The semantic sentiment features of the text content are extracted using a pre-trained language model;

[0023] By fusing the acoustic emotion features and the semantic emotion features, and using a long short-term memory network to perform temporal modeling of the emotion change sequence across dialogue rounds, the emotional state vectors of each speaker are obtained.

[0024] Preferably, the preset conflict mode includes one or more of the following: criticism mode, contempt mode, defense mode, and wall-building mode; the criticism mode corresponds to dialogue content containing accusatory statements, the contempt mode corresponds to dialogue content containing derogatory statements, the defense mode corresponds to dialogue content containing defensive justifications, and the wall-building mode corresponds to a dialogue state in which one party does not respond to the other party's words.

[0025] Preferably, the risk prediction model is a temporal neural network model, which takes the emotional state vector sequence of each speaker and the conflict pattern recognition result sequence as input, and outputs the probability value of the dialogue escalating into a fierce conflict within a preset time window in the future.

[0026] Preferably, the preset trigger threshold is dynamically adjusted based on the family's historical dialogue patterns and / or the user's preset sensitivity level; the risk prediction model adopts a reinforcement learning framework, and after each intervention, it dynamically optimizes the model prediction parameters and / or the preset trigger threshold based on the subsequent development of the dialogue.

[0027] Preferably, generating the mitigation intervention statement includes:

[0028] Obtain the context information of the current dialogue, which includes at least the speaker identifiers, text content, and emotional states of the most recent N rounds of dialogue;

[0029] The context information, detected conflict patterns, and family member relationship configurations are encapsulated as prompt word prefixes;

[0030] The prompt word prefix is ​​input into a generative language model, which is jointly supported by a conflict mitigation strategy library. The strategy library pre-stores various non-judgmental, humorous, distraction, or empathy-inducing mitigation discourse templates and their applicable context labels.

[0031] The generative language model, after being fine-tuned by instructions, is configured to prioritize the output of short, non-judgmental, positively oriented, and emotion-neutral utterances.

[0032] The output of the generative language model is subjected to safety filtering, and the utterances that pass the safety filtering are used as the mitigation intervention statements.

[0033] Preferably, speech recognition, speaker separation, multimodal sentiment analysis, and risk prediction are all performed locally on the device; only when generating the mitigation intervention statement is the desensitized dialogue context data sent to the cloud to invoke the generative language model, and the desensitization includes at least removing speaker-identifiable information and adding differential privacy noise.

[0034] This invention also provides a family conflict perception and proactive intervention system, comprising:

[0035] A microphone array for continuously receiving audio of conversations between at least two family members in a home environment;

[0036] The speaker separation module is used to perform speaker separation processing on the dialogue audio to obtain the speech streams corresponding to each family member, and to identify the acoustic features and text content of each speech stream.

[0037] The multimodal sentiment analysis module is used to perform multimodal sentiment analysis based on the acoustic features and the text content to determine the current emotional state of each speaker and whether there is a preset conflict pattern in the dialogue.

[0038] The risk prediction module is used to input the emotional state and conflict pattern recognition results of each speaker into a pre-trained risk prediction model to predict the risk value of the dialogue escalating into a fierce conflict within a preset time window in the future.

[0039] The intervention statement generation module is used to automatically generate non-judgmental mitigation intervention statements based on the current dialogue context when the risk value exceeds a preset trigger threshold and no active request is received from a family member.

[0040] The voice output module is used to output the mitigation intervention statements in voice form. The beneficial effects of this invention are as follows:

[0041] 1. This invention continuously receives audio recordings of conversations between at least two family members in a home environment and performs real-time speaker separation, voiceprint recognition, speech-to-text processing, and multimodal emotion and conflict analysis. This allows for continuous monitoring of family conversations without requiring user intervention or requests. Furthermore, this invention utilizes a pre-trained risk prediction model to predict the probability of a conversation escalating into a heated conflict within a predetermined time window. When the risk value exceeds a preset trigger threshold, it automatically generates and outputs intervention statements, thus shifting from a passive response to a proactive prevention model. This allows for timely intervention before conflicts escalate, achieving a preventative effect in maintaining family relationships.

[0042] 2. This invention combines mature psychological research findings with deep learning technology. Through automatic identification of preset conflict patterns such as criticism, contempt, defense, and wall-building, and joint analysis of multi-dimensional acoustic features including speech rate, pitch, volume, energy, and Mel-frequency cepstral coefficients, and then using a long short-term memory network to perform temporal modeling of emotional changes across dialogue rounds, it achieves accurate identification and dynamic tracking of complex conflict patterns, surpassing the technical limitations of traditional sentiment analysis systems that can only identify the basic emotions of a single speaker. Simultaneously, this invention employs a generative language model to dynamically generate intervention utterances based on the real-time dialogue context. It combines this with a pre-stored conflict mitigation strategy library containing various de-escalation utterance templates and their applicable context labels. Through instruction fine-tuning and reinforcement learning-based human feedback optimization, the model prioritizes outputting short, non-judgmental, positive-oriented, and emotionally neutral de-escalation utterances. Coupled with a post-processing safety filtering mechanism, the intervention content possesses high context adaptability, flexibility, and safety.

[0043] 3. This invention adopts a local-first, cloud-on-demand edge computing architecture. Core processing tasks such as voice activity detection, speaker separation, voiceprint recognition, and multimodal emotion and conflict analysis are all completed locally on the device. The original dialogue audio does not leave the device without user authorization. Only when it is necessary to call the cloud-based large language model to generate intervention speech or when the user explicitly authorizes data contribution, the desensitized text data (removing speaker identifiable information and adding differential privacy noise) is uploaded to the cloud through an encrypted channel. This maximizes the protection of the security and privacy of private family conversations and effectively responds to public concerns about privacy leaks of smart home devices. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a block diagram of a family conflict perception and proactive intervention system according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the family conflict perception and proactive intervention method according to an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0048] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0049] Example 1:

[0050] This embodiment provides a family conflict perception and proactive intervention system, such as Figure 1 As shown, the system includes a microphone array module, a speech processing module, a speaker separation and voiceprint recognition module, a speech-to-text module, a multimodal emotion and conflict analysis module, a conflict escalation prediction module, an intervention decision-making module, a de-escalation speech generation module, an emotional speech synthesis module, and a speaker module. Its core execution flow is as follows: Figure 2 As shown.

[0051] The microphone array module contains at least two high-sensitivity MEMS microphones arranged in a beamformation within the device housing for continuously capturing audio signals from common areas of the home. A voice processing module is connected to the microphone array module for preprocessing the audio signals, including noise suppression, echo cancellation, beamforming, and voice activity detection.

[0052] The speaker segmentation and voiceprint recognition module employs deep speaker embedding technology based on x-vector or ECAPA-TDNN architecture to segment and cluster the audio stream into speech segments of different speakers. Simultaneously, it compares each segment with pre-registered family member voiceprint templates to identify the speaker. The speech-to-text module, connected to the speaker segmentation and voiceprint recognition module, uses an end-to-end automatic speech recognition model based on the Transformer architecture to convert each segmented speaker's speech into structured text.

[0053] The multimodal emotion and conflict analysis module is connected to the speaker segmentation and voiceprint recognition module and the speech-to-text module. This module contains a first neural network model that simultaneously processes acoustic and textual features. Acoustic features include one or more of Mel-frequency cepstral coefficients, pitch, volume, speech rate, and speech energy. A convolutional neural network sub-model processes the speech spectrogram to extract acoustic emotion features, a BERT or RoBERTa sub-model processes the text to extract semantic emotion features, and a long short-term memory network sub-model captures temporal emotion evolution patterns across conversational turns. The module's output includes real-time emotion state labels for each speaker, such as anger, sadness, contempt, anxiety, etc., with intensity represented by continuous values ​​from 0 to 1, and conflict pattern identifiers, such as whether typical language patterns of criticism, contempt, defense, or defensiveness are present, with each pattern identified by a severity value from 0 to 1.

[0054] The conflict escalation prediction module is connected to the multimodal emotion and conflict analysis module and includes a second neural network model. This model can be a recurrent neural network built based on a temporal convolutional network or gated recurrent units. It receives the emotion state vector and conflict pattern vector sequence output by the analysis module in a time-step manner, and outputs a probability value p of the current conversation escalating into a heated argument within a predetermined time window. The value of p ranges from 0 to 1. The predetermined time window can be, for example, 30 seconds, 60 seconds, or 120 seconds, and can be adjusted according to user presets.

[0055] The intervention decision module connects to the conflict escalation prediction module and compares the obtained predicted probability value p with a dynamic threshold θ. The initial default value of the threshold θ is 0.7, which can be adaptively adjusted based on user feedback and historical intervention effects. When p ≥ θ, an active intervention process is triggered; otherwise, the system continues to passively monitor.

[0056] The de-escalation discourse generation module is connected to the intervention decision-making module. This module contains a generative large language model, such as a 7B to 13B parameter model based on Gemma, Qwen, or LLaMA architectures and specially fine-tuned. The generation process uses multi-turn historical text of the current dialogue, such as the previous 5 to 10 rounds, real-time emotion labels of each speaker, detected conflict patterns, and family member relationship configuration information as context to generate non-judgmental, context-appropriate de-escalation discourse. Generation strategies include humor defusal, attention diversion, guiding empathy, and pause prompts.

[0057] The emotional speech synthesis module connects to the soothing speech generation module, employing emotion-controlled speech synthesis technology based on a diffusion model or VITS architecture. It synthesizes the generated text into speech using a preset, gentle, and friendly tone. The tone is configurable, such as a human-like "little bear friend" or "little elf" tone. A speaker module connects to the emotional speech synthesis module to output the synthesized interventional speech.

[0058] For continuous dialogue tracking and speaker separation, the continuous dialogue tracking method of this invention includes the following steps: continuously acquiring ambient audio through a microphone array module; detecting speech intervals from the continuous audio stream using a speech activity detector; extracting the voiceprint embedding vector, i.e., x-vector, for speaker representation of each speech interval; aggregating similar speech intervals into the same speaker cluster using a clustering algorithm, and accurately locating speaker switching boundaries using a Hidden Markov Model; comparing the clustered speaker clusters with pre-registered family member voiceprint templates to identify speaker identity tags, such as father, mother, eldest son, etc.; and outputting a structured speech segment sequence with timestamps and speaker tags. Even under complex acoustic conditions in a home environment, such as background noise from television and air conditioner operation, this speaker logging technology can still achieve an accuracy rate of over 90% for continuous dialogue tracking of 2 to 4 family members.

[0059] Regarding multimodal sentiment and conflict analysis, this method includes an acoustic sentiment feature extraction path, a linguistic sentiment feature extraction path, a conflict pattern recognition path, and a cross-round temporal modeling path.

[0060] The acoustic emotion feature extraction path includes: extracting linear prediction cepstral coefficients and Mel frequency cepstral coefficients from each speaker's speech segment; converting the speech segment into a Mel spectrogram; inputting the spectrogram into a pre-trained convolutional neural network model, the structure of which includes 2 to 3 convolutional layers, batch normalization layers, pooling layers, and fully connected layers; and outputting an acoustic emotion feature vector for each speech segment, with a dimension of 7, corresponding to the probability distribution of common basic emotions such as happiness, sadness, anger, fear, surprise, disgust, and neutrality.

[0061] The linguistic sentiment feature extraction path includes: inputting the text output from the speech-to-text module into a BERT or RoBERTa pre-trained language model; the model outputs a semantic sentiment score for each text segment, including valence and arousal scores; and simultaneously performing keyword detection to identify phrase patterns containing criticism, contempt, defense, and wall-building patterns, such as accusatory phrases (including "You always..." "You never..."), generalized labels (such as "You're too selfish"), sarcastic remarks, defensive expressions (including "It's not my fault," "That's because you..." "Yeah, how come it's my fault again"), and silent response patterns (such as "Hmm," "Whatever," "Never mind"), etc., which are typical conflict signals.

[0062] The conflict pattern recognition path includes: matching the detected typical conflict signal phrase patterns with a pre-set "Four Horsemen of the Apocalypse" rule base ("Four Horsemen of the Apocalypse" refers to the intimate relationship prediction model proposed by psychologist John Gottman, which includes four destructive interaction patterns: criticism, defense, contempt, and cold war). This rule base contains a feature word list, syntactic template, and emotional tone combination pattern for each horseman pattern; calculating the severity score of each conflict pattern, for example, the severity of the criticism pattern is determined by the number of accusatory phrases, the use of intensity words, and the voice energy value; and outputting a conflict pattern vector containing four categories of scores: criticism, contempt, defense, and wall-building, with each category of score ranging from 0 to 1.

[0063] The cross-turn temporal modeling path includes: concatenating the acoustic sentiment feature vector, semantic sentiment feature vector, and conflict pattern vector obtained from each dialogue turn into a comprehensive feature vector; inputting this comprehensive feature vector into a long short-term memory network in chronological order, which contains 128 hidden units in a 2-layer stacked structure; the long short-term memory network outputs a contextual sentiment state embedding at each time step, which integrates the current dialogue state and the evolution information of previous multiple dialogue turns; and finally outputting a temporal feature sequence that can be used by the conflict escalation prediction module.

[0064] The conflict escalation prediction model is structured as follows: The input layer receives a sequence of temporal features from the output of a Long Short-Term Memory (LSTM) network, with a feature dimension of 256 at each time step. The temporal convolutional layer consists of two stacked one-dimensional convolutional layers with kernel sizes of 3 and 5 and a stride of 1, used to capture local conflict evolution patterns at different time scales. The attention layer employs a multi-head self-attention mechanism, containing four heads, used to identify important key moment signals in the temporal features, such as specific dialogue patterns before a high-energy argument erupts. The feature fusion layer weightedly fuses the convolutional features from different time scales. The prediction output layer is a multilayer perceptron with a sigmoid activation function, containing two hidden layers, each with 64 neurons, outputting a conflict escalation probability value p between 0 and 1.

[0065] This conflict escalation prediction model requires supervised training on a large-scale corpus of family dialogues labeled with escalation events. This corpus can be constructed from publicly available data sources such as emotional dialogue datasets, simulated dialogues from psychological counseling, and professionally annotated data (invited psychologists to annotate dialogues simulating family conflict scenarios), as well as anonymized data collected with user authorization during the collaborative deployment phase. The annotation task includes determining whether a heated argument occurred in the dialogue and when it occurred. The training objective is to minimize the cross-entropy loss between the predicted probability and the actual occurrence.

[0066] Regarding the generation of proactive intervention discourse, this method includes the following steps. The intervention decision module detects that the predicted probability p exceeds a preset threshold θ, activating the proactive intervention process. Contextual fragments of the current dialogue are extracted from the dialogue memory cache. The preset length is the number of the most recent N rounds of dialogue, where N is 5 to 10 and can be adjusted through user preferences. This includes speaker identifiers, text content, and sentiment tags for each round of dialogue, while also extracting a conflict pattern severity vector. The extracted contextual information, the real-time sentiment states of each speaker, the detected conflict pattern severity, and a family member relationship configuration table, such as the roles of father, mother, and child and their preset intimacy coefficients, are collectively encapsulated into a cue word prefix. The cue word prefix is ​​fed into a generative large language model, which has been specifically trained through instruction fine-tuning and reinforcement learning-based human feedback optimization. This model is designed to generate short, non-judgmental, positively oriented, and emotionally neutral discourse, avoiding didactic, accusatory, or obviously biased content. The model generates candidate intervention discourses autoregressively, with a length controlled to approximately 15 to 30 Chinese characters to ensure brevity and naturalness. A post-processing filter performs a security check on the generated candidate utterances, filtering out outputs that may contain potential controversies or ethical risks. If the security check fails, the model regenerates or selects the closest pre-set alternative utterance. The finally confirmed intervention utterance is then fed into the emotional speech synthesis module for speech synthesis and output.

[0067] The following two exemplary application scenarios further illustrate this point.

[0068] For example, in scenario A, which involves spousal conflict: when the system detects criticism and defense patterns in the conversation between the couple regarding the division of household chores, and the emotion scores show that both parties' anger values ​​are at a medium-high level (greater than 0.6), with a predicted probability p reaching 0.75, the system can use a lighthearted and humorous tone to say, "Oh dear, my two masters, I smell a strong gunpowder! How about we pause for a moment and let me dance for you to ease the tension?" The characteristic of this intervention is that it is non-judgmental, that is, it does not point out who is right or wrong, uses humor to reduce tension, and provides an alternative behavior, namely watching the system dance.

[0069] Scenario B involves parent-child conflict: When the system detects a dispute between a father and child over homework, with the father using a critical tone such as "Why are you always so careless!", and the child exhibiting a "wall-building" behavior (silence and non-response), and the system predicts a 0.82% probability of the conflict escalating further, the system can proactively intervene by saying, "Dad, the child seems a little scared now. Should we let him calm down for three minutes first, and then we can work on this problem together?" This intervention provides both parties with an emotional buffer while conveying an actionable goal: three minutes of quiet self-reflection.

[0070] Example 2:

[0071] This embodiment provides a specific system hardware configuration. The system has an anthropomorphic or object-like appearance, with an overall size of approximately 15 to 25 centimeters in height, suitable for placement on a desktop or bedside table. The outer shell can be made of soft silicone material, providing a comfortable feel and a certain degree of impact resistance. A base is located beneath the outer shell, containing a rechargeable battery and the main control circuit board.

[0072] The system features two LED expression light rings on the front, used to express the system's basic emotional state. For example, the LED array forms crescent-shaped eyes to indicate pleasure, round eyes to indicate curiosity, and an inverted V-shape to indicate concern.

[0073] A status indicator bar is located at the top or bottom of the system to clearly display the system's current operating status to the user. A solid green light or a breathing flash indicates sleep mode; in this mode, only the microphone array is in a very low-power silent detection mode, detecting only voice activity without acquiring or processing content. A solid yellow light indicates listening mode; the system is continuously receiving and analyzing audio, but this is happening in the background and has not yet reached the conditions for triggering intervention. A flashing orange light indicates analysis mode; the system has detected a possible conflict signal, and the conflict escalation prediction model is being activated and calculated. A short red flash, followed by one or two flashes before returning to other states, indicates intervention mode; the system has just completed an active intervention statement output. A solid purple light indicates privacy and security mode; the microphone array has been physically or logically muted, and the system does not accept any audio input. This state can be activated by the user pressing a physical switch on the back of the device or with a single click in the accompanying mobile app. This status indicator bar design allows users to clearly see the device's operation at all times, effectively alleviating privacy concerns about "always listening" and enhancing transparency and controllability.

[0074] Example 3:

[0075] This embodiment provides an edge computing and cloud collaborative data architecture. At the system architecture level, the present invention adopts the processing principle of "local priority and cloud on demand".

[0076] At the local processing layer, the raw speech data acquired by the microphone array is preferentially preprocessed, feature extracted, and key calculations are performed locally on the device. Modules for speech activity detection, speaker segmentation, voiceprint recognition, and multimodal emotion and conflict analysis, including convolutional neural network acoustic feature extraction and long short-term memory network temporal modeling, are all executed on the device's local AI acceleration chip, such as a SoC with an NPU unit. This means that the user's original dialogue audio content will not leave the device without the user's authorization. Only when the system determines that it needs to call a large language model in the cloud to generate intervention utterances, or when the user chooses to contribute data to help improve the model, will the anonymized text data, rather than the original audio, be uploaded to the cloud.

[0077] At the data transmission layer, communication between the local machine and the cloud is conducted via an encrypted channel using TLS 1.3 or higher protocols. Before transmission, all text data is anonymized, removing speaker-identifiable information and retaining only role designations such as A, B, and C, excluding names, and differential privacy technology is used to add a certain amount of statistical noise.

[0078] At the cloud processing layer, the cloud server is responsible for tasks including: reasoning and response generation of large language models, which are invoked when local small models are insufficient to generate sufficiently high-quality intervention utterances; optional data anonymization storage and model iterative training, which require explicit user consent; and synchronization and backup of user configuration data.

[0079] Upon initial activation, the system will display a clear privacy policy to users, explicitly stating which data is processed locally, which data may be uploaded to the cloud, the purpose of the data, and how to withdraw consent. All family members, with adult users requiring independent confirmation and minors requiring authorization from their legal guardians, must complete the authorization process before core intervention functions can be activated.

[0080] Through the aforementioned edge computing architecture, this invention minimizes the exposure of sensitive data and effectively addresses public concerns about privacy leaks in smart home devices.

[0081] Example 4:

[0082] This embodiment provides a dedicated authorization and guardianship control mechanism for family scenarios involving minor children.

[0083] To balance the privacy rights of minor children with the guardianship responsibilities of legal guardians, this system implements a tiered authorization management strategy.

[0084] Regarding guardian accounts, parents or legal guardians have the highest management authority. Guardians can create sub-accounts for their minor children, setting usage rules and intervention sensitivity levels, such as low, medium, and high. Simultaneously, guardians can choose to enable an emergency notification function. When the system predicts that the probability of conflict escalation exceeds an extremely high threshold (e.g., p≥0.9) and detects strong emotional signals that may involve personal safety risks, an anonymous summary alert can be sent to the guardian's mobile phone with their prior consent. An example message might be, "The system has detected a significant increase in emotional intensity in the conversation within the last five minutes; it is recommended to pay attention to the family communication atmosphere," but it will never include any specific details of the conversation.

[0085] Regarding the informed consent and experience of minors, although authorization is exercised by guardians, the system still explains its purpose and operation to underage users in language that children can understand. For example, the system will proactively introduce itself: "I am Little AI. My job is to say things that make people feel better when there are minor conflicts between family members. I will only remember the emotions people express, not the specific content they say!"

[0086] Regarding the principle of data minimization, the system implements a stricter data retention policy for dialogue data involving minors. All dialogue context data used for real-time analysis is automatically deleted after being stored locally on the device for no more than 24 hours. Apart from anonymized model training data with the explicit consent of the user, the system does not retain any long-term historical dialogue records.

[0087] Example 5:

[0088] This embodiment provides users with flexible intervention preference settings to meet the personalized needs of different families regarding the degree and style of system intervention.

[0089] Users can configure the following settings via the accompanying mobile app or by making voice commands on the system.

[0090] Regarding intervention threshold adjustment, users can choose from three preset threshold levels: High Sensitivity Mode corresponds to a low threshold θ=0.5, where the system tends to intervene earlier; Balanced Mode corresponds to a medium threshold θ=0.7, which is the default value; and Low Sensitivity Mode corresponds to a high threshold θ=0.85, which intervenes only in situations with a clear high risk. A custom option is also provided, allowing users to manually set any value between 0 and 1.

[0091] Regarding intervention style selection, users can choose from three preset styles: "Humorous and Gentle" uses lighthearted and witty language to ease tension; "Rational Guidance" uses gentle suggestions and reflective questions to guide emotional relief; and "Concise Reminder" provides short, concise reminders, such as "Take a deep breath." The system can also learn from user feedback to gradually adapt to the user's preferred style.

[0092] In terms of member relationship configuration, users can enter the role relationships of family members, such as "father-mother-daughter", as well as relationship tags, such as "husband and wife", "father and son" and "mother and daughter", for the system to refer to when generating intervention discourse, so as to select discourse that is more in line with the context of the relationship.

[0093] Regarding scenario exemption settings, users can specify certain conversation occasions where they will not actively intervene, such as during online work meetings, or time periods explicitly marked as "private conversations." The system can quickly activate the exemption mode via the voice command "Hey AI, please do not disturb for one hour."

[0094] Example 6:

[0095] This embodiment provides a specific application case to illustrate the actual operation process and beneficial effects of the present invention.

[0096] The application scenario is set as a family of four: father, mother, eldest son, and second daughter. The system is presented as a teddy bear plush toy placed on the coffee table in the living room. One day, the eldest son becomes addicted to mobile games because he hasn't finished his homework. The father, upon discovering this, speaks to him sternly, his voice characterized by increased volume, faster speech, and higher vocal energy. The system classifies this as anger (probability 0.82) and criticism (intensity 0.78). The eldest son then responds with an agitated tone, classifying his emotions as anger (0.65) and resentment (0.55), with a defense (intensity 0.71) and criticism. The father responds again, classifying his emotions as anger (0.91) and criticism (intensity 0.85).

[0097] The system has distinguished the dialogue rounds between the father and the eldest son during continuous dialogue tracking. The multimodal sentiment analysis module outputs that the anger scores of both the father and the eldest son exceed the preset high threshold of 0.6. The conflict escalation prediction module, based on the emotional evolution trajectory and the increasing trend of the conflict pattern, calculates the probability of further escalation of the argument within the next 30 seconds as p=0.86, exceeding the preset equilibrium mode threshold θ=0.7. The intervention decision module triggers an active intervention process. The de-escalation speech generation module, using the current dialogue's last three rounds of history, the father and eldest son's emotional labels, conflict patterns, and the father-son relationship as context, generates the following intervention speech: "Dad, Xiaoming just said he wanted to relax because he was too tired. Maybe we can ask him how much longer his homework will take, and discuss a schedule together, instead of making him feel blamed? This will make it easier for him to listen to us." This speech is synthesized and output in a gentle tone by the emotional speech synthesis module.

[0098] System monitoring showed that the anger scores of both parties decreased significantly after the intervention, and no further negative interactions occurred in the conversation. The system marked this intervention as a positive feedback sample and used the anonymized data for reinforcement learning training.

[0099] Example 7:

[0100] This embodiment uses a scenario of stress in a couple as an example to illustrate the progressive intervention capability of the present invention in the early warning stage of low-intensity conflict.

[0101] The application scenario is set as a young couple, with the system placed as a desktop ornament in the living room. One evening, the wife returns home to find unwashed dishes in the kitchen sink while her husband is on the sofa using his phone. The wife speaks to her husband in a complaining tone, with emotions categorized as anger (0.60) and fatigue (0.78), and a criticism intensity of 0.62. The husband responds in a defensive tone, with emotions categorized as anger (0.55) and resentment (0.60), and a defense intensity of 0.58.

[0102] The system detected that both parties' anger scores exceeded the warning threshold, and a conflict pattern sequence of criticism and defense had emerged. Simultaneously, the wife's fatigue score reached 0.78. The probability value p output by the conflict escalation prediction module was 0.65, which did not reach the default threshold θ=0.70, leading the system to determine that the current conversation was in a warning state. Based on a preset gradual intervention strategy, the system implemented a low-intensity intervention, generating the following reminder: "Everyone seems a bit tired. I heard someone got a super-extra meal at the hospital cafeteria today, and someone solved a troublesome bug—you can each tell me one small thing that made you happy today, and exchange your energy levels." This message was synthesized and output in a light and friendly tone by the emotional speech synthesis module.

[0103] System monitoring showed that the intervention guided both parties to begin sharing positive events from their daily lives, the emotional score of the conversation turned from negative to positive, and the conflict pattern sequence ended. The system marked this intervention as a positive feedback sample for continuous optimization training of the model.

[0104] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0105] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0108] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the various method embodiments described above.

[0109] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for perceiving and proactively intervening in family conflicts, characterized in that, Includes the following steps: Acquire audio recordings of conversations between at least two family members in a home environment; The dialogue audio is processed by speaker separation to obtain speech streams corresponding to each family member, and the acoustic features and text content of each speech stream are identified. Multimodal sentiment analysis is performed based on the acoustic features and the text content to determine the current emotional state of each speaker in the dialogue and whether there is a preset conflict pattern in the dialogue. The emotional state and conflict pattern recognition results of each speaker are input into a pre-trained risk prediction model to predict the risk value of the dialogue escalating into a violent conflict within a preset time window in the future. When the risk value exceeds the preset trigger threshold, a mitigation intervention statement is automatically generated based on the current dialogue context without receiving an active request from a family member. The mitigation intervention statement is output in voice form.

2. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The dialogue audio is continuously received via a multi-microphone array, and the speaker separation process includes: Extracting speech regions from a continuous audio stream through speech activity detection; Extract the speaker embedding vector for each speech region; Speech intervals are clustered based on the similarity of voiceprint embedding vectors to separate speech segments corresponding to different speakers; The clustered speaker clusters are compared with pre-registered family member voiceprint templates to identify the family member identities corresponding to each speech segment.

3. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The acoustic features include one or more of the following: Mel frequency cepstral coefficients, pitch, volume, speech rate, and speech energy; the text content is obtained by automatically performing speech recognition on the speech streams of each speaker.

4. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The multimodal sentiment analysis includes: The speech segments of each speaker are converted into Mel spectrograms, and acoustic emotion features are extracted using a convolutional neural network. The semantic sentiment features of the text content are extracted using a pre-trained language model; By fusing the acoustic emotion features and the semantic emotion features, and using a long short-term memory network to perform temporal modeling of the emotion change sequence across dialogue rounds, the emotional state vectors of each speaker are obtained.

5. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The preset conflict modes include one or more of the following: criticism mode, contempt mode, defense mode, and wall-building mode; the criticism mode corresponds to dialogue content containing accusatory statements, the contempt mode corresponds to dialogue content containing derogatory statements, the defense mode corresponds to dialogue content containing defensive justifications, and the wall-building mode corresponds to a dialogue state in which one party does not respond to the other party's words.

6. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The risk prediction model is a temporal neural network model, which takes the emotional state vector sequence of each speaker and the conflict pattern recognition result sequence as input, and outputs the probability value of the dialogue escalating into a fierce conflict within a preset time window in the future.

7. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The preset trigger threshold is dynamically adjusted based on the family's historical dialogue patterns and / or the user's preset sensitivity level. The risk prediction model employs a reinforcement learning framework, dynamically optimizing the model prediction parameters and / or the preset trigger threshold based on the subsequent development of the dialogue after each intervention.

8. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The generated mitigation intervention statements include: Obtain the context information of the current dialogue, which includes at least the speaker identifiers, text content, and emotional states of the most recent N rounds of dialogue; The context information, detected conflict patterns, and family member relationship configurations are encapsulated as prompt word prefixes; The prompt word prefix is ​​input into a generative language model, which is jointly supported by a conflict mitigation strategy library. The strategy library pre-stores various non-judgmental, humorous, distraction, or empathy-inducing mitigation discourse templates and their applicable context labels. The generative language model, after being fine-tuned by instructions, is configured to prioritize the output of short, non-judgmental, positively oriented, and emotion-neutral utterances. The output of the generative language model is subjected to safety filtering, and the utterances that pass the safety filtering are used as the mitigation intervention statements.

9. The method for family conflict perception and proactive intervention according to claim 1, characterized in that, The speech recognition, speaker separation, multimodal sentiment analysis, and risk prediction are all performed locally on the device; only when the mitigation intervention statement is generated is the desensitized dialogue context data sent to the cloud to invoke the generative language model. The desensitization includes at least removing speaker-identifiable information and adding differential privacy noise.

10. A family conflict perception and proactive intervention system, characterized in that, include: A microphone array for continuously receiving audio of conversations between at least two family members in a home environment; The speaker separation module is used to perform speaker separation processing on the dialogue audio to obtain the speech streams corresponding to each family member, and to identify the acoustic features and text content of each speech stream. The multimodal sentiment analysis module is used to perform multimodal sentiment analysis based on the acoustic features and the text content to determine the current emotional state of each speaker and whether there is a preset conflict pattern in the dialogue. The risk prediction module is used to input the emotional state and conflict pattern recognition results of each speaker into a pre-trained risk prediction model to predict the risk value of the dialogue escalating into a fierce conflict within a preset time window in the future. The intervention statement generation module is used to automatically generate non-judgmental mitigation intervention statements based on the current dialogue context when the risk value exceeds a preset trigger threshold and no active request is received from a family member. The voice output module is used to output the mitigation intervention statement in voice form.

Images