An intelligent intervention system and method for personalized parent-child communication based on multi-modal data fusion

Abstract

The application discloses a kind of based on multi-modal data fusion's individualized parent-child communication intelligent intervention system and method, it is related to intelligent wearing equipment technical field.The system includes local device and cloud server;Local device contains multi-modal perception module, intelligent power consumption management module, traffic optimization module, family portrait storage module, local multi-modal analysis model, pre-advice engine and privacy desensitization module;Cloud contains depth individualized analysis engine and federal learning module.The application realizes ultra-low power consumption by three-level hierarchical wake-up architecture, realizes traffic optimization by event-driven upload mechanism, all raw data do not leave device, only upload desensitized text, while protecting user privacy, provide real-time parent-child communication intervention and pre-advice guidance for parents.The application solves the technical blank that existing children's watch only monitors children's behavior, lacks the guidance of parent communication mode, has the beneficial effects of long endurance, low traffic, privacy security.

Description

Technical Field

[0001] This invention relates to the fields of smart wearable devices, affective computing, and personalized recommendation technology, specifically to a smartwatch system that integrates physiological monitoring and voice analysis and can provide real-time and forward-looking parent-child communication guidance based on family profiles. Background Technology

[0002] Existing parent-child communication assistance devices have the following technical shortcomings: 1. Single dimension: Only analyzing voice content, ignoring the speaker's physiological state (such as an elevated heart rate indicating emotional excitement) and differences in family background, resulting in "one-size-fits-all" speech suggestions that lack individuality; 2. Delayed response or limited scenarios: The responses are mostly post-event analysis or passive responses during conversations, failing to meet parents' needs for "pre-event consultation"; 3. Lack of personalization: The suggestions given are too general and cannot be adapted to the actual educational situations of different families because they cannot be combined with profile data such as parents' age, children's school age, and family income. 4. Power consumption versus battery life conflict: In 24-hour continuous monitoring scenarios, traditional solutions consume too much power and are difficult to achieve long battery life; 5. Concerns about data consumption: Frequent cloud interactions generate a large amount of data traffic, increasing the cost for users; 6. The conflict between privacy and real-time performance: Pure cloud solutions pose privacy risks, while pure local solutions lack sufficient computing power. Summary of the Invention

[0003] This invention aims to provide an intelligent wearable system that integrates multimodal data such as voice, physiology, and profiling. It can intervene in inappropriate language in parent-child communication in real time, proactively provide pre-consultation, and generate customized communication plans based on the individual characteristics of families. At the same time, through innovative power consumption and traffic management technology, it achieves a practical solution with long battery life, low data usage, and high privacy protection.

[0004] This invention provides a personalized parent-child communication intelligent intervention system based on multimodal data fusion, comprising two parts: a smart wearable device (in the form of a watch) and a cloud server.

[0005] Voice acquisition unit: Low-power microphone array, continuously monitors ambient sound, and determines parent-child dialogue scenarios based on voiceprint recognition. Physiological monitoring unit: integrates an optical heart rate sensor, a blood pressure monitoring sensor, and a skin conductance sensor to collect real-time data on the wearer's heart rate, blood pressure variability, skin conductance, etc. Motion sensing unit: accelerometer + gyroscope, to determine the wearer's current activity state (stationary / walking / vigorous exercise).

[0006] L1 Deep Sleep State: Only hardware-level Voice Activity Detection (VAD) is retained, detecting ambient sound pressure level, with power consumption in the microwatt range. L2 lightweight monitoring mode: Wakes up upon detecting a human voice, initiates voiceprint recognition and basic physiological monitoring (intermittent sensor sampling), with power consumption in the milliwatt range. L3 Full-Function Analysis Status: Wakes up after confirming a parent-child dialogue scenario, activating full speech recognition, multimodal AI analysis, and cloud collaboration functions.

[0007] —No voice for 30 seconds after the conversation ends, then it drops back to L2 from L3. —After 3 minutes of no speech or movement, the system will revert from L2 to L1. —Forced entry into L1 deep sleep during the late night period (11:00 PM - 6:00 AM)

[0008] —Select a low-power audio codec chip that supports hardware VAD acceleration. —The optical heart rate sensor uses an adaptive sampling rate, reducing the sampling frequency to 1 / 10 in non-interaction scenarios. —The screen uses a low-power OLED that only lights up when there is a notification.

[0009] —Cloud collaboration is triggered only when the local model confidence level is below a threshold (e.g., <60%). —Upload the consultation text only when the user initiates the consultation. —Daily summary reports are automatically uploaded only under WiFi conditions.

[0010] —The text is compressed and encoded before uploading, with a single upload size of <10KB. — Data simplification is achieved by adding noise using differential privacy technology.

[0011] —Pre-download commonly used script templates and model update packages in a WiFi environment —On mobile networks, only the minimum necessary data packets are transmitted.

[0012] —Built-in lightweight neural network model (1.5B-3B parameters) that has undergone knowledge distillation, simultaneously processing: —Voice emotion recognition: tone, volume, speech rate, word choice tendency —Physiological emotion inference: Increased heart rate + enhanced skin conductance → emotional excitement / anxiety —Multimodal fusion judgment: When "intense voice content" + "abnormal physiological indicators" are detected, it is judged as a high-risk communication scenario, triggering intervention.

[0013] —Tone dimension: volume, speaking speed, intonation —Content dimensions: imperative statements, negative evaluations, threatening language, etc. —Emotional dimension: anger, impatience, anxiety, etc. (verified by both voice and physical symptoms)

[0014] —Model quantization (FP32→INT8) reduces computational cost by 75%. —Model trimming, retaining only parent-child communication-related capabilities. —Key phrase wake-up model (<1MB) resident in L2 state

[0015] Parents' information: age, gender, occupation, educational background —Child information: age, gender, grade, academic performance overview, personality traits (e.g., introverted / extroverted) —Family Information: Family Structure (Single Child / Multiple Children), Income Level, Main Educational Philosophy This profile data is used for local personalized dialogue generation, without leaving the device.

[0016] —When the multimodal analysis model detects inappropriate communication, it immediately triggers a device vibration / screen / sound alert.** —Generate personalized communication messages based on family profiles: For example, if a child has average grades and is introverted, the prompts will focus on "encouragement + specific suggestions" rather than "criticism". —If the parents are younger, the prompts can be more conversational; if the parents have a higher level of education, the prompts can be more rational. —Prompt format: vibration + on-screen text + optional voice announcement

[0017] Parents can proactively ask questions via voice or the watch interface, for example: "I want my child to do their homework later, how should I tell them?" How should I comfort my child who didn't do well on a test? "How should I communicate with the teacher about the parent-teacher meeting tomorrow?" —The local model combines family profiles with historical communication data to generate preliminary recommendations. —If the local model has low confidence, cloud collaboration will be triggered (only the anonymized consultation text will be uploaded).

[0018] —All raw audio, physiological data, and family profiles remain on the device. —Upload anonymized data only in the following situations: —Upload the anonymized consultation text (containing no identifiable information) during the pre-consultation session. —When analyzing complex scenarios, upload the anonymized dialogue text (remove identity markers and add differential privacy noise). —Upload anonymized gradient data during model optimization (federated learning).

[0019] — Deploy a full-fledged large language model (e.g., with hundreds of billions of parameters). —The anonymized consultation text / dialogue text uploaded by the receiving device. —A general model trained using data from tens of millions of family communications generates more accurate communication suggestions.

[0020] --Key sales pitch suggestions — Multiple expression options are available (such as a gentle version, an encouraging version, and a rational version). —Explanation of relevant educational psychology principles (optional)

[0021] —Collect desensitization gradient data from different families —Through cluster analysis, identify communication patterns in different family types (such as "highly educated families", "dual-income families", and "families with multiple children"). —Continuously optimize the cloud-based model to make recommendations more targeted to different types of households. —The optimized model is updated to the device via OTA.

[0022] Step A1: The multimodal perception module continuously monitors voice and physiological data, while the power management module dynamically switches between L1 and L2 modes. Step A2: A parent-child dialogue scenario was detected, and the L3 full-function analysis state was activated. Step A3: Use voice analysis and physiological data fusion to identify inappropriate communication. Step A4: Combine local family profiles to generate personalized real-time prompts Step A5: Device vibration / screen display, immediate intervention. Step A6: If the local model has low confidence, trigger cloud collaboration (compress and upload via traffic optimization module). Step A7: Automatically revert to L2 / L1 state after the dialogue ends.

[0023] Step B1: Parents proactively ask questions (such as "How can we get our child to do their homework independently?"). Step B2: Generate preliminary suggestions by combining the local model with the family profile. Step B3: If the parents are not satisfied or the local model has low confidence, trigger cloud collaboration (compress and upload using the traffic optimization module). Step B4: Generate in-depth personalized suggestions from a large cloud-based model and return them to the device. Step B5: Parents can save, rate, and provide suggestions for future optimization.

[0024] Step C1: Generate today's communication report (excluding original content) — Communication duration, emotional distribution, and number of high-risk events —Physiological data trends (e.g., "Heart rate was higher when tutoring homework today") —Personalized recommendations ("Based on your family situation, we recommend...") Step C2: When a WiFi connection is detected, automatically upload a report to the parents' mobile app. Step C3: On mobile networks, only a "New Report" notification will be displayed; the user decides whether to download it.

[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. Multimodal accurate judgment: Verification using both voice and physiological dimensions to avoid misjudgment (such as judging inappropriate communication solely based on volume). 2. Truly Personalized: Scripts are generated based on family profiles (parents' age / income / educational background + child's grades / personality) to avoid a "one-size-fits-all" approach; 3. Dual-mode approach: It can intervene in real time and provide proactive consultation to meet the needs of parents in different scenarios; 4. Privacy and Security: The original audio remains on the device; only the anonymized text is uploaded. Combined with differential privacy technology, this completely eliminates users' privacy concerns. 5. Ultra-low power consumption design: Through a three-level layered wake-up architecture and hardware-level optimization, it achieves 3-5 days of battery life in 24-hour continuous monitoring scenarios; 6. Data usage friendly: Daily data upload volume is less than 100KB, so users don't need to worry about data charges; 7. Intelligent Evolution: Through federated learning, the model moves while the data remains stationary, with cloud-based models continuously optimized and local devices becoming smarter with use. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram of the present invention (showing the modules on the device side, the modules in the cloud, and the data flow). Figure 2 This is a state transition diagram of a three-level hierarchical wake-up architecture (showing the triggering conditions and switching logic for L1 / L2 / L3). Figure 3 This is a flowchart of the automatic recognition process for parent-child scenarios (showing a decision tree from continuous monitoring to activation analysis). Figure 4 This is a workflow diagram for edge-cloud collaboration (showing three processes: real-time intervention, pre-event consultation, and daily summary). Figure 5 This is a diagram illustrating privacy protection and traffic optimization mechanisms (showing how raw data is converted into anonymized text and compressed for uploading). Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1As shown, the local device 100 includes a voice sensor 102, a physiological sensor 104, a local AI model 106, and a privacy module 108; the cloud server 200 includes a deep analysis engine 202 and a federated learning module 204. The arrows in the figure indicate bidirectional data communication between the local device 100 and the cloud server 200.

[0029] like Figure 2 As shown, after the device is turned on, it defaults to L1 deep sleep state 300; when human voice is detected 310, it switches to L2 light monitoring state 302; in L2 state, if a parent-child scenario is confirmed 312, it switches to L3 full-function analysis state 304; in L3 state, if there is no voice for 30 seconds 314, it returns to L2 state 302; in L2 state, if there is no voice for 3 minutes 316, it returns to L1 deep sleep state 300.

[0030] like Figure 3 As shown, after starting at 400, it enters continuous low-power monitoring 402; it checks whether a human voice is detected 404, otherwise it returns to monitoring 402; if so, it activates L2 voiceprint recognition 406; it checks whether there is an adult + child voiceprint 408, otherwise it returns to monitoring 402; if so, it checks whether the conversation exceeds 30 seconds 410, otherwise it returns to monitoring 402; if so, it activates L3 full-function analysis 412, and the process ends 414.

[0031] like Figure 4 The diagram shown is a flowchart of the edge-cloud collaboration process of this invention, which includes three parallel sub-processes.

[0032] A. Real-time intervention process: Collect voice and physiological data 500, analyze locally 502 and determine whether it is high-risk communication 504; if so, immediately prompt to end 506; otherwise, determine whether the confidence level of the local model is too low 508; if so, upload to the cloud 510, receive cloud suggestions 512 and display 514; otherwise, return to local analysis 502.

[0033] B. Pre-consultation process: Receive user questions 520, generate preliminary suggestions locally based on family profile 522; determine if the confidence level is too low 524; if so, upload to the cloud 526, receive in-depth suggestions 528 and display 530; otherwise, directly display the preliminary suggestions.

[0034] C Daily Summary Process: Generate communication report 540; Determine if WiFi is connected 542; If yes, upload to mobile phone 544; Otherwise, wait for connection 546 and return to continue judgment.

[0035] like Figure 5As shown, in local device 600, the raw voice data 602, physiological data 604, and family profile 606 remain on the device and are processed by the local processing module 610. The local processing module performs data anonymization 612, adds differential privacy 614, and compresses the data 616, generating anonymized text 620 and model gradients 622. This data is then encrypted and uploaded 624 to the cloud server 630. The cloud server processes the uploaded data using a deep analysis engine 632 and a federated learning module 634, and returns the optimized model update 636 to local device 600. The label "Raw data never leaves the device" in the diagram indicates that all sensitive data is processed locally and not uploaded to the cloud.

[0036] [Background]: Father (35 years old, mid-level manager in a company), child (8 years old, second grade, above-average grades, lively personality). While helping with homework in the evening, the child was distracted, and the father raised his voice: "Concentrate! How many times have I explained this problem?"

[0037] 1. Power Consumption Status: The device is initially in L2 lightweight monitoring state (because evening is a high-frequency time for parent-child conversations); 2. The voice acquisition unit recognizes a parent-child dialogue scenario, and the device wakes up to L3 full-function analysis state; 3. The physiological monitoring unit detected that the father's heart rate increased from 72 bpm to 98 bpm, and his skin conductance response increased; 4. Local multimodal model determination: Voice content (accusation) + physiological data (emotional agitation) → high-risk communication; 5. Based on the family profile (child is lively, academic performance is acceptable), generate personalized prompts: —Screen / voice display: "Dad, you're getting a little anxious. Your child is lively and needs more specific guidance. Try saying, 'Let's look at this problem. What were you thinking about just now?'" —A heart rate alert will also be displayed: "Your heart rate has increased. Take a deep breath." 6. Upon seeing the prompt, Dad adjusted his tone, and the child refocused his attention; 7. If there is no voice communication for 1 minute after the conversation ends, the device will revert to L2; after 3 minutes, it will revert to L1.

[0038] [Background]: Mother (32 years old, self-media practitioner), child (10 years old, fifth grade, below average grades, sensitive personality). The child did not do well on the midterm exam, and the mother wanted to talk to the child, but was worried about saying the wrong thing, so she took the initiative to ask about the watch.

[0039] [User Question]: "My child's exam results are coming out soon. How should I talk to him about it?"

[0040] 1. The local model receives questions and generates preliminary suggestions based on the family profile (child's academic performance is below average + sensitive personality); 2. The local model had low confidence in "how to talk to sensitive children about grades," triggering cloud-based collaboration; 3. Traffic optimization module activated: The consultation text is compressed and encoded into a 2.5KB data packet, and uploaded through an encrypted channel; 4. The large cloud model returns a depth suggestion (approximately 1.2KB of compressed data): —Key phrase: "Mom noticed that some parts of your exam weren't quite right, but Mom is more concerned about whether you've been facing any difficulties lately." —Extended suggestions: Gentle version: "Let's look at which questions you already knew and which ones you need more practice on." Encouraging version: "You improved on your math word problems last time, but did you encounter too many new words in your Chinese this time?" Psychological principle: Avoid labeling ("You're just careless"), focus on the process rather than the result. 5. Mothers choose a "gentle" approach to communication, leading to smoother communication with their children; 6. The total data usage for this consultation was less than 4KB.

[0041] [Scenario]: Asking a child to organize their schoolbag. Family A: The child is 6 years old, just started first grade, and has an extroverted personality. —The device prompts: "Using a gamified approach: 'Let's have a competition to see who can pack their backpack first!'" Family B: Child is 10 years old, in fifth grade, has excellent grades, and is introverted. —The device prompts: "Use a respectful tone: 'Is everything you need for tomorrow ready? Would you like me to check it for you?'" Family C: Single father, busy with work, 8-year-old child. —The device prompts: "Use concise and clear instructions: 'Put your schoolbag away by the door so you won't forget it tomorrow morning. Let me know if you need anything.'"

[0042] —Weekdays: Parent-child conversations from 7:00-8:00 AM (60 minutes in L3 state), parent-child conversations from 7:00-9:00 PM (120 minutes in L3 state), and intermittent monitoring of L2 state during other times. —Weekend: A total of 4 hours of parent-child conversation throughout the day (Level 3), with the rest of the time at Level 2 / Level 1.

[0043] —Average daily L3 status time: 3 hours on weekdays, 4 hours on weekends —Average daily L2 state duration: 21 hours —L1 state duration: only during deep sleep at night Total battery life: 4.2 days** (300mAh battery) —Daily average data traffic: approximately 85KB** (including 2 cloud collaborations + 1 WiFi report synchronization).

Claims

1. A personalized parent-child communication intelligent intervention system based on multimodal data fusion, characterized in that, include: Deployed locally on smart wearable devices and deployed in the cloud on cloud servers; The local terminal includes: A multimodal perception module is used to collect the wearer's voice and physiological data in real time; The intelligent power management module adopts a layered wake-up architecture to dynamically adjust the device's operating status; The traffic optimization module is used to compress and event-driven transmit the data to be uploaded. The family profile storage module is used for locally encrypted storage of user-preset family characteristic information; A local multimodal analysis model is used to integrate voice features and physiological features to identify communication scenarios and emotional states, and to generate personalized real-time intervention suggestions by combining family profiles. The pre-consultation engine is used to respond to user-initiated questions and generate preliminary suggestions based on family profiles. The privacy desensitization module is used to desensitize the data to be uploaded; The cloud includes: A deep personalization analysis engine is used to receive anonymized data, generate in-depth suggestions using a large language model, and return them. The Federated Learning module is used to optimize cloud-based models using distributed, anonymized data.

2. The system according to claim 1, characterized in that, The intelligent power management module includes a three-level hierarchical wake-up architecture: In the first level of deep sleep, only the hardware-level voice activity detection function is retained to detect the ambient sound pressure level. The second-level lightweight monitoring state is activated when a human voice is detected to perform voiceprint recognition and basic physiological monitoring. The third-level full-function analysis state is activated when a parent-child dialogue scenario is confirmed, and complete voice analysis and physiological monitoring are performed. The states at each level are dynamically switched based on the scenario determination results to achieve power consumption optimization.

3. The system according to claim 1, characterized in that, The traffic optimization module includes: The event-driven upload unit only triggers cloud collaboration when the local model confidence level is below a threshold or when a user actively inquires. The data compression and encoding unit compresses and encodes the uploaded data, ensuring that the amount of data uploaded at one time is less than 10KB. The intelligent pre-loading unit pre-downloads commonly used script templates and model update packages in a WiFi environment.

4. The system according to claim 1, characterized in that, The multimodal sensing module includes: The voice acquisition unit is used to determine parent-child dialogue scenarios based on voiceprint recognition. A physiological monitoring unit is used to collect at least one of heart rate, blood pressure variability, and skin conductance. The local multimodal analysis model identifies a high-risk communication scenario when both speech content recognition and physiological indicator abnormalities occur simultaneously.

5. The system according to claim 1, characterized in that, The information stored in the family portrait storage module includes: Parental information: at least one of the following: age, gender, occupation, and educational background; Child information: at least one of the following: age, grade, academic performance overview, and personality traits; Family information: at least one of the following: family structure, income level, and educational philosophy.

6. A personalized parent-child communication intelligent intervention method based on multimodal data fusion, characterized in that, Includes the following steps: S1: Dynamically monitors device status through a layered wake-up architecture, and collects the wearer's voice and physiological data in real time; S2: Determine whether to enter a parent-child dialogue scenario based on voiceprint recognition; S3: If a dialogue scenario is entered, the full-function analysis state is activated, the local multimodal analysis model is started, and the voice features and physiological features are fused to identify the emotions in the communication. S4: When inappropriate communication is detected, a personalized real-time alert is generated and triggered based on the locally stored family profile; S5: Responding to user-initiated questions, generating preliminary suggestions based on family profiles; S6: If the confidence level of the local model is lower than the threshold, start the traffic optimization module to compress and upload the anonymized data to the cloud-based large model for in-depth analysis, and receive the returned optimization suggestions; S7: Automatically downgrade power consumption status based on the scenario after the conversation ends.

7. The method according to claim 6, characterized in that, The generation of personalized real-time prompts in step S4 includes: Adjust the tone, length, and specific suggestions of the prompts based on the child's age, personality, and academic performance in the family profile.

8. The method according to claim 6, characterized in that, The user-initiated questioning mentioned in step S5 includes: For upcoming parent-child communication scenarios, users can input consultation requests via voice or interface, and the system will return preliminary communication suggestions.

9. A smart wearable device, characterized in that, It includes the hardware module and embedded software corresponding to the method described in claim 6.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of claim 6.

Images