Integrating and summarizing relevant information from family event data

Abstract

This document describes systems and techniques directed to integrating and summarizing relevant information from family event data. Various examples are described herein, including a method that includes receiving, by a machine learning (ML) model, family data, and generating, by the ML model, a relevance based on the family data. The techniques further include generating, based on the relevance, a family summary that includes a textual summary of a subset of the family data that is relevant to the relevance. The techniques can then generate, based on the family summary, a family output for output to a user. By doing so, the user can gain a better understanding of their home through a relevant and useful summary of trends, patterns, and events that occur within their home.

Description

Background Technology

[0001] Home automation, monitoring, and surveillance devices and processes present users with a variety of data. For example, a doorbell camera might display package deliveries, guest arrivals, neighborhood animals, or passing cars. However, not all data is relevant or useful to the user. In the example of video doorbell data, things like neighborhood animals and passing cars might be uninteresting or useless to the user. However, some combinations of other different data might be relevant, such as weather reports and items left outside (e.g., packages left outside when the weather indicates it might rain). Therefore, while these services present users with a large amount of data, some of this data is useful, some is useless, and some is useful when combined with other data. Summary of the Invention

[0002] Home Agent Summaries (simply referred to as "summaries") are created by generative artificial intelligence (GenAI) to discover situations of interest occurring at home. Summaries are the product of AI analyzing raw events to extract relevant and useful information for the user. These summaries can originate from discovered trends, anomalies in patterns, immediate dangers, rare events, and situations reported by the user themselves.

[0003] Electronic devices can utilize, at least in part, large language models (LLMs) to generate summaries for a given situation. Agents (e.g., electronic devices) use LLMs to mimic reasoning, employ tools to interact with external software, use memory to track past conversations, and use reflection to validate their work—all attempts to autonomously achieve their goals. Beyond improving quality through stepwise reasoning, agents also introduce modularity, allowing components to be modified without rebuilding the entire system. For example, when changing command syntax in the future, it would be preferable to reuse the command ranking and selection logic and simply update the syntax generation steps. Similarly, when adding new AI capabilities, it may be advantageous to immediately utilize them as tools rather than making architectural changes.

[0004] This document describes systems and techniques designed to integrate and generalize information from household event data. Various examples are described, including a method that involves: receiving household data generated by one or more household monitoring sensors by a machine learning (ML) model, and generating one or more correlations by the ML model based on the household data. The method further includes: generating a household profile by one or more processors based on the one or more correlations, the household profile including a textual summary of a subset of the household data relating to the one or more correlations. The method further includes: generating a household output by the one or more processors based on the household profile, the household output being configured for output to a user.

[0005] Additionally, an apparatus is disclosed comprising one or more processors and a memory. The memory stores instructions that, when accessed by the one or more processors, cause the one or more processors to perform the described methods. Additionally, a non-transitory computer-readable medium is disclosed that stores instructions that, when accessed by one or more processors, cause the one or more processors to perform the described methods. Additionally, a computer programming product is disclosed that includes memory. The memory stores instructions that, when accessed by one or more processors, cause the one or more processors to perform the described methods.

[0006] The purpose of this invention is to introduce a simplified concept for integrating and generalizing information from household event data, which is further described in the detailed description below and illustrated in the accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0007] This document describes in detail one or more aspects of the systems and techniques used to integrate and generalize information from household event data, with reference to the following figures: Figure 1 An example home environment is shown where information can be integrated and generalized from family event data; Figure 2 An example user environment is shown in which information can be integrated and generalized from household event data; Figure 3 An example user device is shown in which information can be integrated and summarized from household event data; Figure 4 An example user interface (UI) is shown, which can display integrated and summarized information from household event data; Figure 5An example implementation for integrating and summarizing information from household event data is shown; Figure 6 Examples of machine learning trainers for training ML models according to one or more aspects of this disclosure are shown; Figure 7 An example LLM trainer is shown for integrating and generalizing information from household event data; Figure 8 Examples of prompting engineering for LLMs according to one or more aspects of the present invention are shown; Figure 9 Examples of low-rank adaptive (LoRA) training for LLMs are shown according to one or more aspects of this disclosure; Figure 10 An example method for integrating and summarizing information from household event data is shown; Figure 11 This illustrates another example method for integrating and summarizing information from household event data; and Figure 12 Another example method for integrating and summarizing information from household event data is shown.

[0008] Using the same number in different instances can indicate similar features or components. Detailed Implementation

[0009] Overview

[0010] This document describes systems and techniques designed to integrate and generalize information from household event data. Various examples are described, including a method that involves receiving household data by a machine learning (ML) model and generating correlations based on that data. The technique further includes generating a household profile based on these correlations, comprising a textual summary of a subset of the household data related to the correlations. The technique can then generate household outputs based on these profiles for distribution to users. By doing so, users gain a better understanding of their homes through relevant and useful summaries of trends, patterns, and events occurring within their homes.

[0011] Home agent summaries (referred to as "summaries") are created by generative artificial intelligence (GenAI) to discover interesting events occurring at home. A summary is the product of AI analyzing raw events to extract relevant and useful information for the user. The sheer volume and diversity of these raw events present a significant challenge for users. Manually reviewing and analyzing this data to identify anything they should pay particular attention to is time-consuming, inefficient, and often impractical. Existing smart home systems primarily focus on providing these raw events, such as displaying raw event data or allowing users to set predefined rules based on simple triggers. These systems lack the ability to automatically and proactively analyze the complex interactions of raw events to generate higher-level summaries that ignore trivial information (e.g., "The weather is cloudy today") while highlighting important facts (e.g., "We detected an unfamiliar face picking up a package near the front door").

[0012] Platforms for home intelligence profiling can collect historical and current raw events from a user's home and proactively provide personalized, periodic summaries of those events for each user. Artificial intelligence (AI) models (e.g., LLM-based models) can "humanize" these summaries rather than simply presenting a list of events, identifying important summaries while discarding trivial ones.

[0013] Devices that integrate and generalize information from household event data can reason and understand household situations in a near-infinite space. By using an LLM agent to reason about household situations, the device can handle situations it has never encountered before. It can also incorporate world knowledge through training on the internet to understand the relationship between problems and the device's capabilities. In various ways, summaries can help users gain a better understanding of what is happening in their homes, make users feel that something is protecting their homes, highlight the value that scaling events can bring, make summaries proactive so that users do not have to ask for them, serve as a starting point for future searches, and / or learn something simple, such as text, before providing video summaries.

[0014] For example, users could ask questions such as “what happened over (time period)?”, “is my home all good?”, or “is there anything important to report?”. An example goal of integrating and summarizing information from household event data could be to highlight trends in usage across different categories, the reasons for changes in those trends, or unusual events for typical time periods. In some examples, users might have multiple categories of devices in their homes. For instance, a summary based on camera data could include package deliveries and disposals (e.g., “a package was picked up by someone I don’t recognize”), alarms heard or otherwise recorded (e.g., “two alarms were heard on Wednesday”), and / or facial recognition (or unrecognized faces) (e.g., “an unfamiliar face was seen in the backyard and usually is not”). Example summaries based on presence and / or occupancy data may include summaries such as: “you usually leave the house for work at 7:00 am but were late on 1 out of 5 days this week”, “the kids usually get home from school at 3 pm but were early on Wednesday”, and / or “sleep mode was enabled for 4 hours less this week than last week”.

[0015] Additionally or alternatively, summaries based on energy usage may include the following: “6 manual temperature adjustments were made, 4 more than usual”, and / or “the front door was open for 2 hours while the air conditioning ran yesterday”. Summaries based on security may also include: “the garage door was left open Wednesday night” and / or “the front door was unlocked Tuesday night”. Furthermore, summaries based on connectivity may include: “the internet was out from 8 pm to 11 pm on Sunday night” or “your average download speed was 100 Mbps, which is 20% slower than usual”. Examples of summaries based on home environment lighting include: “the living room light was left on overnight on Thursday”. Additionally or alternatively, summaries based on entertainment (e.g., television, speakers) could include: “You spent 12 hours watching TV last week, which is 2 hours more than average.” In various respects, summaries based on home automation could include: “Movie Night Routine was supposed to run but did not because the device could not be reached.”

[0016] This can also include general queries. Consider user input prompts in the form of text input, voice input, etc.: “What is the state of my house right now?” In addition to input prompts, users can also access this type of information through applications (e.g., the Google Home App (GHA)), such as by viewing the “Favorites” or “Devices” tabs. Summaries from user queries can include summaries based on, but not limited to, camera data, occupancy data, energy usage, security, connectivity, lighting, entertainment, home automation, and miscellaneous home data. For example, a security summary could alert the user that “the garage door is closed, the front door is locked, and the alarm is armed.” In another example, a camera data summary could show users recent events from their cameras over the past five minutes. In a further example, the energy usage summary could be: "The HVAC is set to heat / cool at 62℉ / 82℉ and is currently off as the indoor temperature is 68℉." Additional examples include unit status or other information found that is relevant to the user.

[0017] Various methods can be used for content selection. Examples include, but are not limited to, event counting, ranking events (e.g., by count), event filtering, insight-based content, and nearest neighbor clustering. Event counting methods can provide users with a complete count of all events that have occurred in their home. For example, if a user asks, “How many events were detected in the past 12 hours?”, this method can give the user an accurate summary of the event count. Event ranking methods can allow users to rank household events based on importance and can be useful if the events are rare enough. In an example, an alarm sounding in a user's home would be of high importance, and this method would immediately alert the user with a summary of the alarm event. Similarly, event filtering methods can allow users to manually define events using scores or select important events based on user preferences. Users can enter that they are more concerned with an energy usage summary than an entertainment summary, or select a summary they want only about safety and occupancy data. Additionally, insight-based methods can provide users with a summary of patterns or anomalies observed in their home. For example, a user receives a summary stating, "I noticed that your average sleep time over the past week has dropped below six hours due to frequent late-night motion events." Further, the nearest neighbor clustering method takes events and clusters them to identify which events are highly similar and which are unique to the user. This method can provide the user with summaries of events within the same category, such as deliveries and visits, as well as summaries of events only useful to the user, such as a surprise marriage proposal captured on an outdoor camera.

[0018] Summaries can be of various lengths, including but not limited to single sentences, paragraphs, multi-sentence paragraphs, and / or lists. An example of a longer summary for a user is: "Good evening Chang family! Today was definitely a lively one. The morning routine got underway as expected, though the thermostat saw a slightbump, indicating someone was seeking extra warmth. The kids returned from school safely and seemed to relish some quiet time in their rooms. Nikkypicked up two packages from the porch. Later, the kitchen was a hive of activity, preparing for what turned out to be a fantastic game night and guest dinner in the living room! As the evening wound down, the house settled into its nighttime routine, with lights dimming, doors locking, and thetemperature adjusting for a comfortable sleep. A busy, but happy, dayoverall. By the way, here are some camera clips from game night that you mayenjoy Watching! (Good evening, everyone in the Chang family! What a lively day. Morning routines went as expected, though the thermostat looked a little high, indicating someone needed extra warmth. The kids returned safely from school and seemed to enjoy some quiet time in their rooms. Nikky retrieved two packages from the porch. After that, the kitchen was bustling with activity, preparing for what proved to be a fantastic game night and dinner party in the living room! As night fell, the house settled into its nighttime routine: lights dimmed, doors locked, and the temperature adjusted for a comfortable sleep. Overall, a busy and enjoyable day. By the way, here are some video clips from game night that you might enjoy watching!) A shorter summary example could be: “Today, the landscaping crew installed several trees in the backyard, delivering them through the garage door around 10 a.m. They worked for a couple of hours before leaving around 1 p.m. Azmine arrived around 2 p.m. and brought inside several packages, leaving around 6 p.m..” Even shorter examples are possible, such as: “Trash was picked up as scheduled, and a FedEx package was delivered that Angela picked up, but otherwise a normal day.” In some examples, the summary may take the form of a list. For example: Miguel left for an hour in the morning at 7:00 am and was home most of the day. A delivery was made at 11:00 am and Miguel brought it in. Penelope visited for 5 minutes and picked up some clothes. Angela left for work at 6:00 am and got home at 2:00 pm. Hank normally eats at 3:00 am and 3:00 pm.

[0019] Operating environment

[0020] Figure 1 An example smart home environment 100 is shown, in which information can be integrated and generalized from home event data. Typically, home environment 100 includes a network (e.g., a home area network (HAN)) implemented as part of a residence or other type of building with any number of network-connected devices configured to communicate in a wireless network. For example, environment 100 includes a smart fan 102, a smart socket 104, a smart camera 106, a smart outlet 108, a smart home controller 110, a smart sensor 112, a smart alarm clock 114, a smart door lock system 116, a smart home hub 118, a border router 120, a user device 122, a smart thermostat 124, an access point 126 (e.g., a smart networked device), a smart refrigerator 128, and a heating, ventilation, and air conditioning (HVAC) system 130. Any number of these network-connected devices can be implemented for wireless interconnection to communicate and interact wirelessly with each other. Network-connected devices are modular, intelligent, multi-sensing wireless devices that can be seamlessly integrated with each other and / or with a central server or system to provide a wide variety of useful implementations. Network-connected devices can also be configured to communicate via a network, which may include a wireless mesh network, a Wi-Fi™ network, or both.

[0021] As a non-limiting example, network-connected devices may further include: hazard detectors (e.g., for smoke and / or carbon monoxide), cameras (e.g., indoor and outdoor), lighting units (e.g., indoor and outdoor), sensors and detectors (e.g., ambient light detectors, occupancy sensors, doorbells and door lock systems), connected electrical appliances and / or controlled systems (e.g., stoves, ovens, washing machines, dryers, air conditioners, pool heaters, irrigation systems, and security systems), electronic and computing devices (e.g., televisions, entertainment systems, computers, speakers, intercom systems, garage door openers, ceiling fans, and control panels), and any other type of network connectivity implemented inside and / or outside a building (e.g., in home environment 100).

[0022] Consider a scenario where home environment 100 includes a delivery person 132. A smart camera 106 or smart sensor 112 can identify the arrival of delivery person 132 via facial recognition or proximity sensors. Home event data may include the identification of delivery person 132, the lock status of smart door lock system 116, and occupancy data indicating the presence of first child 134 and second child 136 within home environment 100. An example summary may be presented to a user (e.g., an absent parent) indicating that the door is unlocked, first child 134 and second child 136 are home alone, and delivery person 132 has arrived. This summary may be presented to the user periodically, automatically, or upon request. In some examples, the summary is presented on user device 122.

[0023] The home environment 100 may also include a dog 138. Dog 138 may have been left outside by the first child 134, and home event data may include the dog 138's identification, the outside temperature from an outdoor thermometer, and the duration the dog 138 has been outside based on a timestamp from a smart camera 106. An example summary may be presented to the user, indicating that the dog 138 has been outside for 45 minutes, the current temperature is 92℉, and the first child 134 was last seen near the back door 40 minutes ago, prompting the user to text the first child 134 to bring the dog 138 home.

[0024] The home environment 100 may also include a smart alarm clock 114. The smart alarm clock 114 may be located in the user's bedroom, and the home event data may include sleep schedules derived from smart sensors 112, the user's calendar data, ambient light levels from a light detector facing the window, and tracking data from wearable devices (e.g., smartwatches). An example summary might indicate that the user is sleeping later than usual, their calendar shows an early morning meeting, and sunrise will occur earlier than normal due to the time change.

[0025] The home environment 100 may also include a smart refrigerator 128. The smart refrigerator 128 can track food inventory via a weight sensor, and home event data can include door opening frequency, internal temperature levels, and product expiration data. Users can request a summary of their grocery and refrigerator usage over the past day. Example summaries can be presented to the user, indicating that milk is running low, eggs are set to expire in two days, and the refrigerator door was opened 15 times in the past 24 hours.

[0026] Home environment 100 may also include a smart socket 104. The smart socket 104 can control power to various appliances (e.g., HVAC 130, smart refrigerator 128), and home event data can include socket activation history, estimated energy usage, and daytime patterns. An example profile could indicate to a user that a space heater plugged into the smart socket 104 has remained on for five consecutive hours, and that no movement or user presence has been detected in the room. All these example network-connected devices can provide data for this technology to create the profile.

[0027] Figure 2 An example user environment 200 is illustrated, in which information can be integrated and summarized from household event data. In a first example user environment 200-1, user device 202 may be a mobile phone held by user 204. In a second example user environment 200-2, user device 202 may be a laptop computer used by user 204. In a third example user environment 200-3, user device 202 may be a wired earphone in user 204's ear. The user device may have a summary agent located in the memory of user device 202.

[0028] The summarizing agent can collect household event data (home graph devices, current device status, historical events, weather, etc.) from data sources and generate a household summary. The household summary can be a text summary of a subset of the household event data and can be generated based on the correlations between household event data. The household summary can be a historical or real-time summary presented to user 204 from user device 202. Furthermore, the household summary can be presented to user 204 proactively, automatically, or periodically. User 204 can also request a household summary through the summarizing agent on user device 202.

[0029] As an example, consider user 204's plans for their smart home (e.g., Figure 1 A request for a home profile (from user 204) is entered into user device 202. User 204 can query important events that occurred during the week. The profile agent can collect data on home security, energy usage levels, appliance usage, occupancy, and habits. In response, the profile agent can generate a home profile (e.g., a historical profile) presented to user 204 on user device 202. The home profile may include: a package delivered by a stranger on Monday morning, unusually high energy usage detected by the laundry smart plug from the dryer on Thursday afternoon, and the living room light being on all night on Wednesday and Friday.

[0030] In another example, user 204 may request a home summary of the current state of their household. A summary agent may generate a home summary (e.g., a real-time summary) presented to user 204 on user device 202. The summary agent may collect current occupancy data, connectivity data, energy usage levels, and camera status. The home summary may include: two people are home, one person is away, the router is online with a download speed of 100 Mbps, six lights are on, and all cameras are online and fully charged. The summary agent may also have identified that: the smart thermostat has cooled by 0.1℉, and the smart plug has recorded a power consumption of 0.002 kWh. However, this information may not be relevant to user 204, so the summary agent may exclude these findings from the home summary sent to user 204.

[0031] The summary agent can identify information relevant to user 204 and can allow user 204 to provide feedback to further personalize the information. For example, user 204 can request a household summary from the past 24 hours, and the summary agent can show user 204 that their bedroom window was open for two hours on Wednesday afternoon. User 204 can provide feedback by giving a thumbs up or thumbs down to the information in the household summary. If user 204 lives in a city with poor air quality, the household summary about their open windows might be important to user 204, so user 204 could give the summary agent a thumbs up. However, user 204 might frequently open their windows for ventilation and might not want a household summary about their window status, so user 204 might give a thumbs down.

[0032] Example device

[0033] Figure 3A user device 202 in example environment 300 is illustrated, in which information can be integrated and generalized from household event data. User device 202 is shown as having various non-limiting example devices, including a desktop computer 202-1, a tablet computer 202-2, a laptop computer 202-3, a television 202-4, a smartwatch 202-5, smart glasses 202-6, a gaming system 202-7, a smart appliance 202-8, a vehicle 202-9, earphones 202-10 (e.g., true wireless earphones, wired earphones), a hearing aid 202-11, a virtual reality (VR) headset 202-12, and an augmented reality (AR) headset 202-13. Other devices may also be used (e.g., home service devices, smart speakers, smart water monitors, baby monitors, Wi-Fi™ routers, drones, touchpads, drawing tablets, netbooks, e-readers, home automation and control systems, wall displays, or other household appliances). Note that the user device 202 may be wearable, non-wearable but portable, or relatively immobile (e.g., desktop computer and appliance).

[0034] User device 202 may include one or more processors 302 and memory 304 (e.g., non-transitory computer-readable medium). In some examples, memory 304 is part of a computer-programmed product. The one or more processors 302 may include any suitable single-core or multi-core processor (e.g., application processor (AP), digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU)). Memory 304 may include memory media and / or non-transitory storage media. In various respects, memory 304 includes a summary intelligent agent. An operating system (not shown) embodied as computer-readable instructions on memory 304 may be executed by the one or more processors 302.

[0035] User device 202 may additionally include user interface 306. In some examples, user interface 306 is a display on user device 202 in which a family summary (e.g., notification) from family event data can be presented. User device 202 may also include wireless communication module 308 for transmitting data over a wireless network. For example, wireless communication module 308 transmits data over a wireless local area network (WLAN), Bluetooth™, cellular networks (e.g., 4G, 5G), satellite communication networks, mesh networks, the Internet, etc.

[0036] User Interface

[0037] Figure 4 It shows Figure 2The user device 202 provides an example user interface (UI) 400. The user device 202 may display multiple network-connected devices on the UI 400, including a camera 402, a light 404, a thermostat 406, and a Wi-Fi™ router 408. The user device 202 may also present a home summary 410 on the UI 400, indicating important and relevant events from the day. The home summary 410 may be directed to the user (e.g., Figure 2 User device 202 can also display device status on UI 400, including light status 412, fan status 414, thermostat status 416, TV status 418, and speaker status 420. For example, a user might see one of the lights in indicator light 404 at 50% brightness, and thermostat status 416 showing the room thermostat in thermostat 406 at 70℉. Users can also navigate UI 400 via tab selection 422. In some examples, tab selection 422 includes a favorites tab, a device tab, an automation tab, an activity tab, and a settings tab.

[0038] Consider automatically providing users with their family information (e.g., Figure 1 The user device 202 can display a daily summary of the family environment (100) on the UI 400 as a banner, indicating that the gardener is active in the backyard between 9:00 AM and 10:30 AM, the delivery person leaves a package at the front door at 2:15 PM, and the children return home from school at 3:30 PM. The family summary 410 may be useful to the user because these events may not be easily accessible in the UI 400 using tab selection 422. In various ways, the user can decide to provide feedback on the family summary 410 via thumbs-up and thumbs-down icons located on the UI 400. A thumbs-up from the user can reinforce that this type of summary is useful and appropriately defined, while a thumbs-down can trigger improvements, such as omitting daily events or adding more specific details in future summaries. This feedback loop allows for continuous improvement of prompts and user-specific customization over time. In some examples, user feedback is given on the application interface (e.g., GHA).

[0039] In other examples, the home summary 410 may be a home output. The home output may be an audio message and / or audio data related to the home summary 410 configured to be output to a user. For example, a user receives their home summary 410 in the form of an audio message. The home environment 100 may be configured to deliver the audio message via a connected smart speaker and / or via user device 202. Instead of reading the home summary 410 from user device 202, the user can listen to the message. Similarly, the home output may be home video accompanying the home summary 410 configured to be output to a user. For example, user device 202 may present a short video clip of an unfamiliar face detected by an outdoor security camera, along with a summary stating “an unfamiliar face was seen at 3:00 am in the backyard and usually is not.” The user can use the context and video clip from the home summary 410 to determine if the unfamiliar face poses a security risk. In a further example, the home video may be images from two or more home monitoring sensors. Home monitoring sensors may include one or more of the following: smart doorbells, smart locks, security cameras, motion sensors, home hubs, wireless communication devices, smart thermostats, alarms, smart garage door openers, smart smoke detectors, smart water monitors, smart lights, smart sockets, smart switches, smart appliances, smart irrigation systems, smart speakers, smart displays, and smart TVs.

[0040] Figure 5 An example implementation 500 is shown, which enables the integration and summarization of information from family event data. Figure 1 The home environment 100 is configured to output home data 502 (e.g., home event data) to a machine learning (ML) model 504. The ML model 504 can output one or more correlations 506 to... Figure 2 User device 202. Based on one or more correlations 506, user device 202 will Figure 4 Family Summary 410 Output to Figure 2 User 204. In some examples, one or more correlations 506 include one or more of the following: time correlation, object detection correlation, event correlation, home device status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation. For example, one or more correlations 506 are between at least two or more different devices, and one or more correlations 506 are time correlations (e.g., two seconds, one microsecond). User device 202 can trigger the generation of one or more correlations 506.

[0041] In all aspects, home data 502 is generated by one or more home monitoring sensors, and ML model 504 is stored in the memory of the device (e.g., user device 202). Figure 3 In memory 304). One or more processors (e.g., Figure 3 One or more processors (302) can perform the generation of the family profile 410; however, in other examples, the ML model 504 can perform the generation of the family profile 410.

[0042] Machine Learning

[0043] As discussed in this disclosure, a machine learning (ML) model refers to a computer model that has been trained using one or more machine learning techniques. Generally, this training can be performed by providing training inputs to one or more trained models, which in turn can provide outputs. The outputs can be in the form of predictions, confidence scores, or other probability-based measures.

[0044] Figure 6 An example of a machine learning trainer 600 for training a machine learning (ML) model according to one or more aspects of this disclosure is shown. An ML model generator 602 may include training elements in the form of input 604, a training model 606, and output 608, and may be used to generate an ML model 610. Input 604 may be training data. For example, training data 604 may include smart device data, such as the on / off state of an appliance, thermostat readings, motion sensor data, or the brightness of a smart light. Training data 604 may also include user behavior and preferences, such as daily thermostat schedules or manual overclocking of light automation. In other examples, training data 604 includes environmental data, contextual data, user history data, energy usage data, or event log data. Training data 604 may be processed by the training model 606. Examples of training model 606 include multilayer perceptron (MLP) models, convolutional neural networks (CNNs), long short-term memory (LSTM) algorithms, generative adversarial networks (GANs), K-means clustering, Gaussian mixture models (GMMs), or any other variety of machine learning training techniques known to those skilled in the art. The training model 606 may include a single type of model or multiple types, as well as various combinations of types, including single type and combinations of multiple types.

[0045] The training model 606 can use supervised learning methods, where the training data 604 can be labeled, and the outputs 608 can be graded based on their fidelity to the "true" output. The training model 606 can also use unsupervised learning methods, where the training data 604 may be unlabeled, and the training model 606 can classify relevances without reference to the "true" values. The training model 606 can combine supervised and unsupervised techniques. The input 604 can come from the training data used to generate the ML model 610. Once the training of the ML model generator 602 is complete, the ML model 610 can be generated. The ML model 610 can be trained on the same device where it is stored or on at least one other device.

[0046] ML model 610 can be trained based on household data (e.g., Figure 5 Family data (502) is used to generate one or more correlations (e.g., family data 502) to generate one or more correlations. Figure 5 One or more correlations (506). To generate one or more correlations, the ML model 610 can generate one or more correlation values ​​between two or more members of the household data (e.g., network-connected devices). The one or more correlation values ​​can be based on the determined correlation quantities between the two or more members of the household data. As an example, consider a smart motion sensor and a smart light. Since the smart light automatically turns on when the smart motion sensor detects motion, the smart motion sensor can have a high correlation quantity with the smart light (e.g., a similar correlation value). In another example, consider a smart sprinkler system and a smart TV. The smart sprinkler system and the smart TV can have low correlation values ​​relative to each other because the smart sprinkler system waters the lawn, while the smart TV handles media.

[0047] ML model 610 can further compare one or more relevance values ​​with one or more thresholds. The one or more thresholds can be based on one or more of the following: data time, data type, data category, data history, or device type. ML model 610 can also determine that at least two members in the household data exceed one or more thresholds, and can thereby generate a data subset that includes those at least two members in the household data. The household profile can then be based at least on data for the two members identified in the data subset, thereby narrowing down the data from... Figure 1 The amount of incoming data generated by all network-connected devices 102-130. In fact, by pairing at least two members of the home environment 100 via this correlation-based method, the proposed solution limits latency and eliminates ambiguity regarding which data might need to be reported. Only events linked to data relevant between devices are selected as candidates for reporting. This also avoids illusions compared to solutions that would train on each individual sensor. As an example, consider… Figure 1 Dog 138 Figure 1 In a home environment 100, dog 138 may have been left outside by first child 134. ML model 610 generates correlation values ​​between motion sensors in the house, an outdoor camera, and an outdoor thermometer. Furthermore, ML model 610 compares these correlation values ​​to a threshold for the time dog 138 has been outside (e.g., 15 minutes or less) and a historical threshold for when first child 134 responded to dog 138 (e.g., first child 134 left dog 138 outside for 5 minutes). Based on the fact that the outdoor camera detected dog 138 outside for 35 minutes, the current outdoor temperature is 93℉, and the motion sensors in the house indicate the presence of first child 134, ML model 610 determines that the correlation values ​​of all devices exceed the threshold. Based on this indication, ML model 610 generates a subset of data (e.g., a summary) to be configured for output to the user. Example summary (e.g., Figure 4 The family summary 410 can indicate to the user that dog 138 has been outside in a high temperature of 93℉ for 35 minutes.

[0048] According to some examples, the generation of correlation values ​​and / or the comparison of correlation values ​​with thresholds can allow ML model 610 to scale for new devices (such as devices on which ML model 610 has not been trained) (e.g., the ability to use new devices not used in input 604). As an example, consider again a home environment 100, but with a new device: a smart awning. ML model 610 may never have been trained on the smart awning, but in this example, the smart awning may have a high correlation value with an outdoor camera (e.g., a correlation score based on device proximity, grouping from the user, or automatic grouping, etc.). The example summary can further indicate the status of the smart awning to the user based on the correlation value of the smart awning and / or the comparison of the correlation value of the smart awning with one or more thresholds.

[0049] In some examples, ML model 610 can compare correlation values ​​between devices that do not exceed a threshold. Consider the example above, but ML model 610 generates correlation values ​​between an indoor motion sensor, an outdoor camera, an outdoor thermometer, and a refrigerator door sensor. The refrigerator door sensor could indicate that the refrigerator door was opened approximately at the same time the outdoor camera identified Dog 138. However, the refrigerator door sensor does not exceed any threshold and is loosely correlated or not correlated at all with other devices, so ML model 610 does not include the door opening event in the subset of data. The summary delivered to the user includes relevant and helpful information.

[0050] Based on some examples, the summary is generated based on one or more of the following: correlation values ​​between devices, comparisons of correlation values ​​with thresholds, action correlation values, and device thresholds. Device correlation values ​​can be based on a determined correlation quantity between two or more of the following: available network-connected devices, categories of available actions for network-connected devices, categories of network-connected devices, data from network-connected devices, historical data, etc. Device thresholds can be used as comparison values ​​for device correlation values, categories of available actions, etc. For example, consider the previously summarized scenario of a dog being left outside. The proximity of the dog to the garage door coupled with the available action of the garage door opening can be given a high device correlation value and / or exceed a device threshold or device action threshold. In contrast, Figure 1 The refrigerator 128 can have very low correlation values, device correlation values, etc., and the device correlation value can not exceed the device threshold. It should be noted that the use of such values ​​(device correlation value, device threshold, etc.) allows for robust scalability with respect to new devices and / or new device actions, and the ML model 610 has not yet been trained on some of the new devices and / or new device actions.

[0051] The ML model 610 can be updated using additional training after it has undergone initial training. In all respects, the ML model 610 can have the same structure before and after training, but can have one or more different values, such as the initial and final values ​​of the weights and biases. In some examples, the ML model 610 can start with a different architecture before training, and in this way, the generated ML model 610 can have an architecture that is the product of the training completed in the ML model generator 602.

[0052] As an example, training an ML model can be accomplished by incorporating an LSTM algorithm. An LSTM algorithm is a type of recurrent neural network (RNN) that processes data in a time-indexed manner. LSTM can solve the so-called "vanishing gradient problem," in which the gradient used in the fitting may tend to zero and therefore may not produce useful fitting parameters (e.g., weights and biases) for a given model. During the training phase, LSTM allows persistent gradients to be used for fitting when the gradient might otherwise become zero (e.g., in a traditional RNN). Those skilled in the art will understand that other configurations using LSTM can also be used equivalently, and the examples given herein are intended to be illustrative rather than limiting.

[0053] Large Language Models (LLM)

[0054] Large Language Models (LLMs) are typically considered a type of AI. LLMs are trained on massive amounts of data to provide foundational capabilities that can often be used and reused by fine-tuning for specific applications and tasks. In contrast, other software applications are typically built and trained on specific data for each use case. In this way, LLMs are considered a type of foundational model.

[0055] Some LLMs use ML models that can parse language and provide context-aware outputs, such as mimicking human responses. This mimicry of human responses is often directed at prompts (e.g., from a user asking a question). The prompt "ask how to get to the train station in French" can be used as a prompt for the LLM to provide translation services (e.g., a French response to an English prompt).

[0056] As an example, consider Figure 7 The diagram illustrates a trainer 700, which trains an LLM for the system to integrate and generalize information from household event data. Trainer 700 receives training data as input, such as input 702. This training data can be of many different types, such as household event data. Figure 7 In the example shown, training input 702 is a phrase, but it can alternatively be a word, a long text segment (e.g., a book, article, or webpage), or any other data containing comprehensible text. In some examples, the text comes from a screen or image capture. In a process known as “lexicalization,” trainer 700 breaks down training input 702 into lexical units labeled lexical units 702-1, 702-2, 702-3, and 702-4. Here, training input 702 has a missing next word, labeled as blank 702-5. The goal of trainer 700 is to predict blank 702-5.

[0057] The trainer 700 encodes tokens (702-1, 702-2, etc.) into the input tensor through a mapping process. In 704, for example, the word "It" is mapped to the input tensor in 702-1. The first component of 704, 704-1, maps the word “'s” to the input tensor. The second component of 704, 704-2, maps the word "character" to the input tensor. The third component of 704 is 704-3, and the word “ize” is mapped to the input tensor. The fourth component of 704 is 704-4. Although the lexical units “It” 704-1 and “'s” 704-2 are shown as two parts of the word “It’s,” other mapping schemes exist (e.g., mappings based on discrete words or phonemes). In some cases, the ML model or ML component training 700 performs training input 702 to the input tensor. Lexicalization and / or mapping in 704 (e.g., feature extraction CNNs). Lexicalized training input 702 is mapped to the input tensor. The mapping in 704 may involve a lookup table that maps each possible lexical (e.g., 702-1, 702-2, etc.) to a known tensor object in the language space of the training data.

[0058] Transformer 706 will input tensors 704 is used as input, and the goal is to transform the input tensor... 704 Transformation into Transformed Tensors 708 is used to predict the blank 702-5. The transformation process is mathematically represented as follows:

[0059] In Equation 1 T This represents transformer 706. The transformed tensor. 708 includes components 708-1, 708-2, 708-3, 708-4, and 708-5. Component 708-1 is the transformation of component 704-1 through transformer 706 (similarly for component pairs 708-2 / 704-2, 708-3 / 704-3, and 708-4 / 704-4). Component 708-5 corresponds to the blank 702-5, and therefore component 708-5 is the prediction for the blank 702-5. In addition to the contextualization of components 704-1 through 704-4, the final transformed tensor is derived as part of the transformation process. 708 component 708-5.

[0060] Input (e.g., input tensor) Training input 704 and / or training input 702 typically includes multiple lexical units. For example, training input 702 includes lexical units 702-1 through 702-4. Trainer 700 transforms a single training input (e.g., training input 702) into multiple training inputs. For example, by removing lexical unit 702-4, when training input 702 calls trainer 700 to predict lexical unit 702-4, blank 702-5 is shifted to the left, thereby creating a new training input from the original training input 702. Since the value of lexical unit 702-4 is known in this example, the new input is a labeled input, which allows the new input to be used by supervised ML training algorithms (it should be noted that such inputs can also be used by unsupervised ML training algorithms). In this way, a single text containing multiple lexical units (e.g., a book, research paper, etc.) is used as multiple training inputs for trainer 700.

[0061] In some cases, it is desirable to bootstrap the LLM's output without fine-tuning it. Programmers may want to add functionality from an already trained LLM to their application via Application Programming Interface (API) calls, including all the latest features of the LLM, without requiring additional training or maintenance from the programmer. In such cases, the only way to bootstrap the LLM's output is through input hints. Customizing input hints to achieve the desired output is called hint engineering.

[0062] As an example, consider Figure 8 The diagram illustrates prompt engineering 800. Input prompt A is an attempt to obtain the desired result B using LLM. LLM includes different computational pathways (“paths”) based on the form of input prompt A. At least conceptually, there may exist an ideal path 802 that obtains the desired result B from input prompt A in the most efficient way possible. Other paths also exist, such as an erroneous path 804 that yields an undesirable result D, an intermediate path 806 that yields an intermediate result C, another intermediate path 808 that yields the desired result B from the intermediate result C, and an inefficient path 810 that yields the desired result B from input prompt A. Many paths may exist, limited only by the scope of LLM and the scope of input prompt A. The ideal path 802 can be mathematically represented as follows:

[0063] In Equation 2 S This represents the ideal path 802, and This indicates input prompt A, which is generated by the language space. (For example, Figure 7 Lexical units 702-1) and used for language space Path propagation function Characterization. It may be difficult to distinguish the various paths 804-810 from the ideal path 802. To achieve or reasonably approximate the ideal path 802, input prompts should be provided. Make the following changes:

[0064] In Equation 3 This represents a path that is similar to the ideal path 802 ( The deviation is a function ,in From parameters Characterization. During prompting process 800, various iterations of entering prompt A are fed into the LLM to attempt to find an acceptable path. As an example, one form of entering prompt A fails to achieve the desired result B (e.g., incorrect path 804) and is discarded. In another example, another form of entering prompt A gives an intermediate path 806 to reach intermediate result C, and a subsequent prompt gives an intermediate path 808 from intermediate result C to the desired result B. While this achieves the desired result B, it involves multiple steps, which is less efficient than a single direct prompt. In another example, another form of entering prompt A gives an inefficient path 810 that reaches the desired result B. In yet another example, another form of entering prompt A gives an ideal path 802 (or an acceptable approximation) to the desired result B. The characterization of the variations in the form of prompt A can be mathematically represented as follows:

[0065] Equation 4 shows the path deviation With parameters The changes, among which This is the maximum acceptable value. In an ideal scenario, for example, Consider an example where a user requests an application agent to provide a current summary of the user's home's status. The application agent has access to an LLM and utilizes a prompt engineer 800 that has been trained with various forms of input prompts A. Consider a prompt A in the form of "what is the status of my smart home?". Suppose the LLM responds by providing a list of every smart device that has ever been connected to the home network, including devices that are offline or have been disconnected for a long time, but without any indication of their current status or relevance. The user may have intended to receive a concise summary of current activities or important conditions, such as whether the door is locked, the lights are off, or the HVAC is running. Instead, the LLM output may be inconsistent with the user's intent when they entered prompt A. This example output from the LLM is represented by an unexpected result D, which is not an acceptable path for the prompt engineer 800.

[0066] Consider another example of the above form of cue A, but now assume that a subsequent step exists in the training of the applied agent as follows: another cue of the form “summarize only the currently active and relevant smart devices and their statuses” is automatically entered, causing the LLM to filter out offline devices and present a concise report highlighting important systems such as active security cameras, unlocked doors, lighting in occupied rooms, and current HVAC settings. Given an initial overly complex list of all historical devices represented by intermediate result C, the resulting summary is represented by the desired result B. This process is represented by the additive path 806+808. While the desired result B is achieved, deviations may occur. Still higher than the maximum acceptable value This may be due to computational costs preventing the entry of multiple prompts or other limitations.

[0067] Consider an example of an application agent instantiating input prompt A within a smart home control interface, where prompt A has the form "anything important to report this past week?". Further consider the application agent parsing input prompt A via prompt engineering 800 to have an updated form "summarize notable smart home events from the past seven days, prioritizing security alerts, system malfunctions, and unusual patterns" (the updated prompt could be based on knowledge of unsuccessful forms of prompt A, the form of prompt A used in prompt engineering during the application agent's training phase, etc.). LLM could return a list including every door opening, lighting change, and temperature fluctuation, even during routine and insignificant timeframes, leading to information overload and making it difficult for the user to discern anything truly important. Bias It may exceed the maximum acceptable value again. This is illustrated by inefficient path 810. In a similar example, suppose input prompt A is reformatted by the application agent into the form "summarize only high-priority smart home events from the last seven days, including security alerts, system errors, and deviations from normal energy use or occupancy patterns." The result could be a concise and insightful report highlighting missed door lock events, temporary HVAC malfunctions, and unexpected increases in evening energy use. This report provides user awareness without overly complex details, which leads to biases. The value falls within the maximum acceptable value At or below, this is represented by the ideal path 802.

[0068] Each of the forms of input prompt A in the previous examples can be used to train the application agent to parse input prompt A in order to reformat the input prompt in a manner consistent with the intent of input prompt A. Prompt engineering 800 utilizes the failure paths from all forms of input prompt A to obtain an acceptable, efficient, and concise prompt that produces the desired result B. In some examples, the parsing of input prompt A to determine the intent can be done by LLM. LLM can also be used in prompt engineering 800 to generate additional forms of input prompt A, find subsequent prompts, classify intents, or perform other aspects of prompt engineering 800.

[0069] In some examples, hint engineering can be used as part of the overall LLM manipulation scheme. Consider Retrieval Augmentation Generation (RAG). In examples using RAG, the LLM is not fine-tuned, but hint engineering is used to incorporate new data (e.g., data not used to train the LLM). Although other mechanisms (e.g., code for information retrieval or synthetic plotting) can be incorporated into the RAG, hint engineering is an integral part of the RAG. Throughout this disclosure, the idea of ​​hint engineering is intended to cover LLM manipulation methods that do not fine-tune the LLM.

[0070] Figure 9 An example low-rank adaptive (LoRA) training 900 for the LLM 902 is shown. LoRA training 900 can be used to fine-tune the LLM 902. One advantage of LoRA training 900 is that not all parameters 904 of the LLM 902 are tuned, resulting in a much lower computational cost compared to fine-tuning all parameters 904 of an existing LLM 902. In some examples, LoRAs can be generated to customize the LLM 902 for home automation and monitoring tasks, or multiple LoRAs can be generated, each for a specific home automation and / or monitoring application.

[0071] LoRA training 900 uses a trained ML model 906. The trained ML model 906 has LoRA weights 908, which modify only some parameters in parameter 904 of the LLM 902 (indicated by dashed line 910). The LLM 902 can be represented as (e.g., by...) Figure 7 The pre-trained weight matrix of the trainer (700). ,in and Representation matrix The dimension. In Full Fine-Tuning (FT) training, by modifying the matrix To modify the matrix This modified matrix also has dimensions. The matrix. In the LLM 902 example, which is large (e.g., hundreds of billions of parameters), modifying the matrix... It is also large, and therefore potentially intensive in terms of both computational training resources and storage resources. The problem becomes complex in examples seeking multiple LLMs trained by the Fourier Transform (FT).

[0072] LoRA training at 900 can significantly reduce the need to modify the matrix. The cost. Consider the following equation:

[0073] In equation 5, It has dimensions The matrix, and It has dimensions The matrix. In the small limit, Thus, respectively and It is a contravariant and covariant vector of rank 1. In all respects, it is stored and used as a dimension. Matrix In comparison, this significantly reduces dimensionality and thus the computational cost of fine-tuning. Consider the following... The LLM 902 is represented, where A total of 198,025,000,000 parameters are given. 900 are trained using LoRA. At the low end, It can be formed by two vectors with only 445,000 dimensions. and This results in a reduction in the following dimensions:

[0074] Equation 6 shows the modified matrix Size is Examples of 0.00022472% of the size. In some examples, the result is that the LLM 902 can be FT trained using a relatively small set of parameters (e.g., LoRA weights 908). In this way, FT LLMs based on LLM 902 can be created. For example, if LLM 902 is a general LLM for home management and the general LLM for home management is at least in part the product of LoRA training 900, then multiple specialized LLMs based on different aspects of home management can be easily stored on the user device (e.g., Figure 2 On the user device 202). A dedicated LLM may include an appliance maintenance LLM that monitors the performance data of smart devices, a daily automation LLM that personalizes and manages daily routines throughout the smart home, or a family coordination LLM that integrates calendar data from across the home to manage the schedules of family members.

[0075] Example Method

[0076] Figure 10 , Figure 11 and Figure 12 Example methods 1000, 1100, and 1200 for integrating and summarizing information from household event data are described respectively. Methods 1000, 1100, and 1200 are shown as sets of operations (or actions) performed, but are not necessarily limited to the order or combination of operations shown herein. Furthermore, any one or more operations can be repeated, combined, reorganized, or linked to provide a wide range of additional and / or alternative methods. References may be made in the sections discussed below. Figure 1 The family environment is 100, and Figures 1-9 The entities detailed herein are referenced only as examples. This technique is not limited to being performed by one or more entities operating on a device.

[0077] Figure 10 An example method 1000 for integrating and summarizing information from household event data is shown. At 1002, household data is generated by one or more household monitoring sensors (e.g., Figure 5 Home data 502). One or more home monitoring sensors may include one or more of the following: smart doorbells, smart locks, security cameras, motion sensors, home hubs, wireless communication devices, smart thermostats, alarms, smart garage door openers, smart smoke detectors, smart water monitors, smart lights, smart sockets, smart switches, smart appliances, smart irrigation systems, smart speakers, smart displays, and smart TVs. At 1004, by an ML model (e.g., Figure 6 The ML model (610) receives household data. In all respects, the ML model is an LLM (e.g., Figure 9 The LLM 902). An LLM can include one or more LoRA layers and can be trained at least partially using RAG. The ML model can also be stored in a first device (e.g., Figure 2 The memory of the user device 202) (e.g., Figure 3 In the memory 304).

[0078] At position 1006, one or more correlations are generated by the ML model (e.g., Figure 5One or more correlations (506). The generation of one or more correlations may be based on household data. In the example, one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, household device status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation. The generation of one or more correlations may include one or more correlations between two or more household monitoring sensors. The ML model may further generate one or more correlations by generating one or more correlation values ​​between two or more members (e.g., devices) in the household data. One or more correlation values ​​may be based on a determined amount of correlation between two or more members in the household data. The ML model may compare one or more correlation values ​​to one or more thresholds, which may be based on one or more of the following: data time, data type, data history, or device type. If at least two of the two or more members in the household data exceed one or more thresholds, the ML model may generate a subset of data that includes those at least two members in the household data.

[0079] At position 1008, generate a family profile (e.g., Figure 4 Family summaries (410). A family summary may include a textual summary of a subset of family data relating to one or more correlations. In some examples, an ML model generates the family summary, and in other examples, one or more processors (e.g., Figure 3 One or more processors (302) generate a family profile. One or more processors may be housed in a second device, distinct from the first device, and the first and second devices may communicate wirelessly. In some examples, the second device includes a mobile device (e.g., Figure 2The user device 202), and the first device includes one of a cloud computer, a server, a home hub, or a wireless communication device. The generation of the home profile may be based on one or more correlations, and may further be based on the device capabilities of one or more home monitoring sensors. In some examples, the generated home profile includes multiple home profiles. The generation of the home profile is performed automatically or periodically in other examples. In some examples, the ML model generates the home profile based on a subset of data from one or more correlation values. Further, the generation of the home profile may include generating one or more event correlation values ​​between two or more of the one or more home monitoring sensors and / or home data. The one or more event correlation values ​​may be based on a determined amount of correlation between the two or more devices and / or home data. Additionally, at least one of the two or more devices and / or home data is a new device and / or new home data on which the ML model has not yet been trained. The generation of the home profile may further be based on a comparison of one or more event correlation values ​​with one or more event thresholds. In some aspects, the generation of the home profile is based on event correlation values, and the ML model generates the event correlation values.

[0080] At point 1010, a family output is generated based on the family profile. The family output can be configured to be output to users (e.g., ...). Figure 2 (User 204), and the configuration of the home output can be based on UI elements (e.g., Figure 4 The UI (UI 400) format is used. In some examples, the UI element is a mobile device notification. Configuration for output to the user's home output can also be performed by an ML model.

[0081] Figure 11 An example method 1100 for integrating and summarizing information from household event data is illustrated. Method 1100 includes method 1000. At 1102, a household output is output to a user by one or more processors. At 1104, audio data is generated. In various aspects, the audio data includes a reading of a text summary of a subset of the household data relating to one or more correlations. The household output may further include the audio data. At 1106, a household video is generated based on one or more correlations. The household output may further include both the text summary and the household video. In various aspects, the household video includes images from two or more household monitoring sensors.

[0082] Figure 12An example method 1200 for integrating and summarizing information from household event data is shown. Method 1200 includes method 1000. At 1202, a user request for relevance to household data is received. In various aspects, one or more relevances are generated (e.g., Figure 10 At step 1006, the response is to a user request regarding the relevance of household data. At step 1204, the generated household profile includes multiple household profiles and these profiles are ranked. In some examples, the ML model performs the ranking of multiple household profiles. The household output may be further based on the ranking of the multiple household profiles.

[0083] Additional examples

[0084] Some additional examples are described below.

[0085] Example 1: A method for generating a household data summary, the method comprising: receiving household data generated by one or more household monitoring sensors by a machine learning (ML) model, and generating one or more correlations by the ML model based on the household data. The method further comprises: generating a household summary by one or more processors based on the one or more correlations, the household summary including a textual summary of a subset of the household data relating to the one or more correlations; and generating a household output by the one or more processors based on the household summary, the household output being configured for output to a user.

[0086] Example 2: According to the method of Example 1, the generation of the one or more correlations by the ML model based on the family data includes: generating one or more correlation values ​​between two or more members in the family data by the ML model, the one or more correlation values ​​being based on a determined correlation amount between the two or more members in the family data. The generation of the one or more correlations further includes: comparing the one or more correlation values ​​with one or more thresholds by the ML model; determining by the ML model that at least two of the two or more members in the family data exceed the one or more thresholds; and generating a data subset including the at least two members of the two or more members in the family data.

[0087] Example 3: The method described in Example 2 further includes generating the subset of the family data based on one or more correlation values ​​between two or more members in the family data.

[0088] Example 4: The method described in Example 3, wherein the generation of the family profile is further based on the subset of the family data.

[0089] Example 5: The method according to any one of the foregoing examples further includes the one or more processors outputting the home output to the user.

[0090] Example 6: The method according to any one of the preceding examples, wherein the configuration for outputting the family output to the user is based on the format of a user interface (UI) element.

[0091] Example 7: The method described in Example 6, wherein the UI element includes a mobile device notification.

[0092] Example 8: The method described in Example 1 further includes generating audio data. The audio data includes a reading of the text summary of the subset of the household data relating to the one or more correlations, and the household output includes the audio data.

[0093] Example 9: The method described in Example 1 further includes generating a family video based on the one or more correlations. The family output includes the text summary and the family video.

[0094] Example 10: The method described in Example 9, wherein the home video includes images from two or more home monitoring sensors among the one or more home monitoring sensors.

[0095] Example 11: The method according to any one of the preceding examples, wherein the one or more home monitoring sensors include one or more of the following: smart doorbell, smart lock, security camera, motion sensor, home hub, wireless communication device, smart thermostat, alarm, smart garage door opener, smart smoke detector, smart water monitor, smart light, smart socket, smart switch, smart appliance, smart irrigation system, smart speaker, smart display and smart TV.

[0096] Example 12: The method according to any one of the preceding examples, wherein the generation of the family profile is performed by the ML model.

[0097] Example 13: The method according to any one of the preceding examples, wherein the configuration for outputting the family output to the user is performed by the ML model.

[0098] Example 14: The method according to any one of Examples 1 to 11, wherein the generation of the family profile is performed by the one or more processors.

[0099] Example 15: The method according to any one of the preceding examples, wherein the generated one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, household appliance status correlation, weather correlation, pattern correlation, historical correlation, category correlation and energy usage correlation.

[0100] Example 16: The method according to any one of the foregoing examples further includes receiving a user request for relevance to household data. The generation of the one or more relevances is in response to receiving the user request for relevance to the household data.

[0101] Example 17: The method according to any one of the preceding examples, wherein the ML model is a large language model (LLM).

[0102] Example 18: The method described in Example 17, wherein the LLM includes one or more low-rank adaptive (LoRA) layers.

[0103] Example 19: The method described in Example 17, wherein the LLM is trained at least in part using Retrieval Augmentation Generation (RAG).

[0104] Example 20: The method according to any one of the preceding examples, wherein the generated family profile includes multiple family profiles.

[0105] Example 21: The method described in Example 20 further includes ranking the multiple family profiles.

[0106] Example 22: The method described in Example 21, wherein the generation of the family output is further based on the ranking.

[0107] Example 23: The method according to any one of the preceding examples, wherein the generation of the family profile is performed automatically.

[0108] Example 24: The method according to any one of the preceding examples, wherein the generation of the family profile is performed periodically.

[0109] Example 25: The method according to any one of the foregoing examples, wherein the ML model is stored in the memory of a first device, and the one or more processors are housed in a second device, wherein the second device is different from the first device. The first device and the second device communicate wirelessly with each other.

[0110] Example 26: The method according to Example 25, wherein the second device includes a mobile device, and the first device includes one of a cloud computer, a server, a home hub, or a wireless communication device.

[0111] Example 27: The method according to any one of the foregoing examples, wherein the generation of the home profile includes generating one or more event correlation values ​​between two or more of the one or more home monitoring sensors and / or the home data. The one or more event correlation values ​​are based on a determined correlation amount between the two or more devices and / or the home data.

[0112] Example 28: The method described in Example 27 further includes comparing the one or more event relevance values ​​with one or more event thresholds, wherein the generation of the family profile is further based on the comparison.

[0113] Example 29: The method according to any one of Examples 27 and 28, wherein the generation of the family profile is further based on the event relevance value.

[0114] Example 30: The method according to any one of Examples 27-29, wherein the event relevance value is generated by the ML model.

[0115] Example 31: The method according to any one of Examples 27-30, wherein at least one of the two or more devices and / or the family data is a new device and / or new family data on which the ML model has not yet been trained.

[0116] Example 32: The method according to any one of the foregoing examples further includes generating insights from the ML model and from the one or more correlations. The insights include new information derived from the correlations that highlights patterns, behaviors, or data that are not immediately apparent from the household data. The household profile is generated based on these insights.

[0117] Example 33: An electronic device including one or more processors and a memory, the memory including instructions that, when accessed by the one or more processors, cause the one or more processors to perform the method according to any one of Examples 1-32.

[0118] Example 34: A non-transitory computer-readable medium storing instructions that, when accessed by one or more processors, cause the one or more processors to perform the method according to any one of Examples 1-32.

[0119] Example 35: A computer programming product that stores instructions that, when accessed by one or more processors, cause the one or more processors to perform a method according to any one of Examples 1-32.

[0120] in conclusion

[0121] Although techniques for integrating and generalizing information from household event data, including apparatus for doing so, have been described in language specific to features and / or methods, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations of integrating and generalizing information from household event data.

Claims

1. A method for generating a family data summary, the method comprising: The machine learning model receives home data generated by one or more home monitoring sensors. The machine learning model generates one or more correlations based on the family data; A family profile is generated by one or more processors based on the one or more correlations, the family profile comprising a textual summary of a subset of the family data relating to the one or more correlations; as well as The one or more processors generate a family output based on the family profile, the family output being configured to be output to a user.

2. The method of claim 1, wherein the generation of one or more correlations by the machine learning model based on the household data comprises: The machine learning model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on a determined correlation quantity between the two or more members in the family data; The machine learning model compares the one or more correlation values ​​with one or more thresholds; The machine learning model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; as well as Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

3. The method of claim 1, wherein the generation of the family profile is performed by the machine learning model.

4. The method of claim 1, wherein the configuration for outputting the family output to the user is performed by the machine learning model.

5. The method of claim 1, wherein the one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, household appliance status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation.

6. The method of claim 1, wherein the machine learning model is a large language model (LLM).

7. The method of claim 6, wherein the LLM comprises one or more low-rank adaptive LoRA layers.

8. The method of claim 1, further comprising generating insights by the machine learning model based on the one or more correlations, wherein: The insights include new information derived from the one or more correlations, highlighting patterns, behaviors, or data that are not immediately apparent from the household data; and The family profile is generated further based on the insights stated therein.

9. An electronic device comprising: One or more processors; as well as The memory includes instructions that, when executed by the one or more processors, cause the one or more processors to: The machine learning model receives home data generated by one or more home monitoring sensors. The machine learning model generates one or more correlations based on the family data; A family profile is generated based on the one or more correlations, the family profile comprising a textual summary of a subset of the family data relating to the one or more correlations; as well as A family output is generated based on the family profile, and the family output is configured to be output to the user.

10. The electronic device of claim 9, wherein the generation of the one or more correlations by the machine learning model based on the household data comprises: The machine learning model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on a determined correlation quantity between the two or more members in the family data; The machine learning model compares the one or more correlation values ​​with one or more thresholds; The machine learning model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; as well as Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

11. The electronic device of claim 9, wherein the generation of the family profile is performed by the machine learning model.

12. The electronic device of claim 9, wherein the configuration for outputting the home output to the user is performed by the machine learning model.

13. The electronic device of claim 9, wherein the one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, home appliance status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation.

14. The electronic device of claim 9, wherein the machine learning model is a large language model (LLM).

15. The electronic device of claim 14, wherein the LLM comprises one or more low-rank adaptive LoRA layers.

16. The electronic device of claim 9, wherein the instructions further cause the one or more processors to: generate insights by the machine learning model based on the one or more correlations, wherein: The insights include new information derived from the one or more correlations, highlighting patterns, behaviors, or data that are not immediately apparent from the household data; and The family profile is generated further based on the insights stated therein.

17. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: The machine learning model receives home data generated by one or more home monitoring sensors. The machine learning model generates one or more correlations based on the family data; A family profile is generated based on the one or more correlations, the family profile comprising a textual summary of a subset of the family data relating to the one or more correlations; as well as A family output is generated based on the family profile, and the family output is configured to be output to the user.

18. The non-transitory computer-readable medium of claim 17, wherein the generation of the one or more correlations by the machine learning model based on the household data comprises: The machine learning model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on a determined correlation quantity between the two or more members in the family data; The machine learning model compares the one or more correlation values ​​with one or more thresholds; The machine learning model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; as well as Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

19. The non-transitory computer-readable medium of claim 17, wherein the generation of the family profile is performed by the machine learning model.

20. The non-transitory computer-readable medium of claim 17, wherein the configuration for outputting the home output to the user is performed by the machine learning model.

Images