Generating operable insights from smart home event data

Abstract

The invention relates to generating operable insights from smart home event data. This document describes systems and techniques aimed at generating operable insights from smart home event data. Various examples are described herein, including a method that includes generating, by one or more home monitoring sensors, home data. The method further includes receiving, by a machine learning (ML) model, the home data, and generating, by the ML model, one or more correlations based on the home data. The method further includes generating a home operation based on the one or more correlations, and configuring, by the one or more processors, the home operation for output to a user.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 780,826, filed March 31, 2025, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to generating actionable insights from smart home event data. Background Technology

[0004] Home automation, monitoring, and surveillance services present users with a wide variety of data. For example, a doorbell camera might show package deliveries, guest arrivals, neighborhood animals, or passing cars. However, not all data is relevant or useful to the user. In the example of video doorbell data, things like the comings and goings of neighborhood animals and passing cars might be uninteresting or useless to the user. Some combinations of other different data might be relevant, such as weather reports and items left outside (e.g., packages left outside when the weather indicates it might rain). Therefore, while these services can present users with a large amount of data, some of it is useful, some is useless, and some is useful when combined with other data. Additionally, there may be user-specific correlations within the data. Parsing all this home data can be daunting for a user, if not daunting, to the point of being functionally impractical. Summary of the Invention

[0005] Home intelligence agent insights (hereinafter referred to as "insights") bring generative artificial intelligence (GenAI) into the home. Insights are the product of using raw events to discover new information relevant to the user. For example, these insights can come from discovered trends, anomalies in patterns, immediate dangers and rare events, as well as situations reported by the user themselves (e.g., "there's mold in the bathroom").

[0006] Insights can be generated using large language models (LLMs). Agents (e.g., electronic devices) use LLMs to mimic reasoning, interact with external software using tools, use memory to track past conversations, and use reflection to validate their work—all in order to autonomously achieve their goals. Besides improving quality through stepwise reasoning, agents also bring modularity, allowing components to be changed without rebuilding the entire system. For example, when changing command syntax, it is preferable to simply update the syntax generation steps by reusing command ranking and selection logic. Similarly, when adding new artificial intelligence (AI) capabilities, it may be advantageous to directly utilize the new capabilities as tools rather than making architectural changes.

[0007] This document describes systems and technologies designed to generate actionable insights from smart home event data. Various examples are described, including a method that involves generating home data from one or more home monitoring sensors. The method further includes receiving the home data by a machine learning (ML) model, and generating one or more correlations based on the home data by the ML model. The method further includes generating home actions based on the one or more correlations, and configuring the home actions by one or more processors for output to a user.

[0008] Additionally, an electronic device is disclosed, comprising one or more processors and a memory. The memory stores instructions that, when accessed by the one or more processors, cause the one or more processors to perform the described methods. Furthermore, a non-transitory computer-readable medium is disclosed, storing instructions that, when accessed by the one or more processors, cause the one or more processors to perform the described methods. Additionally, a computer programming product is disclosed. This computer programming product stores instructions that, when accessed by the one or more processors, cause the one or more processors to perform the described methods.

[0009] This summary is provided to introduce a simplified concept for generating actionable insights from smart home event data, which is further described in the detailed description below and illustrated in the accompanying drawings. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0010] This document describes in detail one or more aspects of the systems and technologies used to generate actionable insights from smart home event data, with reference to the following figures: Figure 1 An example home environment is shown where actionable insights can be generated from smart home event data; Figure 2 An example user environment is shown in which actionable insights can be generated from smart home event data; Figure 3 An example user device is shown in which actionable insights can be generated from smart home event data; Figure 4 This demonstrates a first example user interface (UI) that enables the generation of actionable insights from smart home event data; Figure 5 This shows a second example UI that enables the generation of actionable insights from smart home event data; Figure 6 An example implementation for generating actionable insights from smart home event data is shown; Figure 7 Examples of machine learning trainers for training machine learning (ML) models are shown, according to one or more aspects of this disclosure; Figure 8 An example LLM trainer for generating actionable insights from smart home event data is shown; Figure 9 Examples of prompting engineering for LLMs according to one or more aspects of the present invention are shown; Figure 10 Examples of low-rank adaptive (LoRA) training for LLMs are shown according to one or more aspects of this disclosure; Figure 11 An example method for generating actionable insights from smart home event data is shown; Figure 12 This demonstrates another example method for generating actionable insights from smart home event data; and Figure 13 Another example method for generating actionable insights from smart home event data is shown.

[0011] Using the same number in different instances can indicate similar features or components. Detailed Implementation

[0012] Overview

[0013] Home agent insights (or simply "insights") are the product of using raw events to discover new information relevant to the user. An insight is not merely a collection of raw events, but rather new information derived from them that highlights patterns, behaviors, and data that are not directly apparent from the collection of raw events alone. An example of an insight is: "By shifting the energy usage to an off-peak period, we saved approximately..." 2.30 today (We saved about $2.30 today by shifting energy use to off-peak hours), "The kitchen light is frequently left on at night between 3 am and 6 am," "Last week it took an average of 7 hours to retrieve packages from the front door," "We've seen an unfamiliar face at the garage door for the past 3 nights at 4 am," "We've detected a degraded performance in the furnace for the past 8 days," "Your battery performance is poor, you might want to consider using the X setting," or "You left your home 30 minutes ago, but the kitchen light is still on and the front door is unlocked."

[0014] This document describes systems and technologies designed to generate actionable insights from smart home event data. Various examples are described, including a method that involves generating home data from one or more home monitoring sensors. The method further includes receiving the home data by a machine learning (ML) model, and generating one or more correlations based on the home data by the ML model. The method further includes generating home actions based on the one or more correlations, and configuring the home actions by one or more processors for output to a user.

[0015] Electronic devices capable of generating actionable insights from smart home event data can reason about and understand virtually an unlimited number of home situations. By using LLM and / or LLM agents to reason about home situations, the device can handle situations it has not yet been explicitly trained on. It can also incorporate world knowledge—for example, through training on the internet—to understand the relationship between problems and device capabilities. For example, if the insight is that a situation is detected as darker than expected in one or more areas of a residence, the proposed action might be to increase the brightness of relevant lights that support such an action. Actionable insights allow for the derivation of dynamic and novel capabilities from, for example, home data and the capabilities of connected devices, without requiring such functionality to be pre-coded or defined. In various ways, the device understands the device's capabilities to leverage a diverse set of action types based on those capabilities. Using examples and formatted contexts that are readable by LLM, the model is able to effectively understand the relevance (e.g., insights) in smart home event data. Additionally, the device can incorporate new capabilities of new devices that have not previously been hard-coded, trained on, or otherwise experienced by the device.

[0016] Insights can be presented to users automatically, upon request, or otherwise. For example, consider a user interaction with a home application that generates actionable insights from smart home event data. The user can query the application (e.g., via a text interface, an LLM agent instantiation) to analyze HVAC optimization potential. The application can use smart home event data (e.g., HVAC usage statistics and dominant weather patterns during timestamps associated with the HVAC usage statistics). When using an LLM agent, the application can generate a response suggesting that the weather is warm enough to allow the user to turn off HVAC heating for 6 hours during normal sleep time (e.g., sleep time derived from home data in the form of motion detection data from within the residence).

[0017] Unprocessed data can be home-related data. For example, unprocessed data could be images from a home security camera or video doorbell. Unprocessed data can also come from smart home monitoring devices, such as smart thermostat data (e.g., HVAC usage statistics, weather-related data, etc.), irrigation or other plumbing-related monitoring data, or door status data (e.g., door lock status, open or closed status, etc.). In some examples, unprocessed data may be related to facial or other recognition methods or to smart appliances (e.g., dishwasher status, refrigerator contents, etc.). Consider a smart lock and smart doorbell on the front door of a residence. Smart home event data could include timestamped door lock status (e.g., unlocked, locked) and timestamped doorbell video events. Example insights could be recognizing that a user has left home (via doorbell video data) and the door lock is unlocked (via door lock data), which could be presented to the user (e.g., on the user's mobile device, on a smart car interface).

[0018] Operating environment

[0019] Figure 1 An example smart home environment 100 is shown in which actionable insights can be generated from smart home event data. Generally, home environment 100 includes a network (e.g., a home area network (HAN)) implemented as part of a residence or other type of building with any number of network-connected devices configured to communicate in a wireless network. For example, environment 100 includes a smart fan 102, a smart socket 104, a smart camera 106, a smart outlet 108, a smart home controller 110, a smart sensor 112, a smart alarm clock 114, a smart door lock system 116, a smart home hub device 118, a border router 120, a user device 122, a smart thermostat 124, an access point 126 (e.g., a smart networked device), a smart refrigerator 128, and a heating, ventilation, and air conditioning (HVAC) system 130. Any number of these network-connected devices can be implemented to wirelessly interconnect to communicate and interact with each other wirelessly. Network-connected devices can be modular, intelligent, multi-sensing wireless devices that can be seamlessly integrated with each other and / or with a central server or system to provide any wide variety of useful implementations. Network-connected devices can also be configured to communicate via a network, which may include a wireless mesh network, a Wi-Fi™ network, or both.

[0020] As a non-limiting example, network-connected devices may further include: hazard detectors (e.g., for smoke and / or carbon monoxide), cameras (e.g., indoor and outdoor), lighting units (e.g., indoor and outdoor), sensors and detectors (e.g., ambient light detectors, occupancy sensors, doorbells and door lock systems), connected electrical appliances and / or controlled systems (e.g., stoves, ovens, washing machines, dryers, air conditioners, pool heaters, irrigation systems, and security systems), electronic and computing devices (e.g., televisions, entertainment systems, computers, speakers, intercom systems, garage door openers, ceiling fans, and control panels), and any other type of network-connected device implemented inside and / or outside a building (e.g., in home environment 100).

[0021] Consider a home environment 100 including a delivery person 132. A smart camera 106 or smart sensor 112 can identify the arrival of the delivery person 132 via facial recognition or proximity sensors. Smart home event data may include the identification of the delivery person 132, the lock status of the smart door lock system 116, and occupancy data indicating the presence of the first child 134 and the second child 136 within the home environment 100. Example insights (e.g., correlations) may be presented to the user (e.g., an absent parent) indicating that the door is unlocked and the first child 134 and the second child 136 are home alone. A potential action derived from the insight could be asking the user if they want to lock the door. In some examples, actionable insights are presented on a user device 122.

[0022] The home environment 100 may also include a dog 138. Dog 138 may have been left outside by the first child 134, and smart home event data may include the dog 138's identification, the outside temperature from an outdoor thermometer, and the duration the dog 138 has been outside based on a timestamp from a smart camera 106. An example insight could indicate to the user that the dog 138 has been outside in a high temperature of 93℉ for 35 minutes. A potential action derived from this insight could be asking the user if they want to open the garage door or unlock the pet door to let the dog 138 in.

[0023] The home environment 100 may also include a smart alarm clock 114. The smart alarm clock 114 may be located in the user's bedroom, and smart home event data may include sleep schedules derived from smart sensors 112, the user's calendar data, ambient light levels from a light detector facing the window, and tracking data from wearable devices (e.g., smartwatches). Example insights could indicate to the user that their average sleep time over the past week has dropped below six hours due to frequent late-night activity events. Potential actions derived from this insight could include asking the user if they want to delay the alarm by 30 minutes and enable a "wind-down" lighting mode at night to promote better sleep.

[0024] The home environment 100 may also include a smart refrigerator 128. The smart refrigerator 128 can track food inventory via a weight sensor, and smart home event data can include door opening frequency, internal temperature levels, and product expiration data. Example insight could indicate to a user that milk is nearing its expiration date, suggesting it is likely to spoil soon. A potential action derived from this insight could be asking the user if they want to add the milk to their grocery list. In another example, a potential action derived from this insight could be asking the user if they want to set a reminder to use the milk within the next 24 hours.

[0025] Home environment 100 may also include a smart socket 104. The smart socket 104 can control power to various appliances (e.g., HVAC 130, smart refrigerator 128), and smart home event data can include socket activation history, estimated energy usage, and daytime patterns. Example insight could indicate to a user that a space heater plugged into the smart socket 104 has been on for five consecutive hours without detected movement in the room. A potential action derived from this insight could be asking the user if they want to turn off the smart socket 104. In another example, an actionable insight could ask the user if they want to set an automatic shut-off rule for the space heater or similar device when the room is unoccupied.

[0026] Figure 2 An example user environment 200 is illustrated in which actionable insights can be generated from smart home event data. In a first example user environment 200-1, user device 202 may be a mobile phone held by user 204. In a second example user environment 200-2, user device 202 may be a laptop computer used by user 204. In a third example user environment 200-3, user device 202 may be a wired earphone in user 204's ear. User device 202 may have an insight service located on the memory of user device 202.

[0027] The Insights service can collect data from data sources (residential map devices, current device status, historical events, weather, etc.) and generate insights. In some examples, no relevant insights may be generated, and the process ends there. In the example of generating insights, a presentation surface can be selected to deliver the insight to user 204 (e.g., user device 202). Potential actions for the insight can be selected by contacting the Action Selection Service. After the insight is generated, the insight, presentation surface, and / or action can be persisted (e.g., stored in a database) for bookkeeping and / or later displayed via a pull request. For example, if the insight needs to be displayed to user 204, a delivery date can be scheduled via a scheduler, or the insight can be pushed to user 204 immediately. Instant insights can directly invoke the presentation router to push the insight to the selected presentation surface. Insights can be published to a home event channel.

[0028] For example, user 204 accesses a page on user device 202 that requires the current insight, such as by opening a thermostat device controller displaying insights related to the general energy usage of that particular device. The insight selector can determine whether any previously generated insights meet the request criteria and retrieve relevant insights from the stored previously generated insights. The insight selector may include existing insights that have been selected to join the reactive display. The insight selector determines whether an insight can be generated for the request and invites an insight service that can act as an insight generator. The insight service can collect all necessary data from external data sources and generate insights. For example, the insight service may collect data from the user's smart home (e.g., Figure 1 The service collects sensor data (e.g., room temperature, refrigerator inventory) and external data (e.g., weather forecasts, grocery delivery services) from the environment (e.g., 100). Based on this input, the insight service generates insights about the user's consumption of fresh produce during warmer months. Alternatively, the insight service may not generate relevant insights. The process can end, and the insight service can send the user a "no insights found" message. For example, the insight service collects appliance usage data, occupancy patterns, and local air quality data to assess the user's smart home. After analyzing the input, the insight service can determine that no relevant insights exist to send to the user and can stop the data collection process.

[0029] When an insight is generated, the insight service can determine the appropriate action. Alternatively, an insight selector can recommend actions based on the insight. After an insight is generated, in some examples, a check may occur on any modifications to the text of the insight to make the text suitable for the request. This check may be merely superficial. Newly generated insights can be stored and / or updated as the information is used for record-keeping purposes.

[0030] Example device

[0031] Figure 3 A user device 202 is illustrated in example environment 300, in which actionable insights can be generated from smart home event data. User device 202 is shown as having various non-limiting example devices, including a desktop computer 202-1, a tablet computer 202-2, a laptop computer 202-3, a television 202-4, a smartwatch 202-5, smart glasses 202-6, a gaming system 202-7, a smart appliance 202-8, a vehicle 202-9, earphones 202-10 (e.g., true wireless earphones, wired earphones), a hearing aid 202-11, a virtual reality (VR) headset 202-12, and an augmented reality (AR) headset 202-13. Other devices may also be used (e.g., home service devices, smart speakers, smart water monitors, baby monitors, Wi-Fi™ routers, drones, touchpads, drawing tablets, netbooks, e-readers, home automation and control systems, wall displays, or other home appliances). Note that the user device 202 may be wearable, non-wearable but portable, or relatively immobile (e.g., desktop computer and appliance).

[0032] User device 202 may include one or more processors 302 and memory 304 (e.g., non-transitory computer-readable medium). In some examples, memory 304 is part of a computer programming product. The one or more processors 302 may include any suitable single-core or multi-core processor (e.g., application processor (AP), digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU)). Memory 304 may include memory media and / or non-transitory storage media. An operating system (not shown) embodied as computer-readable instructions on memory 304 may be executed by the one or more processors 302.

[0033] User device 202 may additionally include user interface 306. In some examples, user interface 306 is a display on user device 202 in which actionable insights (e.g., notifications) from smart home event data can be presented. User device 202 may also include wireless communication module 308 for transmitting data over a wireless network. For example, wireless communication module 308 transmits data via wireless local area network (WLAN), Bluetooth™, cellular networks (e.g., 4G, 5G), satellite communication networks, mesh networks, the Internet, etc.

[0034] Figure 4 This illustrates a first example UI 400 in which actionable insights can be generated from smart home event data. User device 202 can then output these insights to the user (e.g., ...). Figure 2User 204) is presented with suggestion box 402. Figure 4 As shown, suggestion box 402 includes several insights from smart home event data. Suggestion box 402 can be an inbox in an application (e.g., the Google Home App (GHA)). Insights can be requested by the user and / or automatically rendered by a large language model (LLM). Alternatively, insights can be user-generated templates populated with smart home event data. Furthermore, insights can be generated solely from an LLM.

[0035] In many ways, insights are the product of using raw events (e.g., smart home event data) to discover new information relevant to users. Insights are not simply a collection of raw events, but include new information derived from them that highlights patterns, behaviors, and data (e.g., correlations) that are not directly apparent from the collection of raw events alone. Examples of insights include “by shifting the energy usage to an off-peak period, wesaved approximately "We saved about $2.30 today by shifting energy use to off-peak hours," "The kitchen light is frequently left on at night between 3 a.m. and 6 a.m.," "Last week it took an average of 7 hours to retrieve packages from the front door," and "We've seen an unfamiliar face at the garage door for the past 3 nights at 4 a.m.

[0036] Insights can be generated based on a variety of triggers, including but not limited to real-time events, periodic events, app requests, or user queries. The smart home event data used to generate insights can evolve as more types of insights are added. In some examples, to see long-term patterns in the data, backsliding windows (e.g., 1 week, 60 days) can be used for historical home event data. The insight service can generate insights based on triggers. A first example trigger is a periodic trigger, where periodic tasks (e.g., daily) trigger the generation of insights and / or insight actions. A second example trigger is an event trigger, where real-time events (e.g., "lights on," "building enters / leaves mode") trigger the generation of insights and / or insight actions. A third example trigger is an app request trigger, where a user entering a specific page triggers the generation of insights and / or insight actions. A fourth example trigger is a user query trigger, where a specific user query triggers the generation of insights and / or insight actions.

[0037] Insights in discovering information about a user's smart home environment (e.g., Figure 1 New information (e.g., relevance) about the home environment (100) can be useful and can have associated actions. These actions can be requested by the user and / or automated via user device (202). Examples of insights and their potential actions include “we've detected a degraded performance in the furnace for the past eight days; contact an HVAC technician to help diagnose if there is an issue with your furnace” and “by shifting the energy usage to an off-peak period, we saved approximately 2.30 today…set up a routine to further increase savings for those appliances that can be turned off.

[0038] Users can obtain insights about their home environment through various means. The first means is proactive (“push”), where insights and / or insight actions are presented to the user without a specific user request. This proactive approach can include pushing insights to the user in their inbox (e.g., suggestion box 402) or as a notification, or loading insights whenever the user selects a specific page. Additionally, proactive approaches can be scheduled by the insights service (e.g., in the future). The second means is reactive (“pull”), where the user requests insights from the user device 202. For example, a user could ask, “How long is my home typically in away mode?” This reactive approach can include both proactive app request triggers and reactive user query triggers. Pull requests (e.g., for app requests, user queries) can use the interface of an LLM agent. This allows requests to leverage the LLM agent infrastructure for traffic control, logging, or poison checks. The insights service can implement remote procedure calls (RPCs) to handle queries in order to receive requests. For example, reactive requests or reactive responses can be passed as additional parameters to the reactive request. User queries can be passed as part of the query processing protocol.

[0039] The Insight Scheduler can be used to schedule insights for display. The Insight Scheduler provides configurable RPCs for scheduling insights. When scheduling an insight for display, the Insight Scheduler can invoke the Insight RPC on the Insight Service. For example, an insight received by the Insight RPC can be routed to all specified display channels, provided that rate limiting rules are met. If an insight cannot be displayed due to rate limiting, it can be dropped or rescheduled on the Insight Scheduler.

[0040] Potential actions for an insight can be selected by passing the insight text to an action generator. After an insight is generated, the insight, its presentation channel, and / or action can be persisted (e.g., stored in a database) for categorization and / or later presented via a pull request, depending on some examples. When an insight is presented to a user via push notifications (e.g., notifications or inbox), a delivery date can exist, scheduled by a scheduler. When the time for delivery arrives, the insight scheduler can contact the insight service to push the insight to the correct presentation channel. The insight database can contain information for tracking the generated insights or the times they are scheduled. This information can be used to create dashboards and visualize the current status and metrics of the insights, as well as to prevent duplicate insights from being published to users using stored insights.

[0041] Insights can be displayed to users through multiple display channels (e.g., GHA inbox, push notifications, helpful interruptions, reminders on the client's expected display channel, etc.). Additionally, some examples allow each insight to be annotated with a domain that tells you what the insight is about. For example, the insight "we detected you had a poor sleep score last night…your sleep score can improve if you lower your sleep temperature by 1℉" belongs to the energy or health domain.

[0042] In some examples, the Insights Service does not generate insights, but it can attach actions to insights upon request and determine the display used for those insights. Other insights within the service can be displayed in the same way by displaying them to the Home app via a pull mechanism. In some examples, this mechanism can be added to allow insights already created in other services (e.g., Wi-Fi™, thermostat) to utilize the Insights Service without needing to migrate the generation of those insights to this service.

[0043] The generated insights can be compared with previously generated insights already stored in the database. If these new insights are copies and have already been displayed, the new insights can be stored and not displayed. The frequency of displaying copies can be adjusted based on the type of insight. For example, the insight "You just left home, and the door is unlocked" might be displayed multiple times depending on the situation, but "You leave the lights on at night" might only be displayed once or once over a long period (e.g., a month, a quarter). Cooldown periods can be insight-specific.

[0044] Figure 5 A second example UI500 is shown, which can generate actionable insights from smart home event data. User device 202 can then output these insights to the user (e.g., Figure 2 User 204) is presented with suggestion box 402. Figure 5As shown, suggestion box 402 includes several insights 502. A first insight 502-1 can ask the user if they want to set an "away" mode. User device 202 can collect smart home event data from the smart home indicating that the user has left the smart home for an extended period of time. Based on the correlation of the smart home event data, user device 202 can send the first insight 502-1 as a notification to the user, and can also display it on the user device 202's user interface (e.g., ...). Figure 3 The first insight 502-1 is presented on the user interface 306. The second insight 502-2 can remind the user that the living room lights are frequently turned on manually from 7 p.m. to 10 p.m. Based on the smart home event data already collected by the user device 202, the second insight 502-2 can ask the user if they want to set a routine to automatically turn the lights on and / or off at specific times of the day.

[0045] In each aspect, the third insight 502-3 in suggestion box 402 reminds the user about the humidity level for the day. Based on smart home event data (e.g., humidity levels), the third insight 502-3 can ask the user if they want to set a routine to turn on their dehumidifier between 3 AM and 6 AM. User device 202 can use the correlation between the user's sleep schedule and the user's normal use of their humidifier to determine when it would be most useful to turn on the dehumidifier. In each aspect, the fourth insight 502-4 informs the user that their smart ceiling fan in their bedroom has been offline for three days. The fourth insight 502-4 can provide the user with actions that allow them to explore options for resolving the smart ceiling fan issue.

[0046] The logic for generating Insight 502 can be applied to all components in the design (e.g., Figure 3This is shared between one or more processors 302 and memory 304. This allows insights generated from any trigger to be displayed from storage or regenerated in response to pull requests (e.g., triggers, app requests, user queries). Event-triggered insights, typically generated based on household bedtime estimates (e.g., “Looks like you're headed to bed and the front door is still unlocked. Do you want to…”), can be generated reactively based on user queries. For example, a user query “I'm headed to bed, anything I should know?” can generate an insight and the insight action “The front door is still unlocked. Do you want to lock the front door?”. LLM can be used to modify the text of insights to suit different approaches (e.g., proactive, reactive). Previously generated insights from any trigger can be used to respond to app-based pull requests (e.g., requests for controller-related insights). For example, insights previously generated in response to a user request (e.g., the insight "Your HVAC ran for 5 hours while you were on vacation last week" generated from a user query "How often did my HVAC run while I was on vacation last week") can be re-emerged in the app request (e.g., suggestion box 402 now shows the insight "Your HVAC ran for 5 hours while you were on vacation last week").

[0047] Figure 6 An example implementation 600 for generating actionable insights from smart home data is shown. Home environment 100 (about...) Figure 1The described method interacts with the ML model 602 by sending home data 604 (e.g., smart home event data) to the machine learning (ML) model 602 (e.g., LLM). Based on the home data 604, the ML model 602 generates one or more correlations 606 and outputs them to the user device 202 (see reference). Figure 2 and Figure 3 (As described). In some examples, one or more correlations 606 include one or more of the following: time correlation, object detection correlation, event correlation, household device status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation. For example, one or more correlations 606 are between at least two or more different devices, and one or more correlations 606 are time correlations (e.g., two seconds, one microsecond). User device 202 may trigger the generation of one or more correlations 606.

[0048] User device 202 can configure home operation 608 based on one or more correlations 606, and can output home operation 608 to user 204 (see reference). Figure 2 (As described). In various respects, one or more processors 302 of user device 202 (regarding) Figure 3 The described configuration of home operation 608 is for output to user 204. User 204 may send user input 610 to user device 202. User input 610 may be a request to execute home operation 608. Furthermore, user device 202 may send home operation execution 612 to home environment 100. User device 202 may also send home operation request 614 to ML model 602, which may send paused home operation execution 616 to home environment 100.

[0049] Machine Learning

[0050] As discussed in this disclosure, an ML model refers to a computer model that has been trained using one or more machine learning techniques. Generally, this training can be performed by providing training inputs to one or more trained models, which in turn can provide outputs. The outputs can be in the form of predictions, confidence scores, or other probability-based metrics.

[0051] Figure 7An example machine learning trainer 700 for training an ML model is illustrated according to one or more aspects of this disclosure. An ML model generator 702 may include training elements in the form of input 704, a training model 706, and output 708, and may be used to generate an ML model 710. Input 704 may be training data. For example, training data 704 may include smart device data, such as the on / off state of an appliance, thermostat readings, motion sensor data, or the brightness of a smart light. Training data 704 may also include user behavior and preferences, such as daily thermostat schedules or manual overclocking of lighting automation. In other examples, training data 704 includes environmental data, contextual data, user history data, energy usage data, or event log data. Training data 704 may be processed by a training model 706. Examples of training models 706 include multilayer perceptron (MLP) models, convolutional neural networks (CNNs), long short-term memory (LSTM) algorithms, generative adversarial networks (GANs), K-means clustering, Gaussian mixture models (GMMs), or any other variety of machine learning training techniques known to those skilled in the art. The training model 706 may include a single type of model or multiple types, as well as various combinations of types, including single type and combinations of multiple types.

[0052] The training model 706 can use supervised learning methods, where the training data 704 can be labeled, and the outputs 708 can be graded based on their fidelity to the "true" output. The training model 706 can also use unsupervised learning methods, where the training data 704 may be unlabeled, and the training model 706 can classify relevance without reference to the "true" values. The training model 706 can combine supervised and unsupervised techniques. The input 704 can come from the training data used to generate the ML model 710. Once the training of the ML model generator 702 is complete, the ML model 710 can be generated. The ML model 710 can be trained on the same device where it is stored or on at least one other device.

[0053] ML model 710 can be trained based on household data (e.g., Figure 6 Family data (604) is used to generate one or more correlations (e.g., family data 604) to generate one or more correlations. Figure 6One or more correlations (606). To generate one or more correlations, the ML model 710 can generate one or more correlation values ​​between two or more members of the household data (e.g., network-connected devices). One or more correlation values ​​can be based on the determined amount of correlation between the two or more members of the household data. As an example, consider a smart motion sensor and a smart light. Since the smart light automatically turns on when the smart motion sensor detects motion, the smart motion sensor can have a high amount of correlation with the smart light (e.g., a similar correlation value). In another example, consider a smart sprinkler system and a smart TV. The smart sprinkler system and the smart TV can have low correlation values ​​relative to each other because the smart sprinkler system waters the lawn, while the smart TV handles media.

[0054] ML model 710 can further compare one or more relevance values ​​with one or more thresholds. The one or more thresholds can be based on one or more of the following: data time, data type, data category, data history, or device type. ML model 710 can also determine that at least two members of the household data exceed one or more thresholds, and can thereby generate a data subset that includes those at least two members of the household data. Household operations can then be based on the at least two members identified in the data subset, thereby narrowing down the scope of the data. Figure 1 The amount of incoming data generated by all network-connected devices 102-130. In fact, by pairing at least two members of the home environment 100 via this correlation-based approach, the proposed scheme limits latency and resource impact to provide home operation output to the user. As an example, consider... Figure 1 100% of family environments Figure 1 Dog 138. Dog 138 may have been left outside by first child 134. ML model 710 can generate correlation values ​​between motion sensors in the house, an outdoor camera, and an outdoor thermometer. Furthermore, ML model 710 can compare these correlation values ​​to a threshold for the time dog 138 has been outside (e.g., 15 minutes or less) and a historical threshold for when first child 134 handled dog 138 (e.g., first child 134 left dog 138 outside for 5 minutes). Based on the fact that the outdoor camera detected dog 138 being outside for 35 minutes, the current outdoor temperature is 93℉, and the motion sensors in the house indicate the presence of first child 134, ML model 710 can determine that the correlation values ​​of all devices exceed the threshold. Based on this indication, ML model 710 can generate a subset of data (e.g., insights) to be configured for output to the user. Example insights (e.g., Figure 5 Insight 502 can instruct the user that dog 138 has been outside in a high temperature of 93℉ for 35 minutes. Potential actions derived from this insight (e.g., Figure 6The home action 608 could be asking the user if they want to open the garage door or unlock the pet door to let the dog 138 in. This action can be further generated by the ML model 710 based on a comparison of additional relevance values ​​with additional thresholds (e.g., comparing the garage door opener relevance value with the rest of the devices).

[0055] In some examples, ML model 710 can compare correlation values ​​between devices that do not exceed a threshold. Consider the example above, but ML model 710 generates correlation values ​​between an indoor motion sensor, an outdoor camera, an outdoor thermometer, and a refrigerator door sensor. The refrigerator door sensor could indicate that the refrigerator door was opened approximately at the same time the outdoor camera identified Dog138. However, the refrigerator door sensor does not exceed any threshold and is loosely correlated or not correlated at all with other devices, so ML model 710 does not include the door opening event in the subset of data.

[0056] Based on some examples, the generation of relevance values ​​and / or the comparison of relevance values ​​with thresholds can allow the ML model 710 to scale for new devices (such as devices on which the ML model 710 has not been trained) (e.g., the ability to use new devices not used in input 704). As an example, consider again a home environment 100, but with a new device: a smart awning. The ML model 710 may never have been trained on the smart awning, but in this example, the smart awning may have a high relevance value to an outdoor camera (e.g., a relevance score based on device proximity, grouping from the user, or automatic grouping, etc.). A potential action could be to ask the user whether they want to deploy the smart awning based on the relevance value of the smart awning and / or a comparison of the relevance value of the smart awning with one or more thresholds. In this example, the action of deploying the smart awning could also be something that the ML model 710 has not explicitly trained on, but potential actions could include this ability of the smart awning based on relevance values ​​and / or threshold comparisons.

[0057] Based on some examples, the generation of an operation is based on one or more of the following: correlation values ​​between devices, comparison of correlation values ​​with a threshold, operation correlation values, and operation thresholds. Operation correlation values ​​can be determined based on a correlation quantity between two or more of the following: available operations of network-connected devices, categories of available operations, categories of network-connected devices, data of network-connected devices, historical data, etc. Operation thresholds can be used as comparison values ​​for operation correlation values, categories of available operations, etc. For example, consider the scenario of a dog being left outside, as outlined earlier. The proximity of the dog to the garage door, coupled with the available operation of the garage door opening, can be assigned a high operation correlation value and / or exceed the operation threshold. In contrast, Figure 1The refrigerator 128 may have very low correlation values, operational correlation values, etc., and the correlation value may not exceed the operational threshold. It should be noted that the use of such values ​​(operational correlation values, operational thresholds, etc.) allows for robust scalability with respect to new devices and / or new device operations, and the ML model 710 has not yet been trained on some of the new devices and / or new device operations.

[0058] The ML model 710 can be updated using additional training after it has been initially trained. In all respects, the ML model 710 can have the same structure before and after training, but can have one or more different values, such as the initial and final values ​​of the weights and biases. The ML model 710 can start with a different architecture before training, and in this way, the generated ML model 710 can have an architecture that is the product of training performed in the ML model generator 702.

[0059] As an example, training an ML model can be accomplished by combining it with a Long Short-Term Memory (LSTM) algorithm. An LSTM algorithm is a type of recurrent neural network (RNN) that processes data in a time-indexed manner. LSTM can solve the so-called "vanishing gradient problem," in which the gradient used in the fitting may tend to zero and therefore may not produce useful fitting parameters (e.g., weights and biases) for a given model. During the training phase, LSTM allows persistent gradients to be used for fitting when the gradient might otherwise become zero (e.g., in a traditional RNN). Those skilled in the art will understand that other configurations using LSTM can also be used equivalently, and the examples given herein are intended to be illustrative rather than limiting.

[0060] Large Language Models (LLM)

[0061] Typically, LLMs are a type of artificial intelligence (AI). LLMs are trained on large amounts of data to provide foundational capabilities that are often used and reused by fine-tuning for specific applications and tasks. In contrast, other software applications are typically built and trained on specific data for each use case. In this way, LLMs are considered a type of foundational model.

[0062] Some LLMs use ML models that can parse language and provide context-aware outputs (e.g., to mimic human responses). This mimicry of human responses is often directed at prompts (e.g., from a user asking a question). For example, an LLM might use the prompt "ask how to get to the train station in French" to provide a translation service (e.g., responding to an English prompt in French).

[0063] consider Figure 8 , Figure 8 A trainer 800 is shown, through which an LLM is trained to generate actionable insights from smart home event data. The trainer 800 receives training data as training input, such as input 802. This training data can be of many different types, such as home automation and / or monitoring data. Figure 8 In the example shown, training input 802 is a phrase, but it can alternatively be a word, a long text paragraph (e.g., a book, article, or webpage), or any other data containing comprehensible text. In some examples, the text comes from a screen or image capture. In a process known as “lexicalization,” trainer 800 breaks down training input 802 into lexical units labeled as lexical units 802-1, 802-2, 802-3, and 802-4. Here, training input 802 has a missing next word, labeled as blank 802-5. The goal of trainer 800 is to predict blank 802-5.

[0064] The trainer 800 encodes tokens (802-1, 802-2, etc.) into the input tensor through a mapping process. In 804, for example, the word "It" is mapped to the input tensor in 802-1. The first component of 804, 804-1, and the word “'s”, 802-2, are mapped to the input tensor. The second component of 804, 804-2, the word “character”, 802-3, is mapped to the input tensor. The third component of 804 is 804-3, and the word “ize” 802-4 is mapped to the input tensor. The fourth component of 804 is 804-4. Although the lexical units “It” 804-1 and “'s” 804-2 are shown as two parts of the word “It’s”, other mapping schemes exist (e.g., mappings based on discrete words or phonemes). The ML model or ML component training 800 can perform training input 802 to the input tensor. Lexicalization and / or mapping of 804 (e.g., feature extraction convolutional neural networks (CNN)). Lexicalized training input 802 to input tensor The mapping of 804 may involve a lookup table that maps each possible lexical unit (e.g., 802-1, 802-2, etc.) to a known tensor object in the language space of the training data.

[0065] The transformer 806 will input tensors 804 is used as input, and the purpose is to pass the input tensor 804 transformation into the transformed tensor 808 is used to predict the blank 802-5. This transformation process is mathematically represented as follows:

[0066] In Equation 1 T This represents transformer 806. The transformed tensor. 808 includes components 808-1, 808-2, 808-3, 808-4, and 808-5. Component 808-1 is the transformation of component 804-1 through transformer 806 (similar cases apply to components 808-2 / 804-2, 808-3 / 804-3, and 808-4 / 804-4). Component 808-5 corresponds to the blank 802-5, and therefore component 808-5 is a prediction of the blank 804-5. In addition to the contextualization of components 804-1 to 804-4, the final transformed tensor is derived as part of the transformation process. The component of 808 is 808-5.

[0067] Input (e.g., input tensor) Training input 804 and / or training input 802 typically includes multiple lexical units. Training input 802 includes lexical units 802-1 through 802-4. Trainer 800 transforms a single training input (e.g., training input 802) into multiple training inputs. For example, by removing lexical unit 802-4, and shifting the blank 802-5 to the left when training input 802 asks trainer 800 to predict lexical unit 802-4, a new training input is created from the original training input 802. Since the value of lexical unit 802-4 is known in this example, the new input is a labeled input, which allows this new input to be used by supervised ML training algorithms (unsupervised ML training algorithms can also use such an input). In this way, a single text containing multiple lexical units (e.g., a book, a research paper) is used as multiple training inputs for trainer 800.

[0068] In some cases, it is desirable to guide the output of an LLM without fine-tuning it. Programmers may want to add functionality from an already trained LLM, including all the latest features, to their application via Application Programming Interface (API) calls, without requiring additional training or maintenance. In such cases, the only way to guide the LLM's output is through input hints. Customizing input hints to achieve the desired output is known as hint engineering.

[0069] consider Figure 9 , Figure 9The diagram illustrates prompt 900. Input prompt A is an attempt to obtain the desired result B using LLM. LLM includes different computational pathways (“paths”) based on the form of input prompt A. At least conceptually, there may exist an ideal path 902 that obtains the desired result B from input prompt A in the most efficient way possible. Other paths also exist, such as an erroneous path 904 that yields an undesirable result D, an intermediate path 906 that yields an intermediate result C, another intermediate path 908 that yields the desired result B from the intermediate result C, and an inefficient path 910 that yields the desired result B from input prompt A. Many paths may exist, limited only by the scope of LLM and the scope of input prompt A. The ideal path 902 can be mathematically represented as follows:

[0070] In Equation 2 S This represents the ideal path 902, and This indicates input prompt A, which is generated by the language space. and used for language space Path propagation function Characterization. It may be difficult to distinguish the various paths 904-910 from the ideal path 902. To achieve or reasonably approximate the ideal path 902, input prompts should be provided. Make the following changes:

[0071] In Equation 3 This represents a path, which is similar to the ideal path 902. The deviation is a function ,in From parameters Characterization. During prompting process 900, various iterations of entering prompt A are fed into the LLM to attempt to find an acceptable path. As an example, one form of entering prompt A fails to reach the desired result B (e.g., incorrect path 904) and is therefore discarded. Another form of entering prompt A gives an intermediate path 906 to reach intermediate result C, and a subsequent prompt gives an intermediate path 908 from intermediate result C to the desired result B. While this prompt reaches the desired result B, it takes multiple steps, which is less efficient than a single direct prompt. Another form of entering prompt A gives an inefficient path 910 that reaches the desired result B. Yet another form of entering prompt A gives an ideal path 902 (or an acceptable approximation) to the desired result B. The characterization of the variations in the form of entering prompt A can be mathematically expressed as follows:

[0072] Equation 4 shows the path deviation With parameters The changes, among which This is the maximum acceptable value. Ideally, Consider an example where a user requests an application agent to make their house more energy efficient. The application agent has access to an LLM and utilizes a prompting engineer 900 that has been trained with various forms of input prompts A. Consider a prompt A in the form of “make my house more energy efficient”. Suppose the LLM responds by turning off all smart devices, including essential systems such as the refrigerator and HVAC. The user may not have intended to completely disable critical appliances, and doing so could lead to discomfort or even food spoilage. The LLM output may not be consistent with the user's intent when entering prompt A. This example output from the LLM is represented by an unexpected result D, which is not an acceptable path for the prompting engineer 900.

[0073] Consider another example of the above form of prompt A, where there is a subsequent step in the training of the applied agent that causes the LLM to dim the lights in unoccupied areas, adjust the thermostat based on historical weekday patterns, and turn off idle entertainment systems, through an automatically entered prompt in the form of "adjust my energy usage based on my usual weekdayschedule and turn off any unused devices in unoccupied rooms." With all devices initially completely turned off, as represented by intermediate result C, the resulting adjustment of usage patterns and occupancy-based optimization is represented by the desired result B. This process is represented by the additive path 906+908. While the desired result B is achieved, deviations occur... It may still be higher than the maximum acceptable value. This may be due to computational costs, inability to input multiple prompts, or other limitations.

[0074] Consider an example of input prompt A instantiated within a smart home control interface for an application agent, where prompt A has the form "turn off the lights". Further consider the application agent resolving prompt A through prompt engineering to have an updated form: "turn off the lights in rooms that are unoccupied, based on motion sensor data from the last fifteen minutes" (the updated prompt may be based on knowledge of unsuccessful forms of prompt A, the form of prompt A used in prompt engineering during the application agent's training phase, etc.). The LLM could return executable code for use in the smart home that disables the lights in five rooms regardless of occupancy, thus inconveniencing the user if a room in use is mistakenly powered off. Bias. It may exceed the maximum acceptable value again. This is illustrated by inefficient path 910. In a similar example, suppose input prompt A is reformatted by the applied agent to the form "using recent motion data and the time of day, turn off lights only in unoccupied rooms and keep lights on in commonly used evening spaces like the kitchen and living room." The result could be a more subtle response consistent with the user's habits, thus reducing energy consumption while maintaining convenience, which leads to bias. The value falls within the maximum acceptable value At or below, this is represented by the ideal path 902.

[0075] Each of the forms of input prompt A in the previous examples can be used to train the application agent to parse input prompt A in order to reformat the input prompt in a manner consistent with the intent of input prompt A. Prompt engineering 900 utilizes the failure paths from all forms of input prompt A to obtain an acceptable, efficient, and concise prompt that produces the desired result B. In some examples, the parsing of input prompt A to determine the intent can be performed by LLM. LLM can also be used in prompt engineering 900 to generate additional forms of input prompt A, find subsequent prompts, classify intents, or perform other aspects of prompt engineering 900.

[0076] In some examples, cue engineering can be used as part of the overall LLM manipulation scheme. Consider Retrieval Augmentation Generation (RAG). In examples using RAG, the LLM is not fine-tuned; instead, cue engineering is used to incorporate new data (e.g., data not used to train the LLM). While RAG can be combined with other mechanisms (e.g., code for information retrieval, synthetic plotting, etc.), cue engineering is an integral part of RAG. Throughout this disclosure, the idea of ​​cue engineering is intended to cover LLM manipulation methods that do not fine-tune the LLM.

[0077] Figure 10 An example low-rank adaptive (LoRA) training 1000 for LLM 1002 is shown. LoRA training 1000 can be used to fine-tune LLM 1002. One advantage of LoRA training 1000 is that not all parameters 1004 of the LLM 1002 are tuned, resulting in training with significantly lower computational cost compared to fine-tuning all parameters 1004 of an existing LLM 1002. In some examples, LoRAs can be generated to customize the LLM 1002 for home automation and monitoring tasks, or multiple LoRAs can be generated, each for a specific home automation and / or monitoring application.

[0078] LoRA training 1000 uses a trained ML model 1006. The trained ML model 1006 has LoRA weights 1008, which modify only some parameters in the parameters 1004 of the LLM 1002 (indicated by dashed lines 1010). The LLM 1002 can be represented as (e.g., by...) Figure 8 The pre-trained weight matrix of the trainer (800). ,in and Representation matrix The dimension. In Full Fine-Tuning (FT) training, by modifying the matrix To modify the matrix This modified matrix also has dimensions. The matrix. In the case of a very large LLM 1002 (e.g., hundreds of billions of parameters), modifying the matrix... It is also very large, and therefore intensive in terms of both computational training resources and storage resources. The problem becomes complex in examples seeking to train multiple LLMs using the Fourier Transform (FT).

[0079] LoRA training with 1000 iterations can significantly reduce the need to modify the matrix. The cost. Consider the following equation:

[0080] In equation 5, It has dimensions The matrix, and It has dimensions The matrix. In the small limit, Thus, respectively and It is a contravariant vector and a covariant vector of rank 1. Stored and used as a dimension Compared to matrices, this can significantly reduce the dimensionality of the FT and thus reduce computational costs. Consider the following... The LLM 1002 is represented, where Given a total of 198,025,000,000 parameters, train 1000 using LoRA. At the low end, It can be formed by two vectors with only 445,000 dimensions. and This results in a reduction in the following dimensions:

[0081] Equation 6 shows the modified matrix Size is Examples of 0.00022472% of the size. In some examples, the result is that the LLM 1002 can be FT trained using a relatively small set of parameters (e.g., LoRA weights 1008). In this way, an FT LLM based on LLM 1002 can be created. If LLM 1002 is a general LLM for home management and the general LLM for home management is at least in part the product of LoRA training 1000, then multiple specialized LLMs based on different aspects of home management can be easily stored on a user device (e.g., user device 202). Specialized LLMs may include an appliance maintenance LLM that monitors the performance data of smart devices, a routine automation LLM that personalizes and manages routines throughout the smart home, or a home coordination LLM that integrates calendar data from across the home to manage the schedules of family members.

[0082] Example Method

[0083] Figure 11 , Figure 12 and Figure 13 Example methods 1100, 1200, and 1300 for generating actionable insights from smart home event data are described. Methods 1100, 1200, and 1300 are shown as sets of operations (or actions) performed, but are not necessarily limited to the order or combination of operations shown herein. Furthermore, any one or more operations can be repeated, combined, reorganized, or linked to provide a wide range of additional and / or alternative methods. References can be found in the sections discussed below. Figure 1 Environment 100, and Figures 1 to 10 The entities detailed herein are referenced only as examples. The technology is not limited to being performed by one or more entities operating on a device.

[0084] Figure 11 An example method 1100 for generating actionable insights from smart home event data is shown. At 1102, home data is generated by one or more home monitoring sensors (e.g., Figure 6 Home data 604). One or more home monitoring sensors include one or more of the following: smart doorbells, smart locks, security cameras, motion sensors, home hubs, wireless communication devices, smart thermostats, alarms, smart garage door openers, smart smoke detectors, smart water monitors, smart lights, smart sockets, smart switches, smart appliances, smart irrigation systems, smart speakers, smart displays, and smart TVs. At 1104, by an ML model (e.g., Figure 6 The ML model 602) receives household data. In all respects, the ML model is an LLM (e.g., Figure 10 The LLM 1002). An LLM may include one or more LoRA layers and may be trained using RAG at least in part. The ML model may also be stored in a first device (e.g., Figure 2 The memory of the user device 202) (e.g., Figure 3 In the memory 304).

[0085] At position 1106, one or more correlations are generated by the ML model (e.g., Figure 6One or more correlations (606). The generation of one or more correlations may be based on household data. In the example, one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, household device status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation. The generation of one or more correlations may include one or more correlations between two or more household monitoring sensors. The ML model may further generate one or more correlations by generating one or more correlation values ​​between two or more members (e.g., devices) of the household data. One or more correlation values ​​may be based on the determined amount of correlation between two or more members of the household data. The ML model may compare one or more correlation values ​​to one or more thresholds, which may be based on one or more of the following: data time, data type, data history, or device type. If at least two of the two or more members of the household data exceed one or more thresholds, the ML model may generate a subset of data that includes the at least two of the two or more members of the household data.

[0086] At 1108, generate a family operation (e.g., Figure 6 (Home operation 608). In some examples, the ML model generates home operations, and in other examples, one or more processors generate home operations. The generation of home operations may be based on one or more correlations, and may further be based on the device capabilities of one or more home monitoring sensors in one or more home monitoring sensors. In some examples, the generated home operations include multiple home operations. The generation of home operations is performed automatically or periodically in other examples. In some examples, the ML model generates home operations based on a subset of data from one or more correlation values. Further, the generation of home operations may include generating one or more operation correlation values ​​between two or more available operations of one or more home monitoring sensors. The one or more operation correlation values ​​may be based on a determined amount of correlation between the two or more available operations. In addition, at least one of the two or more available operations is a new operation on which the ML model has not yet been trained. The generation of home operations may further be based on a comparison of one or more operation correlation values ​​with one or more operation thresholds. The ML model can generate operation correlation values.

[0087] At 1110, configure home operations for output to users (e.g., Figure 2 User 204). In various aspects, home operation is performed by one or more processors (e.g., Figure 3One or more processors (302) are configured. One or more processors may be housed in a second device, different from the first device, and the first and second devices may communicate wirelessly. In some examples, the second device includes a mobile device (e.g., Figure 2 The user device 202), and the first device includes one of a cloud computer, a server, a home hub, or a wireless communication device. In other aspects, the ML model performs the configuration of home operations for output to the user.

[0088] Figure 12 An example method 1200 for generating actionable insights from smart home event data is shown. Method 1200 includes method 1100. At 1202, one or more processors output home operations to the user. At 1204, one or more processors receive user input (e.g., Figure 6 User input (610). User input may include a request to perform a home operation. At 1206, the home operation is performed. The execution of the home operation is performed by the ML model in some examples, and by one or more processors in other examples. In various aspects, the execution of the home operation includes activating one or more device capabilities of one or more home monitoring sensors in the home monitoring sensor array.

[0089] Figure 13 An example method 1300 for generating actionable insights from smart home event data is shown. Method 1300 includes method 1100. At 1302, a user request for relevance to home data is received (e.g., Figure 6 Family operation request 614). In various aspects, the generation of one or more correlations (e.g., Figure 11 At 1106, the response is a user request regarding the relevance of household data. At 1304, the generated household operations comprise multiple household operations, and these multiple household operations are ranked. In some examples, the ML model performs the ranking of multiple household operations. In various aspects, the household operations are configured for the output (e.g., Figure 11 Step 1110) is based on a ranking of multiple family operations. At 1306, the multiple family operations are compared with one or more family thresholds. In each respect, configuring family operations for output is performed in response to at least one of the multiple family operations being greater than one or more of the one or more family thresholds.

[0090] Additional examples

[0091] Some additional examples are described below.

[0092] Example 1: A method for generating device operations, the method comprising generating household data from one or more household monitoring sensors. The method further comprises receiving the household data by a machine learning (ML) model, and generating one or more correlations by the ML model based on the household data. The method further comprises generating household operations based on the one or more correlations, and configuring the household operations by one or more processors for output to a user.

[0093] Example 2: The method as described in Example 1, wherein generating the one or more correlations by the ML model based on the family data includes: generating one or more correlation values ​​between two or more members in the family data by the ML model, the one or more correlation values ​​being based on a determined correlation amount between the two or more members in the family data. Generating the one or more correlations further includes: comparing the one or more correlation values ​​with one or more thresholds by the ML model; determining by the ML model that at least two of the two or more members in the family data exceed the one or more thresholds; and generating a data subset including the at least two members of the two or more members in the family data.

[0094] Example 3: The method as described in Example 2, wherein the one or more thresholds are based on one or more of the following: data time, data type, data category, data history, or device type.

[0095] Example 4: The method as described in any of the preceding examples, wherein generating the home operation is further based on one or more capabilities of the one or more home monitoring sensors.

[0096] Example 5: The method as described in Example 4, wherein the generation of the home operation is performed by the ML model, which is trained at least in part using an ML training dataset comprising multiple capabilities of multiple home monitoring devices, and at least one of the one or more capabilities of the one or more home monitoring sensors is not a member of the multiple capabilities of the multiple home monitoring devices.

[0097] Example 6: The method as described in any of the foregoing examples further includes the one or more processors outputting the home operation to the user. The method further includes the one or more processors receiving user input including a request to perform the home operation, and performing the home operation.

[0098] Example 7: The method as described in Example 6, wherein the execution of the home operation is performed by the one or more processors.

[0099] Example 8: The method as described in Example 6, wherein the execution of the family operation is performed by the ML model.

[0100] Example 9: The method as described in any one of Examples 6 to 8, wherein the generation of the home operation is further based on the device capabilities of one or more of the home monitoring sensors. The execution of the home operation includes activating the device capabilities of the one or more of the home monitoring sensors.

[0101] Example 10: The method as described in any of the preceding examples, wherein the one or more home monitoring sensors include one or more of the following: smart doorbell, smart lock, security camera, motion sensor, home hub, wireless communication device, smart thermostat, alarm, smart garage door opener, smart smoke detector, smart water monitor, smart light, smart socket, smart switch, smart appliance, smart irrigation system, smart speaker, smart display and smart TV.

[0102] Example 11: The method as described in any of the preceding examples, wherein configuring the home operation for output to the user is performed by the ML model.

[0103] Example 12: The method as described in Example 1, wherein the generation of the home operation is performed by the one or more processors.

[0104] Example 13: The method as described in any of the preceding examples, wherein the generated one or more correlations include one or more of the following: time correlation, object detection correlation, event correlation, household appliance status correlation, weather correlation, pattern correlation, historical correlation, category correlation, and energy usage correlation.

[0105] Example 14: The method as described in any of the foregoing examples further includes receiving a user request for a correlation of household data. The generation of the one or more correlations is in response to receiving the user request for the correlation of the household data.

[0106] Example 15: The method as described in any of the preceding examples, wherein the ML model is a large language model (LLM).

[0107] Example 16: The method as described in Example 15, wherein the LLM includes one or more low-rank adaptive (LoRA) layers.

[0108] Example 17: The method described in Example 15, wherein the LLM is trained at least in part using Retrieval Augmentation Generation (RAG).

[0109] Example 18: The method as described in any of the preceding examples, wherein the generated family operation includes multiple family operations.

[0110] Example 19: The method as described in Example 18 further includes ranking the plurality of family operations.

[0111] Example 20: The method as described in Example 19, wherein the family operation is configured so that the output is based on the ranking.

[0112] Example 21: The method of Example 18 further includes comparing the plurality of family operations with one or more family thresholds. Configuring the family operations for output is performed in response to at least one of the plurality of family operations being greater than one or more of the one or more family thresholds.

[0113] Example 22: The method as described in any of the preceding examples, wherein the generation of the family operation is performed automatically.

[0114] Example 23: The method as described in any of the preceding examples, wherein the generation of the family operation is performed periodically.

[0115] Example 24: The method as described in any of the preceding examples, wherein the ML model is stored in the memory of a first device, and the one or more processors are housed in a second device, which is different from the first device. The first device and the second device communicate wirelessly with each other.

[0116] Example 25: The method as described in Example 24, wherein the second device includes a mobile device, and the first device includes one of a cloud computer, a server, a home hub, or a wireless communication device.

[0117] Example 26: The method as described in any of the preceding examples, wherein generating the home operation includes generating one or more operation correlation values ​​between two or more available operations of the one or more home monitoring sensors, the one or more operation correlation values ​​being based on a determined correlation amount between the two or more available operations.

[0118] Example 27: The method of Example 26 further includes comparing the one or more operation relevance values ​​with one or more operation thresholds, wherein generating the family operation is further based on the comparison.

[0119] Example 28: The method as described in any one of Examples 26 and 27, wherein the generation of the family operation is further based on the operation correlation value.

[0120] Example 29: The method as described in any one of Examples 26 to 28, wherein the operational relevance value is generated by the ML model.

[0121] Example 30: The method as described in any one of Examples 26 to 29, wherein at least one of the two or more available operations is a new operation, and the ML model has not yet been trained on the new operation.

[0122] Example 31: The method as described in any of the preceding examples further includes generating insights from the one or more correlations using the ML model. The insights include new information derived from the correlations that highlights patterns, behaviors, or data that are not directly apparent from the household data. The household operations are generated based on the insights.

[0123] Example 32: An electronic device comprising one or more processors and a memory, the memory including instructions that, when accessed by the one or more processors, cause the one or more processors to perform a method as described in any one of Examples 1 to 31.

[0124] Example 33: A non-transitory computer-readable medium storing instructions that, when accessed by one or more processors, cause the one or more processors to perform a method as described in any one of Examples 1 to 31.

[0125] Example 34: A computer programming product that stores instructions that, when accessed by one or more processors, cause the one or more processors to perform a method as described in any one of Examples 1 to 31.

[0126] in conclusion

[0127] Although techniques and devices incorporating such techniques for generating actionable insights from smart home event data have been described in feature- and / or method-specific language, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations for generating actionable insights from smart home event data.

Claims

1. A method for operating a generating apparatus, the method comprising: Home data is generated by one or more home monitoring sensors; The household data is received by a machine learning (ML) model; The ML model generates one or more correlations based on the family data; Generate family operations based on the one or more of the aforementioned correlations; as well as The home operation is configured by one or more processors for output to the user.

2. The method as described in claim 1, wherein, Generating one or more correlations from the household data using the ML model includes: The ML model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on the determined correlation amount between the two or more members in the family data; The ML model compares the one or more correlation values ​​with one or more thresholds; The ML model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; and Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

3. The method of claim 1, further comprising: The home operation is output to the user by the one or more processors; The one or more processors receive user input including a request to perform the home operation; as well as Perform the aforementioned family operation.

4. The method of claim 1, wherein: The generation of the home operation is further based on the capability of one or more devices of the one or more home monitoring sensors; and The execution of the home operation includes activating the device capability of the one or more home monitoring sensors.

5. The method of claim 1, wherein, The generated correlations include one or more of the following: time correlations, object detection correlations, event correlations, household appliance status correlations, weather correlations, pattern correlations, historical correlations, category correlations, and energy usage correlations.

6. The method of claim 1, wherein, The ML model mentioned is a large language model (LLM).

7. The method of claim 1, further comprising generating insights from the one or more correlations by the ML model, wherein: The insights include new information derived from the one or more correlations, highlighting patterns, behaviors, or data that are not directly apparent from the household data; and The family operations are generated based on the insights provided.

8. An electronic device comprising: One or more processors; as well as The memory includes instructions that, when accessed by the one or more processors, cause the one or more processors to: Home data is received by a machine learning (ML) model, which is generated by one or more home monitoring sensors; The ML model generates one or more correlations based on the family data; Generate family operations based on the one or more of the aforementioned correlations; as well as The home operation is configured by one or more processors for output to the user.

9. The electronic device as claimed in claim 8, wherein, Generating one or more correlations from the household data using the ML model includes: The ML model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on the determined correlation amount between the two or more members in the family data; The ML model compares the one or more correlation values ​​with one or more thresholds; The ML model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; and Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

10. The electronic device of claim 8, wherein, The instruction further causes the one or more processors to: The home operation is output to the user; Receive user input including a request to perform the aforementioned home operation; and Perform the aforementioned family operation.

11. The electronic device of claim 8, wherein: The generation of the home operation is further based on the capability of one or more devices of the one or more home monitoring sensors; and The execution of the home operation includes activating the device capability of the one or more home monitoring sensors.

12. The electronic device of claim 8, wherein, The generated correlations include one or more of the following: time correlations, object detection correlations, event correlations, household appliance status correlations, weather correlations, pattern correlations, historical correlations, category correlations, and energy usage correlations.

13. The electronic device of claim 8, wherein, The ML model mentioned is a large language model (LLM).

14. The electronic device of claim 8, wherein, The instructions further cause the one or more processors to generate insights from the one or more correlations using the ML model, wherein: The insights include new information derived from the one or more correlations, highlighting patterns, behaviors, or data that are not directly apparent from the household data; and The family operations are generated based on the insights provided.

15. A non-transitory computer-readable medium storing instructions that, when accessed by one or more processors, cause the one or more processors to: Home data is generated by one or more home monitoring sensors; The household data is received by a machine learning (ML) model; The ML model generates one or more correlations based on the family data; Generate family operations based on the one or more of the aforementioned correlations; as well as The home operation is configured by one or more processors for output to the user.

16. The non-transitory computer-readable medium of claim 15, wherein, Generating one or more correlations from the household data using the ML model includes: The ML model generates one or more correlation values ​​between two or more members in the family data, the one or more correlation values ​​being based on the determined correlation amount between the two or more members in the family data; The ML model compares the one or more correlation values ​​with one or more thresholds; The ML model determines that at least two of the two or more members in the family data exceed one or more of the thresholds; and Generate a subset of data, the subset of data including at least two of the two or more members in the family data.

17. The non-transitory computer-readable medium of claim 15, wherein, The instruction further causes the one or more processors to: The home operation is output to the user; Receive user input including a request to perform the aforementioned home operation; and Perform the aforementioned family operation.

18. The non-transitory computer-readable medium of claim 15, wherein: The generation of the home operation is further based on the device capabilities of one or more of the one or more home monitoring sensors; and The execution of the home operation includes activating the device capability of the one or more home monitoring sensors.

19. The non-transitory computer-readable medium of claim 15, wherein, The generated correlations include one or more of the following: time correlations, object detection correlations, event correlations, household appliance status correlations, weather correlations, pattern correlations, historical correlations, category correlations, and energy usage correlations.

20. The non-transitory computer-readable medium of claim 15, wherein, The ML model mentioned is a large language model (LLM).

Images