Collaborative perception-based home system control method and device, and electronic device

Abstract

This invention provides a home system control method, device, and electronic device based on collaborative perception, belonging to the field of home equipment. The method includes: generating predicted data for each home device in the current home scenario; determining whether each home device is in a surge state based on the perceived data and predicted data, and counting the number of home devices in a surge state; when the number of home devices in a surge state meets the triggering condition, performing counterfactual causal inference using the deviation between the perceived data and the predicted data as the result variable to determine the attribution result; and determining the corresponding yield control strategy based on the attribution result. This invention enables the system to automatically determine whether a surge state exists based on the comparison of the perceived data and predicted data of each home device, and triggers counterfactual inference when multiple devices are in a surge state, so as to select an automation rule pause strategy within an appropriate range to meet the user's temporary change of needs.

Description

Technical Field

[0001] This invention relates to the field of home appliance technology, and in particular to a home system control method, device, electronic device, and readable storage medium based on collaborative sensing. Background Technology

[0002] A smart home system refers to a system that uses Internet of Things (IoT) technology to interconnect various home devices and enable them to operate automatically and collaboratively based on preset rules or intelligent algorithms. With the development of IoT and AI technologies, smart home systems have evolved from remote control of single devices to whole-house intelligent systems with environmental adaptability. These systems aim to dynamically adjust the operating strategies of home devices according to the user's actual needs, thereby improving living comfort and convenience.

[0003] In related technologies, smart home automation control generally adopts a trigger threshold mechanism. The system pre-binds corresponding trigger conditions to each automation rule. When the collected data can determine that the home device meets the trigger threshold, the corresponding automation execution action is directly triggered to complete the linkage operation of the device such as switching on and off and adjusting parameters, thereby realizing the automated response of the home scene.

[0004] However, the above solutions are mostly based on static rules and are difficult to adapt to changes in user needs in real time, resulting in a poor user experience. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed to provide a home system control method, apparatus, electronic device and readable storage medium based on collaborative perception to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, embodiments of the present invention provide a home system control method based on collaborative sensing, the method comprising: In executing automation rules for home scenarios, the perception data of home devices related to the automation rules are obtained; Generate predictive data for each of the aforementioned home devices in the current home scenario; Based on the perceived data and the predicted data, determine whether each home appliance is in a surge state, and count the number of home appliances in the surge state; When the number of home devices in the surge state meets the triggering condition, counterfactual causal inference is performed using the deviation between the perceived data and the predicted data as the result variable to determine the attribution result; Based on the attribution results, a yield control strategy corresponding to the attribution results is determined; the yield control strategy is used to pause the automation rules of the home system.

[0007] Optionally, generating prediction data for each of the home devices in the current scenario includes: Determine the current scene type; Input the current scene type into the behavior intent layer of the hierarchical generation model to obtain the predicted distribution of the user's acceptance probability of the automated actions of home devices; Using the predicted acceptance probability distribution as a condition, the predicted distribution corresponding to the perceived data of the home appliances is obtained through the multimodal observation layer in the hierarchical generation model, and is used as the predicted data of the home appliances in the current scenario.

[0008] Optionally, determining the current scene type includes: Obtain the hidden state of the scene at the previous time step stored in the scene context layer of the hierarchical generation model; Obtain current time characteristics; The hidden state of the scene at the previous time step and the current time feature are input into the scene transition probability matrix in the scene context layer, and the current scene type is obtained by operating on the scene transition probability matrix.

[0009] Optionally, determining whether each home appliance is experiencing a surge in demand based on the sensed data and the predicted data includes: Determine the deviation between the perceived data of each of the aforementioned home devices and the corresponding predicted data; Determine the surge threshold for each of the aforementioned home appliances; When the deviation of the home appliance exceeds the corresponding surge threshold, the home appliance is determined to be in a surge state.

[0010] Optionally, the step of using the deviation between the perceived data and the predicted data as the outcome variable to perform counterfactual causal inference to determine the attribution result includes: Using the deviation between the perceived data and the predicted data as the outcome variable, an intervention sample is constructed; The intervention sample is input into a preset counterfactual causal inference model, and a preset historical control sample set is obtained through the counterfactual causal inference model. The historical control sample includes historical outcome variables, which are composed of the deviation between the historical perceived data and the predicted data. From the historical control sample set, a preset number of historical control samples that meet the similarity characteristic conditions with the intervention sample are selected. Based on the outcome variables of the intervention sample and the historical outcome variables of the preset number of historical control samples, the estimated value of the control result is determined. Based on the estimated value of the control result, the attribution result is determined.

[0011] Optionally, determining the attribution result based on the estimated value of the control result includes: Determine the interval in which the estimated value of the control result lies; Based on the preset mapping relationship between the interval and the attribution result, the attribution result corresponding to the interval in which the estimated value of the control result is located is determined.

[0012] Optionally, the attribution results include systemic causes and external causes; The determination of the attribution result corresponding to the interval in which the estimated control result falls includes: When the estimated value of the control result is in the first interval, the attribution result is determined to be a systematic cause. When the estimated value of the control result is in the second interval, the attribution result is determined to be an external cause; the value in the first interval is greater than the value in the second interval; The step of executing a yield control strategy corresponding to the attribution result includes: When the attribution result is the system cause, identify the target home device that has performed automated actions within the current time window and suspend the execution of the automation rules for the target home device. When the attribution result is the external cause, the execution of the automation rules for the preset home device type is suspended.

[0013] Optionally, the sensing data includes state change data, audio data, and network traffic data.

[0014] Secondly, embodiments of the present invention provide a home system control device based on collaborative sensing, the device comprising: The perception data acquisition module is used to acquire perception data of home devices related to the automation rules in the execution of home scene automation rules; The prediction data generation module is used to generate prediction data for each of the aforementioned home devices in the current home scenario; The surge state determination module is used to determine whether each home device is in a surge state based on the sensed data and the predicted data, and to count the number of home devices in the surge state. The counterfactual causal inference module is used to perform counterfactual causal inference to determine the attribution result when the number of home devices in the surge state meets the triggering condition, using the deviation between the perceived data and the predicted data as the result variable. The yield control strategy determination module is used to determine a yield control strategy corresponding to the attribution result based on the attribution result; the yield control strategy is used to pause the automation rules of the home system.

[0015] Thirdly, embodiments of the present invention provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the home system control method based on collaborative perception as described in the first aspect.

[0016] Fourthly, embodiments of the present invention provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the home system control method based on collaborative perception as described in the first aspect.

[0017] The embodiments of the present invention have the following advantages: This invention acquires the perception data of home devices related to the automation rules in the execution of home scenarios, and generates predictive data for each home device in the current home scenario. This provides the system with a reference standard for judging whether the actual perception data is normal, and also gives subsequent deviation judgment a clear quantitative benchmark, rather than relying on fixed thresholds or manual settings.

[0018] Based on perceived and predicted data, the system determines whether each home appliance is in a surge state and counts the number of appliances in a surge state. When the number of appliances in a surge state meets the triggering condition, counterfactual causal inference is performed using the deviation between perceived and predicted data as the outcome variable to determine the attribution result. Based on the attribution result, a corresponding yield control strategy is determined. The yield control strategy is used to pause the automation rules of the home system. This enables the system to automatically determine whether a surge state exists based on the comparison of perceived and predicted data for each home appliance, and trigger counterfactual inference when multiple appliances are in a surge state to select an appropriate range of automation rule pause strategies to meet users' temporary changes in needs. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the steps of a home system control method based on collaborative perception provided in an embodiment of the present invention. Figure 2 This is a structural block diagram of a home system control device based on collaborative perception provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] A smart home system refers to a system that uses Internet of Things (IoT) technology to interconnect various home devices and enable them to operate automatically and collaboratively based on preset rules or intelligent algorithms. With the development of IoT and AI technologies, smart home systems have evolved from remote control of single devices to whole-house intelligent systems with environmental adaptability. These systems aim to dynamically adjust the operating strategies of home devices according to the user's actual needs, thereby improving living comfort and convenience.

[0024] In related technologies, smart home automation control commonly employs a trigger threshold mechanism. The system pre-binds corresponding trigger conditions to each automation rule. When the collected data determines that the home device meets the trigger threshold, the corresponding automated action is directly triggered, completing linked operations such as device switching and parameter adjustment, thereby achieving automated response in the home scene. However, the above solutions are mostly based on static rules and are difficult to adapt to changes in user needs in real time, resulting in a poor user experience.

[0025] One of the core concepts of this invention is to enable the system to automatically determine whether there is a surge in demand based on the comparison of the sensing data and the predicted data of each home device, and to trigger counterfactual inference when multiple devices are in a surge state, so as to select an automation rule suspension strategy within the appropriate range to meet the user's temporary change of needs.

[0026] Figure 1 This is a flowchart of the steps of a home system control method based on collaborative perception provided in an embodiment of the present invention.

[0027] like Figure 1 As shown, the method may specifically include the following steps: Step 101: In executing the automation rules of the home scene, obtain the perception data of the home devices related to the automation rules; In practical applications, this step can be executed by a home smart gateway, which connects to smart speakers, smart routers, and other home devices deployed within the home. Automation rules are preset action trigger rules for home devices within the home system, such as "if the current time is 8:00 PM and the user is in a leisure scenario, automatically dim the study lights." During the execution of these home scenario automation rules, the gateway acquires real-time sensing data from the home devices related to those rules.

[0028] In some embodiments, the sensed data includes state change data, audio data, and network traffic data.

[0029] State change data includes state change logs for each home appliance and their corresponding trigger source tags. Specifically, the state change logs record changes in the operating status of home appliances, such as changes in the on / off status of lights or switching modes of a robot vacuum cleaner. When a state change occurs, each home appliance summarizes and reports its state change log and corresponding trigger source tags to the gateway. The trigger source tags distinguish whether the state change is triggered by a system automation rule or by user manual intervention. For example, if the system executes a dimming action according to the automation rule "automatically dim the lights at 20:00," the state change corresponding to this dimming action is marked as automated triggering; if the user adjusts the light brightness via a physical switch or voice command, the state change is marked as user manual triggering. By distinguishing trigger source tags, the system can identify whether the user has resisted the automation rule. For example, if the user manually operates the device within a preset time window after the system executes an automated action, this operation is marked as user resistance event E. override .

[0030] The audio data includes paralinguistic acoustic metadata when the smart speaker is activated. Specifically, when a user wakes the smart speaker with their voice, the smart speaker extracts this paralinguistic acoustic metadata locally, rather than uploading the audio stream to the cloud for semantic parsing. This paralinguistic acoustic metadata may include wake word volume, signal-to-noise ratio, speech rate characteristics, and the number of consecutive retries. After extraction, the smart speaker reports the audio data to the gateway. In essence, the system only acquires the acoustic physical characteristics of the wake word, without parsing the specific semantic content spoken by the user, thus effectively protecting the user's voice privacy while sensing their emotions.

[0031] Network traffic data includes transport layer traffic patterns within the local area network (LAN) as statistically analyzed by the smart router. As an example, the smart router can activate a lightweight traffic probe to passively analyze the transport layer traffic patterns of various devices within the home LAN, particularly home appliances related to current automation rules. This process does not perform deep packet inspection or decrypt any application layer content.

[0032] Transport layer traffic characteristics can include packet length distribution, packet arrival time jitter, uplink / downlink bandwidth ratio, and timing variation rate. Packet length distribution reflects the statistical regularity of data packet size; packet arrival time jitter reflects the uniformity of data packet arrival times; and the uplink / downlink bandwidth ratio reflects the proportional relationship between uplink and downlink traffic. For example, in asymmetric browsing mode, uplink traffic is much less than downlink traffic, while in real-time conferencing mode, uplink and downlink traffic tend to be symmetrical. After obtaining network traffic data, the intelligent router reports this data to the gateway. This network traffic data can then be used to perceive changes in user network behavior patterns without revealing the specific content of communications.

[0033] Step 102: Generate prediction data for each of the aforementioned home devices in the current home scenario; In practical applications, this can be achieved through a hierarchical generative model deployed in a home smart gateway. This hierarchical generative model is a three-layer probabilistic model comprising a scene context layer, a behavioral intent layer, and a multimodal observation layer, used to generate the predicted distribution of various modalities of home devices from top to bottom.

[0034] In some embodiments, step 102 specifically includes the following sub-steps: Sub-step S11: Determine the current scene type; In some embodiments, sub-step S11 specifically includes: Obtain the hidden state of the scene at the previous time step stored in the scene context layer of the hierarchical generative model; Obtain current time characteristics; The hidden state of the scene at the previous time step and the current time feature are input into the scene transition probability matrix in the scene context layer, and the current scene type is obtained by operating on the scene transition probability matrix.

[0035] As an example, the scene context layer in a hierarchical generative model maintains the hidden state z of the current scene. t This hidden state is a discrete hidden variable used to represent the user's current scene type, and its value includes at least one of leisure, work, sleep, outing, and transition. The scene context layer stores the scene hidden state z from the previous time step. t-1 .

[0036] The current time characteristics include at least the hour and whether it is a weekday or weekend, i.e., the date type. The scene transition probability matrix records the transition tendency between different scene types. For example, in the matrix, the probability of transitioning from a sleep scene to a work scene at 8 AM on a weekday is set to 0.7, while the probability of transitioning from a leisure scene to a sleep scene at 10 PM on a weekend is set to 0.6. The system sets the hidden state z of the scene from the previous time step... t-1 Using the current time features as input, and performing calculations through the scene transition probability matrix, the current scene type z is inferred. t In some embodiments, the initial value of the scene transition probability matrix is ​​set based on daily rhythm data from population statistics, and is fine-tuned according to the user's actual work and rest patterns after system deployment.

[0037] Sub-step S12: Input the current scene type into the behavior intent layer in the hierarchical generation model to obtain the predicted distribution of the user's acceptance probability of the automated actions of home devices; The behavioral intent layer in the hierarchical generation model is based on the current scene type z. t Given the condition, generate predictions of user acceptance probability for various automated actions of home appliances. For each type of automated action 'a' in the system... k Such as dimming light control a lightalight Cleaning equipment a clean Audio and video equipment a media The behavioral intent layer outputs a Beta distribution with two parameters α. k and β k This represents the uncertainty the system estimates regarding the event that the user will accept the automated action. The Beta distribution is expressed as:

[0038] Where, α k and β k Having a pseudo-counting meaning, α k β represents the equivalent cumulative number of times the user accepts this automated action in the current scenario; k This represents the equivalent cumulative number of times a user resists the automated action in the current scenario.

[0039] Based on α k and β k The expected acceptance probability can be calculated as follows:

[0040] For example, for automatic dimming lights on weekend evenings, the Beta distribution output by the behavioral intent layer is Beta (12, 3), indicating that the system has equivalently remembered 12 instances of user acceptance and 3 instances of user resistance, with an expected acceptance probability of approximately 80%. For automatic robot vacuum cleaning on weekend evenings, the Beta distribution is Beta (3, 7), indicating that the system has equivalently remembered 3 instances of user acceptance and 7 instances of user resistance, with an expected acceptance probability of approximately 30%. k and β k The specific values ​​are gradually evolved by the system's online learning mechanism based on the user's actual interaction history.

[0041] Sub-step S13: Using the acceptance probability prediction distribution as a condition, the prediction distribution corresponding to the perception data of the home device is obtained through the multimodal observation layer in the hierarchical generation model, and used as the prediction data of the home device in the current scenario.

[0042] In the hierarchical generative model, the multimodal observation layer generates a prediction distribution of specific observations for each modality of home appliances, conditioned on the state of the behavioral intent layer (i.e., the user's acceptance / resistance state to automated actions as represented by the acceptance probability prediction distribution). This prediction distribution can include the following three types of modal prediction distributions: One is the device interaction modality prediction distribution corresponding to state change data. The multimodal observation layer predicts whether the user will trigger the automated resistance event E within the current time window. override The event is modeled as a Bernoulli distribution, which is used to characterize the probability that a user will resist an automated action from the system.

[0043] Secondly, there is the sub-language acoustic modality prediction distribution corresponding to the audio data. The multimodal observation layer predicts the deviation of the wake-up volume of the user when waking up the smart speaker relative to the baseline. This deviation is modeled as a Gaussian distribution N(0, σv2(zt)) centered at 0. Here, σv2(zt) is the variance parameter in the current scene, representing the fluctuation range of the user's volume in this scene.

[0044] Thirdly, the network traffic modality prediction distribution corresponding to the network traffic data. The multimodal observation layer predicts the interaction density feature vector of the current network traffic. The interaction density feature vector includes statistical features such as uplink / downlink packet frequency ratio, burstiness, and quiet period ratio. This feature vector is modeled as a multivariate Gaussian distribution.

[0045] Through the aforementioned top-down, layer-by-layer generation process, the hierarchical generation model ultimately outputs the predicted distribution of each modality of home appliances in the current scenario, serving as the predicted data for the home appliances in the current scenario. These predicted distributions will be compared with the actually observed perceived data in subsequent steps to calculate the prediction error and to perform the subsequent counterfactual causal inference process.

[0046] Step 103: Based on the perceived data and the predicted data, determine whether each home appliance is in a surge state, and count the number of home appliances in the surge state; In some implementation scenarios, the perceived data obtained in step 101 is compared modally with the predicted data generated in step 102 to extract the normalized prediction error for each modality. The prediction error for each modality is then compared with its respective surge threshold to determine whether each modality is in a surge state. It is worth noting that a modality is a data-level classification abstraction of home appliances, and each modality corresponds to the perceived data generated by a specific type of home appliance.

[0047] In some embodiments, step 103 specifically includes the following sub-steps: Sub-step S21: Determine the deviation between the perceived data of each of the home devices and the corresponding predicted data; In some embodiments, sub-step S21 specifically includes: calculating the device interaction modality prediction error, the sub-language acoustic modality prediction error, and the network traffic modality prediction error, respectively, as the deviation between the perceived data and the corresponding prediction data of each modality.

[0048] To address device interaction modality prediction errors, obtain a flag E indicating whether the user has triggered an automated resistance event within the current time window. override，t Among them, E override，t =1 indicates that the user resisted the system's automated action, for example, after the system automatically dimmed the lights according to automation rules, the user manually turned the lights back on using a physical switch. override，t =0 indicates that no resistance event occurred.

[0049] Obtain the automated action 'a' executed within the current time window from the behavior intent layer of the hierarchical generative model. Expected acceptance probability The expected acceptance probability is determined by Beta(αa) ,βa The mean of ) is given.

[0050] Multiplying the resistance event flag by the expected acceptance probability yields the device interaction modality prediction error:

[0051] The essence of this formula is that the more confident the system is that a user will accept a certain automated action, the greater the prediction error will be when actual resistance occurs. For example, suppose the system predicts that the user has a 90% probability of accepting automatic dimming lights (based on the user never resisting during this time of the past four weekends), but the user resists this time... This is characterized as an unexpected resistance event with a high error value. Conversely, assuming the system predicts that the user has only a 30% probability of accepting the robot vacuum's scheduled cleaning (based on the user manually canceling it 2 out of the past 3 times), and the user resists this time, then... This is characterized as a resistance event with a low error value that is within expectations. Through the above mechanism, the system becomes more sensitive to unexpected resistance than to expected resistance, thus enabling it to distinguish between two different situations: a system model error and user behavior that meets expectations.

[0052] To address the prediction error of the secondary language acoustic modality, the wake-up volume V of the smart speaker within the current time window is obtained. wake (t). The wake-up volume is extracted from the wake-up audio by the smart speaker and is expressed in decibels (dB).

[0053] Obtain the multimodal observation layer in the hierarchical generative model for the current scene type z. t The predicted Gaussian distribution parameters of the paralinguistic acoustic modalities, and the personalized baseline mean μ. v (z t ) and standard deviation σ v (z t Among them, the personalized baseline mean μ v (z t The standard deviation σ represents the center level of the normal wake-up volume for a user in the current scenario. v (z t This represents the normal fluctuation range of the volume.

[0054] Substitute the current wake-up volume, baseline mean, and standard deviation into the following formula to calculate the para-language acoustic modality prediction error:

[0055] The formula essentially means that a positive prediction error only occurs when the current wake-up volume deviates from the baseline by more than one standard deviation, thus tolerating normal volume fluctuations by the user. For example, suppose a user's baseline volume in the current scenario has a mean of 55 dB and a standard deviation of 8 dB. When the user wakes up by speaking normally, the wake-up volume is 60 dB, and the deviation is... It did not exceed 1 standard deviation, therefore voice =0, which is considered normal fluctuation. When a user wakes up by yelling due to frustration, the wake-up volume reaches 77.4dB, with a deviation factor of [missing value]. Substituting into the formula, we get voice =max(0, 2.8-1)=1.80, indicating that the user's acoustic modality has deviated significantly from its normal range.

[0056] To address network traffic modality prediction errors, the actual network traffic characteristic distribution within the current time window is obtained from the statistics of the intelligent router. The actual network traffic characteristic distribution includes at least the probability distribution of statistical characteristics such as uplink / downlink packet frequency ratio, burstiness, and quiet period proportion. For example, the uplink / downlink packet frequency ratio can be discretized into three levels: low (<3:1), medium (3:1~10:1), and high (>10:1), and the actual observed distribution is the probability of each level.

[0057] Obtain the multimodal observation layer in the hierarchical generative model for the current scene type z. t Predicted network traffic modal distribution .

[0058] Calculate the KL divergence between the actual distribution and the predicted distribution, and use it as the network traffic modality prediction error:

[0059] The KL divergence measures the distance between two probability distributions; a larger value indicates a greater difference between the observed network traffic distribution and the system's predicted scenario distribution. For example, in a leisure scenario, the system predicts the network traffic uplink / downlink packet ratio distribution as follows: low probability 70%, medium probability 25%, and high probability 5%. However, the router probe actually observes a packet ratio of 1.2:1 for the PC in the study (belonging to a highly symmetrical bidirectional communication mode, falling into the low category), with an actual distribution of (100%, 0%, 0%). Therefore, the KL divergence is calculated as follows:

[0060] In practical systems, network traffic characteristics can be modeled as a multivariate Gaussian distribution, and the KL divergence is calculated using a closed-form solution. Here, a discretization example illustrates the principle. This method can detect significant changes in user network behavior patterns solely through the distribution shift of transport layer statistical characteristics without decrypting any application layer content.

[0061] Through the above calculations, we can obtain the prediction error of device interaction mode, the prediction error of sub-language acoustic mode, and the prediction error of network traffic mode. These three error values ​​together constitute the deviation between the perceived data and the corresponding predicted data of each mode.

[0062] Sub-step S22: Determine the surge threshold for each of the aforementioned home appliances; In some embodiments, the surge threshold for each modality is an adaptive surge threshold. The adaptive surge threshold is dynamically determined based on the historical prediction error distribution under the current scene type, rather than a fixed threshold. The specific determination method is as follows: For each mode i (i∈{device, voice, traffic}), the system maintains a history of prediction errors within an exponentially decaying sliding window to quickly capture recent trends in the errors. The exponentially decaying sliding window smooths the prediction errors using the following formula:

[0063] Where γ∈(0,1) is the attenuation factor (typically 0.9), which makes the impact of recent observations on the smoothed value greater than that of long-term observations. Used for trend monitoring, when the original prediction error... The adaptive threshold has not yet been exceeded, but the smoothing value is... When the error has been continuously increasing for several consecutive time steps, the system recognizes that the prediction error of this mode is forming an upward trend. At this time, the surge threshold of the next time step of this mode is temporarily lowered (for example, temporarily lowered by 10%) to enhance the system's sensitivity to gradual state drift.

[0064] The adaptive surge threshold is defined as the conditional quantile of the recent prediction error for this mode:

[0065] That is, the system takes the current scene type z. t In the same W most recent time steps, the 90th percentile of the modality prediction error is used as the surge threshold for the current time step. Only when the current prediction error exceeds the 90% historical error level of the same historical scenario is it judged as a surge.

[0066] The method for determining this conditional quantile threshold allows the system to automatically adapt to individual differences among users and behavioral variations of the same user in different scenarios. For example, for user A, who consistently speaks loudly, their historical prediction error for paralinguistic modality in leisure scenarios is generally between 0.5 and 1.5, and the 90th quantile is automatically set to approximately 1.4; while for user B, who speaks softly, their historical error is between 0.1 and 0.5, and the 90th quantile is only about 0.45. The same set of judgment logic exceeding the 90th quantile automatically adapts to the individual characteristics of different users, eliminating the need for manual parameter adjustments. Similarly, the baseline fluctuation of network traffic for the same user is inherently greater in work scenarios than in sleep scenarios; therefore, the 90th quantile threshold for traffic modality in work scenarios is automatically higher than that in sleep scenarios, preventing normal fluctuations from being misjudged as abnormal.

[0067] Sub-step S23: When the deviation of the home appliance exceeds the corresponding surge threshold, it is determined that the home appliance is in a surge state.

[0068] In some embodiments, the modal prediction errors calculated in sub-step S21 are... Adaptive surge thresholds for each mode determined in sub-step S22 Compare them one by one, for each mode i, like > If the mode is in a surge state, then the surge flag S is output. i If (t) = 1, otherwise output S. i (t)=0. Formalized:

[0069] For example, suppose the device modality adaptation threshold for a user in the current leisure scenario is 0.15. If the current device interaction modality prediction error is 0.85, exceeding the threshold of 0.15, then the device modality is determined to be in a surge state, i.e., S. device =1.

[0070] After determining the surge in each modality, the system counts the number of modalities in a surge state within the current time window:

[0071] In this embodiment, N t The value of can be 0, 1, 2, or 3, representing the number of modalities that simultaneously experienced a surge in prediction error within the current time window. This collaborative judgment mechanism indicates that a surge in a single modality is insufficient to trigger the subsequent attribution process; only when multiple modalities surge simultaneously does the system determine that the user's cognitive state has significantly shifted, thereby avoiding misjudgments caused by occasional fluctuations in a single modality.

[0072] Step 104: When the number of home devices in the surge state meets the trigger condition, counterfactual causal inference is performed using the deviation between the perceived data and the predicted data as the result variable to determine the attribution result; In some embodiments, the triggering condition specifically refers to calculating the cross-modal collaborative surge index CCI(t) based on the number of surge modes obtained from sub-step S23, and triggering the counterfactual causal inference process when CCI(t) = 1. Here, CCI(t) is defined as:

[0073] That is, when all three modalities (device interaction modality, secondary speech acoustic modality, and network traffic modality) are simultaneously in a surge state (N) t =3, CCI(t)=1), the system determines that the user's cognitive state has shifted significantly, triggering the subsequent attribution analysis process.

[0074] In some embodiments, step 104 specifically includes the following sub-steps: Sub-step S31: Using the deviation between the perceived data and the predicted data as the result variable, construct the intervention sample; In some embodiments, the intervention sample is a formal description of the system state and user state within the current time window, used as input to a counterfactual causal inference model. Specifically, the intervention sample comprises three components: a processing variable T, an outcome variable Y, and a confounding variable X. The processing variable represents whether the system has performed any proactive automation actions within the current time window [t-Δ, t]. If the system has performed at least one proactive automation action within the time window, then T=1; if the system has not performed any proactive automation actions within the time window (the user only performs manual operations, or the device is in standby mode), then T=0.

[0075] The outcome variable is the deviation between the perceived data and the predicted data, specifically the three-modal prediction error vector calculated in sub-step S21. .

[0076] The confounding variables include temporal contextual features and a statistical summary of recent user interaction history. The temporal contextual features include at least the time window information (e.g., 8:00 PM - 9:00 PM) and date type (e.g., weekend or weekday). The statistical summary of recent user interaction history may include statistics such as the frequency of user acceptance / resistance to automated actions and average wake-up volume within a recent preset number of time windows. These confounding variables are used to control for confounding factors during subsequent matching, ensuring that the matched historical control samples are comparable to the current intervention samples in terms of contextual features.

[0077] The above-mentioned treatment variable T, outcome variable Y, and confounding variable X are combined into an intervention sample, denoted as (X, T, Y), which is input into the preset counterfactual causal inference model.

[0078] Sub-step S32: Input the intervention sample into a preset counterfactual causal inference model, and obtain a preset historical control sample set through the counterfactual causal inference model; the historical control sample includes historical outcome variables, which are composed of the deviation between the historical perceived data and the predicted data; select a preset number of historical control samples from the historical control sample set that meet the similarity characteristic conditions with the intervention sample; determine the control result estimate based on the outcome variables of the intervention sample and the historical outcome variables of the preset number of historical control samples; determine the attribution result based on the control result estimate.

[0079] In some embodiments, the counterfactual causal inference model maintains a historical control sample library. This historical control sample library consists of four-tuple data {X, T, Y, z} continuously recorded by the system during daily operation for all time windows.

[0080] Where X is the context feature vector of the historical time window (including time period, date type, scene label, etc.), T is the marker of whether the system performed an active automated action within the historical time window (T=1 indicates that the automated action was performed, and T=0 indicates that it was not performed), Y is the trimodal prediction error vector within the historical time window (i.e., the historical outcome variable), and z is the scene label of the historical time window. The historical outcome variable consists of the deviation between the perceived data and the corresponding predicted data within the historical time window. It can be understood that the historical control sample library naturally contains two types of samples: treatment group samples (T=1, i.e., the time window in which the system performed automated actions) and control group samples (T=0, i.e., the time window in which the system did not perform any automated actions).

[0081] From the historical control sample set, a predetermined number of historical control samples that meet the similarity characteristics with the intervention sample are selected.

[0082] In some embodiments, the similarity feature condition is measured by a weighted Euclidean distance. Specifically, when it is necessary to estimate the counterfactual outcome of the current intervention sample, the system retrieves the K most similar control sample samples (T=0) from the historical control sample library that are most similar to the current confounding variable X. A typical value for K is 5 to strike a balance between estimation accuracy and computational cost. The similarity between samples is defined as the weighted Euclidean distance in the feature space:

[0083] Among them, w m The weight of the m-th feature is determined by the fact that features with a greater impact on matching accuracy have higher weights. For example, the current scene tag (such as leisure, work, sleep, etc.) is the primary factor determining user behavior and is given the highest weight (e.g., w=1.0); the difference in the exact hour of the time period feature has a smaller impact and is given a lower weight (e.g., w=0.3). The initial values ​​of each weight are preset based on domain experience and can also be automatically adjusted by the system during operation based on matching quality feedback. Matching features include the same scene tag (such as leisure scene), similar time periods (such as both being weekend evenings from 20:00 to 21:00), similar recent user interaction frequencies, and similar network activity levels.

[0084] Based on the outcome variables of the intervention sample and the historical outcome variables of a predetermined number of historical control samples, the estimated values ​​of the control results are determined.

[0085] In some embodiments, the counterfactual outcome estimate is used to simulate the user's multimodal prediction error when the system does not perform any automated actions under the current context features. Specifically, the system takes the mean of the outcome variables from the K matched control group samples (i.e., historical samples with T=0) as the counterfactual outcome estimate:

[0086] Among them, NN K (X, T=0) represents the K most similar historical samples in the historical control sample library that did not perform the automated action, which are matched with the current confounding variable X.

[0087] For example, suppose it is a leisure scene at 8:00 PM on a weekend, the system executes dimming control automation (T=1), the user resists, and the current device interaction mode prediction error is... device =0.85. The system retrieved three control samples from the 20:00-21:00 time period over the past four weekends, where the system did not perform any automated actions. The average prediction error of the device interaction modality in these samples was 0.02. .

[0088] In the current weekend evening leisure scenario, the system's execution of automated actions contributes an additional 0.83 to the device modal prediction error compared to doing nothing, and the resistance is almost entirely caused by automated actions.

[0089] The system further subtracts the estimated control result from the outcome variable (current actual deviation) of the intervention sample to obtain the individual treatment effect of the system's active automated actions on the prediction error of each modality:

[0090] The individual treatment effect The incremental contribution of the system performing automated actions to the prediction error of the i-th mode was quantified in the current scenario compared to doing nothing.

[0091] The attribution results are determined based on the estimated values ​​from the control results.

[0092] In some embodiments, the system tests the significance of the individual treatment effect using bootstrap sampling. Specifically, the system repeatedly samples B times (typically B=200) with replacement from the K matched control samples, calculating one result each time. This allows for the construction of a 95% confidence interval for the individual treatment effect. If the confidence interval does not contain 0, the individual treatment effect of that modality is considered significant.

[0093] For example, if the individual processing effect τ of the device interaction mode deviceThe 95% confidence interval is [0.71, 0.92], which does not include 0, so we determine τ. device Significantly positive; if τ device The 95% confidence interval is [-0.08, 0.35], which includes 0, so τ is determined to be... device Not significant; if τ device The 95% confidence interval is [-0.05, 0.41], which includes 0, so τ is determined to be... device Not significant.

[0094] The system determines the attribution result based on the significance of the individual treatment effect. Specifically: If τ device The device modal counterfactual prediction error is significantly positive and after removing the effects of automated actions. If the error is below the corresponding surge threshold, the cause is attributed to systemic reasons, indicating that the surge in the user's prediction error is mainly caused by automated actions performed by the system.

[0095] If τ device If the result is not significant (i.e., the counterfactual prediction error remains high even after removing automated actions), the attribution result is external, indicating that the surge in user prediction error is caused by external factors (such as work stress, emotional fluctuations, etc.) rather than by system behavior.

[0096] If τ device If the attribution is partially significant (i.e., the system behavior exacerbates the user's existing external stress state, but is not the only cause), then the attribution result is a mixed cause, meaning the system behavior exacerbates the user's existing external stress state, but is not the only cause.

[0097] Through the above counterfactual causal inference process, the system can distinguish whether the user's resistance is caused by the system's own automated behavior or by the user's sensitive reaction to the system's actions due to external factors, thus providing accurate attribution basis for the subsequent formulation of differentiated concession strategies.

[0098] In some embodiments, sub-step S32 includes: Determine the interval in which the estimated value of the control result lies; Based on the preset mapping relationship between the interval and the attribution result, the attribution result corresponding to the interval in which the estimated value of the control result is located is determined.

[0099] In some embodiments, determining the attribution result corresponding to the interval in which the estimated control result falls includes: When the estimated value of the control result is in the first interval, the attribution result is determined to be a systematic cause. When the estimated value of the control result is in the second interval, the attribution result is determined to be an external cause; the value in the first interval is greater than the value in the second interval; The estimated control result is the individual treatment effect calculated in sub-step S32. It quantifies the incremental contribution of automated actions to the prediction error of device interaction modes in the current scenario, compared to the system doing nothing. It incorporates individual processing effects. The interval in which the data falls is determined by comparing it with a preset interval boundary value. For example, the interval boundary value can be zero, or it can be dynamically set based on historical data.

[0100] For example, suppose the current device interaction mode prediction error is... device =0.85, the device modal surge threshold is 0.15. If =0.83, confidence interval [0.71, 0.92], significantly positive, and device - If the value is 0.02, which is below the threshold of 0.15, the attribution result is systemic, indicating that the surge in user prediction errors is mainly caused by automated actions performed by the system. The confidence interval is [-0.08, 0.35], which includes 0 and is not significant. Therefore, the attribution result is external, indicating that the surge in user prediction error is caused by external factors (such as work pressure, emotional fluctuations, etc.) rather than by system behavior. Significantly positive, but device - If the value of 0.70 is still higher than the threshold of 0.15, then the attribution result is a mixed cause, indicating that the system behavior exacerbated the user's existing external stress state, but was not the only cause.

[0101] Through the above counterfactual causal inference process, the system can distinguish whether the source of user stress is the system's own automated behavior or external factors, thus providing accurate attribution basis for the formulation of subsequent differentiated concession strategies.

[0102] Step 105: Based on the attribution result, determine the yield control strategy corresponding to the attribution result; the yield control strategy is used to pause the automation rules of the home system.

[0103] In some embodiments, step 105 specifically includes the following sub-steps: Sub-step S41: When the attribution result is the system cause, determine the target home device that has performed automated actions within the current time window, and suspend the execution of the automation rules of the target home device; When the attribution result is systemic, it indicates that the user's current resistance or stress is mainly caused by automated actions performed by the system itself. In this case, the system does not need to take a global retreat, but only needs to accurately locate the rule causing the problem, that is, the automation rule corresponding to the target home appliance that performed an automated action within the current time window [t-Δ, t] and was resisted by the user.

[0104] Specifically, the system can backtrack the automation execution log within the current time window to find all triggered automated actions and filter out the home devices corresponding to the automated actions that were resisted by the user. For example, if the system automatically dims the study light at 8:00 PM, and the user manually dims the light back using a physical switch within 3 minutes, then this dimming control rule is identified as the automation rule corresponding to the target home device. The system only suspends the execution of the automation rule for this target home device; other automation rules unrelated to this causal path (such as ambient music playback rules, humidifier start rules, etc.) continue to operate normally.

[0105] Sub-step S42: When the attribution result is the external cause, pause the execution of the automation rule for the preset home device type.

[0106] When the attribution result is external, it indicates that the user's current surge in prediction error is not caused by system behavior, but by a comprehensive state drift triggered by external stressors. In this case, the user's overall cognitive load is already high, and any unnecessary automation interference could further increase the user's burden. Therefore, the system suspends the execution of automation rules for preset home appliance types.

[0107] The preset home appliance types include non-safety-related categories, such as: pausing scheduled robot vacuum cleaning tasks (to prevent motor noise interference), pausing automatic dimming (to prevent light changes from stimulating the user), and pausing the automatic start / stop of humidifiers or air purifiers (to prevent device switching noise). The system only retains safety-related automation rules (such as smoke alarms and leak detection) and passive responsive services (i.e., responding only when the user actively initiates a command). In some embodiments, the system also proactively lowers the TTS response volume of the whole-house voice assistant to reduce the user's cognitive load.

[0108] In some embodiments, when the attribution result is the mixed cause, the system first executes the sub-step S42 (i.e., suspends the execution of the automation rules for the preset home device type), and after the system detects that the cross-modal collaborative surge index has recovered to 0 and has remained stable for more than a preset time, the system then performs a post-review and parameter correction on the automation rules for the target home device.

[0109] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0110] To enable those skilled in the art to better understand the embodiments of the present invention, an example is provided to illustrate the embodiments of the present invention: Scenario description: A user receives an urgent project from the company while at home on the weekend and is having an intense online meeting behind closed doors in their study.

[0111] Implementation process: T+0min: 20:00 in the evening, the system scene layer infers that the current scene is "Weekend Leisure" (zt=leisure). The generative model outputs a top-down prediction: the predicted user acceptance probability for "automatic dimming of lights" and "automatic playback of background music" is 85% and 72% respectively (based on the user's historical behavior patterns at this time of the past 4 weekends).

[0112] T+2min: The system automatically dims the study lights according to the rules and prepares to play music.

[0113] T+3min: The user's train of thought was interrupted, and he felt extremely irritated. He immediately turned around and pressed the physical switch on the wall to force the light back to its brightest setting. Then he shouted, "Don't play music!" (The voice assistant did not understand, so the user shouted "Stop playing" even louder).

[0114] T+3min: Device interaction mode: Generates 1 automation resistance event E override The system predicts an 85% acceptance probability, but the actual prediction error is... device =1×0.85=0.85 (The system is highly confident that the user will accept it; resistance constitutes a strong unexpected signal).

[0115] Sub-language acoustic modality: The volume baseline for this user's leisure scenario is μv=55dB, σv=8dB. The current volume V for shouting-style wake-up is... wake =77.4dB, the deviation is 77.4-55 / 8=2.8 standard deviations. According to the formula voice =max(0, 2.8-1)=1.80, which exceeds the adaptive threshold of 1.0 for this scenario.

[0116] Network traffic modality: The router probe detected a dramatic change in the network flow pattern of the PC in the study. The predicted probability of a low packet ratio in a leisure scenario was 70%, but in reality, the downlink packet ratio changed from a typical asymmetric browsing pattern (≈5:1) to a highly symmetric RTC pattern (≈1.2:1), meaning the actual distribution would be 100% concentrated in the low packet ratio range. Based on the aforementioned KL divergence calculation method, traffic ≈0.36 × correction factor (obtained from complete multivariate Gaussian KL divergence calculation) traffic =2.10), exceeding the adaptive threshold of 0.50 in this scenario, and surged synchronously.

[0117] T+3min: If the prediction errors of the three modes simultaneously exceed their respective adaptive thresholds, then CCI=1, and the system determines that the user's cognitive state has shifted significantly.

[0118] T+3min: The system constructs a counterfactual question: "If the system's automatic dimming light control and music playback are not executed, what will be the user's current trimodal prediction error mode?" The causal effect estimator makes inferences based on historical matching data. The system finds time windows in similar scenarios (weekend evening leisure scenarios) where the system did not perform proactive automation actions as control samples. In these control samples, the average prediction error for the device interaction modality is 0.02 (occasionally users manually operate the device but do not resist automation), the average prediction error for the paralinguistic modality is 1.65 (users' volume is naturally high when making phone calls), and the average prediction error for the network traffic modality is 1.90 (meeting traffic characteristics remain unchanged).

[0119] Compare the current observed values ​​with the mean of the control sample to calculate the individual treatment effect: τ device =0.85-0.02=0.83: Automated actions contributed approximately 83% of the prediction error to the equipment's resistance mode.

[0120] τ voice =1.80-1.65=0.15: Automated actions have only a slight additional effect on the voice modality (the volume is slightly higher than normal when the user says "Don't play music").

[0121] τ traffic =2.10-1.90=0.20: Automated actions have no substantial impact on traffic modality (conference traffic characteristics are independent of lighting control).

[0122] Counterfactual prediction error after removing the effects of automated actions: devicecf =0.85-0.83=0.02, which is lower than the surge threshold (the device modal threshold θ in this user's leisure scenario).device =0.15) voicecf =1.80-0.15=1.65, still higher than the surge threshold (the threshold is approximately 1.0). trafficcf =2.10-0.20=1.90, still above the surge threshold (the threshold is approximately 0.50). Causal attribution conclusion: Mixed causes. The surge in device resistance modality is primarily caused by system automated actions (lights + music); without these actions, users would not exhibit resistance. However, prediction errors for voice and data traffic modality mainly stem from external factors (call pressure from online meetings and changes in network patterns); these anomalous signals persist even if the system does nothing.

[0123] T+4min: The system triggers a global silent retreat: intercepts any scheduled automatic dimming intervention, suspends the hallway robot vacuum cleaning task originally scheduled to start at 20:30; and silences all notification sounds in the house.

[0124] At the same time, mark the dimming rules and background music rules for this time period as pending review.

[0125] T+120min: The system detected that the CCI remained at 0 for more than 2 hours, and network traffic returned to asymmetric leisure mode. The system automatically deactivated the fallback mode. A silent notification was sent to the user's app suggesting that automatic dimming control be disabled from 20:00 to 22:00 on weekends.

[0126] It should be noted that the home system control method based on collaborative perception provided in this embodiment of the invention can be executed by a home system control device based on collaborative perception, or by a control module within that home system control device for executing the home system control method based on collaborative perception. This embodiment of the invention uses the execution of the home system control method based on collaborative perception by a home system control device as an example to illustrate the home system control method based on collaborative perception provided in this embodiment of the invention.

[0127] Figure 2 This is a structural block diagram of a home system control device based on collaborative perception provided in an embodiment of the present invention.

[0128] like Figure 2 As shown in the figure, a home system control device based on collaborative perception provided by an embodiment of the present invention may specifically include the following modules: The perception data acquisition module 201 is used to acquire perception data of home devices related to the automation rules in the execution of home scene automation rules; Prediction data generation module 202 is used to generate prediction data for each of the aforementioned home devices in the current home scenario; The surge state determination module 203 is used to determine whether each home device is in a surge state based on the perceived data and the predicted data, and to count the number of home devices in the surge state. The counterfactual causal inference module 204 is used to perform counterfactual causal inference to determine the attribution result when the number of home devices in the surge state meets the triggering condition, using the deviation between the perceived data and the predicted data as the result variable. The yield control strategy determination module 205 is used to determine a yield control strategy corresponding to the attribution result based on the attribution result; the yield control strategy is used to pause the automation rules of the home system.

[0129] In some embodiments, the prediction data generation module includes: The current scene type determination submodule is used to determine the current scene type; The acceptance probability determination submodule is used to input the current scene type into the behavior intent layer in the hierarchical generation model to obtain the predicted distribution of the user's acceptance probability of the automated actions of home devices; The prediction data determination submodule is used to obtain the prediction distribution corresponding to the perception data of the home device through the multimodal observation layer in the hierarchical generation model, based on the acceptance probability prediction distribution, as the prediction data of the home device in the current scenario.

[0130] In some embodiments, the current scene type determination submodule includes: The hidden state acquisition unit is used to acquire the hidden state of the scene at the previous time step stored in the scene context layer of the hierarchical generation model. The current time feature acquisition unit is used to acquire the current time feature; The current scene type determination unit is used to input the scene hidden state of the previous time step and the current time feature into the scene transition probability matrix in the scene context layer, and obtain the current scene type through the operation of the scene transition probability matrix.

[0131] In some embodiments, the surge state determination module includes: The deviation determination submodule is used to determine the deviation between the perceived data of each of the home devices and the corresponding predicted data; A surge threshold determination submodule is used to determine the surge threshold for each of the aforementioned home appliances; The surge state determination submodule is used to determine that the home device is in a surge state when the deviation of the home device exceeds the corresponding surge threshold.

[0132] In some embodiments, the counterfactual causal inference module includes: The intervention sample construction submodule is used to construct intervention samples using the deviation between the perceived data and the predicted data as the outcome variable; The attribution result determination submodule is used to input the intervention sample into a preset counterfactual causal inference model, and obtain a preset historical control sample set through the counterfactual causal inference model; the historical control samples include historical outcome variables, which are composed of the deviation between the historical perceived data and the predicted data; from the historical control sample set, a preset number of historical control samples that meet the similarity characteristic conditions with the intervention sample are selected; based on the outcome variables of the intervention sample and the historical outcome variables of the preset number of historical control samples, a control result estimate is determined; based on the control result estimate, the attribution result is determined.

[0133] In some embodiments, the attribution result determination submodule includes: The comparison result interval determination unit is used to determine the interval in which the estimated value of the comparison result lies; The attribution result determination unit is used to determine the attribution result corresponding to the interval in which the estimated value of the control result is located, based on a preset mapping relationship between the interval and the attribution result.

[0134] In some embodiments, the attribution results include systemic causes and external causes; The comparison result interval determination unit includes: The system cause determination subunit is used to determine the attribution result as a system cause when the estimated value of the control result is in the first interval. The external cause determination subunit is used to determine the attribution result as an external cause when the estimated value of the control result is in the second interval; the value in the first interval is greater than the value in the second interval; The yield control strategy determination module includes: The system cause strategy submodule is used to determine the target home device that has performed automated actions within the current time window when the attribution result is the system cause, and to pause the execution of the automation rules of the target home device. The external cause strategy submodule is used to pause the execution of automation rules for preset home device types when the attribution result is the external cause.

[0135] In some embodiments, the sensed data includes state change data, audio data, and network traffic data.

[0136] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0137] This invention also provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described home system control method based on collaborative perception and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0138] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0139] This invention also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described home system control method based on collaborative perception and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0140] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

[0147] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0148] The foregoing has provided a detailed description of a home system control method, device, electronic device, and computer-readable storage medium based on collaborative perception provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A home system control method based on collaborative perception, characterized in that, The method includes: In executing automation rules for home scenarios, the perception data of home devices related to the automation rules are obtained; Generate predictive data for each of the aforementioned home devices in the current home scenario; Based on the perceived data and the predicted data, determine whether each home appliance is in a surge state, and count the number of home appliances in the surge state; When the number of home devices in the surge state meets the triggering condition, counterfactual causal inference is performed using the deviation between the perceived data and the predicted data as the result variable to determine the attribution result; Based on the attribution results, a yield control strategy corresponding to the attribution results is determined; the yield control strategy is used to pause the automation rules of the home system.

2. The home system control method based on collaborative perception according to claim 1, characterized in that, The generation of prediction data for each of the aforementioned home devices in the current scenario includes: Determine the current scene type; Input the current scene type into the behavior intent layer of the hierarchical generation model to obtain the predicted distribution of the user's acceptance probability of the automated actions of home devices; Using the predicted acceptance probability distribution as a condition, the predicted distribution corresponding to the perceived data of the home appliances is obtained through the multimodal observation layer in the hierarchical generation model, and is used as the predicted data of the home appliances in the current scenario.

3. The home system control method based on collaborative perception according to claim 2, characterized in that, Determining the current scene type includes: Obtain the hidden state of the scene at the previous time step stored in the scene context layer of the hierarchical generation model; Obtain current time characteristics; The hidden state of the scene at the previous time step and the current time feature are input into the scene transition probability matrix in the scene context layer, and the current scene type is obtained by operating on the scene transition probability matrix.

4. The home system control method based on collaborative perception according to claim 1, characterized in that, The step of determining whether each home appliance is in a surge state based on the sensed data and the predicted data includes: Determine the deviation between the perceived data of each of the aforementioned home devices and the corresponding predicted data; Determine the surge threshold for each of the aforementioned home appliances; When the deviation of the home appliance exceeds the corresponding surge threshold, the home appliance is determined to be in a surge state.

5. The home system control method based on collaborative perception according to claim 1, characterized in that, The step of using the deviation between the perceived data and the predicted data as the outcome variable to perform counterfactual causal inference to determine the attribution result includes: Using the deviation between the perceived data and the predicted data as the outcome variable, an intervention sample is constructed; The intervention sample is input into a preset counterfactual causal inference model, and a preset historical control sample set is obtained through the counterfactual causal inference model. The historical control sample includes historical outcome variables, which are composed of the deviation between the historical perceived data and the predicted data. From the historical control sample set, a preset number of historical control samples that meet the similarity characteristic conditions with the intervention sample are selected. Based on the outcome variables of the intervention sample and the historical outcome variables of the preset number of historical control samples, the estimated value of the control result is determined. Based on the estimated value of the control result, the attribution result is determined.

6. The home system control method based on collaborative perception according to claim 5, characterized in that, The step of determining the attribution result based on the estimated value of the control result includes: Determine the interval in which the estimated value of the control result lies; Based on the preset mapping relationship between the interval and the attribution result, the attribution result corresponding to the interval in which the estimated value of the control result is located is determined.

7. The home system control method based on collaborative perception according to claim 5, characterized in that, The attribution results include systemic causes and external causes; The determination of the attribution result corresponding to the interval in which the estimated control result falls includes: When the estimated value of the control result is in the first interval, the attribution result is determined to be a systematic cause. When the estimated value of the control result is in the second interval, the attribution result is determined to be an external cause; the value in the first interval is greater than the value in the second interval; The step of executing a yield control strategy corresponding to the attribution result includes: When the attribution result is the system cause, identify the target home device that has performed automated actions within the current time window and suspend the execution of the automation rules for the target home device. When the attribution result is the external cause, the execution of the automation rules for the preset home device type is suspended.

8. The home system control method based on collaborative perception according to claim 1, characterized in that, The sensing data includes state change data, audio data, and network traffic data.

9. A home system control device based on collaborative perception, characterized in that, The device includes: The perception data acquisition module is used to acquire perception data of home devices related to the automation rules in the execution of home scene automation rules; The prediction data generation module is used to generate prediction data for each of the aforementioned home devices in the current home scenario; The surge state determination module is used to determine whether each home device is in a surge state based on the sensed data and the predicted data, and to count the number of home devices in the surge state. The counterfactual causal inference module is used to perform counterfactual causal inference to determine the attribution result when the number of home devices in the surge state meets the triggering condition, using the deviation between the perceived data and the predicted data as the result variable. The yield control strategy determination module is used to determine a yield control strategy corresponding to the attribution result based on the attribution result; the yield control strategy is used to pause the automation rules of the home system.

10. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the home system control method based on collaborative perception as described in any one of claims 1-8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the home system control method based on collaborative perception as described in any one of claims 1-8.

Images