A device control method and apparatus, an electronic device, and a readable storage medium

Abstract

The application discloses a device control method and device, an electronic device and a readable storage medium. The method comprises the following steps: acquiring motion data in a preset area, analyzing the motion data based on human motion characteristics through a first model to obtain a first prediction result; receiving a second prediction result and a third prediction result sent by a cloud server, and performing consistency verification on the first prediction result, the second prediction result and the third prediction result to obtain a verification result; the second prediction result and the third prediction result are prediction results of the motion data obtained by a second model and a third model deployed in the cloud server respectively; and the running state of the device is controlled according to the verification result, so that the accuracy of device control is improved.

Description

Technical Field

[0001] This application belongs to the field of smart home, specifically relating to a device control method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of smart home technology, smart home systems have played an important role in improving users' quality of life and energy efficiency, and smart control devices have provided users with a better living experience.

[0003] Typically, control devices operate by processing sensor data using models and outputting results or control commands. However, current models are prone to misinterpreting moving, non-human objects as human presence, leading to erroneous automatic control of devices like air conditioners and wasting energy. Accuracy is also difficult to guarantee in complex scenarios. Inaccuracy issues are particularly pronounced when sensor data quality fluctuates or environmental conditions change. Furthermore, the control system cannot detect erroneous control actions autonomously, nor can it provide users with decision-making support or explanations, resulting in a degraded user experience. Summary of the Invention

[0004] The purpose of this application is to provide a device control method, apparatus, electronic device, and readable storage medium to improve the accuracy of device control.

[0005] In a first aspect, embodiments of this application provide a device control method applied to a local gateway, the method comprising: Acquire motion data within a preset area, and analyze the motion data based on human motion characteristics using a first model to obtain a first prediction result; The system receives a second prediction result and a third prediction result sent from a cloud server, and performs a consistency check on the first prediction result, the second prediction result, and the third prediction result to obtain a check result; the second prediction result and the third prediction result are prediction results of the second model and the third model deployed on the cloud server for the motion data, respectively. Based on the verification results, control the operating status of the equipment.

[0006] Secondly, embodiments of this application provide a device control method applied to a cloud server, the method comprising: The second model identifies motion data and historical environmental data within a preset area and outputs a second prediction result. A third prediction result is output by performing time-series analysis on the motion data and the historical environmental data using a third model. The second prediction result and the third prediction result are sent to the local gateway so that the local gateway can perform consistency verification on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

[0007] Thirdly, embodiments of this application provide a device control apparatus applied to a local gateway, the apparatus comprising: The first prediction module is used to acquire motion data within a preset area, and analyze the motion data based on human motion characteristics using a first model to obtain a first prediction result. The verification module is used to receive the second prediction result and the third prediction result sent by the cloud server, and to perform consistency verification on the first prediction result, the second prediction result and the third prediction result to obtain the verification result; the second prediction result and the third prediction result are the prediction results of the second model and the third model deployed on the cloud server for the motion data, respectively. The equipment control module is used to control the operating status of the equipment based on the verification results.

[0008] Fourthly, embodiments of this application provide a device control apparatus applied to a cloud server, the apparatus comprising: The second prediction module is used to identify motion data and historical environmental data within a preset area using the second model, and output the second prediction result. The third prediction module is used to perform time-series analysis on the motion data and the historical environmental data through a third model, and output a third prediction result. The result sending module is used to send the second prediction result and the third prediction result to the local gateway, so that the local gateway can perform consistency verification on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

[0009] Fifthly, embodiments of this application provide an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the steps of the method as described in the first or second aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the steps of the method described in the first or second aspect.

[0011] In this embodiment, the local gateway analyzes motion data based on human motion characteristics using a first model to obtain a first prediction result. Simultaneously, it receives a second and a third prediction result from the cloud server and performs a consistency check on the three prediction results. Based on the check result, it controls the device. Since the first, second, and third models process the same motion data based on different inference logics, if any model misjudges a specific scenario, its misjudgment result is unlikely to be consistent with the prediction results of the other two models. Consistency check effectively suppresses erroneous control caused by misjudgment from a single model, thereby improving the accuracy of device control. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the steps of an embodiment of the device control method of the present invention; Figure 2 This is a flowchart illustrating the steps of another embodiment of the device control method of the present invention; Figure 3 This is a flowchart illustrating the steps of another embodiment of the device control method of the present invention; Figure 4 This is a flowchart illustrating the steps of another embodiment of the device control method of the present invention; Figure 5 This is a flowchart illustrating the steps of an embodiment of the device control method of the present invention; Figure 6 This is a structural block diagram of a device control apparatus according to the present invention; Figure 7 This is a structural block diagram of a device control apparatus according to the present invention; Figure 8 This is a structural block diagram of an electronic device provided by the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0017] Method Implementation Examples Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a device control method according to the present invention, applied to a local gateway. The method includes: Step 101: Obtain motion data within a preset area, and analyze the motion data based on human motion characteristics using a first model to obtain a first prediction result; Step 102: Receive the second and third prediction results sent by the cloud server, and perform a consistency check on the first, second, and third prediction results to obtain the check result; the second and third prediction results are prediction results of the second and third models deployed on the cloud server for the motion data, respectively. Step 103: Control the operating status of the equipment based on the verification results.

[0018] Regarding steps 101 to 103, the device control method provided in this embodiment of the invention can be applied to a smart home system. Specifically, the method can be applied to a local gateway device in a smart home system, which performs multi-source sensing and verification of the presence of human beings in the environment through the local gateway, thereby realizing the automated control of home devices such as air conditioners, lighting, and security systems.

[0019] The preset area refers to the detection coverage range of the sensors under the jurisdiction of the local gateway. These sensors can be millimeter-wave radar, infrared sensors, etc. The coverage range can be configured according to the actual application scenario. For example, in a smart home scenario, the preset area can be the interior space of a room, or a specific area such as an entrance hallway or corridor. The boundary and shape of the preset area depend on parameters such as the sensor's installation location, detection angle, and detection distance, and can be set according to actual needs.

[0020] Motion data refers to information collected by sensors deployed within a predetermined area, reflecting the motion state of objects within that area. Specifically, motion data can include velocity, distance, and angle information of moving targets, such as point cloud sequences output by millimeter-wave radar. Motion data can also include pyroelectric signal sequences output by infrared sensors, or motion vector information from video frame sequences captured by cameras.

[0021] The first model is a machine learning model deployed on a local gateway, also known as an edge model, used for real-time inference on motion data. Essentially, it's a lightweight binary classification model that determines whether input motion data corresponds to the presence of a human body. For example, it employs a lightweight neural network architecture like PointNet. Specifically, PointNet aggregates features from all points in the point cloud using symmetric functions, achieving input order invariance. PointNet has a simple structure and few parameters, making it suitable for deployment on local gateways with limited computing resources. For instance, the PointNet structure includes an input layer, a shared multilayer perceptron (MLP), a max-pooling layer, and a fully connected layer. The shared MLP performs independent feature transformations on each point in the point cloud using the same set of weights. The max-pooling layer maximizes the feature matrix along each point's dimension, and the global feature vector is input to the fully connected network, outputting the final classification result.

[0022] It should be noted that the first model has learned and stored the parameters of human motion features during the training phase, and directly judges the input motion data based on the pre-trained parameters during the inference phase.

[0023] Human motion characteristics refer to quantifiable indicators used to distinguish human motion from non-human motion. Non-human motion includes things like flying insects, pets moving, and curtains swaying. Human motion characteristics include, but are not limited to: a reasonable range of motion speed (e.g., human walking speed is typically between 0.5 m / s and 2 m / s); the smoothness of the motion trajectory (e.g., the continuity of the human motion trajectory); the range of the target's volume / cross-sectional area (e.g., the radar cross-sectional area corresponding to a human being falls within a specific range); and the finiteness of acceleration (e.g., human motion is subject to physical constraints, and acceleration has an upper limit). These human motion characteristics are pre-established and stored in the first model through statistical analysis of a large number of human motion samples, forming the basis for the first model's binary classification decision. During inference, the first model compares the input motion data with the aforementioned range thresholds of human motion characteristics to determine whether the motion data corresponds to the presence of a human being.

[0024] The first prediction result is the preliminary judgment of the first model on whether a human body exists within the preset area. Specifically, the first prediction result includes a binary label and its corresponding confidence value, such as "person" or "no one". For example, when the millimeter-wave radar detects a fast-moving small target, the first model, after analyzing the human motion characteristics, may output a prediction result of "no one", with a confidence level of 0.92.

[0025] The second model, also known as the main model, is a machine learning model deployed on a cloud server. Unlike the first model, the second model is a large-scale multimodal model that processes data including not only motion data but also historical environmental data. The second model performs comprehensive identification based on both motion and historical environmental data, outputting a second prediction result. The second model has a larger model structure and more parameters than the first model, enabling it to handle more complex scenarios.

[0026] The third model is a machine learning model deployed on a cloud server, also known as an auxiliary model. It performs time-series analysis and logical judgment on the state changes of historical environmental data within a preset time window, outputting a third prediction result. The reasoning logic of the third model is based on preset rule conditions, and its model structure and complexity differ from the first and second models. The second, third, and first models each employ different reasoning logics to independently reason on the input data.

[0027] The second prediction result is the output of the second model, predicting whether a human body exists within the preset area. The third prediction result is the output of the third model, also predicting whether a human body exists within the preset area. Similar to the first prediction result, both the second and third prediction results include corresponding judgment labels and confidence information. The three prediction results originate from three independent models, each based on different reasoning logic.

[0028] It should be noted that, in addition to receiving the second and third prediction results from the cloud server for consistency verification, the local gateway can also send the first prediction result to the cloud server for consistency verification. Consistency verification refers to the process by which the local gateway comprehensively compares the first, second, and third prediction results. The core of the verification is to determine whether the three prediction results point to the same conclusion and whether the confidence level of each prediction result meets a preset confidence threshold. For example, consistency verification can compare whether the judgment labels of the three prediction results are consistent; or it can be based on confidence-weighted voting, where the verification is considered successful when the total confidence level of a conclusion exceeds a preset threshold; when there are inconsistencies among the three prediction results, the confidence level differences of the inconsistent results are further compared. If the confidence level of one result is significantly higher than the others and exceeds the preset threshold, it can be used as an arbitration basis. Because the three models use different inference logic—the first model performs real-time classification based on human motion characteristics, the second model performs comprehensive identification based on motion data and historical environmental data, and the third model makes judgments based on the temporal logic of historical environmental data—misjudgments produced by a single model under specific interference scenarios are difficult to be simultaneously corroborated by other models. Consistency checks can identify and suppress misjudgments by a single model.

[0029] The verification result is the output of the consistency verification process, used to indicate the degree of consistency among the three prediction results. For example, the verification result includes two states: consistent and inconsistent. Consistent means that the first, second, and third prediction results all point to the same conclusion. Inconsistent means that at least one prediction result differs from the other prediction results. In this case, the system needs to decide whether to trigger the human-machine collaboration process or adopt a conservative control strategy, such as not performing any operation, based on the preset security policy.

[0030] A device refers to a smart home device controlled by a local gateway. For example, a device can be an air conditioner, but it can also include lighting equipment, door locks, security alarms, curtain motors, air purifiers, humidifiers, etc. The device has controllable operating states, such as on / off, temperature setting adjustment, fan speed switching, locking / unlocking, etc. The local gateway changes the device's operating state by sending control commands to it. The device and the local gateway are connected via wired or wireless communication methods, such as wireless networks (Wireless Fidelity, Wi-Fi), Bluetooth, etc.

[0031] Operating status refers to the current working mode or parameter configuration of the equipment. Taking air conditioning equipment as an example, operating status can include: cooling mode, heating mode, air supply mode, dehumidification mode, set temperature value, fan speed level, swing angle, on / off status, etc.

[0032] The local gateway generates corresponding control commands based on the verification results. These commands instruct the device to switch to the target operating state. For example, if the verification results show that all three predictions indicate someone is present, the local gateway generates a command to turn on the air conditioner, switching it from the off state to the running state. If the verification results show that all three predictions indicate no one is present, the local gateway generates a command to turn off the air conditioner, switching it from the running state to the off state.

[0033] For example, a local gateway is deployed in a user's living room, where a millimeter-wave radar sensor is installed to collect motion data within the living room. A second and third model are deployed on a cloud server. The local gateway deploys the first model.

[0034] The local gateway collects motion data in the living room in real time using millimeter-wave radar. Assuming a flying insect quickly flies across the radar's detection area, the radar collects a point cloud sequence containing speed, distance, and angle information as motion data. The local gateway inputs this motion data into a first model, a lightweight binary classification model pre-trained with numerous human motion samples (including breathing, walking, and waving) and non-human motion samples (such as flying insects, pet movement, and curtain swaying). The first model learns and stores human motion characteristics. Based on these characteristics, the first model analyzes the input motion data, extracting motion features such as target speed, trajectory smoothness, and target volume, and comparing them with threshold values ​​for human motion characteristics. The flying insect's speed far exceeds normal human speed, and its trajectory fluctuates significantly, failing to meet human motion characteristics. Therefore, the first model outputs a prediction of "no one," with a confidence level of 0.92.

[0035] The local gateway receives the second and third prediction results from the cloud server. The second prediction result is generated by a cloud-deployed second model, which comprehensively identifies the same motion data and historical environmental data, including information such as the current temperature, humidity, and time in the living room. Because the second model is a large-scale multimodal model, its training samples may lack negative samples of small object movement. Therefore, the second model may misclassify a fast-moving insect as someone, outputting a second prediction result of "someone," with a confidence level of 0.78. The third prediction result is generated by a cloud-deployed third model, which analyzes the temporal state data from historical environmental data, such as whether infrared sensors detected body temperature signals or whether air conditioning was turned on within the past 5 minutes. If the third model detects that infrared sensors did not detect body temperature signals and that the air conditioning was not turned on within the past 5 minutes, and based on preset criteria of no body temperature signal and no continuous movement trajectory, it determines the area to be empty, outputting a third prediction result of "no one," with a confidence level of 0.88.

[0036] The local gateway obtains the first prediction result as "no one" (0.92), the second prediction result as "someone" (0.78), and the third prediction result as "no one" (0.88), and then performs a consistency check on the three. The check finds that the first and third prediction results are both "no one," while the second prediction result is "someone," indicating inconsistency.

[0037] Based on the verification results, the local gateway controls the operating status of the air conditioning equipment according to a preset security policy. For example, during high-energy-consuming operations such as turning on the air conditioner, to avoid energy waste caused by misjudgment by a single model leading to accidental air conditioner activation, the local gateway will not execute the automatic air conditioner activation command when the verification results are inconsistent, but will instead keep the air conditioner in its current off state. The local gateway can also further trigger a human-machine collaboration process, allowing the user to ultimately decide whether to turn on the air conditioner. When the verification results are consistent, the local gateway generates a control command to turn on the air conditioner, switching it from the off state to cooling or heating mode and adjusting the indoor temperature to the preset temperature value.

[0038] In this embodiment, the local gateway analyzes motion data based on human motion characteristics using a first model to obtain a first prediction result. Simultaneously, it receives a second and a third prediction result from the cloud server and performs a consistency check on the three prediction results. Based on the check result, it controls the device. Since the first, second, and third models process the same motion data based on different inference logics, if any model misjudges a specific scenario, its misjudgment result is unlikely to be consistent with the prediction results of the other two models. Consistency check effectively suppresses erroneous control caused by misjudgment from a single model, thereby improving the accuracy of device control.

[0039] Reference Figure 2 The diagram illustrates a flowchart of an embodiment of a device control method according to the present invention, applied to a local gateway. The method includes: Step 201: Obtain motion data within the preset area.

[0040] Step 201 can be referred to step 101 above, and will not be repeated here.

[0041] Step 202: Preprocess the motion data; Step 203: Extract the first motion feature from the preprocessed motion data; Step 204: Compare the first motion feature with the human motion feature to obtain the first prediction result; the first prediction result is used to indicate whether the target corresponding to the motion data is a human body.

[0042] Regarding steps 202 to 204, it should be noted that the raw point cloud data collected by sensors such as millimeter-wave radar contains a large amount of noise and redundant information, such as environmental clutter, false alarm points generated by multipath reflections, and interference signals from non-target sources. The purpose of preprocessing is to remove interference, extract the effective target point cloud, and provide high-quality data for feature extraction.

[0043] For example, preprocessing includes data filtering, noise removal, target clustering, etc.

[0044] Data filtering is used to remove obviously invalid or low-quality data points. For example, it removes points with a signal-to-noise ratio below a preset threshold, points located outside the detection area boundary, and points that do not conform to physical laws. For instance, by setting signal-to-noise ratio and detection range thresholds, noisy points can be effectively filtered out from the original point cloud.

[0045] Noise removal is used to eliminate outliers and isolated points in point cloud data. For example, a neighborhood-based statistical method can be used. For each point in the point cloud, the number of neighboring points within a preset radius is calculated. If the number of neighboring points is less than a preset threshold, the point is identified as an outlier and removed.

[0046] After filtering and denoising, the point cloud may contain multiple targets. Target clustering is used to divide the point cloud into different clusters, each corresponding to a potential target. Clustering algorithms can employ density-based clustering methods, such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN), which clusters points based on spatial distance and velocity similarity, grouping points that are spatially adjacent and have consistent motion states into the same cluster, with different clusters corresponding to different targets. For example, in a flying insect scenario, preprocessing filters and denoises the point cloud sequence acquired by radar, removing discrete noise points caused by the small size of the flying insect and its weak radar cross-section, while also removing environmental clutter, retaining the effective flying insect target point cloud, providing clean data for subsequent motion feature extraction.

[0047] The first motion feature is a quantitative indicator extracted from the preprocessed motion data to describe the target's motion state. This first motion feature is compared with pre-stored human motion features in the first model to determine if the target is a human. The first motion feature can include dimensions such as the object's acceleration, motion structure, and characteristics. For example, the first motion feature may include at least one of average speed, trajectory smoothness, target area, and acceleration. Average speed refers to the average speed of the target's motion over a period of time. Human speeds vary depending on the activity state: walking speed is typically between 0.5 m / s and 2 m / s, running speed is between 3 m / s and 6 m / s, and the micro-movement speed of the chest cavity during breathing is extremely low, approximately 0.01 m / s to 0.05 m / s. The flight speed of flying insects is usually much higher than human walking speed; for example, the flight speed of flies can reach 2 m / s to 7 m / s, accompanied by high-frequency vibrations. The movement speed of pets such as cats and dogs also differs from that of humans. By calculating the target's average speed and comparing it with a human speed range threshold, a preliminary determination can be made as to whether the target is a human.

[0048] Trajectory smoothness describes the continuity and regularity of a target's motion trajectory. Human motion, driven by the musculoskeletal system, exhibits high smoothness and continuity, without physically insignificant abrupt changes or reversals. In contrast, the motion of flying insects displays high-frequency oscillations, sudden turns, and irregular reversals, resulting in a much lower trajectory smoothness than that of humans. For example, trajectory smoothness can be quantified by calculating the rate of change of curvature, the derivative of acceleration, or the angle of directional change between trajectory points. Higher smoothness increases the likelihood that the target is a human.

[0049] Acceleration refers to the rate of change of a target's velocity over time. Human movement is constrained by physics, and there is an upper limit to acceleration. The acceleration change of a normal human walking is usually no more than 2 m / s². The acceleration changes of flying insects due to wing flapping are more dramatic and frequent. The rate of change of human acceleration is limited by human motor capacity, and acceleration is finite in short periods. By calculating the acceleration change of a target and comparing it with the human acceleration range threshold, it is possible to help determine whether the target is a human.

[0050] The first model has learned and stored human motion features during the training phase. Human motion features refer to the reasonable range of various motion indicators obtained through statistical analysis of a large number of human motion samples. The first motion feature is the actual motion indicator value extracted from the currently input motion data, while the human motion feature is a discrimination benchmark pre-stored in the first model, used to compare with the first motion feature to determine whether the current target conforms to the laws of human motion.

[0051] For example, the speed of a flying insect is much greater than that of a human, its trajectory fluctuates wildly, and its target volume is much smaller than that of a human. None of the aforementioned first motion features meet the threshold range for human motion features. Therefore, the first model determines that the current trajectory does not conform to human biological characteristics and outputs a prediction result indicating that no one is present. The first prediction result includes not only a classification label but also the corresponding confidence score. The confidence score can be calculated based on the degree of matching between the first motion feature and the human motion feature. For example, the more matching indicators and the smaller the deviation of each indicator from the threshold, the higher the confidence score.

[0052] It should be noted that the first model can also learn and adjust online based on user feedback during operation, such as when a user confirms the absence of a tagged user via their mobile phone. When the accumulated calibration samples reach a certain number, the first model can fine-tune its parameters locally, further enhancing its adaptability to specific scenarios, such as the unique lighting conditions and spatial layout of the living room.

[0053] In this embodiment, the first model first preprocesses the motion data to remove environmental clutter and outlier interference, ensuring that subsequent feature extraction is based on valid data. Then, it extracts motion features including average velocity, trajectory smoothness, and target cross-sectional area. Finally, it compares the extracted motion features with pre-stored human motion features. This effectively improves the accuracy of local inference and provides reliable local prediction results for subsequent consistency checks.

[0054] Step 205: Receive the second and third prediction results sent by the cloud server, and perform consistency verification on the first, second, and third prediction results to obtain the verification result.

[0055] Step 205 can be referred to step 102 above, and will not be repeated here.

[0056] Step 206: If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are consistent, control the operating status of the device according to any one of the prediction results; Step 207: If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent, control the operating status of the device according to the user's instructions.

[0057] For steps 206 and 207, the consistency check refers to the process of comparing the classification labels of the three prediction results, such as "someone" or "no one", and making a judgment based on the confidence level of each prediction result.

[0058] Consistency verification results can be either consistent or inconsistent. Consistency verification can be implemented using a pre-defined consistency verification algorithm. This algorithm must at least include comparing whether the classification labels of the three predicted results point to the same conclusion, and may also include determining whether the confidence level of each predicted result reaches a pre-defined confidence threshold. Consistency verification can also employ a weighted voting method. For example, the confidence levels of each model's output can be used as weights to calculate a weighted total score for the "person" and "no one" categories. If the weighted total score for a category exceeds a pre-defined threshold, the verification is considered successful.

[0059] When the verification results indicate that the first, second, and third prediction results are consistent, the local gateway controls the device's operating status based on any one of the prediction results. Consistency means that the classification labels of the first, second, and third prediction results are exactly the same, i.e., all three indicate someone is present or all three indicate no one is present. In addition to consistent classification labels, verification can be further determined based on the confidence level of each prediction result reaching a preset confidence threshold. Only when the classification labels are the same and each confidence level is not lower than the preset threshold is the verification result considered consistent. For example, if all three prediction results indicate someone is present, and their respective confidence levels are 0.85, 0.82, and 0.90, all higher than the preset confidence threshold of 0.70, then the verification result is consistent.

[0060] When the three predictions are consistent, it indicates that the three independent models have reached a consensus on the current environment, and this conclusion has high reliability. Therefore, the local gateway can directly control the device's operating status based on this consensus conclusion without user intervention. For example, in a flying insect scenario, the first prediction is "no one is here" (confidence 0.92), the second prediction is "someone is here" (confidence 0.78), and the third prediction is "no one is here" (confidence 0.88). Since the three predictions are inconsistent, the local gateway controls the device's operating status according to user instructions.

[0061] Inconsistency refers to a situation where at least one of the three predicted results differs from the classification labels of the other two. For example, two predictions might indicate "occupied" and one "unoccupied," or one prediction might indicate "occupied" and the other two "unoccupied," or all three might be different. For instance, the main model might predict "occupied" with a confidence level of 0.78, while the marginal model predicts "unoccupied" with a confidence level of 0.92, and the auxiliary model predicts "unoccupied" with a confidence level of 0.88, creating a low-confidence conflict state where one model is inconsistent with the other two. A low-confidence conflict state occurs when the three predicted results diverge, and the confidence level of a single model is insufficient to support the system's automatic execution of equipment control commands due to a lack of validation from other models. In a low-confidence conflict state, the system does not execute control commands but instead triggers a human-machine collaborative process, controlling the equipment's operating status according to user instructions.

[0062] It should be noted that, based on preset security policies, different processing rules can be applied to different devices in low-confidence conflict states. For high-energy-consuming devices such as air conditioners and electric heaters, automatic turn-on commands will not be executed in conflict states to avoid energy waste. However, if the device is currently running but the conflict determines that no one is present, a decision on whether to turn it off will be made based on user instructions or a conservative strategy. For high-security devices such as door locks and security alarms, no automatic commands will be executed in conflict states; user confirmation is required before execution to ensure security. For low-security devices such as lighting equipment, default security policies, such as maintaining the current state, can be used in conflict states, or control can be executed according to more flexible rules. Preset security policies can be configured by system defaults or customized by users according to their personal preferences.

[0063] In this embodiment, when the three prediction results are consistent, the system automatically executes the control command without user intervention, ensuring the convenience and responsiveness of smart homes. When the three prediction results are inconsistent, the system does not execute the automatic command but triggers a human-machine collaborative process for the user to make the final decision, avoiding erroneous operations caused by misjudgment. This ensures that the system can respond quickly in normal scenarios and has a safety backup in abnormal scenarios, balancing efficiency and security.

[0064] Optionally, step 207 may specifically include the following sub-steps: Step 2071: If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent, generate a structured decision request and push it to the user terminal. Step 2072: Receive the decision instruction returned by the user terminal in response to the decision request; Step 2073: Control the operating status of the equipment according to the decision instructions.

[0065] For steps 2071 to 2073, the decision request is information sent by the system to the user to obtain decision instructions when it cannot make a judgment independently. For example, a structured decision request includes a description of the current environmental state and at least two candidate control options for the controlled device. The description informs the user of the current environmental conditions, helping them understand why the system needs their intervention in the decision-making process, such as the presence of a mobile signal. The candidate control options provide the user with a range of selectable decisions. These options include a first option and a second option; the user selects one to instruct the system to perform the corresponding control operation. Taking air conditioning control as an example, two quick options are provided: A. Turn on the air conditioner and B. Keep it off. These two options correspond to the two most core operations in device control; the user only needs to click to complete the decision without manually inputting instructions. When the controlled device is a door lock, the candidate control options can be A. Unlock the door lock and B. Keep it locked. When the controlled device is a lighting device, the candidate control options can be A. Turn on the lights and B. Keep them off.

[0066] Structured processing refers to decision requests having a predefined format, containing fixed information fields, enabling the system to automatically generate and push these requests, and to parse the user's returned selection results. Push processing refers to the system sending structured decision requests to the user's pre-bound terminal device via a push messaging service, such as a user's mobile phone, which maintains communication connections with a local gateway and a cloud server. After the system pushes the decision request, the user's terminal device will receive a notification in real time, allowing the user to view and respond immediately.

[0067] A decision instruction is a control directive returned by the user terminal to the system after the user selects a candidate control option. There is a one-to-one correspondence between decision instructions and candidate control options; whichever option the user selects, the decision instruction instructs the system to execute the corresponding control operation. For example, after seeing a push notification on their phone, the user determines it might be a flying insect or a curtain fluttering, and clicks option B. Keep it off. The user terminal then returns the decision instruction "Keep it off" to the system.

[0068] It should be noted that when a decision request is sent by the local gateway, the decision instruction is returned from the user terminal to the local gateway, which then directly controls the device based on the decision instruction. When a decision request is sent by the cloud server, the decision instruction is returned from the user terminal to the cloud server, which then sends the control instruction to the local gateway for execution.

[0069] After receiving the user's decision command, the local gateway converts it into a device-executable control command and sends it to the controlled device through the device communication interface, causing the device to switch to the target operating state. For example, if the user clicks "B. Keep Off," the system immediately executes the command not to turn on the air conditioner; that is, the local gateway either does not send an on command to the air conditioner or sends a command to keep it off, and the air conditioner remains off. The system executes the corresponding control operation directly upon receiving the user's decision command without additional verification or delay, ensuring a rapid response to the user's decision.

[0070] It should be noted that the mapping between decision commands and control commands is pre-stored in the local gateway. For example, keeping it off corresponds to not sending an on command or sending an off command, while turning on the air conditioner corresponds to sending an on command and setting the temperature to 26℃ in cooling mode. The local gateway converts the user's decision commands into low-level control commands that the air conditioning device can parse, based on the pre-stored mapping. Alternatively, the decision command itself may be an abstract representation of the device control commands, allowing the local gateway to directly control the device based on the decision command without conversion.

[0071] In this embodiment, the system sends structured decision requests to the user terminal via push notifications, allowing users to receive notifications and respond in real time regardless of their location. Users can make control decisions regarding the air conditioner without returning to the living room, improving the convenience and flexibility of the interaction. The system provides users with a description of the current environmental state, enabling them to understand why the system requires their intervention and make judgments based on their observations of the environment. After the user makes a decision, the system immediately executes the corresponding control operation, ensuring rapid response and execution of the user's decision and enhancing the interactive experience.

[0072] In summary, the device control method provided in this application involves a local gateway performing real-time inference locally using a first model, while simultaneously receiving prediction results from a second and third model in the cloud. Each of the three models independently judges the same scenario based on different input data and inference logic. The local gateway performs consistency verification on the three prediction results. When any model misjudges a specific interference scenario, its misjudgment conclusion is unlikely to be consistent with the prediction results of the other two models. Through consistency verification, the system can effectively identify and suppress misjudgments by a single model, avoiding erroneous device control caused by misjudgments by a single model.

[0073] Reference Figure 3 The diagram illustrates a flowchart of an embodiment of a device control method according to the present invention, applied to a cloud server. The method includes: Step 301: Identify the motion data and historical environmental data within the preset area using the second model, and output the second prediction result; Step 302: Perform time-series analysis on the motion data and the historical environmental data using the third model, and output the third prediction result; Step 303: Send the second prediction result and the third prediction result to the local gateway so that the local gateway can perform consistency verification on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

[0074] Regarding steps 301 to 303, it should be noted that the device control method provided in this application embodiment can be applied to a smart home control system. This method can be applied to a cloud server, using multiple models deployed in the cloud to comprehensively perceive and identify motion data and historical environmental data in the environment, and then sending the identification results to a local gateway. The local gateway performs multi-source consistency verification to achieve automated control of home devices. It should also be noted that, in addition to the above method, the cloud server can also receive the first prediction result sent by the local gateway and perform consistency verification thereon.

[0075] The second model is a machine learning model deployed on a cloud server, used for comprehensive identification based on motion data and historical environmental data, outputting a second prediction result. This second model is a large-scale multimodal model, with a larger model structure and more parameters than the first model deployed on the local gateway. The second model can simultaneously process two different types of data, motion data and historical environmental data, and perform comprehensive perception and identification of both. For example, the second model, a large-scale vision-radar multimodal model, can employ a multimodal Transformer-based fusion identification model. This model's structure includes a point cloud encoder, an environmental encoder, a cross-modal fusion module, and a classification output layer.

[0076] The first model performs real-time inference based on motion data, while the second model integrates motion data and historical environmental data for comprehensive identification. The second model focuses on the comprehensive judgment of the spatial characteristics of moving targets and environmental context, while the third model focuses on the changing patterns of environmental states over time.

[0077] The preset area refers to the detection coverage range of the sensor, which can be configured according to the actual application scenario. The motion data acquired by the cloud server comes from the sensor data collection results within this preset area. The cloud server receives motion data forwarded from the local gateway or directly uploaded by the sensor through the communication network.

[0078] Motion data refers to information collected by sensors deployed within a preset area, reflecting the motion state of objects within that area. The motion data acquired by the cloud server and the motion data processed by the local gateway's first model come from the same data source, namely, the same data collected by the same sensor within the same preset area.

[0079] Historical environmental data refers to records of the environmental state within a preset area over a period of time prior to the current moment, including environmental state information at multiple points in time. Historical environmental data may include at least one of the following: indoor temperature data, indoor humidity data, infrared sensor detection data, equipment on / off status data, door lock status data, and window status data.

[0080] Motion data reflects the presence and movement status of a moving target, while historical environmental data reflects the changes in the environment over a past period. The second model uses both types of data for comprehensive identification, checking both the presence of movement and whether the environmental context supports the conclusion that someone is present.

[0081] The third model is a machine learning model deployed on a cloud server, used to perform time-series analysis on motion data and historical environmental data, outputting a third prediction result. The third model is a lightweight time-series analysis model or rule engine. Time-series analysis refers to the process by which the third model analyzes the changing patterns of historical environmental data over time. The third model focuses on the changing patterns of environmental states over time, such as whether the infrared sensor has consistently failed to detect body temperature signals or whether the air conditioning has been turned on in the past 5 minutes. The third model employs different inference logic from the first and second models. The third model can make logical judgments based on preset rule conditions. For example, if the infrared sensor has not detected body temperature signals and the air conditioning has no record of being turned on within a preset time window, it is determined that no one is present. The third model can also be a time-series classification model trained based on machine learning, learning the correlation between environmental state sequences and human presence through a large amount of historical data. For example, the third model can adopt a hybrid time-series judgment model based on a rule engine and a lightweight Long Short-Term Memory (LSTM) network. The structure of this model can include a time-series data input layer, a single-layer or double-layer LSTM network, a conditional judgment module, and an output layer.

[0082] The cloud server sends the second and third prediction results to the local gateway in the form of data packets via the internet. Upon receiving these two prediction results, the local gateway performs a consistency check with the first prediction result output by its local first model. The first prediction result is obtained by the local gateway's first model based on motion data analysis. The first model performs real-time inference on motion data based on human motion characteristics and outputs a judgment result indicating whether a human body exists within a preset area.

[0083] The second and third models perform independent reasoning on the same or related input data based on their respective inference logics. The second model performs comprehensive identification based on motion data and historical environmental data, focusing on spatial motion characteristics and environmental context. The third model performs time-series analysis based on historical environmental data, focusing on state patterns within a time window. After the cloud server sends the prediction results of the two models to the local gateway, the local gateway verifies them in conjunction with the prediction results of the first model, thus realizing a multi-source perception architecture with independent reasoning and unified verification of the three models.

[0084] In this embodiment, the second and third models are deployed on a cloud server, utilizing the powerful computing resources of the cloud for inference. The second model is a multimodal large-scale model capable of processing large-scale point cloud data and multi-dimensional historical environmental data, extracting rich spatial motion features and environmental context features for comprehensive identification; the third model is capable of performing time-series analysis on historical environmental data spanning long periods. Cloud deployment allows the second and third models to be free from the limitations of local gateway computing and storage resources, enabling the use of more complex algorithms and larger model structures, thereby improving the accuracy of identification.

[0085] Reference Figure 4 The diagram illustrates a flowchart of an embodiment of a device control method according to the present invention, applied to a cloud server. The method includes: Step 401: Extract the second motion feature from the motion data using the second model; Step 402: Based on the environmental context information provided by the historical environmental data, classify the second motion feature to obtain the second prediction result.

[0086] Regarding steps 401 and 402, the second motion feature is a multi-dimensional quantitative indicator extracted by the second model from the motion data, used to describe the motion state and spatial distribution of the target within a preset area. Compared with the first motion feature extracted by the first model, the second motion feature has richer dimensions and a wider coverage. The second motion feature extracted by the second model not only includes the motion parameters of a single target, but also includes richer spatial and environmental dimension information, such as: the distribution characteristics of point clouds in space, the position of the target in the room, and the spatial relationship between the target and objects such as furniture.

[0087] The second model processes the overall radar point cloud, meaning it focuses on the overall distribution of the point cloud within the entire detection area, rather than just a single moving target. The first model extracts primary motion features focusing on the kinematic parameters of a single target, such as velocity, trajectory smoothness, and acceleration, for comparison with human motion characteristics. In contrast, the second model extracts secondary motion features focusing more on the overall spatial picture, including the point cloud distribution within the entire preset area, the target's spatial location, and the target's relative relationship to its environment.

[0088] For example, the second motion feature may include at least one of point cloud spatial distribution features, target spatial location features, target motion trajectory spatial pattern, point cloud contour / shape features, and multi-target association features. The point cloud spatial distribution features describe the overall distribution of the point cloud within a preset area. The target spatial location features describe the specific location of the moving target within the preset area and its relative relationship with other objects in the scene. The target motion trajectory spatial pattern describes the motion path features of the target in three-dimensional space. The point cloud contour / shape features describe the shape and structure of the point cloud cluster corresponding to the moving target. The point cloud contour of a human body typically has specific shape features, while the point cloud contour of a flying insect exhibits a small-area, discrete distribution. The multi-target association features describe the spatial relationships and motion coordination between multiple moving targets within the preset area. For example, whether multiple targets maintain a relatively fixed distance, or whether multiple targets move independently.

[0089] For example, the second model has a larger number of parameters and a greater network depth than the first model, thus enabling it to extract richer and more abstract motion features. The second model automatically extracts second motion features from the input motion data through its neural network structure's feature extraction layer. The second model is a multimodal large model, containing multiple layers of convolutional or attention mechanisms, capable of extracting features at different levels of abstraction from the original point cloud data layer by layer. The bottom layer extracts local geometric features such as the spatial coordinates and velocity of points; the middle layer extracts target-level features such as target contours and motion trajectories; and the top layer extracts scene-level features such as spatial layout and the relationship between the target and the scene.

[0090] Environmental context information refers to background information provided by historical environmental data to help determine whether a human body exists within a preset area. For example, this could include actions such as the activation or control of household devices, changes in indoor temperature, or unlocking of doors within the past 10 minutes. It can also include variables from the historical environment, such as temperature and humidity, window and door lock / open status, weather data, and device execution data.

[0091] The second model fuses the second motion feature with environmental context information and then outputs a classifier's judgment on whether a human body exists within a preset region. For example, the fused feature vector is input into a fully connected classification network, which outputs a two-dimensional probability distribution, corresponding to the probability values ​​of "person" and "no one." The second prediction result is the category with the higher probability value, and the corresponding confidence level is also output.

[0092] In this embodiment, the second motion features extracted from the motion data by the second model include multi-dimensional information such as point cloud spatial distribution features, target spatial location features, spatial patterns of motion trajectories, and point cloud contour / shape features, reflecting the complete spatial state of the target within a preset area. The rich spatial features enable the second model to gain a more comprehensive understanding of the scene, improving recognition accuracy. The second model classifies the second motion features based on environmental context information provided by historical environmental data. The introduction of environmental context information means that the second model's recognition is no longer isolated spatial feature matching, but rather a comprehensive judgment combined with the environmental background, thus improving recognition accuracy.

[0093] Optionally, the historical environmental data includes at least one of the following: temperature data, humidity data, sensor detection data, equipment usage record data, weather data, window open / close status, and door lock open / close status.

[0094] It should be noted that historical environmental data refers to environmental status information collected and recorded within a preset area over a period of time prior to the current moment. Historical environmental data reflects environmental changes and equipment usage in the preset area over a past period of time.

[0095] Historical environmental data encompasses multiple dimensions, such as environmental physical parameters, equipment status parameters, environmental configuration parameters, and external environmental parameters. Environmental physical parameters may include temperature and humidity; equipment status parameters may include equipment usage records and on / off status; environmental configuration parameters may include window and door lock status; and external environmental parameters may include weather data. Historical environmental data has a clear temporal sequence characteristic, meaning it is all related to time and constitutes a state sequence that changes over time. The third model is based on the temporal characteristics of the data for analysis and judgment.

[0096] Temperature data refers to the ambient temperature values ​​within a preset area and their changes over time. Temperature data can be collected by temperature sensors, typically recorded at a fixed sampling frequency, forming a sequence of temperature changes over time. As indirect evidence of human presence, temperature data can be useful in air conditioning control scenarios. When the air conditioner is off, the indoor temperature may remain stable during periods of inactivity, but may show a slow upward trend during periods of human presence. Therefore, temperature change trends can serve as an auxiliary reference for determining the presence of human beings within a preset area.

[0097] Humidity data refers to the environmental humidity value within a preset area and its changes over time. Humidity data can be collected by humidity sensors and is usually recorded synchronously with temperature data to form a sequence of humidity changes over time. Humidity is an important environmental factor affecting human comfort. When the humidity is too high (above 70%) or too low (below 30%), people will feel uncomfortable and may be more inclined to turn on the air conditioner or adjust its operating mode. Humidity data provides contextual reference for the second model to understand human behavior patterns in the current environment.

[0098] Sensor detection data refers to detection signals collected by various sensors within a preset area that directly or indirectly reflect the presence of a human body. In one embodiment, sensor detection data includes body temperature detection signals from an infrared sensor. The infrared sensor can detect thermal radiation of a specific wavelength emitted by the human body. When a human body is present in the preset area, the infrared sensor outputs a detection result indicating the presence of a body temperature signal; when no human body is present in the area, it outputs a detection result indicating the absence of a body temperature signal.

[0099] Sensor detection data exhibits a clear temporal sequence, meaning its detection results change over time, forming a sequence of states with and without a signal. Sensor detection data can include at least one of the following: body temperature detection signals from infrared sensors, detection signals from millimeter-wave radar, and detection signals from passive infrared sensors. During the comprehensive identification process, the second model can combine the body temperature detection signals from infrared sensors to verify its identification results. If the second model identifies a person based on motion data, but the infrared sensor continuously outputs no body temperature signal, this contradictory information will reduce the confidence of the second model in its identification result.

[0100] Device usage record data refers to the operation and running records of various smart devices within a preset area, including the time points when devices are started, turned off, or their operating status is adjusted, as well as information such as the type of operation. Device usage record data has a clear temporal sequence characteristic; that is, each record contains a timestamp to record the specific moment when an operation or status change occurred. The third model infers whether anyone is present in the area by analyzing whether device usage records exist within a preset time window.

[0101] Weather data refers to outdoor weather conditions at the location of the preset area, including but not limited to outdoor temperature, weather type, wind speed, and air quality index. Weather data can be obtained from meteorological service providers via network interfaces or collected from outdoor weather stations. Weather data provides the second model with external environmental context information, helping the model understand whether the environmental conditions within the preset area are reasonable. For example, in hot summers when outdoor temperatures exceed 35°C, the likelihood of indoor activity increases because people tend to stay indoors to avoid the heat. On rainy days, people are more likely to stay indoors rather than go out, increasing the probability of human presence within the preset area. Under extreme weather conditions such as storms or high-temperature warnings, the probability of indoor activity is significantly higher than in ordinary weather. By combining weather data with comprehensive identification, the second model can improve the accuracy of prediction results.

[0102] Window open / closed status refers to information about whether a window within a preset area is currently open or closed, and a record of how this status changes over time. Window open / closed status can be obtained through door and window magnetic sensors or intelligent window control systems. Window open / closed status reflects the degree of isolation between the preset area and the external environment. When windows are open, the indoor environment is significantly affected by the external environment, such as outdoor temperature affecting indoor temperature, outdoor noise affecting the indoor acoustic environment, and the possibility of outdoor insects entering the room.

[0103] Door lock status refers to the information of whether the entrance door to the preset area is currently locked or unlocked, and a record of how this status changes over time. Door lock status can be obtained through the smart door lock system, including information such as locked / unlocked status, unlocking method, and unlocking time. The third model can infer whether someone has entered or left the area based on changes in the door lock status within a preset time window, such as whether there are any door unlocking records in the past 5 minutes. A door lock switching from locked to unlocked and opening the door usually means someone has entered the preset area; a door lock switching from unlocked to locked usually means someone has left the preset area. Various historical environmental data play different roles in the reasoning of the second and third models, collectively forming a complete environmental perception system.

[0104] In this embodiment, historical environmental data encompasses multi-dimensional data, providing rich data support and enhancing the system's comprehensive perception of environmental conditions. Different types of sensor data complement and cross-validate each other. Various types of historical environmental data collectively support the system's decision-making from different perspectives, enabling the model to reason and make judgments based on more complete contextual information. Step 403: Obtain the state sequence of the historical environmental data within a preset time window through the third model; Step 404: Determine whether the state sequence meets the preset conditions and obtain the determination result; Step 405: Based on the judgment result and the motion data, output the third prediction result.

[0105] For steps 403 to 405, the preset time window refers to the historical time period that the third model focuses on when performing time series analysis. The length of this time window can be preset according to the actual application scenario and specific judgment requirements. For example, the preset time window is 5 minutes before the current moment. In the past 5 minutes, the infrared sensor did not detect a body temperature signal, and the automatic start record of the air conditioner was empty.

[0106] A state sequence refers to a time-series record of historical environmental data within a preset time window. A state sequence contains multiple snapshots of the environmental state at different times, each snapshot recording the value or state of various environmental parameters at that moment. A state sequence can include at least one of the following: infrared sensor detection signal sequence, device on / off state sequence, temperature change sequence, and door lock state sequence. The device on / off state sequence is the sequence of on / off states of devices such as air conditioners and lights at various times within the preset time window. The temperature change sequence is the sequence of temperature values ​​collected by temperature sensors at various times within the preset time window. The door lock state sequence is the sequence of locked / unlocked states of door locks at various times within the preset time window.

[0107] The third model considers both the infrared sensor detection signal and the continuous movement trajectory characteristics in the motion data. For example, the third model directly judges based on the inter-frame changes in motion data within a preset time window. Specifically, the third model detects whether there is a stable moving target between consecutive frames in the motion data; if the moving target appears and disappears after appearing in a few frames, or if the moving target exhibits a discontinuous movement pattern in the time series, it is judged as having no continuous movement trajectory. For example, when an insect flies quickly across the radar detection area, although there is a motion signal in the motion data, its duration is extremely short and its trajectory is discontinuous in time. The third model detects that the motion signal fails to form a continuous movement trajectory within the preset time window, and therefore judges it as having no continuous movement trajectory.

[0108] Preset conditions refer to a set of judgment rules pre-stored by the third model, used to determine whether the state sequence indicates that there is no one or someone in the preset area. These preset conditions can be pre-stored or pre-trained on the third model using a dedicated training set so that the third model learns these judgment rules.

[0109] For example, a preset condition is that within a preset time window, the infrared sensor continuously fails to detect a body temperature signal, and the air conditioning equipment has no record of being turned on. When the state sequence meets the above preset conditions, the third model determines that there is no one in the preset area. The preset conditions may also include auxiliary conditions such as: within the preset time window, there is no record of the door lock being unlocked, and the temperature change tends to be stable, to enhance the accuracy of the determination.

[0110] The third model's judgment relies on the persistence and / or change of a state within a preset time window. Specifically, the third model focuses on whether a certain state persists or does not persist within the preset time window, rather than the instantaneous state at a certain moment. For example, the third model's judgment that the infrared sensor has not detected a body temperature signal is not based on the detection result at a certain moment, but on the time series state that the infrared sensor has not detected a body temperature signal for the past 5 minutes.

[0111] When the judgment result meets the preset conditions and there is no continuous movement trajectory in the motion data: the third model outputs a prediction result indicating that no one is present. When the judgment result meets the preset conditions but there is a continuous movement trajectory in the motion data: the third model may output a prediction result indicating that someone is present.

[0112] In this embodiment, the third model focuses on the changing patterns of the environmental state within a preset time window, enabling it to analyze the environmental state from a temporal perspective, thus complementing the spatial analysis of the second model. Both models assess the same scene from different dimensions, enhancing the comprehensiveness and robustness of the system's overall perception.

[0113] Step 406: Send the second prediction result and the third prediction result to the local gateway so that the local gateway can perform consistency verification on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

[0114] Step 406 can be referred to step 303 above, and will not be repeated here.

[0115] In summary, the device control method provided in this application allows the cloud server to independently analyze the scene using a second model and a third model from different dimensions. The two models employ completely different inference logics. When one model misjudges a specific interference scenario, the other model, due to its different inference dimensions, is less likely to make a simultaneous error, ensuring the validity and reliability of the two prediction results sent to the local gateway. The second model is a large multimodal model with a large number of parameters and complex calculations, while the third model requires time-series analysis of historical environmental data over a long period. Both require significant computing and storage resources. The cloud server provides elastic and scalable computing resources, enabling the second and third models to use more complex model structures and richer input data for inference, thus improving the accuracy of recognition.

[0116] Reference Figure 5 The diagram illustrates a flowchart of an embodiment of a device control method according to the present invention, the method comprising: Step A11: Multimodal data input and independent reasoning of the three-level model; Step A12: Decision Difference Detection and Conflict Arbitration; Step A13: Structured conflict push and user interaction; Step A14: Feedback loop and incremental model learning.

[0117] For steps A11 to A14, multimodal data refers to data of various types, and independent inference means that the three models process the input data based on different inference logics, without interfering with or affecting each other. The three models run in parallel and each outputs a prediction result. This heterogeneous and redundant architecture design allows subsequent consistency checks to be compared between the three independent judgments, in order to identify and suppress misjudgments by a single model.

[0118] In the three-level independent inference model, the first model refers to the edge model deployed on the local gateway. This first model is a lightweight binary classification model that analyzes motion data based on human motion characteristics and outputs a first prediction result. The second model is a large-scale multimodal model deployed on a cloud server. It performs comprehensive identification based on motion data and historical environmental data and outputs a second prediction result. The third model is deployed on a cloud server and performs logical judgments based on the temporal state of historical environmental data within a preset time window, outputting a third prediction result.

[0119] The inference processes of the three models are performed in parallel in time and are logically independent of each other. Each model makes judgments based on its own input data and inference logic, without referring to the output results of other models during the inference process.

[0120] The system performs consistency checks on the prediction results output by the three models and triggers the corresponding arbitration mechanism when a conflict is detected. After a conflict is detected, the system executes the corresponding arbitration method according to the preset security policy. According to the preset security policy, when high-energy-consuming or high-security-level operations such as air conditioning operation and door lock control are involved, no automatic commands are executed; instead, a human-machine collaborative process is triggered.

[0121] Upon detecting a conflict, the system generates a structured decision request and pushes it to the user terminal, followed by an interaction where the user returns a decision command via the terminal. The system sends the structured decision request to the user's pre-bound terminal device via a message push service. After receiving the user's decision command, the local gateway converts it into a device-executable control command and sends it to the controlled device via the device communication interface, causing the device to switch to the target operating state.

[0122] This application embodiment also includes storing user decisions and related data as training samples, and triggering incremental learning of the model after accumulating sufficient samples to achieve continuous optimization and adaptive evolution of the model. For example, the user's choice to keep the system off and its associated sensor data are stored in a cloud data pool as a high-confidence training sample. After accumulating a certain number of conflict samples, the system triggers an incremental learning task for the main model and the auxiliary model. Through fine-tuning, the main model strengthens its ability to distinguish between the movement of small objects and human movement, reducing the probability of misjudging flying insects by 40%. The auxiliary model and the edge model also benefit from this successful human-machine collaboration, with their confidence weights in similar scenarios being strengthened. By incorporating user feedback into the continuous training of the model, the system can learn from the user's corrective behavior, gradually reducing misjudgments in specific interference scenarios, and achieving self-evolution and performance improvement of the system.

[0123] For example, the second model can use a full-scale fine-tuning approach for incremental learning. Fine-tuning refers to performing a small number of iterative training sessions based on the pre-trained weights of the second model, using accumulated high-confidence training samples to adjust some parameters of the model, making it better adaptable to specific scenarios. The third model's update can be an adjustment of confidence weights. When the system confirms the prediction result of the third model through human-machine collaboration in a certain scenario, the system increases the confidence weight of the third model's prediction result or lowers its conflict-triggering threshold when encountering similar scenarios subsequently, enabling the third model to exert greater decision-making influence in similar scenarios. The first model can use a lightweight update approach. When the accumulated calibration samples reach a certain number, the first model can perform lightweight parameter fine-tuning locally, further enhancing its adaptability to specific scenarios such as the unique lighting conditions and spatial layout of the living room.

[0124] In this embodiment, a distributed heterogeneous redundancy architecture of local + cloud is achieved by deploying three models on a local gateway and a cloud server, respectively. Each of the three models employs different inference logic, independently judging the same scenario from different dimensions. Misjudgments by any one model are suppressed by the effective predictions of the other two models, resulting in significantly better overall decision-making accuracy than a single-model approach. The system automatically executes control when the three models reach a consensus and proactively requests user confirmation when there are disagreements, avoiding unnecessary user interference and preventing erroneous operations by the system under uncertain conditions. The system stores each user's correction feedback as a high-confidence training sample. After accumulating a certain number, it triggers incremental learning of the model, enabling the system to learn from user correction behavior, gradually reducing misjudgments of specific interference scenarios, achieving a positive cycle and continuous evolution where higher accuracy with more use.

[0125] Device Examples Reference Figure 6The diagram illustrates a logic block diagram of a device control apparatus according to an embodiment of the present invention. The apparatus may include: The first prediction module 501 is used to acquire motion data within a preset area, analyze the motion data based on human motion characteristics using a first model, and obtain a first prediction result. The verification module 502 is used to receive the second prediction result and the third prediction result sent by the cloud server, and to perform consistency verification on the first prediction result, the second prediction result and the third prediction result to obtain the verification result; the second prediction result and the third prediction result are the prediction results of the second model and the third model deployed on the cloud server for the motion data, respectively. The device control module 503 is used to control the operating status of the device based on the verification result.

[0126] Optionally, the first prediction module includes: The preprocessing module is used to preprocess the motion data; The first feature extraction module is used to extract the first motion features from the preprocessed motion data. The feature comparison module is used to compare the first motion feature with the human motion feature to obtain the first prediction result; the first prediction result is used to indicate whether the target corresponding to the motion data is a human body.

[0127] Optionally, the device control module includes: The first device control submodule is used to control the operating state of the device according to any one of the prediction results when the verification result indicates that the first prediction result, the second prediction result and the third prediction result are consistent. The second device control submodule is used to control the operating status of the device according to user instructions when the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent.

[0128] Optionally, the second device control submodule includes: A structured request sending module is used to generate a structured decision request and push it to the user terminal when the verification result indicates that the first prediction result, the second prediction result and the third prediction result are inconsistent. A decision instruction receiving module is used to receive a decision instruction returned by the user terminal in response to the decision request; The equipment control unit is used to control the operating status of the equipment according to the decision instructions.

[0129] In summary, the device control apparatus provided in this application allows a local gateway to perform real-time inference locally using a first model, while simultaneously receiving prediction results from a second and third model in the cloud. Each of the three models independently judges the same scenario based on different input data and inference logic. The local gateway performs consistency verification on the three prediction results. When any model misjudges a specific interference scenario, its misjudgment conclusion is unlikely to be consistent with the prediction results of the other two models simultaneously. Through consistency verification, the system can effectively identify and suppress misjudgments by a single model, avoiding erroneous device control caused by misjudgments by a single model.

[0130] Reference Figure 7 The diagram illustrates a logic block diagram of a device control apparatus according to an embodiment of the present invention. The apparatus may include: The second prediction module 601 is used to identify based on motion data and historical environmental data within a preset area using a second model, and output a second prediction result. The third prediction module 602 is used to perform time-series analysis on the motion data and the historical environmental data through a third model, and output a third prediction result. The result sending module 603 is used to send the second prediction result and the third prediction result to the local gateway, so that the local gateway performs consistency verification on the first prediction result, the second prediction result and the third prediction result, and controls the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

[0131] Optionally, the second prediction module includes: The second feature extraction module is used to extract a second motion feature from the motion data through the second model; The first prediction submodule is used to classify the second motion feature based on the environmental context information provided by the historical environmental data, and obtain the second prediction result.

[0132] Optionally, the third prediction module includes: The state sequence acquisition module is used to acquire the state sequence of the historical environmental data within a preset time window through the third model; The judgment module is used to determine whether the state sequence meets the preset conditions and obtain the judgment result; The second prediction submodule is used to output the third prediction result based on the judgment result and the motion data.

[0133] Optionally, the historical environmental data includes at least one of the following: temperature data, humidity data, sensor detection data, equipment usage record data, weather data, window open / close status, and door lock open / close status.

[0134] In summary, the device control apparatus provided in this application allows the cloud server to independently analyze the scene using a second model and a third model from different dimensions. The two models employ completely different inference logics. When one model misjudges a specific interference scenario, the other model, due to its different inference dimensions, is less likely to make a simultaneous error in its prediction, ensuring the validity and reliability of the two prediction results sent to the local gateway. The second model is a large multimodal model with a large number of parameters and complex calculations, while the third model requires time-series analysis of historical environmental data over a long period. Both require significant computing and storage resources. The cloud server provides elastic and scalable computing resources, enabling the second and third models to use more complex model structures and richer input data for inference, thus improving the accuracy of recognition.

[0135] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0136] Reference Figure 8 This is a structural block diagram of an electronic device for device control provided in an embodiment of this application. Figure 8 As shown, it includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method of the foregoing embodiments.

[0137] The processor can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable devices, transistor logic devices, hardware components, or any combination thereof. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0138] The memory may be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it may be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.

[0139] This application also provides a non-transitory computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods of the foregoing embodiments.

[0140] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0141] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0143] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0148] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0149] The above provides a detailed description of a device control method, apparatus, electronic device, and readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A device control method, characterized in that, Applied to a local gateway, the method includes: Acquire motion data within a preset area, and analyze the motion data based on human motion characteristics using a first model to obtain a first prediction result; The system receives a second prediction result and a third prediction result sent from a cloud server, and performs a consistency check on the first prediction result, the second prediction result, and the third prediction result to obtain a check result; the second prediction result and the third prediction result are prediction results of the second model and the third model deployed on the cloud server for the motion data, respectively. Based on the verification results, control the operating status of the equipment.

2. The method according to claim 1, characterized in that, The step of analyzing the motion data based on human motion characteristics using a first model to obtain a first prediction result for the target includes: The motion data is preprocessed; Extract the first motion feature from the preprocessed motion data; The first motion feature is compared with the human motion feature to obtain the first prediction result; the first prediction result is used to indicate whether the target corresponding to the motion data is a human body.

3. The method according to claim 1, characterized in that, The step of controlling the operating status of the device based on the verification result includes: If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are consistent, the operating status of the device shall be controlled according to any one of the prediction results. If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent, the operating status of the control device shall be controlled according to the user's instructions.

4. The method according to claim 1, characterized in that, The step of controlling the operating status of the device according to user instructions when the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent includes: If the verification result indicates that the first prediction result, the second prediction result, and the third prediction result are inconsistent, a structured decision request is generated and pushed to the user terminal. Receive the decision instruction returned by the user terminal in response to the decision request; The operating status of the equipment is controlled according to the decision instructions.

5. A device control method, characterized in that, Applied to a cloud server, the method includes: The second model identifies motion data and historical environmental data within a preset area and outputs a second prediction result. A third prediction result is output by performing time-series analysis on the motion data and the historical environmental data using a third model. The second prediction result and the third prediction result are sent to the local gateway so that the local gateway can perform a consistency check on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the check result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

6. The method according to claim 5, characterized in that, The second model identifies motion data and historical environmental data within a preset area and outputs a second prediction result, including: The second model is used to extract a second motion feature from the motion data; Based on the environmental context information provided by the historical environmental data, the second motion feature is classified to obtain the second prediction result.

7. The method according to claim 5, characterized in that, The step of performing time-series analysis on the motion data and the historical environmental data using a third model to output a third prediction result includes: The third model is used to obtain the state sequence of the historical environmental data within a preset time window; Determine whether the state sequence meets the preset conditions, and obtain the determination result; Based on the judgment result and the motion data, the third prediction result is output.

8. The method according to claim 5, characterized in that, The historical environmental data includes at least one of the following: temperature data, humidity data, sensor detection data, equipment usage record data, weather data, window open / close status, and door lock open / close status.

9. A device control system, characterized in that, The device, applied to a local gateway, includes: The first prediction module is used to acquire motion data within a preset area, and analyze the motion data based on human motion characteristics using a first model to obtain a first prediction result. The verification module is used to receive the second prediction result and the third prediction result sent by the cloud server, and to perform consistency verification on the first prediction result, the second prediction result and the third prediction result to obtain the verification result; the second prediction result and the third prediction result are the prediction results of the second model and the third model deployed on the cloud server for the motion data, respectively. The equipment control module is used to control the operating status of the equipment based on the verification results.

10. A device control apparatus, characterized in that, The device, used in a cloud server, includes: The second prediction module is used to identify motion data and historical environmental data within a preset area using the second model, and output the second prediction result. The third prediction module is used to perform time-series analysis on the motion data and the historical environmental data through a third model, and output a third prediction result. The result sending module is used to send the second prediction result and the third prediction result to the local gateway, so that the local gateway can perform consistency verification on the first prediction result, the second prediction result and the third prediction result, and control the operating status of the device according to the verification result. The first prediction result is obtained by the first model of the local gateway based on the motion data analysis.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 4 or the method as described in any one of claims 5 to 8.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as claimed in any one of claims 1 to 4 or the method as claimed in any one of claims 5 to 8.

Images