An adaptive intelligent internet of things execution method and system based on cooperation of smart glasses and agent system

Abstract

The application discloses an adaptive intelligent Internet of Things execution method and system based on cooperation of intelligent glasses and intelligent agent system. The method collects visual, audio, environmental and positioning multi-modal data through the intelligent glasses worn on the head of a user, and jointly collects room side sensing data with fixed auxiliary sensors deployed on the room side; the intelligent agent system performs scene recognition, spatial positioning, user voice instruction analysis, target device multi-factor weighted matching, device operation parameter calculation and control instruction generation; during device operation, off-site detection and gradual shutdown are performed based on the presence review channel of the fixed auxiliary sensors on the room side; and a user habit model is updated through a closed-loop feedback mechanism. The application effectively solves the problem that Internet of Things device control lacks spatial perception, behavior understanding and adaptive adjustment, and can improve target device matching accuracy, reduce false shutdown rate and reduce the number of user interactions.

Description

Technical Field

[0001] This invention relates to the technical field of collaborative control of artificial intelligence and the Internet of Things, and in particular to the technical field of an adaptive intelligent Internet of Things execution method and system based on the collaboration of smart glasses and intelligent agent system. Background Technology

[0002] With the rapid development of IoT technology, smart homes have become an important part of modern family life. Homes typically house various smart appliances such as air conditioners, fans, water heaters, washing machines, lights, and curtains. However, existing smart home control systems still suffer from the following prominent technical problems: First, device control lacks spatial awareness. When a user issues a voice command like "turn on the air conditioner," the existing system cannot automatically determine the user's current room location, often requiring the user to explicitly specify "turn on the bedroom air conditioner" or "turn on the living room air conditioner," increasing the user's interaction burden. While existing technologies offer near-field device binding solutions using smart speakers in fixed locations, this requires deploying independent control terminals in each room and cannot follow the user's movements. Second, device control lacks the ability to understand continuous user actions. For example, after a user puts clothes in the washing machine, adds detergent, and closes the door, the existing system still requires the user to manually start the washing machine or trigger it via voice command. The system cannot automatically infer the user's intention and trigger corresponding device operations by observing continuous user actions. Third, device operating parameters lack adaptive adjustment capabilities. Existing systems typically use fixed preset values ​​or manually set values ​​when controlling parameters such as air conditioner and water heater temperatures. They cannot automatically adjust operating parameters based on environmental factors such as current outdoor temperature, number of people indoors, and time of day, nor can they personalize adjustments based on users' long-term usage habits. Fourth, the equipment lacks intelligent off-site management capabilities. Users frequently forget to turn off air conditioners, lights, and other equipment after leaving the room, causing unnecessary energy waste. Existing timed shutdown schemes are too rigid and cannot distinguish between short-term (e.g., going to the bathroom) and long-term (e.g., going out) departures, easily leading to accidental or delayed shutdowns. Furthermore, relying solely on wearable devices for on-site detection means that since the devices leave the room with the user, on-site verification of the original room is impossible. Existing solutions primarily using fixed sensors lack precise spatial perception capabilities that allow for movement with the user, making it difficult to organically integrate these two types of solutions within the same system. Fifth, equipment linkage decisions lack external environmental awareness. For example, whether to start the drying program after washing should be decided comprehensively based on external factors such as current weather conditions and electricity prices. Existing systems typically only support user-preset fixed linkage rules and lack the ability to automatically adjust the device's working mode based on the real-time external environment.

[0003] As an example of prior art closely related to the technical field of this application, U.S. Patent Application US2024 / 0094822A1 (Applicant: Snap Inc., Publication Date: March 21, 2024; subsequently granted and published as Patent No. US12,468,398, Publication Date: November 11, 2025, entitled "AR glasses as IoT remote control") discloses a solution for using smart glasses with augmented reality capabilities as remote controls for IoT devices. The core of this solution lies in: utilizing the combined tracking capabilities of the smart glasses' built-in six-degrees-of-freedom (6DoF) inertial measurement unit and camera, tracking the position of the target display screen in the real-world coordinate system based on the principle of raycasting, mapping the intersection of the user's gaze and the screen to the cursor position on the screen; and then recognizing the user's clicks, selections, and other gestures through a hand and head gesture recognition module, using the user's input through gaze and gestures as control commands for IoT-enabled devices such as smart TVs and computer monitors. However, the existing technical solution has the following corresponding shortcomings in relation to the technical problem to be solved in this application: (1) The solution only covers the single-device interaction scenario where "the user has directly looked at the target device", and does not solve the problem of how to automatically match the target device based on the user's spatial location and usage habits when there are multiple similar devices in the space where the user is located (e.g., multiple rooms have air conditioners) and the user's current gaze may not be directed at any specific device; (2) The solution uses cursor plus gesture as the main interaction channel, requiring the user to actively look at the target device and complete the precise gesture, and does not cover the scenario where precise gesture control is achieved by combining natural voice commands with spatial perception; (3) The solution does not involve the decision-making process of on-site verification and automatic shutdown of the devices still running in the original room after the user leaves the original room, nor does it provide the source of on-site verification data after the smart glasses leave the original room with the user, making it difficult to solve the problem of continuing to make departure judgments in the original room after the wearable perception leaves the site; (4) The solution does not introduce a device working mode decision-making mechanism driven by real-time data such as external weather and electricity prices, nor does it provide a closed-loop learning mechanism for online adaptive updating of matching weights and preference parameters through explicit and behavioral feedback from the user.

[0004] Therefore, there is an urgent need for a technical solution that can deeply integrate wearable sensing devices with intelligent agent decision-making systems to achieve precise device matching based on spatial perception, automatic task triggering based on behavior understanding, adaptive parameter calculation based on multi-source information, and off-site management based on room-side auxiliary sensors, thereby improving the reliability and control accuracy of IoT home systems. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the prior art by proposing an adaptive intelligent Internet of Things (IoT) execution method and system based on the collaboration of smart glasses and intelligent agent systems. This solves the technical problems of lack of spatial perception, continuous behavior understanding, parameter adaptation, and accurate departure management in IoT device control.

[0006] To achieve the above objectives, an adaptive intelligent IoT execution method based on the collaboration of smart glasses and an intelligent agent system is provided, applied to a system comprising smart glasses, an intelligent agent decision-making terminal, a protocol adaptation gateway, and multiple IoT devices; the IoT devices are deployed in a space comprising at least two rooms; each room is further equipped with fixed auxiliary sensors, the fixed auxiliary sensors including at least one or more of a fixed camera, an infrared thermal sensor, and an ambient audio sensor; the method includes: Step S1, Multimodal Perception Data Acquisition: Through the visual sensing module, audio sensing module, environmental sensing module, and positioning sensing module integrated on the smart glasses worn on the user's head, visual image data, environmental audio data, temperature, humidity, light data, and spatial location data of the user's space are collected simultaneously; at the same time, room-side perception data is collected through fixed auxiliary sensors deployed in each room, and the room-side perception data includes at least one of the following: room video data collected by a fixed camera, infrared heat source data collected by an infrared heat source sensor, and room environmental audio data collected by an environmental audio sensor; Step S2, Scene Recognition and Spatial Positioning: Input the visual image data into a pre-trained scene classification model to identify room type. At the same time, combine the spatial location data to determine the user's current location coordinates in the home space topology map through an indoor positioning algorithm, and query the list of registered IoT devices in the area where the location coordinates are located. Step S3, User voice command parsing: Perform voice activity detection and voice recognition on the environmental audio data, and extract the device type slot, action type slot and parameter slot from the user voice command; Step S4, Multi-factor weighted matching of target devices: Based on the current region device list obtained in step S2 and the device type slots obtained in step S3, a multi-factor weighted matching algorithm is used to calculate the matching score S_match(d) for each candidate device: S_match(d) = w1·f_dist(d) + w2·f_sem(d) + w3·f_habit(d) + w4·f_state(d) + w5·f_time(d) Where f_dist(d) is the spatial distance factor, f_sem(d) is the semantic matching factor, f_habit(d) is the user habit factor, f_state(d) is the device current state factor, and f_time(d) is the time period preference factor; w1 to w5 are weight coefficients and w1+w2+w3+w4+w5=1; the candidate device with the highest S_match(d) is selected as the target device; when the highest S_match(d) is lower than the preset matching confidence threshold S_th, the AR display module of the smart glasses requests manual confirmation from the user; Step S5, Calculation of equipment operating parameters: Based on the current environmental sensor data and user habit model, determine the operating parameters of the target equipment according to the preset parameter calculation formula; for temperature control equipment, the operating parameters include the target temperature T_target calculated by the following formula: T_target = T_pref(k) + β·(T_out − T_ref) + γ·(N_person − 1)·ΔT_person Where T_pref(k) is the user's historical preferred temperature in time period k, T_out is the current outdoor temperature, T_ref is the reference outdoor temperature, β is the outdoor temperature compensation coefficient, N_person is the number of people in the room obtained by visual human detection, ΔT_person is the temperature compensation amount for each additional person, and γ is the personnel density influence coefficient. Step S6, Control Command Generation and Security Verification: A control command is generated based on the target device and the operating parameters calculated in step S5. This control command is then subjected to device capability boundary verification, device conflict detection, and security confirmation verification in sequence. The device capability boundary verification checks whether the command parameters are within the allowable operating range of the target device. Device conflict detection checks whether two or more devices from a pre-registered set of mutually exclusive devices are simultaneously enabled in the same room; high-priority devices are retained according to a pre-configured device priority list, while low-priority devices are disabled or have their startup delayed. Security confirmation verification checks whether the target device belongs to a pre-registered set of high-power devices or a set of security-sensitive devices; a confirmation request is sent to the user through the AR display module of the smart glasses, and the control command is only issued after receiving confirmation from the user. After successful verification, the control command is converted to a communication protocol format supported by the target device via a protocol adaptation gateway before being issued and executed. Step S7, Departure Detection and Automatic Shutdown Decision Based on Room-Side Sensors: While the target device is in operation, when the positioning sensor module detects that a user has moved from the current room to another room, it starts a buffer wait timer T_wait. During the countdown of T_wait, the presence of personnel is verified based on the fixed auxiliary sensors deployed in the original room. The presence verification calculates the probability of presence in the original room P_presence using the following fusion determination formula: P_presence = max(P_visual, P_audio, P_thermal) Where P_visual is the visual presence probability obtained by human detection from room video data collected by the fixed camera in the original room; P_audio is the audio presence probability obtained by voiceprint activity detection from room environmental audio data collected by the environmental audio sensor in the original room; P_thermal is the heat source presence probability obtained from infrared heat source data collected by the infrared heat source sensor in the original room; when P_presence is still lower than the preset presence threshold P_th when the T_wait countdown expires, a phased shutdown operation is performed: first, the target device's operating parameters are switched to the preset energy-saving parameters and then wait for an additional buffer time T_extra. If P_presence does not rise back to P_th during the T_extra period, the target device is completely shut down. Step S8, User Habit Learning and Strategy Update: Collect explicit and behavioral feedback from users regarding device control results, and update the parameter vector θ_t of the user habit model according to the following exponentially weighted moving average update formula: θ_t = λ·θ_{t−1} + (1 − λ)·θ_new Where λ is the forgetting factor, 0 < λ < 1; the parameters of the user habit model include the weight coefficients w1 to w5 in step S4, the historical preferred temperature T_pref(k) calculated in step S5, and the base value of the buffer waiting time T_wait in step S7.

[0007] Preferably, the indoor positioning algorithm in step S2 adopts a fusion positioning of UWB ultra-wideband positioning and WiFi fingerprint positioning: P_fused = α·P_uwb + (1 − α)·P_wifi Where P_uwb is the UWB positioning result, P_wifi is the WiFi fingerprint positioning result, and α is the dynamic fusion coefficient; when the UWB signal strength index RSSI_uwb is greater than the preset threshold RSSI_th, α takes the value of [0.7, 0.9]; when RSSI_uwb is less than or equal to RSSI_th, α takes the value of [0.3, 0.5].

[0008] Preferably, step S3 also includes referential resolution processing: when the user's voice command contains an indicator pronoun, the specific IoT device referred to by the indicator pronoun is determined by combining the user's head orientation information and gaze direction information in the visual image data; the method is to determine the gaze cone angle with the user as the vertex based on the head orientation and gaze direction, and sort the candidate devices located within the cone angle and whose spatial distance from the user does not exceed the preset maximum referential distance D_ref according to their visual proportion in the user's field of vision and gaze duration from high to low, and take the device ranked first as the referential target.

[0009] 4. Preferably, the spatial distance factor f_dist(d) in step S4 is calculated using the normalized inverse distance formula: f_dist(d) = 1 − dist(P_user, P_device(d)) / D_max Where P_user is the user's current location coordinates, P_device(d) is the registered location coordinates of candidate device d, dist(·) is the Euclidean distance function, and D_max is the maximum distance between the user and all candidate devices in the current room.

[0010] Preferably, the method further includes a behavior sequence recognition and automatic triggering step for linked tasks: the user's continuous operation behavior is detected sequentially using the visual sensing module of the smart glasses; upon detecting the first action A_1, the system enters a sequence monitoring state and starts a preset time window T_window; within T_window, for each detected action A_k (k=1,…,K), the current sequence integrity confidence C_seq is calculated using the following formula: C_seq = ∏_{k=1}^{K} P(A_k | A_1, A_2, …, A_{k−1}) Where P(A_k | A_1, …, A_{k−1}) is the conditional probability that the k-th action is correctly identified given that the first k−1 actions have been observed, and K is the total number of actions in the predefined behavior sequence; when K is equal to the total number of actions in the predefined behavior sequence and C_seq is greater than or equal to the preset sequence confidence threshold C_th, the behavior sequence is determined to be complete and the device linkage task associated with the behavior sequence is triggered; if the determination condition is not met after the T_window timeout, the sequence monitoring state is exited and no linkage task is triggered.

[0011] Preferably, the execution of the linkage task further includes an external environment perception and equipment operating mode decision-making step: after the linkage task is triggered, the intelligent agent decision-making terminal obtains current weather data and electricity price data through an external data interface; the operating mode decision is executed according to the following rules: When the precipitation probability R_rain in the weather data is greater than the preset precipitation threshold R_th, the working mode of the washing equipment will be set to washing and drying mode. When R_rain is less than or equal to R_th, and the current electricity price E_price in the electricity price data is higher than the preset electricity price threshold E_th, the operation of the washing equipment will be delayed until the current day when the electricity price is lower than E_th. When R_rain is less than or equal to R_th and E_price is less than or equal to E_th, the working mode of the washing equipment is set to washing mode only.

[0012] Preferably, in step S7, the buffer waiting time T_wait is adaptively adjusted based on the device type and user history behavior: T_wait = T_base · (1 + μ · P_return) Where T_base is the preset base buffer time related to the device type, P_return is the probability that a user will return to the room within the T_base duration after a brief departure in the historical data, and μ is the adjustment coefficient and μ∈[0.1, 1.0].

[0013] Preferably, in step S8, the forgetting factor λ is adaptively adjusted according to the number of times U_act the user has explicitly modified or rejected the control instructions generated by the system in the most recent N days of the user habit model: when U_act is greater than or equal to the preset activity threshold U_th, λ is [0.6, 0.8]; when U_act is less than U_th, λ is [0.85, 0.98].

[0014] Preferably, the method further includes an offline degradation control step: when the network connection interruption between the smart glasses and the intelligent agent decision-making terminal continues to exceed a preset network interruption threshold T_net, the smart glasses enable a locally cached copy of the home space topology map and a copy of the device status table. Based on the locally cached copies, steps S2 to S6 are executed in the edge preprocessing unit of the smart glasses, and control commands are directly sent to the target devices through the local communication protocol stack pre-configured by the smart glasses. The local communication protocol stack supports at least one of BLE and Zigbee. During the offline network configuration phase, the pairing credentials of each target device are pre-issued and securely stored by the intelligent agent decision-making terminal. After the network is restored, the smart glasses send the operation logs during the offline period back to the intelligent agent decision-making terminal to synchronously update the user habit model.

[0015] An adaptive intelligent IoT execution system based on the collaboration of smart glasses and an intelligent agent system, characterized in that it is used to execute the above-described method, the system comprising: The smart glasses' sensing end includes a visual sensing module, an audio sensing module, an environmental sensing module, a positioning sensing module, an edge preprocessing unit, and an AR feedback display module; Room-side fixed auxiliary sensors include one or more of the following: fixed cameras, infrared thermal sensors, and ambient audio sensors deployed in each room; The intelligent agent decision-making end includes a scene understanding engine, a user intent reasoning engine, an adaptive decision engine, a context memory, and an external data interface. The intelligent agent decision-making end is used to receive multimodal data collected by the intelligent glasses sensing end and room-side sensing data collected by the room-side fixed auxiliary sensor, perform scene recognition, user intent parsing, target device multi-factor weighted matching, operating parameter calculation, departure detection and linkage task decision-making, and generate control commands. The IoT device execution end includes a protocol adaptation gateway and the plurality of IoT devices. The protocol adaptation gateway is used to convert the control command into a communication protocol format supported by the target device and send it for execution. An adaptive feedback closed-loop module is used to collect device execution status feedback and user behavior feedback, and update the user habit model parameters in the context memory according to the exponentially weighted moving average update formula described in step S8 of claim 1.

[0016] Preferably, the context memory includes: a user habit model storage area for storing user device usage preference parameters and temperature control preference parameters under different time periods, seasons, and weather conditions; a home space topology map storage area for storing the type labels, spatial coordinate ranges, and location coordinates and device attribute information of registered IoT devices in each room; a device status table storage area for recording the current operating status, operating parameters, and cumulative runtime of each IoT device in real time; and a behavior sequence template library for storing predefined user behavior sequence templates and their corresponding linkage task configuration information.

[0017] Preferably, the protocol adapter gateway supports at least three of the communication protocols Zigbee, Z-Wave, MQTT, CoAP, and BLE, and encapsulates IoT devices with different protocols into standardized device control interfaces through a unified device abstraction layer.

[0018] An electronic device includes a processor and a memory, the memory storing computer-executable instructions, wherein the processor, when executing the computer-executable instructions, implements the method according to any one of claims 1 to 9.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

[0020] The beneficial effects of this invention are: The core of this invention lies in using wearable sensing that moves with the user and fixed auxiliary sensors on the room side as two parallel data sources. Both provide decision-making basis during the device matching stage, and the room side channel independently completes the presence verification after the user leaves the original room. Therefore, in complex scenarios with multiple rooms and multiple similar devices, the accuracy of target device matching and the reliability of departure detection are significantly improved compared with solutions that rely solely on voice and text or a single wearable sensing device. Correspondingly, it also reduces accidental shutdown and cross-room miscontrol.

[0021] At the interaction level, the joint determination of the behavior sequence integrity confidence C_seq and the time window T_window ensures that the linkage task will only be triggered when the action sequence is complete and C_seq is not lower than the threshold C_th. Under this condition, the false trigger rate is significantly lower than that of the single-frame action triggering method. In addition, the local cache topology map, the device status table copy, and the pre-configured local communication protocol stack constitute the degradation control path when the network is interrupted, so that the control will not be interrupted as a whole when the main link is abnormal.

[0022] For online learning, an exponentially weighted moving average parameter update strategy is adopted. The base values ​​of multi-factor matching weights, temperature preferences, and buffer waiting time can be continuously fine-tuned based on user feedback. Attached Figure Description

[0023] The above and other features, properties and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always denote the same features, wherein: Figure 1 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention; Figure 2 This is a main flowchart of the method provided in an embodiment of the present invention; Figure 3 A detailed flowchart of scene recognition and target device matching provided in embodiments of the present invention; Figure 4 A flowchart of departure detection and automatic shutdown decision-making based on room-side sensors provided in an embodiment of the present invention; Figure 5 This is a flowchart of behavior sequence recognition and linkage task triggering provided in an embodiment of the present invention; Figure 6 This is a flowchart illustrating the user habit learning and strategy update process provided in an embodiment of the present invention.

[0024] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] The present invention will be further described below with a set of specific embodiments. However, it should be understood that the following embodiments are merely specific examples of the inventive concept and are not intended to limit the scope of protection of the present invention. See below for further details. Figures 1 to 6 The following explanation is provided.

[0026] Let's start here. Figure 2 Let me make a brief explanation. Figure 2 This is a schematic diagram of the overall main flow of the method of the present invention, in which the method is divided into seven functional stages, S1 to S7, based on functional modules, to facilitate the reader's understanding of the method flow from an overall perspective. The correspondence between the functional stages and the specific method steps described in claim 1 is as follows: Figure 2 S1 (multimodal sensing data acquisition) in this context corresponds to step S1 in claim 1; Figure 2 S2 (scene recognition and spatial localization) in claim 1 corresponds to step S2 in claim 1; Figure 2 S3 (user intent parsing and target device matching) in claim 1 covers step S3 (user voice command parsing) and step S4 (target device multi-factor weighted matching). Figure 2 S4 (calculation of operating parameters and generation of control instructions) in claim 1 covers steps S5 (calculation of equipment operating parameters) and S6 (generation of control instructions and safety verification). Figure 2 The "whether the network interruption exceeds T_net" judgment branch and its "offline degradation control" branch correspond to the offline degradation control steps described in claim 9; Figure 2 S5 (departure detection and automatic shutdown decision) corresponds to step S7 in claim 1; Figure 2 S6 (behavior sequence recognition and linkage task triggering) in claim 5 corresponds to the behavior sequence recognition and linkage task automatic triggering step described in claim 5. Figure 2 S7 (user feedback learning and strategy update) corresponds to step S8 in claim 1. To avoid ambiguity, the rest of this specification refers to the method steps as S1 to S8 in claim 1. like Figure 1 and Figure 2 As shown, the sensing system in this embodiment consists of two types of data sources: (1) Smart glasses body perception. The smart glasses integrate a visual sensing module (a front-facing wide-angle RGB camera with a resolution of no less than 1080P and a frame rate of no less than 30fps; and a depth camera based on structured light or ToF), an audio sensing module (a microphone array consisting of at least two microphones, using beamforming technology for directional sound pickup), an environmental sensing module (temperature and humidity, light and IMU) and a positioning sensing module (UWB chip and WiFi / BLE receiver module).

[0027] The output signal of beamforming is: y(t) = Σ_{i=1}^{M} w_i·x_i(t − τ_i), where M is the number of microphones, w_i is the weighting coefficient of the i-th microphone, x_i(t) is the input signal of the i-th microphone, and τ_i is the delay compensation amount of the i-th microphone.

[0028] (2) Room-side fixed auxiliary sensors. In each room that needs to be managed by this system, one or more of the following fixed auxiliary sensors are pre-deployed: fixed cameras (e.g., wide-angle surveillance cameras on the ceiling or in the corners of the walls, with a resolution of not less than 1080P) to continue collecting room video for human detection after the user leaves the original room; infrared heat source sensors (e.g., passive infrared PIR or infrared thermal imaging array sensors) to detect whether there are still human heat sources in the original room; and environmental audio sensors (e.g., fixed microphone modules in the room) to detect human activity sounds such as speaking and footsteps. The fixed auxiliary sensors are connected to the agent decision-making end via Zigbee, WiFi, or wired network, and transmit the most recent P_visual, P_audio, and P_thermal calculation results back to the agent decision-making end in each sampling period; if only one of the above three types of sensors is deployed in a room, the presence probability of the undeployed channel is recorded as 0, and P_presence is still calculated by taking the maximum value.

[0029] The data collected by the smart glasses is first formatted, noise-filtered, and feature-compressed by a local edge preprocessing unit, and then transmitted to the intelligent agent decision-making end via high-speed wireless communication protocols such as WiFi 6E or Bluetooth 5.3. The intelligent agent decision-making end can be implemented on any intelligent agent platform with multimodal data processing, intent reasoning, and decision optimization capabilities. As an optional implementation, this embodiment uses the OpenClaw platform as the platform for the intelligent agent decision-making end; in fact, this invention does not depend on any specific intelligent agent platform or brand, and any platform with equivalent capabilities can be used as a substitute for this embodiment.

[0030] like Figure 3 As shown, the scene recognition and spatial positioning module receives visual image data and positioning data, and performs the following processing: (2a) Room type recognition. RGB images are input into a pre-trained lightweight convolutional neural network scene classification model, which can recognize common family room types such as bedroom, living room, kitchen, bathroom, study, and balcony. The model outputs the probability distribution of each room type, and a voting mechanism with a time window of N consecutive frames (e.g., N=10) is used to improve the robustness of recognition.

[0031] (2b) Precise location positioning. The time difference of arrival is measured using a UWB positioning chip and at least three UWB anchor points deployed in the home. The precise coordinates of the user are calculated using a trilateration algorithm and then fused with the WiFi fingerprint positioning results to obtain P_fused.

[0032] (2c) Area Device Query. Based on the mapping location of P_fused in the home space topology map, query the list of all registered IoT devices in that area. The home space topology map supports both manual configuration and semi-automatic scanning based on visual SLAM for construction.

[0033] (3a) The data collected by the audio sensing module is processed by voice activity detection, automatic speech recognition and natural language understanding to output a structured slot representation, including device type slot, action type slot and parameter slot.

[0034] (3b) When the user’s speech contains an indicative pronoun (e.g., “open this”), the user’s gaze cone angle is calculated by combining the head orientation given by the IMU and the gaze direction estimated by the depth camera and head pose. Candidate devices within the cone angle range and at a distance of no more than D_ref from the user are sorted by visual proportion and gaze duration, and the first-ranked device is taken as the referent.

[0035] (3c) In addition, the system also supports the timing detection of continuous user operation behavior through the vision sensing module of smart glasses. If an action sequence matching the predefined behavior sequence template is detected (e.g., "walking towards the washing machine → opening the door → putting clothes in → adding detergent → closing the door"), the behavior sequence recognition step is triggered (see Section 5 for details).

[0036] like Figure 3 As shown, after obtaining the current region's device list and voice command slots, the matching score for each candidate device is calculated using a multi-factor weighted matching formula: S_match(d) = w1·f_dist(d) + w2·f_sem(d) + w3·f_habit(d) + w4·f_state(d) + w5·f_time(d) The spatial distance factor f_dist(d) = 1 − dist(P_user, P_device(d)) / D_max; the semantic matching factor f_sem(d) is the semantic cosine similarity between the "user voice device type slot" and the "device type label" calculated by the pre-trained word vector model; the usage habit factor f_habit(d) is the historical usage frequency of the device in similar scenarios obtained from the user habit model; the device state factor f_state(d) is 1 for closed devices and 0 for open devices under the "on" command (inverted for "off" command), and 0 for faulty / maintenance devices; the time period preference factor f_time(d) is the historical probability of the user using the device in the current time period (morning / afternoon / evening / late night).

[0037] The initial values ​​of the weight coefficients w1 to w5 can be set empirically (e.g., w1=0.3, w2=0.3, w3=0.15, w4=0.15, w5=0.1) and adjusted online via S8. The device with the highest S_match(d) is selected as the target device; when the highest score is lower than the preset matching confidence threshold S_th (e.g., S_th=0.6), the AR display module requests manual confirmation from the user.

[0038] Taking air conditioning temperature control as an example, the target temperature is calculated using the following formula: T_target = T_pref(k) + β·(T_out − T_ref) + γ·(N_person − 1)·ΔT_person Where T_pref(k) is obtained by weighted average of the air conditioner setting temperature set by the user in the same time period over the past 30 days (recent data has higher weight than distant data); T_ref is the reference outdoor temperature (e.g., 25°C); β is the outdoor temperature compensation coefficient (e.g., 0.1); N_person is the number of people in the room currently detected by visual human figure detection; ΔT_person is the temperature compensation amount for each additional person (e.g., −0.5°C); and γ is the personnel density influence coefficient. For water heaters, the system predicts water demand based on the current time period and historical water usage habits and preheats the water in advance. For lighting equipment, the system automatically adjusts the brightness and color temperature based on light sensor data and user-preferred brightness.

[0039] The control command format is {device_id, action, parameters, priority, timestamp}, and three levels of verification are performed sequentially before being issued: Level 1 (Equipment Capability Boundary Verification): Checks whether the command parameters are within the target equipment's permissible operating range. For example, the air conditioning temperature command must be within the range of 16–30°C.

[0040] Level 2 (Device Conflict Detection): The system maintains a pre-registration table of "mutually exclusive device sets." The set members are provided by the product configuration; for example, (air conditioner, electric heater) is one mutually exclusive pair, and (humidifier, dehumidifier) ​​is another. When two or more devices from the same mutually exclusive pair are detected to be running simultaneously in the same room, the system retains the higher-priority device and shuts down or delays the start of the lower-priority device according to the pre-configured "device priority list." Example priority list: Safety alarms > Temperature control > Cleaning > Lighting.

[0041] Level 3 (Security Confirmation Verification): The system maintains two pre-registration forms: a "High-Power Device Set" and a "Safety-Sensitive Device Set." In this embodiment, the high-power device set includes, for example, electric water heaters, ovens, and ceramic cooktops; the safety-sensitive device set includes, for example, gas stoves, door locks, and security windows. For control commands to devices within these two sets, the system sends a confirmation request to the user via the AR display module, only issuing the command after receiving a voice "confirm" or a head nod of confirmation from the user.

[0042] like Figure 4 As shown, the detailed process is as follows: (7a) Departure Trigger. When the positioning sensor module detects that a user has moved from the current room to another room, the system starts a buffer wait timer T_wait for the original room. If the user passes through multiple rooms in a short period of time, an independent T_wait timer is started for each room the user leaves and subsequent steps are executed independently.

[0043] (7b) Room-side presence verification. Since the smart glasses leave with the user, this invention uses pre-deployed fixed auxiliary sensors in the original room for presence verification: Visual verification: P_visual is obtained by using the room video captured by the fixed camera in the original room and then processing it through the human detection model; Audio verification: P_audio is obtained by detecting the ambient audio of the room environment collected by the original room environment audio sensor and then detecting the voiceprint activity; Heat source verification: P_thermal is obtained from the infrared heat source signal collected by the original infrared heat source sensor (PIR or infrared thermal imaging array) in the room.

[0044] The overall presence probability P_presence = max(P_visual, P_audio, P_thermal). If only one or two of the above three types of sensors are deployed in the original room, the probability of no channel being deployed is taken as 0, and P_presence is still calculated by taking the maximum value.

[0045] (7c) Adaptive T_wait. T_wait = T_base·(1 + μ·P_return), where T_base is related to the device type (e.g., air conditioner T_base=5 minutes, lighting T_base=2 minutes, water heater T_base=10 minutes), and P_return is the historical probability of the user returning to the room after a short absence, μ∈[0.1, 1.0]. The higher the P_return, the longer the T_wait.

[0046] (7d) Gradual shutdown. When T_wait times out and P_presence is still lower than P_th (e.g., P_th = 0.2–0.3), the system does not directly shut down the device. Instead, it first switches the target device to preset energy-saving parameters (air conditioner is first set to 28°C, lights are first set to the lowest brightness), and then waits for an additional buffer time T_extra. If P_presence has not risen back to P_th by the end of T_extra, the target device is completely shut down. This phased shutdown strikes a balance between avoiding accidental shutdown and achieving energy-saving goals.

[0047] (7e) Event logging. Record a complete log after each shutdown event (departure time, shutdown waiting time, list of shutdown devices, whether anyone returned, etc.) for subsequent habit learning and T_wait parameter optimization.

[0048] like Figure 5 As shown, taking a washing machine linkage scenario as an example, the system predefines a behavior sequence template: A1 = Put ​​clothes in, A2 = Add detergent, A3 = Close the washing machine door. The visual sensing module of the smart glasses continuously monitors the user's actions.

[0049] Upon detection of A1, the sequence monitoring state is entered, and a time window T_window is started (for complex sequences containing more than 3 actions, T_window is 8–15 minutes; for simple sequences containing 2 actions, T_window is 3–8 minutes). Each time a new action A_k is detected, the sequence integrity confidence is calculated using the following formula: C_seq = ∏_{k=1}^{K} P(A_k | A_1, …, A_{k−1}) When K equals the total number of predefined actions and C_seq ≥ C_th (e.g., C_th=0.85), the linkage task is triggered; if the conditions are not met after the T_window timeout, the sequence monitoring state is exited.

[0050] After the linkage is triggered, the intelligent agent's decision-making end decides the device's operating mode according to the following rules: R1: If R_rain > R_th (e.g., R_th=0.5), set the washing device to "wash + dry mode"; R2: If R_rain ≤ R_th and E_price > E_th, delay the task until the current day when E_price ≤ E_th. R3: If R_rain ≤ R_th and E_price ≤ E_th, set it to "wash only mode".

[0051] The decision results are displayed to the user through an AR module, and the user can confirm or modify them by voice.

[0052] like Figure 6 As shown, the system continuously learns user habits through a closed-loop feedback mechanism: (9a) Feedback collection. Explicit feedback includes the user's voice confirmation / rejection / parameter modification of the system decision; behavioral feedback includes subsequent behavioral patterns such as the user restarting the device within N minutes after the system automatically shuts down (considered as a mistaken shutdown signal).

[0053] (9b) Online Update. An exponentially weighted moving average strategy is used to update θ_t = λ·θ_{t−1} + (1 − λ)·θ_new. The forgetting factor λ is adaptively set according to the number of explicit modifications / rejections of system commands by the user in the most recent N days, U_act: when U_act ≥ U_th, it is set to [0.6, 0.8] to allow the model to quickly adapt to new preferences; otherwise, it is set to [0.85, 0.98] to maintain model stability. Updated parameters include w1 to w5, the usage frequency of each device at different times, T_pref(k), the base value of T_wait, P_return, and threshold parameters in the linked decision tree, etc.

[0054] The AR display module of smart glasses follows the principles of non-intrusion (feedback information is superimposed on the edge of the field of vision as a semi-transparent overlay), hierarchical presentation (normal confirmations are small icons, and safety-related operations are prominent pop-ups), and adaptive disappearance (confirmation prompts disappear immediately after user confirmation, and notification prompts disappear automatically after 3-5 seconds).

[0055] When the network connection between the smart glasses and the smart agent's decision-making end is interrupted for more than the preset network interruption threshold T_net (e.g., T_net = 30 seconds), the smart glasses activate the local cache copy and enter offline degradation control mode: (11a) During the factory configuration phase, the smart glasses are issued a local cache by the protocol adaptation gateway, which includes a copy of the home space topology map, a copy of the device status table, and BLE / Zigbee pairing credentials for each managed device. The pairing credentials in the cache are encrypted by AES-128 or an algorithm of equivalent strength and then stored in the secure storage area of ​​the smart glasses. The cache is incrementally synchronized by the intelligent agent decision-making end as the configuration changes during online periods.

[0056] (11b) During offline periods, the edge preprocessing unit of the smart glasses executes S2 to S6 based on the local cache and sends control commands directly to the target device through the BLE or Zigbee local communication protocol stack pre-configured by the smart glasses. For IoT devices that do not support BLE or Zigbee direct connection (e.g., cloud access devices that only support Wi-Fi / MQTT), the operating state of the device remains unchanged during offline periods and does not participate in offline control.

[0057] (11c) After the network is restored, the smart glasses will send the operation logs during the offline period back to the intelligent agent decision-making end to update the user habit model and device status table; if a control command during the offline period conflicts with the cloud status after restoration, the status of the intelligent agent decision-making end shall prevail.

[0058] like Figure 1 As shown, the system in this embodiment includes four levels: The smart glasses perception layer is responsible for multimodal data acquisition and edge preprocessing, including an RGB+depth camera, microphone array, temperature and humidity / light / IMU, UWB / BLE / WiFi and edge preprocessing unit.

[0059] Room-side fixed auxiliary sensing layer: It consists of one or more of fixed cameras, infrared heat source sensors, and environmental audio sensors deployed in each room, providing S7 on-site verification with a data source independent of wearable devices.

[0060] The intelligent agent decision-making layer includes a scene understanding engine, a user intent reasoning engine, an adaptive decision-making engine, a context memory, and an external data interface. This embodiment, as an optional implementation, uses the OpenClaw platform as the platform for this layer; however, the system implementation of this invention does not depend on any specific intelligent agent platform, and any equivalent platform providing multimodal processing, reasoning, and decision-making capabilities can be used as a substitute.

[0061] IoT device execution layer: includes device groups such as temperature control, cleaning, hot water, lighting, and security, as well as protocol adapter gateways. The protocol adapter gateways support protocols such as Zigbee, Z-Wave, MQTT, CoAP, and BLE and encapsulate them into standardized interfaces through the device abstraction layer.

[0062] Adaptive feedback closed-loop layer: runs through the entire system and is responsible for device status feedback, implicit evaluation of user satisfaction, online updates of habit models, and self-optimization of strategies.

[0063] It should be noted that the specific embodiments described in this specification are merely examples of the inventive concept. Those skilled in the art, after reading this specification, can derive other reasonable modifications based on the same technical concept. The true scope of protection of this invention is determined by the claims. All equivalent structural or process changes made using the content of this specification and the accompanying drawings, as well as solutions directly or indirectly applied to other related technical fields, fall within the scope of protection of this invention.

[0064] This invention presents an adaptive intelligent IoT execution method and system based on the collaboration of smart glasses and an intelligent agent system. Through a dual-path sensing design of "smart glasses multimodal perception + room-side fixed auxiliary sensor," it solves the inherent problem of wearable sensing being unable to verify presence status after leaving the original room, improving the reliability of departure detection and reducing the false shutdown rate. By employing a multi-factor weighted matching formula, five dimensions—spatial distance, semantic similarity, usage habits, device status, and time period preference—are jointly used for target device selection. Compared to schemes that only match voice commands and text, this improves the accuracy of target device matching and reduces cross-room false control in scenarios with multiple rooms and multiple similar devices. Through the joint determination of behavior sequence integrity confidence C_seq and time window T_window, the linkage task is triggered only when a complete behavior sequence is achieved and the confidence exceeds C_th, reducing the false trigger rate. By using offline degradation control steps based on local cached topology map copies, device status table copies, and pre-configured local communication protocol stacks, control continuity under network interruption conditions is improved. By employing an online parameter update strategy based on exponentially weighted moving averages, parameters such as multi-factor matching weights, temperature preferences, and the base value of the buffer waiting time T_wait can be continuously adjusted based on user feedback.

[0065] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An adaptive intelligent Internet of Things (IoT) execution method based on the collaboration of smart glasses and an intelligent agent system, applied to a system comprising smart glasses, an intelligent agent decision-making terminal, a protocol adaptation gateway, and multiple IoT devices; wherein the IoT devices are deployed in a space comprising at least two rooms; each room is further equipped with a fixed auxiliary sensor, the fixed auxiliary sensor comprising at least one or more of a fixed camera, an infrared thermal sensor, and an ambient audio sensor; characterized in that, The method includes: Step S1, Multimodal Perception Data Acquisition: Through the visual sensing module, audio sensing module, environmental sensing module, and positioning sensing module integrated on the smart glasses worn on the user's head, visual image data, environmental audio data, temperature, humidity, light data, and spatial location data of the user's space are collected simultaneously; at the same time, room-side perception data is collected through fixed auxiliary sensors deployed in each room, and the room-side perception data includes at least one of the following: room video data collected by a fixed camera, infrared heat source data collected by an infrared heat source sensor, and room environmental audio data collected by an environmental audio sensor; Step S2, Scene Recognition and Spatial Positioning: Input the visual image data into a pre-trained scene classification model to identify room type. At the same time, combine the spatial location data to determine the user's current location coordinates in the home space topology map through an indoor positioning algorithm, and query the list of registered IoT devices in the area where the location coordinates are located. Step S3, User voice command parsing: Perform voice activity detection and voice recognition on the environmental audio data, and extract the device type slot, action type slot and parameter slot from the user voice command; Step S4, Multi-factor weighted matching of target devices: Based on the current region device list obtained in step S2 and the device type slots obtained in step S3, a multi-factor weighted matching algorithm is used to calculate the matching score S_match(d) for each candidate device: S_match(d) = w1·f_dist(d) + w2·f_sem(d) + w3·f_habit(d) + w4·f_state(d) + w5·f_time(d) Where f_dist(d) is the spatial distance factor, f_sem(d) is the semantic matching factor, f_habit(d) is the user habit factor, f_state(d) is the device current state factor, and f_time(d) is the time period preference factor; w1 to w5 are weight coefficients and w1+w2+w3+w4+w5=1; the candidate device with the highest S_match(d) is selected as the target device; when the highest S_match(d) is lower than the preset matching confidence threshold S_th, the AR display module of the smart glasses requests manual confirmation from the user; Step S5, Calculation of equipment operating parameters: Based on the current environmental sensor data and user habit model, determine the operating parameters of the target equipment according to the preset parameter calculation formula; for temperature control equipment, the operating parameters include the target temperature T_target calculated by the following formula: T_target = T_pref(k) + β·(T_out − T_ref) + γ·(N_person − 1)·ΔT_person Where T_pref(k) is the user's historical preferred temperature in time period k, T_out is the current outdoor temperature, T_ref is the reference outdoor temperature, β is the outdoor temperature compensation coefficient, N_person is the number of people in the room obtained by visual human detection, ΔT_person is the temperature compensation amount for each additional person, and γ is the personnel density influence coefficient. Step S6, Control Command Generation and Security Verification: A control command is generated based on the target device and the operating parameters calculated in step S5. This control command is then subjected to device capability boundary verification, device conflict detection, and security confirmation verification in sequence. The device capability boundary verification checks whether the command parameters are within the allowable operating range of the target device. Device conflict detection checks whether two or more devices from a pre-registered set of mutually exclusive devices are simultaneously enabled in the same room; high-priority devices are retained according to a pre-configured device priority list, while low-priority devices are disabled or have their startup delayed. Security confirmation verification checks whether the target device belongs to a pre-registered set of high-power devices or a set of security-sensitive devices; a confirmation request is sent to the user through the AR display module of the smart glasses, and the control command is only issued after receiving confirmation from the user. After successful verification, the control command is converted to a communication protocol format supported by the target device via a protocol adaptation gateway before being issued and executed. Step S7, Departure Detection and Automatic Shutdown Decision Based on Room-Side Sensors: While the target device is in operation, when the positioning sensor module detects that a user has moved from the current room to another room, it starts a buffer wait timer T_wait. During the countdown of T_wait, the presence of personnel is verified based on the fixed auxiliary sensors deployed in the original room. The presence verification calculates the probability of presence in the original room P_presence using the following fusion determination formula: P_presence = max(P_visual, P_audio, P_thermal) Where P_visual is the visual presence probability obtained by human detection from room video data collected by the fixed camera in the original room; P_audio is the audio presence probability obtained by voiceprint activity detection from room environmental audio data collected by the environmental audio sensor in the original room; P_thermal is the heat source presence probability obtained from infrared heat source data collected by the infrared heat source sensor in the original room; when P_presence is still lower than the preset presence threshold P_th when the T_wait countdown expires, a phased shutdown operation is performed: first, the target device's operating parameters are switched to the preset energy-saving parameters and then wait for an additional buffer time T_extra. If P_presence does not rise back to P_th during the T_extra period, the target device is completely shut down. Step S8, User Habit Learning and Strategy Update: Collect explicit and behavioral feedback from users regarding device control results, and update the parameter vector θ_t of the user habit model according to the following exponentially weighted moving average update formula: θ_t = λ·θ_{t−1} + (1 − λ)·θ_new Where λ is the forgetting factor, 0 < λ < 1; the parameters of the user habit model include the weight coefficients w1 to w5 in step S4, the historical preferred temperature T_pref(k) calculated in step S5, and the base value of the buffer waiting time T_wait in step S7.

2. The method as described in claim 1, characterized in that: The indoor positioning algorithm in step S2 uses a fusion positioning method combining UWB ultra-wideband positioning and WiFi fingerprint positioning. P_fused = α·P_uwb + (1 − α)·P_wifi Where P_uwb is the UWB positioning result, P_wifi is the WiFi fingerprint positioning result, and α is the dynamic fusion coefficient; when the UWB signal strength index RSSI_uwb is greater than the preset threshold RSSI_th, α takes the value of [0.7, 0.9]; when RSSI_uwb is less than or equal to RSSI_th, α takes the value of [0.3, 0.5].

3. The method as described in claim 1, characterized in that: Step S3 also includes referential resolution processing: when the user's voice command contains an indicator pronoun, the specific IoT device referred to by the indicator pronoun is determined by combining the user's head orientation information and gaze direction information in the visual image data; specifically, based on the head orientation and gaze direction, a gaze cone angle with the user as the vertex is determined, and candidate devices located within the cone angle range and whose spatial distance from the user does not exceed the preset maximum referential distance D_ref are sorted from high to low according to their visual proportion in the user's field of vision and gaze duration, and the device ranked first is taken as the referential target.

4. The method as described in claim 1, characterized in that: In step S4, the spatial distance factor f_dist(d) is calculated using the normalized inverse distance formula: f_dist(d) = 1 − dist(P_user, P_device(d)) / D_max Where P_user is the user's current location coordinates, P_device(d) is the registered location coordinates of candidate device d, dist(·) is the Euclidean distance function, and D_max is the maximum distance between the user and all candidate devices in the current room.

5. The method as described in claim 1, characterized in that: The method also includes a behavior sequence recognition and automatic triggering step for linked tasks: the visual sensing module of the smart glasses performs time-series detection on the user's continuous operation behavior; when the first action A_1 is detected, the sequence monitoring state is entered and a preset time window T_window is started; Within the T_window, for each detected action A_k (k=1,…,K), the current sequence integrity confidence C_seq is calculated using the following formula: C_seq = ∏_{k=1}^{K} P(A_k | A_1, A_2, …, A_{k−1}) Where P(A_k | A_1, …, A_{k−1}) is the conditional probability that the k-th action is correctly identified given that the first k−1 actions have been observed, and K is the total number of actions in the predefined behavior sequence; when K is equal to the total number of actions in the predefined behavior sequence and C_seq is greater than or equal to the preset sequence confidence threshold C_th, the behavior sequence is determined to be complete and the device linkage task associated with the behavior sequence is triggered; if the determination condition is not met after the T_window timeout, the sequence monitoring state is exited and no linkage task is triggered.

6. The method as described in claim 5, characterized in that: The execution of the linkage task also includes external environment perception and equipment operating mode decision-making steps: after the linkage task is triggered, the intelligent agent decision-making terminal obtains current weather data and electricity price data through an external data interface; the operating mode decision is executed according to the following rules: When the precipitation probability R_rain in the weather data is greater than the preset precipitation threshold R_th, the working mode of the washing equipment will be set to washing and drying mode. When R_rain is less than or equal to R_th, and the current electricity price E_price in the electricity price data is higher than the preset electricity price threshold E_th, the operation of the washing equipment will be delayed until the current day when the electricity price is lower than E_th. When R_rain is less than or equal to R_th and E_price is less than or equal to E_th, the working mode of the washing equipment is set to washing mode only.

7. The method as described in claim 1, characterized in that: In step S7, the buffer waiting time T_wait is adaptively adjusted based on the device type and user history behavior: T_wait = T_base · (1 + μ · P_return) Where T_base is the preset base buffer time related to the device type, P_return is the probability that a user will return to the room within the T_base duration after a brief departure in the historical data, and μ is the adjustment coefficient and μ∈[0.1, 1.0].

8. The method as described in claim 1, characterized in that: In step S8, the forgetting factor λ is adaptively adjusted based on the number of times U_act, which the user has explicitly modified or rejected the control instructions generated by the system in the most recent N days according to the user habit model: when U_act is greater than or equal to the preset activity threshold U_th, λ takes the value of [0.6, 0.8]; when U_act is less than U_th, λ takes the value of [0.85, 0.98].

9. The method as described in claim 1, characterized in that: The method further includes an offline degradation control step: when the network connection interruption between the smart glasses and the intelligent agent decision-making terminal continues to exceed a preset network interruption threshold T_net, the smart glasses enable a locally cached copy of the home space topology map and a copy of the device status table. Based on the locally cached copies, steps S2 to S6 are executed in the edge preprocessing unit of the smart glasses, and control commands are directly sent to the target devices through the local communication protocol stack pre-configured by the smart glasses. The local communication protocol stack supports at least one of BLE and Zigbee. During the offline network configuration phase, the pairing credentials of each target device are pre-issued by the intelligent agent decision-making terminal and securely stored. Once the network is restored, the smart glasses will send the operation logs from the offline period back to the intelligent agent's decision-making terminal to synchronously update the user habit model.

10. An adaptive intelligent Internet of Things execution system based on the collaboration of smart glasses and an intelligent agent system, characterized in that, The system for performing the method according to any one of claims 1 to 9, the system comprising: The smart glasses' sensing end includes a visual sensing module, an audio sensing module, an environmental sensing module, a positioning sensing module, an edge preprocessing unit, and an AR feedback display module; Room-side fixed auxiliary sensors include one or more of the following: fixed cameras deployed in each room, infrared thermal sensors, and ambient audio sensors; The intelligent agent decision-making end includes a scene understanding engine, a user intent reasoning engine, an adaptive decision engine, a context memory, and an external data interface. The intelligent agent decision-making end is used to receive multimodal data collected by the intelligent glasses sensing end and room-side sensing data collected by the room-side fixed auxiliary sensor, perform scene recognition, user intent parsing, target device multi-factor weighted matching, operating parameter calculation, departure detection and linkage task decision-making, and generate control commands. The IoT device execution end includes a protocol adaptation gateway and the plurality of IoT devices. The protocol adaptation gateway is used to convert the control command into a communication protocol format supported by the target device and send it for execution. An adaptive feedback closed-loop module is used to collect device execution status feedback and user behavior feedback, and update the user habit model parameters in the context memory according to the exponentially weighted moving average update formula described in step S8 of claim 1.

Images