Smart home AI control system

Abstract

The invention relates to the technical field of smart home, in particular to a smart home AI control system. The system comprises an equipment initialization and environment perception module which adopts an improved ORB-SLAM3 algorithm to realize centimeter-level high-precision pose tracking and construct a spatial memory map; the triple interaction fusion module fuses high-precision eye movement tracking, micro-gesture recognition and intelligent voice, and obtains user intentions through dynamic weight adjustment; the scenarized control and execution module is used for constructing a state-action scenarized control matrix and executing an intelligent linkage decision based on a multi-factor scoring system; the self-adaptive rendering optimization module is used for reducing GPU load through action prediction and fixation point differential rendering; and the behavior reinforcement learning module continuously optimizes system response and service recommendation by using a hierarchical deep learning network and a deep Q network mechanism. According to the system, the interaction precision is remarkably improved, the equipment energy consumption is reduced, and active learning and self-adaption of the system are realized.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and in particular to a smart home AI control system. Background Technology

[0002] With the deep integration of the Internet of Things (IoT) and immersive technologies such as Virtual Reality (VR), smart home control is evolving towards a more natural and efficient interaction mode. However, existing technologies face several key bottlenecks in achieving this goal. First, there's the issue of interaction accuracy. Traditional VR interaction relies on precise sensory input, but current systems lack robustness in gesture recognition, and in crucial eye tracking, the error is typically greater than 2%. This low accuracy directly results in a poor user experience and imprecise control when users attempt to control devices or fine-tune parameters in a VR environment. Second, there's the system performance bottleneck. High-performance VR rendering causes excessive GPU load, leading to severe device overheating and short battery life, especially on mobile VR devices where performance degradation is significant. This severely restricts long-term system use and user comfort.

[0003] Furthermore, existing smart home systems generally suffer from insufficient scene intelligence. These systems lack continuous perception and understanding of the user's precise pose in three-dimensional space, and cannot adaptively adjust control strategies based on the user's accurate location within the space. Scene transitions often lack smooth, context-based decision-making, resulting in abrupt and unresponsive user experiences. More importantly, existing systems offer limited personalized services and lack effective behavioral learning models. They cannot analyze and predict users' complex behavioral habits and preferences across multiple time scales, and cannot provide truly targeted proactive services. The level of system intelligence remains at the static, preset scene stage, lacking the ability to self-optimize and continuously adapt to user behavior through reinforcement learning mechanisms. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides an intelligent home AI control system, which aims to solve the problems of low interaction accuracy, insufficient spatial intelligence, high rendering energy consumption, and poor system adaptability in the existing technology.

[0005] This invention provides the following technical solution: a smart home AI control system, comprising:

[0006] The device initialization and environmental perception module is used to connect the terminal device to the smart home hub via wireless communication and load a preset three-dimensional home environment map. It combines visual-inertial fusion technology to track the high-precision pose of the terminal device in three-dimensional space in order to build and update the spatial memory map of the environment.

[0007] The triple interaction fusion module is used to collect user eye movement data, gesture data and voice data in parallel, and through multimodal fusion, combined with time synchronization and confidence assessment, it identifies and analyzes the collected data to obtain user intent;

[0008] The scenario-based control and execution module is used to construct a scenario-based control matrix from the user's multi-dimensional state to the device control actions. It combines a spatial memory algorithm to execute scenario decisions, select the optimal linkage scenario scheme, and has the execution controller execute scenario switching.

[0009] The adaptive rendering optimization module is used to dynamically adjust the rendering strategy based on the user's gaze point, head movement information, and system load information provided by the monitoring unit.

[0010] The behavior reinforcement learning module is used to analyze users' short-term, medium-term and long-term behavior across multiple time scales, use hierarchical deep learning networks to achieve accurate prediction of user behavior, and continuously optimize system response and personalized service recommendations based on reinforcement learning mechanisms and user feedback.

[0011] Preferably, the step of tracking the high-precision pose of the terminal device in three-dimensional space includes:

[0012] An improved ORB feature point extraction algorithm is used to obtain feature points, a bag-of-words model is used to accelerate feature matching, and PnP combined with RANSAC algorithm is used to achieve initial estimation of camera pose.

[0013] Real-time high-precision pose tracking is performed based on the improved ORB-SLAM3 algorithm. Sliding window optimization is used to preserve keyframes and reduce computation, and graph optimization is used to minimize reprojection error using the g2o optimization library.

[0014] The bag-of-words model is used to identify visited scenarios and loop closure optimization is performed to eliminate accumulated errors and improve global consistency.

[0015] Preferably, the user eye-tracking data acquisition steps include:

[0016] Eye tracking is achieved using a pupil center detection algorithm based on corneal reflection;

[0017] Run an adaptive pupil model to dynamically adjust recognition parameters based on the user's eye characteristics;

[0018] A drift correction mechanism is implemented, and periodic calibration ensures accuracy for long-term use.

[0019] Preferably, the user gesture data collection step includes:

[0020] Hand localization is performed using an object detection model;

[0021] The key points of the hand and their corresponding 3D coordinates are extracted using Media Pipe, and the 3D coordinates are converted into feature vectors.

[0022] The feature vectors are input into a CNN-LSTM hybrid model to predict the gesture category and confidence level.

[0023] Preferably, the user voice data collection step includes:

[0024] The target sound source is enhanced directionally using beamforming algorithms, and environmental interference is filtered out using adaptive noise cancellation and echo cancellation techniques.

[0025] An end-to-end speech recognition model is adopted, combined with domain-specific vocabulary enhancement, to achieve offline recognition of smart home control commands;

[0026] By understanding the context and combining it with information from the current scene, the accuracy of understanding voice commands can be improved.

[0027] Preferably, the step of obtaining user intent includes:

[0028] The eye-tracking, gesture, and speech recognition results are time-series aligned within a preset time window;

[0029] The confidence assessment of each modality recognition result is calculated, and when determining the user's intent, eye-tracking localization, gesture execution, and voice settings are combined and coordinated.

[0030] Based on confidence assessment, prediction priority rules, and user context information under different time windows, the weights of each modality are dynamically adjusted to achieve the final user intent decision.

[0031] Preferably, the scenario-based control matrix construction step includes:

[0032] Define a multidimensional state variable that includes user location, time, and environmental parameters;

[0033] Define a set of actions that include device control, scene switching, and parameter adjustment;

[0034] Construct a scenario-based control matrix that transforms state variables into action sets.

[0035] Preferably, the scenario decision-making steps combined with the spatial memory algorithm include:

[0036] The location and attitude information of the terminal device are continuously updated through a spatial memory algorithm, and related spatial events are triggered, including area entry detection and position maintenance detection.

[0037] Receive the spatial event, activate the relevant scene rules, query the scene-based control matrix, and calculate the matching score of the candidate scene set based on the multi-factor scoring system;

[0038] Select the scenario solution with the highest matching degree and output it as the optimal linkage scenario solution.

[0039] Preferably, the step of dynamically adjusting the rendering strategy includes:

[0040] Predict the user's head and eye movements within a preset time window in the future, and adjust the rendering refresh rate of the immersive interactive terminal based on the prediction results and system load;

[0041] Based on the real-time gaze tracking results, multi-resolution differential rendering is performed, with high-resolution rendering applied to the user's visual center area and progressive resolution downgrade applied to the surrounding areas.

[0042] Adjusting the level of detail based on scene complexity and adjusting shadow precision based on object distance and importance achieves a balance between rendering effects and computing resources.

[0043] Preferably, the steps for optimizing system response and personalized service recommendations include:

[0044] User behavior is analyzed using a hierarchical deep learning network, which includes a short-term behavior recognition network, a medium-term pattern learning network, and a long-term preference modeling network.

[0045] A deep Q-network mechanism is used to optimize system response and service recommendation. This mechanism selects the best action based on the state space and action space, and continuously updates the decision-making strategy through reward signals.

[0046] The present invention has the following beneficial effects:

[0047] 1. In this invention, a triple interaction fusion module is used, employing high-precision eye tracking based on corneal reflection and micro-gesture recognition using a CNN-LSTM hybrid model, as well as domain-specific vocabulary-enhanced speech data processing. Combined with time synchronization and confidence assessment, the interaction accuracy and naturalness are greatly improved, ensuring that user intent is captured accurately and efficiently.

[0048] 2. In this invention, an improved version of the ORB-SLAM algorithm of the device initialization and environmental perception module is used to achieve centimeter-level high-precision pose tracking. Combined with a scenario-based control matrix and a multi-factor scoring system, intelligent scene decision-making based on the user's spatial location is realized, which completely solves the problem of insufficient spatial intelligence.

[0049] 3. In this invention, an adaptive rendering optimization module dynamically adjusts the rendering refresh rate based on an LSTM motion prediction model and performs foveated differential rendering. This strategy effectively reduces GPU load and energy consumption while ensuring the clarity of the core field of view, significantly improving dizziness rate and battery life.

[0050] 4. In this invention, a behavior reinforcement learning module is established, which uses a short-term / medium-term / long-term hierarchical deep learning network to accurately predict user behavior, and through a deep Q-network reinforcement learning mechanism, it achieves continuous self-optimization of system response and personalized service recommendation, breaking through the bottleneck of poor adaptive capability of traditional systems. Attached Figure Description

[0051] Figure 1 A structural diagram of a smart home AI control system provided in an embodiment of the present invention;

[0052] Figure 2 A flowchart for obtaining user intent through multimodal fusion provided in an embodiment of the present invention;

[0053] Figure 3 This is a flowchart illustrating the scenario-based control matrix combined with a spatial memory algorithm for executing scenario decisions, as provided in an embodiment of the present invention.

[0054] Figure 4 A flowchart illustrating the dynamic adjustment rendering strategy provided in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This invention provides the following technical solution: a smart home AI control system, comprising:

[0057] The device initialization and environmental perception module is used to connect the terminal device to the smart home hub via wireless communication and load a preset three-dimensional home environment map. It combines visual-inertial fusion technology to track the high-precision pose of the terminal device in three-dimensional space in order to build and update the spatial memory map of the environment.

[0058] Preferably, the step of tracking the high-precision pose of the terminal device in three-dimensional space includes:

[0059] An improved ORB feature point extraction algorithm is used to obtain feature points, a bag-of-words model is used to accelerate feature matching, and PnP combined with RANSAC algorithm is used to achieve initial estimation of camera pose.

[0060] Real-time high-precision pose tracking is performed based on the improved ORB-SLAM3 algorithm. Sliding window optimization is used to preserve keyframes and reduce computation, and graph optimization is used to minimize reprojection error using the g2o optimization library.

[0061] The bag-of-words model is used to identify visited scenarios and loop closure optimization is performed to eliminate accumulated errors and improve global consistency.

[0062] Specifically, the user-worn device establishes a stable connection with the smart home hub via Bluetooth and Wi-Fi. The hub then transmits pre-collected or modeled 3D map data of the home environment to the device. This map includes not only geometric information but also semantic information such as furniture and devices. The device continuously collects environmental image data and its own motion data using its built-in visual sensors and inertial measurement unit.

[0063] After an image frame enters the system, an improved ORB feature point extraction algorithm is first used to obtain visual feature points with good repeatability and discriminative power on the high-resolution image. To improve real-time performance, the system uses a bag-of-words model to index and quantize feature descriptors, enabling accelerated feature matching between images. Based on the successfully matched feature points and 3D coordinates in the map, the system uses an n-point perspective algorithm (PnP) combined with the Random Sample Consensus (RANSAC) algorithm, aided by pre-integrated data from the inertial measurement unit, to achieve initial estimation of the camera pose and effectively eliminate erroneous matches.

[0064] The system performs real-time pose tracking based on an improved version of the ORB-SLAM3 algorithm, with its core being a tightly coupled vision-inertial odometry (VIO) mechanism. The VIO mechanism integrates visual observation and inertial measurement unit measurement, improving the frequency and robustness of pose estimation, and performing particularly well in fast-moving or texture-deficient scenarios.

[0065] To maintain high accuracy while meeting the real-time requirements of terminal devices, a sliding window optimization strategy is adopted to incorporate the most recent keyframes and their associated map points into the optimization problem, thereby maintaining the number of keyframes and reducing the amount of computation. This step models the localization problem as a graph optimization problem, and by constructing an optimization graph, the reprojection error is minimized using the g2o optimization library.

[0066] The system continuously compares the current frame with historical keyframes. When a similarity between the current scene and a previously visited scene is detected, loop closure detection is triggered. This detection process uses a bag-of-words model to identify visited scenes, achieving fast and accurate scene recognition. Once a loop is confirmed, the system performs loop closure optimization. This optimization step uses global graph optimization to evenly distribute accumulated pose errors across the loop path, thereby eliminating accumulated errors and ensuring the global consistency and long-term stability of the spatial memory map.

[0067] During high-precision pose tracking, the system continuously builds and updates a spatial memory map of the environment. This map not only contains sparse feature points for localization but also an octree map. This map structure provides centimeter-level spatial localization and environmental understanding capabilities for the upper-level scene-based control and execution modules.

[0068] This module achieves centimeter-level, highly robust pose tracking of the terminal in three-dimensional space based on the improved ORB-SLAM3, and constructs a globally consistent spatial memory map.

[0069] The triple interaction fusion module is used to collect user eye movement data, gesture data and voice data in parallel, and through multimodal fusion, combined with time synchronization and confidence assessment, it identifies and analyzes the collected data to obtain user intent;

[0070] Preferably, the user eye-tracking data acquisition steps include:

[0071] Eye tracking is achieved using a pupil center detection algorithm based on corneal reflection;

[0072] Run an adaptive pupil model to dynamically adjust recognition parameters based on the user's eye characteristics;

[0073] A drift correction mechanism is implemented, and periodic calibration ensures accuracy for long-term use.

[0074] Specifically, an eye-tracking sensor array composed of a built-in infrared camera and an infrared light source is used to capture images of the user's eyes. Eye tracking is achieved using a pupil center detection algorithm based on corneal reflection. This algorithm uses an infrared light source to illuminate the eye, and the image sensor simultaneously captures the pupil center and the corneal light spot. By calculating the relative position and distance between the pupil center and the corneal light spot, the direction of gaze and the coordinates of the fixation point in three-dimensional space can be accurately calculated.

[0075] To ensure accuracy and stability under diverse user conditions and prolonged use, the system integrates two adaptive and correction mechanisms. The first is an adaptive pupil model. Since users exhibit significant differences in physiological characteristics such as eye color, pupil size, and eye curvature, the system utilizes an adaptive pupil model. This model dynamically adjusts the parameters for pupil recognition and gaze mapping based on each user's eye characteristics, thereby improving universality and recognition accuracy across different user groups.

[0076] Next is the drift correction mechanism. After prolonged use or slight movement of the head-mounted display, the eye-tracking baseline may drift, affecting positioning accuracy. The system operates a drift correction mechanism, performing periodic calibration or background calibration when the user performs non-critical operations to achieve automatic or semi-automatic baseline adjustment, ensuring accuracy over long-term use. Methods such as the five-point calibration method are typically used to reduce the calibration burden on the user.

[0077] This module achieves stable eye tracking with an error of less than ±0.5° through a high-precision pupil center detection algorithm based on corneal reflection. Coupled with an adaptive pupil model and drift correction mechanism, the system can overcome the effects of individual differences and prolonged use, providing highly reliable gaze point information and offering crucial localization support for multimodal fusion decision-making.

[0078] Preferably, the user gesture data collection step includes:

[0079] Hand localization is performed using an object detection model;

[0080] The key points of the hand and their corresponding 3D coordinates are extracted using Media Pipe, and the 3D coordinates are converted into feature vectors.

[0081] The feature vectors are input into a CNN-LSTM hybrid model to predict the gesture category and confidence level.

[0082] Specifically, gesture data is mainly collected through depth cameras and inertial measurement units to obtain high-precision hand data with six degrees of freedom.

[0083] Images captured by a depth camera are first fed into an object detection model, such as YOLOv5. This model quickly and accurately locates the hand in the image, determines the bounding box and position of the hand to eliminate background interference, and provides the region of interest for subsequent keypoint extraction.

[0084] For the located hand area, the system uses the Media Pipe keypoint extraction tool to extract 21 finger joints and palm keypoints from the image. Combined with depth information, the system simultaneously obtains the coordinates of these keypoints in 3D space, ensuring that gesture recognition possesses 3D spatial attributes. Since gestures are essentially temporal actions, the system processes and encodes the 3D coordinate data from the continuous time series, transforming it into a feature vector. This vector captures key features such as hand posture, movement speed, relative joint angles, and trajectory.

[0085] Feature vectors are input into a hybrid model architecture for gesture classification and prediction. The system employs a CNN-LSTM hybrid model, which effectively handles the spatiotemporal characteristics of gesture data. The CNN part of this model preferably uses ResNet18, primarily for extracting spatial features of hand poses; the LSTM part preferably uses a bidirectional long short-term memory network, primarily for learning the temporal dependencies of gesture sequences and recognizing the order and trajectory of actions. To improve recognition accuracy, the hybrid model optionally integrates SE attention to automatically enhance the weights of joints or time frames that play a crucial role in recognition. After receiving the feature vectors, the hybrid model predicts the category and confidence level of the current gesture. Gesture categories include micro-gestures such as clicking, swiping, grasping, and zooming.

[0086] This module obtains the 3D temporal coordinates of the hand through object detection and key point extraction, and uses a CNN-LSTM hybrid model to achieve spatiotemporal feature fusion recognition of gestures. This mechanism ensures high accuracy and low latency in micro-gesture recognition, providing core support for precise control of smart home devices.

[0087] Preferably, the user voice data collection step includes:

[0088] The target sound source is enhanced directionally using beamforming algorithms, and environmental interference is filtered out using adaptive noise cancellation and echo cancellation techniques.

[0089] An end-to-end speech recognition model is adopted, combined with domain-specific vocabulary enhancement, to achieve offline recognition of smart home control commands;

[0090] By understanding the context and combining it with information from the current scene, the accuracy of understanding voice commands can be improved.

[0091] Specifically, voice data is collected via a dual-microphone array. To ensure recognition accuracy in complex home environments, the system employs multiple noise reduction processes at the front end. First, a beamforming algorithm is used to process the signals collected by the dual-microphone array. This algorithm can directionally amplify sound sources in the direction the user is speaking, while suppressing noise interference from other directions. Next, adaptive noise cancellation technology is used to identify and filter continuous or sudden environmental noises in real time, such as air conditioner noise and television sound. Finally, echo cancellation technology is run to eliminate echo interference generated in the microphone after sound effects are played from the terminal device's own speakers, ensuring clear input of voice commands.

[0092] After noise reduction, the speech signal is sent to the core recognition module for analysis. The system employs an end-to-end speech recognition model that directly maps acoustic features to text commands. To ensure response speed and data privacy, the model supports offline recognition, avoiding excessive reliance on cloud services. For the characteristics of smart home control, the model incorporates domain-specific vocabulary enhancement. The system maintains a dedicated vocabulary for smart home control commands, such as "open the curtains" and "switch to movie mode," to improve the accuracy of recognizing these technical terms.

[0093] Simply recognizing text accurately is not enough to achieve intelligent control. The system uses a contextual understanding mechanism, combined with current scene information, to perform semantic analysis on voice commands, thereby improving the accuracy of command understanding and eliminating ambiguity. For example, if the user is looking at the living room light, or it is midnight, when the user says "turn it on," the system will combine eye tracking and the current scene to determine that "it" refers to the device the user is looking at.

[0094] This module utilizes a dual-microphone array and employs techniques such as beamforming, adaptive noise cancellation, and echo cancellation to achieve high-fidelity noise reduction for speech data. The core recognition function uses an end-to-end offline model and leverages domain-specific vocabulary enhancement and contextual understanding to ensure accurate and intelligent voice command recognition.

[0095] Preferably, the steps for obtaining user intent are as follows: Figure 2 As shown, it includes:

[0096] The eye-tracking, gesture, and speech recognition results are time-series aligned within a preset time window;

[0097] The confidence assessment of each modality recognition result is calculated, and when determining the user's intent, eye-tracking localization, gesture execution, and voice settings are combined and coordinated.

[0098] Based on confidence assessment, prediction priority rules, and user context information under different time windows, the weights of each modality are dynamically adjusted to achieve the final user intent decision.

[0099] Specifically, since the acquisition frequencies and processing delays of eye-tracking, gesture, and speech data differ, the system first aligns their recognition results within a preset time window. This time window is an optimized short period of time that ensures all relevant input signals correspond accurately in time, thus forming a complete interactive event unit.

[0100] The confidence level of each modality's recognition result is calculated. For example, the gesture recognition model outputs the probability of the gesture category, and the speech recognition model outputs the text recognition score. The confidence level represents the reliability of the modal input and is a crucial basis for subsequent decision-making and dynamic weight adjustment. When determining user intent, the system does not simply superimpose the results of each modality, but rather leverages their complementary strengths to work collaboratively. Eye-tracking data is typically used to locate the target device or virtual interface element that the user is focusing on, gesture data is typically used to perform instantaneous operations, and speech data is typically used to set parameters or provide supplementary information. Through this collaborative mode, the system can parse more complex and natural interactive commands, such as: "Eye-track to the TV, grasp the gesture for confirmation, and say 'Play a science fiction movie'."

[0101] Intent decisions are dynamically adjusted based on confidence assessment, prediction priority rules, and user context information under different time windows. For example, in quiet environments, voice has a higher weight; in noisy environments, voice has a lower weight, while the weight of eye movements and gestures increases accordingly. When the system determines that the user is in a fine-grained operation state, the weight of gestures increases; when the user is in a fast browsing state, the weight of eye movements increases.

[0102] This module achieves multimodal data collaboration through time series alignment, confidence assessment, and complementary enhancement patterns. Its core innovation lies in a dynamic weight adjustment mechanism based on confidence levels, priority rules, and user context information, ensuring accurate and efficient acquisition of the user's final operational intent across different environments and interaction styles.

[0103] The scenario-based control and execution module is used to construct a scenario-based control matrix from the user's multi-dimensional state to the device control actions. It combines a spatial memory algorithm to execute scenario decisions, select the optimal linkage scenario scheme, and has the execution controller execute scenario switching.

[0104] Preferably, the scenario-based control matrix construction step includes:

[0105] Define a multidimensional state variable that includes user location, time, and environmental parameters;

[0106] Define a set of actions that include device control, scene switching, and parameter adjustment;

[0107] Construct a scenario-based control matrix that transforms state variables into action sets.

[0108] Specifically, the state dimension is the set of user and environmental factors that the system needs to perceive in real time. It includes all key information obtained from the device initialization and environmental perception module and the triple interaction fusion module. User location: High-precision pose information provided by the ORB-SLAM3 spatial positioning algorithm, including x, y, z coordinates and attitude angles. Time: Includes absolute and relative time, where absolute time refers to a specific moment, date, and holiday, and relative time refers to a time period within a day, weekday, or weekend. Environmental parameters: Includes physical and spatial environments. The physical environment includes temperature, humidity, light intensity, noise, and air quality, while the spatial environment includes the user's location and furniture layout.

[0109] The action dimension refers to the complete set of operations that the system can perform on smart home devices. Device control: such as turning specific devices on / off, like lights, air conditioners, and curtains. Scene switching: such as switching all devices to preset "movie mode" or "sleep mode". Parameter adjustment: such as adjusting light brightness, air conditioner temperature, and volume.

[0110] Mapping rules define the specific actions or interactive scenarios that the system should execute under certain multidimensional states. Example rule: If [user location = sofa area] and [time = 8 PM] and [ambient light < 10 lux] → [execute action = switch viewing mode]. Each element of the scenario-based control matrix represents a potential intelligent interactive scenario solution. Its input is multidimensional state variables collected in real time, and its output is a set of matching actions. The mapping rule set can be manually configured by the user, preset by the system, and continuously optimized and adjusted by the behavior reinforcement learning module based on long-term user feedback and preferences.

[0111] This module completes the construction of a scenario-based control matrix by defining multidimensional state variables that include location, time, and environmental parameters, defining a set of actions that include control, switching, and adjustment, and establishing a set of mapping rules from state variables to action sets.

[0112] Preferably, the scenario decision-making steps are performed in conjunction with a spatial memory algorithm, such as... Figure 3 As shown, it includes:

[0113] The location and attitude information of the terminal device are continuously updated through a spatial memory algorithm, and related spatial events are triggered, including area entry detection and position maintenance detection.

[0114] Receive the spatial event, activate the relevant scene rules, query the scene-based control matrix, and calculate the matching score of the candidate scene set based on the multi-factor scoring system;

[0115] Select the scenario solution with the highest matching degree and output it as the optimal linkage scenario solution.

[0116] Specifically, the ORB-SLAM3 spatial memory algorithm continuously and accurately updates the terminal's position and posture information. Based on this real-time pose information, the decision engine triggers relevant spatial events. These spatial events are preset behavioral patterns related to physical space, mainly including area entry detection and position holding detection. The former detects whether the user has entered a specific preset area in the home environment, such as stepping into the living room sofa area, while the latter detects whether the user stays in a specific area for more than a preset time threshold, such as staying in front of the kitchen counter for more than 10 seconds.

[0117] When a spatial event is triggered, the decision engine receives the triggered spatial event, activates the relevant scene rules in the scene-based control matrix, and then queries the scene-based control matrix to obtain a set of all candidate scenes. The system calculates the matching score for each candidate scene based on a multi-factor scoring system. These factors include assessing the positional matching degree between the current location, gaze point, and the spatial region required by the scene; assessing the temporal matching degree between the current time and the scene's preset time period; and evaluating the user's historical preference for activating which scene the user tends to activate in similar states, based on user historical preference data.

[0118] The decision engine sorts the calculated matching scores and selects the scenario with the highest matching score, outputting this scenario as the optimal linkage scenario to the execution controller. The execution controller then translates the scenario into a series of specific device control commands and executes the scenario switching, such as: drawing the curtains, dimming the main light, and turning on the ambient light.

[0119] This module uses a spatial memory algorithm to monitor and trigger spatial events such as area entry and position maintenance in real time, thereby activating the scene decision-making process. The core mechanism lies in a multi-factor scoring system based on location, time, and historical preferences, which accurately matches and evaluates candidate scenes, and finally outputs the linkage scene solution with the highest matching degree, realizing intelligent control based on user spatial behavior.

[0120] The adaptive rendering optimization module is used to dynamically adjust the rendering strategy based on the user's gaze point, head movement information, and system load information provided by the monitoring unit.

[0121] Preferably, the steps for dynamically adjusting the rendering strategy are as follows: Figure 4 As shown, it includes:

[0122] Predict the user's head and eye movements within a preset time window in the future, and adjust the rendering refresh rate of the immersive interactive terminal based on the prediction results and system load;

[0123] Based on the real-time gaze tracking results, multi-resolution differential rendering is performed, with high-resolution rendering applied to the user's visual center area and progressive resolution downgrade applied to the surrounding areas.

[0124] Adjusting the level of detail based on scene complexity and adjusting shadow precision based on object distance and importance achieves a balance between rendering effects and computing resources.

[0125] Specifically, the system utilizes an LSTM network action prediction model trained in the behavior reinforcement learning module to analyze the user's head and eye movement data in real time over a recent period. This model can predict the user's movement trends and amplitude within a preset time window. The system continuously monitors the GPU's system load. If the predicted user movement amplitude is small or stable, and the system load is high, the system will lower the rendering refresh rate, for example, from 120Hz to 90Hz. If the system predicts the user will perform large, rapid head or eye movements, it will immediately increase the rendering refresh rate to ensure smooth visuals during critical moments. This predictive adjustment minimizes performance bottlenecks, reducing GPU load by an average of 30%, while also reducing unnecessary rendering, thereby extending the device's battery life.

[0126] To improve perceived clarity and conserve rendering resources, the system utilizes high-precision eye-tracking technology for gaze-based rendering. Based on real-time gaze tracking results provided by the triple-interaction fusion module, the system determines the central focal point of the user's current vision. Using this gaze point as the center, multi-resolution differentiated rendering is performed. High-resolution rendering is applied to the central area of ​​the user's vision to ensure clear perception of details; for peripheral areas far from the gaze point, the system implements progressive resolution downgrading, gradually reducing the rendering resolution to fully utilize the physiological characteristic of the human eye being less sensitive to peripheral vision details.

[0127] To further balance rendering quality and computational resources, the system dynamically adjusts rendering pipeline parameters based on scene context. It adjusts the level of detail based on scene complexity; for 3D objects at the edge of the user's view or in the distance, the system automatically loads models with lower detail, reducing the computational load of drawing calls and vertex processing. Furthermore, it dynamically adjusts shadow precision based on the object's distance in 3D space and its importance in the current scene. For example, high-precision shadows are used for nearby or interactive targets, while low-precision shadows or shadows are disabled for distant or unimportant backgrounds.

[0128] This module achieves comprehensive dynamic optimization of the rendering strategy for terminal devices through dynamic refresh rate adjustment based on motion prediction, multi-resolution differential rendering based on real-time foveated point tracking, and adaptive adjustment of detail level and shadow accuracy. This enables the system to provide a highly immersive experience while significantly reducing GPU load and system power consumption, effectively improving the user experience.

[0129] The behavior reinforcement learning module is used to analyze users' short-term, medium-term and long-term behavior across multiple time scales, use hierarchical deep learning networks to achieve accurate prediction of user behavior, and continuously optimize system response and personalized service recommendations based on reinforcement learning mechanisms and user feedback.

[0130] Preferably, the steps for optimizing system response and personalized service recommendations include:

[0131] User behavior is analyzed using a hierarchical deep learning network, which includes a short-term behavior recognition network, a medium-term pattern learning network, and a long-term preference modeling network.

[0132] A deep Q-network mechanism is used to optimize system response and service recommendation. This mechanism selects the best action based on the state space and action space, and continuously updates the decision-making strategy through reward signals.

[0133] Specifically, the system employs a hierarchical deep learning architecture to analyze user behavior data, achieving accurate predictions of short-term, medium-term, and long-term behaviors. For the short-term behavior recognition network, a CNN-LSTM hybrid model is used to analyze user interactions and real-time environmental changes over the past 10 minutes, instantly predicting the user's possible next action within the next 0 to 30 seconds. For the medium-term pattern learning network, a Transformer encoder analyzes aggregated user behavior data and system status over the past 24 hours, predicting user behavior patterns and scenario needs for the remainder of the day. For the long-term preference modeling network, a graph neural network analyzes the aggregated features of historical behavior data and environmental data, establishing a complex correlation graph between user behavior and devices / scenes, predicting long-term seasonal changes, long-term lifestyle habits, and preference trends.

[0134] The system transforms prediction results into inputs in a reinforcement learning environment, employing a deep Q-network mechanism to optimize system response and personalized service recommendations, achieving a shift from passive response to proactive service. First, the state space is defined as the system's current multidimensional state, including user behavior characteristics, environmental parameters, and device status. The action space is defined as the set of possible system responses and service recommendations, such as switching scenarios, adjusting recommended device parameters, or maintaining the current state. For the reward signal, the decision-making strategy is continuously updated based on user feedback; for example, accepting a recommendation earns a +1 reward, ignoring or manually canceling a recommendation earns a -1 reward, and improving interaction efficiency earns a +0.5 reward. The system inputs the current state into the DQN, selects the optimal action, and observes the user's reaction to the recommendation and the system performance results. The reward value is calculated and the Q-network is updated to optimize the decision-making strategy. The training process is stabilized through strategies such as experience replay and target networks.

[0135] The ultimate output of reinforcement learning is personalized service recommendations and system response optimization. For example: proactive scenario recommendation: when the system predicts a 70% probability that a user will enter a "movie-watching" state within 10 minutes, the system will proactively recommend executing "movie-watching mode" to the user. Parameter fine-tuning optimization: based on long-term preferences, the system can personalize and fine-tune preset parameters such as air conditioning temperature and lighting brightness in "movie-watching mode." Interaction efficiency optimization: when the system detects that a user frequently uses a certain gesture to control a device, it will increase the response priority of that interaction to optimize interaction efficiency.

[0136] This module achieves accurate prediction of user behavior across multiple time scales through hierarchical deep learning networks with short-term, medium-term, and long-term timeframes. Building upon this, the system employs a deep Q-network reinforcement learning mechanism to continuously learn and iterate based on state, action, and reward signals, enabling self-optimization of system responses and highly personalized service recommendations, thus endowing the system with true self-learning and adaptive capabilities.

[0137] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart home AI control system, characterized in that, include: The device initialization and environmental perception module is used to connect the terminal device to the smart home hub via wireless communication and load a preset three-dimensional home environment map. It combines visual-inertial fusion technology to track the high-precision pose of the terminal device in three-dimensional space in order to build and update the spatial memory map of the environment. The triple interaction fusion module is used to collect user eye movement data, gesture data and voice data in parallel, and through multimodal fusion, combined with time synchronization and confidence assessment, it identifies and analyzes the collected data to obtain user intent; The scenario-based control and execution module is used to construct a scenario-based control matrix from the user's multi-dimensional state to the device control actions. It combines a spatial memory algorithm to execute scenario decisions, select the optimal linkage scenario scheme, and has the execution controller execute scenario switching. The adaptive rendering optimization module is used to dynamically adjust the rendering strategy based on the user's gaze point, head movement information, and system load information provided by the monitoring unit. The behavior reinforcement learning module is used to analyze users' short-term, medium-term and long-term behavior across multiple time scales, use hierarchical deep learning networks to achieve accurate prediction of user behavior, and continuously optimize system response and personalized service recommendations based on reinforcement learning mechanisms and user feedback.

2. The smart home AI control system according to claim 1, characterized in that, The steps for tracking the high-precision pose of the terminal device in three-dimensional space include: An improved ORB feature point extraction algorithm is used to obtain feature points, a bag-of-words model is used to accelerate feature matching, and PnP combined with RANSAC algorithm is used to achieve initial estimation of camera pose. Real-time high-precision pose tracking is performed based on the improved ORB-SLAM3 algorithm. Sliding window optimization is used to preserve keyframes and reduce computation, and graph optimization is used to minimize reprojection error using the g2o optimization library. The bag-of-words model is used to identify visited scenarios and loop closure optimization is performed to eliminate accumulated errors and improve global consistency.

3. The smart home AI control system according to claim 1, characterized in that, The steps for collecting user eye-tracking data include: Eye tracking is achieved using a pupil center detection algorithm based on corneal reflection; Run an adaptive pupil model to dynamically adjust recognition parameters based on the user's eye characteristics; A drift correction mechanism is implemented, and periodic calibration ensures accuracy for long-term use.

4. The smart home AI control system according to claim 1, characterized in that, The steps for collecting user gesture data include: Hand localization is performed using an object detection model; The key points of the hand and their corresponding 3D coordinates are extracted using Media Pipe, and the 3D coordinates are converted into feature vectors. The feature vectors are input into a CNN-LSTM hybrid model to predict the gesture category and confidence level.

5. The smart home AI control system according to claim 1, characterized in that, The steps for collecting user voice data include: The target sound source is enhanced directionally using beamforming algorithms, and environmental interference is filtered out using adaptive noise cancellation and echo cancellation techniques. An end-to-end speech recognition model is adopted, combined with domain-specific vocabulary enhancement, to achieve offline recognition of smart home control commands; By understanding the context and combining it with information from the current scene, the accuracy of understanding voice commands can be improved.

6. The smart home AI control system according to claim 1, characterized in that, The steps to obtain user intent include: The eye-tracking, gesture, and speech recognition results are time-series aligned within a preset time window; The confidence assessment of each modality recognition result is calculated, and when determining the user's intent, eye-tracking localization, gesture execution, and voice settings are combined and coordinated. Based on confidence assessment, prediction priority rules, and user context information under different time windows, the weights of each modality are dynamically adjusted to achieve the final user intent decision.

7. The smart home AI control system according to claim 1, characterized in that, The steps for constructing a scenario-based control matrix include: Define a multidimensional state variable that includes user location, time, and environmental parameters; Define a set of actions that include device control, scene switching, and parameter adjustment; Construct a scenario-based control matrix that transforms state variables into action sets.

8. The smart home AI control system according to claim 1, characterized in that, The steps for performing scenario decision-making using spatial memory algorithms include: The location and attitude information of the terminal device are continuously updated through a spatial memory algorithm, and related spatial events are triggered, including area entry detection and position maintenance detection. Receive the spatial event, activate the relevant scene rules, query the scene-based control matrix, and calculate the matching score of the candidate scene set based on the multi-factor scoring system; Select the scenario solution with the highest matching degree and output it as the optimal linkage scenario solution.

9. A smart home AI control system according to claim 1, characterized in that, The steps for dynamically adjusting the rendering strategy include: Predict the user's head and eye movements within a preset time window in the future, and adjust the rendering refresh rate of the immersive interactive terminal based on the prediction results and system load; Based on the real-time gaze tracking results, multi-resolution differential rendering is performed, with high-resolution rendering applied to the user's visual center area and progressive resolution downgrade applied to the surrounding areas. Adjusting the level of detail based on scene complexity and adjusting shadow precision based on object distance and importance achieves a balance between rendering effects and computing resources.

10. A smart home AI control system according to claim 1, characterized in that, The steps to optimize system response and recommend personalized services include: User behavior is analyzed using a hierarchical deep learning network, which includes a short-term behavior recognition network, a medium-term pattern learning network, and a long-term preference modeling network. A deep Q-network mechanism is used to optimize system response and service recommendation. This mechanism selects the best action based on the state space and action space, and continuously updates the decision-making strategy through reward signals.

Images