A multi-modal fusion-based intelligent home central control system and method

Abstract

This invention discloses a smart home central control system and method based on multimodal fusion. The control system includes a multimodal interaction interface layer, a device execution layer, and a central controller. The central controller is communicatively connected to both the multimodal interaction interface layer and the device execution layer. The central controller includes an instruction receiving and parsing unit, a context awareness unit, a multimodal fusion decision engine, a scheduling and execution unit, and a feedback unit. The instruction receiving and parsing unit receives and parses interactive instructions; the context awareness unit acquires various dynamic data to provide the context information required for decision-making; the multimodal fusion decision engine is trained based on a hierarchical hybrid decision architecture, which includes a rule engine layer, a probabilistic inference layer, and a reinforcement learning layer. This invention achieves a more accurate, robust, and personalized smart home control experience through a dedicated fusion algorithm, a hierarchical conflict arbitration strategy, hardware and software co-optimization, and an adaptive feedback mechanism.

Description

Technical Field

[0001] This invention relates to the field of smart home control technology, specifically to a smart home central control system and method based on multimodal fusion; it is suitable for high-precision, personalized control scenarios in complex home environments, and can achieve seamless integration and intelligent arbitration of conflicts between multimodal commands such as voice, gesture, and APP. Background Technology

[0002] With the development of IoT technology, smart home devices are becoming increasingly popular, and multimodal interaction has become an important development direction for smart home control. Currently, existing technologies have proposed various multimodal smart home control systems. For example, patent application CN120386449A discloses a smart interaction system based on a large multimodal model, including a multimodal input layer, a context-aware module, and a cross-modal fusion module. It generates response strategies through hierarchical decision-making. However, this system lacks a dedicated probabilistic model for instruction complementarity in its cross-modal fusion, does not establish a dynamic weight adjustment mechanism for conflict resolution, and does not consider the impact of hardware selection on the fusion effect. Another example is patent CN105892302B, which discloses a multimodal control method for intelligent robots. However, this method only implements the parsing and parameter extraction of multimodal instructions, without addressing complex instruction complementarity fusion and conflict arbitration mechanisms. For example, patent application CN112054946A discloses a smart home control system with multimodal information fusion. However, the information fusion processing subsystem only performs simple information aggregation, does not adopt a hierarchical hybrid decision-making architecture, and does not consider the dynamic adaptation of fusion strategies to hardware performance differences. In summary, the existing technology still has the following technical shortcomings:

[0003] 1. Single fusion mechanism: Cross-modal fusion adopts feature splicing method, and the intention recognition accuracy in complementary instruction scenarios (such as gesture + voice) is only 65%, which cannot handle compound instructions with "missing target + missing action".

[0004] 2. Crude conflict resolution: Conflict arbitration relies on fixed rules and does not consider user identity or environmental factors, resulting in a misoperation rate as high as 18% in multi-user family scenarios;

[0005] 3. Lack of hardware and software synergy: Due to the lack of consideration for differences in hardware performance, the accuracy of gesture recognition using a 2D camera is only 30% in a 5 lux light environment, and the misrecognition rate of voice commands reaches 25% in a noisy environment (75dB).

[0006] 4. Fixed feedback method: 80% of the feedback in the existing system is based on single voice feedback. In environments with noise levels above 70dB, the user feedback reception rate is only 40%, which seriously affects the interactive experience.

[0007] Therefore, there is an urgent need in this field for a multimodal integrated smart home central control system with a refined integrated decision-making mechanism, deep software and hardware collaboration, and personalized adaptive capabilities to address the aforementioned shortcomings of existing technologies. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing technologies and provide a multimodal fusion smart home central control system and method based on a hierarchical hybrid decision architecture. Through a dedicated fusion algorithm, hierarchical conflict arbitration strategy, software and hardware co-optimization and adaptive feedback mechanism, it achieves a more accurate, robust and personalized smart home control experience.

[0009] To achieve the above objectives, the present invention provides a smart home central control system based on multimodal fusion, comprising:

[0010] A multimodal interaction interface layer is used to receive interaction commands from the user in at least two different modalities. This multimodal interaction interface layer includes, but is not limited to, a voice interaction module, a gesture interaction module, and an APP interaction module. Wherein:

[0011] Voice interaction module: Noise reduction is performed using spectral subtraction + LMS adaptive filtering algorithm, and speech recognition is achieved based on MFCC feature extraction + CNN-LSTM hybrid model. The recognition confidence threshold is set to 0.7, and commands below this threshold are marked as "low confidence commands".

[0012] Gesture interaction module: The core sensor is primarily a structured light depth-sensing camera (resolution ≥1920×1080, frame rate ≥30fps, depth measurement range 0.3-5m, depth error ≤±2%), and secondarily a regular 2D RGB camera (resolution ≥1280×720). Hand detection is achieved through YOLOv8, 21 hand key points are extracted through MediaPipe, and gesture classification is completed using a Transformer model. The output is the command type, target device ID, and confidence level (value range 0-1).

[0013] APP Interaction Module: Transmits commands based on HTTPS+MQTT protocol, supports command priority marking (levels 1-5), transmission latency ≤100ms, and output fields include command content, trigger timestamp, user identity ID, and device target ID.

[0014] The device execution layer includes multiple controlled smart home devices, supports Wi-Fi 802.11n / ac, Zigbee 3.0, and Bluetooth 5.2 protocols, with device response latency ≤200ms and execution status feedback accuracy of 100%.

[0015] A central controller is communicatively connected to both the multimodal interaction interface layer and the device execution layer. The central controller can be a standalone embedded hardware device, a functional module integrated into a smart speaker / smart screen, or a cloud-edge collaborative distributed system. The central controller includes: an instruction receiving and parsing unit, a context-aware unit, a multimodal fusion decision engine, a scheduling and execution unit, and a feedback unit. Users generate interaction commands through the multimodal interaction interface layer. The instruction receiving and parsing unit receives these commands from the multimodal interaction interface layer and parses them using natural language processing or protocol parsing techniques. The context-aware unit obtains various dynamic data from the device execution layer regarding the environment in which each smart home device is located. The system provides the contextual information required for decision-making. The multimodal fusion decision engine is trained based on a hierarchical hybrid decision architecture, which includes a rule engine layer, a probabilistic inference layer, and a reinforcement learning layer. The rule engine layer uses the CLIPS rule language to build a rule base, the probabilistic inference layer uses a Bayesian network to achieve intent fusion, and the reinforcement learning layer uses a deep Q-network. The parsed instructions, along with various dynamic data acquired by the context-aware unit, are fed into the multimodal fusion decision engine for intelligent analysis and decision-making. Finally, the scheduling and execution unit generates specific control commands, which are sent to the corresponding smart home devices in the device execution layer to drive their operation. The feedback unit informs the user of the execution results.

[0016] Furthermore, the core sensor of the gesture interaction module adopts a structured light depth-sensing camera or an RGB camera; the specific weight adjustment rule of the gesture interaction module is as follows:

[0017] When the core sensor is a depth-sensing camera, if the ambient light intensity is ∈ [0, 10) lux, the gesture command weight adjustment coefficient is -0.2; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.1; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.05; and the adjustment range does not exceed ±30% in any case.

[0018] When the core sensor uses a regular 2D camera, if the ambient light intensity is <10 lux, the gesture command weight is set to 0; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.5; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.1.

[0019] Furthermore, the context information includes environmental context information, user state context information, device state context information, and time context information. The environmental context information includes ambient light intensity, ambient noise level (decibels), and whether there is human activity. The user state context information includes user identity, user location, user movement trajectory, and user activity status. The device state context information includes operating status and device location. The time context information includes time period and instruction reception time sequence.

[0020] Furthermore, the multimodal fusion decision engine includes a complementary fusion module and an arbitration and conflict resolution module. The complementary fusion module is used to process collaborative multimodal instructions, and the arbitration and conflict resolution module is used to process conflicting multimodal instructions according to a preset priority strategy and contextual relevance, and to decide the instruction that should be executed in the end.

[0021] Furthermore, the Bayesian network uses user intent as the hidden parent node, modal instructions and context as child nodes, and the relationships between nodes are determined by expert knowledge combined with data mining.

[0022] Furthermore, the deep Q-network includes an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer has a 32-dimensional dimension and is used to receive raw feature data and pass it to the first hidden layer. The first hidden layer performs preliminary feature extraction and nonlinear transformation on the input features, mapping the 32-dimensional input to a 64-dimensional space. The second hidden layer further abstracts and compresses the features, compressing the 64-dimensional features back to 32 dimensions. The output layer has a 4-dimensional dimension and is used to generate the final prediction result. The first and second hidden layers are activated using the ReLU function. The output layer uses linear activation.

[0023] Furthermore, the feedback methods of the feedback unit include voice feedback, light feedback, APP push feedback, and screen text feedback.

[0024] This invention also provides a smart home control method based on multimodal fusion, comprising the following steps:

[0025] S1. Multimodal command reception: The multimodal interaction interface layer receives one or more commands from voice commands, gesture commands, and APP commands in parallel through a multi-threaded listening mechanism, processes the commands, and obtains the raw command data; if the timestamp difference of multiple commands is ≤500ms, it is determined to be a synchronous command; otherwise, it is an asynchronous command and processed according to the time sequence.

[0026] S2, Command Parsing and Context Awareness: The command receiving and parsing unit receives interactive commands from the multimodal interaction interface layer and parses the interactive commands through natural language processing or protocol parsing technology; synchronously with command parsing, context awareness collects various dynamic data in the environment where each smart home device is located, performs filtering and normalization processing, generates a context vector, and updates the system global variables;

[0027] S3. Layered Hybrid Decision Making: The multimodal fusion decision engine intelligently analyzes and makes decisions based on the parsed instructions and various dynamic data acquired by the context-aware unit.

[0028] S4. Command Scheduling and Execution: The scheduling and execution unit generates specific control commands based on the final decision result of the multimodal fusion decision engine;

[0029] S5. Adaptive Feedback: The feedback unit selects an appropriate method to inform the user of the execution result based on the adaptive selection and decision process.

[0030] Furthermore, in step S5, the specific determination condition of the adaptive selection logic of the feedback unit is as follows:

[0031] (1) If the ambient noise level is >70dB, speech is excluded; if the ambient noise level is ∈[50,70]dB, the weight of speech is reduced; if the ambient noise level is <50dB, speech is given priority.

[0032] (2). If the ambient light intensity is <10 lux, then exclude the lights; if the ambient light intensity is ∈ [10,50] lux, then adjust the light brightness to the maximum; if the ambient light intensity is ≥50 lux, then adjust the lights to normal brightness.

[0033] (3). If the distance between the user and the terminal is <1m, then voice or light feedback will be given priority; if the distance between the user and the terminal is ∈[1,5]m, then voice / light / screen are all acceptable); if the distance between the user and the terminal is >5m, then APP feedback will be given priority.

[0034] User history preferences: If a user selects APP push notifications 80 times in their last 100 interactions, then APP push notifications will still be prioritized even if the voice condition is met.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) High fusion accuracy: By using a dedicated Bayesian network model to achieve complementary fusion of multimodal instructions, the pain point of incomplete information of single-modal instructions is solved, and the accuracy of intent recognition is improved by more than 30% compared with existing technologies.

[0037] (2) Personalized conflict arbitration: Combining hierarchical weighting strategy and deep reinforcement learning, dynamic and personalized arbitration of conflict commands is achieved, adapting to different user habits and scenario changes, reducing the error rate by 40%.

[0038] (3) Robust hardware and software collaboration: The depth-sensing camera is used first, and the hardware performance difference is dynamically adapted through the decision algorithm. The interaction reliability in complex environments such as dim light and noisy environments is improved by 50% compared with the existing technology.

[0039] (4) Seamless interactive experience: The adaptive multimodal feedback mechanism ensures that the feedback method is accurately matched with the scene and user status, avoids feedback interference, and improves the smoothness of interaction.

[0040] (5) Balancing security and privacy: Independent hardware form supports local decision-making, core data does not rely on external network transmission, response latency is less than 100ms, and absolute security rules protect the safety of users' life and property.

[0041] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a schematic diagram of the overall architecture of a smart home central control system based on multimodal fusion according to the present invention;

[0044] Figure 2 This is a detailed internal structural block diagram of the central controller according to a preferred embodiment of the present invention;

[0045] Figure 3 This is a flowchart illustrating a smart home central control method based on multimodal fusion according to the present invention.

[0046] Figure 4 This is a schematic diagram of a specific embodiment of the present invention;

[0047] Figure 5 This is the overall flowchart of the hierarchical hybrid decision-making system of the present invention;

[0048] Figure 6 This is a flowchart illustrating the core decision-making process in this invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the embodiments shown in the accompanying drawings. However, it should be noted that these embodiments are not intended to limit the present invention. Equivalent transformations or substitutions in function, method, or structure made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0050] Please see Figures 1 to 4 This invention provides a smart home central control system based on multimodal fusion, comprising a multimodal interaction interface layer, a device execution layer, and a central controller; the specific structure is as follows:

[0051] The multimodal interaction interface layer is used to receive interaction commands from the user in at least two different modalities. The multimodal interaction interface layer includes a voice interaction module, a gesture interaction module, an APP interaction module, and other interaction modules; specifically:

[0052] 1. Voice interaction module:

[0053] Front-end processing: 16kHz sampling rate, 16-bit quantization, 20ms frame length, 10ms frame shift, steady-state noise is removed by spectral subtraction, and LMS adaptive filtering is used to eliminate echo (echo suppression ≥20dB).

[0054] Speech recognition: MFCC features have 13 dimensions, plus first-order and second-order differences, for a total of 39 dimensions. The input is a CNN-LSTM model (2 CNN layers, 1 LSTM layer, and softmax output layer). The recognition vocabulary contains 500+ home control-related words, and the real-time recognition latency is ≤300ms.

[0055] 2. Gesture Interaction Module:

[0056] Depth-sensing camera: Uses Intel RealSense D435i, with built-in IMU sensor, supports simultaneous localization and mapping (SLAM), hand detection range of 0.3-5m, and supports 16 preset gestures (such as "click", "swipe", "rotate", "point" etc.).

[0057] Gesture recognition process:

[0058] Image acquisition: RGB+depth image acquisition at 30fps;

[0059] Hand detection: The YOLOv8 model takes ≤10ms for inference on a GPU (NVIDIA Jetson Nano), with a detection confidence threshold of 0.5;

[0060] Key point extraction: MediaPipe extracts 21 key points of the hand, with a key point positioning error of ≤2 pixels;

[0061] Gesture classification: Transformer model input key point coordinate sequence (5 frames), classification accuracy ≥95%, inference time ≤20ms.

[0062] 3. APP Interaction Module:

[0063] Client: Supports Android 8.0+ and iOS 12.0+ systems, developed using Flutter cross-platform, with an interface including a device list, scene modes, and command history;

[0064] Communication mechanism: based on MQTT 3.1.1 protocol, QoS level 2 (ensuring messages are delivered exactly once), heartbeat interval of 30 seconds, and reconnection time ≤ 3 seconds.

[0065] The device execution layer includes multiple controlled smart home devices, such as smart lights, air conditioning systems, curtain motors, and other smart devices. Device access protocol: Must support at least one mainstream communication protocol (Wi-Fi / Zigbee / Bluetooth) and have a status feedback interface. Execution accuracy is specifically as follows: (1) Light brightness: Adjustment accuracy ≤1% (0-100% range); (2) Air conditioning temperature: Adjustment accuracy ≤0.5℃ (16-30℃ range); (3) Curtains: Positioning accuracy ≤1cm, operating noise ≤35dB. Status feedback: After execution, a "success / failure" status must be returned, with a feedback delay ≤100ms.

[0066] The central controller communicates with both the multimodal interaction interface layer and the device execution layer. The central controller can be a standalone embedded hardware device, a functional module integrated into a smart speaker / smart screen, or a cloud-edge collaborative distributed system. The central controller includes an instruction receiving and parsing unit, a context-aware unit, a multimodal fusion decision engine, a scheduling and execution unit, and a feedback unit; wherein:

[0067] The instruction receiving and parsing unit adopts a multi-threaded design, specifically using the Python threading module. Each modality is allocated an independent thread, with the main thread responsible for data integration. The semantic mapping dictionary is stored in JSON format, as shown in the example below.

[0068] "Enlarge": {"Action Type": "Increase Brightness", "Confidence Correction Factor": 0.9},

[0069] "Turn down": {"Action type": "Reduce brightness", "Confidence correction factor": 0.9},

[0070] "Too cold": {"Action type": "Warming up", "Confidence correction coefficient": 0.85}.

[0071] Confidence calibration formula: Calibrated confidence = Original confidence × Modal reliability coefficient × Semantic matching coefficient.

[0072] Context-aware unit:

[0073] Unit sensor data fusion: Kalman filtering is used to fuse data from multiple sensors. For example, the presence of a human body is jointly determined by an infrared sensor (detecting presence) and UWB positioning (detecting location). The accuracy after fusion is ≥99%.

[0074] Data storage: Context data is cached in Redis with an expiration time of 5 minutes, while historical data is stored in a MySQL database for user habit analysis.

[0075] Multimodal fusion decision engine:

[0076] Bayesian network training:

[0077] Training data: 10,000 labeled samples (including multimodal instructions, context, and user intent);

[0078] Training tools: PyMC3 framework, maximum number of iterations 1000, convergence threshold 1e-6;

[0079] Model validation: Five-fold cross-validation was used, and the intent recognition accuracy was ≥95.2%.

[0080] DQN model training:

[0081] Training data: 5000 sets of conflict scenario samples (including instructions, context, and user feedback);

[0082] Training environment: TensorFlow 2.8 framework, GPU (NVIDIA RTX 3090) acceleration;

[0083] Model validation: In 1000 test samples, the arbitration accuracy rate was ≥92% and the user satisfaction rate was ≥85%.

[0084] Scheduling and Execution Unit:

[0085] Protocol adaptation layer: It adopts a plug-in design and supports the addition of new protocols (such as Matter protocol). The plug-in development interface includes three core functions: "command encoding", "data decoding" and "status query".

[0086] Routing algorithm: Dijkstra's algorithm is used to optimize the communication path and reduce transmission latency. The path update cycle is 10 seconds.

[0087] Feedback Unit:

[0088] Multimodal feedback collaboration: Supports mixed feedback (such as voice + light), for example, when the voice feedback "the air conditioner is off" is given, the indicator light stays green for 2 seconds;

[0089] Personalized feedback: Adjust the tone of voice according to the user's identity (e.g., use a gentler voice for children and a concise voice for administrators).

[0090] In one specific implementation, the user generates interaction commands through the multimodal interaction interface layer. The command receiving and parsing unit receives the interaction commands from the multimodal interaction interface layer and parses them using natural language processing or protocol parsing technology. The context awareness unit obtains various dynamic data of the environment in which each smart home device is located from the device execution layer to provide the context information required for decision-making. The multimodal fusion decision engine is trained based on a hierarchical hybrid decision architecture, which includes a rule engine layer, a probabilistic inference layer, and a reinforcement learning layer. The rule engine layer uses CLIPS rule language to build a rule base, the probabilistic inference layer uses a Bayesian network to achieve intent fusion, and the reinforcement learning layer uses a deep Q-network. The parsed commands and various dynamic data obtained by the context awareness unit are sent to the multimodal fusion decision engine for intelligent analysis and decision-making. Finally, the scheduling and execution unit generates specific control commands, which are sent to the corresponding smart home devices in the device execution layer to drive them to perform operations. The feedback unit is used to inform the user of the execution results.

[0091] Furthermore, the core sensor of the gesture interaction module adopts a structured light depth-sensing camera or an RGB camera; the specific weight adjustment rules of the gesture interaction module are as follows:

[0092] When the core sensor is a depth-sensing camera, if the ambient light intensity is ∈ [0, 10) lux, the gesture command weight adjustment coefficient is -0.2; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.1; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.05; and the adjustment range does not exceed ±30% in any case.

[0093] When the core sensor uses a regular 2D camera, if the ambient light intensity is <10 lux, the gesture command weight is set to 0; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.5; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.1.

[0094] Furthermore, the context information includes environmental context information, user state context information, device state context information, and time context information. Environmental context information includes ambient light intensity, ambient noise level, and whether there is human activity. User state context information includes user identity, user location, user movement trajectory, and user activity status. Device state context information includes operating status and device location. Time context information includes time period and instruction reception time sequence.

[0095] Furthermore, the multimodal fusion decision engine includes a complementary fusion module and an arbitration and conflict resolution module. The complementary fusion module is used to process collaborative multimodal instructions, while the arbitration and conflict resolution module is used to process conflicting multimodal instructions according to preset priority strategies and contextual relevance, and to adjudicate the instruction that should be executed in the end.

[0096] Furthermore, the Bayesian network uses user intent as the hidden parent node, modal instructions and context as child nodes, and the relationships between nodes are determined by expert knowledge combined with data mining.

[0097] Furthermore, the deep Q-network includes an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer has a 32-dimensional dimension and is used to receive the raw feature data and pass it to the first hidden layer. The first hidden layer performs preliminary feature extraction and nonlinear transformation on the input features, mapping the 32-dimensional input to a 64-dimensional space. The second hidden layer further abstracts and compresses the features, compressing the 64-dimensional features back to 32 dimensions. The output layer has a 4-dimensional dimension and is used to generate the final prediction result. The first and second hidden layers are activated using the ReLU function. The output layer uses linear activation.

[0098] Furthermore, the feedback methods of the feedback unit include voice feedback, light feedback, APP push feedback, and on-screen text feedback.

[0099] Example 1 (Complementary Fusion: Bayesian Network Intent Mining)

[0100] Scene parameters

[0101] Environment: Living room (30m2), ambient light 50 lux, noise 45dB, no one moving;

[0102] User: User A (Administrator), Location (2.5, 3.0, 2.8) m (0.6 m from the terminal);

[0103] Device: Chandelier (ID:Light_001), current brightness 50%, operating normally;

[0104] Command: Gesture (depth sensing camera) → point to the chandelier (confidence 0.95, target ID: Light_001), Voice → "Turn it brighter" (confidence 0.92, action type: brighten).

[0105] Execution process

[0106] Command reception: If the timestamp difference between two commands is 300ms, they are determined to be synchronous commands;

[0107] Analysis and Perception:

[0108] Voice command parsing: "Brighten it a little" → Action type = brighten, target missing, confidence after calibration 0.92 × 0.95 (environmental coefficient) = 0.874;

[0109] Gesture command parsing: Pointing to the chandelier → Target ID=Light_001, action missing, confidence after calibration 0.95×1.0 (environmental coefficient) = 0.95;

[0110] Context vector: [50,45,True, Administrator, (2.5,3.0,2.8), Still, Running, (2.0,3.5,2.8), Daytime, [t1,t2]];

[0111] Decision-making process:

[0112] Rule engine layer: No security instructions, both modalities are valid;

[0113] Command relationship: complementary (gestures lack action, voice lacks target);

[0114] Bayesian inference: Input evidence (target = Light_001, action = turn on brightness, user = administrator, location = living room), calculate posterior probability:

[0115] P (Intention = Increase chandelier brightness) = 0.97;

[0116] P (Intention = Increase the brightness of the living room spotlights) = 0.02;

[0117] The sum of probabilities of other intentions = 0.01;

[0118] Output composite command: {Command ID:CMD_1234, Target Device ID:Light_001, Action Type: Increase Brightness, Adjustment Amount: 30%, Confidence Level: 0.97};

[0119] Scheduled execution:

[0120] Protocol conversion: Zigbee cluster command 0x0008 (brightness adjustment), data = 80 (50% + 30%);

[0121] Device response: The chandelier returns "Execution successful" within 50ms after receiving the command;

[0122] Feedback: The environment is quiet (noise level 45dB), and voice feedback ("The chandelier brightness has been increased") + light feedback (green light stays on for 2 seconds) is used.

[0123] Actual measurement results

[0124] Intent recognition accuracy: 97% (consistent with the posterior probability of Bayesian inference);

[0125] Command execution latency: The total latency from command issuance to feedback completion is 420ms (receiving 10ms + parsing 30ms + decision 30ms + execution 50ms + feedback 200ms).

[0126] User experience: No lengthy instructions required, natural interaction, 98% satisfaction rate (based on 100 tests in the same scenario).

[0127] Example 2 (Conflict Arbitration: Hierarchical Weights + Deep Reinforcement Learning)

[0128] Scene parameters

[0129] Environment: Bedroom (15m2), time 22:30 (night), light 5 lux, noise 35dB;

[0130] Users: User A (Administrator, location (1.8, 2.2, 2.8)m, distance from terminal 0.5m), User B (Child, location (5.0, 3.0, 2.8)m, distance from terminal 6m);

[0131] Equipment: Air conditioner (ID: AC_001), current temperature 26℃, operating normally;

[0132] Command: User A gestures (depth sensing camera) → turn off the air conditioner (confidence 0.88), User B APP → adjust to 30℃ (confidence 1.0).

[0133] Execution process

[0134] Command reception: If the timestamp difference between two commands is 200ms, they are determined to be synchronization commands;

[0135] Analysis and Perception:

[0136] Gesture command analysis: Action = Turn off air conditioner, Target = AC_001, Post-calibration confidence 0.88 × 0.9 (light 5 lux adjustment factor - 0.1) = 0.792;

[0137] APP instruction parsing: Action = Increase temperature to 30℃, Target = AC_001, Post-calibration confidence level 1.0 × 1.0 = 1.0;

[0138] Context vector: [5,35,True,Admin / (Child),(1.8,2.2,2.8) / (5.0,3.0,2.8),Still / Still,Running,(2.0,2.5,2.8),Night,[t1,t2]];

[0139] Decision-making process:

[0140] Rule engine layer: No security instructions, both modalities are valid;

[0141] Command relationship: Conflict (shutdown vs. warmup);

[0142] Hierarchical weight calculation:

[0143] User A (gesture, base weight 0.7): Environment adjustment coefficient -0.1, user status adjustment coefficient +0.2 (administrator) +0.1 (near distance) = +0.3, time coefficient 0 → final weight = 0.7×(1-0.1+0.3+0)=0.84;

[0144] User B (APP, base weight 0.8): Environment adjustment coefficient 0, user status adjustment coefficient -0.3 (child) -0.2 (distant distance) = -0.5, time coefficient 0 → final weight = 0.8 × (1 + 0 - 0.5 + 0) = 0.4;

[0145] DQN arbitration: Input state vector (including weights 0.84 / 0.4 and context information), output execute user A's command;

[0146] Scheduled execution:

[0147] Protocol conversion: Wi-Fi command (UDP port 5005), data = turn off air conditioner;

[0148] Equipment response: The air conditioner returns "Execution successful" within 100ms and stops running;

[0149] Feedback: It's quiet at night. User A receives voice feedback ("Air conditioner is off"), and User B receives push notification from the app ("Command ignored, current operation initiated by administrator").

[0150] Actual measurement results

[0151] Arbitration accuracy rate: 100% (meets user expectations, no complaints);

[0152] Error rate: 0 (100 tests of the same scenario, no erroneous execution occurred);

[0153] Latency: Total latency 550ms (receive 10ms + parsing 30ms + decision 50ms + execution 100ms + feedback 200ms).

[0154] Example 3 (Software-Hardware Collaboration: Robustness in Complex Environments)

[0155] Scene parameters

[0156] Environment: Kitchen (10m2), ambient noise 75dB, light 8lux, user position (1.0, 1.5, 2.8)m (0.3m away from terminal);

[0157] Device: Range Hood (ID: RangeHood_001), currently off;

[0158] Commands: Gesture (depth sensing camera) → Turn on the range hood (confidence 0.90), Voice → "Turn off the range hood" (confidence 0.65).

[0159] Execution process:

[0160] Command reception: The timestamp difference between two commands is 150ms, indicating synchronized commands;

[0161] Analysis and Perception:

[0162] Gesture command parsing: Action = Open, Target = RangeHood_001, Post-calibration confidence 0.90 × 0.95 (light 8 lux adjustment coefficient - 0.05) = 0.855;

[0163] Voice command parsing: Action = Off, Target = RangeHood_001, Post-calibration confidence 0.65 × 0.8 (noise 75dB adjustment factor - 0.2) = 0.52;

[0164] Context vector: [8,75,True, Normal user, (1.0,1.5,2.8), Move, Close, (1.2,1.8,2.8), Daytime, [t1,t2]];

[0165] Decision-making process:

[0166] Rule Engine Layer: Noise 75dB > 70dB, speech confidence 0.52 < 0.7 → Mark the speech command as "unreliable" and reduce its weight to 0;

[0167] Command relationship: Single effective modality (gesture);

[0168] Output command: Turn on the range hood;

[0169] Scheduled execution:

[0170] Protocol conversion: Bluetooth 5.2 command, data = start (medium speed);

[0171] Device response: The range hood starts within 50ms and returns "Execution successful";

[0172] Feedback: In noisy environments (75dB), feedback is provided via light (the range hood indicator light stays on) + APP push notification ("The range hood has been started").

[0173] Actual test results:

[0174] Accuracy of complex environment recognition: 85.5% (gesture commands are executed effectively);

[0175] Interaction reliability in noisy environments: 90% (90 out of 100 tests correctly recognized valid commands);

[0176] No feedback omissions: Users receive feedback from both the light and the app, achieving a 100% reception rate (compared to the 40% reception rate of single voice feedback).

[0177] Technical verification data:

[0178] 1. Core performance indicator testing

[0179] 2. Hardware compatibility testing

[0180] 3. Long-term stability testing

[0181] Test conditions: Continuous operation for 72 hours, simulating 1000 daily interactions (including voice, gestures, and APP commands, in a ratio of 3:3:4).

[0182] Test results:

[0183] The system experienced no crashes or lags, with an average response latency of around 450ms.

[0184] The accuracy of intent recognition fluctuates by ≤±1.5%;

[0185] Device connection stability ≥99.5% (only 5 brief offline incidents, with automatic reconnection successful);

[0186] Online learning performance of the reinforcement learning model: After 72 hours, the accuracy of conflict arbitration increased from 92% to 93.5%.

[0187] Please see Figure 3 As shown, the present invention also provides a smart home control method based on multimodal fusion, comprising the following steps:

[0188] S1. Multimodal command reception: The multimodal interaction interface layer receives one or more commands from voice commands, gesture commands, and APP commands in parallel through a multi-threaded listening mechanism, processes the commands, and obtains the raw command data; if the timestamp difference of multiple commands is ≤500ms, it is determined to be a synchronous command; otherwise, it is an asynchronous command and processed according to the time sequence.

[0189] S2, Command Parsing and Context Awareness: The command receiving and parsing unit receives interactive commands from the multimodal interaction interface layer and parses the interactive commands through natural language processing or protocol parsing technology; synchronously with command parsing, context awareness collects various dynamic data in the environment where each smart home device is located, performs filtering and normalization processing, generates context vectors, and updates the system global variables;

[0190] S3, Hierarchical Hybrid Decision Making: The multimodal fusion decision engine intelligently analyzes and makes decisions based on the parsed instructions and various dynamic data acquired by the context-aware unit;

[0191] S4. Command Scheduling and Execution: The scheduling and execution unit generates specific control commands based on the final decision results of the multimodal fusion decision engine;

[0192] S5. Adaptive Feedback: The feedback unit selects an appropriate method to inform the user of the execution result based on the adaptive selection decision process. The specific decision conditions for the adaptive selection logic of the feedback unit are as follows:

[0193] (1) If the ambient noise level is >70dB, speech is excluded; if the ambient noise level is ∈[50,70]dB, the weight of speech is reduced; if the ambient noise level is <50dB, speech is given priority.

[0194] (2). If the ambient light intensity is <10 lux, then exclude the lights; if the ambient light intensity is ∈ [10,50] lux, then adjust the light brightness to the maximum; if the ambient light intensity is ≥50 lux, then adjust the lights to normal brightness.

[0195] (3). If the distance between the user and the terminal is <1m, then voice or light feedback will be given priority; if the distance between the user and the terminal is ∈[1,5]m, then voice / light / screen are all acceptable); if the distance between the user and the terminal is >5m, then APP feedback will be given priority.

[0196] User history preferences: If a user selects APP push notifications 80 times in their last 100 interactions, then APP push notifications will still be prioritized even if the voice condition is met.

[0197] Please see Figure 4 As shown in the figure, a typical application scenario of "complementary integration" is clearly illustrated, in which:

[0198] User intent: The user wants to adjust the brightness of a specific light fixture (chandelier).

[0199] The interaction methods include gestures (target location) and voice (action specification): gestures involve pointing to the chandelier that needs to be controlled, the gesture recognition module (such as a camera) captures this action, and the algorithm identifies that the user is pointing to the "chandelier"; at the same time, the user can speak the command "brighten it a little", the voice recognition module converts the speech into text and parses the action as "brighten".

[0200] System Integration Processing: Taken individually, the gesture command "chandelier" is not a complete control command, and the voice command "brighten" also lacks a control object. The complementary integration module of the central controller combines these two incomplete but complementary pieces of information to accurately generate the final executable command: "Increase the brightness of the chandelier by 30%".

[0201] Execution and Feedback: The system sends the instruction to the chandelier, which smoothly increases its brightness and provides intuitive feedback to the user through the change in the light effect itself.

[0202] like Figure 5 As shown, the fusion decision-making algorithm of this invention does not rely on a single technical path, but instead adopts a hierarchical hybrid architecture that combines rule-based rapid response with intelligent reasoning based on probabilistic models and deep learning to balance reliability, flexibility, and adaptability. The overall architecture of the fusion decision-making algorithm adopts a hierarchical hybrid decision system, and the overall process of the algorithm is as follows: Figure 5 As shown, it clearly illustrates the flow and processing of data across different levels:

[0203] The first layer: rapid filtering and response based on the rule engine; this layer acts as the system's "spinal reflex," handling the most direct and urgent situations to ensure response speed and absolute safety.

[0204] Algorithm selection: Production rule system.

[0205] Rule base example:

[0206] Safety First Rule:

[0207] If any modality contains the keywords “emergency stop” or “alarm”, then immediately execute the safety protocol (such as stopping all mobile devices, turning on all lights in the house, and sounding an alarm) and ignore all other instructions.

[0208] Modal validity rules:

[0209] If ambient noise > 70 dB AND the instruction source is "voice" AND the voice recognition confidence < 0.7, then mark the voice instruction as "unreliable" and reduce its weight to 0.

[0210] If ambient light is less than 10 lux and the gesture command originates from a "2D camera", then mark the gesture command as "unavailable".

[0211] Simple conflict resolution rules:

[0212] If the user is a "child" and the instruction is "raise the air conditioner temperature to 30℃", then the operation will be refused and the message "This is beyond the comfort range" will be displayed.

[0213] Function: This layer can quickly process a large number of clear scenarios without going into complex calculations, which reduces system load and ensures core security.

[0214] The second layer: Intelligent fusion and arbitration based on probability and learning models

[0215] For more complex interactive intentions that pass through the first layer, this layer acts as the "brain" of the system, performing in-depth reasoning.

[0216] a) Algorithms for complementary fusion: Bayesian network-based

[0217] Application scenario: When multiple modal commands arrive successively within a time window, and the information is incomplete but may be complementary (such as the aforementioned "pointer light" + "brighten").

[0218] Model description:

[0219] In this embodiment, a Bayesian network is constructed, where the nodes represent user intent, observed modal instructions (voice text, gesture type, APP event), and contextual variables (user location, time).

[0220] User intent is to hide variables, for example, Intent={TurnOn_LivingRoom_Light, Adjust_Thermostat_Temperature, ...}.

[0221] The probability of an observation node depends on the intent node (i.e., the probability of observing a specific instruction given a certain intent).

[0222] Reasoning process: Set the observed evidence (Speech_Text = "Turn up", Gesture_Target = "Chandelier", User_Location = "Living room") into the network.

[0223] Perform probabilistic reasoning through the network to calculate the posterior probability of all possible intentions: P(Intent | Speech_Text, Gesture_Target, User_Location).

[0224] The intent with the highest posterior probability is selected as the final decision. For example, the probability of P(Intent=Increase_Brightness_of_Chandelier | Evidence) is much higher than other intents, so the system executes "increase the brightness of the chandelier".

[0225] Advantages: It can naturally handle the uncertainty and incompleteness of information, mathematically fuse soft evidence from multiple sources, and derive the most likely user intent.

[0226] b) Algorithm for conflict arbitration: based on deep reinforcement learning

[0227] Application scenario: When multiple modal commands conflict with each other in a short period of time and cannot be resolved by simple rules (such as voice "turn on the light" vs. gesture "turn off the light").

[0228] Model description: A deep Q-network is used as the arbitrator.

[0229] State: s = [Modality_1_Instruction, Modality_1_Confidence, Modality_2_Instruction, Modality_2_Confidence, Context_Vector]. The context vector includes time, environment, user history preferences, etc.

[0230] Actions: a = {Execute_Instruction_1, Execute_Instruction_2, Ask_For_Confirmation, Do_Nothing}. The action space includes executing a specific instruction or taking security measures such as asking for confirmation.

[0231] award:

[0232] Positive reward: After the system performs an action, if the device status is consistent with the user's subsequent behavior or feedback (e.g., the user does not correct it immediately).

[0233] Negative reward: Users quickly cancel or complain after execution.

[0234] Training and online learning:

[0235] In the initial stage of the system, predefined strategies are used (such as prioritizing high confidence and modal default priority).

[0236] Over long-term operation, by collecting actual user interaction data, the DQN network is continuously fine-tuned, enabling it to learn the optimal arbitration strategy for specific users and scenarios. For example, the system may learn that "at night, for user A, when voice and gesture conflict, gesture commands should be executed first because user A does not like to talk at night."

[0237] Advantages: It has long-term learning and adaptive capabilities, enabling it to transcend fixed priority rules and form personalized, scenario-based optimal decisions.

[0238] like Figure 3 As shown, the priority strategy for resolving multimodal instruction conflicts

[0239] This prioritization system is a weighted decision-making model, and its core decision-making process can be summarized as shown in the diagram below. When a conflict occurs, the system will traverse this process and comprehensively calculate to arrive at the final decision:

[0240] Priority core rules (ranked by weight from highest to lowest)

[0241] Level 1: Absolute safety rules (with veto power)

[0242] Rule 1.1: Life safety is the highest priority

[0243] Content: Any identified emergency command issued by any modality (such as "emergency call", "alarm", "stop operation" etc.) or command strongly related to safety (such as the system recommending "open the window" after a gas leak is detected) shall be executed immediately, ignoring all other conflicting commands.

[0244] Example: A user gestures to "turn off the air conditioner," but simultaneously says in a panicked voice, "Alarm!" The system should ignore the "turn off the air conditioner" command and immediately execute the alarm procedure.

[0245] Level 2: Modal Basic Priority (Static Weights)

[0246] When no security instructions are provided, the system presets a basic priority for different interaction modalities, reflecting their default reliability and intent clarity. This priority can be dynamically adjusted by subsequent context.

[0247] Rule 2.1: Physical contact > Non-contact

[0248] Content: App / Touchscreen Control > Gesture Control ≈ Voice Control

[0249] Reason: Physical contact (such as clicking an app button) is generally considered to be more deliberate and intentional, with the lowest probability of accidental triggering.

[0250] Rule 2.2: High-precision mode > Low-precision mode

[0251] Content: If the gesture recognition module is a depth-sensing camera, its recognition accuracy is high, so gesture control > voice control. If it is a regular 2D camera, the priority is lower.

[0252] Reason: High-precision input methods can more accurately reflect the user's intent.

[0253] Level 3: Context-based dynamic adjustment rules (dynamic weight correction)

[0254] This level of rules, based on real-time context information, weights and modifies the basic priorities of the second level, which is key to achieving intelligent decision-making.

[0255] Rule 3.1: Environmental Adaptation

[0256] content:

[0257] In noisy environments: Voice command weight - significantly reduced.

[0258] In dimly lit environments: 2D camera gesture weight - significantly reduced; depth-sensing camera gesture weight - slightly reduced or unchanged; voice / APP weight + increased.

[0259] In public places / quiet environments: the weight of voice commands is reduced (to avoid disturbing others); the weight of gestures / app commands is increased.

[0260] Rule 3.2: User Status and Location Adjustment

[0261] content:

[0262] User proximity principle: Instructions received from the user's most recent interaction modality are given higher weight and boosted. For example, a gesture made by a user standing in front of the gesture camera has a higher weight than an instruction from a distant mobile app.

[0263] User identity: The weight of instructions from administrators / owners is greater than the weight of instructions from guests / children.

[0264] User activity status: If the system detects through sensors that the user is gazing at the target device and making a gesture, the weight of that gesture command is significantly increased.

[0265] Rule 3.3: Time and Series Adjustment

[0266] content:

[0267] Last valid instruction takes precedence: If conflicting instructions are received consecutively within a very short period of time (e.g., 500 milliseconds), the instruction that arrives last has higher weight because it may represent the user's final decision.

[0268] Historical habits: If historical data shows that users use a certain modality more than 80% of the time in a specific scenario (for example, always turning off the lights by voice at night), then the command weight of that modality will be implicitly increased in that scenario.

[0269] Level 4: Final Arbitration and Learning

[0270] If, after weighting according to all the above rules, the priorities of two conflicting instructions are still very close, the system initiates final arbitration.

[0271] Rule 4.1: Conservative Implementation and Confirmation

[0272] Content: Perform operations with lower risk or without changing the status, or proactively send a confirmation request to the user.

[0273] Example: Turning the light on vs. turning the light off conflict, and their weights are similar. If the current ambient light is adequate, prioritize turning the light off (energy saving); if the environment is dark, prioritize turning the light on (safety), or ask the user, "Do you want to turn the light on or off?"

[0274] Rule 4.2: Recording and Learning

[0275] Content: The resolution and outcome of each conflict are recorded and used as training data for the reinforcement learning model, enabling the system to continuously optimize its weight allocation strategy and achieve personalization.

[0276] Examples of strategy application in embodiments of the present invention:

[0277] Scene: Nighttime, dimly lit living room. User A (the host) gestures to turn off the TV in the living room. Almost simultaneously, User B (the guest) issues a command to "increase the TV volume" via a mobile app in the bedroom.

[0278] Conflict resolution process:

[0279] No security instructions.

[0280] Basic priority: App control (physical contact) > Gesture control (contactless).

[0281] Dynamic adjustment:

[0282] Environment: Dim lighting significantly reduces the weight of gestures (assuming a regular camera is being used).

[0283] User location: User A is in the living room, closer to the TV, so their gesture weight is increased to some extent, but not enough to completely offset the weight reduction caused by the environment.

[0284] User identity: User A is the owner, and the weight of their gestures is increased again.

[0285] Overall calculation: After dynamic adjustment, the weight of APP commands is still slightly higher than that of gesture commands.

[0286] Final ruling: The system executes the app's command – "Increase TV volume". Simultaneously, the system provides user A with a gentle feedback through an on-screen notification: "Gesture detected to turn off, ignored."

[0287] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A smart home central control system based on multimodal fusion, characterized in that, include: A multimodal interaction interface layer is used to receive interaction commands from users in at least two different modalities. The multimodal interaction interface layer includes, but is not limited to, a voice interaction module, a gesture interaction module, and an APP interaction module. The device execution layer includes multiple controlled smart home devices; The central controller is communicatively connected to both the multimodal interaction interface layer and the device execution layer. The central controller can be a standalone embedded hardware device, a functional module integrated into a smart speaker / smart screen, or a cloud-edge collaborative distributed system. The central controller includes an instruction receiving and parsing unit, a context-aware unit, a multimodal fusion decision engine, a scheduling and execution unit, and a feedback unit. Users generate interaction instructions through the multimodal interaction interface layer. The instruction receiving and parsing unit receives these instructions from the multimodal interaction interface layer and parses them using natural language processing or protocol parsing techniques. The context-aware unit obtains various dynamic data from the device execution layer regarding the environment in which each smart home device is located, providing the contextual information needed for decision-making. The multimodal fusion decision engine is trained based on a hierarchical hybrid decision architecture, which includes a rule engine layer, a probabilistic inference layer, and a reinforcement learning layer. The rule engine layer uses CLIPS rule language to build a rule base, the probabilistic inference layer uses a Bayesian network to achieve intent fusion, and the reinforcement learning layer uses a deep Q-network. The parsed instructions and various dynamic data acquired by the context awareness unit are sent to the multimodal fusion decision engine for intelligent analysis and decision-making. Finally, the scheduling and execution unit generates specific control commands, which are sent to the corresponding smart home devices in the device execution layer to drive them to perform operations. The feedback unit is used to inform the user of the execution results.

2. The smart home central control system according to claim 1, characterized in that, The core sensor of the gesture interaction module is a structured light depth-sensing camera or an RGB camera; the specific weight adjustment rules of the gesture interaction module are as follows: When the core sensor is a depth-sensing camera, if the ambient light intensity is ∈ [0, 10) lux, the gesture command weight adjustment coefficient is -0.2; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.1; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.05; and the adjustment range does not exceed ±30% in any case. When the core sensor uses a regular 2D camera, if the ambient light intensity is <10 lux, the gesture command weight is set to 0; if the ambient light intensity is ∈ [10, 50) lux, the gesture command weight adjustment coefficient is -0.5; if the ambient light intensity is ∈ [50, 1000) lux, the gesture command weight adjustment coefficient is 0; if the ambient light intensity is ≥1000 lux, the gesture command weight adjustment coefficient is -0.

1.

3. The smart home central control system according to claim 1, characterized in that, The context information includes environmental context information, user state context information, device state context information, and time context information. The environmental context information includes ambient light intensity, ambient noise level, and whether there is human activity. The user state context information includes user identity, user location, user movement trajectory, and user activity status. The device state context information includes operating status and device location. The time context information includes time period and instruction reception time sequence.

4. The smart home central control system according to claim 1, characterized in that, The multimodal fusion decision engine includes a complementary fusion module and an arbitration and conflict resolution module. The complementary fusion module is used to process collaborative multimodal instructions, and the arbitration and conflict resolution module is used to process conflicting multimodal instructions according to a preset priority strategy and contextual relevance, and to decide the instruction that should be executed in the end.

5. The smart home central control system according to claim 1, characterized in that, The Bayesian network uses user intent as the hidden parent node and modal instructions and context as child nodes. The relationships between nodes are determined by expert knowledge combined with data mining.

6. The smart home central control system according to claim 1, characterized in that, The deep Q-network comprises an input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer has a 32-dimensional dimension and is used to receive raw feature data and pass it to the first hidden layer. The first hidden layer performs preliminary feature extraction and nonlinear transformation on the input features, mapping the 32-dimensional input to a 64-dimensional space. The second hidden layer further abstracts and compresses the features, compressing the 64-dimensional features back to 32 dimensions. The output layer has a 4-dimensional dimension and is used to generate the final prediction result. The first and second hidden layers are activated using the ReLU function. The output layer uses linear activation.

7. The smart home central control system according to claim 1, characterized in that, The feedback methods of the feedback unit include voice feedback, light feedback, APP push feedback, and screen text feedback.

8. A smart home control method for the system as described in any one of claims 1 to 7, characterized in that, Includes the following steps: S1. Multimodal command reception: The multimodal interaction interface layer receives one or more commands from voice commands, gesture commands, and APP commands in parallel through a multi-threaded listening mechanism, processes the commands, and obtains the raw command data; if the timestamp difference of multiple commands is ≤500ms, it is determined to be a synchronous command; otherwise, it is an asynchronous command and processed according to the time sequence. S2, Command Parsing and Context Awareness: The command receiving and parsing unit receives interactive commands from the multimodal interaction interface layer and parses the interactive commands through natural language processing or protocol parsing technology; synchronously with command parsing, context awareness collects various dynamic data in the environment where each smart home device is located, performs filtering and normalization processing, generates a context vector, and updates the system global variables; S3. Layered Hybrid Decision Making: The multimodal fusion decision engine intelligently analyzes and makes decisions based on the parsed instructions and various dynamic data acquired by the context-aware unit. S4. Command Scheduling and Execution: The scheduling and execution unit generates specific control commands based on the final decision result of the multimodal fusion decision engine; S5. Adaptive Feedback: The feedback unit selects an appropriate method to inform the user of the execution result based on the adaptive selection and decision process.

9. The method according to claim 8, characterized in that, In step S5, the specific determination condition of the adaptive selection logic of the feedback unit is as follows: (1) If the ambient noise level is >70dB, speech is excluded; if the ambient noise level is ∈[50,70]dB, the weight of speech is reduced; if the ambient noise level is <50dB, speech is given priority. (2). If the ambient light intensity is <10 lux, then exclude the lights; if the ambient light intensity is ∈ [10,50] lux, then adjust the light brightness to the maximum; if the ambient light intensity is ≥50 lux, then adjust the lights to normal brightness. (3) If the distance between the user and the terminal is <1m, then voice or light feedback will be given priority; if the distance between the user and the terminal is ∈[1,5]m, then voice / light / screen are all acceptable); if the distance between the user and the terminal is >5m, then APP feedback will be given priority.

Images