An intelligent hub system and control method

Abstract

The application provides a smart hub system and a control method. A hardware platform in the smart hub system is used to collect video data and audio data, and send the video data and the audio data to an AI agent through a hardware management module; the AI agent performs AI inference based on the video data and the audio data, and sends a control signal to the hardware platform through the hardware management module; the hardware platform is also used to control an external device and / or send a prompt message to the outside in response to the control signal. The smart hub system provided by the application is provided with a hardware platform, the hardware platform has multiple external interfaces, thereby improving the open expansibility of device access; the AI agent and the hardware platform are communicatively connected through the hardware management module, thereby improving the hardware deep integration of the agent framework; the smart hub system is provided with the AI agent, thereby improving the real-time processing of visual AI and the semantic understanding ability of a cloud large model.

Description

Technical Field

[0001] This invention relates to the field of smart home technology, and in particular to a smart central system and control method. Background Technology

[0002] After years of development, the smart home industry has seen continuous evolution in its technological approaches. Currently, the main technological approaches used in home smart hubs include the following: 1. Pure cloud-based processing architecture. This architecture uploads all audio and video data from the environment to cloud servers for AI inference and storage. It suffers from significant data privacy and security risks, insufficient real-time performance, and strong network dependence.

[0003] 2. Local closed control architecture. This architecture connects to specific brands of home appliances through built-in protocol stacks such as Matter, Zigbee, and Bluetooth Mesh. It suffers from drawbacks such as a closed ecosystem, poor scalability, and a lack of AI inference capabilities.

[0004] 3. Edge computing gateway architecture. This architecture uses a two-dimensional multimodal data fusion algorithm and triple compression technology to compress large multimodal models and deploy them at the edge. However, it suffers from a disconnect between AI capabilities and device control, and insufficiently intelligent task scheduling.

[0005] 4. Desktop application of intelligent agent framework. Although the currently popular AI Agent framework has multi-channel access, memory system and autonomous decision-making capabilities, its application scenarios are limited to desktop and server environments and lack deep integration with embedded edge NPU hardware: it has the following defects: (1) It does not support edge visual reasoning acceleration; (2) It lacks a hardware abstraction layer and cannot directly call peripherals such as cameras and microphones; (3) It lacks a docking mechanism with lightweight edge models.

[0006] 5. Single-function AI visual detection solution. This architecture is limited to visual alarm capabilities and does not include device control, voice interaction, data management, or other functions, thus failing to form a true home smart hub system.

[0007] Therefore, existing technologies are developing independently in four dimensions: real-time processing of visual AI, semantic understanding capabilities of large cloud models, open scalability of device access, and deep hardware integration of intelligent agent frameworks. They have not yet formed an organically integrated technical solution. Summary of the Invention

[0008] The main objective of this invention is to provide an intelligent central system and control method, which aims to solve the technical problem that existing technologies have developed independently in four dimensions: real-time processing of visual AI, semantic understanding capability of large cloud models, open scalability of device access, and deep hardware integration of intelligent agent framework, without forming an organically integrated technical solution.

[0009] To achieve the above objectives, the present invention provides an intelligent central system, which includes a hardware platform, an AI agent, and a hardware management module. The AI ​​agent and the hardware platform are communicatively connected through the hardware management module. The hardware platform is used to collect video data and audio data, and send the video data and audio data to the AI ​​agent through the hardware management module. The AI ​​agent performs AI reasoning based on the video data and the audio data, and sends control signals to the hardware platform through the hardware management module. The hardware platform is also used to control external devices and / or send prompt messages in response to the control signal.

[0010] Optionally, the hardware platform includes a control chip; The control chip includes a central processing unit, a neural network processor, a graphics processor, a multimedia encoding / decoding module, and an ISP image signal processor; The control chip is also equipped with a video acquisition interface, a voice input interface, a voice output interface, a video output interface, a network communication interface, a USB interface, a GPIO interface, and a debugging interface.

[0011] Optionally, the AI ​​agent includes a gateway layer, an agent runtime layer, a skill layer, and a memory system that are interconnected. The gateway layer is used for protocol conversion and routing, session management, authentication and security management, and traffic scheduling and management. The Agent runtime layer is used to perform intent parsing operations, task planning operations, skill invocation operations, state management, and feedback. The skill layer is used for visual perception processing, voice input processing, voice output processing, plugin management, and media playback. The memory system is used to cache dialogue context, store operation logs, store long-term memory data, and vectorize historical dialogues and knowledge bases.

[0012] Optionally, the AI ​​agent further includes a processing module and a plug-in management module, both of which are deployed on the AI ​​agent.

[0013] Optionally, the processing module includes a visual skill unit, a model dynamic scheduling unit, and a voice interaction channel; The visual skill unit is used to load AI models, preprocess video frames, perform inference scheduling operations, perform post-processing operations, distribute results externally, and issue external alarms. The model dynamic scheduling unit is used to dynamically allocate computing power according to the operating mode and task priority; The voice interaction channel is used to detect wake words, recognize speech, synthesize speech, and perform intent routing operations.

[0014] Optionally, the plug-in management module includes a device abstraction layer, a plug-in engine, and pre-built access plug-ins; The device abstraction layer is used to define a unified device model interface; The plugin engine is used to support the dynamic installation, uninstallation, activation, and disabling of plugins.

[0015] Optionally, the AI ​​agent adopts a two-level task diversion mechanism: tasks requiring real-time response, such as visual AI reasoning, are deployed on the edge NPU for execution; tasks requiring deep semantic understanding are matched through a local edge rule base, and if a match is found, they are executed directly, otherwise they are forwarded to the cloud for processing by a large model.

[0016] Optionally, the intelligent central system further includes a data management module, which is electrically connected to the hardware platform and includes a backup unit and a web management interface. The backup unit is used to perform automatic backup operations; The web management interface is used to provide a web entry point.

[0017] Furthermore, to achieve the above objectives, the present invention also provides a control method for an intelligent central system, wherein the method is applied to the intelligent central system as described in any of the preceding claims, the method comprising: Collect video and audio data; AI reasoning is performed based on the video data and the audio data to determine the corresponding working mode of the intelligent central system; Based on the aforementioned working mode, control external devices and / or send notification messages to external devices.

[0018] Optionally, the method further includes: Collect audio data; If the audio data is detected to include a preset wake word, the audio data is converted into text data; Perform intent recognition on the text data; If the intent recognition result of the text data represents device control, then the external device is controlled based on the text data; If the intent recognition result of the text data represents a question-and-answer dialogue, then the question-and-answer voice is sent out based on the text data.

[0019] Optionally, the method further includes: Configure the first process to call the AI ​​model deployed in the cloud; Configure a second process to run the home smart service system; A third process is configured to run voice acquisition, wake word detection, speech recognition and synthesis, and overlay display functions. The third process displays alarm information through the overlay, and the alarm information does not interfere with the output screen of the preset media player. When the second process receives a voice command or text command sent by the third process, it matches the voice command or text command in the locally configured edge rule base. If a match is successful, control commands are sent out based on the matching result; If a match fails, the voice command or text command is forwarded to the first process, so that the first process invokes an AI model deployed in the cloud to respond to the voice command or text command. Compared with the prior art, the embodiments of this application have the following main advantages: This application provides an intelligent central system and control method. The intelligent central system includes a hardware platform, an AI agent, and a hardware management module. The AI ​​agent and the hardware platform are communicatively connected through the hardware management module. The hardware platform is used to collect video and audio data and send the video and audio data to the AI ​​agent through the hardware management module. The AI ​​agent performs AI inference based on the video and audio data and sends control signals to the hardware platform through the hardware management module. The hardware platform is also used to control external devices and / or send prompt messages in response to the control signals. The intelligent central system provided by this application has a hardware platform with multiple external interfaces, thereby improving the openness and scalability of device access. The AI ​​agent and the hardware platform are communicatively connected through the hardware management module, improving the hardware depth integration of the agent framework. The intelligent central system includes an AI agent, thereby improving the real-time processing of visual AI and the semantic understanding capabilities of large cloud models. Attached Figure Description

[0020] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is one of the structural schematic diagrams of the intelligent central system provided in the embodiments of this application; Figure 2 This is a second schematic diagram of the intelligent central system provided in the embodiments of this application; Figure 3 This is one of the flowcharts illustrating the control method of the intelligent central system provided in the embodiments of this application; Figure 4 This is the second flowchart illustrating the control method of the intelligent central system provided in the embodiments of this application.

[0022] Explanation of reference numerals in the attached figures: 100. Hardware Platform; 200. Hardware Management Module; 300. AI Agent; 310. Gateway Layer; 320. Agent Runtime Layer; 330. Skill Layer; 340. Memory System; 400. Processing Module; 500. Plugin Management Module; 410. Visual Skill Unit; 420. Model Dynamic Scheduling Unit; 430. Voice Interaction Channel; 510. Device Abstraction Layer; 520. Plugin Engine; 530. Pre-built Access Plugin; 600. Data Management Module; 610. Backup Unit; 620. Web Management Interface; 700. Playback Module. Detailed Implementation

[0023] All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a particular order.

[0024] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0025] Please see Figure 1 , Figure 1 This is one of the structural schematic diagrams of the intelligent central system provided in the embodiments of this application. For example... Figure 1 As shown, the intelligent central system provided in this application includes a hardware platform 100, an AI agent 300, and a hardware management module 200. The AI ​​agent 300 and the hardware platform 100 are communicatively connected through the hardware management module 200. The hardware platform 100 is used to collect video data and audio data, and send the video data and audio data to the AI ​​agent 300 through the hardware management module 200. The AI ​​agent 300 performs AI reasoning based on the video data and the audio data, and sends control signals to the hardware platform 100 through the hardware management module 200. The hardware platform 100 is also used to control external devices and / or send prompt messages in response to the control signal.

[0026] The intelligent central system provided in this application includes a hardware platform 100, an AI agent 300, and a hardware management module 200. The AI ​​agent 300 and the hardware platform 100 are connected through the hardware management module 200.

[0027] It should be understood that an AI agent comprises multiple independent processes deployed on an edge computing platform, including at least: The OpenClaw gateway process provides a WebSocket interface, agent looping, skill registration, and cloud-based LLM calls. The HISS Core service process includes a message bus, business modules, API services, and gateway clients. The voice and display overlay process is managed independently, including microphone acquisition, wake word detection, speech recognition / synthesis, and HDMI overlay display.

[0028] The aforementioned hardware management module includes a Hardware Abstraction Layer (HAL), which encapsulates driver interfaces such as V4L2, ALSA, GPIO, and DRM / KMS into standardized skills for unified invocation by the AI ​​agent. The workflow of the intelligent central system provided in this embodiment is as follows: the hardware platform 100 collects video and audio data and sends them to the AI ​​agent 300 via the hardware management module 200. The AI ​​agent 300 performs AI inference based on the video and audio data and sends control signals to the hardware platform 100 via the hardware management module 200. In response to the control signals, the hardware platform 100 controls external devices and / or sends notification messages.

[0029] This embodiment achieves deep integration between the AI ​​agent 300 and the hardware platform 100. The hardware management module 200 decouples the upper-layer AI application from the lower-layer hardware driver, enabling the AI ​​agent 300 to uniformly call various hardware peripherals and realize a complete closed loop of perception, reasoning, decision-making, and execution.

[0030] It should be understood that the hierarchical structure of the hardware management module 200 in this embodiment is as follows: The top layer is a standardized API interface: providing a unified JSON-RPC call endpoint.

[0031] The middle layer handles business logic: it converts API calls into specific operations for the underlying driver subsystem.

[0032] The lowest layer is the driver interface encapsulation: directly calling the Linux kernel driver interface. For the V4L2 video driver, the ioctl() system call is used to read the frame buffer; for the ALSA audio driver, the asoundlib library is used for reading and writing PCM data and controlling mixing; for the GPIO subsystem, control is achieved through the / sys / class / gpio / file system; and for the DRM / KMS display driver, the libdrm library is used for frame buffer management and display output.

[0033] In this embodiment, by developing a hardware abstraction layer, hardware peripherals such as cameras, microphones, and GPIOs are encapsulated into standardized skills that can be called by the skill layer 330, filling a technological gap in the field of embedded edge hardware integration.

[0034] The intelligent central system provided in this application embodiment is equipped with a hardware platform 100, which has a variety of external interfaces to improve the openness and scalability of device access; the AI ​​agent 300 is connected to the hardware platform 100 through a hardware management module 200 to improve the hardware deep integration of the agent framework; the intelligent central system is equipped with an AI agent 300 to improve the real-time processing of visual AI and the semantic understanding capability of cloud-based large models.

[0035] Optionally, the hardware platform 100 includes a control chip; The control chip includes a central processing unit, a neural network processor, a graphics processor, a multimedia encoding / decoding module, and an ISP image signal processor; The control chip is also equipped with a video acquisition interface, a voice input interface, a voice output interface, a video output interface, a network communication interface, a USB interface, a GPIO interface, and a debugging interface.

[0036] Optionally, the aforementioned control chip adopts an 8nm advanced process technology and integrates a central processing unit, a neural network processor, a graphics processor, a multimedia codec module, and an ISP image signal processor.

[0037] In one alternative implementation, the neural network processor is a tri-core NPU. The three NPU cores can work collaboratively or run independently, converting pre-trained visual AI models into RKNN format and utilizing the neural network processor for hardware-accelerated inference. The neural network processor supports dynamic voltage and frequency adjustment technology, achieving different computing power outputs within a power consumption range of 0.5W to 3W.

[0038] In an alternative implementation, the graphics processor supports OpenGL ES 3.2, OpenCL 2.2, and Vulkan 1.2 for touchscreen UI rendering and graphical interface acceleration.

[0039] In an optional implementation, the multimedia codec module supports 8K@60fps video decoding and 8K@30fps video encoding, providing support for home entertainment functions.

[0040] In one alternative implementation, the ISP image signal processor can perform functions such as lens shadow correction, 2D / 3D noise reduction, sharpening and dehazing, and fisheye correction to improve the image quality captured by the camera.

[0041] In one alternative implementation, the control chip connects to a 720P or 1080P resolution camera via a video capture interface. Multiple USB cameras can also be added via a USB 3.0 interface to achieve multi-area monitoring.

[0042] In an alternative implementation, a dual-microphone or quad-microphone array is connected to the control chip via a voice input interface to achieve far-field sound pickup and echo cancellation.

[0043] In an alternative implementation, the speaker is connected to the control chip via a voice output interface.

[0044] In one alternative implementation, the video output interface supports a maximum resolution output of 4K@120fps or 8K@60fps for connecting an external TV as an extended display terminal to simultaneously display the system interface and media content.

[0045] In one optional implementation, the Gigabit Ethernet physical layer chip is connected to the control chip via a network communication interface; the wireless communication network is connected via the network communication interface and supports dual-band communication of 2.4 GHz and 5 GHz.

[0046] In one alternative implementation, the USB interface is a USB 3.0 Type-A interface, used to expand peripherals such as USB cameras, external storage devices, keyboards, and mice.

[0047] In one alternative implementation, the GPIO interface has 2-4 programmable GPIO pins for connecting external devices such as relays and sensors.

[0048] In an alternative implementation, the test interface is used for system debugging and log output.

[0049] Optionally, the control chip is also connected to a storage module, specifically including: Memory: 4GB or 8GB LPDDR4X memory chips, soldered to the adjacent position of the controller chip and connected via a high-speed DDR bus.

[0050] System storage: 32GB eMMC 5.1 memory, connected to the control chip via SDIO interface, used to store the Ubuntu 22.04 operating system, OpenClaw runtime, RKNN-Toolkit runtime library, and pre-installed visual model files.

[0051] Data storage: NVMe SSDs are connected to the PCIe bus of the controller chip via an M.2 interface for storing photo backups, video surveillance recordings, and multimedia files.

[0052] Optionally, the control chip housing adopts a passive heat dissipation design, and no active fan is installed on the circuit board. Heat is conducted to the metal heat sink fins through thermal grease on the top of the chip. The heat sink fins are in contact with the inner surface of the housing, and natural air convection is used to dissipate heat from the surface of the housing.

[0053] This embodiment provides a hardware platform 100 with powerful AI computing power, rich peripheral interfaces and good heat dissipation performance, providing a hardware foundation for the efficient operation of the AI ​​agent 300.

[0054] It should be understood that the integration of the OpenClaw framework with the control chip adopts the following technical path: Path 1: Environment Adaptation Layer. Configure the Node.js runtime environment, Python environment, and system dependencies required by OpenClaw on the control chip's system.

[0055] Path Two: Channel Development. Develop a custom LocalVoiceChannel that inherits from OpenClaw's standard Channel interface.

[0056] Path 3: Skill Development. Develop two customized skills: VisionSkill and PluginManagerSkill.

[0057] Path 4: HAL Service Deployment. Encapsulate V4L2 / ALSA / GPIO / DRM as a JSON-RPC service for use by skills such as VisionSkill.

[0058] Path 5: System startup process: After the system is powered on, Ubuntu starts → HAL service starts (systemd) → OpenClaw service starts → Loads Channel and Skill configurations → Gateway listens on the port and maintains long connections with each Channel → Immich / Kodi / Homeassistant → HISS Core → Voice / Overlay → Enters standby mode.

[0059] Path 6: Hybrid communication between OpenClaw and HISS, where WebSocket is used for task / event pushing and REST is used for status query.

[0060] Path 7: The design motivation for the independent process of voice and display (hiss_voice_overlay) is to avoid resource competition between ALSA / Qt and business logic, and to ensure that the alarm overlay is displayed normally when playing in full screen in Kodi.

[0061] Please see Figure 2 Optionally, the AI ​​agent 300 includes a gateway layer 310, an agent runtime layer 320, a skill layer 330, and a memory system 340 that are interconnected. The gateway layer 310 is used for protocol conversion and routing, session management, authentication and security management, and traffic scheduling and management. The Agent runtime layer 320 is used to perform intent parsing operations, task planning operations, skill invocation operations, state management, and feedback. The skill layer 330 is used for visual perception processing, voice input processing, voice output processing, plugin management, and media playback. The memory system 340 is used to cache dialogue context, store operation logs, store long-term memory data, and perform vectorized storage of historical dialogues and knowledge bases.

[0062] The AI ​​agent 300 in this embodiment includes a gateway layer 310, an agent runtime layer 320, a skill layer 330, and a memory system 340 that are interconnected. Optionally, the AI ​​agent 300 is built based on the OpenClaw open-source AI Agent framework and uses Node.js as its runtime environment.

[0063] The aforementioned gateway layer 310 is used for protocol conversion and routing, session management, authentication and security management, as well as traffic scheduling and management.

[0064] Specifically, the gateway layer 310 is the central control plane of the AI ​​agent 300, and undertakes the following core responsibilities: Protocol Conversion and Routing: Gateway Layer 310 supports multiple communication protocols such as HTTP / HTTPS, WebSocket, and gRPC. Gateway Layer 310 uniformly converts requests from the voice channel, touch GUI, and external APIs into standardized internal event streams and routes them to the corresponding Agent or Skill based on the event type.

[0065] Session Management: Gateway layer 310 maintains a unified session context across channels, ensuring the continuity of conversations and consistency of identity for the same user across different interaction methods. Session state is persistently stored using Redis.

[0066] Authentication and security management: Gateway layer 310 implements API access control based on JWT tokens, supports API Key whitelists and rate limiting policies to prevent API abuse and unauthorized access.

[0067] Traffic scheduling and management: The gateway layer 310 has built-in rate limiting policies and queue buffering mechanisms to protect the stability of the backend agent in high-concurrency scenarios.

[0068] In this embodiment, the gateway layer 310 is configured to establish connections with the local voice channel, touch GUI service and cloud LLM API provider, becoming a unified entry point for all user requests.

[0069] The Agent runtime layer 320 described above is used to perform intent parsing, task planning, skill invocation, state management, and feedback. The Agent runtime layer 320 is the core decision-making unit of the AI ​​agent 300, responsible for transforming natural language objectives into an executable sequence of operations. Specifically: Intent parsing: Agent runtime layer 320 uses a lightweight natural language understanding model to extract entities and operation intents from user requests and identify the type of task the user wants to complete.

[0070] Task planning: Agent runtime layer 320 implements skill call sequence planning based on DAG graph, decomposing complex tasks into a series of atomic operation steps.

[0071] Skill Invocation: The Agent runtime layer 320 converts the planned steps into function calls to specific skills one by one, and sends execution instructions to the skill layer 330 through a standardized interface.

[0072] State Management and Feedback: The Agent runtime layer 320 maintains persistent storage for workflow instances via Redis, supporting breakpoint resumption. After receiving the Skill execution result, the Agent runtime layer 320 organizes it into natural language feedback content and sends it back to the user through the gateway layer 310.

[0073] The aforementioned skill layer 330 is used for visual perception processing, voice input processing, voice output processing, plugin management, and media playback.

[0074] Skill Layer 330 is a pluggable business capability library for AI Agent 300, employing a three-tier loading mechanism: Bundled (built-in skills), Managed (managed skills), and Workspace (workspace skills). Each skill must implement a standardized interface, including input specifications, execution logic, and output format.

[0075] Optionally, the skill layer 330 in this embodiment has developed and integrated the following customized skills: VisionSkill (Visual Perception Skill): Used for visual perception processing. VisionSkill calls the NPU of the hardware platform 100 to perform YOLOv8 model inference. VisionSkill receives JSON commands in the format {task: "detect", model: "fall_detection", input: "camera_frame"}, calls the NPU inference engine, and returns detection results in the format {result: "fall_detected", confidence: 0.87, bbox: [x1,y1,x2,y2]}.

[0076] AudioInputSkill: Used for voice input processing. AudioInputSkill captures audio streams by calling the microphone array via HAL, and includes integrated calls to Vosk's local ASR.

[0077] AudioOutputSkill: Used for voice output processing. AudioOutputSkill calls the speaker to play TTS audio through HAL, including integrated calls to the Piper / eSpeak native TTS engine.

[0078] PluginManagerSkill: Used for plugin management. PluginManagerSkill manages the installation, uninstallation, activation, and deactivation of third-party device access plugins by calling the plugin manager API.

[0079] MediaPlaybackSkill: Used for media playback. MediaPlaybackSkill utilizes the VPU hardware decoder of the hardware platform 100 to play multimedia files and supports HDMI output.

[0080] The aforementioned memory system 340 is used to cache dialogue context, store operation logs, store long-term memory data, and vectorize historical dialogues and knowledge bases. The memory system 340 presents a four-layer memory architecture, enabling the AI ​​agent 300 to possess stateful agent behavior.

[0081] Specifically, the memory system 340 has the following functions: Session Memory: The memory system caches context information from the current conversation turn, supporting coherent understanding of multi-turn conversations. The session context is stored in Redis and has a time limit.

[0082] Daily Log: The memory system 340 records the user's daily operations and the agent's execution records, storing them in a structured log format in a local database to form user behavior profiles and activity summaries.

[0083] Long-term memory data storage (MEMORY.md): The memory system 340 stores persistent data such as user long-term preferences, family member information, and device configuration in the form of a Markdown file.

[0084] Vector Memory is used to store historical dialogues and knowledge bases: The memory system 340 stores historical dialogues and knowledge bases in a vectorized manner, supporting semantic similarity retrieval. When a user asks a question, the memory system 340 first retrieves relevant historical records from the vector memory, which serve as context aids the Agent runtime layer 320 to provide a more accurate answer.

[0085] In this embodiment, the modular and scalable design of the AI ​​agent 300 is realized through the above four-layer architecture, which can realize intelligent task scheduling and context-driven automated decision-making, breaking through the limitations of traditional fixed rule scheduling.

[0086] Please see Figure 2 Optionally, the AI ​​agent 300 further includes a processing module 400 and a plug-in management module 500, both of which are deployed on the AI ​​agent 300.

[0087] In this embodiment, the processing module 400 and the plug-in management module 500 are deployed on the AI ​​agent 300 as a core functional extension of the AI ​​agent 300.

[0088] The processing module 400 is responsible for the multimodal perception and decision-making capabilities of the AI ​​agent 300, including three major functions: visual perception, model scheduling, and voice interaction.

[0089] Among them, the plug-in management module 500 is responsible for the device access expansion capability of the AI ​​intelligent agent 300, realizing the unified access and management of heterogeneous home devices.

[0090] In this embodiment, AI inference capabilities and device access capabilities are deployed on the AI ​​agent 300 in a modular manner, so that the AI ​​agent 300 has both powerful edge AI perception capabilities and open device access expansion capabilities.

[0091] Please see Figure 2 Optionally, the processing module 400 includes a visual skill unit 410, a model dynamic scheduling unit 420, and a voice interaction channel 430; The visual skill unit 410 is used to load AI models, preprocess video frames, perform inference scheduling operations, perform post-processing operations, distribute results externally, and issue external alarms. The model dynamic scheduling unit 420 is used to dynamically allocate computing power according to the operating mode and task priority; The voice interaction channel 430 is used to detect wake words, recognize speech, synthesize speech, and perform intent routing operations.

[0092] In this embodiment, the processing module 400 includes a visual skill unit 410, a model dynamic scheduling unit 420, and a voice interaction channel 430.

[0093] The aforementioned visual skill unit 410 is a customized unit developed based on the OpenClaw skill layer 330 and deployed on the NPU of the hardware platform 100. It can perform the following functions: Loading AI Models: The Visual Skills Unit 410 loads three pre-converted RKNN model files (fall detection, open flame and smoke detection, and personnel intrusion detection) through the RKNN-Toolkit 2 runtime library. The models reside in NPU memory to avoid the time-consuming process of repeated loading.

[0094] Video frame preprocessing: The camera captures a 1080P video stream at 30 frames per second. The vision skill unit 410 periodically captures video frames, scales and normalizes them, converts them to the input size required by the model, and converts the image data into INT8 quantization format before writing it to the NPU input buffer. The preprocessing operation is executed in parallel by the CPU and does not consume NPU computing power.

[0095] Execution of inference scheduling operations: The visual skill unit 410 sends inference requests to the NPU based on the NPU computing power allocated by the model dynamic scheduling unit 420. The three models can be executed concurrently or in a time-sharing manner.

[0096] Post-processing: After NPU inference is completed, the visual skill unit 410 parses the output tensor to extract the bounding box coordinates, confidence scores, and class labels. Overlapping bounding boxes are filtered using non-maximum suppression, retaining valid detection results with confidence scores above a threshold.

[0097] External distribution of results: The detection results are distributed externally through gateway layer 310 in a standardized JSON event format.

[0098] External alarm: When a person falls, open flames or smoke, or suspicious intrusion is detected, the Agent runtime layer 320 receives the event and automatically executes the preset alarm workflow: it sends a message push service request to the guardian's mobile phone, carrying the alarm type, timestamp, and alarm screenshot (Base64 encoded JPEG image); at the same time, it calls the TTS engine to issue a voice alarm through the speaker.

[0099] The dynamic scheduling unit 420 in the above model is used to dynamically allocate computing power based on the operating mode and task priority. The scheduling strategy is configured as follows: First Priority (Highest Priority, Continuously Executed): Configure fall detection and intrusion detection as the highest priority, allocating approximately 60% of the total NPU computing power, and continuously running them at a rate of no less than 30 frames per second. Intrusion detection is activated when the system is in "Security Alert Mode"; fall detection is activated when the system detects someone in the house who is registered as an elderly person / child.

[0100] Second priority (periodic frequency reduction): Configure open flame and smoke detection as the second priority, allocate approximately 10% of the total NPU computing power, and run in a frame rate reduction mode of 1 to 5 frames per second. This detection has lower real-time frame rate requirements (fire and smoke develop relatively slowly), and frequency reduction can significantly save NPU computing power.

[0101] Third Priority (Triggered on Demand): Configure other non-urgent visual analysis tasks as the third priority, including but not limited to activity statistics, device control linkage, and behavior analysis statistics. This priority task will only be executed when the system detects that the NPU utilization rate is below 30% for 3 consecutive minutes, or when the user explicitly triggers it, allocating the remaining approximately 30% of computing power resources.

[0102] The aforementioned voice interaction channel 430 is used for detecting wake words, recognizing speech, synthesizing speech, and performing intent routing operations. The voice interaction channel 430 is integrated with the gateway layer 310 to form a local voice channel. Specifically, the voice interaction channel 430 has the following functions: Wake-up word detection (wake-up word detection unit): Employs the Porcupine offline wake-up engine (provided by PicoVoice), supporting custom wake-up words that can be adjusted via parameters. Porcupine processes audio stream segments at 200ms intervals, confirming wake-up when the same wake-up word is detected three consecutive times with a confidence level higher than 0.8. Wake-up words can be configured as "Hello Little Butler," "Little Butler," "HiHISC," etc., and users can modify them themselves via the web interface.

[0103] Speech Recognition (Local Speech Recognition Unit ASR): The Vosk offline ASR engine is activated upon wake-up. Vosk uses a small edge-side Chinese speech recognition model built on Kaldi, with a vocabulary covering common household control commands and question-and-answer vocabulary. The word error rate is less than 15% in a home environment (SNR≥15dB).

[0104] Audio processing flow: The microphone acquires a 16kHz / 16-bit PCM audio stream, which is then processed by VAD (Voice Activity Detection) to extract valid speech segments. These segments are then fed into the Vosk recognizer to output the recognized text.

[0105] Synthesized speech (local speech synthesis unit TTS): using the local Piper engine or the open-source eSpeak engine. Local TTS is used in the following scenarios: (1) feedback of device control operations; (2) system status announcements; (3) voice prompts for security alarms. The TTS synthesis speed is about 2 to 3 times the real-time rate, that is, 1 second of text requires about 0.3 to 0.5 seconds to synthesize.

[0106] Intent routing operation (intent routing unit): After receiving the text output by ASR, the Agent runtime layer 320 classifies it using a lightweight intent classifier (a hybrid method based on keyword matching, regular expressions, and a rule engine). The intent classifier predefines the following categories: Device control category: Regular expression pattern matching such as "on / off + {device name}" or "adjust {device name} to {brightness / temperature}". For example, "turn on living room light" → "device control category".

[0107] Information query type: Regular expression pattern matching such as "{device name} status" or "{device name} is on or off". For example, "What is the current status of the living room light?" → "Information query type".

[0108] Task execution class: Regular expression pattern matching "help me {execute task description}". For example, "help me check today's weather" → "transfer to cloud LLM API processing".

[0109] Natural Language Question Answering: Text that does not match any of the above categories is directly routed to the cloud LLM API for processing.

[0110] In this embodiment, by configuring the visual skill unit 410, the model dynamic scheduling unit 420, and the voice interaction channel 430, the deep integration of edge-side visual AI and voice interaction is realized, enabling the AI ​​agent 300 to have multimodal perception capabilities, and to process visual events and voice commands simultaneously, thereby realizing intelligent home services.

[0111] Please see Figure 2 Optionally, the plug-in management module 500 includes a device abstraction layer 510, a plug-in engine 520, and a pre-made access plug-in 530. The device abstraction layer 510 is used to define a unified device model interface; The plugin engine 520 is used to support the dynamic installation, uninstallation, activation, and disabling of plugins.

[0112] The aforementioned device abstraction layer 510 is used to define a unified device model interface, which all third-party device access plugins must implement. Thus, once all plugins implement this interface, the AI ​​agent 300 can call devices of different brands and types in a unified way, without needing to worry about differences in the underlying protocols.

[0113] The aforementioned plugin engine 520 supports the dynamic installation, uninstallation, enabling, and disabling of plugins. Plugin engine 520 is implemented based on OpenClaw's Channel and Tool plugin mechanism. Specifically: Plugin package format: The plugin is packaged in ZIP format and contains the following files: plugin.json: Metadata file containing plugin name, version number, author information, description of supported device types, and a list of dependent third-party Python libraries; plugin.py: The core implementation file, which contains concrete class implementations that inherit from DevicePlugin and implement the four interface methods mentioned above; Optionally, it includes pip-formatted dependency files that define the third-party Python packages and versions required for the plugin to run.

[0114] Plugin installation process: Upload ZIP file via Web management interface 620 → Plugin engine 520 verifies ZIP package structure and plugin.json validity → Unzip → Install → Dynamically import → Call the discover() method to discover devices → Register the discovery results to the unified device registry of device abstraction layer 510.

[0115] Dynamic loading and hot-plugging: The plugin manager scans all plugin subdirectories and dynamically imports Python modules that implement standardized plugin interfaces. The plugin management service provides a RESTful API to support the dynamic installation, uninstallation, enabling, and disabling of plugins without requiring a system restart.

[0116] In this embodiment, a pre-installed access plug-in 530 is provided. After installation, all smart devices under the user's account can be seen in the device list and controlled.

[0117] In this embodiment, by configuring a plug-in management module 500 including a device abstraction layer 510, a plug-in engine 520, and pre-made access plug-ins 530, a highly open and scalable device access architecture is achieved. Users can install plug-ins to access third-party devices as needed without waiting for system firmware upgrades, thus breaking through the ecological barriers of traditional intelligent hub systems.

[0118] Please see Figure 2 Optionally, the intelligent central system further includes a data management module 600, which is electrically connected to the hardware platform 100. The data management module 600 includes a backup unit 610 and a Web management interface 620.

[0119] Enhanced home data management and AI integration based on Immich, including: The Immich Docker service (PostgreSQL + Redis + ML) is responsible for automatic backup of mobile phone photos, face recognition, and CLIP semantic search.

[0120] The HISS DataManager module encapsulates the Immich API, providing voice search, daily recall, storage space reminders, and intelligent cleanup suggestions.

[0121] Cross-module integration: Photos can be pushed to Kodi for slideshow playback, creating family commemorative short films.

[0122] The backup unit 610 is used to perform automatic backup operations; The Web management interface 620 is used to provide a Web entry point.

[0123] The backup unit 610 described above is used to perform automatic backup operations. Backup unit 610 uses the open-source photo backup engine Immich to achieve automatic synchronization and backup of photos on the mobile terminal. The specific technical implementation is as follows: Immich uses Docker containerization, and the following services are defined in the docker-compose.yml file: immich-server: The core service, which includes photo upload, management, and browsing functions.

[0124] immich-microservices: Background microservices that handle tasks such as thumbnail generation.

[0125] database: SQLite, persists the database file to the local database in volume mode.

[0126] Redis: A caching service used for task queue management.

[0127] Backup unit 610 has the following optimized features: Disable the built-in AI intelligent classification function of Immich to reduce the additional usage of NPU and CPU.

[0128] The photo storage directory is mapped to the NVMe SSD directory to ensure high write performance for large-capacity photos.

[0129] Configure a regular cleanup task to delete temporary cache files and thumbnail preview cache, freeing up storage space.

[0130] The aforementioned web management interface 620 provides a web entry point. Developed based on the Flask / FastAPI framework, the web management interface 620 provides a unified web entry point for device control, plugin management, system configuration, and photo browsing. Specific functions include: Device control page: Displays a list of all connected devices, supports clicking the switch button to control device on / off, and sliders to adjust parameters such as brightness and temperature; Plugin Management Page: Displays a list of installed plugins, supports uploading new plugin ZIP packages, enabling / disabling plugins, and uninstalling plugins; System configuration page: Supports modifying wake word, configuring guardian's mobile phone push address, setting security alert time period, setting medication schedule, etc.; Photo browsing page: Embedded or reverse-proxied through Immich's web interface, it allows users to browse, download, and manage backed-up photo sets via a browser.

[0131] In this embodiment, by configuring a data management module 600 including a backup unit 610 and a web management interface 620, localized storage management of family data is achieved. All photo data is stored in a local database, and users have complete control over data access permissions, avoiding the privacy leakage risks associated with cloud storage. Meanwhile, the web management interface 620 provides a unified and convenient system management entry point, lowering the barrier to entry for users. Please see Figure 2 Optionally, the intelligent central system further includes a playback module 700, which is electrically connected to the hardware platform 100.

[0132] Based on Kodi's home entertainment playback module and the viewing environment, including: Kodi operates as a separate process, exclusively controlling the HDMI main screen, while the HISS EntertainmentModule controls playback via JSON-RPC.

[0133] Voice on demand: Local rules connect directly to Kodi, and complex searches are parsed by OpenClaw LLM and then called to search in Kodi.

[0134] HDMI Hardware Overlay: The third process displays alarm information through the DRM hardware overlay plane, which is different from the primary display plane occupied by the media player. The two are independent of each other at the hardware level and do not interfere with each other.

[0135] Movie-viewing scene synchronization: Automatically dim the lights and close the curtains when playing a movie, and restore them after the movie stops.

[0136] In this embodiment, the playback module 700 utilizes the 8K video decoding unit built into the control chip to support external display devices connected via the HDMI interface to decode and output 4K and 8K resolution video content.

[0137] Specifically, the playback module 700 utilizes the hardware decoding capability of the control chip to achieve the following functions: Access media files in a home NAS or shared directory via SMB / NFS / WebDAV protocols.

[0138] Real-time hardware decoding of 4K / 8K video, supporting H.265 / VP9 / AV1 formats, with CPU utilization below 10%.

[0139] The HDMI 2.1 interface connects to an external TV to output 1080P / 4K resolution images and supports CEC protocol for linking TV power on / off.

[0140] Receive video streams pushed by your phone via DLNA / AirPlay to achieve "one-click screen mirroring".

[0141] The playback module 700 can also provide local media playback capabilities, supporting playback operations triggered by natural language commands.

[0142] Specifically, the following natural language control commands are supported: "Play / Pause / Stop" → Calls Kodi's JSON-RPC API to send playback control commands.

[0143] "Previous / Next" → Switch between media items in the playlist.

[0144] "Volume Up / Down" → Adjust system volume via ALSA audio interface.

[0145] "Play movie xxx" → After the Agent runtime layer 320 parses the intent, it searches for the video file with the specified name in the shared directory and starts playing it.

[0146] In this embodiment, by configuring the playback module 700, intelligent control of home entertainment functions is realized. Users can control multimedia playback through natural language, which improves the convenience of the home entertainment experience.

[0147] Please see Figure 3 , Figure 3 This is one of the flowcharts illustrating the control method of the intelligent central system provided in this application embodiment. For example... Figure 3 As shown, the control method provided in this application includes the following steps: Step 301: Collect video and audio data.

[0148] Step 302: Perform AI reasoning based on the video data and the audio data to determine the corresponding working mode of the intelligent central system.

[0149] Step 303: Based on the aforementioned working mode, control the external device and / or send a notification message to the outside world.

[0150] In this embodiment, the hardware platform acquires video streams in the home environment in real time through the video acquisition module and continuously acquires ambient sound signals through the voice interaction module.

[0151] Optionally, the camera captures a 1080P video stream at a rate of 30 frames per second and transmits it to the hardware platform; the microphone array continuously captures an audio stream and transmits it to the hardware platform.

[0152] Video and audio data are sent to the AI ​​agent through the hardware management module. The AI ​​agent 300 performs AI inference based on the video and audio data to determine the working mode of the intelligent central system.

[0153] In one optional implementation, a lightweight visual AI model performs real-time inference on video and audio data; the model dynamic scheduling unit configures NPU computing power allocation according to the operating mode; if a person falls, open flames, smoke, or suspicious person intrusion is detected, the system is determined to enter "alarm mode"; if activity is detected in the elderly's activity area during a preset activity period, the system is determined to enter "daily monitoring mode"; if the current time is during nighttime security hours (e.g., 22:00 to 6:00 the next day), the system is determined to enter "nighttime security mode".

[0154] After determining the working mode, based on the determined working mode, control external devices and / or send prompt messages to the outside world.

[0155] Optionally, in alarm mode, an audible and visual alarm is emitted via a speaker; a red warning image flashes on an external TV connected via HDMI; and a warning message containing an alarm screenshot is pushed to a preset guardian's mobile phone via a network communication module.

[0156] Optionally, in daily monitoring mode, greetings and reminders can be broadcast; medication can be taken according to the medication schedule; and activity intensity scores can be generated periodically and broadcast or pushed to the guardian's mobile phone.

[0157] Optionally, in nighttime security mode, personnel intrusion detection is performed at a high frequency of 30 frames per second, and open flame and smoke detection is performed at 2 frames per second; an alarm workflow is triggered when an intrusion or fire is detected.

[0158] The following details the workflow of the daily monitoring mode: (1) System status: At 7:00 a.m., the system automatically switches to daily monitoring mode according to the preset "elderly activity period" (7:00-21:00).

[0159] (2) Activity monitoring and proactive greeting: The location and posture of people in the elderly's activity area are detected at 30 frames / second. When the elderly are detected to be active in the living room, the preset greeting workflow is triggered: "Good morning, the weather is sunny today, remember to eat breakfast on time", and the greeting event is recorded in the log file.

[0160] (3) Medication time reminder: The system will remind the user to take the antihypertensive medication on time when the scheduled time arrives, based on the user's preset medication schedule (e.g., 8:00 for antihypertensive medication). The system will also send a medication reminder notification to the guardian's mobile phone at the same time.

[0161] (4) Medication Verification (Optional Implementation): Point the camera at the area where the medicine bottle is placed and run a dedicated medicine bottle posture detection model. If the medicine bottle is detected to be picked up within 30 minutes after the reminder, it is recorded as "medication taken"; if it is not detected, a voice reminder will be sent again after 1 hour and a reminder will be pushed to the guardian.

[0162] (5) Daily activity report generation: Every 2 hours, the number of people detected and their movement trajectory in the activity area of ​​the elderly are automatically counted and broadcast or pushed to the guardian's mobile phone.

[0163] The following details the workflow of the nighttime security mode: (1) Mode switching: At 22:00 at night, the system will automatically switch to night security mode according to the time. The scheduling strategy will be adjusted as follows: personnel intrusion detection will be performed at a high frequency of 30 frames / second, and open flame and smoke detection will be performed at 2 frames / second.

[0164] (2) Intrusion detection: When the camera detects an unregistered person in the preset "restricted area", it immediately triggers the alarm workflow: the preset alarm audio is emitted through the speaker, the red warning screen flashes through the external TV connected via HDMI, and the warning information containing the alarm screenshot is pushed to the guardian's mobile phone. The complete link latency is less than 200 milliseconds.

[0165] (3) Fire warning: When the confidence level of open flame or smoke detection exceeds 0.7, local audible and visual alarms and remote push notifications are triggered simultaneously. The alarm information has a higher priority than intrusion detection.

[0166] The following details the workflow of the child companionship mode: (1) Content on demand: The child says, "I want to hear the story of Snow White." After the voice command is converted to text by ASR, the intent routing unit recognizes it as a content on demand class, calls the cloud LLM API to generate the story text, plays it sentence by sentence, and caches the text and voice locally for repeated playback.

[0167] (2) Screen Time Monitoring: The system uses a timer function to detect whether a child stays in front of the TV / screen for more than a preset time (e.g., 30 minutes). After the timeout, a voice reminder will sound, "Little friend, it's time to take a break and let your eyes rest for a while," and the TV screen will automatically turn off via the HDMI CEC protocol.

[0168] (3) Educational Q&A: When a child asks "Why is the sky blue?", the cloud-based LLM API provides an explanatory answer suitable for the child's age group, which is then played via TTS.

[0169] In this embodiment, by performing AI reasoning on video and audio data, the corresponding working mode of the intelligent central system is determined. Based on the working mode, external devices are controlled and / or prompt messages are sent to the outside world. This realizes the adaptive working mode switching of the intelligent central system based on multimodal perception. The intelligent central system can automatically determine the working mode and perform corresponding operations according to visual events and voice commands without the need for manual intervention by the user, thus realizing intelligent home services.

[0170] Furthermore, the control method provided in this embodiment employs a two-level task offloading mechanism: tasks requiring real-time response, such as visual AI inference, are all deployed on the edge-side NPU for execution. The real-time inference of the edge-side visual model ensures millisecond-level responses to emergencies such as falls and fires, triggering local alarms without relying on the network. Tasks requiring deep semantic understanding, such as natural language question answering, are offloaded to the cloud-based large language model, ensuring both question-answering quality and the depth of semantic understanding. This overcomes both the privacy leakage risks and network dependency issues of a pure cloud architecture, as well as the insufficient computing power and response latency issues of edge-side local LLM inference.

[0171] Optionally, please refer to Figure 4 , Figure 4 This is a second schematic flowchart of the control method for the intelligent central system provided in this application embodiment. For example... Figure 4 As shown, the control method provided in this application includes the following steps: Step 401: Collect audio data.

[0172] Step 402: If the audio data is detected to include a preset wake word, the audio data is converted into text data.

[0173] Step 403: Perform intent recognition on the text data.

[0174] Step 404: If the intent recognition result of the text data represents device control, then control the external device based on the text data.

[0175] Step 405: If the intent recognition result of the text data represents a language question and answer, then send the question and answer voice message based on the text data.

[0176] In this embodiment, the hardware platform continuously collects ambient sound signals through a microphone array.

[0177] Upon detecting a preset wake-up word, the wake-up voice command is converted into text. The intent routing unit in the Agent runtime layer performs intent classification on the converted text.

[0178] Optionally, the intent classifier is based on a hybrid approach of keyword matching, regular expressions, and rule engines, and predefines the following categories: Device control type: Matches regular expression patterns such as "on / off + {device name}" and "adjust {device name} to {brightness / temperature}".

[0179] Information query type: Matching regular expression patterns such as "the status of {device name}" and "whether {device name} is on or off".

[0180] Task execution type: Matches regular expression patterns such as "help me {execute task description}".

[0181] Natural Language Question Answering: Text that does not match any of the above categories.

[0182] In this embodiment, if the intent recognition result is a device control command, the command is routed to the plug-in management module, and the plug-in engine calls the corresponding third-party device access skill to perform the control operation.

[0183] For example, let's illustrate the complete workflow using the user's voice command "turn on the living room lights": (1) The microphone array continuously collects ambient sound, and the wake word detection unit runs the Porcupine wake-up engine in real time.

[0184] (2) Porcupine detected the wake word "Hello, little housekeeper" and activated the voice interaction channel.

[0185] (3) The Vosk offline ASR engine processes the voice command fragment after the wake word and outputs the text "Turn on the living room light".

[0186] (4) The intent routing unit of the Agent runtime layer analyzes the text: it matches the keyword "open" and the device type "light" (predefined device type list), extracts the device identifier "living room light", and classifies it as "device control".

[0187] (5) The Agent runtime layer queries the device registry to find the device ID and the plugin to which the “living room light” belongs.

[0188] (6) Call the plugin and send it to the device IP via UDP.

[0189] (7) The device performs the operation and returns the execution result (success / failure).

[0190] (8) The plugin returns the execution result to Agent Runtime.

[0191] (9) The Agent runtime layer calls the local TTS engine to synthesize the audio of "Okay, the living room light is on" and plays it through the speaker.

[0192] (10) The user hears voice feedback and the interaction is complete.

[0193] In this embodiment, if the intent recognition result is a natural language question-and-answer type instruction, then the cloud-based large language model is invoked. The specific process is as follows: (1) The Agent runtime layer encapsulates the user's question text into an API request and sends it to the cloud-based large language model API through the gateway layer.

[0194] (2) The request adopts the streaming mode, and the model layer will transmit the generated answer content back through the gateway layer as it is generated.

[0195] (3) The answer content is synthesized into speech by calling the local TTS engine through AudioOutputSkill sentence by sentence.

[0196] (4) Play the TTS synthesized response voice in real time through the speaker.

[0197] (5) The conversation context is maintained and managed by the memory system 240, which supports coherent understanding of multi-turn dialogues.

[0198] This embodiment achieves a complete closed loop of voice interaction, automating the entire process from wake word detection, speech recognition, intent understanding to device control or question-and-answer feedback. Device control commands are executed locally, offering fast response times and being network-independent; natural language question-and-answer commands leverage cloud-based large-scale model capabilities to ensure answer quality and depth of semantic understanding. The two modes work together, balancing real-time performance and intelligence.

[0199] Optionally, the method further includes: Configure the first process to call the AI ​​model deployed in the cloud; Configure a second process to run the home smart service system; A third process is configured to run voice acquisition, wake word detection, speech recognition and synthesis, and overlay display functions. The third process displays alarm information through the overlay, and the alarm information does not interfere with the output screen of the preset media player. When the second process receives a voice command or text command sent by the third process, it matches the voice command or text command in the locally configured edge rule base. If a match is successful, control commands are sent out based on the matching result; If a match fails, the voice command or the text command is forwarded to the first process, so that the first process can invoke the AI ​​model deployed in the cloud to respond to the voice command or the text command.

[0200] Specifically, the first process is configured to run the AI ​​Agent gateway, which is used to receive voice / text commands and call the cloud-based large model; the second process is configured to run the HISS core service, including message bus, business modules and API interfaces; and the third process is configured to independently run voice acquisition, wake word detection, speech recognition and synthesis, and HDMI overlay display.

[0201] In this embodiment, the second process calls peripherals such as camera, microphone, and GPIO through the hardware abstraction layer, and communicates with the first and third processes through the message bus.

[0202] When the second process receives the voice command text from the third process, it first matches it in the local edge rule base. If a match is found, it directly executes device control or local feedback. If no match is found, it forwards the command to the first process, which then calls the cloud-based big model to perform semantic understanding and return the execution action.

[0203] It should be understood that the third process displays alarm information through the DRM Plane overlay, and does not interfere with the main screen of media players such as Kodi.

[0204] To further illustrate the technical effects of the intelligent central system provided in this embodiment, a detailed comparative test is conducted between the solution provided in this application embodiment and four existing technical solutions.

[0205] Existing Solution 1: Pure Cloud Solution Testing method: The video stream captured by the camera is uploaded in full to the cloud AI service for inference, and the voice question and answer calls the cloud LLM.

[0206] Test results: Under stable network conditions, the uplink bandwidth usage for video data is approximately 2.5 GB / hour, the end-to-end response latency for voice commands is approximately 300-800 milliseconds, and the security alarm function is completely disabled under network outage conditions. Since all home audio and video data is stored on a cloud server, the privacy risk assessment is high.

[0207] Existing Solution 2: Local Closed Architecture Solution Testing method: Only devices with pre-configured Zigbee / Bluetooth Mesh protocols are supported; edge-side visual AI capabilities are not supported.

[0208] Test results: Device access requires waiting for system firmware upgrades, with an extension period of several weeks to months, and does not support visual AI functions such as fall detection and smoke detection.

[0209] Existing Solution 3: Local LLM Architecture Solution Test method: A 7B-parameter large language model was fully deployed on the hardware platform, with a memory usage of over 5.2GB and an inference speed of approximately 7 tokens / s.

[0210] Test results: The dialogue response delay exceeded 5 seconds, resulting in a poor user experience; the chip temperature remained above 85°C, requiring an active fan for forced cooling, with noise levels around 35dB(A); the model loading time was approximately 12 seconds, severely delaying the first-time user experience.

[0211] Existing Solution 4: Edge Computing Gateway Solution Testing method: Multimodal data is fused and processed at the edge, visual reasoning and speech recognition are completed on the local side, and core semantic understanding is still in the cloud.

[0212] Test results: Security alarm latency is approximately 150-300ms, visual inference frame rate is approximately 15-25 FPS, natural language question answering still requires network connectivity, and device access range is limited by the pre-built protocol stack.

[0213] The intelligent central system provided in this application embodiment: Security functions (i.e., detecting personnel falls, intrusions, and open flames): It can still operate independently when the network is interrupted, with a local alarm latency of less than 100 milliseconds and a trigger accuracy of over 88%. Visual inference frame rate: 35-50 FPS.

[0214] Device access: Supports access and control of over 2000 device models. A new device access plugin can be developed in approximately 2-4 hours.

[0215] Natural language interaction: The latency of cloud API calls is about 800-1500 milliseconds, the quality of responses is comparable to the native capabilities of large cloud models, and the additional memory usage is less than 200MB.

[0216] Data privacy: All video data is processed on the local NPU, and only anonymized text commands are uploaded to the cloud. Photo data is stored on the local SSD, and users have complete control over data access permissions.

[0217] Heat dissipation and power consumption: Utilizing a passive cooling design, the core temperature remains stable below 65℃ under full load, with power consumption of approximately 5W-6.5W. No active fan is required, achieving silent operation. Power consumption in standby mode is approximately 3.2W.

[0218] Photo backup performance: Uploading a single photo (e.g., JPEG format, 5MB storage space) from the phone to the local SSD takes about 2-3 seconds, and supports concurrent uploads from multiple devices.

[0219] Process separation effect: This application deploys the voice acquisition and display overlay function as an independent process, separating it from the core business logic process. This effectively avoids resource competition between audio devices and business modules, while ensuring that alarm information can still be displayed normally through the hardware overlay layer when the media player is in full-screen output. This significantly improves the system's stability and multi-task concurrency capabilities.

[0220] The above comparison results demonstrate that the intelligent central system provided in this application embodiment is significantly superior to existing technical solutions in multiple dimensions, including real-time security response capability, edge-side visual reasoning performance, natural language interaction quality, device access openness, data privacy and security, automated decision execution, and hardware cost.

[0221] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. An intelligent central control system, characterized in that, The intelligent central system includes a hardware platform, an AI agent, and a hardware management module. The AI ​​agent and the hardware platform are connected through the hardware management module. The hardware platform is used to collect video data and audio data, and send the video data and audio data to the AI ​​agent through the hardware management module. The AI ​​agent performs AI reasoning based on the video data and the audio data, and sends control signals to the hardware platform through the hardware management module. The hardware platform is also used to control external devices and / or send prompt messages in response to the control signal.

2. The intelligent central control system according to claim 1, characterized in that, The hardware platform includes a control chip; The control chip includes a central processing unit, a neural network processor, a graphics processor, a multimedia encoding / decoding module, and an ISP image signal processor; The control chip is also equipped with a video acquisition interface, a voice input interface, a voice output interface, a video output interface, a network communication interface, a USB interface, a GPIO interface, and a debugging interface.

3. The intelligent central control system according to claim 1, characterized in that, The AI ​​agent includes a gateway layer, an agent runtime layer, a skill layer, and a memory system that are interconnected. The gateway layer is used for protocol conversion and routing, session management, authentication and security management, and traffic scheduling and management. The Agent runtime layer is used to perform intent parsing operations, task planning operations, skill invocation operations, state management, and feedback. The skill layer is used for visual perception processing, voice input processing, voice output processing, plugin management, and media playback. The memory system is used to cache dialogue context, store operation logs, store long-term memory data, and vectorize historical dialogues and knowledge bases.

4. The intelligent central control system according to claim 1, characterized in that, The AI ​​agent also includes a processing module and a plug-in management module, both of which are deployed on the AI ​​agent.

5. The intelligent central system according to claim 4, characterized in that, The processing module includes a visual skill unit, a model dynamic scheduling unit, and a voice interaction channel; The visual skill unit is used to load AI models, preprocess video frames, perform inference scheduling operations, perform post-processing operations, distribute results externally, and issue external alarms. The model dynamic scheduling unit is used to dynamically allocate computing power according to the operating mode and task priority; The voice interaction channel is used to detect wake words, recognize speech, synthesize speech, and perform intent routing operations.

6. The intelligent central control system according to claim 4, characterized in that, The plug-in management module includes a device abstraction layer, a plug-in engine, and pre-built access plug-ins; The device abstraction layer is used to define a unified device model interface; The plugin engine is used to support the dynamic installation, uninstallation, activation, and disabling of plugins.

7. The intelligent central control system according to claim 1, characterized in that, The intelligent central system also includes a data management module, which is electrically connected to the hardware platform. The data management module includes a backup unit and a web management interface. The backup unit is used to perform automatic backup operations; The web management interface is used to provide a web entry point.

8. A control method for an intelligent central system, characterized in that, The method is applied to the intelligent central system as described in any one of claims 1-7, and the method includes: Collect video and audio data; AI reasoning is performed based on the video data and the audio data to determine the corresponding working mode of the intelligent central system; Based on the aforementioned working mode, control external devices and / or send notification messages to external devices.

9. The control method according to claim 8, characterized in that, The method further includes: Collect audio data; If the audio data is detected to include a preset wake word, the audio data is converted into text data; Perform intent recognition on the text data; If the intent recognition result of the text data represents device control, then the external device is controlled based on the text data; If the intent recognition result of the text data represents a question-and-answer dialogue, then the question-and-answer voice is sent out based on the text data.

10. The control method according to claim 8, characterized in that, The method further includes: Configure the first process to call the AI ​​model deployed in the cloud; Configure a second process to run the home smart service system; A third process is configured to run voice acquisition, wake word detection, speech recognition and synthesis, and overlay display functions. The third process displays alarm information through the overlay, and the alarm information does not interfere with the output screen of the preset media player. When the second process receives a voice command or text command sent by the third process, it matches the voice command or text command in the locally configured edge rule base. If a match is successful, control commands are sent out based on the matching result; If a match fails, the voice command or the text command is forwarded to the first process, so that the first process can invoke the AI ​​model deployed in the cloud to respond to the voice command or the text command.

Images