An intelligent housekeeper system based on artificial intelligence

Abstract

This invention discloses an artificial intelligence-based intelligent butler system, comprising: a multimodal data acquisition module for collecting users' multimodal data, including voice data, image data, text data, physiological signal data, and environmental sensor data. Through multimodal feature fusion and contextual understanding, it achieves cross-modal deep semantic alignment, improving the system's accuracy in understanding user intentions. It predicts intentions using a temporal diffusion model, transforming passive response into proactive service. Through multi-agent collaborative decision-making, it achieves globally optimal control of multiple devices. Through an emotion perception-driven module, it dynamically adjusts emotion-adaptive service strategies. Through federated continuous learning, it achieves personalized model evolution while ensuring privacy, providing users with a proactive, intelligent, collaborative, and secure butler service experience.

Description

Technical Field

[0001] This invention belongs to the technical field of home management systems, specifically an intelligent home management system based on artificial intelligence. Background Technology

[0002] With the rapid development of IoT, AI, and big data technologies, smart home systems have evolved from traditional single-device remote control to whole-house intelligent interconnection. The integration of AI technology provides smart homes with a more intelligent and personalized service experience. Breakthroughs in Transformer architecture and multimodal large models have given home appliances the ability to "see," automatically recognizing the environment and objects, understanding user intent, and providing personalized services, driving the transformation of home appliances from execution tools to intelligent butlers. Users' needs for home appliances have moved beyond simple voice control of appliance switches; they now expect an intelligent butler that understands user habits, anticipates user needs, and proactively provides services.

[0003] However, existing smart home systems still have many shortcomings in terms of technical implementation:

[0004] First, the interaction methods are limited, and the multimodal fusion capabilities are insufficient. Most smart home systems on the market currently rely on single voice interaction, and the accuracy of voice recognition is greatly affected by environmental noise, failing to accurately understand complex semantics and implicit intentions. While some systems incorporate touch or gestures, multimodal fusion remains at a simple superposition level, lacking deep semantic alignment and fusion reasoning capabilities. Existing systems struggle to meet human emotional needs and cannot ensure that the generated responses match the user's emotional state.

[0005] Second, the service model is passive, lacking proactive prediction and intent reasoning capabilities. Existing systems are essentially command-response based, only executing corresponding operations after the user issues a clear command. Current smart home systems remain in a "rule-driven, passive response" stage. The systems cannot anticipate needs based on user behavior sequences and lifestyle habits, lacking time-series modeling of user behavior and intent prediction mechanisms. Existing smart home devices can only sense physical parameters, lacking emotion and behavior recognition capabilities. Most of the "AI proactive butlers" advertised on the market simply apply AI concepts to existing timed, sensor-triggered automation rules, failing to autonomously understand needs and plan complete household chores.

[0006] Third, the system's decision-making capabilities are limited, and multi-device collaboration is poor. The decision-making logic of existing systems is mostly based on preset rules or simple classifiers, which cannot make globally optimal decisions when faced with complex home scenarios. There is a lack of a unified collaborative decision-making center among the devices, resulting in a lack of coordination and interoperability between multiple intelligent agents. Users are forced to break down their own needs and issue commands step by step, reducing the so-called "intelligent interaction" to tedious and mechanical operations.

[0007] Fourth, there is a disconnect between emotion perception and service decision-making. While existing research can identify users' facial expressions or vocal emotions, the results of emotion recognition have not been effectively translated into service strategies. Current systems cannot accurately identify users' emotional states, leading to inaccurate device responses. Emotion perception remains at the "recognition" level and fails to effectively "drive" adjustments to service strategies.

[0008] Fifth, insufficient user privacy protection hinders model continuous evolution. Smart home systems require the collection of vast amounts of household privacy data; existing systems upload raw data to the cloud for processing, posing a data leakage risk. Centralized learning methods suffer from privacy risks, high communication costs, and limited scalability. Furthermore, model training is often completed offline in a single instance, making personalized online updates based on each household's usage habits impossible. Standard federated learning techniques struggle to effectively address the data diversity and individual user preferences present in smart home scenarios.

[0009] In summary, existing smart home systems have significant shortcomings in terms of interactive intelligence, proactive service, collaborative decision-making, emotion-driven features, and privacy security. There is an urgent need for a new system architecture that can integrate multimodal deep understanding, temporal intent prediction, multi-agent collaborative decision-making, emotion-driven adaptation, and privacy-preserving continuous learning. Summary of the Invention

[0010] The technical solution adopted in this invention is as follows: An intelligent butler system based on artificial intelligence, comprising:

[0011] A multimodal data acquisition module is used to collect multimodal data from users, including voice data, image data, text data, physiological signal data, and environmental sensor data.

[0012] A multimodal feature fusion module, the input of which is connected to the output of the multimodal data acquisition module, is used to extract and fuse features from the multimodal data to generate a multimodal fusion feature vector.

[0013] A contextual understanding module, the input of which is connected to the output of the multimodal feature fusion module, is used to perform contextualized semantic understanding based on the multimodal fusion feature vector and generate contextual understanding results;

[0014] An intent prediction module, whose input is connected to the output of the context understanding module, is used to perform time-series modeling based on the context understanding results and user historical behavior data, predict the user's potential service needs within a preset future time window, and generate intent prediction results.

[0015] A multi-agent collaborative decision-making module, wherein the input end of the multi-agent collaborative decision-making module is connected to the output end of the intent prediction module, is used to receive the intent prediction result, decompose the intent prediction result into multiple sub-tasks, and schedule multiple specialized agents to execute the multiple sub-tasks respectively, thereby generating a collaborative decision result;

[0016] An adaptive execution module, the input of which is connected to the output of the multi-agent collaborative decision-making module, is used to generate device control commands based on the collaborative decision-making results and send the device control commands to the corresponding intelligent devices for execution;

[0017] An emotion perception driving module is provided, wherein the first input of the emotion perception driving module is connected to the output of the multimodal feature fusion module, and the output of the emotion perception driving module is connected to the control of the adaptive execution module. The module is used to perform user emotion recognition based on the multimodal fusion feature vector, generate emotion recognition results, and dynamically adjust the service strategy of the adaptive execution module according to the emotion recognition results.

[0018] The federated continuous learning module has its first data terminal connected to the data terminal of the multimodal data acquisition module, its second data terminal connected to the data terminal of the multimodal feature fusion module, its third data terminal connected to the data terminal of the context understanding module, its fourth data terminal connected to the data terminal of the intent prediction module, its fifth data terminal connected to the data terminal of the multi-agent collaborative decision-making module, and its sixth data terminal connected to the data terminal of the adaptive execution module. This module is used to perform federated learning updates on the model parameters of the multimodal feature fusion module, the context understanding module, the intent prediction module, the multi-agent collaborative decision-making module, and the adaptive execution module on a local device based on the multimodal data. The encrypted model parameter update data is then uploaded to a cloud server for global aggregation.

[0019] Furthermore, the multimodal feature fusion module employs a multimodal feature fusion algorithm based on a cross-modal attention mechanism to perform feature extraction and fusion processing. The multimodal feature fusion algorithm includes the following steps:

[0020] The first of the multimodal data The raw data of each modality is mapped into modal feature vectors through the corresponding modal encoder. ,in N is the total number of modes;

[0021] The following cross-modal attention calculation formula is used to calculate the first... The modal eigenvector relative to the th modality eigenvector Cross-modal attention weights of modal feature vectors : , in, To query the weight matrix, The key weight matrix, Let be the dimension of the key vector. This represents the matrix transpose operation;

[0022] The following multi-head cross-modal attention calculation formula is used to calculate the first... Enhanced feature vectors of each modality : , in, M represents the number of attention heads. To output the weight matrix, For value weight matrix, This represents a vector concatenation operation;

[0023] The multimodal fusion feature vector is generated using the following modal fusion calculation formula: , in, For the first The adaptive fusion weights for each modality are dynamically generated through a learnable gating network and satisfy the following conditions: , For layer normalization operation, This is the fused feature vector from the previous time step.

[0024] Furthermore, the intent prediction module employs an intent prediction algorithm based on a temporal diffusion model to perform the temporal modeling and intent prediction. The temporal diffusion model includes a forward diffusion process and a reverse denoising process.

[0025] The forward diffusion process gradually adds Gaussian noise to the historical behavior sequence until it becomes pure noise using the following formula: , in, For the original historical behavior sequence, Let be the noisy sequence for the t-th diffusion step. , The total number of diffusion steps, Let be the preset noise variance at step t. Indicates a Gaussian distribution. It is the identity matrix;

[0026] The reverse denoising process uses the following formula to gradually remove noise from pure noise and restore the predicted value of the future behavior sequence: , in, For learnable parameters, The mean function predicted by the neural network. The variance function predicted by the neural network;

[0027] The intent prediction module generates future behavior sequence predictions based on the context understanding result c using the following conditional diffusion formula. : ,

[0028] in, The initial prediction sequence generated for the unconditional diffusion model. To guide the weighting coefficients, The guided loss function is based on the context understanding result c, and is calculated using the following formula: , in, For a given prediction sequence The log-likelihood probability of the situational understanding result c under the given conditions.

[0029] Furthermore, the multi-agent collaborative decision-making module uses a collaborative decision-making framework based on a multi-agent deep deterministic policy gradient algorithm to generate the collaborative decision-making result. The collaborative decision-making framework includes K specialized agents, each of which maintains its own policy network and evaluation network. The policy network outputs actions based on the local observation state of the specialized agent, and the evaluation network is used to evaluate the value of the actions.

[0030] The multi-agent collaborative decision-making module adopts an architecture that combines centralized training and distributed execution. During the training phase, all professional agents share a centralized evaluation function. The centralized evaluation function takes the global state and the actions of all professional agents as input, outputs the global action value, and takes maximizing the cumulative discount reward as the optimization objective.

[0031] Each specialized agent's policy network updates its parameters through the policy gradient method, introducing an entropy regularization term into the policy gradient to encourage exploration.

[0032] The multi-agent collaborative decision-making module also applies consensus and consistency constraints to the action outputs of each professional agent. Each professional agent adjusts its original action in coordination with the action outputs of other professional agents it communicates with, so that the final actions of each professional agent reach a consistent and coordinated state.

[0033] Furthermore, the emotion perception driving module includes a multi-branch feature extraction network, a dynamic weight allocation unit, an emotion evolution analysis unit, and an emotion-service mapping unit;

[0034] The multi-branch feature extraction network is used to extract facial expression temporal features, speech prosody temporal features, physiological signal temporal features, and text semantic features from the multimodal fusion feature vector;

[0035] The dynamic weight allocation unit is used to dynamically calculate the contribution weight of each modality feature in emotion recognition according to the current context, and to perform weighted fusion of each modality feature based on the contribution weight to obtain fused emotion features;

[0036] The emotion evolution analysis unit is used to predict the future trend of emotion changes based on the current emotion state and the context environment, and generate emotion evolution trend information.

[0037] The emotion-service mapping unit is used to generate service strategy adjustment parameters for adjusting the adaptive execution module based on the fused emotion features and the emotion evolution trend information, and outputs the service strategy adjustment parameters to the control terminal of the adaptive execution module.

[0038] Furthermore, the federated continuous learning module employs a differential privacy-based federated averaging algorithm to perform the federated learning update. This differential privacy-based federated averaging algorithm includes the following steps:

[0039] The federated continuous learning module calculates the local model parameter update amount based on local multimodal data on the local device, and performs gradient pruning on the local model parameter update amount to obtain the pruned local model parameter update amount.

[0040] The federated continuous learning module adds Gaussian difference privacy noise to the cropped local model parameter update amount to obtain the noisy local model parameter update amount, and then encrypts the noisy local model parameter update amount before uploading it to the cloud server.

[0041] The cloud server receives encrypted, noisy local model parameter update data uploaded by all local devices participating in federated learning. After decrypting the encrypted, noisy local model parameter update data, it performs a weighted average aggregation based on the number of local training samples of each local device to generate global model parameter update data.

[0042] The cloud server distributes the global model parameter update to each local device, and each local device updates its own local model parameters according to the global model parameter update.

[0043] Furthermore, the intelligent butler system also includes a cross-device context-aware scene automatic orchestration module. The input of the scene automatic orchestration module is connected to the output of the context understanding module and the output of the intent prediction module, respectively. The output of the scene automatic orchestration module is connected to the input of the multi-agent collaborative decision-making module.

[0044] The automatic scene arrangement module is used to match the target scene most similar to the current scene from the preset scene library based on the context understanding result and the intent prediction result. When matching, the semantic features, time tags and location tags of the current scene are comprehensively considered to calculate the comprehensive similarity with the preset scene.

[0045] The scene automatic orchestration module is also used to automatically generate a device linkage scheme corresponding to the target scene when the comprehensive similarity exceeds a preset threshold, and send the device linkage scheme to the multi-agent collaborative decision-making module.

[0046] Furthermore, the multimodal data acquisition module includes a voice acquisition unit, an image acquisition unit, a text input interface, a physiological signal acquisition interface, and an environmental sensor interface;

[0047] The voice acquisition unit is used to collect the user's voice data;

[0048] The image acquisition unit is used to acquire the user's image data;

[0049] The text input interface is used to receive text data input by the user;

[0050] The physiological signal acquisition interface is used to acquire the user's physiological signal data from the wearable device;

[0051] The environmental sensor interface is used to acquire environmental sensor data from environmental sensors deployed in the home environment, including temperature sensors, humidity sensors, light sensors, air quality sensors, and human infrared sensors.

[0052] Furthermore, the multiple specialized intelligent agents in the multi-agent collaborative decision-making module include an environmental control intelligent agent, a health management intelligent agent, a safety monitoring intelligent agent, and an entertainment service intelligent agent;

[0053] The environmental control agent is used to perform environmental parameter adjustment sub-tasks;

[0054] The health management intelligent agent is used to perform sub-tasks of monitoring and early warning of user health status;

[0055] The security monitoring intelligent agent is used to perform the sub-task of detecting and alarming home security anomalies;

[0056] The entertainment service agent is used to perform the sub-tasks of multimedia content recommendation and playback.

[0057] Furthermore, the system also includes at least one processor and at least one non-volatile computer-readable storage medium, the at least one non-volatile computer-readable storage medium being connected to the at least one processor, and storing a computer program in the non-volatile computer-readable storage medium;

[0058] When the computer program is executed by the at least one processor, the at least one processor enables the at least one processor to implement the functions of the multimodal data acquisition module, the multimodal feature fusion module, the context understanding module, the intent prediction module, the multi-agent collaborative decision-making module, the adaptive execution module, the emotion perception-driven module, the federated continuous learning module, and the scene automatic orchestration module;

[0059] The non-volatile computer-readable storage medium is used to persistently store the multimodal data acquired by the multimodal data acquisition module, the multimodal fusion feature vector generated by the multimodal feature fusion module, the context understanding result generated by the context understanding module, the intent prediction result generated by the intent prediction module, the collaborative decision-making result generated by the multi-agent collaborative decision-making module, the emotion recognition result generated by the emotion perception driving module, and the local model parameters of the federated continuous learning module.

[0060] The at least one processor is connected to the at least one non-volatile computer-readable storage medium via a data bus, and the at least one processor is also connected to an external intelligent device via a communication interface for sending device control commands to the external intelligent device.

[0061] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0062] (1) This invention collects multi-dimensional data such as voice, image, text, physiological signals, and environmental sensors through a multi-modal data acquisition module, and uses a multi-modal feature fusion algorithm based on a cross-modal attention mechanism for feature extraction and fusion processing. This effectively integrates heterogeneous multi-source information, achieves comprehensive perception of user status and environment, and significantly improves the accuracy and robustness of system context understanding. Compared with single-modal smart home systems, the multi-modal fusion architecture of this invention can more accurately capture the user's true intentions and needs.

[0063] (2) This invention employs an intent prediction algorithm based on a temporal diffusion model through an intent prediction module. By combining contextual understanding results and user historical behavior data for temporal modeling, it can proactively predict users' potential service needs within a preset future time window. Compared to traditional passive response systems, this invention achieves an intelligent leap from "passively executing instructions" to "proactively predicting needs," enabling the corresponding services to be launched in advance before users generate needs, greatly improving the smoothness and intelligence of the user experience.

[0064] (3) This invention employs a collaborative decision-making framework based on a multi-agent deep deterministic policy gradient algorithm through a multi-agent collaborative decision-making module. This framework decomposes the complex intent prediction results into multiple sub-tasks, which are then executed by various specialized agents, including environmental control agents, health management agents, security monitoring agents, and entertainment service agents. Through an architecture combining centralized training and distributed execution, along with a consensus constraint mechanism, efficient collaboration and global optimization among agents are achieved, effectively resolving decision-making conflicts and resource competition issues in multi-task concurrent scenarios.

[0065] (4) This invention achieves user emotion recognition and emotion evolution trend prediction based on multimodal fusion feature vectors through a multi-branch feature extraction network, dynamic weight allocation unit, emotion evolution analysis unit, and emotion-service mapping unit of the emotion perception-driven module. The system can dynamically adjust the service strategy of the adaptive execution module according to the user's real-time emotional state. For example, it can automatically adjust the light color temperature and play soothing music when the user is tired, and provide calming guidance when the user is anxious, thus realizing a truly warm and intelligent butler service.

[0066] (5) This invention employs a federated averaging algorithm based on differential privacy through a federated continuous learning module. On the local device, it updates the model parameters of each functional module using local multimodal data through federated learning, and then uploads the encrypted model parameter updates to the cloud server for global aggregation. This architecture fully utilizes cloud computing power to achieve continuous model optimization and upgrading, while ensuring that user privacy data does not leave the local device through multiple privacy protection mechanisms such as local training, gradient pruning, differential privacy noise addition, and encrypted transmission. This fundamentally resolves the contradiction between user privacy protection and continuous model learning in smart home scenarios.

[0067] (6) This invention utilizes a cross-device context-aware scene automatic orchestration module to automatically match target scenes from a preset scene library and generate device linkage schemes based on context understanding results and intent prediction results. This module comprehensively considers multi-dimensional information such as semantic features of the context, time tags, and location tags to calculate the comprehensive similarity, realizing automated orchestration and intelligent linkage across devices and scenes. Users do not need to manually set scene rules, which greatly reduces the usage threshold and configuration complexity of smart home systems. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the system architecture of the present invention;

[0069] Figure 2 This is a schematic diagram of the multimodal data acquisition module of the present invention;

[0070] Figure 3 This is a schematic diagram of the multi-agent collaborative decision-making module of the present invention;

[0071] Figure 4 This is a schematic diagram of the adaptive execution module of the present invention;

[0072] Figure 5 This is a schematic diagram of the emotion perception driving module of the present invention;

[0073] Figure 6 This is a schematic diagram of the federated continuous learning module of the present invention.

[0074] In the diagram: 1. Multimodal data acquisition module; 2. Multimodal feature fusion module; 3. Contextual understanding module; 4. Intent prediction module; 5. Multi-agent collaborative decision-making module; 6. Adaptive execution module; 7. Emotion perception-driven module; 8. Federated continuous learning module; 9. Scene automatic orchestration module. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0076] Example 1, please refer to Figures 1-6A smart home management system based on artificial intelligence includes: a multimodal data acquisition module 1, used to collect multimodal data from users, including voice data, image data, text data, physiological signal data, and environmental sensor data; a multimodal feature fusion module 2, whose input is connected to the output of the multimodal data acquisition module 1, used to extract and fuse features from the multimodal data to generate a multimodal fusion feature vector; a contextual understanding module 3, whose input is connected to the output of the multimodal feature fusion module 2, used to perform contextualized semantic understanding based on the multimodal fusion feature vector to generate a contextual understanding result; and an intent prediction module 4, whose input is connected to the contextual understanding module 2. The output of module 3 is connected to the context understanding module and the user's historical behavior data to perform time-series modeling, predict the user's potential service needs within a preset future time window, and generate intent prediction results. The multi-agent collaborative decision-making module 5, whose input is connected to the output of the intent prediction module 4, receives the intent prediction results, decomposes them into multiple sub-tasks, and schedules multiple specialized agents to execute these sub-tasks, generating collaborative decision results. The adaptive execution module 6, whose input is connected to the output of the multi-agent collaborative decision-making module 5, generates device control commands based on the collaborative decision results and sends these commands to the corresponding smart devices for execution. Emotion... The perception-driven module 7 has its first input connected to the output of the multimodal feature fusion module 2, and its output connected to the control of the adaptive execution module 6. It is used to perform user emotion recognition based on the multimodal fusion feature vector, generate emotion recognition results, and dynamically adjust the service strategy of the adaptive execution module 6 according to the emotion recognition results. The federated continuous learning module 8 has its first data terminal connected to the data terminal of the multimodal data acquisition module 1, its second data terminal connected to the data terminal of the multimodal feature fusion module 2, its third data terminal connected to the data terminal of the context understanding module 3, and its fourth data terminal connected to the data terminal of the context understanding module 3. The fifth data terminal of the federated continuous learning module 8 is connected to the data terminal of the multi-agent collaborative decision-making module 5, and the sixth data terminal of the federated continuous learning module 8 is connected to the data terminal of the adaptive execution module 6. This is used to perform federated learning updates on the model parameters of the multi-modal feature fusion module 2, contextual understanding module 3, intent prediction module 4, multi-agent collaborative decision-making module 5, and adaptive execution module 6 based on multi-modal data on the local device. The encrypted model parameter updates are then uploaded to the cloud server for global aggregation. First, the multi-modal data acquisition module 1 uses voice acquisition units, image acquisition units, text input interfaces, physiological signal acquisition interfaces, and environmental sensor interfaces deployed in the home environment.The system collects multimodal data from users in real time, including voice data, image data, text data, physiological signal data, and environmental sensor data. All data is synchronized and aligned using timestamps as indices and preprocessed before being used as the system's raw input data. Subsequently, the multimodal feature fusion module 2 receives the multimodal data output from the multimodal data acquisition module 1 and performs feature extraction and fusion processing using a multimodal feature fusion algorithm based on a cross-modal attention mechanism. Specifically, the raw data for each modality is first mapped to a modal feature vector hi through the corresponding modality encoder, and then the first feature vector hi is calculated using the cross-modal attention calculation formula. The modal eigenvector relative to the th modality eigenvector The cross-modal attention weights of each modal feature vector will and The dot product similarity is calculated by projecting the query weight matrix and key weight matrix, respectively, and then softmax normalized. Next, multi-head cross-modal attention is used to calculate the enhanced feature vector by concatenating the outputs of multiple attention heads and projecting them onto the output weight matrix. Finally, the enhanced feature vectors of each modality are weighted and summed according to the adaptive fusion weights dynamically generated by the learnable gating network using the modality fusion calculation formula. This summation is then overlaid with the fusion feature vector from the previous time step and normalized layer by layer to generate a multimodal fusion feature vector. This achieves deep interaction and fusion of cross-modal information. Then, the contextual understanding module 3 receives the multimodal fusion feature vector output by the multimodal feature fusion module 2 and contextualizes it using a pre-trained context encoder network. Semantic understanding: The context encoder employs a Transformer-based context modeling structure, mapping multimodal fusion feature vectors to contextual semantic vectors. It combines current time information, user location information, and historical contextual information for comprehensive reasoning, generating contextual understanding results. These results include the current home scene category, the user's current activity state, environmental state descriptions, and potential explicit user needs. An attention mechanism automatically focuses on the most critical semantic elements within the current context, filtering out redundant information. Simultaneously, the intent prediction module 4 receives the contextual understanding results from the contextual understanding module 3 and uses a temporal diffusion model-based intent prediction algorithm to perform temporal modeling and intent prediction. This temporal diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process uses the user's historical behavior sequence as the original sequence, gradually adding Gaussian noise according to a preset total diffusion step count until it becomes pure noise. The reverse denoising process starts from the pure noise and gradually denoises and restores the predicted value of the future behavior sequence through the mean function and variance function predicted by the neural network. The intent prediction module 4 also introduces the contextual understanding result as conditional information into the generation process, generating a future behavior sequence prediction based on the contextual understanding result through a conditional diffusion formula. Finally, the intent prediction result is output, including the type of service demand that the user may generate within a preset time window, the expected time of occurrence of the demand, and the demand priority. Then, the multi-agent collaborative decision-making module 5 receives the intent prediction result output by the intent prediction module 4 and adopts a multi-agent collaborative decision-making approach. A collaborative decision-making framework based on a deep deterministic policy gradient algorithm for agents generates collaborative decision-making results. This framework includes four specialized agents: an environmental control agent, a health management agent, a security monitoring agent, and an entertainment service agent. Each specialized agent maintains its own policy network and evaluation network. The policy network outputs actions based on the local observation state of the specialized agent. The multi-agent collaborative decision-making module 5 adopts an architecture combining centralized training and distributed execution. During the training phase, all specialized agents share a centralized evaluation function, taking the global state and the actions of all specialized agents as input, and outputting the global action value. The optimization objective is to maximize the cumulative reward with discounts. The policy network of each specialized agent updates its parameters using the policy gradient method.To encourage exploration, an entropy regularization term is introduced into the policy gradient. The multi-agent collaborative decision-making module 5 further decomposes the intent prediction result into multiple sub-tasks, scheduling corresponding specialized agents to execute them separately. Consensus constraints are applied between the action outputs of each specialized agent to ensure that their final actions reach a consistent collaborative state, generating a collaborative decision result. Subsequently, the adaptive execution module 6 receives the collaborative decision result output by the multi-agent collaborative decision-making module 5 and, according to the standard communication protocols of each smart device, translates the collaborative decision result into device control commands recognizable by each smart device, including on / off commands, parameter adjustment commands, and mode switching commands. These device control commands are then sent to the home IoT gateway. The instruction is executed by the corresponding smart device, and the execution status information fed back by each smart device is received to confirm whether the instruction has been successfully executed. In the entire process, the emotion perception driving module 7 receives the multimodal fusion feature vector output by the multimodal feature fusion module 2 in real time, and extracts facial expression temporal features, speech prosody temporal features, physiological signal temporal features, and text semantic features from it through a multi-branch feature extraction network. The dynamic weight allocation unit dynamically calculates the contribution weight of each modality feature in emotion recognition according to the current context environment, and performs weighted fusion of each modality feature to obtain the fused emotion feature. The emotion evolution analysis unit predicts the future trend of emotion state based on the current emotion state and context environment, and generates the emotion evolution trend. Based on the fusion of emotional features and emotional evolution trend information, the emotion-service mapping unit generates service strategy adjustment parameters and outputs them to the control terminal of the adaptive execution module 6. This dynamically adjusts the service strategy of the adaptive execution module 6, achieving emotion-driven personalized service adjustment. Simultaneously, the scene automatic orchestration module 9 receives the context understanding results from the context understanding module 3 and the intent prediction results from the intent prediction module 4. It matches the target scene most similar to the current context from a preset scene library, comprehensively considering the semantic features, time tags, and location tags of the current context with the preset scenes to calculate a comprehensive similarity score. When the comprehensive similarity score exceeds a preset threshold, it automatically generates a device linkage scheme corresponding to the target scene and sets the devices accordingly. The linkage scheme is sent to the multi-agent collaborative decision-making module 5. Simultaneously, the federated continuous learning module 8 performs federated learning updates using a differential privacy-based federated averaging algorithm. On the local device, it calculates the local model parameter update based on locally collected multimodal data, performs gradient pruning on the local model parameter update, adds Gaussian differential privacy noise, encrypts it, and uploads it to the cloud server. The cloud server decrypts the parameter update uploads from all participating local devices, performs a weighted average aggregation based on the number of local training samples on each local device, generates a global model parameter update, and distributes it to each local device. Each local device then updates its local model parameters according to the global model parameter update.The model parameters of the multimodal feature fusion module 2, contextual understanding module 3, intent prediction module 4, multi-agent collaborative decision-making module 5, and adaptive execution module 6 continuously evolve while protecting user privacy. Finally, at least one processor executes a computer program stored in a non-volatile computer-readable storage medium to implement the functions of all the above modules. It also connects to external intelligent devices via a communication interface to send device control commands. The non-volatile computer-readable storage medium persistently stores multimodal data, multimodal fusion feature vectors, contextual understanding results, intent prediction results, collaborative decision-making results, emotion recognition results, and local model parameters. High-speed data reading, writing, and transmission are achieved through a data bus.

[0077] Please see Figures 1-6 The multimodal feature fusion module 2 employs a multimodal feature fusion algorithm based on a cross-modal attention mechanism to perform feature extraction and fusion processing. The multimodal feature fusion algorithm includes the following steps: [The algorithm then describes the steps of extracting and fusing features from the multimodal data.] The raw data of each modality is mapped into modal feature vectors through the corresponding modal encoder. ,in N is the total number of modalities. The cross-modal attention calculation is performed using the following formula. The modal eigenvector relative to the th modality eigenvector Cross-modal attention weights of modal feature vectors : in, To query the weight matrix, The key weight matrix, Let be the dimension of the key vector. This represents the matrix transpose operation, calculated using the following multi-head cross-modal attention calculation formula. Enhanced feature vectors of each modality : in, M represents the number of attention heads. To output the weight matrix, For value weight matrix, The vector concatenation operation generates a multimodal fusion feature vector using the following modal fusion calculation formula: in, For the first The adaptive fusion weights for each modality are dynamically generated through a learnable gating network and satisfy the following conditions: , For layer normalization operation, This is the fused feature vector from the previous time step.

[0078] The intent prediction module 4 uses an intent prediction algorithm based on a temporal diffusion model to perform temporal modeling and intent prediction. The temporal diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the historical behavior sequence until it becomes pure noise using the following formula: in, For the original historical behavior sequence, Let be the noisy sequence for the t-th diffusion step. , The total number of diffusion steps, Let be the preset noise variance at step t. Indicates a Gaussian distribution. Given an identity matrix, the inverse denoising process uses the following formula to gradually remove noise from pure noise and reconstruct the predicted value of the future behavior sequence: in, For learnable parameters, The mean function predicted by the neural network. Given the variance function predicted by the neural network, the intent prediction module 4 generates future behavior sequence predictions conditioned on the context understanding result c using the following conditional diffusion formula. : in, The initial prediction sequence generated for the unconditional diffusion model. To guide the weighting coefficients, The guided loss function is based on the context-understanding result c, and is calculated using the following formula: in, For a given prediction sequence Given the contextual understanding result c, the log-likelihood probability is calculated. The multi-agent collaborative decision-making module 5 uses a collaborative decision-making framework based on a multi-agent deep deterministic policy gradient algorithm to generate collaborative decision results. This framework includes K specialized agents, each maintaining its own policy network and evaluation network. The policy network outputs actions based on the agent's local observation state, and the evaluation network assesses the value of the actions. The multi-agent collaborative decision-making module 5 employs an architecture combining centralized training and distributed execution. During the training phase, all agents share a centralized evaluation function. This function takes the global state and the actions of all agents as input, outputs the global action value, and optimizes by maximizing the cumulative discount reward. The policy network of each specialized agent updates its parameters using the policy gradient method, introducing an entropy regularization term to encourage exploration. The multi-agent collaborative decision-making module 5 also imposes consensus constraints on the action outputs of each specialized agent. Each specialized agent coordinates its original actions based on the action outputs of other specialized agents it communicates with, ensuring that the final actions of all specialized agents reach a consistent and coordinated state. The emotion perception driving module 7 includes a multi-branch feature extraction network, a dynamic weight allocation unit, an emotion evolution analysis unit, and an emotion-service mapping unit. The multi-branch feature extraction network is used to extract facial expression temporal features, speech prosody temporal features, physiological signal temporal features, and text semantic features from the multimodal fusion feature vector. The dynamic weight allocation unit dynamically calculates the contribution weight of each modality feature in emotion recognition based on the current context, and performs weighted fusion of each modality feature based on the contribution weight to obtain fused emotion features. The emotion evolution analysis unit predicts the future trend of emotion state based on the current emotion state and context, generating emotion evolution trend information. The emotion-service mapping unit generates service policy adjustment parameters for adjusting the adaptive execution module 6 based on the fused emotion features and emotion evolution trend information, and outputs the service policy adjustment parameters to the control terminal of the adaptive execution module 6. The federated continuous learning module 8 performs federated learning updates using a differential privacy-based federated averaging algorithm, which includes the following steps. The Federated Continuous Learning Module 8 calculates the local model parameter update based on local multimodal data on the local device and performs gradient pruning on the local model parameter update to obtain the pruned local model parameter update. It then adds Gaussian difference privacy noise to the pruned local model parameter update to obtain the noisy local model parameter update, which is then encrypted and uploaded to the cloud server. The cloud server receives the encrypted noisy local model parameter update from all participating local devices, decrypts the noisy local model parameter update, and performs a weighted average aggregation based on the number of local training samples on each local device to generate the global model parameter update.The cloud server distributes global model parameter updates to each local device. Each local device updates its own local model parameters according to the global model parameter updates. The smart home system also includes a cross-device context-aware scene auto-arrangement module 9. The input of the scene auto-arrangement module 9 is connected to the output of the context understanding module 3 and the intent prediction module 4, respectively. The output of the scene auto-arrangement module 9 is connected to the input of the multi-agent collaborative decision-making module 5. The scene auto-arrangement module 9 is used to match the target scene most similar to the current context from a preset scene library based on the context understanding results and intent prediction results. When matching, it comprehensively considers the similarity between the semantic features, time tags, and location tags of the current context and the preset scenes, and calculates the comprehensive similarity. The scene automatic orchestration module 9 is also used to automatically generate a device linkage scheme corresponding to the target scene when the comprehensive similarity exceeds a preset threshold, and send the device linkage scheme to the multi-agent collaborative decision-making module 5. The multimodal data acquisition module 1 includes a voice acquisition unit, an image acquisition unit, a text input interface, a physiological signal acquisition interface, and an environmental sensor interface. The voice acquisition unit is used to acquire the user's voice data, the image acquisition unit is used to acquire the user's image data, the text input interface is used to receive the user's input text data, the physiological signal acquisition interface is used to acquire the user's physiological signal data from the wearable device, and the environmental sensor interface is used to acquire environmental sensor data from environmental sensors deployed in the home environment. The environmental sensors include temperature sensors. The system incorporates sensors including humidity, light, air quality, and infrared human body sensors. The multi-agent collaborative decision-making module 5 comprises several specialized agents: an environmental control agent, a health management agent, a security monitoring agent, and an entertainment service agent. The environmental control agent performs environmental parameter adjustment sub-tasks; the health management agent performs user health status monitoring and early warning sub-tasks; the security monitoring agent performs home security anomaly detection and alarm sub-tasks; and the entertainment service agent performs multimedia content recommendation and playback sub-tasks. At the data perception level, the solution constructs a multimodal acquisition system covering voice, image, text, physiological signals, and environmental sensor data, providing the system with rich perceptual information and significantly improving performance. The solution enhances perception accuracy and robustness. In multimodal fusion, it innovatively introduces a cross-modal attention mechanism to automatically capture deep semantic relationships between different modalities. Through a learnable gating network, it dynamically adjusts the fusion weights of each modality, enabling the system to adaptively highlight the most relevant modal features based on context, effectively solving the problem of traditional methods struggling to handle cross-modal interactions. At the contextual understanding level, the solution employs a Transformer-based context encoder for contextual modeling, fully utilizing temporal information, user location, and historical context for comprehensive reasoning, enabling the system to accurately identify complex home scenarios and user activity states. For intent prediction, the solution innovatively uses a temporal diffusion model instead of traditional sequence prediction methods.By employing a dual-process mechanism of forward diffusion noise addition and reverse denoising to learn complex patterns of user behavior, and combining it with a conditional guidance mechanism, the prediction results are highly consistent with the current context, solving the problem that traditional models struggle to effectively incorporate contextual information. In multi-agent collaborative decision-making, the solution constructs a collaborative decision-making framework based on a multi-agent deep deterministic policy gradient algorithm. It decomposes complex tasks into four specialized sub-tasks: environmental control, health management, security monitoring, and entertainment services. A centralized training and distributed execution architecture, along with consensus constraints, are introduced to enable agents to cooperate while maintaining independent decision-making capabilities, resolving action conflict issues. Regarding emotion perception, the solution designs a complete emotion perception driving module, using a multi-branch network to improve... This solution extracts various temporal features and dynamically adjusts modal contribution weights. It also introduces an emotion evolution analysis mechanism to predict emotion change trends, enabling the system to proactively adjust service strategies. Regarding data privacy protection, the solution employs a federated continuous learning framework. All raw data is stored locally, with only model parameter updates after gradient pruning and differential privacy noise enhancement uploaded. The cloud aggregates these local updates using a federated averaging algorithm, achieving collaborative training without sharing raw data and effectively protecting user privacy. For scene automation, the solution includes a cross-device context-aware scene auto-orchestration module. This module automatically matches preset scenes and generates device linkage plans based on context and intent, achieving end-to-end automation from perception to execution. In summary, this solution achieves technological innovation in the perception, fusion, understanding, prediction, decision-making, execution, and learning layers. It constructs a complete intelligent home management system architecture that combines high-precision perception, accurate intent prediction, collaborative decision-making, emotion-driven regulation, and privacy-preserving continuous learning, significantly improving the intelligence level and user experience of smart home systems.

[0079] Please see Figures 1-6The system also includes at least one processor and at least one non-volatile computer-readable storage medium. The at least one non-volatile computer-readable storage medium is connected to the at least one processor and stores a computer program. When the computer program is executed by the at least one processor, it enables the at least one processor to implement the functions of a multimodal data acquisition module 1, a multimodal feature fusion module 2, a contextual understanding module 3, an intent prediction module 4, a multi-agent collaborative decision-making module 5, an adaptive execution module 6, an emotion perception-driven module 7, a federated continuous learning module 8, and a scene automatic orchestration module 9. The non-volatile computer-readable storage medium is used to persistently store the multimodal data acquired by the multimodal data acquisition module 1, the multimodal fusion feature vector generated by the multimodal feature fusion module 2, the contextual understanding results generated by the contextual understanding module 3, and the scene automatic orchestration module 9. The system includes the intent prediction results generated by the multi-agent collaborative decision-making module 5, the emotion recognition results generated by the emotion perception-driven module 7, and the local model parameters of the federated continuous learning module 8. At least one processor is connected to at least one non-volatile computer-readable storage medium via a data bus. At least one processor is also connected to an external intelligent device via a communication interface to send device control commands. The solution configures at least one processor and at least one non-volatile computer-readable storage medium, storing the computer program within them. This allows the processor to uniformly implement all functions of the multi-modal data acquisition module, multi-modal feature fusion module, contextual understanding module, intent prediction module, multi-agent collaborative decision-making module, adaptive execution module, emotion perception-driven module, federated continuous learning module, and scene automatic orchestration module when executing the program. This integrated program deployment method integrates all the aforementioned intelligent algorithm modules onto the same computing platform, eliminating communication delays and interface compatibility issues that may arise from dispersed deployment between modules, ensuring the efficiency and consistency of data flow between all parts of the system. Non-volatile computer-readable storage media are not only used to store computer programs, but are also designed to persistently store multimodal data, multimodal fusion feature vectors, contextual understanding results, intent prediction results, collaborative decision-making results, emotion recognition results, and local model parameters. This persistent data design enables the system to have state memory capabilities, supporting contextual reasoning across time steps and local model accumulation in federated learning, avoiding data loss and model reset issues caused by power outages or restarts. The processor and storage media are connected via a data bus, ensuring high-throughput data transfer capabilities and meeting the stringent bandwidth and latency requirements of real-time multimodal data processing.Meanwhile, the processor connects to external smart devices via a communication interface to send device control commands. This interface supports multiple communication protocols, including WiFi, Zigbee, Bluetooth, and Ethernet. This multi-protocol compatibility design allows the system to seamlessly integrate with smart home device ecosystems from different manufacturers and using different standards, avoiding the device compatibility limitations of a single communication protocol and significantly expanding the system's applicability and deployment flexibility. Overall, this hardware architecture is tightly coupled with the aforementioned software modules, providing ample local computing and storage resources for complex artificial intelligence algorithms while enabling reliable interaction with the external environment through standardized communication interfaces. This forms a complete closed-loop hardware support system from data acquisition, fusion and understanding, intent prediction, collaborative decision-making to final device control execution, ensuring the operational stability, data security, and device compatibility of the smart home system in actual home deployments.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent butler system based on artificial intelligence, characterized in that, include: The multimodal data acquisition module (1) is used to acquire the user's multimodal data, which includes voice data, image data, text data, physiological signal data and environmental sensor data; A multimodal feature fusion module (2) is provided, the input of which is connected to the output of the multimodal data acquisition module (1), and is used to perform feature extraction and fusion processing on the multimodal data to generate a multimodal fusion feature vector. The context understanding module (3) is connected to the output of the multimodal feature fusion module (2) for contextual semantic understanding based on the multimodal fusion feature vector and generating contextual understanding results. The intention prediction module (4) is connected to the output of the context understanding module (3) and is used to perform time series modeling based on the context understanding results and user historical behavior data to predict the user's potential service needs within a preset future time window and generate intention prediction results. Multi-agent collaborative decision-making module (5), the input end of the multi-agent collaborative decision-making module (5) is connected to the output end of the intention prediction module (4), and is used to receive the intention prediction result, decompose the intention prediction result into multiple sub-tasks, and schedule multiple professional agents to execute the multiple sub-tasks respectively to generate collaborative decision results; An adaptive execution module (6) is provided, the input of which is connected to the output of the multi-agent collaborative decision-making module (5), for generating device control instructions based on the collaborative decision-making results and sending the device control instructions to the corresponding intelligent device for execution. The emotion perception driving module (7) is connected to the output of the multimodal feature fusion module (2) and the output of the emotion perception driving module (7) is connected to the control of the adaptive execution module (6). It is used to perform user emotion recognition based on the multimodal fusion feature vector, generate emotion recognition results, and dynamically adjust the service strategy of the adaptive execution module (6) according to the emotion recognition results. The federated continuous learning module (8) has its first data terminal connected to the data terminal of the multimodal data acquisition module (1), its second data terminal connected to the data terminal of the multimodal feature fusion module (2), its third data terminal connected to the data terminal of the context understanding module (3), its fourth data terminal connected to the data terminal of the intent prediction module (4), its fifth data terminal connected to the data terminal of the multi-agent collaborative decision-making module (5), and its sixth data terminal connected to the data terminal of the adaptive execution module (6). It is used to perform federated learning updates on the model parameters of the multimodal feature fusion module (2), the context understanding module (3), the intent prediction module (4), the multi-agent collaborative decision-making module (5), and the adaptive execution module (6) on the local device based on the multimodal data, and upload the encrypted model parameter update amount to the cloud server for global aggregation.

2. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The multimodal feature fusion module (2) uses a multimodal feature fusion algorithm based on a cross-modal attention mechanism to perform feature extraction and fusion processing. The multimodal feature fusion algorithm includes the following steps: The first of the multimodal data The raw data of each modality is mapped into modal feature vectors through the corresponding modal encoder. ,in N is the total number of modes; The following cross-modal attention calculation formula is used to calculate the first... The modal eigenvector relative to the th modality eigenvector Cross-modal attention weights of modal feature vectors : ， in, To query the weight matrix, The key weight matrix, Let be the dimension of the key vector. This represents the matrix transpose operation; The following multi-head cross-modal attention calculation formula is used to calculate the first... Enhanced feature vectors of each modality : ， in, M represents the number of attention heads. To output the weight matrix, For value weight matrix, This represents a vector concatenation operation; The multimodal fusion feature vector is generated using the following modal fusion calculation formula: ， in, For the first The adaptive fusion weights for each modality are dynamically generated through a learnable gating network and satisfy the following conditions: , For layer normalization operation, This is the fused feature vector from the previous time step.

3. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The intent prediction module (4) uses an intent prediction algorithm based on a time-series diffusion model to perform the time-series modeling and intent prediction. The time-series diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the historical behavior sequence until it becomes pure noise using the following formula: ， in, For the original historical behavior sequence, Let be the noisy sequence for the t-th diffusion step. , The total number of diffusion steps, Let be the preset noise variance at step t. Indicates a Gaussian distribution. It is the identity matrix; The reverse denoising process uses the following formula to gradually remove noise from pure noise and restore the predicted value of the future behavior sequence: ， in, For learnable parameters, The mean function predicted by the neural network. The variance function predicted by the neural network; The intent prediction module (4) generates a future behavior sequence prediction based on the context understanding result c using the following conditional diffusion formula. : ， in, The initial prediction sequence generated for the unconditional diffusion model. To guide the weighting coefficients, The guided loss function is based on the context understanding result c, and is calculated using the following formula: ， in, For a given prediction sequence The log-likelihood probability of the situational understanding result c under the given conditions.

4. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The multi-agent collaborative decision-making module (5) generates the collaborative decision-making result using a collaborative decision-making framework based on the multi-agent deep deterministic policy gradient algorithm. The collaborative decision-making framework includes K professional agents, each of which maintains its own policy network and evaluation network. The policy network outputs actions based on the local observation state of the professional agent, and the evaluation network is used to evaluate the value of the actions. The multi-agent collaborative decision-making module (5) adopts an architecture that combines centralized training and distributed execution. During the training phase, all professional agents share a centralized evaluation function. The centralized evaluation function takes the global state and the actions of all professional agents as input, outputs the global action value, and takes maximizing the cumulative discount reward as the optimization objective. Each specialized agent's policy network updates its parameters through the policy gradient method, introducing an entropy regularization term into the policy gradient to encourage exploration. The multi-agent collaborative decision-making module (5) also applies consensus and consistency constraints to the action outputs of each professional agent. Each professional agent adjusts its original action in coordination with the action outputs of other professional agents it communicates with, so that the final actions of each professional agent reach a consistent and coordinated state.

5. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The emotion perception driving module (7) includes a multi-branch feature extraction network, a dynamic weight allocation unit, an emotion evolution analysis unit, and an emotion-service mapping unit; The multi-branch feature extraction network is used to extract facial expression temporal features, speech prosody temporal features, physiological signal temporal features, and text semantic features from the multimodal fusion feature vector; The dynamic weight allocation unit is used to dynamically calculate the contribution weight of each modality feature in emotion recognition according to the current context, and to perform weighted fusion of each modality feature based on the contribution weight to obtain fused emotion features; The emotion evolution analysis unit is used to predict the future trend of emotion changes based on the current emotion state and the context environment, and generate emotion evolution trend information. The emotion-service mapping unit is used to generate service strategy adjustment parameters for adjusting the adaptive execution module (6) based on the fused emotion features and the emotion evolution trend information, and output the service strategy adjustment parameters to the control terminal of the adaptive execution module (6).

6. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The federated continuous learning module (8) performs the federated learning update using a differential privacy-based federated averaging algorithm, which includes the following steps: The federated continuous learning module (8) calculates the local model parameter update amount based on local multimodal data on the local device, and performs gradient pruning on the local model parameter update amount to obtain the pruned local model parameter update amount. The federated continuous learning module (8) adds Gaussian difference privacy noise to the cropped local model parameter update amount to obtain the noisy local model parameter update amount, and then encrypts the noisy local model parameter update amount and uploads it to the cloud server. The cloud server receives encrypted, noisy local model parameter update data uploaded by all local devices participating in federated learning. After decrypting the encrypted, noisy local model parameter update data, it performs a weighted average aggregation based on the number of local training samples of each local device to generate global model parameter update data. The cloud server distributes the global model parameter update to each local device, and each local device updates its own local model parameters according to the global model parameter update.

7. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The intelligent butler system also includes a cross-device context-aware scene automatic arrangement module (9). The input end of the scene automatic arrangement module (9) is connected to the output end of the context understanding module (3) and the output end of the intent prediction module (4), respectively. The output end of the scene automatic arrangement module (9) is connected to the input end of the multi-agent collaborative decision-making module (5). The scene automatic arrangement module (9) is used to match the target scene most similar to the current situation from the preset scene library based on the situation understanding result and the intention prediction result. When matching, the semantic features, time tags and location tags of the current situation are comprehensively considered to calculate the comprehensive similarity with the preset scene. The scene automatic arrangement module (9) is also used to automatically generate the device linkage scheme corresponding to the target scene when the comprehensive similarity exceeds the preset threshold, and send the device linkage scheme to the multi-agent collaborative decision module (5).

8. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The multimodal data acquisition module (1) includes a voice acquisition unit, an image acquisition unit, a text input interface, a physiological signal acquisition interface, and an environmental sensor interface; The voice acquisition unit is used to collect the user's voice data; The image acquisition unit is used to acquire the user's image data; The text input interface is used to receive text data input by the user; The physiological signal acquisition interface is used to acquire the user's physiological signal data from the wearable device; The environmental sensor interface is used to acquire environmental sensor data from environmental sensors deployed in the home environment, including temperature sensors, humidity sensors, light sensors, air quality sensors, and human infrared sensors.

9. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The multiple professional intelligent agents in the multi-agent collaborative decision-making module (5) include an environmental control intelligent agent, a health management intelligent agent, a safety monitoring intelligent agent, and an entertainment service intelligent agent; The environmental control agent is used to perform environmental parameter adjustment sub-tasks; The health management intelligent agent is used to perform sub-tasks of monitoring and early warning of user health status; The security monitoring intelligent agent is used to perform the sub-task of detecting and alarming home security anomalies; The entertainment service agent is used to perform the sub-tasks of multimedia content recommendation and playback.

10. The intelligent butler system based on artificial intelligence according to claim 1, characterized in that, The system further includes at least one processor and at least one non-volatile computer-readable storage medium, the at least one non-volatile computer-readable storage medium being connected to the at least one processor, and storing a computer program in the non-volatile computer-readable storage medium; When the computer program is executed by the at least one processor, the at least one processor enables the at least one processor to perform the functions of the multimodal data acquisition module (1), the multimodal feature fusion module (2), the context understanding module (3), the intent prediction module (4), the multi-agent collaborative decision-making module (5), the adaptive execution module (6), the emotion perception-driven module (7), the federated continuous learning module (8), and the scene automatic orchestration module (9); The non-volatile computer-readable storage medium is used to persistently store the multimodal data collected by the multimodal data acquisition module (1), the multimodal fusion feature vector generated by the multimodal feature fusion module (2), the context understanding result generated by the context understanding module (3), the intent prediction result generated by the intent prediction module (4), the collaborative decision-making result generated by the multi-agent collaborative decision-making module (5), the emotion recognition result generated by the emotion perception driving module (7), and the local model parameters of the federated continuous learning module (8). The at least one processor is connected to the at least one non-volatile computer-readable storage medium via a data bus, and the at least one processor is also connected to an external intelligent device via a communication interface for sending device control commands to the external intelligent device.

Images