Household appliance and control method and system thereof

Abstract

The invention discloses a household appliance and a control method and system thereof. The method comprises the following steps: acquiring multi-modal data; analyzing the multi-modal data by using a large language model to obtain at least two tasks; and allocating a corresponding target agent to each task by using the agent manager, so that the target agent controls the corresponding household appliance. By means of the mode, household appliance control with higher flexibility and higher intelligence is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of household electrical appliances, and in particular to household electrical appliances and control methods and systems thereof. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, smart home systems are moving towards a more intelligent, personalized and adaptive direction. However, the control strategies of related smart home systems lack flexibility and are difficult to adapt to the personalized needs of users. Summary of the invention

[0003] The household appliance and control method and system provided in the present application realize more flexible and intelligent control of the household appliance.

[0004] In a first aspect, the present application provides a method for controlling household appliances, the method comprising: acquiring multimodal data; parsing the multimodal data using a large language model to obtain at least two tasks; and using an agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding household appliance.

[0005] Among them, after using the agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance, it includes: recording the working results of the home appliance; and optimizing the agent manager and / or the large language model using the working results.

[0006] The method also includes: collecting user feedback information; optimizing the intelligent agent manager and / or the large language model using the work results, including: optimizing the intelligent agent manager and / or the large language model using the work results and user feedback information.

[0007] Among them, using a large language model to parse multimodal data to obtain at least two tasks, including: using a large language model to parse multimodal data to obtain intent corresponding to the multimodal data; decomposing tasks according to the intent to obtain at least two tasks.

[0008] Among them, the intelligent agent manager is used to assign a corresponding target intelligent agent to each task so that the target intelligent agent controls the corresponding home appliance, including: using the intelligent agent manager to assign a corresponding target intelligent agent to each task so that the target intelligent agent controls the home appliance corresponding to the target intelligent agent, and / or, the target intelligent agent cooperates with other intelligent agents to control the corresponding home appliance.

[0009] Among them, the intelligent agent manager is used to assign a corresponding target intelligent agent to each task, so that the target intelligent agent and other target intelligent agents collaborate to control the corresponding home appliances, including: using the intelligent agent manager to assign a corresponding target intelligent agent to each task; receiving an execution plan for collaboration between the target intelligent agent and other intelligent agents; and feeding back the confirmation result of the execution plan to the target intelligent agent, so that the target intelligent agent and other intelligent agents collaborate to control the corresponding home appliances according to the execution plan.

[0010] Among them, the multimodal data includes user instructions; the household appliances include at least two water-using devices, and the multimodal data is parsed using a large language model to obtain at least two tasks, including: using a large language model to parse the user instructions to obtain the water-saving intention corresponding to the user instructions; decomposing the tasks according to the water-saving intention to obtain the tasks corresponding to each water-using device.

[0011] Among them, the multimodal data includes user instructions; the household appliances include at least two kitchen appliances, and the multimodal data is parsed using a large language model to obtain at least two tasks, including: parsing the user instructions using a large language model to obtain the cooking intentions corresponding to the user instructions; decomposing the tasks according to the cooking intentions to obtain the tasks corresponding to each kitchen appliance.

[0012] In a second aspect, the present application provides a control system for a household appliance, the control system comprising: a communication interface for communicating with the household appliance; and a processor coupled to the communication interface for implementing the method provided in the first aspect.

[0013] In a third aspect, the present application provides a method for controlling a household appliance, the method comprising: receiving a control instruction; wherein the control instruction is obtained by the method provided in the first aspect; and operating according to the control instruction.

[0014] In a fourth aspect, the present application provides a household appliance, which includes a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method provided in the first aspect.

[0015] The beneficial effects of the present application are as follows: different from the prior art, the household appliance equipment and its control method and system provided by the present application use a large language model to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand the multimodal data; and use an intelligent agent manager to assign a corresponding target intelligent agent to each task, so that the target intelligent agent controls the corresponding household appliance equipment; wherein the target intelligent agent can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of household appliance equipment, and thus can handle more complex home control scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a structural schematic diagram of an embodiment of a control system for household appliances provided by the present application;

[0018] Figure 2 It is a flowchart of an embodiment of a multimodal data processing method provided by the present application;

[0019] Figure 3 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0020] Figure 4 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0021] Figure 5 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0022] Figure 6 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0023] Figure 7 This application provides Figure 6 A flow chart of an embodiment of step 62 in FIG.

[0024] Figure 8 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0025] Fig. 9 It is a flowchart of an embodiment of a control method for household electrical appliances provided by the present application;

[0026] Fig.10 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0027] Fig.11 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0028] Fig.12 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0029] Fig.13 It is a flowchart of an embodiment of a method for controlling a household appliance provided by the present application;

[0030] Fig.14 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0031] Fig.15 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0032] Fig.16 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0033] Fig.17 This application provides Fig.16 A flow chart of an embodiment of step 163;

[0034] Fig.18 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0035] Fig.19 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0036] Fig. 20 is a structural schematic diagram of another embodiment of a control system for a household appliance provided by the present application;

[0037] Fig.21 It is a flowchart of the cooperation mechanism between multiple agents provided by this application;

[0038] Fig. 22 This is a schematic diagram of another application scenario of the control system of the household appliance provided by the present application;

[0039] Fig.23 It is a structural schematic diagram of an embodiment of a household appliance provided by the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] See also Figure 1 , Figure 1 1 is a schematic diagram of a control system for a household appliance provided by the present application. The control system 1000 includes: a communication interface 100 and a processor 200. The communication interface 100 is used to communicate with the household appliance. The processor 200 is coupled to the communication interface 100 and is used to implement a corresponding method according to relevant data of the household appliance. For specific implementation methods, please refer to any of the following embodiments.

[0043] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a multimodal data processing method provided by the present application. The method comprises:

[0044] Step 21: Obtain multimodal data.

[0045] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0046] In some embodiments, text data may be user input commands and / or historical interaction records, etc. Image data may be environmental images captured by cameras on home appliances, user gestures, etc. Voice data may be user voice commands and / or environmental sounds, etc. Sensor data may be various types of sensor data such as temperature, humidity, light, and energy consumption.

[0047] In some embodiments, the multimodal data includes textual data and sensor data.

[0048] In some embodiments, the multimodal data includes image data and sensor data.

[0049] In some embodiments, the multimodal data includes voice data and sensor data.

[0050] In some embodiments, the multimodal data includes text data, voice data, and sensor data.

[0051] In some embodiments, the multimodal data includes text data, image data, and sensor data.

[0052] In some embodiments, the multimodal data includes voice data, image data, and sensor data.

[0053] In some embodiments, the multimodal data includes text data, image data, voice data, and sensor data.

[0054] In some embodiments, the sensor data may be temperature data, humidity data, or light data. That is, corresponding data may be collected using corresponding sensors. For example, temperature data may be collected using a temperature sensor, humidity data may be collected using a humidity sensor, and light data may be collected using a light sensor.

[0055] Step 22: Use at least two fusion methods to fuse the multimodal data to obtain multimodal fused data with a unified semantic space.

[0056] In some embodiments, a fusion method can be selected according to the specific type of multimodal data, and then the corresponding fusion method can be used to fuse the multimodal data to obtain multimodal fusion data with a unified semantic space. That is, by fusing the multimodal data, the multimodal data can be aligned in the semantic space, so that the correlation between the multimodal data is enhanced, which is more conducive to the subsequent recognition of the large language model and improves the reasoning accuracy of the large language model.

[0057] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0058] See also Figure 3 , Figure 3 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0059] Step 31: Acquire multimodal data; wherein the multimodal data at least includes text data and sensor data.

[0060] The sensor data is collected by the sensors corresponding to the home appliances.

[0061] Step 32: pre-process the sensor data in the multimodal data to obtain a first vector representation; and pre-process the text data in the multimodal data to obtain a text vector.

[0062] In some embodiments, the text data may be segmented and stop words may be removed, and the text may be converted into a vector using a pre-trained word embedding model to obtain a text vector.

[0063] In some embodiments, the text data may be segmented and stop words may be removed, and the text may be converted into a vector using a pre-trained word embedding model to obtain a text vector.

[0064] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0065] Step 33: Using at least two fusion methods, perform data fusion on the text vector and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0066] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0067] See also Figure 4 , Figure 4 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0068] Step 41: Acquire multimodal data; wherein the multimodal data includes image data and sensor data.

[0069] The sensor data is collected by the sensors corresponding to the home appliances.

[0070] Step 42: preprocess the sensor data in the multimodal data to obtain a first vector representation; preprocess the image data in the multimodal data to obtain image features.

[0071] In some embodiments, image data may be subjected to operations such as image scaling, cropping, and data enhancement, and then a pre-trained convolutional neural network may be used to extract image features.

[0072] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0073] Step 43: Using at least two fusion methods, perform data fusion on the image features and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0074] In this embodiment, at least two fusion methods are used to fuse multimodal data including image data and sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0075] See also Figure 5 , Figure 5 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0076] Step 51: Acquire multimodal data; wherein the multimodal data at least includes voice data and sensor data.

[0077] The sensor data is collected by the sensors corresponding to the home appliances.

[0078] Step 52: pre-process the sensor data in the multimodal data to obtain a first vector representation; pre-process the speech data in the multimodal data to obtain a second vector representation.

[0079] In some embodiments, the speech signal may be preprocessed to extract MFCC features, and the speech may be converted into text using a pre-trained speech recognition model, and then converted into a vector representation, that is, finally obtaining the above-mentioned second vector representation.

[0080] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0081] Step 53: Using at least two fusion methods, perform data fusion on the second vector representation and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0082] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least voice data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0083] See also Figure 6 , Figure 6 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0084] Step 61: Acquire multimodal data; wherein the multimodal data at least includes text data, image data, voice data and sensor data.

[0085] Step 62: Use at least two fusion methods to perform data fusion on the multimodal data to obtain multimodal fusion data with a unified semantic space.

[0086] Among them, at least two fusion methods include: attention mechanism fusion method; weighted fusion method; splicing fusion method.

[0087] Attention mechanism is a resource allocation scheme that aims to allocate limited computing resources to more important tasks. It originates from the study of human vision, which shows that humans selectively focus on some information and ignore other information when processing information. There are two main types of attention mechanisms: focused attention and saliency-based attention.

[0088] How the attention mechanism works: In deep learning, the attention mechanism enables the model to selectively focus on different parts of the input sequence by dynamically generating weights for different connections. For example, in the Transformer model, the degree of attention to each input vector is dynamically determined by calculating the degree of association between each input vector and other vectors. This mechanism can be applied to fields such as natural language processing and computer vision to help the model better capture key information.

[0089] That is, the attention mechanism can quickly capture the key information in different modal data in multimodal data, and then fuse the key information.

[0090] The weighted fusion method can set weights for data of different modalities and perform weighted fusion of modal data according to the corresponding weights. For example, the data of different modalities are processed in a unified dimension, and after the dimension is unified, the modal data is weighted fused according to the corresponding weights.

[0091] The splicing and fusion method can splice data of different modalities. In some embodiments, the priority of different modal data can be set and spliced ​​according to the priority. For example, data with high priority is spliced ​​in the front, and data with low priority is spliced ​​in the back. That is, sort from high to low according to priority, and then splice. In this way, when extracting subsequent data, high-priority data can be quickly extracted, thereby improving the accuracy of the high-priority data, making it easier for the subsequent large language model to more accurately obtain the features of the modal data and understand its semantic information.

[0092] In some embodiments, different modal data can be initially fused by selecting corresponding fusion methods, and then the initially fused data can be fused again in a unified manner to obtain multi-modal fusion data.

[0093] In some embodiments, see Figure 7 , step 62 may be the following process:

[0094] Step 621: Use the attention mechanism fusion method to fuse the multimodal data to obtain the first fused data.

[0095] In some embodiments, the attention mechanism is used to extract the value vector, key vector, and query vector from the multimodal data. The key vector and the query vector are then used to calculate the correlation between the key vector and the query vector, and the attention score is obtained by the softmax function. Then, a weighted sum is performed based on the attention score to obtain a value vector with an attention score. That is, the first fused data can reflect the degree of matching between different modal data, so that data alignment can be achieved.

[0096] In some embodiments, the attention mechanism may include a self-attention mechanism, a cross-attention mechanism, and / or a multi-head attention mechanism.

[0097] Step 622: Perform data fusion on the multimodal data using a weighted fusion method to obtain second fused data.

[0098] Step 623: Use a splicing fusion method to fuse the multimodal data to obtain third fused data.

[0099] In some embodiments, the weighted fusion method and the splicing fusion method may use residual connection and layer normalization technology.

[0100] Residual connection refers to the process of passing the input of the previous layer directly to the next layer in a deep neural network, that is, adding cross-layer connections. This method can alleviate the problems of gradient disappearance and gradient explosion, making the effective depth of the deep neural network deeper, thereby improving the performance of the network.

[0101] Layer Normalization is a normalization technique that is more suitable for RNN and long-term series than batch normalization. The basic idea is to normalize all neurons of each sample in the neural network, that is, to normalize the mean and variance of the neurons with respect to the sample, thereby reducing the dependency between neurons and avoiding model instability caused by changes in batch size. It can effectively eliminate the correlation between each layer and improve the convergence and performance of the network. Compared with batch normalization, layer normalization is more stable and more robust to noise and small batches.

[0102] Step 624: Fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0103] In some embodiments, a fusion representation generation technology may be used to fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0104] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data, image data, voice data and sensor data, and obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0105] See also Figure 8 , Figure 8 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0106] Step 81: Acquire multimodal data.

[0107] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0108] Step 82: Use at least two fusion methods to perform data fusion on the multimodal data to obtain multimodal fusion data with a unified semantic space.

[0109] In some embodiments, steps 81 to 82 have the same or similar technical solutions as any embodiment of the present application, and are not repeated here.

[0110] Step 83: Input the multimodal fusion data into the large language model to obtain the contrast loss and the multi-task learning loss.

[0111] In some embodiments, the large language model can use a pre-trained GPT model as the core and fine-tune it for the home appliance field. The main features include:

[0112] 1. Using the Transformer architecture, it has powerful sequence modeling capabilities.

[0113] 2. Conduct pre-training on large-scale household appliance corpus to master domain knowledge.

[0114] 3. Support multi-modal input and can process fused feature representation.

[0115] 4. Use autoregressive generation to achieve flexible text generation and conversation capabilities.

[0116] In some embodiments, multi-task learning includes at least two of anomaly detection tasks, energy consumption optimization tasks, behavior prediction tasks, state estimation tasks, and instruction understanding tasks.

[0117] The anomaly detection task mainly relies on multimodal fusion data to detect anomalies in the working status of home appliances. For example, sensor data and status data of home appliances can be extracted from multimodal fusion data to perform anomaly detection.

[0118] The main purpose of the energy consumption optimization task is to control household appliances to achieve corresponding tasks with the lowest energy consumption, so as to optimize the energy consumption of household appliances.

[0119] The main purpose of the behavior prediction task is to predict user behavior through voice data and text data in multimodal fusion data, so as to control the corresponding home appliances according to the predicted behavior.

[0120] The state estimation task mainly estimates the working state of home appliances based on multimodal fusion data. For example, the state of home appliances can be estimated by extracting sensor data from multimodal fusion data, and then the corresponding estimation results can be given.

[0121] The main purpose of the command understanding task is to understand the command through the voice data and text data in the multimodal fusion data, so as to control the corresponding home appliances according to the understood commands.

[0122] Step 84: Perform joint optimization based on contrast loss and multi-task learning loss, and adjust the weights of relevant data in multimodal data.

[0123] In some embodiments, the contrast loss and the multi-task learning loss may be weightedly summed to obtain a final total loss, and the total loss may be used to adjust the weights of relevant data in the multimodal data.

[0124] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0125] See also Fig. 9 , Fig. 9 This is a flow chart of an embodiment of a method for controlling a household appliance provided by the present application. The method includes:

[0126] Step 91: Acquire multimodal fusion data.

[0127] The multimodal fusion data is obtained by using the method provided in any embodiment of the present application.

[0128] Step 92: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0129] Among them, the large language model is a large language model that has been trained in advance.

[0130] Step 93: Control the corresponding home appliances to work according to the multi-task decision.

[0131] In some embodiments, a corresponding intelligent agent is provided in the household appliance, and the intelligent agent can generate corresponding control instructions according to the corresponding task decision, thereby controlling the operation of the corresponding household appliance.

[0132] In some embodiments, the agent has the ability to make autonomous decisions. For example, if the multi-task decision determines that the home appliance needs to be turned on, the agent can independently determine the turning-on mode of the home appliance in combination with the current environmental data. For example, if the home appliance is an air conditioner, when the air conditioner needs to be turned on in the multi-task decision, a command to turn on the air conditioner is sent to the agent. The agent determines the turning-on temperature in combination with the current environmental data, and after determining the turning-on temperature, controls the air conditioner to start and work according to the turning-on temperature.

[0133] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0134] See also Fig.10 , Fig.10 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0135] Step 101: Acquire multimodal fusion data.

[0136] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0137] Step 102: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0138] Step 103: Perform safety check on the multi-task decision.

[0139] In some embodiments, the safety check mainly checks whether the home appliance control solution corresponding to the multi-task decision is safe during execution, such as whether there is an abnormal execution risk.

[0140] Step 104: After the safety check is passed, the corresponding home appliance is controlled to work according to the multi-task decision; or, when the safety check is not passed, the multi-task decision is corrected.

[0141] After the safety check is passed, the corresponding home appliances are controlled to work according to the multi-task decision. That is, after the safety check is passed, the task decision is sent to the corresponding home appliance. The home appliance is equipped with a corresponding intelligent agent, which can generate corresponding control instructions according to the corresponding task decision, and then control the corresponding home appliance.

[0142] When the safety check fails, the multi-task decision is revised so that the revised multi-task decision can successfully pass the safety check. After the safety check passes, the corresponding home appliances are controlled to work according to the revised multi-task decision.

[0143] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0144] Furthermore, by performing a safety check on the multi-task decision, the task decision is sent to the corresponding home appliance after the check is passed, or when the safety check fails, the multi-task decision is corrected to reduce the occurrence of sending erroneous task decisions to home appliances, thereby improving the control accuracy of home appliances.

[0145] See also Fig.11 , Fig.11 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0146] Step 111: Acquire multimodal fusion data.

[0147] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0148] Step 112: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0149] Step 113: Control the corresponding home appliances to work according to the multi-task decision.

[0150] In some embodiments, steps 111 to 113 have the same or similar technical solutions as any embodiment of the present application, and are not described in detail here.

[0151] Step 114: Record the working results of the home appliance and collect user feedback information.

[0152] Step 115: Update the large language model based on the work results and user feedback information.

[0153] In this embodiment, updating the large language model in combination with work results and user feedback information can improve the accuracy of the large language model in task reasoning, and make the reasoning of the large language model more in line with user preferences, thereby improving user experience.

[0154] See also Fig.12 , Fig.12 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0155] Step 121: Acquire multimodal fusion data.

[0156] Among them, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0157] Step 122: Input the multimodal fusion data into the large language model so that the large language model determines at least two task types.

[0158] In some embodiments, the large language model is trained for multiple tasks during the training process, so the large language model can perform multi-task reasoning. When the multimodal fusion data indicates that multiple tasks need to be implemented, the large language model can obtain at least two task types based on the multimodal fusion data.

[0159] Step 123: Outputting a task decision corresponding to each task type according to at least two task types to control the operation of the corresponding household electrical appliance.

[0160] Step 124: Control the corresponding home appliance to work according to the multi-task decision.

[0161] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0162] Furthermore, the large language model can process more complex multimodal fusion data, determine at least two task types from the multimodal fusion data, and then generate corresponding task decisions according to the task types to control the operation of home appliances.

[0163] In the above embodiments, effective fusion of multimodal data in the field of home appliances is achieved, breaking down data silos; precise alignment of data of different modalities is achieved in a unified semantic space, improving the model's understanding ability; significantly improving the generalization ability and robustness of the model in home appliance scenarios; achieving a more natural and intelligent human-computer interaction experience, improving user satisfaction; improving the accuracy and efficiency of intelligent control of home appliances, and reducing misoperation and energy waste.

[0164] With the rapid development of artificial intelligence and Internet of Things technologies, smart home systems are moving towards a more intelligent, personalized and adaptive direction. However, the control strategies of related smart home systems lack flexibility and are difficult to adapt to the personalized needs of users.

[0165] Based on this, the present application proposes to use a large language model to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand multimodal data; and use an agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein the target agent can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent home appliance control, and thus be able to handle more complex home control scenarios. For details, please refer to any of the following embodiments.

[0166] See also Fig.13 , Fig.13 This is a flow chart of an embodiment of a method for controlling a household appliance provided by the present application. The method includes:

[0167] Step 131: Acquire multimodal data.

[0168] The multimodal data may be the multimodal data mentioned in any of the above embodiments.

[0169] Step 132: Utilize the large language model to parse the multimodal data and obtain at least two tasks.

[0170] In some embodiments, a large language model may be used to parse multimodal data, and then perform task decomposition to obtain at least two tasks.

[0171] Step 133: Utilize the agent manager to assign a corresponding target agent to each task, so that the target agent controls the corresponding home appliance.

[0172] In some embodiments, each target agent is deployed in a corresponding home appliance. The target agent has autonomous decision-making capabilities.

[0173] The agent manager can be set in the central control device of the home system. The central control device can communicate with the home appliances corresponding to each agent.

[0174] The agent manager finds the agent corresponding to each task and assigns the corresponding task to the agent. After receiving the task, the agent generates the corresponding control instruction according to the task and the environment data, and uses the control instruction to control the corresponding home appliance.

[0175] In some embodiments, one target agent can be deployed for the same type of home appliances. That is, one target agent can control at least one home appliance. In other embodiments, one target agent can control multiple home appliances.

[0176] In some embodiments, an agent group may be formed according to the type of home appliances. That is, each target agent in the agent group corresponds to a type of home appliance. The target agent may control the home appliance of the type. The target agent is deployed in one of the home appliances of the type, and the home appliances of the type may communicate with each other. For example, the target agent may be deployed in the home appliance with the best device performance among the home appliances of the type.

[0177] After the target agent receives the task, it generates corresponding control instructions according to the task and the environmental data, and determines the target home appliance from a number of home appliances of the same type, and uses the control instructions to control the corresponding target home appliance to work. In some embodiments, the number of target home appliances is determined according to the task. If the task requires multiple home appliances, the number of target home appliances is multiple.

[0178] In this embodiment, a large language model is used to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand the multimodal data; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein the target agent is able to understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus be able to handle more complex home control scenarios.

[0179] See also Fig.14 , Fig.14 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0180] Step 141: Acquire multimodal data.

[0181] Step 142: Utilize the large language model to parse the multimodal data and obtain at least two tasks.

[0182] Step 143: Utilize the agent manager to assign a corresponding target agent to each task, so that the target agent controls the corresponding home appliance.

[0183] Among them, each target intelligent agent is deployed in the corresponding home appliance, and the target intelligent agent has the ability to make autonomous decisions.

[0184] Steps 141 to 143 have the same or similar technical solutions as any embodiment of the present application.

[0185] Step 144: Record the working results of the household electrical appliance.

[0186] In some embodiments, after the household appliances perform corresponding tasks, the working results of each household appliance are recorded.

[0187] Step 145: Use the work results to optimize the agent manager and / or large language model.

[0188] In some embodiments, the agent manager and / or the large language model are updated using the work results. The large language model can quickly learn the user's actual preferences and habits, so that the large language model can more accurately identify the intention results corresponding to the multimodal data, and improve the accuracy of subsequent control of home appliances. The agent manager can also better adjust the results of the assigned tasks based on the work results.

[0189] Further, user feedback information is collected; the agent manager and / or the large language model are optimized using the work results and user feedback information. In some embodiments, the large language model is updated in combination with the work results and user feedback information. The user feedback information allows the large language model to quickly learn the user's actual preferences and habits, so that the large language model can more accurately identify the intention results corresponding to the multimodal data, thereby improving the accuracy of subsequent control of home appliances.

[0190] In this embodiment, a large language model is used to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand the multimodal data; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein the target agent is able to understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus be able to handle more complex home control scenarios.

[0191] Furthermore, the work results are used to optimize the agent manager and / or large language model, which enhances the system's adaptability and learning ability, and can continuously optimize the control strategy according to user behavior. It can also achieve more sophisticated and personalized home appliance control, which can adapt to the needs and preferences of different users.

[0192] See also Fig.15 , Fig.15 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0193] Step 151: Acquire multimodal data.

[0194] Step 152: Analyze the multimodal data using a large language model to obtain the intent corresponding to the multimodal data.

[0195] In some embodiments, before the large language model parses the multimodal data, the multimodal data is fused. Then the fused multimodal data is parsed using the large language model to obtain the intent result corresponding to the multimodal data. The fusion process of multimodal data can refer to the corresponding embodiment of this application, which will not be described here.

[0196] Step 153: Decompose the tasks according to the intention to obtain at least two tasks.

[0197] In some embodiments, the multimodal data corresponds to multiple sub-intents, and each sub-intent can be realized by a corresponding home appliance. Therefore, the task is decomposed according to the intent to obtain at least two tasks. In this way, a large language model can be used to convert complex multimodal data into the form of multiple tasks, which facilitates the rapid understanding of complex instructions, improves the execution efficiency of home appliances, and thus improves the user experience.

[0198] Step 154: Utilize the agent manager to assign a corresponding target agent to each task, so that the target agent controls the corresponding home appliance.

[0199] In some embodiments, each target agent is deployed in a corresponding home appliance. The target agent has autonomous decision-making capabilities.

[0200] In some embodiments, one target agent can be deployed for the same type of home appliances. That is, one target agent can control at least one home appliance. In other embodiments, one target agent can control multiple home appliances.

[0201] In some embodiments, an agent group may be formed according to the type of home appliances. That is, each target agent in the agent group corresponds to a type of home appliance. The target agent may control the home appliance of the type. The target agent is deployed in one of the home appliances of the type, and the home appliances of the type may communicate with each other. For example, the target agent may be deployed in the home appliance with the best device performance among the home appliances of the type.

[0202] After the target agent receives the task, it generates corresponding control instructions according to the task and the environmental data, and determines the target home appliance from a number of home appliances of the same type, and uses the control instructions to control the corresponding target home appliance to work. In some embodiments, the number of target home appliances is determined according to the task. If the task requires multiple home appliances, the number of target home appliances is multiple.

[0203] In this embodiment, a large language model is used to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand multimodal data; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein each target agent is deployed in a corresponding home appliance, and the target agent has autonomous decision-making capabilities, can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus can handle more complex home control scenarios.

[0204] See also Fig.16 , Fig.16 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0205] Step 161: Acquire multimodal data.

[0206] Step 162: Utilize the large language model to parse the multimodal data and obtain at least two tasks.

[0207] Step 163: Use the agent manager to assign a corresponding target agent to each task, so that the target agent controls the home appliance corresponding to the target agent, and / or the target agent collaborates with other agents to control the corresponding home appliance.

[0208] In some embodiments, an agent manager is used to assign a corresponding target agent to each task, so that the target agent controls the home appliance corresponding to the target agent.

[0209] In some embodiments, an agent manager is used to assign a corresponding target agent to each task, so that the target agent controls the home appliance corresponding to the target agent, and the target agent collaborates with other agents to control the corresponding home appliance.

[0210] In some embodiments, an agent manager is used to assign a corresponding target agent to each task, so that the target agent collaborates with other agents to control the corresponding home appliances.

[0211] In some embodiments, each target agent is deployed in a corresponding home appliance. The target agent has autonomous decision-making capabilities.

[0212] In some embodiments, the remaining agents may not be the target agents assigned to each task by the agent manager this time, but non-remaining agents among the agents. For example, the target agents assigned to each task by the agent manager include agent A, agent B, and agent C. If agent A finds that it needs to cooperate with agent D, then agent A is the target agent and agent D is the remaining target agent.

[0213] In some embodiments, the remaining agents can use the agent manager to assign corresponding target agents to each task except the current target agent. For example, the target agents assigned to each task using the agent manager include agent A, agent B, and agent C. Agent A finds that it needs to cooperate with agent B, then agent A is the target agent and agent B is the remaining target agent. It can be understood that the target agent may need at least one remaining target agent to cooperate in controlling the corresponding home appliance. For example, two remaining target agents are needed to cooperate in controlling the corresponding home appliance, and three remaining target agents are needed to cooperate in controlling the corresponding home appliance.

[0214] In some embodiments, if multiple tasks or a single task is involved, the household appliances need to collaborate to complete the task. When the target agent makes a decision and detects that assistance from other agents is needed, assistance information is sent to the corresponding other target agents, so that the other target agents combine the assistance information with their own tasks to generate corresponding execution plans.

[0215] In some embodiments, see Fig.17 , step 163 may be the following process:

[0216] Step 1631: Use the agent manager to assign a corresponding target agent to each task.

[0217] Step 1632: Receive an execution plan for collaboration between the target agent and other agents.

[0218] When the target intelligent agent makes a decision and detects that it needs assistance from other intelligent agents, it will send assistance information to the corresponding other target intelligent agents, so that the other target intelligent agents can combine the assistance information and their own tasks to generate corresponding execution plans, thereby cooperating with the target intelligent agent to generate a collaborative execution plan between the target intelligent agent and the other intelligent agents.

[0219] The agent manager receives the execution plan of the collaboration between the target agent and other agents, checks the execution plan, and obtains the corresponding confirmation result.

[0220] Step 1633: Feedback the confirmation result of the execution plan to the target agent, so that the target agent and other agents can collaborate to control the corresponding home appliances according to the execution plan.

[0221] When the result determines that the execution plan is feasible, the target agent and other agents control the corresponding home appliances according to the execution plan.

[0222] When it is determined that the execution plan is not feasible, the target agent and other target agents need to re-plan their collaboration, formulate a new execution plan, and control the corresponding home appliances according to the new execution plan when the new execution plan is feasible.

[0223] In this embodiment, a large language model is used to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand multimodal data; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein each target agent is deployed in a corresponding home appliance, and the target agent has autonomous decision-making capabilities, can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus can handle more complex home control scenarios.

[0224] Furthermore, designing intelligent entities in home appliances can improve the collaboration between home appliances and realize complex cross-device control scenarios.

[0225] See also Fig.18 , Fig.18 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0226] Step 181: Acquire multimodal data; the multimodal data includes user instructions.

[0227] In some embodiments, the user instruction may include a voice instruction or a text instruction. The voice instruction may be output by the user according to the water-saving suggestion. The text instruction may be output by the user according to the water-saving suggestion.

[0228] Step 182: Analyze the user instruction using the large language model to obtain the water-saving intention corresponding to the user instruction.

[0229] In some embodiments, before the large language model parses the multimodal data, the multimodal data is fused. Then the fused multimodal data is parsed using the large language model to obtain the intent result corresponding to the multimodal data. The fusion process of multimodal data can refer to the corresponding embodiment of this application, which will not be described here.

[0230] Step 183: Decompose the tasks according to the water-saving intention to obtain the tasks corresponding to each water-using equipment.

[0231] In the embodiments of the present application, since water saving is involved, the household appliances may be various types of water-consuming appliances, such as a water dispenser, a washing machine, a water heater, a water softener and / or a dishwasher.

[0232] These water-consuming appliances work according to the corresponding intelligent tasks to achieve the overall water-saving task.

[0233] Step 184: Use the agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding water-using equipment.

[0234] Among them, each target intelligent agent is deployed in the corresponding home appliance, and the target intelligent agent has the ability to make autonomous decisions.

[0235] In this embodiment, a large language model is used to parse user instructions of multimodal data to obtain at least two tasks, which can effectively integrate and understand user instructions; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein each target agent is deployed in a corresponding home appliance, and the target agent has autonomous decision-making capabilities, can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus can handle more complex home control scenarios.

[0236] See also Fig.19 , Fig.19 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0237] Step 191: Acquire multimodal data; the multimodal data includes user instructions.

[0238] In some embodiments, the user instruction may indicate that the user wants to cook a dish.

[0239] Step 192: Analyze the user instruction using the large language model to obtain the cooking intention corresponding to the user instruction.

[0240] In some embodiments, cooking intentions may include intentions such as dishes and ingredients corresponding to the dishes.

[0241] Step 193: Decompose the tasks according to the cooking intention to obtain the tasks corresponding to each kitchen equipment.

[0242] Step 194: Use the agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding kitchen equipment.

[0243] Among them, each target intelligent agent is deployed in the corresponding home appliance, and the target intelligent agent has the ability to make autonomous decisions.

[0244] In some embodiments, when determining that the user instruction corresponds to the cooking intention, it is necessary to detect the food inventory in the refrigerator and perform a replenishment operation based on the food inventory. For example, if the food inventory is insufficient, the refrigerator automatically places an order to purchase the corresponding food in an Internet store. Specifically, the corresponding amount of food can be ordered according to the needs of the dish.

[0245] When the food inventory meets the cooking needs, the corresponding home appliances are controlled to work. For example, the cooking process is displayed on the smart chopping board, and the corresponding video is played to guide the user to cook. For example, the oven is preheated in advance. The hot water temperature of the faucet is controlled in advance. The rice cooker is controlled in advance. In other words, according to the actual dishes to be cooked, the home appliances related to the dishes are determined, and the corresponding tasks are assigned to these home appliances, thereby assisting the user to complete the cooking of the dishes.

[0246] In this embodiment, a large language model is used to parse user instructions of multimodal data to obtain at least two tasks, which can effectively integrate and understand user instructions; and an agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance; wherein each target agent is deployed in a corresponding home appliance, and the target agent has autonomous decision-making capabilities, can understand and execute tasks corresponding to complex, context-related multimodal data, achieve more flexible and intelligent control of home appliances, and thus can handle more complex home control scenarios.

[0247] See also Fig. 20 The control system 1000 of the household appliance mainly includes the following modules: a multimodal input interface 201, a multimodal basic model 202, an intelligent agent manager 203, a household appliance control intelligent agent group 204, an environment perception module 205 and a data storage and analysis module 206.

[0248] The multimodal input interface 201 is responsible for receiving and preliminarily processing the multimodal input of the user, such as voice, image, text, etc.

[0249] The multimodal basic large model 202 is the core component of the system, responsible for understanding user intentions, parsing complex instructions, and generating advanced control strategies.

[0250] The agent manager 203 is responsible for task decomposition and allocation, and coordinates the work of the agents of various household appliances.

[0251] The home appliance control agent group 204 includes multiple agents, each of which is responsible for controlling a specific type of home appliance.

[0252] The environment sensing module 205 is used to collect and process real-time data of the home environment.

[0253] The data storage and analysis module 206 is used to record system operation data and support continuous learning and optimization.

[0254] The working process of the control system 1000 of the household appliance is as follows:

[0255] Users input commands through multimodal methods (such as voice, gestures, APP operations, etc.).

[0256] The multimodal input interface 201 receives and pre-processes user input.

[0257] The multimodal basic model 202 combines the data provided by the environmental perception module to understand the user's intention and generate a task plan.

[0258] The agent manager 203 decomposes the tasks and assigns them to corresponding home appliance control agents.

[0259] Each intelligent agent works together to execute control commands.

[0260] The control system collects execution results and user feedback and stores them in the data storage and analysis module.

[0261] The control system continuously learns and optimizes based on accumulated data to improve performance.

[0262] In some embodiments, Fig.21 As shown in the figure, the collaboration mechanism between multiple agents is as follows:

[0263] The user sends a multimodal instruction. The multimodal big model parses the instruction and assigns tasks.

[0264] The agent manager receives task instructions from the multimodal large model.

[0265] The agent manager 203 assigns tasks to relevant agents, such as agent 1, agent 2, and agent 3. Among them, agent 1, agent 2, and agent 3 correspond to different home appliances.

[0266] Each agent formulates a preliminary execution plan based on its own knowledge base and current status.

[0267] Agents exchange information, coordinate actions, and resolve potential conflicts through communication interfaces.

[0268] The final execution plan is returned to the agent manager for confirmation.

[0269] After confirmation, each agent executes the corresponding control command.

[0270] The execution results are reported to the agent manager and ultimately fed back to the user.

[0271] In some embodiments, in combination Fig. 22,In the smart cooking scenario, the workflow of the control system is as follows:

[0272] The user selects the dish to be made: "Braised Fish".

[0273] The control system understands the task and starts coordinating the various kitchen devices. For example, the refrigerator agent checks the ingredients needed. The smart cutting board agent shows the steps to cut the fish. The induction cooker agent presets the heat and cooking time. The range hood agent is ready.

[0274] During the cooking process, the control system makes intelligent adjustments based on real-time data. For example, the induction cooker automatically adjusts the firepower based on the temperature sensor data. Image recognition is used to determine the degree of cooking of the fish, and the user is reminded to turn it over or add ingredients. When changes in oil smoke concentration are detected, the range hood automatically adjusts the suction power. After cooking is completed, the system pushes reminders, records data for future optimization, and recommends tableware and drinks.

[0275] In some embodiments, the workflow of the household water optimization scenario is as follows:

[0276] The user issues a command to "optimize water use at home". The control system analyzes recent water use data, detects water quality, and analyzes water use habits. The control system formulates and executes optimization plans. For example:

[0277] Smart faucet agent: adjust water flow and set automatic shut-off time. Water heater agent: optimize heating time and avoid long standby time. Washing machine agent: recommend water-saving washing mode. Toilet agent: adjust flushing volume according to usage.

[0278] The system continuously monitors water usage, conducts regular water quality tests, and pushes water-saving suggestions and water quality reports to users.

[0279] The data collection function of the control system, the control system continuously collects information such as user operations, equipment status, and environmental data.

[0280] The data analysis function of the control system can use machine learning algorithms to analyze the collected data and identify user habits and preferences.

[0281] Model update capability: Regularly update multimodal large models with new data to improve understanding and decision-making capabilities.

[0282] Intelligent agent optimization function: optimize the control strategy and cooperation mechanism of each intelligent agent based on the analysis results.

[0283] Personalized recommendation function: Based on the learning results, personalized scenarios and control solutions are recommended to users.

[0284] In some embodiments, the present application also provides a method for controlling a household appliance, the method comprising: receiving a control instruction; wherein the control instruction is obtained by a multi-task decision generated by the method of any of the above embodiments; and working according to the control instruction.

[0285] See also Fig.23 , Fig.23 It is a structural diagram of an embodiment of a household appliance provided in the present application. The household appliance 300 includes a communication interface 301 and a processor 302. The processor 302 is coupled to the communication interface 301 to implement the method of any of the above embodiments.

[0286] In summary, the household appliance equipment and its control method and system provided by the present application use a large language model to parse multimodal data to obtain at least two tasks, which can effectively integrate and understand multimodal data; and use an intelligent agent manager to assign a corresponding target intelligent agent to each task so that the target intelligent agent controls the corresponding household appliance equipment; wherein each target intelligent agent is deployed in the corresponding household appliance equipment, and the target intelligent agent has autonomous decision-making capabilities, can understand and execute tasks corresponding to complex, context-related multimodal data, and achieve more flexible and intelligent control of household appliance equipment, thereby being able to handle more complex home control scenarios.

[0287] Finally, the above terms are explained:

[0288] Multimodal input: refers to the system's ability to process multiple forms of input information at the same time, such as text, voice, image, video, etc. In this system, it mainly refers to the user's ability to interact with the system through multiple methods such as voice, gesture, and text.

[0289] Multimodal base large model: A large-scale AI model that can process and understand multiple forms of input (such as text, images, audio, etc.). For example, the model used is similar to GPT-4Vision, which can understand and process information from multiple modalities at the same time.

[0290] Agent: In this application, it refers to a software entity that can make decisions and perform tasks autonomously. Each agent is responsible for controlling a specific type of home appliance and has an independent knowledge base and decision-making capabilities.

[0291] Agent Manager: A module responsible for coordinating and managing all agents in the system. It receives instructions from the multimodal large model, decomposes and assigns tasks to corresponding agents, and coordinates cooperation among agents.

[0292] Environmental perception module: A system component responsible for collecting and processing real-time data about the home environment, including temperature, humidity, light, air quality, etc. This data is used to assist the system in making more accurate decisions.

[0293] Continuous Learning: The process by which a system continuously learns from its daily operations and user interactions, optimizing its performance and decision-making capabilities. This enables the system to become more intelligent and personalized over time.

[0294] Distributed control: A control method that distributes control tasks to multiple independent but cooperative control units (in this system, intelligent agents). This method improves the flexibility and scalability of the system.

[0295] Dynamic scene understanding: The system’s ability to understand and analyze the current home environment and user needs in real time, including identifying user intent, environmental status, and potential control needs.

[0296] GPT (Generative Pre-trained Transformer): A large-scale language model based on the Transformer architecture that can understand and generate natural language. The multimodal base model of this system is based on similar principles, but extended to handle multimodal inputs.

[0297] Smart home appliances: Home appliances that can be connected through the Internet and accept intelligent control, such as smart air conditioners, smart refrigerators, smart lighting, etc. These devices are the objects controlled by this system.

[0298] Fine-tuning: Based on the pre-trained model, further training is performed using data from a specific field to make the model better suited to a specific task. In this system, the multimodal large model is fine-tuned to adapt to the smart home scenario.

[0299] Collaborative decision-making: The process in which multiple agents participate together, exchange information and coordinate actions to reach the optimal control solution. This approach can handle complex cross-device control scenarios.

[0300] User portrait: A comprehensive description of user habits, preferences, and behavior patterns formed by the system through long-term learning and analysis. User portraits are used to personalize services and predict user needs.

[0301] Scene mode: refers to specific home usage scenarios, such as "getting up in the morning", "going home mode", "cinema mode", etc. The system can recognize and automatically trigger these scene modes and perform corresponding device control.

[0302] Adaptive Optimization: The process by which the system is able to automatically adjust and optimize control strategies based on real-time feedback and historical data. This enables the system to continually improve its performance and user satisfaction.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0304] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processing circuit component (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0305] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for controlling a household appliance, characterized in that: The method comprises: Acquire multimodal data; Parsing the multimodal data using a large language model to obtain at least two tasks; The agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding household electrical appliance.

2. The method according to claim 1, characterized in that After the agent manager is used to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance, the method includes: Recording the working results of the household electrical appliances; The working results are used to optimize the agent manager and / or the large language model.

3. The method according to claim 2, characterized in that The method further comprises: Collect user feedback information; The step of optimizing the agent manager and / or the large language model using the work results includes: The working results and the user feedback information are used to optimize the agent manager and / or the large language model.

4. The method according to claim 1, characterized in that: The method of using the agent manager to assign a corresponding target agent to each task so that the target agent controls the corresponding home appliance includes: The agent manager is used to assign a corresponding target agent to each task, so that the target agent controls the home appliance corresponding to the target agent, and / or the target agent cooperates with other agents to control the corresponding home appliance.

5. The method according to claim 4, characterized in that The method of using the agent manager to assign a corresponding target agent to each task, so that the target agent cooperates with other agents to control the corresponding home appliance, includes: Use the agent manager to assign the corresponding target agent to each task; receiving an execution plan for collaboration between the target agent and the remaining agents; The confirmation result of the execution plan is fed back to the target agent, so that the target agent and the other agents can collaboratively control the corresponding household appliances according to the execution plan.

6. The method according to claim 1, characterized in that The multimodal data includes user instructions; the household appliances include at least two water-using appliances, and the multimodal data is parsed using a large language model to obtain at least two tasks, including: Analyzing the user instruction using a large language model to obtain a water-saving intention corresponding to the user instruction; The tasks are decomposed according to the water-saving intention to obtain the tasks corresponding to each water-using equipment.

7. The method according to claim 1, characterized in that The multimodal data includes user instructions; the household appliances include at least two kitchen appliances, and the multimodal data is parsed using a large language model to obtain at least two tasks, including: parse the user instruction using a large language model to obtain a cooking intention corresponding to the user instruction; The tasks are decomposed according to the cooking intention to obtain the tasks corresponding to each kitchen equipment.

8. A control system for household electrical appliances, characterized in that: The control system comprises: A communication interface, used to communicate with home appliances; A processor, coupled to the communication interface, configured to implement the method according to any one of claims 1 to 7.

9. A method for controlling a household appliance, characterized in that: The method comprises: Receiving a control instruction; wherein the control instruction is obtained by any method described in claims 1-7; Works according to the control instructions described.

10. A household appliance, characterized in that: The household appliance comprises a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method according to claim 9.

Images