Household appliance and control method thereof

Abstract

The invention discloses a household appliance and a control method thereof. The method comprises the steps of obtaining a user instruction; performing scene recognition and task decomposition on the user instruction by using a large language model to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene; wherein each scene at least corresponds to one household appliance, and each scene corresponds to one scene agent; allocating a corresponding equipment agent to each task by using the scene agent, so that the equipment agent controls the corresponding household appliance; wherein each device intelligent agent is deployed in the corresponding household appliance. Through the above mode, complex cross-equipment scene linkage can be realized, and the scene control capability is improved.

Description

Technical Field

[0001] The present application relates to the technical field of household electrical appliances, and in particular to household electrical appliances and control methods thereof. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, smart home systems are developing in a more intelligent, personalized and adaptive direction. However, related smart home systems are difficult to achieve complex cross-device scene linkage. Summary of the invention

[0003] The household appliance device and control method thereof provided in this application can realize complex cross-device scene linkage and improve scenario-based control capabilities.

[0004] In a first aspect, the present application provides a method for controlling household appliances, the method comprising: obtaining user instructions; using a large language model to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions, and a task corresponding to each scene; wherein each scene corresponds to at least one household appliance, and each scene corresponds to a scene agent; using the scene agent to assign a corresponding device agent to each task, so that the device agent controls the corresponding household appliance; wherein each device agent is deployed in the corresponding household appliance.

[0005] Among them, a scene agent is used to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliances, including: based on each task, at least two scene agents perform scene collaboration to obtain an execution plan corresponding to each device agent; and the execution plan is assigned to the corresponding device agent so that the device agent controls the corresponding home appliances.

[0006] Among them, at least two scene agents include a first scene agent and a second scene agent. Based on each task, at least two scene agents perform scene collaboration, including: in response to a conflict between the execution plans formulated between the first scene agent and the second scene agent, coordinating the first scene agent and the second scene agent to re-formulate the execution plan; and / or, in response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, integrating the first execution plan and the second execution plan to obtain a third execution plan of the device agent corresponding to the target home appliance; and / or, in response to the first scene agent formulating a fourth execution plan, using the second scene agent to adjust the fourth execution plan.

[0007] Among them, the scenes include at least one of a living scene, a work and study scene, an entertainment scene, a sleeping scene, a cooking scene, a health scene, a water use scene and an environmental control scene.

[0008] Among them, a large language model is used to perform scene recognition and task decomposition on user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene, including: using a large language model to perform scene recognition and task decomposition on user instructions to obtain work and study scenes, living scenes and health scenes corresponding to the user instructions; assigning a first task to the work and study scene, a second task to the living scene, and a third task to the health scene.

[0009] Among them, the scene agent includes a first scene agent, a second scene agent and a third scene agent, and the scene agents are used to assign corresponding device agents to each task, including: using the first scene agent to formulate a first execution plan according to the first task, and determining the corresponding device agent based on the first execution plan; the first execution plan includes lighting control and / or audio control; using the second scene agent to formulate a second execution plan according to the second task, and determining the corresponding device agent based on the second execution plan; the second execution plan includes air conditioning control and / or curtain control; using the third scene agent to formulate a third execution plan according to the third task, and determining the corresponding device agent based on the third execution plan; the third execution plan includes exercise reminders and / or air purification control.

[0010] Among them, a large language model is used to perform scene recognition and task decomposition on user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene, including: using a large language model to perform scene recognition and task decomposition on user instructions to obtain cooking scenes, living scenes and health scenes corresponding to user instructions; assigning a fourth task to the cooking scene, a fifth task to the living scene, and a sixth task to the health scene.

[0011] Among them, the scene intelligent agent includes a fourth scene intelligent agent, a fifth scene intelligent agent and a sixth scene intelligent agent, and the scene intelligent agent is used to assign a corresponding equipment intelligent agent to each task, including: using the fourth scene intelligent agent to formulate a fourth execution plan according to the fourth task, and determining the corresponding equipment intelligent agent based on the fourth execution plan; the fourth execution plan includes ingredient identification control, menu recommendation and / or oven control; using the fifth scene intelligent agent to formulate a fifth execution plan according to the fifth task, and determining the corresponding equipment intelligent agent based on the fifth execution plan; the fifth execution plan includes lighting control and / or air conditioning control; using the sixth scene intelligent agent to formulate a sixth execution plan according to the sixth task, and determining the corresponding equipment intelligent agent based on the sixth execution plan; the sixth execution plan includes data analysis control and / or dietary recommendations.

[0012] Among them, a large language model is used to perform scene recognition and task decomposition on user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene, including: using a large language model to perform scene recognition and task decomposition on user instructions to obtain water use scenes, environmental control scenes and health scenes corresponding to user instructions; assigning the seventh task to the water use scene, the eighth task to the environmental control scene and the ninth task to the health scene.

[0013] Among them, the scene agents include the seventh scene agent, the eighth scene agent and the ninth scene agent, and the scene agents are used to assign corresponding equipment agents to each task, including: using the seventh scene agent to formulate the seventh execution plan according to the seventh task, and determining the corresponding equipment agent according to the seventh execution plan; the seventh execution plan includes water temperature adjustment, water quality adjustment and / or water pressure adjustment; using the eighth scene agent to formulate the eighth execution plan according to the eighth task, and determining the corresponding equipment agent according to the eighth execution plan; the eighth execution plan includes lighting control and / or air conditioning control; using the ninth scene agent to formulate the ninth execution plan according to the ninth task, and determining the corresponding equipment agent according to the ninth execution plan; the ninth execution plan includes data analysis control and / or dietary recommendations.

[0014] Among them, after the scene ends, the scene agent is used to assign the scene end task to the device agent, so that the device agent controls the corresponding home appliances to achieve the scene end task.

[0015] In a second aspect, the present application provides a control system for a household appliance, the control system comprising: a communication interface for communicating with the household appliance; and a processor coupled to the communication interface for implementing the method provided in the first aspect.

[0016] In a third aspect, the present application provides a method for controlling a household appliance, the method comprising: receiving a control instruction; wherein the control instruction is obtained by the method provided in the first aspect; and operating according to the control instruction.

[0017] In a fourth aspect, the present application provides a household appliance, which includes a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method provided in the first aspect.

[0018] The beneficial effects of the present application are as follows: Different from the prior art, the household appliance device and its control method and system provided by the present application obtain user instructions; use a large language model to perform scene recognition and task decomposition on the user instructions, and obtain at least one scene corresponding to the user instructions, and the task corresponding to each scene; wherein each scene corresponds to at least one household appliance device, and each scene corresponds to a scene agent; use the scene agent to assign a corresponding device agent to each task, so that the device agent controls the corresponding household appliance device; wherein each device agent is deployed in the corresponding household appliance device, and the scene management of large-scale household appliances is realized by setting up a scene agent, which can realize complex cross-device scene linkage and enhance the scene control capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0020] Figure 1 It is a structural schematic diagram of an embodiment of a control system for household appliances provided by the present application;

[0021] Figure 2 It is a flowchart of an embodiment of a multimodal data processing method provided by the present application;

[0022] Figure 3 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0023] Figure 4 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0024] Figure 5 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0025] Figure 6 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0026] Figure 7 This application provides Figure 6 A flow chart of an embodiment of step 62 in FIG.

[0027] Figure 8 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0028] Fig. 9It is a flowchart of an embodiment of a control method for household electrical appliances provided by the present application;

[0029] Fig.10 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0030] Fig.11 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0031] Fig.12 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0032] Fig.13 It is a flowchart of an embodiment of a method for controlling a household appliance provided by the present application;

[0033] Fig.14 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0034] Fig.15 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0035] Fig.16 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0036] Fig.17 is a flowchart of another embodiment of the household appliance control method provided by the present application;

[0037] Fig.18 is a structural schematic diagram of another embodiment of a control system for a household appliance provided by the present application;

[0038] Fig.19 It is a structural schematic diagram of an embodiment of a household appliance provided by the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] See also Figure 1 , Figure 1 1 is a schematic diagram of a control system for a household appliance provided by the present application. The control system 1000 includes: a communication interface 100 and a processor 200. The communication interface 100 is used to communicate with the household appliance. The processor 200 is coupled to the communication interface 100 and is used to implement a corresponding method according to relevant data of the household appliance. For specific implementation methods, please refer to any of the following embodiments.

[0042] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a multimodal data processing method provided by the present application. The method comprises:

[0043] Step 21: Obtain multimodal data.

[0044] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0045] In some embodiments, text data may be user input commands and / or historical interaction records, etc. Image data may be environmental images captured by cameras on home appliances, user gestures, etc. Voice data may be user voice commands and / or environmental sounds, etc. Sensor data may be various types of sensor data such as temperature, humidity, light, and energy consumption.

[0046] In some embodiments, the multimodal data includes textual data and sensor data.

[0047] In some embodiments, the multimodal data includes image data and sensor data.

[0048] In some embodiments, the multimodal data includes voice data and sensor data.

[0049] In some embodiments, the multimodal data includes text data, voice data, and sensor data.

[0050] In some embodiments, the multimodal data includes text data, image data, and sensor data.

[0051] In some embodiments, the multimodal data includes voice data, image data, and sensor data.

[0052] In some embodiments, the multimodal data includes text data, image data, voice data, and sensor data.

[0053] In some embodiments, the sensor data may be temperature data, humidity data, or light data. That is, corresponding data may be collected using corresponding sensors. For example, temperature data may be collected using a temperature sensor, humidity data may be collected using a humidity sensor, and light data may be collected using a light sensor.

[0054] Step 22: Use at least two fusion methods to fuse the multimodal data to obtain multimodal fused data with a unified semantic space.

[0055] In some embodiments, a fusion method can be selected according to the specific type of multimodal data, and then the corresponding fusion method can be used to fuse the multimodal data to obtain multimodal fusion data with a unified semantic space. That is, by fusing the multimodal data, the multimodal data can be aligned in the semantic space, so that the correlation between the multimodal data is enhanced, which is more conducive to the subsequent recognition of the large language model and improves the reasoning accuracy of the large language model.

[0056] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0057] See also Figure 3 , Figure 3 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0058] Step 31: Acquire multimodal data; wherein the multimodal data at least includes text data and sensor data.

[0059] The sensor data is collected by the sensors corresponding to the home appliances.

[0060] Step 32: pre-process the sensor data in the multimodal data to obtain a first vector representation; and pre-process the text data in the multimodal data to obtain a text vector.

[0061] In some embodiments, the text data may be segmented and stop words may be removed, and the text may be converted into a vector using a pre-trained word embedding model to obtain a text vector.

[0062] In some embodiments, the text data may be segmented and stop words may be removed, and the text may be converted into a vector using a pre-trained word embedding model to obtain a text vector.

[0063] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and the sensor data may be converted into a low-dimensional vector representation using a dimensionality reduction technique such as an autoencoder or PCA (Principal Component Analysis) to obtain a first vector representation.

[0064] Step 33: Using at least two fusion methods, perform data fusion on the text vector and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0065] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0066] See also Figure 4 , Figure 4 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0067] Step 41: Acquire multimodal data; wherein the multimodal data includes image data and sensor data.

[0068] The sensor data is collected by the sensors corresponding to the home appliances.

[0069] Step 42: pre-process the sensor data in the multimodal data to obtain a first vector representation; pre-process the image data in the multimodal data to obtain image features.

[0070] In some embodiments, image data may be subjected to operations such as image scaling, cropping, and data enhancement, and then a pre-trained convolutional neural network may be used to extract image features.

[0071] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0072] Step 43: Using at least two fusion methods, perform data fusion on the image features and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0073] In this embodiment, at least two fusion methods are used to fuse multimodal data including image data and sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0074] See also Figure 5 , Figure 5 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0075] Step 51: Acquire multimodal data; wherein the multimodal data at least includes voice data and sensor data.

[0076] The sensor data is collected by the sensors corresponding to the home appliances.

[0077] Step 52: pre-process the sensor data in the multimodal data to obtain a first vector representation; pre-process the voice data in the multimodal data to obtain a second vector representation.

[0078] In some embodiments, the speech signal may be preprocessed to extract MFCC features, and the speech may be converted into text using a pre-trained speech recognition model, and then converted into a vector representation, that is, finally obtaining the above-mentioned second vector representation.

[0079] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0080] Step 53: Using at least two fusion methods, perform data fusion on the second vector representation and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0081] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least voice data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0082] See also Figure 6 , Figure 6 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0083] Step 61: Acquire multimodal data; wherein the multimodal data at least includes text data, image data, voice data and sensor data.

[0084] Step 62: Use at least two fusion methods to perform data fusion on the multimodal data to obtain multimodal fusion data with a unified semantic space.

[0085] Among them, at least two fusion methods include: attention mechanism fusion method; weighted fusion method; splicing fusion method.

[0086] The attention mechanism is a resource allocation scheme that aims to allocate limited computing resources to more important tasks. It originates from the study of human vision, which shows that humans selectively focus on some information and ignore other information when processing information. There are two main types of attention mechanisms: focused attention and saliency-based attention.

[0087] How the attention mechanism works: In deep learning, the attention mechanism enables the model to selectively focus on different parts of the input sequence by dynamically generating weights for different connections. For example, in the Transformer model, the degree of attention to each input vector is dynamically determined by calculating the degree of association between each input vector and other vectors. This mechanism can be applied to fields such as natural language processing and computer vision to help the model better capture key information.

[0088] That is, the attention mechanism can quickly capture the key information in different modal data in multimodal data, and then fuse the key information.

[0089] The weighted fusion method can set weights for data of different modalities and perform weighted fusion of modal data according to the corresponding weights. For example, the data of different modalities are processed in a unified dimension, and after the dimension is unified, the modal data is weighted fused according to the corresponding weights.

[0090] The splicing and fusion method can splice data of different modalities. In some embodiments, the priority of different modal data can be set and spliced ​​according to the priority. For example, data with high priority is spliced ​​in the front, and data with low priority is spliced ​​in the back. That is, sort from high to low according to priority, and then splice. In this way, when extracting subsequent data, high-priority data can be quickly extracted, thereby improving the accuracy of the high-priority data, making it easier for the subsequent large language model to more accurately obtain the features of the modal data and understand its semantic information.

[0091] In some embodiments, different modal data can be initially fused by selecting corresponding fusion methods, and then the initially fused data can be fused again to obtain multi-modal fusion data.

[0092] In some embodiments, see Figure 7 , step 62 may be the following process:

[0093] Step 621: Use the attention mechanism fusion method to fuse the multimodal data to obtain the first fused data.

[0094] In some embodiments, the attention mechanism is used to extract the value vector, key vector, and query vector from the multimodal data. The key vector and the query vector are then used to calculate the correlation between the key vector and the query vector, and the attention score is obtained by the softmax function. Then, a weighted sum is performed based on the attention score to obtain a value vector with an attention score. That is, the first fused data can reflect the degree of matching between different modal data, so that data alignment can be achieved.

[0095] In some embodiments, the attention mechanism may include a self-attention mechanism, a cross-attention mechanism, and / or a multi-head attention mechanism.

[0096] Step 622: Perform data fusion on the multimodal data using a weighted fusion method to obtain second fused data.

[0097] Step 623: Use a splicing fusion method to fuse the multimodal data to obtain third fused data.

[0098] In some embodiments, the weighted fusion method and the splicing fusion method may use residual connection and layer normalization technology.

[0099] Residual connection refers to the process of passing the input of the previous layer directly to the next layer in a deep neural network, that is, adding cross-layer connections. This method can alleviate the problems of gradient disappearance and gradient explosion, making the effective depth of the deep neural network deeper, thereby improving the performance of the network.

[0100] Layer Normalization is a normalization technique that is more suitable for RNN and long-term series than batch normalization. The basic idea is to normalize all neurons of each sample in the neural network, that is, to normalize the mean and variance of the neurons with respect to the sample, thereby reducing the dependency between neurons and avoiding model instability caused by changes in batch size. It can effectively eliminate the correlation between each layer and improve the convergence and performance of the network. Compared with batch normalization, layer normalization is more stable and more robust to noise and small batches.

[0101] Step 624: Fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0102] In some embodiments, a fusion representation generation technology may be used to fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0103] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data, image data, voice data and sensor data, and obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0104] See also Figure 8 , Figure 8 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0105] Step 81: Acquire multimodal data.

[0106] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0107] Step 82: Use at least two fusion methods to perform data fusion on the multimodal data to obtain multimodal fusion data with a unified semantic space.

[0108] In some embodiments, steps 81 to 82 have the same or similar technical solutions as any embodiment of the present application, and are not repeated here.

[0109] Step 83: Input the multimodal fusion data into the large language model to obtain the contrast loss and the multi-task learning loss.

[0110] In some embodiments, the large language model can use a pre-trained GPT model as the core and fine-tune it for the home appliance field. The main features include:

[0111] 1. Using the Transformer architecture, it has powerful sequence modeling capabilities.

[0112] 2. Conduct pre-training on large-scale household appliance corpus to master domain knowledge.

[0113] 3. Support multi-modal input and can process fused feature representation.

[0114] 4. Use autoregressive generation to achieve flexible text generation and conversation capabilities.

[0115] In some embodiments, multi-task learning includes at least two of anomaly detection tasks, energy consumption optimization tasks, behavior prediction tasks, state estimation tasks, and instruction understanding tasks.

[0116] The anomaly detection task mainly relies on multimodal fusion data to detect anomalies in the working status of home appliances. For example, sensor data and status data of home appliances can be extracted from multimodal fusion data to perform anomaly detection.

[0117] The main purpose of the energy consumption optimization task is to control household appliances to achieve corresponding tasks with the lowest energy consumption, so as to optimize the energy consumption of household appliances.

[0118] The main purpose of the behavior prediction task is to predict user behavior through voice data and text data in multimodal fusion data, so as to control the corresponding home appliances according to the predicted behavior.

[0119] The state estimation task mainly estimates the working state of home appliances based on multimodal fusion data. For example, the state of home appliances can be estimated by extracting sensor data from multimodal fusion data, and then the corresponding estimation results can be given.

[0120] The main purpose of the command understanding task is to understand the commands through the voice data and text data in the multimodal fusion data, so as to control the corresponding home appliances according to the understood commands.

[0121] Step 84: Perform joint optimization based on contrast loss and multi-task learning loss, and adjust the weights of relevant data in multimodal data.

[0122] In some embodiments, the contrast loss and the multi-task learning loss may be weightedly summed to obtain a final total loss, and the total loss may be used to adjust the weights of relevant data in the multimodal data.

[0123] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0124] See also Fig. 9 , Fig. 9 This is a flow chart of an embodiment of a control method for household appliances provided by the present application. The method includes:

[0125] Step 91: Acquire multimodal fusion data.

[0126] The multimodal fusion data is obtained by using the method provided in any embodiment of the present application.

[0127] Step 92: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0128] Among them, the large language model is a large language model that has been trained in advance.

[0129] Step 93: Control the corresponding home appliances to work according to the multi-task decision.

[0130] In some embodiments, a corresponding intelligent agent is provided in the household appliance, and the intelligent agent can generate corresponding control instructions according to the corresponding task decision, thereby controlling the operation of the corresponding household appliance.

[0131] In some embodiments, the agent has the ability to make autonomous decisions. For example, if the multi-task decision determines that the home appliance needs to be turned on, the agent can independently determine the turning-on mode of the home appliance in combination with the current environmental data. For example, if the home appliance is an air conditioner, when the air conditioner needs to be turned on in the multi-task decision, a command to turn on the air conditioner is sent to the agent. The agent determines the turning-on temperature in combination with the current environmental data, and after determining the turning-on temperature, controls the air conditioner to start and work according to the turning-on temperature.

[0132] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0133] See also Fig.10 , Fig.10 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0134] Step 101: Acquire multimodal fusion data.

[0135] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0136] Step 102: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0137] Step 103: Perform safety check on the multi-task decision.

[0138] In some embodiments, the safety check mainly checks whether the home appliance control solution corresponding to the multi-task decision is safe during execution, such as whether there is an abnormal execution risk.

[0139] Step 104: After the safety check is passed, the corresponding home appliance is controlled to work according to the multi-task decision; or, when the safety check is not passed, the multi-task decision is corrected.

[0140] After the safety check is passed, the corresponding home appliances are controlled to work according to the multi-task decision. That is, after the safety check is passed, the task decision is sent to the corresponding home appliance. The home appliance is equipped with a corresponding intelligent agent, which can generate corresponding control instructions according to the corresponding task decision, and then control the corresponding home appliance.

[0141] When the safety check fails, the multi-task decision is revised so that the revised multi-task decision can successfully pass the safety check. After the safety check passes, the corresponding home appliances are controlled to work according to the revised multi-task decision.

[0142] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0143] Furthermore, by performing a safety check on the multi-task decision, the task decision is sent to the corresponding home appliance after the check is passed, or when the safety check fails, the multi-task decision is corrected to reduce the occurrence of sending erroneous task decisions to home appliances, thereby improving the control accuracy of home appliances.

[0144] See also Fig.11 , Fig.11 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0145] Step 111: Acquire multimodal fusion data.

[0146] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0147] Step 112: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0148] Step 113: Control the corresponding home appliances to work according to the multi-task decision.

[0149] In some embodiments, steps 111 to 113 have the same or similar technical solutions as any embodiment of the present application, and are not described in detail here.

[0150] Step 114: Record the working results of the home appliance and collect user feedback information.

[0151] Step 115: Update the large language model based on the work results and user feedback information.

[0152] In this embodiment, updating the large language model in combination with work results and user feedback information can improve the accuracy of the large language model in task reasoning, and make the reasoning of the large language model more in line with user preferences, thereby improving user experience.

[0153] See also Fig.12 , Fig.12 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0154] Step 121: Acquire multimodal fusion data.

[0155] Among them, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0156] Step 122: Input the multimodal fusion data into the large language model so that the large language model determines at least two task types.

[0157] In some embodiments, the large language model is trained for multiple tasks during the training process, so the large language model can perform multi-task reasoning. When the multimodal fusion data indicates that multiple tasks need to be implemented, the large language model can obtain at least two task types based on the multimodal fusion data.

[0158] Step 123: Outputting a task decision corresponding to each task type according to at least two task types to control the operation of the corresponding household electrical appliance.

[0159] Step 124: Control the corresponding home appliance to work according to the multi-task decision.

[0160] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0161] Furthermore, the large language model can process more complex multimodal fusion data, determine at least two task types from the multimodal fusion data, and then generate corresponding task decisions according to the task types to control the operation of home appliances.

[0162] In the above embodiments, effective fusion of multimodal data in the field of home appliances is achieved, breaking down data silos; precise alignment of data of different modalities is achieved in a unified semantic space, improving the model's understanding ability; significantly improving the generalization ability and robustness of the model in home appliance scenarios; achieving a more natural and intelligent human-computer interaction experience, improving user satisfaction; improving the accuracy and efficiency of intelligent control of home appliances, and reducing misoperation and energy waste.

[0163] With the rapid development of artificial intelligence and Internet of Things technologies, smart home systems are developing in a more intelligent, personalized and adaptive direction. However, related smart home systems are difficult to achieve complex cross-device scene linkage.

[0164] Based on this, the present application proposes to obtain user instructions; use a large language model to perform scene recognition and task decomposition on user instructions, obtain at least one scene corresponding to the user instructions, and the task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; use the scene agent to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and the scene agent is set to achieve scene management of large-scale home appliances, which can realize complex cross-device scene linkage and improve scene control capabilities. Please refer to any of the following embodiments for details.

[0165] See also Fig.13 , Fig.13 This is a flow chart of an embodiment of a method for controlling a household appliance provided by the present application. The method includes:

[0166] Step 131: Obtain user instructions.

[0167] In some embodiments, the user instruction may be a voice instruction and / or a text instruction.

[0168] In some embodiments, the user instruction may express the user demand, and the user demand may require the cooperation of at least one household appliance in the scene.

[0169] Step 132: Use the large language model to perform scene recognition and task decomposition on the user instruction to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene.

[0170] Among them, each scene corresponds to at least one household appliance, and each scene corresponds to a scene agent.

[0171] In some embodiments, the large language model can learn scene recognition functions in combination with historical data. The trained large language model can then be used to perform scene recognition and task decomposition on user instructions to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene.

[0172] In some embodiments, home appliances in multiple scenarios are usually required to work together to meet user needs.

[0173] In some embodiments, a large language model is used to perform scene recognition and task decomposition on user instructions to obtain at least two scenes corresponding to the user instructions and tasks corresponding to each scene; wherein each scene corresponds to at least two home appliances, and each scene corresponds to a scene agent. That is, one scene agent can manage at least two home appliances.

[0174] In some embodiments, the scenarios involved in the present application can be set in advance by the user, or obtained through self-learning of a large language model.

[0175] Step 133: Use the scene agent to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance.

[0176] Each device agent is deployed in a corresponding home appliance.

[0177] In this embodiment, a corresponding scene agent can be set for several household appliances corresponding to each scene, and the scene agent can be used as a control transfer station. The scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding household appliance.

[0178] In this embodiment, user instructions are obtained; a large language model is used to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; the scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and scene management of large-scale home appliances is achieved by setting up scene agents, which can realize complex cross-device scene linkage and enhance scene control capabilities.

[0179] See also Fig.14 , Fig.14 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0180] Step 141: Obtain user instructions.

[0181] Step 142: Use the large language model to perform scenario recognition and task decomposition on the user instruction to obtain at least one scenario corresponding to the user instruction and a task corresponding to each scenario.

[0182] Among them, each scene corresponds to at least one household appliance, and each scene corresponds to a scene agent.

[0183] Step 143: Based on each task, at least two scene agents perform scene collaboration to obtain an execution plan corresponding to each device agent.

[0184] In some embodiments, due to the control logic of multiple home appliances in a task, there may be a situation where multi-scenario collaboration is required. Based on this, at least two scenario agents can collaborate with each other to obtain an execution plan corresponding to each device agent.

[0185] In some embodiments, in response to a conflict between the execution plans formulated between the first scene agent and the second scene agent, the first scene agent and the second scene agent are coordinated to reformulate the execution plan. For example, the water temperature recommended by the health scene agent is different from the user's default preference. The system will weigh the pros and cons and may choose a slightly lower temperature and explain the reason to the user. For example, the execution plan of the first scene agent chooses to turn on the speakers to the maximum volume. The execution plan of the second scene agent chooses to turn on the TV to silent mode. At this time, there is a conflict between the first scene agent and the second scene agent, so the first scene agent and the second scene agent are coordinated to reformulate the execution plan. In some embodiments, users can coordinate by voice playback.

[0186] In some embodiments, in response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, the first execution plan and the second execution plan are combined to obtain a third execution plan of the device agent corresponding to the target home appliance. For example, the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent are both for lighting equipment, so the first execution plan and the second execution plan can be combined according to the scene type to obtain the third execution plan of the device agent corresponding to the target home appliance, and the best lighting plan for the lighting equipment can be obtained.

[0187] In some embodiments, in response to the first scene agent developing the fourth execution plan, the second scene agent is used to adjust the fourth execution plan. For example, the cooking scene agent develops a recommended menu, and the health scene agent can analyze the user's recent dietary records and health data, adjust the recommended menu based on the analysis results, and give dietary suggestions.

[0188] In some embodiments, in response to a conflict between the execution plans formulated by the first scene agent and the second scene agent, the first scene agent and the second scene agent are coordinated to re-formulate the execution plan. In response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, the first execution plan and the second execution plan are combined to obtain a third execution plan of the device agent corresponding to the target home appliance.

[0189] In some embodiments, in response to a conflict between the execution plans formulated by the first scene agent and the second scene agent, the first scene agent and the second scene agent are coordinated to re-formulate the execution plan. And in response to the first scene agent formulating a fourth execution plan, the second scene agent is used to adjust the fourth execution plan.

[0190] In some embodiments, in response to a conflict between the execution plans formulated by the first scene agent and the second scene agent, the first scene agent and the second scene agent are coordinated to re-formulate the execution plan. In response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, the first execution plan and the second execution plan are combined to obtain a third execution plan of the device agent corresponding to the target home appliance. In response to the first scene agent formulating a fourth execution plan, the second scene agent is used to adjust the fourth execution plan.

[0191] In some embodiments, in response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, the first execution plan and the second execution plan are combined to obtain a third execution plan of the device agent corresponding to the target home appliance. In response to the first scene agent formulating a fourth execution plan, the second scene agent is used to adjust the fourth execution plan.

[0192] Step 144: Allocate the execution plan to the corresponding device agent so that the device agent controls the corresponding home appliance.

[0193] Each device agent is deployed in a corresponding home appliance.

[0194] In some embodiments, the scene includes at least one of a living scene, a work and study scene, an entertainment scene, a sleeping scene, a cooking scene, a health scene, a water use scene, and an environmental control scene.

[0195] In this embodiment, user instructions are obtained; a large language model is used to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; the scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and scene management of large-scale home appliances is achieved by setting up scene agents, which can realize complex cross-device scene linkage and enhance scene control capabilities.

[0196] Furthermore, based on each task, at least two scene agents collaborate to obtain an execution plan corresponding to each device agent, which can realize collaborative control of smart homes based on diversified life scenarios, greatly improving the intelligence level of the system.

[0197] See also Fig.15 , Fig.15 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0198] Step 151: Obtain user instructions.

[0199] In some embodiments, the user instruction may be a voice instruction and / or a text instruction.

[0200] In some embodiments, the user demand may be expressed in the user instruction.

[0201] Step 152: Use the large language model to perform scene recognition and task decomposition on the user instructions to obtain the work and study scenes, living scenes, and health scenes corresponding to the user instructions.

[0202] Step 153: Allocate a first task to the work and study scenario, a second task to the living scenario, and a third task to the health scenario.

[0203] In some embodiments, the large language model can assign a first task to a work and study scenario, a second task to a living scenario, and a third task to a health scenario. That is, different scenarios have their own corresponding tasks. The specific content of these tasks can be to control corresponding home appliances or devices corresponding to users.

[0204] Step 154: Use the scene agent to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance.

[0205] Each device agent is deployed in a corresponding home appliance.

[0206] In this embodiment, each scene corresponds to a scene agent.

[0207] In some embodiments, a first scene agent is used to formulate a first execution plan according to a first task, and a corresponding device agent is determined according to the first execution plan. If the first task requires control of light and / or sound, the first execution plan includes light control and / or sound control. That is, the specific execution plan needs to be determined according to the task content.

[0208] The second scene agent can be used to formulate a second execution plan according to the second task, and the corresponding device agent can be determined according to the second execution plan. If the second task requires the control of air conditioning and / or curtains, the second execution plan includes air conditioning control and / or curtain control.

[0209] The third scene agent can be used to formulate a third execution plan according to the third task, and the corresponding device agent can be determined according to the third execution plan. If the third task requires air purification and / or exercise reminder, the third execution plan includes exercise reminder and / or air purification control.

[0210] After the corresponding device agent receives the corresponding execution plan, the device agent can automatically control the corresponding device to complete the execution plan in combination with the environmental data.

[0211] In this embodiment, user instructions are obtained; a large language model is used to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; the scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and scene management of large-scale home appliances is achieved by setting up scene agents, which can realize complex cross-device scene linkage and enhance scene control capabilities.

[0212] Furthermore, based on each task, at least two scene agents collaborate to obtain an execution plan corresponding to each device agent, which can realize collaborative control of smart homes based on diversified life scenarios, greatly improving the intelligence level of the system.

[0213] See also Fig.16 , Fig.16 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0214] Step 161: Obtain user instructions.

[0215] Step 162: Use the large language model to perform scene recognition and task decomposition on the user's instructions to obtain the cooking scene, living scene, and health scene corresponding to the user's instructions.

[0216] Step 163: Assign a fourth task to the cooking scene, a fifth task to the living scene, and a sixth task to the health scene.

[0217] Step 164: Use the scene agent to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance.

[0218] Each device agent is deployed in a corresponding home appliance.

[0219] In some embodiments, a fourth scenario agent is used to formulate a fourth execution plan according to a fourth task, and a corresponding device agent is determined according to the fourth execution plan. The fourth execution plan includes ingredient recognition control, menu recommendation and / or oven control.

[0220] The fifth scene agent can be used to formulate a fifth execution plan according to the fifth task, and the corresponding device agent can be determined according to the fifth execution plan. The fifth execution plan includes lighting control and / or air conditioning control.

[0221] The sixth scenario agent can be used to formulate a sixth execution plan according to the sixth task, and the corresponding device agent can be determined according to the sixth execution plan. The sixth execution plan includes data analysis control and / or dietary advice.

[0222] After the corresponding device agent receives the corresponding execution plan, the device agent can automatically control the corresponding device to complete the execution plan in combination with the environmental data.

[0223] In this embodiment, user instructions are obtained; a large language model is used to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; the scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and scene management of large-scale home appliances is achieved by setting up scene agents, which can realize complex cross-device scene linkage and enhance scene control capabilities.

[0224] Furthermore, based on each task, at least two scene agents collaborate to obtain an execution plan corresponding to each device agent, which can realize collaborative control of smart homes based on diversified life scenarios, greatly improving the intelligence level of the system.

[0225] See also Fig.17 , Fig.17 1 is a flow chart of another embodiment of the household appliance control method provided by the present application. The method comprises:

[0226] Step 171: Obtain user instructions.

[0227] Step 172: Use the large language model to perform scenario recognition and task decomposition on the user instructions to obtain the water use scenario, environmental control scenario, and health scenario corresponding to the user instructions.

[0228] Step 173: Assign the seventh task to the water use scenario, the eighth task to the environment control scenario, and the ninth task to the health scenario.

[0229] Step 174: Use the scene agent to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance.

[0230] Each device agent is deployed in a corresponding home appliance.

[0231] In some embodiments, the seventh scenario agent is used to formulate a seventh execution plan according to the seventh task, and the corresponding device agent is determined according to the seventh execution plan. The seventh execution plan includes water temperature adjustment, water quality adjustment and / or water pressure adjustment.

[0232] The eighth scene agent can be used to formulate an eighth execution plan according to the eighth task, and the corresponding device agent can be determined according to the eighth execution plan. The eighth execution plan includes lighting control and / or air conditioning control.

[0233] The ninth scenario agent can be used to formulate a ninth execution plan according to the ninth task, and the corresponding device agent can be determined according to the ninth execution plan. The ninth execution plan includes data analysis control and / or dietary advice.

[0234] After the corresponding device agent receives the corresponding execution plan, the device agent can automatically control the corresponding device to complete the execution plan in combination with the environmental data.

[0235] In this embodiment, user instructions are obtained; a large language model is used to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; the scene agent is used to assign a corresponding device agent to each task, so that the device agent controls the corresponding home appliance; wherein each device agent is deployed in the corresponding home appliance, and scene management of large-scale home appliances is achieved by setting up scene agents, which can realize complex cross-device scene linkage and enhance scene control capabilities.

[0236] Furthermore, based on each task, at least two scene agents collaborate to obtain an execution plan corresponding to each device agent, which can realize collaborative control of smart homes based on diversified life scenarios, greatly improving the intelligence level of the system.

[0237] After the scene ends, the scene agent is used to assign the scene ending task to the device agent, so that the device agent controls the corresponding home appliances to achieve the scene ending task.

[0238] See also Fig.18 The control system 1000 of the household appliance mainly includes the following modules: a scene intelligent agent layer 201, a scene collaboration layer 202 and a user interaction layer 203.

[0239] The scenario agent layer 201 includes multiple agents based on specific scenarios, each of which has a unique skill tree. For example:

[0240] Living scene agent:

[0241] Input: light sensor, temperature sensor, human presence sensor, voice command.

[0242] Skill tree: lighting control, temperature regulation, air quality management, furniture layout optimization.

[0243] Output: lighting control commands, air conditioning control commands, curtain control commands, air purifier control commands.

[0244] Work and study scenario agents:

[0245] Input: Noise sensor, light sensor, computer usage status, schedule data.

[0246] Skill tree: focus mode management, environmental noise control, work area lighting optimization, schedule management.

[0247] Output: smart speaker control commands, lighting control commands, schedule reminders, and device usage suggestions.

[0248] Entertainment scene agent:

[0249] Input: audio device status, video device status, human posture sensor, voice commands.

[0250] Skill tree: coordinated control of audio and video equipment, atmosphere lighting management, multimedia content recommendation, interactive game control.

[0251] Output: TV / projector control commands, audio system control commands, lighting control commands, content recommendations.

[0252] Sleep scene agent:

[0253] Input: biorhythm data, environmental noise sensor, light sensor, temperature and humidity sensor.

[0254] Skill tree: sleep environment optimization, biorhythm synchronization, noise elimination, comfort adjustment.

[0255] Output: air conditioning control command, curtain control command, humidifier control command, white noise playback control.

[0256] Cooking scene agent:

[0257] Input: cooking equipment status, food identification camera, temperature sensor, gas sensor.

[0258] Skill tree: recipe recommendation, cooking process optimization, kitchen safety monitoring, and food management.

[0259] Output: oven control commands, stove control commands, exhaust fan control commands, recipe display.

[0260] Healthy scene agent:

[0261] Input: wearable device data, air quality sensors, motion detection sensors, diet records.

[0262] Skill tree: health data analysis, exercise recommendation generation, diet plan formulation, environmental health optimization.

[0263] Output: health reports, exercise reminders, dietary recommendations, air purifier control commands.

[0264] The scene collaboration layer 202 is responsible for coordinating the collaboration between multiple scene agents to achieve cross-scene intelligent control.

[0265] The user interaction layer 203 provides a natural language interaction interface for users, supporting users to directly express scenario requirements.

[0266] In some embodiments, the workflow of the control system is as follows:

[0267] Users enter commands through natural language, such as "I want to work from home."

[0268] The user interaction layer passes instructions to the scene collaboration layer for scene recognition.

[0269] The scene collaboration layer identifies relevant scenes, decomposes tasks and assigns them to corresponding scene agents.

[0270] Each scene agent formulates an execution plan based on its own skill tree, equipment status and environmental information.

[0271] Scene agents exchange information and make collaborative decisions through the scene collaboration layer.

[0272] Each scene agent executes the finalized control command.

[0273] The execution results are fed back to the user interaction layer through the scene collaboration layer and finally presented to the user.

[0274] The control system records this interaction and control process for subsequent learning and optimization.

[0275] In some embodiments, the process of the smart home office scenario is as follows:

[0276] User says: "I want to work from home".

[0277] Scene recognition: The system recognizes that this involves work and study scenes, living scenes, and health scenes.

[0278] Task decomposition and allocation:

[0279] Work-study scenario agent: Preparing the office environment.

[0280] Living Scenario Agent: Adapts living spaces to work needs.

[0281] Healthy Scenario Agent: Monitor and optimize work health.

[0282] Each agent executes the plan:

[0283] Work and study scenario agents:

[0284] Check your calendar and be reminded of important meetings.

[0285] Adjust the brightness and color temperature of the lights in the work area (Output: Lighting Control Commands).

[0286] Turn on the noise cancellation function (output: smart speaker control command).

[0287] Living scene agent:

[0288] Adjust the indoor temperature to the optimum working temperature (output: air conditioning control command).

[0289] Adjust curtains according to light conditions (output: curtain control command).

[0290] Healthy scene agent:

[0291] Set a sedentary reminder (output: exercise reminder).

[0292] Adjust the air purifier to optimize the working environment (output: air purifier control command).

[0293] Scene collaboration:

[0294] The work-study scene agent collaborates with the living scene agent to jointly decide the best lighting plan based on the type of work and natural light conditions.

[0295] The health scenario agent coordinates with the work and study scenario agent to take care of physical health while ensuring work efficiency.

[0296] Results feedback:

[0297] The system informs the user via voice: "The home office environment is ready. You have a video conference at 3 pm today. I have set a reminder every hour for you to get up and move around. I wish you good work!".

[0298] In some embodiments, the process of the smart cooking scene is as follows:

[0299] The user said: "I want to make a healthy dinner".

[0300] Scene recognition: The system recognizes that this involves cooking scenes, health scenes, and living scenes.

[0301] Task decomposition and allocation:

[0302] Cooking scene agent: prepares the cooking environment and suggests menus.

[0303] Healthy Scenario Agent: Provides healthy eating recommendations.

[0304] Living scene agent: adjust the kitchen environment.

[0305] Each agent executes the plan:

[0306] Cooking scene agent:

[0307] Scan the ingredients in the refrigerator (input: ingredient recognition camera).

[0308] Menu recommendations based on available ingredients and health needs.

[0309] Preheat the oven (output: oven control command).

[0310] Healthy scene agent:

[0311] Analyze the user's recent diet records and health data (input: wearable device data, diet records).

[0312] Adjust menu recommendations based on analysis results to ensure nutritional balance (output: dietary recommendations).

[0313] Living scene agent:

[0314] Adjust kitchen lighting (output: light control commands).

[0315] Set the kitchen temperature (output: air conditioning control command).

[0316] Scene collaboration:

[0317] The cooking scene agent collaborates with the health scene agent to jointly develop a menu that is both delicious and healthy.

[0318] The cooking scene agent coordinates with the living scene agent to create the best cooking environment.

[0319] Results feedback:

[0320] The system informs the user by voice: "I have prepared a healthy dinner menu for you. Considering your recent increase in exercise, I have added some high-protein foods. The kitchen environment has been adjusted to the best cooking state. Do you want me to play a cooking instruction video for you?"

[0321] In some embodiments, the process of the smart bathing scene is as follows:

[0322] The user said: "I want to take a comfortable bath."

[0323] Scene recognition: Through speech recognition and semantic analysis, the system recognizes that the user's needs involve the "bath" scene. The system activates three main agents related to this scene: water scene agent, environmental control agent, and health scene agent.

[0324] Task decomposition and allocation: The system decomposes the overall task and allocates it to each agent:

[0325] Water use scene agent: responsible for water quality and temperature control.

[0326] Environmental control agent: responsible for optimizing the bathroom environment.

[0327] Health scenario agent: responsible for health-related recommendations.

[0328] Each agent executes the plan: a) Water use scenario agent:

[0329] Check the status of the water heater and adjust it to the appropriate temperature (38°C by default, adjustable according to user preference).

[0330] Activate the water softener to ensure soft water.

[0331] Set water pressure according to user preference.

[0332] b) Environment Control Agent:

[0333] Adjust the bathroom temperature to a comfortable range (default 26°C).

[0334] Set the appropriate humidity (default 60%).

[0335] Adjust the lighting to a soothing setting (warm tones, medium brightness).

[0336] Control ventilation equipment to ensure air circulation without causing chilliness.

[0337] c) Healthy scene agent:

[0338] Retrieve the user's recent health data (such as blood pressure, heart rate, etc.).

[0339] Recommend appropriate water and bathroom temperatures based on health data.

[0340] Set a bathing time reminder (20 minutes by default) to prevent the adverse effects of long bathing on health.

[0341] Scenario collaboration: Each agent submits its execution plan to the scenario collaboration layer. The collaboration layer comprehensively considers the suggestions of each agent and resolves possible conflicts, such as:

[0342] If the water temperature recommended by the health scenario agent is different from the user's default preference, the system will weigh the pros and cons and may choose a slightly lower temperature and explain the reason to the user.

[0343] The environmental control agent and the water usage scene agent coordinate to ensure a comfortable transition between bathroom temperature and water temperature.

[0344] Execute control commands: The collaboration layer formulates the final execution plan and sends control commands to each smart device:

[0345] Water heater: Set the water temperature to 37.5℃.

[0346] Water Softener: Activate.

[0347] Bathroom thermostat system: set the temperature to 25.5℃.

[0348] Bathroom humidifier: Set humidity to 58%.

[0349] Lighting system: Set to warm, soft yellow light, 70% brightness.

[0350] Ventilation system: Set to low speed.

[0351] Smart speaker: ready to play the user's favorite light music.

[0352] Result feedback: The system uses the voice assistant to provide feedback on the preparation status to the user: "Dear [user name], your bathing environment is ready! The water temperature has been adjusted to 37.5℃, which is slightly lower than your usual preference, considering your large amount of exercise today. The bathroom temperature is set to 25.5℃ and the humidity is 58%, creating a comfortable bathing environment. The lighting has been adjusted to soothing mode, and I have also prepared light music for you. It is recommended that you keep your bathing time within 20 minutes. I wish you a wonderful bathing time!"

[0353] Continuous monitoring and adjustment: During the user's bathing process, the system will continuously monitor various parameters:

[0354] If the water temperature fluctuates, the water use scene intelligent experience will adjust the water heater output in time.

[0355] If the humidity in the bathroom is too high, the environmental control agent will increase the ventilation.

[0356] The health scene intelligent experience will track bathing time and gently remind users through the bathroom speaker when it is close to 20 minutes.

[0357] Scenario End and Learning: After the user finishes taking a shower, the system will:

[0358] Turn off hot water supplies, lower bathroom temperatures, and increase ventilation.

[0359] Record the various parameters of this bath and user feedback (if any).

[0360] Use the collected data to optimize the algorithm so as to provide users with more accurate services next time.

[0361] In some embodiments, the process of the smart morning scene is as follows:

[0362] User settings: "Wake me up at 7 o'clock tomorrow morning, I need to be at the company before 8 o'clock."

[0363] Scene recognition and initial planning: The system recognizes that this is a complex scene involving multiple aspects such as sleeping, living, eating and traveling. The system activates the following agents:

[0364] Sleeping scene intelligent agent, living scene intelligent agent, eating scene intelligent agent, travel scene intelligent agent, and health scene intelligent agent.

[0365] Night preparation stage: a) Sleep scene agent:

[0366] Analyze the user's sleep cycle data.

[0367] According to the requirement of getting up at 7 o'clock, the best time to go to bed is calculated and the user is notified at the appropriate time.

[0368] b) Living scene agent:

[0369] Set up smart curtains to gradually open at 6:55am.

[0370] Adjust the bedroom temperature to the most suitable temperature for sleeping.

[0371] c) Travel scenario agent:

[0372] Check the weather forecast and traffic conditions to estimate travel times for the next morning.

[0373] Early morning wake-up phase (6:55-7:00): a) Sleeping scene agent:

[0374] Monitor the user's sleeping status.

[0375] The wake-up procedure is triggered at the optimal wake-up time (light sleep stage) between 6:55-7:00.

[0376] b) Living scene agent:

[0377] After receiving the signal from the sleeping scene agent, gradually increase the room brightness.

[0378] Play soothing music to wake you up.

[0379] c) Health scenario agent:

[0380] Get the user's sleep quality data and prepare a morning health report.

[0381] Morning activity phase (7:00-7:30): a) Living scene agent:

[0382] Adjust the room temperature to a comfortable level.

[0383] Turn on the bathroom lights and heater.

[0384] b) Eating scene agent:

[0385] Adjust breakfast recommendations based on sleep quality data provided by the health scenario agent.

[0386] Start the smart coffee machine and prepare coffee.

[0387] c) Travel scenario agent:

[0388] Adjust departure time suggestions based on real-time traffic conditions.

[0389] Share information with living scene agents and eating scene agents to coordinate schedules.

[0390] d) Healthy scene agent:

[0391] Generate a morning health report, including sleep quality, recommended water intake, etc.

[0392] Share information with eating scene agents and living scene agents.

[0393] Preparation phase (7:30-8:00): a) Living scene agent:

[0394] Adjust the time to remind users to prepare to go out based on the suggestions of the travel scenario agent.

[0395] Make sure all unnecessary electrical appliances are turned off.

[0396] b) Eating scene agent:

[0397] Adjust the pace of breakfast preparation according to travel time.

[0398] If time is of the essence, prepare portable breakfast options.

[0399] c) Travel scenario agent:

[0400] Continuously monitor road conditions and notify users in a timely manner if there are any changes.

[0401] Preheat the vehicle or book an online taxi (based on user habits).

[0402] Scenario coordination and execution: The scenario coordination layer integrates the inputs of each agent and coordinates possible conflicts:

[0403] If the sleep quality is poor, the health scenario agent may suggest delaying the wake-up time, but the travel scenario agent recommends leaving earlier based on the road conditions. The system will weigh the pros and cons and may choose to slightly delay the wake-up time and adjust the breakfast plan to a quick option.

[0404] Living scene agents and eating scene agents coordinate to ensure optimal arrangements for bathroom use and breakfast time.

[0405] User interaction and feedback: The system provides information and suggestions to users through voice assistants or mobile apps: "Good morning, [user name]. It is 7:00 now. You have a good sleep quality and slept for 7 hours and 20 minutes. The weather is fine today, but traffic congestion is expected during the morning rush hour. It is recommended to leave at 7:40 to ensure that you arrive at the company before 8 o'clock. Coffee is ready, and the recommended quick and healthy breakfast is whole wheat toast with eggs and avocado. The bathroom is preheated. I wish you a good day!".

[0406] Continuous optimization: The system records the user's actual behavior (such as actual wake-up time, breakfast choice, departure time, etc.) for optimization of future scenarios. Each agent uses this data to adjust its decision model to provide more accurate services.

[0407] In some embodiments, the present application also provides a method for controlling a household appliance, the method comprising: receiving a control instruction; wherein the control instruction is obtained by a multi-task decision generated by the method of any of the above embodiments; and working according to the control instruction.

[0408] See also Fig.19 , Fig.19 It is a structural diagram of an embodiment of a household appliance provided in the present application. The household appliance 300 includes a communication interface 301 and a processor 302. The processor 302 is coupled to the communication interface 301 to implement the method of any of the above embodiments.

[0409] In summary, the household appliance equipment and its control method and system provided by the present application obtain user instructions; use a large language model to perform scene recognition and task decomposition on the user instructions to obtain at least one scene corresponding to the user instructions, and the task corresponding to each scene; wherein each scene corresponds to at least one household appliance equipment, and each scene corresponds to a scene agent; use the scene agent to assign a corresponding device agent to each task, so that the device agent controls the corresponding household appliance equipment; wherein each device agent is deployed in the corresponding household appliance equipment, and the scene management of large-scale household appliances is realized by setting up a scene agent, which can realize complex cross-device scene linkage and improve the scene control capability.

[0410] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0411] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processing circuit component (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0412] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for controlling a household appliance, characterized in that: The method comprises: Get user instructions; Using a large language model to perform scene recognition and task decomposition on the user instruction, obtaining at least one scene corresponding to the user instruction and a task corresponding to each scene; wherein each scene corresponds to at least one home appliance, and each scene corresponds to a scene agent; The scene agent is used to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance; wherein each of the device agents is deployed in the corresponding home appliance.

2. The method according to claim 1, characterized in that The method of using the scene agent to assign a corresponding device agent to each task so that the device agent controls the corresponding home appliance includes: Based on each task, at least two of the scene agents perform scene collaboration to obtain an execution plan corresponding to each of the device agents; The execution plan is distributed to the corresponding device agent so that the device agent controls the corresponding home appliance.

3. The method according to claim 2, characterized in that The at least two scene agents include a first scene agent and a second scene agent, and the scene collaboration between the at least two scene agents based on each task includes: In response to a conflict between the execution plans formulated by the first scene agent and the second scene agent, coordinating the first scene agent and the second scene agent to reformulate the execution plan; and / or, In response to the first execution plan formulated by the first scene agent and the second execution plan formulated by the second scene agent corresponding to the same target home appliance, a third execution plan of the device agent corresponding to the target home appliance is obtained by combining the first execution plan and the second execution plan; and / or, In response to the first scene agent formulating a fourth execution plan, the second scene agent is used to adjust the fourth execution plan.

4. The method according to any one of claims 1 to 3, characterized in that: The scenes include at least one of a living scene, a work and study scene, an entertainment scene, a sleeping scene, a cooking scene, a health scene, a water use scene, and an environmental control scene.

5. The method according to claim 4, characterized in that The using of the large language model to perform scene recognition and task decomposition on the user instruction to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene includes: Using a large language model to perform scene recognition and task decomposition on the user instructions, to obtain the work and study scenes, living scenes, and health scenes corresponding to the user instructions; A first task is assigned to the work and study scenario, a second task is assigned to the living scenario, and a third task is assigned to the health scenario.

6. The method according to claim 5, characterized in that The scene agent includes a first scene agent, a second scene agent and a third scene agent, and the method of using the scene agents to assign a corresponding device agent to each task includes: Using the first scene agent to formulate a first execution plan according to the first task, and determining a corresponding device agent according to the first execution plan; the first execution plan includes lighting control and / or audio control; Using the second scene agent to formulate a second execution plan according to the second task, and determining a corresponding device agent according to the second execution plan; the second execution plan includes air conditioning control and / or curtain control; The third scene agent is used to formulate a third execution plan according to the third task, and the corresponding device agent is determined according to the third execution plan; the third execution plan includes exercise reminders and / or air purification control.

7. The method according to claim 4, characterized in that The using of the large language model to perform scene recognition and task decomposition on the user instruction to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene includes: Using a large language model to perform scene recognition and task decomposition on the user command, to obtain the cooking scene, living scene and health scene corresponding to the user command; A fourth task is assigned to the cooking scene, a fifth task is assigned to the living scene, and a sixth task is assigned to the health scene.

8. The method according to claim 7, characterized in that The scene agent includes a fourth scene agent, a fifth scene agent and a sixth scene agent, and the method of using the scene agents to assign a corresponding device agent to each task includes: Utilizing the fourth scene agent to formulate a fourth execution plan according to the fourth task, and determining a corresponding device agent according to the fourth execution plan; the fourth execution plan includes ingredient recognition control, menu recommendation and / or oven control; Utilizing the fifth scene agent to formulate a fifth execution plan according to the fifth task, and determining a corresponding device agent according to the fifth execution plan; the fifth execution plan includes lighting control and / or air conditioning control; The sixth scene agent is used to formulate a sixth execution plan according to the sixth task, and a corresponding device agent is determined based on the sixth execution plan; the sixth execution plan includes data analysis control and / or dietary recommendations.

9. The method according to claim 4, characterized in that The using of the large language model to perform scene recognition and task decomposition on the user instruction to obtain at least one scene corresponding to the user instruction and a task corresponding to each scene includes: Using a large language model to perform scene recognition and task decomposition on the user instructions, to obtain the water use scene, environmental control scene, and health scene corresponding to the user instructions; A seventh task is assigned to the water use scenario, an eighth task is assigned to the environmental control scenario, and a ninth task is assigned to the health scenario.

10. The method according to claim 9, characterized in that The scene agents include a seventh scene agent, an eighth scene agent, and a ninth scene agent, and the step of using the scene agents to assign a corresponding device agent to each task includes: Utilizing the seventh scenario agent to formulate a seventh execution plan according to the seventh task, and determining a corresponding device agent according to the seventh execution plan; the seventh execution plan includes water temperature regulation, water quality regulation and / or water pressure regulation; Using the eighth scene agent to formulate an eighth execution plan according to the eighth task, and determining a corresponding device agent according to the eighth execution plan; the eighth execution plan includes lighting control and / or air conditioning control; The ninth scene agent is used to formulate a ninth execution plan according to the ninth task, and a corresponding device agent is determined based on the ninth execution plan; the ninth execution plan includes data analysis control and / or dietary recommendations.

11. The method according to any one of claims 1 to 10, characterized in that: After the scene ends, the scene agent is used to assign a scene ending task to the device agent, so that the device agent controls the corresponding home appliance to achieve the scene ending task.

12. A method for controlling a household appliance, characterized in that: The method comprises: Receiving a control instruction; wherein the control instruction is obtained by any method of claims 1-11; Works according to the control instructions described.

13. A household appliance, characterized in that: The household appliance comprises a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method according to claim 12.

Images