Multi-modal data processing method, household appliance and control method and system thereof

Abstract

The invention discloses a multi-modal data processing method, a household appliance and a control method and system of the household appliance. The method comprises the following steps: acquiring multi-modal data; wherein the multi-modal data at least comprises at least one of text data, image data and voice data, and sensor data; the sensor data is acquired by a sensor corresponding to the household appliance; and carrying out data fusion on the multi-modal data by utilizing at least two fusion modes to obtain multi-modal fusion data with a unified semantic space. Through the above mode, effective fusion of multi-modal data in the field of household appliances can be realized, data islands can be broken, further, accurate alignment of different modal data in a unified semantic space can be realized, and the understanding ability of a large language model can be effectively improved after the multi-modal fusion data is input into the large language model.

Description

Technical Field

[0001] The present application relates to the technical field of household appliances, and in particular to a multimodal data processing method, household appliances and control methods and systems thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful capabilities in various fields. However, in the field of home appliances, language models that rely solely on a single modality often cannot meet complex interaction and control requirements. Summary of the invention

[0003] The multimodal data processing method, household appliance device and its control method and system provided in the present application can realize the effective fusion of multimodal data in the household appliance field, break the data silos, and further realize the precise alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, it can effectively improve the understanding ability of the large language model.

[0004] In a first aspect, the present application provides a multimodal data processing method, the method comprising: acquiring multimodal data; wherein the multimodal data includes at least one of text data, image data and voice data, and sensor data; the sensor data is collected by a sensor corresponding to a household appliance; and using at least two fusion methods to fuse the multimodal data to obtain multimodal fused data with a unified semantic space.

[0005] Among them, using at least two fusion methods to perform data fusion on multimodal data to obtain multimodal fusion data with a unified semantic space, including: preprocessing sensor data in the multimodal data to obtain a first vector representation; and preprocessing text data in the multimodal data to obtain a text vector; and / or, preprocessing image data in the multimodal data to obtain image features; and / or, preprocessing voice data in the multimodal data to obtain a second vector representation; using at least two fusion methods to perform data fusion on multimodal data to obtain multimodal fusion data with a unified semantic space, including: using at least two fusion methods, at least one of the text vector, image features and the second vector representation and the first vector representation are data fused to obtain multimodal fusion data with a unified semantic space.

[0006] Among them, at least two fusion methods include: attention mechanism fusion method; weighted fusion method; splicing fusion method.

[0007] Among them, at least two fusion methods are used to fuse the multimodal data to obtain multimodal fusion data with a unified semantic space, including: using an attention mechanism fusion method to fuse the multimodal data to obtain first fusion data; using a weighted fusion method to fuse the multimodal data to obtain second fusion data; using a splicing fusion method to fuse the multimodal data to obtain third fusion data; fusing the first fusion data, the second fusion data and the third fusion data to obtain multimodal fusion data.

[0008] Among them, after using at least two fusion methods to fuse multimodal data and obtain multimodal fusion data with a unified semantic space, it includes: inputting the multimodal fusion data into a large language model to obtain contrast loss and multi-task learning loss; wherein multi-task learning includes at least two of anomaly detection tasks, energy consumption optimization tasks, behavior prediction tasks, state estimation tasks and instruction understanding tasks; performing joint optimization based on contrast loss and multi-task learning loss, and adjusting the weights of related data in the multimodal data.

[0009] In a second aspect, the present application provides a method for controlling a household appliance, the method comprising: obtaining multimodal fusion data; wherein the multimodal fusion data is obtained using the method provided in the first aspect; inputting the multimodal fusion data into a large language model to obtain a multi-task decision output by the large language model; and controlling the operation of the corresponding household appliance based on the multi-task decision.

[0010] Among them, controlling the operation of corresponding household appliances according to the multi-task decision includes: performing a safety check on the multi-task decision; after the safety check is passed, controlling the operation of corresponding household appliances according to the multi-task decision; or, when the safety check fails, correcting the multi-task decision.

[0011] Among them, after the corresponding home appliance is working according to the multi-task decision control, it includes: recording the working results of the home appliance and collecting user feedback information; and updating the large language model in combination with the working results and user feedback information.

[0012] In a third aspect, the present application provides a control system for a household appliance, the control system comprising: a communication interface for communicating with the household appliance; and a processor coupled to the communication interface for implementing the method provided in the first aspect or the second aspect.

[0013] In a fourth aspect, the present application provides a method for controlling a household appliance, the method comprising: receiving a control instruction; wherein the control instruction is obtained by a multi-task decision generated by the method provided in the second aspect; and working according to the control instruction.

[0014] In a fifth aspect, the present application provides a household appliance, which includes a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method provided in the fourth aspect.

[0015] The beneficial effects of the present application are as follows: Different from the prior art, the multimodal data processing method, home appliance device and control method and system provided by the present application use at least two fusion methods to fuse multimodal data with sensor data, and obtain multimodal fusion data with a unified semantic space, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of intelligent control of home appliances, reducing misoperation and energy waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1 It is a structural schematic diagram of an embodiment of a control system for household appliances provided by the present application;

[0018] Figure 2 It is a flowchart of an embodiment of a multimodal data processing method provided by the present application;

[0019] Figure 3 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0020] Figure 4 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0021] Figure 5 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0022] Figure 6 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0023] Figure 7 This application provides Figure 6A schematic diagram of a flow chart of an embodiment of step 62;

[0024] Figure 8 It is a flowchart of another embodiment of the multimodal data processing method provided by the present application;

[0025] Fig. 9 It is a flowchart of an embodiment of a control method for household electrical appliances provided by the present application;

[0026] Fig.10 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0027] Fig.11 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0028] Fig.12 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0029] Fig.13 is a structural schematic diagram of another embodiment of a control system for a household appliance provided by the present application;

[0030] Fig.14 This is a schematic diagram of an application scenario of a control system for household appliances provided by this application;

[0031] Fig.15 This is a schematic diagram of another application scenario of the control system of the household appliance provided by the present application;

[0032] Fig.16 This is a schematic diagram of another application scenario of the control system of the household appliance provided by the present application;

[0033] Fig.17 This is a schematic diagram of another application scenario of the control system of the household appliance provided by the present application;

[0034] Fig.18 It is a structural schematic diagram of an embodiment of a household appliance provided by the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful capabilities in various fields. However, in the field of home appliances, language models that rely solely on a single modality often cannot meet complex interaction and control requirements.

[0038] Related home appliance intelligent control systems usually have the following problems:

[0039] 1. Data silos: Different types of data are often processed independently.

[0040] 2. Difficulty in modal alignment: Data from different modalities have different characteristics and expressions.

[0041] Based on this, the present application proposes to use at least two fusion methods to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of intelligent control of home appliances, reducing misoperation and energy waste. Please refer to any of the following embodiments for details.

[0042] See also Figure 1 , Figure 1 1 is a schematic diagram of a control system for a household appliance provided by the present application. The control system 1000 includes: a communication interface 100 and a processor 200. The communication interface 100 is used to communicate with the household appliance. The processor 200 is coupled to the communication interface 100 and is used to implement a corresponding method according to relevant data of the household appliance. For specific implementation methods, please refer to any of the following embodiments.

[0043] See also Figure 2 , Figure 2 1 is a flow chart of an embodiment of a multimodal data processing method provided by the present application. The method comprises:

[0044] Step 21: Obtain multimodal data.

[0045] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0046] In some embodiments, text data may be user input commands and / or historical interaction records, etc. Image data may be environmental images captured by cameras on home appliances, user gestures, etc. Voice data may be user voice commands and / or environmental sounds, etc. Sensor data may be various types of sensor data such as temperature, humidity, light, and energy consumption.

[0047] In some embodiments, the multimodal data includes textual data and sensor data.

[0048] In some embodiments, the multimodal data includes image data and sensor data.

[0049] In some embodiments, the multimodal data includes voice data and sensor data.

[0050] In some embodiments, the multimodal data includes text data, voice data, and sensor data.

[0051] In some embodiments, the multimodal data includes text data, image data, and sensor data.

[0052] In some embodiments, the multimodal data includes voice data, image data, and sensor data.

[0053] In some embodiments, the multimodal data includes text data, image data, voice data, and sensor data.

[0054] In some embodiments, the sensor data may be temperature data, humidity data, or light data. That is, corresponding data may be collected using corresponding sensors. For example, temperature data may be collected using a temperature sensor, humidity data may be collected using a humidity sensor, and light data may be collected using a light sensor.

[0055] Step 22: Use at least two fusion methods to fuse the multimodal data to obtain multimodal fused data with a unified semantic space.

[0056] In some embodiments, a fusion method can be selected according to the specific type of multimodal data, and then the corresponding fusion method can be used to fuse the multimodal data to obtain multimodal fusion data with a unified semantic space. That is, by fusing the multimodal data, the multimodal data can be aligned in the semantic space, so that the correlation between the multimodal data is enhanced, which is more conducive to the subsequent recognition of the large language model and improves the reasoning accuracy of the large language model.

[0057] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0058] See also Figure 3 , Figure 3 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0059] Step 31: Acquire multimodal data; wherein the multimodal data at least includes text data and sensor data.

[0060] The sensor data is collected by the sensors corresponding to the home appliances.

[0061] Step 32: pre-process the sensor data in the multimodal data to obtain a first vector representation; and pre-process the text data in the multimodal data to obtain a text vector.

[0062] In some embodiments, the text data may be segmented and stop words may be removed, and the text may be converted into a vector using a pre-trained word embedding model to obtain a text vector.

[0063] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0064] Step 33: Using at least two fusion methods, perform data fusion on the text vector and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0065] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0066] See also Figure 4 , Figure 4 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0067] Step 41: Acquire multimodal data; wherein the multimodal data includes image data and sensor data.

[0068] The sensor data is collected by the sensors corresponding to the home appliances.

[0069] Step 42: pre-process the sensor data in the multimodal data to obtain a first vector representation; pre-process the image data in the multimodal data to obtain image features.

[0070] In some embodiments, image data may be subjected to operations such as image scaling, cropping, and data enhancement, and then a pre-trained convolutional neural network may be used to extract image features.

[0071] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0072] Step 43: Using at least two fusion methods, perform data fusion on the image features and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0073] In this embodiment, at least two fusion methods are used to fuse multimodal data including image data and sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0074] See also Figure 5 , Figure 5 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0075] Step 51: Acquire multimodal data; wherein the multimodal data at least includes voice data and sensor data.

[0076] The sensor data is collected by the sensors corresponding to the home appliances.

[0077] Step 52: pre-process the sensor data in the multimodal data to obtain a first vector representation; pre-process the speech data in the multimodal data to obtain a second vector representation.

[0078] In some embodiments, the speech signal may be preprocessed to extract MFCC features, and the speech may be converted into text using a pre-trained speech recognition model, and then converted into a vector representation, that is, finally obtaining the above-mentioned second vector representation.

[0079] In some embodiments, data normalization and time series feature extraction may be performed on the sensor data, and a dimensionality reduction technique such as an autoencoder or PCA may be used to convert the sensor data into a low-dimensional vector representation to obtain a first vector representation.

[0080] Step 53: Using at least two fusion methods, perform data fusion on the second vector representation and the first vector representation to obtain multimodal fusion data with a unified semantic space.

[0081] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least voice data and sensor data, and multimodal fusion data with a unified semantic space is obtained, which can realize the effective fusion of multimodal data in the field of home appliances, break the data island, and further realize the precise alignment of different modal data in the unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of home appliance intelligent control, reducing misoperation and energy waste.

[0082] See also Figure 6 , Figure 6 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0083] Step 61: Acquire multimodal data; wherein the multimodal data at least includes text data, image data, voice data and sensor data.

[0084] Step 62: Use at least two fusion methods to perform data fusion on the multimodal data to obtain multimodal fusion data with a unified semantic space.

[0085] Among them, at least two fusion methods include: attention mechanism fusion method; weighted fusion method; splicing fusion method.

[0086] Attention mechanism is a resource allocation scheme that aims to allocate limited computing resources to more important tasks. It originates from the study of human vision, which shows that humans selectively focus on some information and ignore other information when processing information. There are two main types of attention mechanisms: focused attention and saliency-based attention.

[0087] How the attention mechanism works: In deep learning, the attention mechanism enables the model to selectively focus on different parts of the input sequence by dynamically generating weights for different connections. For example, in the Transformer model, the degree of attention to each input vector is dynamically determined by calculating the degree of association between each input vector and other vectors. This mechanism can be applied to fields such as natural language processing and computer vision to help the model better capture key information.

[0088] That is, the attention mechanism can quickly capture the key information in different modal data in multimodal data, and then fuse the key information.

[0089] The weighted fusion method can set weights for data of different modalities and perform weighted fusion of modal data according to the corresponding weights. For example, the data of different modalities are processed in a unified dimension, and after the dimension is unified, the modal data is weighted fused according to the corresponding weights.

[0090] The splicing and fusion method can splice data of different modalities. In some embodiments, the priority of different modal data can be set and spliced ​​according to the priority. For example, data with high priority is spliced ​​in the front, and data with low priority is spliced ​​in the back. That is, sort from high to low according to priority, and then splice. In this way, when extracting subsequent data, high-priority data can be quickly extracted, thereby improving the accuracy of the high-priority data, making it easier for the subsequent large language model to more accurately obtain the features of the modal data and understand its semantic information.

[0091] In some embodiments, different modal data can be initially fused by selecting corresponding fusion methods, and then the initially fused data can be fused again to obtain multi-modal fusion data.

[0092] In some embodiments, see Figure 7 , step 62 may be the following process:

[0093] Step 621: Use the attention mechanism fusion method to fuse the multimodal data to obtain the first fused data.

[0094] In some embodiments, the attention mechanism is used to extract the value vector, key vector, and query vector from the multimodal data. The key vector and the query vector are then used to calculate the correlation between the key vector and the query vector, and the attention score is obtained by the softmax function. Then, a weighted sum is performed based on the attention score to obtain a value vector with an attention score. That is, the first fused data can reflect the degree of matching between different modal data, so that data alignment can be achieved.

[0095] In some embodiments, the attention mechanism may include a self-attention mechanism, a cross-attention mechanism, and / or a multi-head attention mechanism.

[0096] Step 622: Perform data fusion on the multimodal data using a weighted fusion method to obtain second fused data.

[0097] Step 623: Use a splicing fusion method to fuse the multimodal data to obtain third fused data.

[0098] In some embodiments, the weighted fusion method and the splicing fusion method may use residual connection and layer normalization technology.

[0099] Residual connection refers to the process of passing the input of the previous layer directly to the next layer in a deep neural network, that is, adding cross-layer connections. This method can alleviate the problems of gradient disappearance and gradient explosion, making the effective depth of the deep neural network deeper, thereby improving the performance of the network.

[0100] Layer Normalization is a normalization technique that is more suitable for RNN and long-term series than batch normalization. The basic idea is to normalize all neurons of each sample in the neural network, that is, to normalize the mean and variance of the neurons with respect to the sample, thereby reducing the dependency between neurons and avoiding model instability caused by changes in batch size. It can effectively eliminate the correlation between each layer and improve the convergence and performance of the network. Compared with batch normalization, layer normalization is more stable and more robust to noise and small batches.

[0101] Step 624: Fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0102] In some embodiments, a fusion representation generation technology may be used to fuse the first fused data, the second fused data, and the third fused data to obtain multimodal fused data.

[0103] In this embodiment, at least two fusion methods are used to fuse multimodal data including at least text data, image data, voice data and sensor data, and obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0104] See also Figure 8 , Figure 8 : is a flowchart of another embodiment of the multimodal data processing method provided by the present application. The method includes:

[0105] Step 81: Acquire multimodal data.

[0106] Among them, the multimodal data includes at least one of text data, image data and voice data, as well as sensor data; the sensor data is collected by the sensor corresponding to the home appliance.

[0107] Step 82: Use at least two fusion methods to fuse the multimodal data to obtain multimodal fused data with a unified semantic space.

[0108] In some embodiments, steps 81 to 82 have the same or similar technical solutions as any embodiment of the present application, and are not repeated here.

[0109] Step 83: Input the multimodal fusion data into the large language model to obtain the contrast loss and the multi-task learning loss.

[0110] In some embodiments, the large language model can use a pre-trained GPT model as the core and fine-tune it for the home appliance field. The main features include:

[0111] 1. Using the Transformer architecture, it has powerful sequence modeling capabilities.

[0112] 2. Conduct pre-training on large-scale household appliance corpus to master domain knowledge.

[0113] 3. Support multi-modal input and can process fused feature representation.

[0114] 4. Use autoregressive generation to achieve flexible text generation and conversation capabilities.

[0115] In some embodiments, multi-task learning includes at least two of anomaly detection tasks, energy consumption optimization tasks, behavior prediction tasks, state estimation tasks, and instruction understanding tasks.

[0116] The anomaly detection task mainly relies on multimodal fusion data to detect anomalies in the working status of home appliances. For example, sensor data and status data of home appliances can be extracted from multimodal fusion data to perform anomaly detection.

[0117] The main purpose of the energy consumption optimization task is to control household appliances to achieve corresponding tasks with the lowest energy consumption, so as to optimize the energy consumption of household appliances.

[0118] The main purpose of the behavior prediction task is to predict user behavior through voice data and text data in multimodal fusion data, so as to control the corresponding home appliances according to the predicted behavior.

[0119] The state estimation task mainly estimates the working state of home appliances based on multimodal fusion data. For example, the state of home appliances can be estimated by extracting sensor data from multimodal fusion data, and then the corresponding estimation results can be given.

[0120] The main purpose of the command understanding task is to understand the commands through the voice data and text data in the multimodal fusion data, so as to control the corresponding home appliances according to the understood commands.

[0121] Step 84: Perform joint optimization based on contrast loss and multi-task learning loss, and adjust the weights of relevant data in multimodal data.

[0122] In some embodiments, the contrast loss and the multi-task learning loss may be weightedly summed to obtain a final total loss, and the total loss may be used to adjust the weights of relevant data in the multimodal data.

[0123] In this embodiment, at least two fusion methods are used to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In turn, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of smart control of home appliances, reducing misoperation and energy waste.

[0124] See also Fig. 9 , Fig. 9 This is a flow chart of an embodiment of a control method for household appliances provided by the present application. The method includes:

[0125] Step 91: Acquire multimodal fusion data.

[0126] The multimodal fusion data is obtained by using the method provided in any embodiment of the present application.

[0127] Step 92: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0128] Among them, the large language model is a large language model that has been trained in advance.

[0129] Step 93: Control the corresponding home appliances to work according to the multi-task decision.

[0130] In some embodiments, a corresponding intelligent agent is provided in the household appliance, and the intelligent agent can generate corresponding control instructions according to the corresponding task decision, thereby controlling the operation of the corresponding household appliance.

[0131] In some embodiments, the agent has the ability to make autonomous decisions. For example, if the multi-task decision determines that the home appliance needs to be turned on, the agent can independently determine the turning-on mode of the home appliance in combination with the current environmental data. For example, if the home appliance is an air conditioner, when the air conditioner needs to be turned on in the multi-task decision, a command to turn on the air conditioner is sent to the agent. The agent determines the turning-on temperature in combination with the current environmental data, and after determining the turning-on temperature, controls the air conditioner to start and work according to the turning-on temperature.

[0132] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0133] See also Fig.10 , Fig.10 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0134] Step 101: Acquire multimodal fusion data.

[0135] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0136] Step 102: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0137] Step 103: Perform safety check on the multi-task decision.

[0138] In some embodiments, the safety check mainly checks whether the home appliance control solution corresponding to the multi-task decision is safe during execution, such as whether there is an abnormal execution risk.

[0139] Step 104: After the safety check is passed, the corresponding home appliance is controlled to work according to the multi-task decision; or, when the safety check is not passed, the multi-task decision is corrected.

[0140] After the safety check is passed, the corresponding home appliances are controlled to work according to the multi-task decision. That is, after the safety check is passed, the task decision is sent to the corresponding home appliance. The home appliance is equipped with a corresponding intelligent agent, which can generate corresponding control instructions according to the corresponding task decision, and then control the corresponding home appliance.

[0141] When the safety check fails, the multi-task decision is revised so that the revised multi-task decision can successfully pass the safety check. After the safety check passes, the corresponding home appliances are controlled to work according to the revised multi-task decision.

[0142] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0143] Furthermore, by performing a safety check on the multi-task decision, the task decision is sent to the corresponding home appliance after the check is passed, or when the safety check fails, the multi-task decision is corrected to reduce the occurrence of sending erroneous task decisions to home appliances, thereby improving the control accuracy of home appliances.

[0144] See also Fig.11 , Fig.11 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0145] Step 111: Acquire multimodal fusion data.

[0146] In some embodiments, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0147] Step 112: Input the multimodal fusion data into the large language model to obtain the multi-task decision output by the large language model.

[0148] Step 113: Control the corresponding home appliances to work according to the multi-task decision.

[0149] In some embodiments, steps 111 to 113 have the same or similar technical solutions as any embodiment of the present application, and are not described in detail here.

[0150] Step 114: Record the working results of the home appliance and collect user feedback information.

[0151] Step 115: Update the large language model based on the work results and user feedback information.

[0152] In this embodiment, updating the large language model in combination with work results and user feedback information can improve the accuracy of the large language model in task reasoning, and make the reasoning of the large language model more in line with user preferences, thereby improving user experience.

[0153] See also Fig.12 , Fig.12 1 is a flow chart of another embodiment of the control method of household appliances provided by the present application. The method comprises:

[0154] Step 121: Acquire multimodal fusion data.

[0155] Among them, the multimodal fusion data is obtained using the method provided in any embodiment of the present application.

[0156] Step 122: Input the multimodal fusion data into the large language model so that the large language model determines at least two task types.

[0157] In some embodiments, the large language model is trained for multiple tasks during the training process, so the large language model can perform multi-task reasoning. When the multimodal fusion data indicates that multiple tasks need to be implemented, the large language model can obtain at least two task types based on the multimodal fusion data.

[0158] Step 123: Outputting a task decision corresponding to each task type according to at least two task types to control the operation of the corresponding household electrical appliance.

[0159] Step 124: Control the corresponding home appliance to work according to the multi-task decision.

[0160] In this embodiment, the fusion data is input into the large language model, which effectively improves the understanding ability of the large language model, can improve the accuracy of the multi-task decision-making output by the large language model, and thus significantly improve the generalization ability and robustness of the large language model in the home appliance scenario. In the process of home appliance control, a more natural and intelligent human-computer interaction experience is achieved, user satisfaction is improved, and the accuracy and efficiency of smart control of home appliances can be improved, reducing misoperation and energy waste.

[0161] Furthermore, the large language model can process more complex multimodal fusion data, determine at least two task types from the multimodal fusion data, and then generate corresponding task decisions according to the task types to control the operation of home appliances.

[0162] In the above embodiments, effective fusion of multimodal data in the field of home appliances is achieved, breaking down data silos; precise alignment of data of different modalities is achieved in a unified semantic space, improving the model's understanding ability; significantly improving the generalization ability and robustness of the model in home appliance scenarios; achieving a more natural and intelligent human-computer interaction experience, improving user satisfaction; improving the accuracy and efficiency of intelligent control of home appliances, and reducing misoperation and energy waste.

[0163] In some embodiments, see Fig.13The control system 1000 of the household appliance includes: a multimodal data acquisition module 201, a data preprocessing and encoding module 202, a multimodal fusion module 203, a GPT large model module 204, a multi-task learning module 205, a comparative learning module 206, a dynamic weight adjustment module 207 and an output processing and execution module 208.

[0164] The multimodal data acquisition module 201 is responsible for acquiring multimodal data from different sources, including but not limited to: text data, image data, voice data and sensor data.

[0165] Text data can be user input commands, historical interaction records, etc. Image data can be environmental images captured by home appliance cameras, user gestures, etc. Voice data can be user voice commands, environmental sounds, etc. Sensor data can be various sensor data such as temperature, humidity, light, energy consumption, etc.

[0166] The data preprocessing and encoding module 202 preprocesses and encodes the collected multimodal data, and converts different types of data into a unified vector representation. The specific steps include: text data processing, image data processing, voice data processing and sensor data processing.

[0167] Among them, text data processing can include word segmentation, removal of stop words, and converting text into vectors using a pre-trained word embedding model.

[0168] Image data processing can include image scaling, cropping, data augmentation, and extracting image features using pre-trained convolutional neural networks.

[0169] Speech data processing can include speech signal preprocessing, extracting MFCC features, converting speech to text using a pre-trained speech recognition model, and then converting it to a vector representation.

[0170] Sensor data processing can include data normalization, time series feature extraction, and converting sensor data into low-dimensional vector representations using dimensionality reduction techniques such as autoencoders or PCA.

[0171] The multimodal fusion module 203 adopts a multimodal fusion algorithm based on the attention mechanism to achieve effective integration of information from different modalities. The specific steps are as follows: self-attention calculation, cross-attention calculation, multi-head attention mechanism, residual connection and layer normalization, and fusion representation generation.

[0172] The GPT large model module 204 uses the pre-trained GPT model as the core and is fine-tuned for the home appliance field. The main features include:

[0173] Using the Transformer architecture, it has powerful sequence modeling capabilities.

[0174] Pre-training is performed on a large-scale corpus in the field of home appliances to master domain knowledge.

[0175] Supports multimodal input and can process fused feature representations.

[0176] Autoregressive generation is used to achieve flexible text generation and conversation capabilities.

[0177] The multi-task learning module 205 can design a multi-task learning framework and optimize multiple home appliance related tasks at the same time to improve the generalization ability of the model. The main tasks include: command understanding, state estimation, behavior prediction, energy consumption optimization and anomaly detection.

[0178] The contrastive learning module 206 introduces a contrastive learning strategy to enhance the alignment effect of different modal data in the semantic space. The main steps include: data enhancement, negative sample selection, contrastive loss calculation and joint optimization.

[0179] The dynamic weight adjustment module 207 can design a dynamic weight adjustment mechanism to adaptively adjust the importance of each modality according to different scenarios. The specific method includes: scene recognition, importance evaluation, Softmax normalization and dynamic fusion.

[0180] The output processing and execution module 208 is responsible for processing the output of the model and converting it into specific home appliance control instructions. The main functions include: output analysis, safety check, instruction conversion and execution feedback.

[0181] In one application scenario, combined with Fig.14 To illustrate, the multimodal data acquisition module 201 performs multimodal data acquisition, then performs data preprocessing and encoding, performs multimodal fusion on the encoded data, and then inputs the fused multimodal fusion data into the GPT large model for processing.

[0182] The GPT large model can perform multi-task learning, such as Task 1: instruction understanding; Task 2: state estimation; Task 3: behavior prediction; Task 4: energy consumption optimization; Task 5: anomaly detection.

[0183] The GPT large model can perform contrastive learning, data enhancement and negative sample selection, and then obtain contrastive loss. Combine multi-task learning and contrastive learning for joint optimization.

[0184] Dynamic weight adjustment is performed based on the results of the joint optimization, and the adjustment results are fed back to the multimodal fusion module.

[0185] Similarly, the processing result is output according to the result of dynamic weight adjustment.

[0186] The processing result is subjected to a safety check, and if it passes, it is sent to the home appliance to execute the corresponding control instruction. If it fails, the processing result is corrected and a new processing result is re-output.

[0187] During and after the household appliance executes the corresponding control instruction, execution feedback is performed, and the GPT large model in the GPT large model module 204 is updated using the execution feedback.

[0188] In one application scenario, combined with Fig.15 For explanation, the multimodal data acquisition module 201 performs multimodal data acquisition. Such as sensors, voice, images, and text. The data preprocessing and encoding module 202 performs sensor data processing, voice processing, image processing, and text processing. The multimodal fusion module 203 performs multimodal fusion on the processed data. Then the fused multimodal fusion data is input into the GPT large model module 204 for GPT model processing. Such as energy consumption optimization, indoor state estimation, and instruction understanding. And make decisions based on the results of energy consumption optimization, indoor state estimation, and instruction understanding.

[0189] After making a decision, the decision is checked for safety. If it passes, it is sent to the home appliance to execute the corresponding control instructions. If it fails, the decision is revised and re-made.

[0190] During and after the household appliance executes the corresponding control instruction, execution feedback is performed, and the GPT large model in the GPT large model module 204 is updated using the execution feedback.

[0191] In one application scenario, combined with Fig.16 To illustrate, the multimodal data acquisition module 201 performs multimodal data acquisition. Such as sensors, voice, images, and text. The data preprocessing and encoding module 202 performs sensor data processing, voice processing, image processing, and text processing. The multimodal fusion module 203 performs multimodal fusion on the processed data. The fused multimodal fusion data is then input into the GPT large model module 204 for GPT model processing. Such as recipe recommendation, freshness estimation, and ingredient identification. And based on recipe recommendation, freshness estimation, and ingredient identification, information is integrated to generate management suggestions. For example, the management interface is pushed to a mobile phone APP and displayed on the user interface.

[0192] Receive user feedback and use the user feedback to update the GPT large model in the GPT large model module 204.

[0193] In the smart water use scenario, the user says through the voice assistant: "I need to wash some greasy pots."

[0194] The control system of the home appliance responds: the voice recognition module converts the command into text.

[0195] The multimodal fusion module combines the current water quality data, water heater status, and dishwasher status.

[0196] The large language model understands the user's intent and recognizes that greasy dishes need to be disposed of.

[0197] The control system makes task decisions: a. Instruct the water softener to temporarily increase the hardness of the water to better remove oil stains. b. Notify the water heater to prepare enough hot water at 65°C. c. Select a strong degreasing program for the dishwasher and increase the amount of detergent.

[0198] The control system responds to the user via voice: "OK, I have prepared the best solution for you to wash greasy pots. The water heater is preparing 65°C hot water, and the dishwasher has been set to the strong degreasing mode. Please put the pot into the dishwasher and I will automatically start cleaning."

[0199] After the pots are placed in the dishwasher, the dishwasher, water softener, and water heater work according to the task decision to complete the cleaning of the pots.

[0200] Through this embodiment, the intelligent water management system can realize the intelligent linkage of the water softener, water heater and dishwasher, improve water use efficiency, and provide a personalized water use experience to meet the needs of different users. In this process, it can reasonably optimize energy use and reduce water and electricity consumption. It can extend the life of the equipment and reduce maintenance costs, thereby improving user satisfaction and enhancing product competitiveness.

[0201] In one application scenario, combined with Fig.17 To explain:

[0202] The user said through the voice assistant: "I need to wash some greasy pots."

[0203] The control system of the home appliance responds: the voice recognition module converts the command into text. That is, after the system is initialized, multimodal data collection is performed and the data is preprocessed.

[0204] The multimodal fusion module combines the current water quality data, water heater status and dishwasher status to perform multimodal fusion.

[0205] The large language model understands the user's intention and recognizes that greasy dishes need to be disposed of. At this time, the large language model processes the fused multimodal data, determines the task type, and obtains tasks such as water demand prediction, equipment status optimization, and anomaly detection (fault diagnosis).

[0206] The control system makes task decisions: a. Instruct the water softener to temporarily increase the hardness of the water outlet to better remove oil stains (water softener control). b. Notify the water heater to prepare enough hot water at 65℃ (water heater control). c. Select a strong degreasing program for the dishwasher and increase the amount of detergent (dishwasher control). And send alarms / maintenance recommendations.

[0207] The control system responds to the user via voice: "OK, I have prepared the best solution for washing greasy pots. The water heater is preparing 65°C hot water, and the dishwasher is set to the strong degreasing mode. Please put the pot into the dishwasher and I will start cleaning automatically."

[0208] After the pots are placed in the dishwasher, the dishwasher, water softener, and water heater execute control instructions according to the task decision.

[0209] And record the execution results and collect user feedback. Then use the execution results and user feedback to update and optimize the model. And detect whether the control system needs to continue to run. If so, continue to collect multimodal data. If not, end.

[0210] In the intelligent air conditioning control scenario, data collection is performed first, such as collecting text data, image data, voice data, and sensor data.

[0211] Among them, the text data can be generated by control instructions input by the user through a mobile phone APP.

[0212] The image data may be generated by an indoor environment image captured by a camera in the room.

[0213] Voice data can be generated by commands issued by users through voice assistants. Large language models can determine the position of human bodies based on human voice.

[0214] The sensor data may be sensor data such as indoor temperature, humidity, and air quality.

[0215] The above-mentioned text data, image data, voice data and sensor data are multimodally fused and processed. For example, the collected multimodal data is preprocessed and encoded. The multimodal fusion module is used to fuse the data of different modes into a unified representation.

[0216] The large language model processes the fused data to understand user intent and environmental conditions.

[0217] The multi-task learning module in the control system simultaneously performs command understanding, room state estimation and energy consumption optimization.

[0218] Contrastive learning modules in control systems enhance semantic alignment of data from different modalities.

[0219] The dynamic weight adjustment module in the control system adjusts the importance of each mode according to the current scenario.

[0220] The output of the large language model is parsed into specific air conditioning control instructions (such as adjusting temperature, wind speed, mode, etc.).

[0221] The execution module in the control system converts the instructions into signals that the air conditioner can recognize and executes them.

[0222] The control system records execution results and user feedback for continuous optimization of large language models.

[0223] In this way, the control system can intelligently adjust air-conditioning parameters based on the user's multimodal input (such as the voice command "a bit hot" combined with indoor temperature data) and environmental conditions to provide a more comfortable indoor environment while optimizing energy use.

[0224] In the above embodiments, effective fusion of multimodal data in the field of home appliances is achieved, breaking down data silos; precise alignment of data of different modalities is achieved in a unified semantic space, improving the model's understanding ability; significantly improving the generalization ability and robustness of the model in home appliance scenarios; achieving a more natural and intelligent human-computer interaction experience, improving user satisfaction; improving the accuracy and efficiency of intelligent control of home appliances, and reducing misoperation and energy waste.

[0225] In the present application, a method for controlling a household appliance is also provided, the method comprising: receiving a control instruction; wherein the control instruction is obtained by a multi-task decision generated by any of the above embodiments; and operating according to the control instruction.

[0226] See also Fig.18 , Fig.18 1 is a schematic diagram of a structure of an embodiment of a household appliance provided by the present application. The household appliance 180 includes a communication interface 181 and a processor 182, and the processor 182 is coupled to the communication interface 181 to implement the method of any of the above embodiments.

[0227] In summary, the multimodal data processing method, home appliance device and control method and system provided by the present application use at least two fusion methods to fuse multimodal data with sensor data to obtain multimodal fusion data with a unified semantic space, which can achieve effective fusion of multimodal data in the field of home appliances, break data silos, and further achieve accurate alignment of different modal data in a unified semantic space. After the multimodal fusion data is input into the large language model, the understanding ability of the large language model can be effectively improved, thereby significantly improving the generalization ability and robustness of the large language model in the home appliance scenario. In addition, a more natural and intelligent human-computer interaction experience can be achieved in the process of home appliance control, improving user satisfaction, and also improving the accuracy and efficiency of intelligent control of home appliances, reducing misoperation and energy waste.

[0228] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0229] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processing circuit component (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0230] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multimodal data processing method, characterized in that: The method comprises: Acquire multimodal data; wherein the multimodal data includes at least one of text data, image data and voice data, and sensor data; the sensor data is collected by a sensor corresponding to the home appliance; The multimodal data is fused using at least two fusion methods to obtain multimodal fused data with a unified semantic space.

2. The method according to claim 1, characterized in that Before fusing the multimodal data using at least two fusion methods to obtain multimodal fusion data with a unified semantic space, the method includes: Preprocessing the sensor data in the multimodal data to obtain a first vector representation; and preprocessing the text data in the multimodal data to obtain a text vector; and / or, preprocessing the image data in the multimodal data to obtain image features; and / or, preprocessing the speech data in the multimodal data to obtain a second vector representation; The method of fusing the multimodal data using at least two fusion methods to obtain multimodal fusion data with a unified semantic space includes: By using at least two fusion methods, data fusion is performed on at least one of the text vector, the image feature and the second vector representation and the first vector representation to obtain multimodal fusion data with a unified semantic space.

3. The method according to claim 1, characterized in that The at least two fusion modes include: Attention mechanism fusion method; Weighted fusion method; Splicing and fusion method.

4. The method according to claim 3, characterized in that The method of fusing the multimodal data using at least two fusion methods to obtain multimodal fusion data with a unified semantic space includes: Using the attention mechanism fusion method to fuse the multimodal data, to obtain first fused data; Performing data fusion on the multimodal data by using the weighted fusion method to obtain second fused data; Performing data fusion on the multimodal data by using the splicing and fusion method to obtain third fused data; The first fused data, the second fused data and the second fused data are fused to obtain the multimodal fused data.

5. The method according to claim 1, characterized in that After fusing the multimodal data using at least two fusion methods to obtain multimodal fusion data with a unified semantic space, the method further comprises: Inputting the multimodal fusion data into a large language model to obtain a contrast loss and a multi-task learning loss; wherein the multi-task learning includes at least two of anomaly detection tasks, energy consumption optimization tasks, behavior prediction tasks, state estimation tasks, and instruction understanding tasks; Joint optimization is performed based on the contrast loss and the multi-task learning loss, and the weights of relevant data in the multimodal data are adjusted.

6. A method for controlling a household appliance, characterized in that: The method comprises: Acquire multimodal fusion data; wherein the multimodal fusion data is obtained by using the method according to any one of claims 1 to 5; Inputting the multimodal fusion data into a large language model to obtain a multi-task decision output by the large language model; The corresponding household electrical appliances are controlled to work according to the multi-task decision.

7. The method according to claim 6, characterized in that The controlling the corresponding household appliance to work according to the multi-task decision includes: performing a safety check on the multi-tasking decision; After the safety check is passed, the corresponding household appliances are controlled to work according to the multi-task decision; Or, when the security check fails, modifying the multi-task decision.

8. A control system for household electrical appliances, characterized in that: The control system comprises: A communication interface, used to communicate with home appliances; A processor, coupled to the communication interface, configured to implement the method according to any one of claims 1 to 7.

9. A method for controlling a household appliance, characterized in that: The method comprises: Receiving a control instruction; wherein the control instruction is obtained by a multi-task decision generated by any of the methods described in claims 6-7; Works according to the control instructions described.

10. A household appliance, characterized in that: The household appliance comprises a communication interface and a processor, wherein the processor is coupled to the communication interface and is used to implement the method according to claim 9.

Images