Home control system, household appliance control method and household appliance control equipment

Abstract

The invention discloses a home control system, a household appliance control method and equipment. The home control system comprises: a cloud; the wearable device is in communication connection with the cloud end and sends the collected multi-modal data to the cloud end, and the multi-modal data comprises at least one of voice data, image data and posture data; the cloud analyzes the multi-modal data by using a large language model, generates a control instruction, sends the control instruction to the household appliance corresponding to the multi-modal data, and receives an equipment state fed back by the household appliance; and the wearable device receives and displays the device state. Through the above mode, the wearable device is used as an interaction medium, so that household appliance control is more natural and convenient.

Description

Technical Field

[0001] The present application relates to the field of home appliance technology, and in particular to a home control system, a control method and a device for home appliance equipment. Background Art

[0002] With the development of Internet of Things and artificial intelligence technologies, smart home control systems are becoming increasingly popular.

[0003] However, related smart home control methods still have some problems, such as a single input mode, lack of deep understanding of user intentions, and unintuitive presentation of home appliance status information. Summary of the invention

[0004] The home control system, home appliance control method and device provided in this application can use wearable devices as interactive media to make home appliance control more natural and convenient.

[0005] In a first aspect, the present application provides a home control system, which includes: a cloud; a wearable device, which is communicatively connected to the cloud and sends collected multimodal data to the cloud, wherein the multimodal data includes at least one of voice data, image data, and posture data; the cloud uses a large language model to analyze the multimodal data, generate control instructions, and send the control instructions to the home appliance corresponding to the multimodal data, and receive the device status feedback from the home appliance; the wearable device receives the device status and displays it.

[0006] Among them, the home control system also includes: a home gateway, which is connected to the cloud, wearable devices and home appliances respectively, receives multimodal data sent by wearable devices and forwards it to the cloud, and receives control instructions sent by the cloud and forwards them to home appliances.

[0007] Among them, the cloud selects the target device state that meets the user's preference from the device state and sends the target device state to the wearable device.

[0008] Among them, the wearable device is smart glasses, and the multimodal data includes voice data, image data and posture data; the cloud uses a large language model to analyze the voice data, image data and posture data, obtains the corresponding home appliance type in the voice data, and the home appliance that meets the home appliance type in the image data, and generates control instructions for the corresponding home appliance.

[0009] After receiving and displaying the device status, the smart glasses receive gesture commands and / or voice commands, and send the gesture commands and / or voice commands to the cloud, where the gesture commands and / or voice commands are related to the device status.

[0010] Among them, the smart glasses superimpose the device status within the visual range of the smart glasses for display.

[0011] In a second aspect, the present application provides a control method for household appliances, which is applied to wearable devices, and the method includes: collecting multimodal data, the multimodal data including at least one of voice data, image data and posture data; sending the multimodal data to the cloud, so that the cloud uses a large language model to analyze the multimodal data and generate control instructions; receiving the device status of the household appliance sent by the cloud and displaying it; wherein the device status is obtained after the household appliance executes the control instruction.

[0012] In a third aspect, the present application provides a wearable device, which includes a processor and a memory and a communication interface coupled to the processor, the memory storing at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the method provided in the second aspect.

[0013] In a fourth aspect, the present application provides a method for controlling household appliances, which is applied to the cloud. The method includes: receiving multimodal data sent by a wearable device, the multimodal data including at least one of voice data, image data and posture data; analyzing the multimodal data using a large language model, generating control instructions, and sending the control instructions to the household appliance corresponding to the multimodal data; receiving device status feedback from the household appliance; and sending the device status to the wearable device.

[0014] In a fifth aspect, the present application provides an electronic device comprising a processor and a memory and a communication interface coupled to the processor, wherein the memory stores at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the method provided in the fourth aspect.

[0015] The beneficial effects of the present application are as follows: Different from the prior art, the home control system, the control method and the device for home appliances provided by the present application utilize wearable devices to collect multimodal data, improve the diversity of data collection, and increase the interactive input methods of the home control system. Furthermore, the cloud utilizes a large language model to analyze the multimodal data, generate control instructions, enhance the in-depth understanding of the multimodal data, improve the accuracy of the control instructions, and utilize wearable devices as an interactive medium to make home appliance control more natural and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0017] Figure 1It is a structural schematic diagram of an embodiment of a home control system provided by the present application;

[0018] Figure 2 is a structural schematic diagram of another embodiment of the home control system provided by the present application;

[0019] Figure 3 It is a flowchart of an embodiment of a control method for household electrical appliances provided by the present application;

[0020] Figure 4 is a flow chart of another embodiment of the control method of household electrical appliances provided by the present application;

[0021] Figure 5 It is a flowchart of an embodiment of an interaction method for household appliances provided by the present application;

[0022] Figure 6 is a flowchart of another embodiment of the interactive method for household appliances provided by the present application;

[0023] Figure 7 is a flowchart of another embodiment of the interactive method for household appliances provided by the present application;

[0024] Figure 8 This application provides Figure 7 A flow chart of an embodiment of step 72 in FIG.

[0025] Fig. 9 is a flowchart of another embodiment of the interactive method for household appliances provided by the present application;

[0026] Fig.10 is a structural schematic diagram of an embodiment of a wearable device provided by the present application;

[0027] Fig.11 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] With the development of Internet of Things and artificial intelligence technologies, smart home control systems are becoming increasingly popular.

[0031] However, related smart home control methods still have some problems, such as a single input mode, lack of deep understanding of user intentions, and unintuitive presentation of home appliance status information.

[0032] Based on this, the present application proposes to use wearable devices to collect multimodal data, improve the diversity of data collection, increase the interactive input methods of the home control system, and further, use a large language model in the cloud to analyze the multimodal data, generate control instructions, improve the in-depth understanding of the multimodal data, improve the accuracy of the control instructions, and use wearable devices as interactive media to make home appliance control more natural and convenient. Please refer to any of the following embodiments for details.

[0033] See also Figure 1 , Figure 1 1 is a schematic diagram of a structure of an embodiment of a home control system provided by the present application. The home control system 100 includes: a cloud 10 and a wearable device 20.

[0034] The wearable device 20 is connected to the cloud 10 for communication, and sends the collected multimodal data to the cloud 10, where the multimodal data includes at least one of voice data, image data, and posture data.

[0035] The cloud 10 uses a large language model to analyze the multimodal data, generates control instructions, sends the control instructions to the home appliance corresponding to the multimodal data, and receives the device status feedback from the home appliance; the wearable device 20 receives the device status and displays it.

[0036] In some embodiments, the cloud 10 selects a target device state that meets the user's preference from the device states, and sends the target device state to the wearable device 20 .

[0037] In some embodiments, the wearable device 20 is a pair of smart glasses, and the multimodal data includes voice data, image data, and posture data; the cloud 10 uses a large language model to analyze the voice data, image data, and posture data, obtains the type of home appliance corresponding to the voice data, and the home appliance that meets the type of home appliance in the image data, and generates control instructions for the corresponding home appliance.

[0038] In some embodiments, after receiving and displaying the device status, the smart glasses receive gesture instructions and send the gesture instructions to the cloud 10, where the gesture instructions are related to the device status.

[0039] In some embodiments, after receiving and displaying the device status, the smart glasses receive voice instructions and send the voice instructions to the cloud 10, where the voice instructions are related to the device status.

[0040] In some embodiments, after receiving and displaying the device status, the smart glasses receive gesture instructions and voice instructions, and send the gesture instructions and voice instructions to the cloud 10, where the gesture instructions and voice instructions are related to the device status.

[0041] In some embodiments, the smart glasses display the device status superimposed within the visual range of the smart glasses.

[0042] In this embodiment, the wearable device 20 is used to collect multimodal data, improve the diversity of data collection, and increase the interactive input methods of the home control system. Furthermore, the cloud 10 uses a large language model to analyze the multimodal data, generate control instructions, enhance the in-depth understanding of the multimodal data, improve the accuracy of the control instructions, and use the wearable device 20 as an interactive medium to make home appliance control more natural and convenient.

[0043] See also Figure 2 , Figure 2 1 is a schematic diagram of another embodiment of a home control system provided by the present application. The home control system 100 comprises: a cloud 10, a home gateway 30 and a wearable device 20.

[0044] The home gateway 30 is respectively connected to the cloud 10, the wearable device 20 and the home appliance (not shown), receives the multimodal data sent by the wearable device 20 and forwards it to the cloud 10, and receives the control instructions sent by the cloud 10 and forwards them to the home appliance.

[0045] In some embodiments, the cloud 10 selects a target device state that meets the user's preference from the device state, and sends the target device state to the wearable device 20 via the home gateway 30 .

[0046] In some embodiments, the wearable device 20 is a pair of smart glasses, and the multimodal data includes voice data, image data, and posture data; the cloud 10 uses a large language model to analyze the voice data, image data, and posture data, obtains the type of home appliance corresponding to the voice data, and the home appliance that meets the type of home appliance in the image data, and generates control instructions for the corresponding home appliance.

[0047] In some embodiments, after receiving and displaying the device status, the smart glasses receive gesture instructions and send the gesture instructions to the cloud 10 through the home gateway 30, where the gesture instructions are related to the device status.

[0048] In some embodiments, after receiving and displaying the device status, the smart glasses receive voice commands and send the voice commands to the cloud 10 through the home gateway 30, where the voice commands are related to the device status.

[0049] In some embodiments, after receiving and displaying the device status, the smart glasses receive gesture commands and voice commands, and send the gesture commands and voice commands to the cloud 10 through the home gateway 30, where the gesture commands and voice commands are related to the device status.

[0050] In some embodiments, the smart glasses display the device status superimposed within the visual range of the smart glasses.

[0051] In this embodiment, wearable devices are used to collect multimodal data, improve the diversity of data collection, and increase the interactive input methods of the home control system. Furthermore, the cloud uses a large language model to analyze the multimodal data, generate control instructions, enhance the in-depth understanding of the multimodal data, improve the accuracy of the control instructions, and use wearable devices as an interactive medium to make home appliance control more natural and convenient.

[0052] See also Figure 3 , Figure 3 1 is a flow chart of an embodiment of a control method for a household appliance provided by the present application. Applied to a wearable device, the method includes:

[0053] Step 31: Collect multimodal data, where the multimodal data includes at least one of voice data, image data, and posture data.

[0054] In some embodiments, multimodal data can be collected using relevant sensors on the wearable device. For example, voice data can be collected using a microphone on the wearable device. Image data can be collected using an image sensor on the wearable device. Gyroscopes and inertial navigation components on the wearable device can be used to collect attitude data. In other embodiments, attitude data can be collected in combination with image data collected by an image sensor.

[0055] Step 32: Send the multimodal data to the cloud, so that the cloud uses a large language model to analyze the multimodal data and generate control instructions.

[0056] Step 33: Receive and display the device status of the home appliance sent from the cloud; wherein the device status is obtained after the home appliance executes the control instruction.

[0057] In this embodiment, wearable devices are used to collect multimodal data, improve the diversity of data collection, and increase the interactive input methods of the home control system. Furthermore, the cloud uses a large language model to analyze the multimodal data, generate control instructions, enhance the in-depth understanding of the multimodal data, improve the accuracy of the control instructions, and use wearable devices as an interactive medium to make home appliance control more natural and convenient.

[0058] See also Figure 4 , Figure 4 This is a flow chart of another embodiment of the control method of the household appliance provided by the present application. Applied in the cloud, the method includes:

[0059] Step 41: receiving multimodal data sent by a wearable device, where the multimodal data includes at least one of voice data, image data, and posture data.

[0060] Step 42: Analyze the multimodal data using the large language model, generate control instructions, and send the control instructions to the home appliance corresponding to the multimodal data.

[0061] Step 43: Receive the device status fed back by the home appliance.

[0062] Step 44: Send the device status to the wearable device.

[0063] In the present application, the wearable device may be a smart bracelet, a smart watch and / or smart glasses.

[0064] In an application scenario, taking the wearable device as smart glasses as an example, the control process of home appliances with multi-modal input is as follows:

[0065] 1) The user issues a control command through the smart glasses, such as saying "turn on the water heater" and looking at the target water heater.

[0066] 2) Smart glasses collect multimodal data such as voice, vision, and head posture.

[0067] 3) The multimodal data is uploaded to the cloud server after preprocessing.

[0068] 4) The cloud server uses a large multimodal model to analyze data and combines contextual information to understand user intent.

[0069] 5) The cloud server generates precise control instructions and sends them to the target water heater through the home gateway.

[0070] 6) The water heater starts and feeds back the result. The user can intuitively view the remaining hot water and water temperature of the water heater through the smart glasses.

[0071] In an application scenario, taking smart glasses as an example, the process of visualizing the status of home appliances based on AR is as follows:

[0072] 1) Smart home appliances (such as dishwashers) report their operating status to the home gateway in real time.

[0073] 2) The home gateway uploads the status information to the cloud server.

[0074] 3) The cloud server generates AR visualization content based on user preferences, such as displaying the current step and remaining time of the dishwasher.

[0075] 4) AR content is sent to smart glasses.

[0076] 5) Smart glasses overlay AR content on the user’s field of view.

[0077] 6) Users can interact with the AR interface through sight, gestures, or voice, such as adjusting the water temperature in a water dispenser.

[0078] In one application scenario, taking the wearable device as smart glasses as an example, the water heater status visualization and control are as follows:

[0079] 1) The user uses the voice command "check the status of the water heater" and looks at the water heater, and the smart glasses collect relevant multimodal input.

[0080] 2) The cloud server uses a multimodal large model to understand the user's intention and retrieve the current operating status data of the water heater, including the remaining hot water, water temperature, etc.

[0081] 3) The cloud server generates an AR interface based on the user's location and line of sight, which is superimposed on the user's field of view, intuitively displaying the various status indicators of the water heater.

[0082] 4) The user can further control the water heater through gestures or voice, such as adjusting the water temperature. The smart glasses send the user's control instructions to the home gateway, which is ultimately communicated to the water heater device.

[0083] 5) The water heater executes the user's control instructions and provides real-time feedback on the latest operating status data. The cloud server updates the display content of the AR interface accordingly.

[0084] In one application scenario, taking the wearable device as smart glasses as an example, the dishwasher status visualization and control are as follows:

[0085] 1) The user says the voice command "check the status of the dishwasher" while focusing on the dishwasher at home.

[0086] 2) Smart glasses collect multimodal inputs such as voice and sight and send them to cloud servers for processing.

[0087] 3) The cloud server multimodal large model understands the user's query intention and pulls the current operating status of the dishwasher from the home gateway, including the washing program stage, remaining time, etc.

[0088] 4) The cloud server generates an AR interface based on the user's location and line of sight, which is superimposed on the user's field of view, intuitively displaying the dishwasher's status information.

[0089] 5) The user can further control the dishwasher through gestures or voice, such as adjusting the water temperature or pausing / resuming the washing process. The smart glasses send the user's control instructions to the home gateway, which is ultimately conveyed to the dishwasher device.

[0090] 6) The dishwasher executes the user's control instructions and provides real-time feedback on the latest operating status data. The cloud server updates the display content of the AR interface accordingly.

[0091] In the above embodiments of the present application, multimodal input improves the accuracy of home appliance control and reduces operation ambiguity. Context awareness makes the system smarter and can predict user needs. AR status visualization provides intuitive and real-time home appliance information and improves user experience. Smart glasses as an interactive medium make home appliance control more natural and convenient. The home control system has good scalability and is easy to integrate new home appliances and functions, such as water dispensers, TVs, etc.

[0092] In some embodiments, the instructions of home appliances in the related art are usually presented in the form of static text and pictures, which makes it difficult for users to understand the complex operation steps and lacks interactivity. The recipes are also mainly presented in the form of text and pictures, lacking an intuitive display of the cooking process and failing to meet the learning needs of users.

[0093] Based on this, the present application proposes to use a wearable device to interact with a user, receive interaction data between the user and the home appliance, and use a large language model to identify the intent of the interaction data to obtain the target intent, generate a dynamic manual according to the target intent, and display the dynamic manual to solve the problems of the lack of interactivity and intuitiveness of the manual of the home appliance in the related art. Please refer to any of the following embodiments for details.

[0094] See also Figure 5 , Figure 5 1 is a flow chart of an embodiment of an interaction method for household appliances provided by the present application. Applied to a wearable device, the method includes:

[0095] Step 51: Receive interaction data corresponding to the user.

[0096] The interaction data includes at least one of voice data, image data and gesture data.

[0097] The interactive data at least includes a target object, which may be a function in a home appliance, a functional component in a home appliance, or a home appliance.

[0098] Step 52: Use the large language model to perform intent recognition on the interaction data to obtain the target intent.

[0099] In some embodiments, the large language model is a model trained in advance. In some embodiments, the target intent may represent an understanding intent of a target object of the home appliance.

[0100] Step 53: Generate a dynamic specification based on the target intent.

[0101] In some embodiments, the target object that needs to be understood can be obtained from the target intent, and then the text information, image information and voice information corresponding to the target object can be obtained; and a dynamic description can be generated based on the text information, image information and voice information.

[0102] Step 54: Display the dynamic instruction manual.

[0103] Wearable devices are used to display dynamic instructions so that users can intuitively understand the detailed instructions of the target object and facilitate user learning.

[0104] In this embodiment, a wearable device is used to interact with a user and receive interaction data between the user and the home appliance. A large language model is used to identify the intent of the interaction data to obtain the target intent. A dynamic instruction manual is generated based on the target intent. The dynamic instruction manual is displayed to solve the problems of the lack of interactivity and intuitiveness of the manuals for home appliances in related technologies.

[0105] See also Figure 6 , Figure 6 1 is a flow chart of another embodiment of the interaction method of the household appliance provided by the present application. Applied to a wearable device, the method includes:

[0106] Step 61: Receive the user's voice data.

[0107] The voice data at least includes a target object. The target object may be a function in a home appliance and / or a functional component in a home appliance. For example, the corresponding text content in the voice data is: I want to know what functions a dishwasher has. Then the target object is the dishwasher.

[0108] Step 62: Use the large language model to perform intent recognition on the voice data to obtain the home appliances in the voice data and the target intent for the home appliances.

[0109] In some embodiments, the speech data is converted into text using a large language model to obtain text data, and then the text data is subjected to intent recognition to obtain the home appliance in the speech data and the target intent for the home appliance. For example, the corresponding text content in the speech data is: I want to know what functions the dishwasher has. Then the home appliance in the speech data is a dishwasher, and the target intent is to know the functions of the dishwasher.

[0110] Step 63: Generate a dynamic specification based on the target intent.

[0111] In some embodiments, the target object that needs to be understood can be obtained from the target intent as a dishwasher, and then the text information, image information and voice information corresponding to all functions of the dishwasher can be obtained; and a dynamic instruction manual can be generated based on the text information, image information and voice information.

[0112] Step 64: Display the dynamic instruction manual.

[0113] In this embodiment, a wearable device is used to interact with a user and receive the user's voice data; a large language model is used to perform intent recognition on the interaction data to obtain the target intent; a dynamic instruction manual is generated based on the target intent; and the dynamic instruction manual is displayed to solve the problems of the lack of interactivity and intuitiveness of the manuals of home appliances in related technologies.

[0114] See also Figure 7 , Figure 7 1 is a flow chart of another embodiment of the interaction method of the household appliance provided by the present application. Applied to a wearable device, the method includes:

[0115] Step 71: Receive the user's voice data and image data.

[0116] In some embodiments, the image data can be collected within the collection field of view of the image sensor of the household appliance on the wearable device.

[0117] For example, when the user's voice data is collected, the image data is collected synchronously.

[0118] Step 72: Use the large language model to perform intent recognition on the voice data and image data to obtain the target intent.

[0119] In some embodiments, a large language model can be used to simultaneously perform intent recognition on voice data and image data, and then combine the content in the voice data and the target object in the image data to obtain the target intent.

[0120] In some embodiments, see Figure 8 , step 72 may be the following process:

[0121] Step 721: Use a large language model to perform content recognition on the speech data to obtain key content.

[0122] For example, the corresponding text content in the voice data is: What is this useful for? Using the large language model to perform content recognition on the voice data, the key content obtained is: I want to know the function of something. At this time, we do not know what this thing is specifically.

[0123] Step 722: Determine the target object corresponding to the key content from the image data.

[0124] Furthermore, after obtaining the key content from the voice data, the large language model determines the target object corresponding to the key content from the image data. For example, if the large language model recognizes the button of the range hood from the image data, the button can be used as the target object.

[0125] Step 723: Obtain target intent based on key content and target object.

[0126] In some embodiments, when the key content of the voice data is: wanting to know the function of something, and the target object in the image data is the buttons of a range hood, the target intention is to know the functions of the buttons of the range hood.

[0127] Step 73: Generate a dynamic specification based on the target intent.

[0128] Step 74: Display the dynamic instruction manual.

[0129] In some embodiments, when a dynamic manual is displayed using a wearable device, the user's age stage can be obtained; the voice playback speed can be determined according to the age stage; and the dynamic manual can be displayed according to the voice playback speed. For example, the higher the age stage, the older the user is, and the slower the voice playback speed. For example, if the user is elderly, when the dynamic manual is displayed, the relevant content of the manual is played at the voice playback speed corresponding to the elderly stage.

[0130] In this embodiment, a wearable device is used to interact with a user to receive voice data and image data between the user and the home appliance. A large language model is used to perform intent recognition on the voice data and image data to obtain the target intent. A dynamic instruction manual is generated based on the target intent. The dynamic instruction manual is displayed to solve the problems of lack of interactivity and intuitiveness in the manuals of home appliances in related technologies, allowing users to understand the operating steps more intuitively and improve the user experience.

[0131] See also Fig. 9 , Fig. 9 This is a flow chart of another embodiment of the method for interacting with household appliances provided by the present application. The method includes:

[0132] Step 91: Receive the user's cooking requirements.

[0133] The cooking requirement can be sent to the wearable device in the form of voice, image, text, etc. That is, the wearable device receives voice data, image data and / or text data with the cooking requirement.

[0134] Step 92: Input the dish demand into the large language model to obtain the dish production data corresponding to the dish demand.

[0135] Step 93: Display the dish preparation data.

[0136] In the process of displaying the dish preparation data, user instructions are received; and the presentation method and content of the dish preparation data are adjusted according to the user instructions.

[0137] In this embodiment, wearable devices are used to interact with users and receive users' dish making requirements; and dish making data corresponding to the dish making requirements are displayed using a large language model. This solves the problems of lack of interactivity and intuitiveness of recipes in related technologies, allowing users to understand the operating steps more intuitively and improve the user experience.

[0138] In one application scenario, taking the wearable device as smart glasses as an example, a dynamic manual for a water heater based on smart glasses. The elderly user wears smart glasses, looks at a function button on the water heater, and issues a voice command in dialect, "I want to know what this button does." Smart glasses capture the user's voice and visual information and transmit it to a multimodal large model (large language model). The multimodal large model recognizes the dialect of the elderly user, understands that the user's intention is to understand the function of the function button, and generates a dynamic manual containing multiple modal information such as large font text, high-contrast pictures, slow speech, simplified AR animation, etc. The dynamic manual is presented in the user's field of view through the AR display function of the smart glasses, for example, the function of the function button is displayed on the screen in the form of large font text and slow speech, and the operation steps of the function are superimposed on the image of the water heater in the form of a simplified AR animation.

[0139] In one application scenario, taking wearable devices as smart glasses as an example, a multimodal recipe for braised pork based on a multimodal large model. An elderly user selects the braised pork dish he wants to learn through his mobile phone. The system transmits the dish information to the multimodal large model (large language model). The multimodal large model generates a braised pork recipe containing multiple modal information such as detailed text steps, clear pictures, slow voice explanations, and segmented video demonstrations. Users can watch the video demonstration of the recipe, listen to the slow voice explanation, or browse the detailed text steps and clear pictures through their mobile phones. The system can adjust the presentation and content of the recipe based on the user's learning progress and feedback, such as pausing the video, repeating the explanation, etc., such as providing a more detailed explanation or demonstration when the user encounters difficulties.

[0140] See also Fig.10 , Fig.10 1 is a schematic diagram of a structure of an embodiment of a wearable device provided by the present application. The wearable device 20 includes a processor 201, a memory 202 and a communication interface 203 coupled to the processor 201, and the memory 202 stores at least one computer program, which is used to implement the method of any of the above embodiments when loaded and executed by the processor.

[0141] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of an embodiment of an electronic device provided by the present application. The electronic device 300 includes a processor 301, a memory 302 and a communication interface 303 coupled to the processor 301, and the memory 302 stores at least one computer program, which is used to implement the method of any of the above embodiments when loaded and executed by the processor.

[0142] The electronic device 300 may be any of the above-mentioned cloud devices, wearable devices, and home gateways.

[0143] In some embodiments, the wearable device in the above-mentioned home control system 100 is connected to the cloud for communication, receives the interaction data between the user and the home appliance, and sends the interaction data to the cloud. The cloud uses a large language model to perform intent recognition on the interaction data, obtains the target intent, and generates a dynamic instruction manual based on the target intent; the wearable device receives the dynamic instruction manual and displays it. That is, the above-mentioned home control system 100 can implement the method of any embodiment of the present application.

[0144] In summary, the interaction method, wearable device, and home control system for home appliances provided in the present application utilize wearable devices to interact with users and receive interaction data between users and home appliances; and utilize a large language model to perform intent recognition on the interaction data to obtain the target intent; generate a dynamic instruction manual based on the target intent; and display the dynamic instruction manual to solve the problems of lack of interactivity and intuitiveness in home appliance manuals in related technologies.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0146] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processing circuit component (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0147] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A home control system, characterized in that: The home control system comprises: Cloud; A wearable device is connected to the cloud for communication and sends the collected multimodal data to the cloud, wherein the multimodal data includes at least one of voice data, image data and posture data; The cloud uses a large language model to analyze the multimodal data, generates a control instruction, sends the control instruction to the home appliance corresponding to the multimodal data, and receives the device status fed back by the home appliance; The wearable device receives and displays the device status.

2. The home control system according to claim 1, characterized in that: The home control system also includes: A home gateway, wherein the home gateway is connected to the cloud, the wearable device and the home appliance respectively, receives the multimodal data sent by the wearable device and forwards it to the cloud, and receives the control instruction sent by the cloud and forwards it to the home appliance.

3. The home control system according to claim 1, characterized in that: The cloud selects a target device state that meets the user's preference from the device states, and sends the target device state to the wearable device.

4. The home control system according to claim 1, characterized in that: The wearable device is a smart glasses, and the multimodal data includes the voice data, image data and posture data; The cloud uses a large language model to analyze the voice data, the image data and the posture data, obtains the home appliance type corresponding to the voice data, the home appliance that matches the home appliance type in the image data, and generates the control instructions corresponding to the home appliance.

5. The home control system according to claim 4, characterized in that: After receiving and displaying the device status, the smart glasses receive gesture instructions and / or voice instructions, and send the gesture instructions and / or voice instructions to the cloud, where the gesture instructions and / or voice instructions are related to the device status.

6. The home control system according to claim 4, characterized in that: The smart glasses overlay the device status within the visual range of the smart glasses for display.

7. A method for controlling a household appliance, characterized in that: Applied to a wearable device, the method comprises: Collecting multimodal data, wherein the multimodal data includes at least one of the voice data, image data, and posture data; Sending the multimodal data to the cloud, so that the cloud analyzes the multimodal data using a large language model and generates control instructions; Receive and display the device status of the home appliance sent from the cloud; wherein the device status is obtained after the home appliance executes the control instruction.

8. A wearable device, characterized in that: The wearable device includes a processor and a memory and a communication interface coupled to the processor, the memory stores at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the method as claimed in claim 7.

9. A method for controlling a household appliance, characterized in that: Applied to the cloud, the method includes: Receiving multimodal data sent by a wearable device, wherein the multimodal data includes at least one of the voice data, image data, and posture data; Analyzing the multimodal data using a large language model to generate a control instruction, and sending the control instruction to the home appliance corresponding to the multimodal data; Receiving device status feedback from the home appliance; The device status is sent to the wearable device.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory and a communication interface coupled to the processor, the memory stores at least one computer program, and when the at least one computer program is loaded and executed by the processor, it is used to implement the method as claimed in claim 9.

Images