Terminal equipment and electronic equipment control method
Through standardized functions, the electronic device functions and parameters are described, and the multi-modal large model is trained to independently control home appliances, which solves the problem that the existing home appliance control system cannot meet the user's changing needs and complex environmental tasks, and realizes integrated decision-making control in the whole house , improves control efficiency.
Patent Information
- Application Number
- CN202410946689.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing home appliance control system cannot meet the changing needs of users, and when facing complex environments and tasks, its control effect often cannot meet expectations and cannot achieve integrated decision-making control in the whole house.
By using standardized functions to describe the functions and parameters of electronic devices, a multimodal large model is trained to independently and flexibly control multiple electronic devices to achieve integrated decision-making control. Specific steps include receiving control tasks, constructing prompt instructions, inputting to multimodal large model, outputting thinking, action and action outputs, and starting the corresponding electronic device to perform control operations.
The efficiency of multimodal large model decision-making and generation planning is improved, electronic equipment control becomes more convenient, and the whole-house integrated decision-making control is realized, which can better meet the changing needs of users.
Smart Images

Figure CN118921241A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of device control, and in particular to a terminal device and an electronic device control method. Background Art
[0002] With the continuous development of science and technology, artificial intelligence technology has been widely used in various fields, especially in the field of home appliance control, where the application of artificial intelligence technology has become a trend. However, most of the current home appliance control systems are still based on programmed control strategies, and use internally designed integrated control commands to achieve whole-house device control. This strategy often cannot meet the changing needs of users, and when faced with complex environments and tasks, its control effect often fails to meet expectations. Some home appliance control systems also use state and intent recognition detection solutions to achieve decision-making control of home appliances through state threshold point detection, but this solution can only perform modular detection and execution in different regions, and cannot truly achieve whole-house integrated decision-making control. Summary of the invention
[0003] Some embodiments of the present application provide a terminal device and an electronic device control method. By using standardized functions to describe the functions and input parameters of electronic devices, the efficiency of multimodal large model decision-making and generation planning is improved, making electronic device control more convenient, and the multimodal large model autonomously and flexibly controls multiple electronic devices to achieve integrated decision-making control.
[0004] In a first aspect, some embodiments of the present application provide a terminal device, including:
[0005] The controller is configured as:
[0006] Receive control tasks;
[0007] constructing prompt instructions based on the control task;
[0008] Input the prompt instruction into the multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter;
[0009] The electronic device corresponding to the device name is started and the parameters are sent to the electronic device, so that the electronic device performs the control operation corresponding to the parameters.
[0010] In some embodiments, the controller performs a receiving control task and is further configured to:
[0011] Receiving voice data input by a user;
[0012] Identify the voice text corresponding to the voice data;
[0013] If no control instruction corresponding to the voice text is matched, it is determined that the voice text is a control task.
[0014] In some embodiments, the controller performs a receiving control task and is further configured to:
[0015] Receive a timed control task set by a user, wherein the timed control task includes a timed time and a control task;
[0016] When the timing time is reached, the control task is received.
[0017] In some embodiments, the controller performs a receiving control task and is further configured to:
[0018] Receive image data or abnormal message sent by an electronic device, wherein the image data is an image taken by the electronic device, and the abnormal message is a message sent by the electronic device after detecting an abnormal condition, and the abnormal message includes abnormal data.
[0019] In some embodiments, the controller performs a receiving control task and is further configured to:
[0020] Receiving image data sent by an electronic device;
[0021] Identify the image data to obtain image description information;
[0022] The control task is determined to be the image description information.
[0023] In some embodiments, the controller executes the prompt instruction based on the control task, and is further configured to:
[0024] A prompt instruction is constructed based on the control task and user preference information, wherein the user preference information includes device parameters of the electronic device related to the control task, and the device parameters include the device parameters of the electronic device most frequently used by the user within a preset time.
[0025] In some embodiments, the controller executes the prompt instruction based on the control task, and is further configured to:
[0026] A prompt instruction is constructed based on the control task and a function description of a target electronic device, wherein the target electronic device is an electronic device that does not appear in the data set, and the function description of the target electronic device is manually added by a user after establishing a connection between the target electronic device and a terminal device.
[0027] In some embodiments, after sending the parameters to the electronic device, the controller is further configured to:
[0028] The electronic device is controlled to feed back the execution result and status of the control operation to the multimodal large model, so that the multimodal large model continues to output thoughts, actions and action outputs until the control task is completed or an abnormality occurs in the control task.
[0029] In a second aspect, some embodiments of the present application provide a method for controlling an electronic device, including:
[0030] Receive control tasks;
[0031] constructing prompt instructions based on the control task;
[0032] Input the prompt instruction into the multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter;
[0033] The electronic device corresponding to the device name is started and the parameters are sent to the electronic device, so that the electronic device performs the control operation corresponding to the parameters.
[0034] In some embodiments, the step of receiving a control task includes:
[0035] Receiving voice data input by a user;
[0036] Identify the voice text corresponding to the voice data;
[0037] If no control instruction corresponding to the voice text is matched, it is determined that the voice text is a control task.
[0038] Some embodiments of the present application provide a terminal device and an electronic device control method. A modal big model is trained through an electronic device control field data set. Wherein, the data set includes dialog format data related to electronic device control. The function description of the electronic device is standardized and written, and the function description includes the device name, function description, input parameter name and input parameter description of the electronic device. After receiving the control task, a prompt instruction is constructed based on the control task; the prompt instruction is input into the trained multimodal big model, so that the multimodal big model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device. Wherein, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter. Parse the device name and device input parameter output of the multimodal big model. Start the electronic device corresponding to the device name and send the parameter to the electronic device so that the electronic device performs the control operation corresponding to the parameter. The embodiment of the present application uses a standardized function to describe the function and input parameter of the electronic device, improves the efficiency of the multimodal big model decision and generation planning, makes the electronic device control more convenient, and the multimodal big model autonomously and flexibly controls multiple electronic devices to achieve integrated decision control. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 A schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;
[0041] Figure 2 A schematic diagram of the hardware configuration of a display device provided in some embodiments of the present application;
[0042] Figure 3 A schematic diagram of software configuration of a display device provided in some embodiments of the present application;
[0043] Figure 4 A flowchart of an electronic device control method provided in some embodiments of the present application;
[0044] Figure 5 A schematic diagram of a home page provided for some embodiments of the present application;
[0045] Figure 6 A schematic diagram of an add device page provided in some embodiments of the present application;
[0046] Figure 7 A schematic diagram of another home page provided for some embodiments of the present application;
[0047] Figure 8 A schematic diagram of a prompt page provided for some embodiments of the present application;
[0048] Fig. 9 A schematic diagram of a function description filling page provided in some embodiments of the present application;
[0049] Fig.10 A flowchart of another electronic device control method provided for some embodiments of the present application. DETAILED DESCRIPTION
[0050] The following embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following embodiments do not represent all implementations consistent with the present application. They are only examples of systems and methods consistent with some aspects of the present application as detailed in the claims.
[0051] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0052] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.
[0053] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0054] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0055] In the embodiments of the present application, terminal devices generally refer to electronic devices with communication processing and control functions. Terminal devices include but are not limited to mobile terminals, display devices, smart speakers, and image acquisition devices. Mobile terminals include but are not limited to mobile phones, computers, and tablet computers.
[0056] In the embodiment of the present application, the display device 200 generally refers to a device with image display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart TV, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, etc.
[0057] Figure 1 This is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application. Figure 1 As shown in FIG. 1 , a user can operate the display device 200 through touch operation, the mobile terminal 300 and the control device 100. The control device 100 is used to receive operation instructions input by the user and convert the operation instructions into control instructions that can be recognized and responded to by the display device 200. For example, the control device 100 can be a remote controller, a stylus pen, a handle, etc.
[0058] The mobile terminal 300 can be used as a control device for performing human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device for establishing a communication connection with the display device 200 and performing data interaction. In some embodiments, the mobile terminal 300 can install software applications with the display device 200, and achieve connection and communication through a network communication protocol to achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300 can also be transmitted to the display device 200 to achieve a synchronous display function.
[0059] In some embodiments, the mobile terminal 300 or other electronic devices may also simulate the functions of the control device 100 by running an application program for controlling the display device 200 .
[0060] like Figure 1 As also shown in FIG. 4 , the display device 200 also communicates data with the server 400 through various communication methods. The display device 200 may be allowed to communicate and connect through a local area network (LAN), a wireless local area network (WLAN), and other networks.
[0061] The display device 200 may provide a broadcast receiving television function, and may also additionally provide an intelligent network television function with a computer support function, including but not limited to network television, smart television, Internet Protocol television (IPTV), and the like.
[0062] Figure 2 Some embodiments of the present application provide Figure 1 2 is a block diagram of the hardware configuration of the display device 200.
[0063] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.
[0064] In some embodiments, the detector 230 is used to collect signals of the external environment or external interaction. For example, the detector 230 includes a light receiver, a sensor for collecting the intensity of ambient light; or, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or, the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.
[0065] In some embodiments, the display 260 includes a display function component for presenting a picture, and a driving component for driving an image display. The display 260 is used to receive an image signal output from the controller 250 for display. For example, the display 260 can be used to display video content, image content, and components of a menu control interface and a user control UI interface.
[0066] In some embodiments, the communication device 220 is a component for communicating with an external device or server 400 according to various communication protocol types. The display device 200 may be provided with a plurality of communication devices 220 according to different supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 may be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.
[0067] The communication device 220 can enable the display device 200 to communicate with the external device or server 400 by wireless or wired connection. Among them, the wired connection can connect the display device 200 with the external device through components such as data cables and interfaces. The wireless connection can connect the display device 200 with the external device through wireless signals or wireless networks. The display device 200 can establish a connection relationship with the external device directly, or indirectly establish a connection relationship through a gateway, a router, a connection device, etc.
[0068] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first interface to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.
[0069] In some embodiments, the controller 250 and the tuner-demodulator 210 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.
[0070] In some embodiments, the user may input a user command through a graphical user interface (GUI) displayed on the display 260 , and the user input interface receives the user input command through the graphical user interface (GUI).
[0071] In some embodiments, the audio output device 270 may be a local speaker of the display device 200, or may be an external audio output device of the display device 200. In particular, for the external audio output device of the display device 200, the display device 200 may also be provided with an external audio output terminal, and the audio output device may be connected to the display device 200 through the external audio output terminal to output the sound of the display device 200.
[0072] In some embodiments, the user input interface 280 may be used to receive instructions from a user.
[0073] In order to perform user interaction, in some embodiments, the display device 200 may run an operating system. The operating system is a computer program for managing and controlling hardware resources and software resources in the display device 200. The operating system can control the display device to provide a user interface, for example, the operating system can directly control the display device to provide a user interface, or can provide a user interface by running an application program. The operating system also allows the user to interact with the display device 200.
[0074] It should be noted that the operating system may be a native operating system based on a specific operating platform, or a third-party operating system deeply customized based on a specific operating platform, or an independent operating system specially developed for the display device.
[0075] The operating system can be divided into different modules or layers according to the functions implemented, such as Figure 3 As shown, in some embodiments, the system is divided into four layers, from top to bottom, namely, the application layer (Applications) layer (referred to as "application layer"), the application framework layer (Application Framework) layer (referred to as "framework layer"), the system library layer and the kernel layer.
[0076] In some embodiments, the application layer is used to provide services and interfaces for applications so that the display device 200 can run applications and interact with users based on the applications. At least one application can be run in the application layer, and these applications can be window programs, system settings programs, clock programs, etc. that come with the operating system; they can also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the above examples.
[0077] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications. The application framework layer includes some predefined functions. The application framework layer is equivalent to a processing center that determines the actions that applications in the application layer take. Through the API interface, applications can access system resources and obtain system services during execution.
[0078] like Figure 3 As shown, in the embodiment of the present application, the application framework layer includes a view system, managers, content providers, etc., wherein the view system can design and implement the interface and interaction of the application, and the view system includes lists, grids, text boxes, buttons, etc. The manager includes at least one of the following modules: an activity manager for interacting with all activities running in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to the application package currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.
[0079] In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application. The window manager is used to manage all window programs, such as obtaining the display screen size, determining whether there is a status bar, locking the screen, capturing the screen, and controlling the display window changes, for example, reducing the display window, shaking the display, distorting the display, etc.
[0080] In some embodiments, the system runtime layer can provide support for the framework layer. When the framework layer is used, the operating system will run the instruction library contained in the system runtime layer, such as the C / C++ instruction library, to implement the functions to be implemented by the framework layer.
[0081] In some embodiments, the kernel layer is a functional layer between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. Figure 3 As shown, the kernel layer may be configured with hardware drivers, and the drivers included in the kernel layer may be at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.
[0082] It should be noted that the above example is only a simple division of the operating system functions and does not constitute a limitation on the specific operating system form of the display device 200 in the embodiment of the present application. Depending on factors such as the function of the display device and the type of operating system, the number of levels and specific level types contained in the operating system may be expressed in other forms.
[0083] With the continuous development of science and technology, artificial intelligence technology has been widely used in various fields, especially in the field of home appliance control, the application of artificial intelligence technology has become a trend.
[0084] In some embodiments, most home appliance control systems are still based on programmed control strategies, and whole-house device control is achieved through internally designed integrated control commands. The programmed control strategy includes presetting trigger conditions corresponding to multiple electronic device / smart home device control instructions. For example: the trigger condition is that the user is in the corridor, and the control instruction is to turn on the bedroom lights and the corridor lights. When the camera captures the user appearing in the corridor, the instructions to turn on the bedroom lights and the corridor lights are executed. However, this strategy requires users to preset in advance, which often cannot meet the changing needs of users, and when faced with complex environments and tasks, its control effect often cannot meet expectations.
[0085] In some embodiments, some home appliance control systems adopt state and intention recognition detection schemes. The intention recognition detection scheme is to perform operations corresponding to the intention by identifying the intention of the user input command. However, intention recognition can only perform a single operation with clear instructions. For example: when the user inputs a voice command to start the movie viewing mode, it may not be clear what operation the command corresponds to, or it may just execute the operation of turning on the TV. The state detection scheme is to achieve decision-making control of home appliances through state threshold point detection. For example: the camera can capture the image of the user's indoor floor in real time. When it is detected that the indoor floor image collected when there is no object moving is different from the preset indoor clean floor image, a start command is sent to the sweeping robot to make the sweeping robot start cleaning the floor. However, this solution can only detect and execute modularly in different areas, and cannot truly achieve integrated decision-making control for the whole house.
[0086] In order to solve the above technical problems, the present application embodiment provides a terminal device, and the present embodiment further improves some functions of the terminal device. Figure 4 As shown, the controller (at least one processor) enables the terminal device to perform the following steps by running the application:
[0087] Step S401: receiving a control task;
[0088] Control tasks include wake-up tasks, scheduled tasks, and inspection tasks. Wake-up tasks are immediate tasks for users, which immediately wake up the big model to make decisions, such as turning on the water heater, opening the curtains, etc. Scheduled tasks are set time period tasks, which require waking up the big model at a specific time to make decisions, such as feeding cat food at 8 o'clock every morning, and the big model decides how much cat food to add. Inspection tasks are daily monitoring tasks, which require monitoring equipment and sensor equipment to cooperate with the big model to make decisions, such as sweeping tasks. When the monitoring equipment finds abnormalities on the ground, it will wake up the big model and feedback information. The big model decides whether to start the sweeping robot for cleaning.
[0089] In some embodiments, the step of receiving a control task includes:
[0090] Receiving voice data input by a user;
[0091] In some embodiments, the step of receiving voice data input by a user includes:
[0092] Real-time monitoring of environmental sounds;
[0093] Determine whether the ambient sound is greater than a preset volume;
[0094] If the ambient sound is not greater than the preset volume, continue to collect the ambient sound;
[0095] If the ambient sound is greater than a preset volume, the sound collector is controlled to collect audio data.
[0096] Determining whether the audio data includes a wake-up keyword;
[0097] If the audio data does not include the wake-up keyword, continue to monitor the ambient sound;
[0098] If the audio data includes a wake-up keyword, a voice function wake-up page is displayed, a voice function prompt tone is played, and a sound collector is controlled to collect voice data.
[0099] In some embodiments, the step of receiving voice data input by a user includes:
[0100] The voice data sent by the control device is received, wherein the voice data is collected by the control device after the user presses a voice key of the control device and sent to the terminal device.
[0101] In some embodiments, the step of receiving voice data input by a user includes:
[0102] In response to a user input, the electronic device control application is opened, and the display is controlled to display a home page of the control application, wherein the home page includes an open session window control;
[0103] In response to a user inputting an instruction to select the open conversation window control, controlling the display to display a conversation window, wherein the conversation window includes a voice input control;
[0104] In response to the user inputting an instruction to select the input voice control, the sound collector is controlled to collect the voice data input by the user.
[0105] Identify the voice text corresponding to the voice data;
[0106] In some embodiments, a pre-processing model of input data is provided inside the terminal device, and the pre-processing model is used to process the data input by the user, that is, the control task, and process the data input by the user into structured data, and the structured data is input data that can be recognized by the large model. The pre-processing model includes a speech recognition model, and the speech data can be recognized by the speech recognition model to obtain speech text. The speech recognition model provided inside the terminal can protect user privacy, has a fast recognition speed, and can still be used when the network is disconnected.
[0107] In some embodiments, the multimodal large model has speech recognition capabilities, can recognize speech data as speech text, and can directly input the speech data into the multimodal large model.
[0108] In some embodiments, the voice data is sent to a server so that the server recognizes the voice text corresponding to the voice data through voice recognition technology and sends the voice text to the terminal device.
[0109] In some embodiments, the control task is determined to be the voice text, that is, prompt instructions can be constructed based on the voice text later.
[0110] In some embodiments, after acquiring the voice text, determining whether a control instruction corresponding to the voice text is matched;
[0111] The control instructions include power on / off instructions, terminal device parameter adjustment instructions, and terminal device installed application instructions. If the similarity between the voice text and the setting text corresponding to the control instruction reaches a preset percentage, it can be determined that the voice text matches the control instruction. If the similarity between the voice text and the setting text corresponding to the control instruction does not reach a preset percentage, it can be determined that the voice text does not match the control instruction.
[0112] If a control instruction corresponding to the voice text is matched, the operation corresponding to the control instruction is executed.
[0113] Exemplarily, if the voice text is shutdown, a shutdown instruction is matched and the shutdown operation is performed.
[0114] If no control instruction corresponding to the voice text is matched, the voice text is determined to be a control task.
[0115] If no control instruction corresponding to the voice text is matched, the control application of the electronic device is called, the voice text or voice data is sent to the control application, and the control application constructs a prompt instruction based on the voice text or voice data.
[0116] In some embodiments, the step of receiving a control task includes:
[0117] Receive a timed control task set by a user, wherein the timed control task includes a timed time and a control task;
[0118] The control task and its corresponding time can be input in the conversation window, and the voice data input by the user can also be recognized, and the voice text includes the scheduled time and the control task. For example, the input text or voice text is: listen to music in the living room at 8 o'clock tomorrow morning.
[0119] When the timing time is reached, the control task is received.
[0120] After the timing time is reached, the control application of the electronic device is called, the voice text or the input text is sent to the control application, and the control application constructs a prompt instruction based on the voice text or the input text.
[0121] In some embodiments, the step of receiving a control task includes:
[0122] Receive image data or abnormal message sent by an electronic device, wherein the image data is an image taken by the electronic device, and the abnormal message is a message sent by the electronic device after detecting an abnormal condition, and the abnormal message includes abnormal data.
[0123] The electronic device has an image capture function and can send the real-time captured image to the terminal device, and the control application of the terminal device processes the image data. For example, the camera can send the captured image to the terminal device, and the terminal device constructs a prompt instruction based on the image data. Among them, the multimodal large model has image recognition capabilities and can directly input image data into the multimodal large model.
[0124] The electronic device has the ability to detect abnormal states or specific states, and can send abnormal state or specific state messages to the terminal device. The control application of the terminal device processes the abnormal state or specific state messages. For example, after the monitoring device identifies the ground abnormality, it can send a ground abnormality message to the terminal device, wherein the ground abnormality message includes an abnormal image, and the terminal device constructs a prompt instruction based on the ground abnormality message. For example, after the smart door lock detects that someone is standing at the door, it sends a message that there is someone at the door to the terminal device, and the message that there is someone at the door includes an image of the person at the door, and the terminal device constructs a prompt instruction based on the message that there is someone at the door.
[0125] In the embodiment of the present application, the electronic device determines abnormal situations and special situations by itself, reduces the burden of data processing on the terminal device, and sends abnormal data to the terminal device, and the multimodal large model of the terminal device decides how to handle it.
[0126] In some embodiments, the step of receiving a control task includes:
[0127] Receiving image data sent by an electronic device;
[0128] Identify the image data to obtain image description information;
[0129] In some embodiments, a pre-processing model of input data is provided inside the terminal device, and the pre-processing model is used to process the data sent by the electronic device, that is, the control task, and process the data sent by the electronic device into structured data, and the structured data is input data that can be recognized by the large model. The pre-processing model includes an image recognition model, and the image data can be recognized by the image recognition model to obtain image description information. The image recognition model provided inside the terminal can protect user privacy, has a fast recognition speed, and can still be used when the network is disconnected.
[0130] In some embodiments, the image data is sent to a server so that the server recognizes the image data through an image recognition model, obtains image description information, and sends the image description information to a terminal device.
[0131] The control task is determined to be the image description information.
[0132] Step S402: constructing prompt instructions based on the control task;
[0133] In some embodiments, the control task includes text, voice, and / or images.
[0134] In some embodiments, the control task is input as structured data through pre-processing models such as a speech recognition model and an image processing model.
[0135] Prompt instructions refer to the input information provided by users to the multimodal large model, which usually includes keywords, questions, and instructions. Prompt instructions can guide the multimodal large model to generate high-quality, accurate, and targeted responses.
[0136] Exemplarily, the prompt instructions are as follows:
[0137] "You are an AI assistant that can control various home appliances. In each step, you need to provide your thoughts to analyze the current status and decide on the next action. Your output should follow the following format:\nThinking:\nAction:\nAction output:\nAfter executing the action command, you will get feedback from the home appliances. Then, you need to analyze your status based on this feedback and decide on the next action.\nAfter a series of decisions and executions, if all devices are operating normally, you need to output:\nFinish: Task completed, if a device returns an exception and all alternative solutions fail to solve the problem, you need to output:\nFinish: XX device is abnormal, please check the device status and re-enter the task.\nThe task you are currently performing belongs to {{Task Category}}, Task Description: {{Input Task}},\nLet's get started."
[0138] It should be noted that the task classification can be determined according to the source of the task. For example, if the voice service calls the control service / multimodal large model, the task classification can be determined as a wake-up task. If the timed service calls the control service / multimodal large model, the task classification can be determined as a timed task. If the electronic device directly calls the control service / multimodal large model by sending data, the task classification can be determined as an inspection task.
[0139] In some embodiments, the step of constructing a prompt instruction based on the control task includes:
[0140] A prompt instruction is constructed based on the control task and user preference information, wherein the user preference information includes device parameters of the electronic device related to the control task, and the device parameters include the device parameters of the electronic device most frequently used by the user within a preset time.
[0141] In some embodiments, the electronic device may count the device parameters that are most frequently used by the user during a preset time, and send the device parameters to the terminal device, which stores them in the user preference information.
[0142] In some embodiments, the electronic device may send device parameters of the electronic device used by the user to the terminal device, and the terminal device collects statistics on the device parameters of the electronic device used most frequently, and records them in the user preference information.
[0143] For example, it can be statistically determined that in the past week, the user has set the TV volume to 8 most frequently, so the user likes the information: TV volume is 8. It can also be statistically determined that in the past week, the user has set the bedroom light brightness to 20 most frequently during the period from 21:00 to 00:00, so the user likes the information: 21:00-00:00-bedroom light brightness is 20.
[0144] Exemplarily, the prompt instructions are as follows:
[0145] "You are an AI assistant that can control various home appliances. In each step, you need to provide your thoughts to analyze the current status and decide on the next action. ……\nThe task you are currently performing belongs to {{Task Category}}. Before making a decision, please refer to some user preferences {{User Preference Information}}. Task Description: {{Enter Task}}, \nLet's get started."
[0146] In some embodiments, the user preference information in the prompt instruction includes preference information related to all electronic devices.
[0147] In some embodiments, the user preference information in the prompt instruction includes preference information of electronic devices related to the control task. A mapping relationship between keywords and electronic devices can be established, and the electronic devices related to the task can be determined based on the keywords. Specifically, it includes:
[0148] Filter out the keywords of the control task;
[0149] Determine electronic devices related to the control task based on keywords;
[0150] Acquiring user preference information related to the electronic device;
[0151] A prompt instruction is constructed based on the control task and the user preference information.
[0152] For example, the control task is "starting the movie watching mode". The keyword is "movie watching", and the electronic devices related to movie watching include televisions, audio equipment and smart lamps. The user preference information related to the television, audio equipment and smart lamps can be constructed into prompt instructions.
[0153] A mapping relationship between the sending device of the control task and the electronic device can be established. For example, after the camera sends image data to the terminal device, the terminal device may turn on the bedroom light or notify the sweeping robot to clean the floor based on the image data. Therefore, the camera of the sending device of the control task forms a mapping relationship with the bedroom light and the sweeping robot. The electronic devices related to this task can be determined based on the sending device of the control task. Specifically, it includes:
[0154] Determine the electronic device to be selected according to the sending device of the control task;
[0155] Acquiring user preference information related to the selected electronic device;
[0156] A prompt instruction is constructed based on the control task and the user preference information.
[0157] Exemplarily, the sending device of the control task is a camera, and the electronic devices related to the camera include a sweeping robot and a smart lamp, etc. The user preference information related to the sweeping robot and the smart lamp can be constructed into a prompt instruction.
[0158] Compared with the solution of controlling home appliances through written programs, the embodiment of the present application does not require users to participate in complex configurations. Instead, the user's preferences are recorded through a series of interactions, such as the volume, color and brightness of the lights, etc. Before the next decision, the task and preferences are simultaneously input into the multimodal big model, allowing the multimodal big model to make decisions that are more biased towards the user without the need for additional configuration.
[0159] In some embodiments, the step of constructing a prompt instruction based on the control task includes:
[0160] A prompt instruction is constructed based on the control task and a function description of a target electronic device, wherein the target electronic device is an electronic device that does not appear in the data set, and the function description of the target electronic device is manually added by a user after establishing a connection between the target electronic device and a terminal device.
[0161] It should be noted that in order for a terminal device to control an electronic device, the terminal device and the electronic device need to be in the same local area network and establish a communication connection, or the terminal device and the electronic device need to be bound to the same family. The terminal device can send control instructions to the electronic device through the server, and the electronic device can also send the instruction execution results and status back to the terminal device through the server.
[0162] After receiving a user input command to open the control application, the terminal device can display the home page of the control application, the home page includes a device list and an add device control, the device list includes devices that have established a communication connection with the terminal device. The devices in the device list can be controlled by the terminal device.
[0163] After receiving the user input instruction to select to add a device control, the terminal device displays the device control corresponding to the device that is in the same local area network as the terminal device and is not in the device list;
[0164] After receiving the user input instruction to select the device control, if the device name corresponding to the device control is a known device, the device corresponding to the device control is directly added to the device list. The known device is an existing electronic device in the electronic device control field data set.
[0165] If the device name corresponding to the device control is an unknown device, that is, the target electronic device, the control display displays a prompt message, wherein the prompt message is used to prompt the user that the device is an unknown device, and if the user wants to control the device, the function description of the device needs to be added, and the user is guided to fill in the device function description using the device manual.
[0166] For example, Figure 5 As shown, the home page of the control application includes a device list 51 and an add device control 52. After receiving the user's instruction to select the add device control 52, the add device page is displayed. Figure 6 As shown, the add device page includes a smart speaker control 61 and a sweeping robot control 62. After receiving the user's selection of the smart speaker control 61, since the smart speaker is a known device, the home page may be displayed as follows Figure 7 After receiving the user's selection of the sweeping robot control 62, since the sweeping robot is an unknown device, a prompt page may be displayed, such as Figure 8 The prompt page includes a yes control 81 indicating that the function description should be filled in and a no control 82 indicating that the function description should not be filled in. After receiving the user input and selecting the yes control 81, the function description filling page is displayed, as shown in FIG. Fig. 9 As shown. Users can fill in the relevant content according to the prompts on the device manual and function description page, so that the terminal device can control unknown devices.
[0167] After obtaining the function description of the target electronic device, a prompt instruction can be constructed. For example, the prompt instruction is as follows:
[0168] "You are an AI assistant that can control various home appliances. At each step, you need to provide your thinking to analyze the current status and decide on the next action. ...\nThe task you are currently performing belongs to {{Task Category}}. In addition to the home appliances you know, you can also control the following devices: [Function description of new device 1, ...]. Task description: {{Input task}}, \nLet's get started."
[0169] Step S403: inputting the prompt instruction into the multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device;
[0170] Among them, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking about instruction intentions, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameters.
[0171] Write a standard function description for the electronic device. The function description is in JSON format, including the device name, device function description, the name and description of each input parameter of the device, and the name of the required input parameter.
[0172] There are many smart home appliances, and the instructions and control methods are different. In order to standardize the use and improve the efficiency of decision-making and generation planning of home appliances by multimodal large models, each home appliance is described using json functions to introduce the functions and input parameters of the device. The format is as follows: {"name":"Device name","description":"Device function description","parameters":{"Device input parameter name 1":{"type":"Parameter type","description":"Device input parameter description"}},"required":["Required parameter name 1",]}. The field "name" is the device name, the field "description" is the specific function description of the device, and the field "parameters" is a dictionary that stores all the input parameter information of the device. The key of the dictionary is the name of the parameter. The field "required" is a list, which contains the names of the parameters that must be passed in and the key values of "parameters". If there are no required parameters, "required" is an empty list. If the required parameters are missing, the relevant operations cannot be performed.
[0173] For example, the function description of the smart lamp is as follows:
[0174] "Device Name":"Smart Lighting",
[0175] "Device function description": "Lighting that can be remotely controlled through a mobile phone APP or voice assistant, with functions such as adjusting brightness, color, and switching";
[0176] "dictionary":
[0177] "brightness":"type(parameter type)":"integer(integer)",
[0178] "description(device input parameter description)":"Used to adjust the brightness of the lamp, the value range is 0-100";
[0179] "color":"type":"string(string)",
[0180] "description":"Used to adjust the color of lamps. The selectable colors include red, orange, yellow, green, blue, indigo, purple, etc.";
[0181] "power":"type":"boolean",
[0182] "description":"Used to control the switch of the lamp, true means on, false means off";
[0183] "required":["Power supply"].
[0184] Clarify the scope of the field under the intelligent home appliance control scenario, collect various home appliance control-related data and information through various channels, and use multimodal large models to organize, annotate, verify, and expand the data to build a home appliance control field data set. In order to ensure the quality of the data, manual verification and annotation are also required.
[0185] The dataset is constructed in the form of ChatML (Chat Markup Language) dialogues, for example:
[0186] {"task":"Start viewing mode",
[0187] "conversations":
[0188] [{"role":"system";"content":"You are an AI assistant that can control various home appliances. In each step, you need to provide your thoughts to analyze the current status and decide on the next action. Your output should follow the following format:\nThinking:\nAction:\nAction output:\nAfter executing the action command, you will get feedback from the home appliances. Then, you need to analyze your status based on this feedback and decide on the next action.\nAfter a series of decisions and executions, if all devices are operating normally, you need to output:\nFinish: The task is completed. If a device returns an exception and all alternative solutions fail to solve the problem, you need to output:\nFinish: XX device is abnormal. Please check the device status and re-enter the task."},
[0189] {"role":"user";"content":"The task you are currently executing belongs to {{wake-up task}}. In addition to the home appliances you already know [smart lamps, smart TVs, smart speakers, ...], you can also control the following devices: [device 1, device 2, ...]. Task description: {{Start viewing mode}}, \nLet's get started."},
[0190] {"role":"assistant";"content":"Thinking: First, I need to understand the meaning of "starting movie mode". In movie mode, you usually need to dim the lights, start the smart TV, connect the smart speaker to the TV, and adjust the volume to an appropriate level. Next, I will control the smart lamp and adjust its brightness to the appropriate brightness in movie mode.\nAction: Smart lamp\nAction output: {"brightness":5, "power":true}"},
[0191] {"role":"user";"content":"{"device_status":"normal","response":adjust the brightness to 5}"},
[0192] {"role":"assistant";"content":"Thinking: The light dimming is successful. Next, I will turn on the smart TV and adjust the channel to the movie.\nAction: Smart TV\nAction output:{"channel":"movie","power":true}"},
[0193] {"role":"user";"content":"{"device_status":"Normal","response":"The TV is turned on and has been adjusted to the movie channel}"},
[0194] {"role":"assistant";"content":"Thinking: Movie mode requires connecting to an audio device. Next, I will turn on the smart speaker, connect it to the smart TV, and set the volume to an appropriate level.\nAction: Smart speaker\nAction output: {"connectDevice":"TV","volume":40}"},
[0195] {"role":"user";"content":"{"device_status":"Normal","response":Connected to TV, volume adjusted to 40}"},
[0196] {"role":"assistant";"content":"Thinking: I have turned on the TV, and the light and volume are controlled to the appropriate level.\nAction: Finish\nAction output: Task completed"},]}.
[0197] In the above example, "task" is a specific task, "conversations" is the training conversation data of the multimodal large model, "role" is the role label, "system" is the system description responsible for defining the model and making clear requirements, "user" is the input information, and "assistant" is the output of the multimodal large model. In the first user-content, "known home appliances: [smart lamps, smart TVs, smart speakers]", the smart lamps, smart TVs, and smart speakers here have appeared in the training data, so only the names are listed. If unknown devices appear in actual applications, you need to replace [device 1, device 2, ...] in the list with the json format data in the function description of the home appliances given in the previous part.
[0198] The response of the large model in the dataset consists of three parts: thinking about the command intent and analyzing subsequent actions; outputting the action name, that is, the device name; and generating the action input, that is, the device input parameter, in JSON format.
[0199] According to the constructed ChatML dialogue format dataset, the multimodal large model is fine-tuned. During the fine-tuning process, a variety of methods can be flexibly selected, such as full parameter fine-tuning (full) and efficient fine-tuning methods LoRA, Freeze, Galore, etc. The embodiment of this application does not limit the fine-tuning method. With the help of the dataset, the multimodal large model learns the structured input and output of home appliance control data, and strengthens the multimodal large model's understanding of home appliance control tasks and decision-making planning capabilities.
[0200] Input prompt instructions into the multimodal big model, and the big model outputs thinking (status analysis and planning ideas), actions (device names), and action outputs (device input parameters). The output is structured data, which is convenient for control application analysis.
[0201] In some embodiments, the function description of the electronic device used by the multimodal large model is the function description of all electronic devices included in the training set.
[0202] In some embodiments, the function description of the electronic device used by the multimodal large model is a function description of the electronic device that establishes a communication connection with the terminal device. The embodiment of the present application can make the output of the multimodal large model more suitable for the electronic device used by the user.
[0203] Step S404: start the electronic device corresponding to the device name and send the parameters to the electronic device, so that the electronic device executes the control operation corresponding to the parameters.
[0204] Parse the device name and device input parameters (device input parameters) output by the multimodal large model, start the electronic device corresponding to the device name and send the parameters to the electronic device.
[0205] After receiving the parameters, the electronic device executes the control operation corresponding to the parameters. The execution result and status are fed back to the terminal device. The feedback information is also in JSON format, containing two fields: device status (device_status) and execution result (response). When the device is normal, the data corresponding to the device status field is "normal", and the data corresponding to the execution result field is the parameters after execution; when the device is abnormal, the data corresponding to the device status field is "error", and the data corresponding to the execution result field is the cause of the device abnormality.
[0206] After receiving the execution result and status sent by the electronic device, the multimodal large model determines whether to continue to execute the control task.
[0207] When receiving the normal status of the device, determine whether the control task is completed;
[0208] If the control task has been completed, there is no need to continue executing the control task;
[0209] If the control task is not completed, continue to execute the control task.
[0210] When receiving the abnormal device status, it is determined whether there is an unexecuted alternative solution, which is an alternative solution for the operation after the abnormal device status. For example, when a TV is connected to two speakers, when one speaker is not working well, the alternative solution is to enable the other speaker.
[0211] If there is an alternative that has not been executed, then continue to execute the control task;
[0212] If there are no unexecuted alternatives, there is no need to continue with the control task.
[0213] If the task continues to be executed, the multimodal large model continues to output thoughts, actions, and action outputs based on the prompt instructions, the function description of the electronic device, and the execution results and status of the control operation.
[0214] If the device is abnormal and all alternative solutions fail to solve the problem, the output is: \n Finish: XX device is abnormal, the reason for the abnormality is xxx, please check the device status and re-enter the task, and terminate the program.
[0215] If the task is completed, Finish is output: the task is completed and the program is terminated.
[0216] Exemplarily, a scheduled task is to wake up the control application / multimodal big model at a specific time to make a decision. For example, a scheduled task is to feed the cat at 8 o'clock every morning. The multimodal big model is woken up at 8 o'clock in the morning. The multimodal big model calls the feeder to detect the remaining cat food, and decides how much cat food to add based on the cat's health or the cat's recent food intake. The amount of cat food to be added is sent to the feeder so that the feeder can perform the operation of adding cat food. This type of task does not require the user to enter complex information, only the time and specific tasks need to be entered, and it is executed on a scheduled basis.
[0217] Exemplarily, the inspection task is a daily monitoring task used to handle emergencies, which requires monitoring equipment and sensor equipment to cooperate with the multimodal large model to make decisions. For example: a sweeping task, when the monitoring equipment finds an abnormality on the ground, it wakes up the large model and sends feedback information to the multimodal large model. The multimodal large model decides whether to start the sweeping robot for cleaning. Another example: face recognition of a smart door lock, when it detects someone standing at the door, it will wake up the multimodal large model. The multimodal large model identifies the person based on the user's album of relatives and friends. If the person is in the first sequence set by the user, the door will be opened directly to welcome him. Otherwise, the user will be notified who is at the door and asked whether to open the door. This type of task requires the user to give sufficient permissions to obtain real-time data from monitoring equipment and sensor equipment, and wake up the multimodal large model when an abnormal situation occurs to solve the emergency.
[0218] Exemplarily, the wake-up task is an immediate task for the user, which immediately wakes up the multimodal big model to make a decision. It can be a simple task, such as turning on the water heater, opening the curtains, etc. It can be a complex task, for example, the user is watching a TV series in the living room, and the user wants to switch to the bedroom to watch it. The multimodal big model decides to turn on the bedroom viewing mode, and at the same time transfers the episodes of the living room TV to the bedroom TV, so that the user can continue to watch the TV series seamlessly. This will also be the task that users use the most and interact with the most frequently, which can meet the various needs of users.
[0219] In some embodiments, Fig.10As shown, a control task is received, and the control task includes a wake-up task, a timing task, and an inspection task. A prompt instruction is constructed based on the control task, user preference information, and the function description of the unknown device, and the prompt instruction is input into the fine-tuned multimodal large model. Among them, the multimodal large model is fine-tuned using a data set in the field of electronic device control. The multimodal large model outputs planning ideas, device names, and device input parameters. Parse the device name and device input parameters. Start the electronic device corresponding to the device name and pass the parameters to the device. After the electronic device executes the operation, the execution result and status are sent to the multimodal large model. The multimodal large model determines whether to continue to execute the task. If the task is to be continued, the planning ideas, device name, and device input parameters are output again; if the task is not to be continued, the task completion or the reason for the device abnormality is output.
[0220] The multimodal big model has powerful natural language processing capabilities, dialogue generation, command following, text and image understanding, and logical reasoning. Applying the multimodal big model to the field of home appliance control not only obtains good home appliance decision-making and planning capabilities, but also outputs structured responses according to user needs, making it easier for home appliances to parse operating instructions and provide direction for the integrated smart home appliance autonomous control system.
[0221] This application writes a standardized function description for each home appliance, including the functional description and input parameter description of the device, so that the multimodal large model can determine the need to use the device and generate control input parameters. When an instruction is passed in, the multimodal large model will consider the instruction intent and analyze subsequent actions, output the action name, that is, the device name, and generate the action input, that is, the device input parameter. When encountering complex intentions, the multimodal large model will plan each device action step by step, and will also modify the planned route in time according to the device feedback.
[0222] Some embodiments of the present application provide an electronic device control method, the method is applicable to a terminal device, the terminal device includes a controller, the controller is configured to: receive a control task; construct a prompt instruction based on the control task; input the prompt instruction into a multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter; start the electronic device corresponding to the device name and send the parameter to the electronic device, so that the electronic device performs the control operation corresponding to the parameter. The embodiment of the present application uses a standardized function to describe the function and input parameter of the electronic device, improves the efficiency of the multimodal large model decision and generation planning, makes the electronic device control more convenient, and the multimodal large model autonomously and flexibly controls multiple electronic devices to achieve integrated decision control.
[0223] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0224] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. A terminal device, characterized in that: include: The controller is configured as: Receive control tasks; constructing prompt instructions based on the control task; Input the prompt instruction into the multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter; The electronic device corresponding to the device name is started and the parameters are sent to the electronic device, so that the electronic device performs the control operation corresponding to the parameters.
2. The terminal device according to claim 1, characterized in that: The controller performs a receiving control task and is further configured to: Receiving voice data input by a user; Identify the voice text corresponding to the voice data; If no control instruction corresponding to the voice text is matched, it is determined that the voice text is a control task.
3. The terminal device according to claim 1, characterized in that: The controller performs a receiving control task and is further configured to: Receive a timed control task set by a user, wherein the timed control task includes a timed time and a control task; When the timing time is reached, the control task is received.
4. The terminal device according to claim 1, characterized in that: The controller performs a receiving control task and is further configured to: Receive image data or abnormal message sent by an electronic device, wherein the image data is an image taken by the electronic device, and the abnormal message is a message sent by the electronic device after detecting an abnormal condition, and the abnormal message includes abnormal data.
5. The terminal device according to claim 1, characterized in that: The controller performs a receiving control task and is further configured to: Receiving image data sent by an electronic device; Identify the image data to obtain image description information; The control task is determined to be the image description information.
6. The terminal device according to claim 1, characterized in that: The controller executes a prompt instruction based on the control task, and is further configured to: A prompt instruction is constructed based on the control task and user preference information, wherein the user preference information includes device parameters of the electronic device related to the control task, and the device parameters include the device parameters of the electronic device most frequently used by the user within a preset time.
7. The terminal device according to claim 1, characterized in that: The controller executes a prompt instruction based on the control task, and is further configured to: A prompt instruction is constructed based on the control task and a function description of a target electronic device, wherein the target electronic device is an electronic device that does not appear in the data set, and the function description of the target electronic device is manually added by a user after establishing a connection between the target electronic device and a terminal device.
8. The terminal device according to claim 1, characterized in that: After sending the parameters to the electronic device, the controller is further configured to: The electronic device is controlled to feed back the execution result and status of the control operation to the multimodal large model, so that the multimodal large model continues to output thoughts, actions and action outputs until the control task is completed or an abnormality occurs in the control task.
9. A method for controlling an electronic device, characterized in that: include: Receive control tasks; constructing prompt instructions based on the control task; Input the prompt instruction into the multimodal large model, so that the multimodal large model outputs thinking, action and action output based on the prompt instruction and the function description of the electronic device, the multimodal large model is trained based on a data set in the field of electronic device control, the data set includes dialogue format data related to electronic device control, the function description includes the device name, function description, input parameter name and input parameter description of the electronic device, the thinking includes thinking instruction intention, state analysis and planning ideas, the action includes the device name, and the action output includes the device input parameter; The electronic device corresponding to the device name is started and the parameters are sent to the electronic device, so that the electronic device performs the control operation corresponding to the parameters.
10. The method according to claim 9, characterized in that The step of receiving the control task includes: Receiving voice data input by a user; Identify the voice text corresponding to the voice data; If no control instruction corresponding to the voice text is matched, it is determined that the voice text is a control task.
