A method for controlling home appliances based on command words, and a device for implementing the same.

Abstract

The present invention relates to a method for controlling home appliances based on command words, and an apparatus for implementing the same. [Solution] A method for controlling a home appliance based on a command word according to one embodiment of the present invention includes: a first step of checking whether the command word input to the electronic device corresponds to direct control; a second step in which, if the command word input in the first step does not correspond to direct control, the electronic device inputs, along with the command word, an example sentence extracted from a guide and database containing categories for the command word, which has a similarity to the input command word of a certain standard or higher, to a generative AI module; and a third step in which the result calculated via the generative AI module includes the category of the input command word, and a control command is generated using the calculated result.

Description

Technical Field

[0001] The present invention relates to a method for controlling home appliances based on command words and an apparatus for embodying the same.

Background Art

[0002] Control methods for devices such as home appliances can be classified into a method of direct human operation control and a method of control via natural language commands. In the case of direct operation, the user can control the operation of home appliances by operating a remote control, buttons on the home appliance, a dial, etc. In the case of control via natural language commands, when the user inputs a natural language command word for the home appliance to the home appliance, the home appliance recognizes and operates based on this.

[0003] However, since natural language command words spoken or input by the user are based on different natural languages for each person, it is very difficult to interpret them and change them into actual commands. In particular, accuracy is required for the interpretation of natural language command words depending on each user's past command word input habits and the usage environment of the home appliance.

[0004] Therefore, this specification intends to describe a method and an apparatus for interpreting the intention from the command words spoken by the user, calculating corresponding control commands, and using them to control home appliances.

Summary of the Invention

Problems to be Solved by the Invention

[0005] This specification is for solving the above-described problems, and aims to improve speech recognition errors and flexibly recognize the commands of the user.

[0006] In addition, this specification enables various controls to be operated at once when processing the command words of the user.

[0007] Furthermore, this specification enables users to control home appliances via voice recognition even when they do not use prescribed commands.

[0008] The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned can be understood from the following description and more clearly from the embodiments of the present invention. Furthermore, it is readily apparent that the objectives and advantages of the present invention can be achieved by the means and combinations thereof described in the claims. [Means for solving the problem]

[0009] A method for controlling a home appliance based on a command word according to one embodiment of the present invention includes: a first step of checking whether a command word input to an electronic device corresponds to direct control; a second step in which, if the command word input in the first step does not correspond to direct control, the electronic device inputs, along with the command word, an example sentence extracted from a guide and database containing categories for the command word, which has a similarity to the input command word above a certain standard, to a generative AI module; and a third step in which the result calculated via the generative AI module includes the category of the input command word, and the calculated result is used to generate a control command.

[0010] A server that controls a home appliance based on a command word according to one embodiment of the present invention includes a server control unit that makes a judgment about the input command word and generates a corresponding control command, and a server communication unit that communicates with a database containing a large number of example sentences and the home appliance. After the server communication unit receives the command word input to the home appliance from the home appliance, if the input command word does not correspond to direct control, the server control unit inputs the command word, along with a guide that includes a category for the command word and example sentences extracted from the database that have a similarity of above a certain standard to the input command word, into a generative AI module. The result calculated by the generative AI module includes the category of the input command word, and the server communication unit generates a control command using the calculated result and transmits the control command to the home appliance so that the home appliance can provide functions according to the generated control command. [Effects of the Invention]

[0011] When the present invention is applied, speech recognition errors can be improved, and user commands can be recognized flexibly.

[0012] When applying the present invention, various controls can be operated simultaneously when processing the user's command words.

[0013] When applying the present invention, home appliances can be controlled via voice recognition even when the user does not use prescribed commands.

[0014] The effects of the present invention are not limited to those described above; various other effects of the present invention in the configuration of the present invention can be easily conceived. [Brief explanation of the drawing]

[0015] [Figure 1] This diagram shows the process by which a home appliance operates when a command word according to one embodiment of the present invention is input. [Figure 2]This diagram shows the process by which, when an instruction word according to another embodiment of the present invention is input, the home appliance transmits the instruction word to a server, and then receives a control command from the server and operates. [Figure 3] This figure shows the configuration of a server according to one embodiment of the present invention. [Figure 4] This figure shows the classification of input command words according to one embodiment of the present invention. [Figure 5] This figure shows the process by which a server according to one embodiment of the present invention processes an input command word. [Figure 6] This figure shows the structure of a prompt according to one embodiment of the present invention. [Figure 7] This figure shows an exemplary base according to one embodiment of the present invention. [Figure 8] This figure shows the flow of interpreting input command words using a generative AI module according to one embodiment of the present invention. [Figure 9] This figure details the operation process of a server according to one embodiment of the present invention. [Figure 10] This figure shows the configuration of a home appliance according to one embodiment of the present invention. [Modes for carrying out the invention]

[0016] Hereinafter, with reference to the drawings, embodiments of the present invention will be described in detail so that those with ordinary skill in the art to which the present invention pertains can easily implement them. The present invention can be embodied in various different forms and is not limited to the embodiments described herein.

[0017] To clearly explain the present invention, parts not related to the explanation are omitted, and the same reference numerals are used for the same or similar components throughout the specification. In addition, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When attaching reference numerals to the components of each drawing, for the same components, even if they are shown on other drawings, they may preferably have the same reference numerals. Also, in the description of the present invention, when it is determined that a specific description of a related known configuration or function obscures the gist of the present invention, the detailed description thereof can be omitted.

[0018] In describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc. can be used. These terms are for distinguishing the components from other components, and the essence, order, sequence, or number of the components are not limited by these terms. When a component is described as being "connected", "coupled", or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but other components may be "interposed" between the components, or each component may be "connected", "coupled", or "connected" through other components.

[0019] Also, in implementing the present invention, although the components can be described in a subdivided manner for convenience of explanation, these components may be implemented in one device or module, or one component may be divided and implemented in a number of devices or modules.

[0020] Hereinafter, the household electrical appliances described in this specification are devices including electronic products. The household electrical appliances may be arranged in a home, an office, etc., and the household electrical appliances may also be moved by a person and arranged in other places.

[0021] The devices described herein, such as home appliances or server devices, preprocess voice or text commands entered by a user, classify the entered commands into specific categories, interpret the entered commands (user commands) according to the classified categories, and control the home appliances so that they can operate in accordance with the results of the interpretation.

[0022] A consumer electronics appliance, a server, or a system consisting of one or more devices can store hardware or software for processing feature instructions. Alternatively, a consumer electronics appliance, a server, or a system consisting of one or more devices can process feature instructions by receiving software transmitted from a remote third device. Processing feature instructions using hardware built into the device, software stored within the device, or transmitted and executable software can be performed by a processor within each device. Alternatively, the hardware or software itself can function as a processor.

[0023] Therefore, the processor can store software or program code that can perform the aforementioned tasks. This software or program code can be received from other external devices, stored on a storage medium used by the processor, and then the processor can execute the software or program code.

[0024] Furthermore, the processor may include hardware components such as a programmable chip, and the processor can store data and program code to be input to the hardware components in a predetermined storage medium before inputting them to the hardware components.

[0025] The hardware or software may be the processor itself. Alternatively, the hardware or software can work in conjunction with the processor to embody embodiments of the present invention.

[0026] Figure 1 is a diagram illustrating the process by which a home appliance operates when a command word according to one embodiment of the present invention is input.

[0027] User 1 inputs a predetermined command word (voice or text) to the home appliance 100 (S3). The command word can be input by voice (VOICE) or by text. The input command word (or input command word, Input Command) is converted to text (Speech To Text), and the home appliance (device) 100 determines a control command corresponding to the command word based on the pre-processed result (S5). Once the control command is determined, the home appliance 100 executes the determined control command to perform a predetermined function (S7).

[0028] Figure 2 shows the process by which, when an instruction word according to another embodiment of the present invention is input, the home appliance transmits the instruction word to the server, and then receives a control command from the server and operates.

[0029] Refer to Figure 1 for S3. The home appliance 100 transmits the input command word to the server 500 (S11). At this time, if the input command word is a voice command, the home appliance 100 can convert the voice command to text and transmit the command word, which is the result of pre-processing the text, to the server 500.

[0030] In this case, the server 500 can determine the control command corresponding to the received command word (S15). The preprocessing process can also be performed by the server 500. In this case, the home appliance 100 can transmit the input command word directly to the server 500.

[0031] Once a control command is determined, the server 500 transmits information about the determined control command to the home appliance 100 (S16), and the home appliance 100 executes the received control command and performs a predetermined function (S17). For example, a home appliance sends a user command to the server, the server interprets the command to understand the user's intent, and determines the specific action that the home appliance should perform. The home appliance executes the command determined by the server, but the embodiment is not limited to this. In another embodiment, the home appliance 100 may locally perform a decision process to determine the user's intent and generate a corresponding control command.

[0032] As discussed in Figures 1 and 2, the server 500 or the home appliance 100 can calculate a suitable control instruction from the pre-processed instruction words.

[0033] The following description will focus on the server 500's operations for command recognition, but the present invention is not limited thereto, and the home appliance 100 can also provide some or all of the functions offered by the server 500. Furthermore, for the sake of clarity, an air conditioner will be described as one embodiment of the home appliance 100, but the present invention is not limited thereto.

[0034] The following describes an apparatus that embodies an embodiment of the present invention.

[0035] Embodiments of the present invention can be embodied in various devices. These devices include servers, home appliances, electronic devices, computer devices, and many others. In addition to physical devices, the devices of the present invention may also include hardware or software components that perform embodiments of the present invention. Furthermore, embodiments of the present invention include programs, hardware, chips, etc., that are stored or embodied in a form that enables them to perform predetermined tasks.

[0036] In the case of programs, software, etc., they may be stored permanently within the device, or they may be temporarily transmitted from an external source, stored in the device, and then executed. In the case of a permanent storage method, the device may include a non-transitory computer-readable medium.

[0037] In other words, embodiments of the present invention can be embodied in a computer-readable storage medium by one or more computer programs, or any combination of any or more of the above.

[0038] The functions of the elements disclosed herein can be embodied using circuits or processing circuits that include general-purpose processors, special-purpose processors, integrated circuits, ASICs ("Application-Specific Integrated Circuits"), existing circuits, and / or combinations thereof. These circuits may be processors configured or programmed to perform the disclosed functions. Since processors include transistors and other circuits, they can be considered processing circuits or circuits.

[0039] Circuits, units, or means in this specification may be hardware that performs or is programmed to perform the functions mentioned in the detailed description. Hardware may be the hardware disclosed herein or other known hardware, and may be hardware that is programmed or configured to perform the functions mentioned in the detailed description of the specification. If the hardware is a processor that is considered a type of circuit, then the circuit, means, or unit may be software used to constitute the hardware and / or processor in combination with the hardware and software. Furthermore, computer storage media may be non-transitory computer readable medium. For example, they may be executable by a cloud server-based system. Computer storage media may be located in the same single device or distributed across two or more different devices. Thus, logically one computer storage medium may physically include two or more computer storage media, and they may be located in one or more locations. Computer storage media include various storage media such as hard disks, CD / DVD discs, memory cards, and memory chips.

[0040] Furthermore, the data described in the specification can be computed or performed in various environments, including cloud server-based systems, on-device systems, and distributed server systems (multiple servers). Processing may be performed in a distributed manner on cloud servers or locally on an on-device processor, and the results processed in each environment can be stored in non-volatile memory.

[0041] Electronic devices such as home appliances and server computer equipment may be connected to one or more storage devices via a network. The storage devices may be a combination of volatile and non-volatile memory, and may or may not be located in the same physical location as the computer equipment.

[0042] A server computer system may include one or more processors and memory. The memory may store information accessible to the processor and may contain data capable of processing, storing, or modifying instruction words executed by the processor. This memory may also consist of volatile and non-volatile memory. The processor may include a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a tensor processing unit (TPU).

[0043] Instructions may be configured to perform specific actions when a processor executes an instructed task, and may be stored in the form of object code or an interpretable script. These instructions can be used to embody a system and can be executed on a local or remote processor. Data can be retrieved, stored, or modified by instructions and may consist of a database, JSON, YAML, or XML.

[0044] These commands may include executable files, source code files, or metadata.

[0045] Electronic devices, including home appliances and hubs, can be configured similarly to server computer devices. Electronic devices, including home appliances and hubs, may include a processor, memory, instructions, data, and user input and output devices. The server computer device can transmit data to the electronic devices, and the electronic devices can display a portion of the received data via a display. Furthermore, data transmission and communication between the server computer device and the electronic devices are possible via networks such as Bluetooth®, Wi-Fi, wired, and wireless networks, and various protocols and connection methods are supported. Direct and indirect communication between computer devices is possible, and various protocols and connection methods are supported.

[0046] Furthermore, the functions of the server computer device of the present invention can be performed by smartphones, tablets, etc.

[0047] The methods or processes described herein involve sequentially performing one or more tasks, and each task can be performed by either hardware, software, or a combination of both. For example, the first task may be performed by hardware and the second by software. Of course, either hardware or software can perform the entire task.

[0048] Figure 3 shows the configuration of a server according to one embodiment of the present invention.

[0049] The server 500 includes a server control unit 550, a control example database 550, a status example database 520, a control command database 530, and a server communication unit 590. It also optionally includes a generative AI module 300. That is, the generative AI module 300 may be embodied in the server 500 or in an external device. The generative AI module 300 may be a single piece of software, hardware, or a storage medium executable by a computer. The server 500 and the home appliance 100, as embodiments of electronic devices, can process various information, communicate, and process data stored in a storage medium such as memory. The server 500 and the home appliance 100 may include the generative AI module in the form of software, hardware, or a storage medium. Alternatively, the generative AI module may be embodied in an external device other than the server 500 or the home appliance 100 in the form of software, hardware, or a storage medium. Therefore, the generative AI module 300 described herein can be embodied in a generative AI unit, a generative AI controller, a generative AI program, generative AI software, generative AI hardware, and the like.

[0050] The server control unit 550 determines which category the input instruction belongs to or which control instruction it corresponds to, and generates the corresponding control instruction. During the determination process, the server control unit 550 can input predetermined information to the generative AI module 300. That is, in order to confirm the category, the server control unit 550 inputs the input instruction to the generative AI module 300, and as a result can obtain the category. This can also be applied to the device control unit 150.

[0051] Databases 510 and 520 can store a large number of example sentences. Furthermore, the situation example database 520 can store predefined routines.

[0052] The server communication unit 590 communicates with the home appliance 100. Furthermore, if the generative AI module 300 is located on an external device (external server), the server communication unit 590 also communicates with the external device.

[0053] When the server communication unit 590 receives a command word input to the home appliance 100 from the home appliance 100, the server control unit 550 checks whether the command word input to the home appliance 100 corresponds to Direct Control. For example, in one embodiment, Direct Control may refer to a command from the user that the system can immediately understand and execute without requiring interpretation by an artificial intelligence model. Alternatively, Direct Control commands may be predefined commands that directly match a specific set of functions or characteristics in an existing database of the home appliance or server. The system can recognize the command and generate a corresponding control command with high accuracy without further analysis. For example, if a user says "Turn on the air conditioner," and the command is of a pre-programmed type, it can be considered Direct Control. The system does not need to guess the user's intent and executes the "Power On" function of the air conditioner. On the other hand, non-Direct Control commands may be more ambiguous or creative, such as "It's very cold." Such commands can be interpreted using a generation AI module to generate a specific command. For example, in one embodiment, this method may include determining whether the input command corresponds to this type of Direct Control. If not supported, the system can interpret it using a generated AI module.

[0054] If, upon verification, the input command word does not correspond to a direct control, the server control unit 550 extracts example sentences from the database 520 whose similarity to the input command word is above a certain standard and inputs them to the generative AI module 300. In one embodiment, similar sample text can be searched based on text matching or a text matching score. In another embodiment, sample text can be saved as an embedding vector and similar sample text can be searched based on a vector similarity function such as cosine similarity or Euclidean distance.

[0055] For example, the server control unit 550 can input an example sentence (an example sentence whose similarity to the input instruction is above a certain standard) extracted from a guide and database containing categories for the instruction, along with the instruction, into the generative AI module. The result calculated via the generative AI module will include the category of the input instruction, and the server control unit 550 can use the calculated result to generate a control instruction. These processes can also be performed in the device control unit 150.

[0056] Similarity can be determined based on the type of input command words, specifically whether they belong to the same type. For example, in the case of an air conditioner, temperature control, airflow direction control, and time setting are all types of commands. If a command word relates to temperature control, the server control unit 550 can extract existing example sentences related to temperature from the database 520. Of course, these similarity determinations can also be performed by the equipment control unit 150.

[0057] If the input command corresponds to direct control, the server control unit 550 can generate a control command corresponding to the input command without inputting the command to the generative AI module 300.

[0058] The results calculated by the generative AI module may include categories of the input command words (i.e., the user's input command words).

[0059] Then, if the input command word calculated by the generative AI module 300 corresponds to post-interpretational control, the server control unit 550 generates a control command using the calculated result.

[0060] This refers to the category descriptions in Figure 4.

[0061] Then, so that the home appliance 100 can provide functions according to the generated control commands, the server communication unit 590 transmits the control commands to the home appliance, allowing the server 500 to control the home appliance 100 based on the command words. For example, in situations where it is not a direct command, the generating AI module interprets the user's command to understand the user's intent, and the server generates a corresponding direct control command based on the output of the AI ​​module. This new direct control command is sent to the home appliance, instructing it to perform functions that satisfy the user's original requests and intentions.

[0062] Figure 4 shows the classification of input command words according to one embodiment of the present invention.

[0063] In the following, input voice commands, input text commands, or pre-processed utterances thereof will all be referred to simply as input commands.

[0064] The input commands belong to either direct control (Category 1) or non-direct control categories (Categories 2, 3, and 4). In other words, the input commands can be distinguished as either requiring interpretation by a generative AI module (Categories 2, 3, and 4) or not (Category 1).

[0065] Direct control refers to a situation where an input instruction word directly matches the control command required to control the home appliance. This occurs when the server 500 does not apply a separate interpretation process to the input instruction word. An input instruction word that qualifies as direct control can correspond to one or more control commands.

[0066] If the input command word belongs to the direct control category, the server 500 generates a control command necessary to control the home appliance 100 in response to the input command word. For example, if it corresponds to one control command (consisting of a single utterance), such as "Turn on the air conditioner" or "Set the temperature to 25 degrees," or if it corresponds to two control commands (consisting of a compound utterance), such as "Use the air conditioner with four fan speed settings," the server 500 can use the control command to control the home appliance 100. In other words, the server 500 can generate a control command in response to an input command word that falls under direct control (input command word) and provide it to the home appliance 100.

[0067] The server 500 determines only whether the input command word corresponds to direct control, and if it does not correspond to direct control, it inputs it to the generative AI module 300. In this case, the information (prompt) that the server 500 inputs to the generative AI module 300 may include a base, guide, and example sentences.

[0068] The Base contains information that instructs the generative AI module 300 on the roles it must perform and the results it must calculate in order to interpret the input command. For example, the Base may refer to the basic set of instructions provided to the generative AI module (for example, the Base is also called a system prompt). The function of the Base is to define the operating context of the AI ​​model, thereby guiding the interpretation of commands from the user (e.g., the type of appliance, a description of the result, constraints on the result, etc.). This also includes establishing specific roles that the AI ​​should perform, such as acting as an expert system for controlling an appliance. Furthermore, the Base can also specify the structure and format required for the AI ​​model's output.

[0069] The Guide includes information to distinguish which category the input command belongs to (i.e., which of categories 2, 3, or 4 in Figure 4 it belongs to) and the components of the control command that the generative AI module 300 must calculate in response. For example, the Guide can be part of the prompt, providing the generative AI module with specific contextual information about the command being processed. Unlike a base where general operating parameters can be set, the Guide can provide detailed information relevant to the task, such as predefined categories related to the intent of the user's command (e.g., interpreter control, interpreter routine, interpreter chat, etc.). By limiting the AI ​​module's interpretation to a specific functional area, the Guide can reduce ambiguity and prevent the generation of irrelevant or incorrect responses. By narrowing the AI ​​model to a specific category, the Guide can improve the accuracy and relevance of the final control commands generated for home appliances.

[0070] Prompts can include one-time examples or a small number of examples. For example, the sample text portion of a prompt can be a dynamic component of the prompt that provides the generating AI module with concrete examples of commands that have been previously interpreted. These examples can be selected from a database and can be selected based on their high semantic similarity to the user's current input command. The sample text can provide the AI ​​module with a contextual model of successful interpretations. For example, it can show how similar phrases or intents have been translated into structured commands in the past. By providing these relevant examples, the AI ​​model can better understand nuances and process variations in the user's phrasing to generate more accurate and reliable control commands for home appliances. For example, the sample text is an example sentence whose similarity to the input command word is above a certain threshold. Server 500 can use BM25 or Cosine similarity techniques to extract N example sentences (where N is a natural number greater than or equal to 1) from databases 510 and 520 whose similarity to the input command word is above a certain threshold. In other words, the example sentences may be those sentences stored in databases 510 and 520 that have a similarity score above a certain threshold based on BM25 or cosine similarity to the input command word. In addition to the method described above, various other methods can be used to extract example sentences.

[0071] The process for calculating similarity according to one embodiment of the present invention is as follows: In order to calculate similarity, the command words entered by the user can be preprocessed. For example, the server control unit 550 or the device control unit 150 can extract nouns, foreign words, verbs, roots, etc. from the command words entered by the user by cutting them based on spaces, or by morphological analysis.

[0072] The server control unit 550 or the device control unit 150 can perform a primary filtering (e.g., 30, 40, etc.) by comparing the pre-processed user command words with example sentences stored in the database 520. In this process, the server control unit 550 or the device control unit 150 can reorder the filtered results according to BM25 based on frequency count, etc., by applying cosine similarity.

[0073] Subsequently, the server control unit 550 or the device control unit 150 can select i samples of the intent with the highest similarity score and j samples of the next highest intent as samples.

[0074] Based on the prompt input to the generative AI module 300, the generative AI module 300 can calculate a predetermined result, and the server 500 determines, through the calculated result, that the input command word falls into one of categories 2 / 3 / 4.

[0075] If the input instruction belongs to the second category, Post-Interpretational Control, the server 500 can use the calculated result to generate the control instructions necessary to control the home appliance.

[0076] On the other hand, if the input instruction is a post-interpretational routine of the third category, the server 500 can generate a control instruction corresponding to that routine. In this case, the routine may be stored in the situation example database 520, and the routine may include common routines that apply to all home appliances and personalized routines that apply to home appliances used by a specific user.

[0077] If the input command is a Post-Interpretational Chat (Category 4), the server 500 outputs the calculated result as audio or text. The server 500 can generate control commands so that the speaker or screen of a home appliance can output the audio or text, so that the home appliance can output it.

[0078] Categories 1, 2, and 3 all control home appliances to perform specific actions. Therefore, input commands corresponding to categories 1, 2, and 3 are control utterance commands. However, users may explicitly or implicitly instruct home appliances to operate, and if the server 500 cannot calculate a control command that matches the input command, the generative AI module 300 can input the command, along with a base, guide, and example sentence, to confirm the user's intent.

[0079] In other words, even when the user explicitly instructs a function to operate, if the input command contains words or sentences unrelated to control, or if the input command contains erroneous sentences, the server 500 may not be able to calculate the corresponding control command.

[0080] In this case, the server 500 can improve the accuracy of instruction processing using the generative AI module 300.

[0081] When a user implicitly inputs voice / text commands to operate a home appliance, the server 500 can process these as context-related commands. Context-related commands are divided into those defined / stored by the server 500 and those that are undefined.

[0082] In other words, the server 500 stores routines, which are bundles of functions provided by the home appliance 100. However, even when a user instructs the execution of these routines, if the user cannot clearly articulate the name of the routine, the generative AI module 300 can be used to improve the accuracy of the instruction processing.

[0083] Furthermore, server 500 can interpret the user's intent even when it is not defined in the routine.

[0084] This specification describes a process in which, after the home appliance or server preprocesses the command words input via the home appliance, the home appliance operates in response to the input command words if the preprocessed command words provide the necessary information for controlling the home appliance (Category 1). On the other hand, if the preprocessed command words do not provide the necessary information for controlling the home appliance, the home appliance operates after performing an interpretation process for the input command words and using the information calculated after the interpretation (Categories 2, 3, and 4).

[0085] In order for the generative AI module 300 to calculate accurate results, the server 500 can extract example sentences from the control example database 510 or the situation example database 520 whose similarity to the input command word is above a certain standard, and input them to the generative AI module 300 along with the input command word.

[0086] As a result, the generative AI module 300 interprets the input command words using examples and calculates a result containing the information necessary to generate control commands that enable the home appliance to operate, based on the input command words, i.e., the user's input command words. The server 500 generates control commands corresponding to the calculated results and provides them to the home appliance 100.

[0087] The server 500 of the present invention, when a voice / text command is input to a home appliance 100 such as an air conditioner, analyzes and extracts the user's intent and controls the home appliance 100 to operate accordingly.

[0088] If the sentence spoken by the user (input command word) corresponds directly to an existing set format, the server 500 can control the home appliance 100 in response to the input command word without using a separate generative AI module 300.

[0089] On the other hand, if the input command word differs from the format set in the server 500, the generative AI module 300 is used to interpret it and ensure the result is obtained, but the intent / control for the result is provided with predetermined information (such as base and guide) to prevent errors in the generative AI module 300.

[0090] Furthermore, it is possible to control home appliances in a customized manner using the user's personalized information, distinguishing between routine and non-routine commands.

[0091] The server 500 improves the accuracy of the results by selecting similar command words into example sentences during the voice command processing process and providing them to a generative AI module 300 such as chatGPT. The generative AI module 300's output can also be applied in both routine and non-routine forms, thus fully reflecting the user's intent. In one embodiment, the generative AI module 300 may be independent of an LLM such as ChatGPT, but the embodiments are not limited thereto. For example, in another embodiment, the generative AI module 300 may operate based on a proprietary LLM model that has been trained or fine-tuned for a specific task, such as controlling home appliances.

[0092] Figure 5 shows the process by which a server according to one embodiment of the present invention processes an input command word.

[0093] The home appliance 100, or a user terminal or remote control linked to the home appliance 100, receives command words via voice or text. The home appliance 100 or the user terminal then transmits the input command words (input command words) to the server 500 (S21).

[0094] The server 500 checks if there is a control instruction corresponding to the input instruction word, and if there is, it generates the control instruction (S22) and transmits it to the home appliance 100 (S23). As a result, the home appliance 100 operates according to the control instruction (S24).

[0095] On the other hand, if there is no control instruction corresponding to the input instruction (S31), the server 500 extracts example sentences similar to the input instruction from the database and generates a prompt that includes the base, side, example sentences and the input instruction (S32). For example sentences are stored in the control example database 510 or the situation example database 520, and the server 500 can extract example sentences from the respective databases 510 and 520 whose similarity to the input instruction is above a certain threshold.

[0096] Server 500 can generate prompts containing the aforementioned base, guide, example sentences, and input commands. The generative AI module 300 can also generate prompts. The prompt configuration will be described later.

[0097] Then, the server 500 inputs a prompt to the generative AI module 300 (S33). If the generative AI module 300 is implemented in an external server that is a generative AI service provider, the server 500 can input a prompt to the generative AI module 300 via an API agreed upon in advance with the generative AI service provider. One embodiment in which the generative AI module 300 is implemented in a generative AI service provider may include generative AI models such as various large language models including chatGPT. Alternatively, a generative AI module such as chatGPT may be located within the server 500.

[0098] In response to the prompt input in S33, the generative AI module 300 generates and provides the result. That is, the generative AI module 300 returns the result in response to the prompt input in S33 (S34).

[0099] The generative AI module 300 can be implemented on the server 500 or on a separate external server (generative AI service provider). The server 500 can include guiding content in its prompts to interpret which category the input command belongs to.

[0100] Server 500 examines the validity of the result (S35). If, as a result of the examination, the result contains a control instruction or is capable of generating a control instruction, i.e., is category 2, then Server 500 generates a control instruction (S36a).

[0101] On the other hand, if the results show that the output does not contain control instructions but contains routines, i.e., it is category 3, the server 500 extracts information from the situation example database 520 corresponding to the output and generates control instructions (S36b).

[0102] The server 500 transmits the control command generated in S36a or S36b to the home appliance 100 (S37). As a result, the home appliance 100 operates according to the control command (S38).

[0103] The functions of server 500 in Figure 5 can be provided by home appliance 100.

[0104] The process shown in Figure 5 can be summarized as follows: The server 500 or the home appliance 100 checks whether the input command corresponds to direct control (S22, S31).

[0105] If the command words entered in S22 and S31 do not fall under direct control (Category 1), the server 500 or the home appliance 100 extracts example sentences from databases 510 and 520 whose similarity to the entered command words is above a certain standard and inputs them into the generative AI module 300 (S32 and S33).

[0106] If the generative AI module 300 calculates (S34) that the input command word corresponds to Post-Interpretational Control (Category 2), the server 500 or the home appliance 100 generates a control command using the calculated result (S35a). The home appliance 100 then provides (operates) its function according to the generated control command (S37, S38).

[0107] If the input command word, as in S22, falls under direct control (Category 1), the server 500 or the home appliance 100 generates a control command corresponding to the input command word, and the home appliance 100 provides a function according to the control command corresponding to the input command word (S23).

[0108] If the input does not fall under Category 1 in S32, the server 500 or the home appliance 100 can input the example sentence and the input command word into the generative AI module 300. At this time, the server 500 or the home appliance 100 can extract example sentences from databases 510 and 520 whose similarity to the input command word is above a certain threshold and input them into the generative AI module 300. See Tables 3 to 7 below for example sentences.

[0109] Then, if the input instruction belongs to category 3, that is, if it corresponds to the execution of a routine stored in the server 500 or the home appliance 100, as shown in Table 6 below, the server 500 or the home appliance 100 acquires a control instruction corresponding to the stored routine and uses it to control the home appliance 100.

[0110] In other words, if the input instruction word calculated by the generative AI module 300 corresponds to a Post-Interpretational Routine (Category 3), the server 500 or home appliance 100 can generate a control instruction corresponding to the routine included in the calculated result. For this reason, the server 500 or home appliance 100 can extract control instructions corresponding to already stored routines from memory or a database.

[0111] The server 500 or home appliance 100 of the present invention implements a GPT prompt to improve speech recognition so that even when a user inputs a command word containing a control function or a sentence relating to a specific situation (=uncontrolled utterance), the server 500 or home appliance 100 can interpret the utterance text. Furthermore, by using representative utterance learning data and a sample extraction algorithm that are examples of exemplary sentences, the server 500 or home appliance 100 can interpret the intent of the command word (or sentence) input by the user and recommend a suitable function.

[0112] In particular, by interpreting the command words entered by the user using the generative AI module 300, it is possible to extract the user's intention to control the home appliance even from complex utterances that do not have a predetermined form or format. Furthermore, the user can give voice commands related to their situation, in which case the home appliance can operate in a customized manner suitable for the user's situation. In addition, if a specific routine operation is set for the home appliance, the server 500 or the home appliance 100 can load control commands related to the routine corresponding to the sentence entered by the user describing their situation and use these to control the operation of the home appliance 100.

[0113] In this embodiment of the present invention, if the text spoken or entered by the user directly corresponds to a control command for controlling the home appliance, the home appliance is controlled based on the corresponding control command; otherwise, the home appliance is controlled based on the result interpreted via the generative AI module 300. As a result, the server or home appliance can quickly determine whether the user's command falls under direct control category 1 in Figure 4, thus enabling rapid voice / text control of the home appliance.

[0114] Furthermore, if the user's command does not fall under Category 1 direct control, the server or home appliance can work with the generative AI module 300 to obtain control commands corresponding to the input command words, thereby enabling accurate voice / text control of the home appliance.

[0115] When using the generative AI module 300, the server 500 or the home appliance 100 can process both controlled and uncontrolled speech.

[0116] In the case of control function utterances, the system can process complex utterances that instruct various control commands, sentences that contain uncontrolled words, and short utterances that may contain errors or mispronunciations during the recognition process (Tables 3-5).

[0117] Furthermore, in the case of situational awareness speech, the generative AI module 300 can provide explanations and interpretations of situations based on predefined situations (configured routines), situations registered in ThinQ routines, and undefined situations. In addition, utterances in everyday chat can be processed by the server 500 or home appliances (Table 6 or Table 7).

[0118] In other words, by dividing control commands for controlling home appliances into intent control commands and slot control commands, the server 500 or home appliance 100 can flexibly process sentence-type commands entered by the user, extract the user's intent to control the device even from complex spoken commands, and control the functions of the home appliance 100. Intent control commands and slot control commands will be described later.

[0119] Furthermore, even for undefined routines or situations, by inputting example sentences and guides into the generative AI module 300, the server 500 or the home appliance 100 can combine two or three control commands for the functions that the home appliance (e.g., an air conditioner) can provide.

[0120] Based on the interpretation of the user's utterance, a pre-specified routine (for example, a routine stored in the ThinQ server or application) can be executed.

[0121] Control commands for home appliances 100 are instructions that control the operation of the home appliance, that is, the operation of the home appliance so that it provides a specific function. Control commands for home appliances are classified into intent control commands and slot control commands.

[0122] An Intent control instruction specifies the higher-level function of the home appliance 100, while a Slot control instruction specifies the subdivided functions of the Intent control instruction. Both Intent and Slot control instructions must be provided for the home appliance to operate. In other words, the set of Intent and Slot control instructions specifies the function that the home appliance performs.

[0123] In other words, intent control commands are control commands for higher-level concepts within the functions of home appliances, such as on / off, temperature control, and fan speed control. Slot control commands correspond to the detailed settings of these intent control commands. For example, detailed on / off control commands can be on (turn on) and off (turn off), and the slot control command that makes up the intent control command for temperature control can be a specific temperature value. Home appliances perform their functions through a collection of intent control commands and slot control commands. For example, a slot control command is a structured, machine-readable instruction generated as the final output of the interpretation process. A slot control command can be configured so that each "slot" represents a specific parameter of the target (target device, action to perform, specific setting value, etc.). For example, the natural language input "The kitchen is too dark" can be converted into a slot control command like {device: “kitchen_light”, action: “set_brightness”, value: “80%”}. This format eliminates the ambiguity inherent in human speech and provides home appliances with precise commands that can be executed directly without requiring further interpretation. Furthermore, intent control commands can represent higher-level conceptual goals derived from the user's natural language input, encompassing multiple actions or more abstract objectives. Unlike slot control commands, which specify concrete actions, intent control commands can identify the user's overall objective, such as "prepare for a movie" or "enhance home security." For example, this method first identifies the intent from the user's utterance. Then, using the identified intent, it generates one or more specific slot control commands to achieve the user's objective. For instance, the intent "movie_night" might trigger the generation of individual slot control commands such as dimming the lights, turning on the TV, and adjusting the sound system. In this way, a single abstract objective is transformed into a set of specific, actionable actions.

[0124] In the case of an air conditioner, Table 1 shows a list of intent control commands and some slot control commands. The appliance can perform functions that turn on according to the intent control command AIR_POWER_REQUEST and the slot control command ON.

[0125] [Table 1]

[0126] In the case of an oven, some of the intent control instructions and slot control instructions are shown in Table 2.

[0127] [Table 2]

[0128] In other words, an intent control instruction is an instruction that corresponds to the user's intention to control the device, and a slot control instruction is the detailed content of such an intention. Therefore, when intent control instructions and slot control instructions are combined, the home appliance can operate to perform a specific function. When the contents of Table 1 or Table 2 are input to the generative AI module 300 by a home appliance, the generative AI module 300 can calculate the intent control instruction and slot control instruction corresponding to the input instruction words.

[0129] Furthermore, the generative AI module 300 can interpret the intent behind the voice / text commands entered by the user.

[0130] Furthermore, example sentences may be input to the generative AI module 300 so that the generative AI module 300 can calculate the corresponding intent control instruction and slot control instruction for the input instruction word.

[0131] To obtain accurate results via the generative AI module 300, it is necessary to generate prompts.

[0132] Figure 6 shows the structure of a prompt according to one embodiment of the present invention. The information input to the generative AI module 300 is referred to as a prompt, but the present invention is not limited thereto. The server 500 or home appliance 100 can input text classification and few-shot learning based on prompting of a GPT LLM (Generative Pre-trained Transformer Large Language Model) to obtain the necessary information from the generative AI module 300, and can extract intent control instructions and slot control instructions from the input command words.

[0133] As described above, the prompt 400 may consist of a base, guide, example sentences, and input command words necessary for controlling the home appliance, and in some embodiments, it may consist of only a portion of these.

[0134] The Base 410 contains information that instructs the generative AI module 300 on the roles it must perform and the results it must calculate in order to interpret the input command. For example, a server 500 or a home appliance 100 can generate a Base as follows: The Base includes basic rules, but the details are the type of home appliance to which it applies, the type of results to be calculated 411, a description 412 for each of the results to be calculated, and limitations 413 for the results.

[0135] The generative AI module 300 is instructed to generate the results in the order of [1], [2], and [3], by specifying the results in three fields. See Figure 7.

[0136] The Guide includes information for distinguishing which category the input command belongs to (i.e., which of categories 2, 3, or 4 in Figure 4 it belongs to), and corresponding components of the control command that the generative AI module 300 must calculate. In one embodiment, the Guide includes a Text Classification method and definitions of Intent / Slot. The Guide may also include explanations and definitions of Intent control commands and Slot control commands. The Guide 420 can be created in various ways depending on the characteristics or implementation method of the generative AI module 300.

[0137] Figure 7 shows an exemplary base according to one embodiment of the present invention. Each corresponding element is indicated by an instruction number. 411 indicates the type of result to be calculated and that the equipment to which it applies is an air conditioner. 412 relates to control and defines three results (whether or not it is a control command, intent, and slot). 413 indicates that three contents must be included in the output.

[0138] For example, the guide could explain the rules for each result, list the intent and slot control instructions that must be met, and clearly indicate which rules must be followed to fulfill the requirements.

[0139] The guide configuration can consist of three areas. One embodiment of these three areas is a guide for the computed object, a guide for the computed object of intent control instructions, and a guide for the computed object of slot control instructions.

[0140] The guidance for the output guides the system to output content indicating whether the input command is for control purposes. For home appliances related to air conditioners, weather, conditions, and emotions are considered "non-control," while power, temperature settings, and fan speed, which are related to the control of the air conditioner, belong to "control."

[0141] Table 1 shows an intent control instruction and its corresponding explanation, which constitute a single set. An intent control instruction can instruct the input instruction word in the higher-level concept of the slot to select one of a specific category.

[0142] The area guiding the content of slot control instructions can instruct different intent control instructions to be processed separately. If the calculation is "control", slot control instructions related to air conditioner control may be listed together. These may each be guided according to intent control instructions.

[0143] Intent control instructions - slot control instruction 1, slot control instruction 2, etc.

[0144] Guides and example sentences (Sample Text) (Few-Shot learning) are input into the LLM. Example sentence 430 is an example sentence whose similarity to the input command word is above a certain standard, and the server 500 can use BM25 or Cos similarity techniques to extract N example sentences (where N is a natural number greater than or equal to 1) whose similarity to the input command word is above a certain standard from databases 510 and 520.

[0145] The example sentences used as training data are various utterances (representative utterances), which may be stored in the control example database 510 or the situation example database 520.

[0146] The following table shows the example sentences stored in the control example database 510, and their corresponding intent control commands and slot control commands. Table 3 shows compound utterance example sentences with two or more functions to control (intent control commands). Table 4 shows example sentences that contain words or sentences unrelated to control. Table 5 shows example sentences that contain some errors.

[0147] [Table 3]

[0148] [Table 4]

[0149] [Table 5]

[0150] When example sentences like those in Tables 3 to 5 are input, and the aforementioned base and guide are input, the generative AI module 100 calculates a result that includes intent control instructions and slot control instructions.

[0151] For example, if the user inputs the command "Set the temperature to 23 degrees and increase the fan speed," the generative AI module 100 can calculate the following result using the example sentences shown in Table 3 above.

[0152] [{`intent`:`AIR_TEMPERATURE_REQUEST`, `slots`:[{ `temperature`: `23`}]}] "{`intent`: `AIR_WIND_STRENGTH_REQUEST`, `slots`:"{`whind_speed`: `up`}''}”

[0153] On the other hand, example sentences that instruct routines for specific situations can also be divided into two categories: predefined routines and undefined routines.

[0154] A predefined routine is a set of situations that correspond to home appliances, such as exercise, studying, job hunting, or going out, and the user can instruct the appliance to perform these routines. Table 6 shows example sentences related to routines.

[0155] [Table 6]

[0156] Next, for any unspecified routines, the generative AI module 300 can output the corresponding information.

[0157] [Table 7]

[0158] Next, if the input command is completely unrelated to controlling home appliances, this falls under category 4, "post-interpretation chat," and the generative AI module 300 can output the result accordingly. For example, if the user inputs the command "Who is the author who received the Novel Prize in Physics this year?", the generative AI module 300 can output the result as follows:

[0159] "{`INTENT`:`CHAT_REQUEST`, `SLOTS`: "{`ANSWER`: `The recipient of this year's Novell Prize in Physics is A.`}]}]

[0160] In other words, if the result calculated by the generative AI module 300 for an input command word corresponds to a Post-Interpretational Chat, the server 500 or the home appliance 100 generates a control command to output the calculated result in voice or text. Subsequently, the home appliance 100 can output a portion of the calculated result in voice or text.

[0161] In this case, the user can ask questions and hear responses as if having a conversation with the home appliance 100.

[0162] As shown in Tables 3 to 7, the example sentences stored in databases 510 and 520 can consist of sentences for controlling home appliances and a set of corresponding intent control instructions and slot control instructions.

[0163] Of these, Tables 3 to 5 may be stored in the control example database 510, and Tables 6 and 7 may be stored in the situation example database 520.

[0164] Alternatively, these may not be separated, and the example statements in Tables 3 to 7 and their corresponding intent control instructions and slot control instructions may be stored in a single database.

[0165] Using various example sentences and routines stored on the server, the server 500 can generate control commands suitable for the user's intent. While the user can instruct the home appliance to perform a specific function using voice or text commands, these commands may not always directly match into control commands that control the home appliance 100.

[0166] If the command is matched directly, it corresponds to direct control as shown in Figure 4 above. For example, to start an air conditioner, the user can include commands such as "turn it on / turn it on / put it in / put it in / start it / start it." However, the user can also input a command to start the air conditioner into the appliance that includes different words.

[0167] For example, if a user types "Air conditioner, it's hot," the entered command does not specify a particular function of the air conditioner because it does not contain any of the verbs mentioned above, such as "turn on / turn it on / put it in / put it in / start / start it." However, the user's intention is to either turn on the air conditioner or adjust the fan speed to be stronger, so the user's intention is to instruct the operation of the air conditioner's function.

[0168] The server 500 inputs example sentences to the generative AI module 300 so that the generative AI module 300 can interpret the command words even if they do not correspond to direct control. In other words, the generative AI module 300, which has been inputted example sentences corresponding to intent control commands and slot control commands that instruct the functions provided by home appliances, can calculate intent control commands and slot control commands for the input command words as well.

[0169] Furthermore, home appliances can operate according to various routines. For example, in the case of an air conditioner, there may be study routines, exercise routines, etc., and the wind speed, airflow, target temperature, etc., may be set according to these routines. Similarly, in the case of an oven, there may be routines for warming up packaged rice, boiling ramen noodles, etc., and the heating temperature, time, etc., may be set according to these routines.

[0170] Furthermore, even if the command words entered by the user do not precisely correspond to a routine, if the server 500 inputs a situational example sentence to the generative AI module 300, the generative AI module 300 can calculate a result reflecting the situational example sentence, and if the calculated result corresponds to a specific routine, the server 500 can generate a control command based on that routine.

[0171] In other words, the server 500 extracts example utterances from the situational utterance database 510 whose similarity to the input command word is above a certain standard, and inputs them to the generative AI module 300 along with the input command word. As a result, the generative AI module 300 interprets the input command word using the example utterances and calculates a result. The server 500 generates a control command corresponding to the routine instructed by the calculated result and provides it to the home appliance 100.

[0172] If the input command does not correspond to a predefined routine, the server 500 guides the generative AI module 300 to calculate the result related to the undefined routine, thereby enabling the generative AI module 300 to calculate the result.

[0173] The server 500 according to an embodiment of the present invention is a server that provides artificial intelligence services and may include databases 510, 520 that store customers' past personalized information and routine information.

[0174] Based on learning about the user's lifestyle, it can provide a spatial solution optimized for each individual. For example, if a user says, "I studied well last week, please set it up exactly the same way," using voice or text commands, the system can load the settings from that time stored in databases 510 and 520 to create a personalized environment.

[0175] In one embodiment of the present invention, the server 500 includes embodiments in which the server is a group of servers, which is a collection of various servers. Therefore, the configuration of the server 500 can be easily modified at the level of those skilled in the art.

[0176] Figure 8 shows the flow of interpreting input command words using a generative AI module according to one embodiment of the present invention. It is a detailed embodiment of the embodiment shown in Figure 5.

[0177] When a command word is input to the home appliance 100, the server 500 receives it (S21). The server 500 then checks whether the control command corresponds directly to it (S31 / S32). If it does not correspond directly, it generates a prompt and inputs it to the generative AI module 300, as discussed in Figure 5 (S33). The server 500 receives the result calculated by the generative AI module 300 (S34), examines the validity of the result, and after performing S36a / S36b depending on the type of result, transmits the control command to the home appliance 100, after which the home appliance 100 provides (operates) its function according to the control command (S38).

[0178] Figure 9 is a diagram illustrating in detail the operation process of a server according to one embodiment of the present invention. The process in S35 will be examined in detail.

[0179] The server control unit 550 of server 500 checks whether the intent control instruction and slot control instruction included in the result calculated by the generative AI module 300 are valid (S41). This can be checked using the intent control instruction and the corresponding slot control instruction, as shown in Table 1 or Table 2. For example, the server control unit 550 determines that the result is valid if the intent control instruction calculated in relation to the air conditioner is AIR_POWER_REQUEST and the slot control instruction is "ON".

[0180] On the other hand, if the intent control command calculated for the air conditioner is AIR_POWER_REQUEST and the slot control command is "5 minutes", the server control unit 550 can instruct the home appliance 100 to request that further command words be input, because the calculated result is incorrect.

[0181] Alternatively, the server control unit 550 can also verify whether the calculated result instructs a routine, and if so, it can perform a validation check by verifying whether it corresponds to a stored routine.

[0182] After completing the validation check in S41, the server control unit 550 checks whether the calculated result is an already defined routine (S42). If it is an already defined routine, the server control unit 550 retrieves the routine stored in the database (S43) and controls the home appliance 100 according to the stored routine.

[0183] On the other hand, if it is not a routine that has already been defined, the server control unit 550 generates a control command in response to the intent control command and the slot control command, and transmits the control command to the home appliance 100 to control the home appliance 100 (S45).

[0184] The command words (Voice, Text) entered by the user undergo a predetermined preprocessing process in the server 500 or the home appliance 100. If the entered command words directly correspond to predefined and stored words or sentences that control the home appliance, the server 500 or the home appliance 100 converts the entered command words into control commands.

[0185] On the other hand, if interpretation is required for the input command, the server 500 or home appliance 100 performs a command interpretation process. Generative AI can be used in this process, and the server 500 or home appliance 100 may include a generative AI module. Alternatively, the server 500 or home appliance 100 can interpret the input command using a generative AI module located on an external server.

[0186] The generative AI module 300 first interprets the input command word to determine whether it is a command word related to the control of a home appliance, and if it is a command word related to the control of a home appliance, it can calculate an intent control command and a slot control command using example sentences or guides provided by the server 500. Alternatively, the generative AI module 300 calculates a result that instructs the server 500 to execute a specific routine stored therein.

[0187] A specific routine refers to a command that specifies a predefined situation, i.e., a routine command. The server 500 or home appliance 100 can obtain intent control commands and slot control commands related to the predefined situation from the database.

[0188] For example, if a predefined routine is "exercise," even if the user says words like "home training," "running," or "cycling," the generative AI module 300 will interpret this as "exercise," and the server 500 or home appliance 100 will perform the functions of the exercise routine.

[0189] Furthermore, personalized routines can also be stored. If a user sets their own routines in the home appliance 100 and inputs words related to those routines, these routines can be stored in the home appliance 100 or the server 500. The generative AI module 300 can also interpret the words a user says to invoke these personalized routines.

[0190] If the instruction specifies a situation that is not predefined, that is, an instruction that has not been "chinned," the generative AI module 300 can extract the most similar routine from its pre-stored routines, or generate intent control instructions and slot control instructions based on the interpretation result of the input instruction word.

[0191] Figure 10 shows the configuration of a home appliance according to one embodiment of the present invention.

[0192] The structure is similar to that of the server 500 described above. However, the home appliance 100 differs from the server 500 in that it provides specific functions.

[0193] The home appliance 100 includes a device control unit 150, a control example database 110, a status example database 120, a control command database 130, and a device communication unit 190. It also optionally includes a generative AI module 300. That is, the generative AI module 300 may be implemented within the home appliance 100 or in an external device (e.g., a server 500 or an external provider).

[0194] The functional module 180 provides functions specific to the home appliance 100. For example, if the home appliance 100 is an air conditioner, the functional module 180 includes both an indoor unit and an outdoor unit. If the home appliance 100 is an oven, the functional module 180 includes components for heating. If the home appliance 100 is a refrigerator, the functional module 180 includes components that provide freezing and refrigeration functions.

[0195] Depending on the implementation method of the home appliance 100, the functional module 180 is configured separately and includes an equipment control unit 150, and the other components (110, 120, 130, 190, and optionally 300) can be composed of one or more software or hardware components. According to one embodiment of the present invention, the equipment control unit 150 and the other components (110, 120, 130, 190, and optionally 300) can be implemented on a single chip.

[0196] The device control unit 150 determines which category or control command an input command belongs to and generates the corresponding control command. During the determination process, the device control unit 150 can input predetermined information to the generative AI module 300.

[0197] Databases 110 and 120 can store a large number of example sentences. Furthermore, the situation example database 120 can store predefined routines.

[0198] The device communication unit 190 communicates with other devices (for example, other servers or mobile terminals). If the generative AI module 300 is located on an external device (external server), the device communication unit 190 also communicates with the external device.

[0199] The device control unit 150 checks whether the command word input to the home appliance 100 corresponds to direct control.

[0200] If, upon verification, the input command word does not correspond to direct control, the device control unit 150 extracts an example sentence from the database 120 that has a similarity to the input command word above a certain standard and inputs it into the generative AI module 300.

[0201] If the input command corresponds to direct control, the device control unit 150 can generate a control command corresponding to the input command without inputting the command to the generative AI module 300.

[0202] The device control unit 150 generates a control command using the calculated result if the input command word, as calculated by the generative AI module 300, corresponds to post-interpretational control. See the category explanation in Figure 4 for details.

[0203] The home appliance 100 then controls the function module 180 to provide functions according to the generated control commands.

[0204] When applying embodiments of the present invention, speech recognition errors can be improved, and user commands can be recognized flexibly. Furthermore, situation-specific customized control allows multiple functions of home appliances to be controlled and operated through a single spoken command. In addition, even when the user inputs not only defined keywords and control sentences, but also undefined keywords and non-control sentences, the server 500 can recognize these and control the home appliances. Moreover, routines pre-stored on the server (e.g., ThinQ routines) can be executed as voice / text command utterances.

[0205] Furthermore, to improve the voice recognition capabilities of home appliances, a separate generative AI module 300 (e.g., Chat GPT API) can be utilized, enabling recognition of utterances that could not be recognized based on existing defined command words. In addition, for utterances that invoke ThinQ routines, which are a customer's custom mode, the home appliance can be controlled by utilizing routines stored on the server (e.g., routines stored on the ThinQ Cloud server).

[0206] The various types of data that a prompt can contain, that is, the data included in a prompt, may include identification information about home appliances owned by the user, and configuration information about home appliances owned by the user. For example, identification information about home appliances refers to devices actually owned by the user, such as home appliances registered on the server or home appliances registered in the user account on the server.

[0207] To generate accurate prompts, the prompts may include important information specific to each home appliance, such as various pieces of information about the appliance's configuration.

[0208] A method for controlling a home appliance based on an instruction word according to one embodiment of the present invention includes: a first step in which a server or home appliance checks whether the input instruction word corresponds to direct control; a second step in which, if the input instruction word in the first step does not correspond to direct control, the server or home appliance extracts an example sentence from a database whose similarity to the input instruction word is above a certain standard and inputs it to a generative AI module; a third step in which, if the input instruction word corresponds to post-interpretational control as calculated by the generative AI module, the server or home appliance generates a control command using the calculated result; and a fourth step in which the home appliance provides a function according to the generated control command.

[0209] A server that controls a home appliance based on a command word according to one embodiment of the present invention includes a server control unit that makes a judgment on the input command word and generates a corresponding control command, and a server communication unit that communicates with a database containing a large number of example sentences and the home appliance. After the server communication unit receives the command word input to the home appliance from the home appliance, the server control unit checks whether the input command word corresponds to direct control. If the input command word does not correspond to direct control, the server control unit extracts an example sentence from the database whose similarity to the input command word is above a certain standard and inputs it to a generative AI module. If the input command word in the result calculated by the generative AI module corresponds to post-interpretational control, the server control unit generates a control command using the calculated result, and the server communication unit transmits the control command to the home appliance so that the home appliance can provide functions according to the generated control command.

[0210] In this method, the electronic device is a consumer electronics product, and the method may further include a fourth operation that provides the consumer electronics product with a function corresponding to a generated control command. This method may further include providing the home appliance with a function corresponding to the control command that corresponds to the input command, if the input command directly corresponds to the control in the first operation. In this method, the sample text may have a similarity to the input command that is equal to or greater than a threshold value, based on BM25 or cosine similarity. This method may further include, if the input command corresponds to post-interpretation control in the result generated by the generating AI module in the third operation, generating a control command using the generated result by an electronic device. This method may further include, if the input command corresponds to a post-interpretation routine or control in the result generated by the generating AI module in the third operation, generating a control command corresponding to a routine included in the generated result by an electronic device. This method may further include, in the third operation, if the input command corresponds to a post-interpretation chat in the result generated by the generating AI module, generating a control command by an electronic device to output the generated result in voice or text. In this method, the third operation further includes generating a control command using intent control commands and slot control commands included in the generated result, and the second operation may include, by an electronic device, inputting to the generating AI module a sample text containing one or more intent control commands and one or more slot control commands necessary to control a home appliance, along with the input command.

[0211] In one or more embodiments, the input commands entered by the user may correspond to a pre-configured format. In one or more embodiments, the input commands provided by the user may be classifiable into direct control input command categories and post-interpretation input command categories. In one or more embodiments, a guide including categories may be provided. If the input command corresponds to direct control, the server control unit does not need to input the input command to the generation AI module. The sample text may have a similarity of BM25 or higher to the input command based on cosine similarity. In the third process, if the input command corresponds to post-interpretation control among the results generated by the generation AI module, the server control unit may generate a control command using the generated results. Also, if the input command corresponds to post-interpretation control or a routine among the results generated by the generation AI module, the server control unit may generate a control command corresponding to the routine included in the generated results.

[0212] In the third operation described above, if the input command corresponds to a post-interpretation chat in the result generated by the generating AI module, the server control unit can generate a control command to output the generated result in voice or text. The server control unit can generate a control command using intent control commands and slot control commands included in the generated result, and the server control unit can input sample text, sample text, and input commands to the generating AI module, which include one or more intent control commands and one or more slot control commands necessary to control the home appliance. In one or more embodiments, a method is provided for controlling a home appliance in response to an input command provided by a user, or for controlling a home appliance that communicates with a server. The home appliance or server is configured to recognize input commands having a set format, and the input commands provided by a user can be classified into a direct control input command category and a post-interpretation input command category, the direct control input command category represents input commands that have a set format, are directly recognized by the home appliance or server, and can generate a corresponding control command, and the post-interpretation input command category represents input commands other than those in the direct control input command category, and represent input commands that require interpretation in order to generate a corresponding control command. Furthermore, the post-interpretation input command category includes one or more of the post-interpretation control category, post-interpretation routine category, and post-interpretation chat category, as well as combinations thereof.

[0213] In one or more embodiments, the method may include the steps of: determining whether an input command provided by a user corresponds to a direct control input command category by a home appliance or server; if the input command provided by a user does not correspond to a direct control input command category, inputting the input command and prompt provided by the user to a generating artificial intelligence module; the prompt including a guide containing information for distinguishing which post-interpretation input command category the input command belongs to, and sample text having a similarity of a predetermined threshold or higher to the input command, wherein the sample text is extracted from a database by the home appliance or server; receiving the result generated by the generating artificial intelligence module by the home appliance or server; the generated result identifying the post-interpretation input command category of the input command provided by the user; and generating a control command using the generated result by the home appliance or server. In one or more embodiments, when determining whether an input command provided by a user corresponds to a direct control input command category, the home appliance or server may determine a match between the input command provided by the user and one or more predefined input commands pre-configured in the home appliance or server.

[0214] In one or more embodiments, when determining whether an input command provided by a user corresponds to a direct control input command category, the home appliance or server determines the degree of match between the input command provided by the user and one or more predefined input commands pre-configured on the home appliance or server, and if the degree of match is greater than or equal to a threshold, it can determine that the input command provided by the user corresponds to a direct control input command category. In one or more embodiments, if an input command provided by a user corresponds to a direct control input command category, the home appliance or server generates and executes a corresponding control command based on the input command provided by the user, and the home appliance performs a function or operation corresponding to the executed control command. In one or more embodiments, if an input command provided by a user falls under the category of a direct control input command, the input command is not input to the generation AI module. In one or more embodiments, the generated result may include information that the home appliance or server can recognize as an input command in a configured format, or as a control command that can be executed to perform a function or operation on the home appliance, and the home appliance or server uses this information to generate a control command when generating a control command using the generated result.

[0215] In one or more embodiments, a home appliance or server executes a generated control command using the generated result, and the home appliance performs a corresponding function or operation in response to the execution of the control command. In one or more embodiments, sample text has a similarity to an input command of a threshold value or higher, based on BM25 or cosine similarity. In one or more embodiments, the home appliance or server can determine the similarity of the sample text. Furthermore, the home appliance or server may generate a control command to control a function or operation of the home appliance if the generated result specifies that the post-interpretation input command category of the input command provided by the user corresponds to a post-interpretation control. In one or more embodiments, generating a control command using the generated result may include the home appliance or server generating a control command corresponding to a routine for one or more functions or operations performed by the home appliance, if the generated result specifies that the post-interpretation input command category of the input command provided by the user corresponds to a post-interpretation routine.

[0216] In one or more embodiments, generating control commands using the generated results may further include generating control commands for the home appliance or server to output chat or announcement, or voice or text response by the home appliance, if the generated results specify a post-interpretation input command category for which the input command provided by the user corresponds to a post-interpretation chat.

[0217] In one or more embodiments, inputting user-provided input commands and prompts into a generating AI module may include inputting sample text into the generating AI module, which includes one or more intent control commands and one or more slot control commands necessary for controlling the appliance, by the appliance or server. In one or more embodiments, generating control commands using the generated results may further include generating control commands using the intent control commands and slot control commands contained in the generated results, or using the information for generating the intent control commands and slot control commands contained in the generated results. In one or more embodiments, input commands may be provided to the appliance by the user via text and / or voice.

[0218] In another embodiment, a server is provided that controls a home appliance in response to input commands provided by a user. The server comprises a server communication unit configured to communicate with the home appliance, a server communication unit configured to receive input commands entered by the user into the home appliance and to provide control commands to the home appliance, a database storing a plurality of sample texts, and a server control unit configured to receive input commands from the server communication unit and to generate control commands, wherein the server control unit is configured to perform a method according to any one of the features of the above-described embodiment or an optional embodiment. In another embodiment, a home appliance is provided that performs a function or operation in response to input commands provided by a user, the home appliance comprising a device communication unit configured to receive input commands entered by the user into a remote controller that communicates with the home appliance or a device communication unit, a database storing a plurality of sample texts, and a device control unit configured to receive input commands from the device communication unit and to generate control commands, wherein the device control unit is configured to perform a method according to any one of the features of the above-described embodiment or an optional embodiment. In another embodiment, a system is provided that includes a server and home appliances configured to communicate with each other.

[0219] In this application, the input commands entered by the user may be default commands or predefined commands that are pre-configured in the home appliance 100 and / or server 500, or may be similar to or identical to them. For example, the home appliance and / or server may include, or have, a database containing one or more default commands or predefined commands (e.g., a list of input commands programmed or predefined by the home appliance manufacturer or user to be entered by the user to perform the corresponding function of the home appliance) and one or more corresponding control commands. The control commands are associated with one or more functions or operations of the home appliance, and therefore, when the home appliance executes a control command, the home appliance performs the associated function or operation of the home appliance.

[0220] Such user input commands that can be recognized by a home appliance / server as default / programmed / defined / predefined input commands are sometimes called direct control, direct control commands, defined input commands, or input commands that are categorized or listed in the direct control category of input commands, or recognized input command category. This is because such user commands can be easily recognized by home appliances, and their corresponding control commands are determined / recognized / selected / generated by the home appliance / server and executed on the home appliance by the home appliance / server to perform the relevant or specified function or operation of the home appliance.

[0221] Alternatively, an input command entered by the user may differ from, be similar to, or not match any of the default / programmed / defined / predefined input commands, and therefore, the corresponding control command cannot be determined directly from the input command. Such input commands are sometimes called input commands requiring interpretation (i.e., input commands to be interpreted, non-default input commands, input commands that deviate from default commands, or undefined input commands). In the case of undefined input commands, i.e., input commands that could not be determined or recognized as direct control commands by the appliance / server, i.e., input commands that do not correspond to direct control, this method uses the generating artificial intelligence module to interpret the input command based on the input command and additional information provided to the generating AI module from the server / appliance (i.e., prompts) (e.g., one or more of base or base information, guide or guide information, sample or sample information), and as a result, the generating AI module produces a result. Input commands requiring interpretation are classified into indirect control categories, unrecognized input command categories, or post-interpretation categories, which may include one or more post-interpretation control categories, post-interpretation routine categories, and post-interpretation chat categories.

[0222] Prompts provided to a generating AI module from a home appliance and / or server can instruct, restrict, or command the generating AI to provide / generate results such that the generated results resemble default / programmed / defined / predefined input commands recognizable by the home appliance / server, and thus cause the home appliance / server to determine / recognize / generate corresponding control commands to perform a function or operation, or the generated results resemble control commands executable on the home appliance to perform a function or operation, and thus cause the home appliance / server to execute control commands to perform an associated / specified / corresponding function or operation of the home appliance.

[0223] Although it has been stated that all components constituting embodiments of the present invention are combined into one or operate in combination, the present invention is not necessarily limited to these embodiments, and any components within the scope of the present invention can also be selectively combined into one or more components and operate. Furthermore, although each component can be embodied in a single independent piece of hardware, some or all of the components can be selectively combined to be embodied as a computer program having program modules that perform some or all of the combined functions in one or more pieces of hardware. The code and code segments constituting the computer program can be easily inferred by those skilled in the art of the present invention. These computer programs can embody embodiments of the present invention by being stored on a computer-readable medium and read and executed by a computer. Storage media for computer programs include magnetic recording media, optical recording media, and storage media containing semiconductor recording elements. Furthermore, computer programs embodying embodiments of the present invention include program modules that are transmitted in real time via an external device.

[0224] The above description has focused on embodiments of the present invention, but various modifications and variations can be made at the level of an ordinary engineer. Therefore, as long as these modifications and variations do not deviate from the scope of the present invention, they can be understood to be included within the scope of the present invention.

Claims

1. The first stage involves verifying whether the input command word to the electronic device corresponds to direct control, In the first stage, if the input command word does not correspond to direct control, the electronic device inputs, along with the command word, an example sentence extracted from a database containing a category for the command word and whose similarity to the input command word is above a certain standard into the generative AI module. The results calculated via the generative AI module include the category of the input command word, and the third stage involves generating a control command using the calculated results. including, A method of controlling home appliances based on command words.

2. The electronic device is a home appliance and includes a fourth stage in which it provides a function according to the generated control command. A method for controlling a home appliance based on the command word described in claim 1.

3. In the first stage, if the input command word corresponds to direct control, The aforementioned home appliance further includes the step of providing a function in accordance with a control command corresponding to the input command word, A method for controlling a home appliance based on the command word described in claim 1.

4. The above example sentence is characterized in that the input command word, BM25, or cosine similarity score is above a certain threshold. A method for controlling a home appliance based on the command word described in claim 1.

5. In the XL3 stage, If the input command word calculated by the generative AI module corresponds to post-interpretation control, the electronic device includes the step of generating a control command using the calculated result. A method for controlling a home appliance based on the command word described in claim 1.

6. If the input command word in the result calculated by the generative AI module corresponds to a post-interpretation routine, the electronic device further includes the step of generating a control command corresponding to the routine included in the calculated result. A method for controlling a home appliance based on the command word described in claim 1.

7. If the input command word in the result calculated by the generative AI module corresponds to a Post-Interpretation Chat, the electronic device further includes the step of generating a control command to output the calculated result in voice or text. A method for controlling a home appliance based on the command word described in claim 1.

8. XL Stage 3 is, The electronic device further includes the step of generating the control command using the intent control command and slot control command included in the calculated result. A method for controlling a home appliance based on the command word described in claim 1.

9. The second stage described above is, The electronic device includes an example statement containing one or more intent control commands and one or more slot control commands necessary for controlling the home appliance, and a step of inputting the input command words to the generative AI module. A method for controlling a home appliance based on the command word described in claim 8.

10. The above example sentence includes a sentence for controlling the home appliance and a set of corresponding intent control instructions and slot control instructions. A method for controlling a home appliance based on the command word described in claim 9.

11. The set of intent control instructions and slot control instructions instructs the function to be performed by the home appliance. A method for controlling a home appliance based on the command word described in claim 9.

12. A server control unit that makes a judgment about the input command word and generates a corresponding control command, A database containing numerous example sentences, It includes a server communication unit that communicates with home appliances, After the server communication unit receives a command word input to the home appliance from the home appliance, If the command word input to the home appliance does not correspond to Direct Control, the server control unit inputs the command word, along with a guide including the category for the command word and example sentences extracted from the database whose similarity to the input command word is above a certain standard, into the generative AI module. The results calculated by the generative AI module include the category of the input instruction word, and control instructions are generated using the calculated results. The server communication unit transmits the control commands to the home appliance so that the home appliance can provide functions in accordance with the generated control commands. A server that controls home appliances based on command words.

13. If the input command word corresponds to direct control, The server control unit is characterized by not inputting the input command words to the generative AI module. A server that controls home appliances based on the command words described in claim 12.

14. The above example sentence is characterized in that its similarity to the input command word, calculated using BM25 or cosine similarity, is above a certain threshold. A server that controls home appliances based on the command words described in claim 12.

15. If the input command word calculated by the generative AI module corresponds to post-interpretation control, the server control unit generates a control command using the calculated result. A server that controls home appliances based on the command words described in claim 12.

16. If the input command word in the result calculated by the generative AI module corresponds to a post-interpretation routine, the server control unit generates a control command corresponding to the routine included in the calculated result. A server that controls home appliances based on the command words described in claim 12.

17. If the input command word in the result calculated by the generative AI module corresponds to a Post-Interpretation Chat, the server control unit generates a control command to output the calculated result in voice or text. A server that controls home appliances based on the command words described in claim 12.

18. The server control unit generates the control command using the intent control command and slot control command included in the calculated result. A server that controls home appliances based on the command words described in claim 12.

19. The server control unit inputs an example statement containing one or more intent control commands and one or more slot control commands necessary for controlling the home appliance, the example statement, and the input command words to the generative AI module. A server that controls home appliances based on the command words described in claim 18.

20. The above example sentence includes a sentence for controlling the home appliance and a set of corresponding intent control instructions and slot control instructions. A server that controls home appliances based on the command words described in claim 19.

21. The set of intent control instructions and slot control instructions instructs the function to be performed by the home appliance. A server that controls home appliances based on the command words described in claim 19.

Images