Electronic device and control method therefor

Abstract

This electronic device comprises: a memory for storing a plurality of pieces of primitive task information, a plan pool including pieces of code information generated by combining primitive tasks, and a large language model (LLM); and a processor for generating response information corresponding to a user language command when information corresponding to the user language command is acquired, wherein the processor checks whether the task corresponding to the user language command is included in the plan pool by using the large language model, and generates the response information by using the code information in the plan pool when the task corresponding to the user language command is included in the plan pool.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device and a method for controlling the same that can provide deterministic response information to a user even when using a large-scale language model (LLM).

[0002] Recently, event information is being extracted from voice or text input through message applications (or messenger applications, chat applications) or voice recognition applications, and services corresponding to the extracted event information (e.g., schedule management services, reservation services, shopping services, etc.) are being provided.

[0003] Recently, large-scale language models have been introduced, and the services described above are also considering using large-scale language models.

[0004] An electronic device according to one embodiment of the present disclosure includes a memory storing a plan pool including a plurality of primitive task information, code information generated by combining the primitive tasks, and a large language model (LLM), and a processor that generates response information corresponding to a user language command when information corresponding to the user language command is acquired.

[0005] The processor can use the large-scale language model to determine whether a task corresponding to the user language command is included in the plan pool, and if the task corresponding to the user language command is included in the plan pool, can generate response information using code information within the plan pool.

[0006] The processor may generate code information corresponding to the user language command using the large-scale language model and the plurality of primitive task information, if a task corresponding to the user language command is not included in the plan pool.

[0007] The above processor can obtain an execution result of the generated code information using the generated code information, and can confirm whether the obtained execution result corresponds to the user language command.

[0008] The processor may repeat the process of generating code information if the acquired execution result does not correspond to the user language command, and may include the generated code information in the plan pool if the acquired execution result corresponds to the user language command.

[0009] The processor can generate candidate tasks by combining primitive tasks in various ways using the large-scale language model, check whether the generated candidate tasks are included in the plan pool, and if the generated candidate tasks are not included in the plan pool, include the generated candidate tasks in the plan pool.

[0010] The above processor can repeatedly generate the candidate tasks when a new primitive task is added or when the number of stored code information in the plan pool is less than or equal to a preset number.

[0011] The above code information may include a first program code for performing a primitive task and a second program code for handling an exception of the first program code.

[0012] The electronic device further includes a microphone and a speaker, and the processor recognizes a user's spoken voice acquired through the microphone, and when response information corresponding to the voice recognition result is generated, the processor can control the speaker to output guidance information corresponding to the response information.

[0013] The electronic device further includes a manipulation input device and a display, and the processor can control the display so that when text information is input through the manipulation input device, response information corresponding to the input text information is generated and a screen corresponding to the response information is displayed.

[0014] The electronic device further includes a communication device that performs communication with an external device, and the processor can control the communication device to transmit a program code executable by the external device to the external device when information corresponding to the user language command is received.

[0015] A control method in an electronic device according to one embodiment of the present disclosure includes a step of storing a plan pool including a plurality of primitive task information and code information generated by combining the primitive tasks, a step of obtaining information corresponding to a user language command, a step of using a large language model (LLM) to determine whether a task corresponding to the user language command is included in the plan pool, and a step of using code information in the plan pool to generate response information.

[0016] The present control method may further include a step of generating code information corresponding to the user language command using the large-scale language model and the plurality of primitive task information, if a task corresponding to the user language command is not included in the plan pool.

[0017] The present control method may further include a step of obtaining an execution result of the generated code information using the generated code information, and a step of confirming whether the obtained execution result corresponds to the user language command.

[0018] The present control method may further include a step of repeating a process of generating new code information if the acquired execution result does not correspond to the user language command, and a step of including the generated code information in the plan pool if the acquired execution result corresponds to the user language command.

[0019] The above-described storing step includes a step of generating candidate tasks by combining primitive tasks in various ways using the large-scale language model, a step of checking whether the generated candidate tasks are included in the plan pool, and a step of including the generated candidate tasks in the plan pool if the generated candidate tasks are not included in the plan pool.

[0020] The step of generating the above candidate task may repeatedly generate the candidate task when a new primitive task is added or the number of stored code information in the plan pool is less than or equal to a preset number.

[0021] The above code information may include a first program code for performing a primitive task and a second program code for handling an exception of the first program code.

[0022] The step of obtaining information corresponding to the user language command may further include obtaining the information by recognizing the user's spoken voice through a microphone, and the control method may further include a step of outputting guidance information corresponding to the response information through a speaker.

[0023] The step of obtaining information corresponding to the user language command may further include a step of receiving the information from an external device, and the control method may further include a step of transmitting a program code corresponding to the response information and executable on the external device to the external device.

[0024] Meanwhile, in a non-transitory computer-readable recording medium including a program for executing a control method of an electronic device according to an embodiment of the present disclosure, the control method of the electronic device includes a step of storing a plan pool including a plurality of primitive task information and code information generated by combining the primitive tasks, a step of obtaining information corresponding to a user language command, a step of using a Large Language Model (LLM) to determine whether a task corresponding to the user language command is included in the plan pool, and a step of using code information in the plan pool to generate response information if the task corresponding to the user language command is included in the plan pool.

[0025] The above-described and other aspects, features, and advantages of the embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0026] FIG. 1 is a diagram showing a conversation system according to one embodiment of the present disclosure;

[0027] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure;

[0028] FIG. 3 is a block diagram illustrating another configuration of an electronic device according to an embodiment of the present disclosure;

[0029] Figure 4 is a diagram for explaining the service provision operation using a large-scale language model.

[0030] FIG. 5 is a diagram illustrating a process for generating plan information according to an embodiment of the present disclosure;

[0031] Figure 6 is a diagram illustrating an example of information of a primitive task used in the present disclosure.

[0032] FIG. 7 is a diagram illustrating an example of a plan code generated according to one embodiment of the present disclosure;

[0033] FIG. 8 is a flowchart for explaining the control operation of an electronic device according to an embodiment of the present disclosure, and

[0034] FIG. 9 is a flowchart illustrating a method for generating response information according to one embodiment of the present disclosure.

[0035] Hereinafter, various embodiments of this document are described with reference to the attached drawings. However, this is not intended to limit the technology described in this document to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of this document are included. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0036] In this document, the expressions "has," "may have," "includes," or "may include" indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), but do not exclude the presence of additional features.

[0037] In this document, the expressions "A or B," "at least one of A and / or B," or "one or more of A or / and B" can include all possible combinations of the listed items. For example, "A or B," "at least one of A and B," or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) both at least one A and at least one B are included.

[0038] The expressions "first," "second," "first," or "second," etc., used in this document can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When a component (e.g., a first component) is referred to as being "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be connected via another component (e.g., a third component). Conversely, when a component (e.g., a first component) is referred to as being "directly coupled to" or "directly connected to" another component (e.g., a second component), it should be understood that no other component (e.g., a third component) exists between said component and said other component.

[0040] The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" does not necessarily mean something is "specifically designed to" in hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device, together with other devices or components, is "capable of." For example, the phrase "a coprocessor configured (or set) to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing those operations, or a generic-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0041] In this disclosure, the term "value" is defined as a concept that includes not only scalar values ​​but also vectors. In this disclosure, expressions such as "produce" and "calculate" may be replaced with expressions that generate the result of the corresponding production or calculation.

[0042] Meanwhile, the conversation history information of the present disclosure refers to text information comprised of responses from a user and an electronic device within a conversation window. Such conversation history information may be a conversation history between a single user and a single response bot, or between multiple users and a single response bot. Here, the response bot is an automated machine program that provides a specific service at the user's request. Furthermore, this conversation history information includes not only text input directly from the user but also voice input.

[0043] And in the present disclosure, character data means data excluding numerical data such as real numbers and imaginary numbers, and includes not only data composed of text, but also all forms of data (e.g., image data, voice data) that can convert specific information into numerical values.

[0044] The mathematical operations and calculations of each step of the present disclosure described below can be implemented as computer operations by a known coding method for performing the operations or calculations and / or by coding designed to suit the present disclosure.

[0045] Hereinafter, the present invention will be described in detail with reference to the drawings.

[0046] FIG. 1 is a diagram illustrating a conversation system according to one embodiment of the present disclosure.

[0047] Referring to FIG. 1, a conversation system (1000) may include an electronic device (100) and a server (200).

[0048] The electronic device (100) can receive conversation information from a user and transmit the received conversation information to a server (200).

[0049] Here, ‘conversation information’ can be text directly entered by the user, or the result of voice recognition that converts the user’s spoken voice into text through voice recognition.

[0050] Here, "voice recognition" involves the process of converting a user's voice into a form that an electronic device can process. For example, speech recognition involves converting the acoustic speech signal acquired by an electronic device into text, such as words or sentences, and may also be referred to as computer speech recognition or speech-to-text (STT).

[0051] This conversational information may only include spoken speech or conversational text input from the current user, and in some cases, may also include content from previous conversations. This conversational information may also be referred to as a dialogue or prompt. This conversational information may include the user's verbal commands requesting the execution (or search) of a specific function.

[0052] These user verbal commands can be intuitive and explicit, such as "Turn on the air conditioner," or more abstract, such as "It's hot." Furthermore, user verbal commands can be complex, requiring the processing of multiple tasks, such as "Turn on the TV and set it to channel 11."

[0053] The electronic device (100) can output the response information received from the server (200). For example, the electronic device (100) can output the response information audibly or on a screen. Such output methods may be based on user commands.

[0054] The server (200) can receive conversation information from the electronic device (100), generate response information corresponding to the received conversation information, and transmit the response information to the electronic device (100).

[0055] At this time, the server (200) can generate the aforementioned response information using a large-scale language model. For example, if the received conversation information includes the text "Play music A," the server (200) can use the LLM to obtain a program code recognizable by the electronic device (100) and transmit the obtained program code to the electronic device (100). More specific operations are described with reference to FIG. 4.

[0056] Here, "LLM (Large Language Model)" can be referred to as a large-scale language model, an artificial intelligence model that utilizes machine learning (or machine learning) techniques to understand and generate human language. Such LLMs can utilize deep learning models that process, analyze, and predict complex data using multilayer neural networks to generate natural language responses. LLMs can also be expressed as deep language models, AI language models, deep language networks, and chat models.

[0057] Meanwhile, LLMs are a type of artificial intelligence that generates content based on input. They have the advantage of being able to actively generate results based on the user's specific needs. However, LLMs have two problems when applied to conversational services. First, they require long processing times to obtain responses. Second, while conversational responses require deterministic responses, they often provide non-deterministic responses.

[0058] In other words, conversational or voice recognition services require immediate processing of user commands, and deterministic processing (i.e., consistent results every time) is required. However, as described above, LLM presents two challenges that make it difficult to directly apply to conversational services.

[0059] To address these issues, the present disclosure utilizes a plan pool. Specifically, a server or electronic device utilizing LLM can predict various user inquiries or questions, and create a pool by pre-generating codes corresponding to each prediction.

[0060] Here, the plan pool is a pool that stores information about code information generated by LLM. It can be composed of multiple code information and a lookup table for indexing (or searching) each piece of code information. This lookup table can include characteristic information corresponding to each piece of code information (e.g., primitive codes used in the code information, examples of using the code, descriptive information, etc.), and location information where the code information is stored.

[0061] Code information is information about primitive tasks and specific objects, or codes that combine multiple primitive tasks, and is the program code required to execute a user-requested command. Here, a primitive task is a basic unit of program control controlled by the operating system of an electronic device.

[0062] For example, to perform a user request such as "Call A~", an action is required to find a contact A and an action is required to call that contact A. Therefore, code information that combines a first code that uses a primitive task for finding a contact and a second code that uses a primitive task for making a call can be generated by LLM. Meanwhile, this is because the user request is related to the execution of a specific service, and a primitive task for retrieving specific information can be used to use a ~ search site and a primitive task for determining the search purpose.

[0063] Additionally, the aforementioned programming languages ​​may vary, and the electronic device may utilize code in a programming language that can be executed on the device into which the user's command is input. That is, if the electronic device and the server can use a common program, code in a common programming language may be utilized. If the user terminal device cannot recognize the programming language corresponding to the code in the server (or LLM), the LLM may additionally perform an operation of converting pre-generated code information into a programming language that the user terminal device can recognize.

[0064] This behavior can also be used as a translation mechanism. For example, if the server's LLM model operates in English, or if the user's terminal device's default language is a different language, such as Korean, information can be translated and provided in the user's terminal's default language.

[0065] The electronic device (100) and server (200) can perform the aforementioned response using a pre-created code pool. For example, if the user's request corresponds to a code within the pre-created pool, the response process can be performed using that pool. If the user's request does not exist within the plan pool, a plan corresponding to the request can be created using the existing method, and the created code can be added to the pool.

[0066] By using a pre-made plan like this, responses are performed according to user requests, so that a uniform response (i.e., a deterministic response) can be performed instead of a different response for each user request.

[0067] Additionally, it takes a lot of time or resources to create code that responds to the user requests described above, but by generating this code in advance, responses to user requests can be performed more quickly.

[0068] More specific operations of the conversation service according to the present disclosure are described with reference to FIGS. 2 and 3.

[0069] Meanwhile, in FIG. 1, the electronic device (100) and the server (200) are depicted as being directly connected, but when implemented, they may be connected via a router, repeater, etc. In addition, multiple electronic devices, not just one, may be connected to the server.

[0070] In addition, although the conversation system in Figure 1 is depicted as consisting of two devices, when implemented, it may be implemented in a form where a user's request is received and processed by one device.

[0071] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.

[0072] Specifically, in the system of FIG. 1, devices that directly generate response information, such as a server, and devices that collect conversation messages or voices and generate necessary conversation information, may be referred to as electronic devices. That is, the electronic device of FIG. 2 can operate as the user terminal device of FIG. 1 or the server of FIG. 1. Furthermore, it is also possible to perform the functions of the user terminal device and the server of FIG. 1 in a single device.

[0073] The electronic device (100) of FIG. 2 may be a variety of devices such as a personal computer (PC), laptop, smartphone, tablet, server, AI speaker, vehicle infotainment, TV, set-top box, and home appliance.

[0074] Referring to FIG. 2, the electronic device (100) may include a memory (110) and a processor (120).

[0075] Memory (110) is a component for storing O / S, various software, data, etc. for driving an electronic device (100). Memory (110) can be implemented in various forms such as RAM, ROM, flash memory, HDD, external memory, memory card, etc., and is not limited to any one.

[0076] The memory (110) stores at least one instruction. These instructions (or commands) may include those for executing various applications necessary for performing the above-described conversation system, applications for voice recognition, and services corresponding to response information.

[0077] The memory (110) stores information on a plurality of primitive tasks. Here, a "primitive task" is the smallest unit of a function (or code) performed in an electronic device, and the primitive task information may include the name of the task and the code corresponding to the task. Examples of primitive tasks are described below in FIG. 6.

[0078] The memory (110) stores a plan pool. Specifically, the plan pool is a pool that stores multiple pieces of code information. The code information may refer to a code (or a predefined plan) generated using the aforementioned primitive task. This code information may be a combination of a primitive task and a specific object (or objects), a combination of multiple primitive tasks, or a combination of multiple primitive tasks and a single object.

[0079] Additionally, the code information may include a first program code for performing a primitive task and a second program code for handling exceptions to the first program code. Such exceptions are described in detail in FIG. 7.

[0080] Meanwhile, the memory embedded in the electronic device (100) is implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard disk drive, or solid state drive (SSD)), and the memory that can be detachably attached to the electronic device (100) is implemented as a form such as a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. Can be.

[0081] Meanwhile, in the illustrated example, the electronic device (100) is depicted as being composed of one memory, but when referring to volatile memory and non-volatile memory separately, the electronic device (100) may be referred to as including multiple memories.

[0082] The processor (120) can perform overall control operations of the electronic device (100). Specifically, the processor (120) has a function of controlling the overall operations of the electronic device (100).

[0083] The processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an ARM processor, or may be defined by the relevant terminology. In addition, the processor (120) may be implemented as a system on chip (SoC), a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA) having a built-in processing algorithm. In addition, the processor (120) may perform various functions by executing computer executable instructions stored in a memory. Meanwhile, in FIG. 2, only one processor is illustrated in the electronic device (100), but the implementation A poem may contain multiple processors (e.g., CPU + GPU, CPU + DSP).

[0084] The processor (120) can pre-create a plan pool using a large-scale language model. Specifically, since, as previously described, responding using a large-scale language model (more precisely, directly generating code at each step, as described in FIG. 4 ) takes time, the processor (120) can predict candidate commands corresponding to speech commands (or requests) likely to be used by the user in advance and pre-create code information corresponding to the candidate commands.

[0085] For example, the processor (120) may generate candidate tasks by combining primitive tasks in various ways using a large-scale language model, check whether the generated candidate tasks are included in the plan pool, and if the generated candidate tasks are not included in the plan pool, include the generated candidate tasks in the plan pool. This operation is described in more detail in FIG. 5. Meanwhile, this operation of generating a plan pool may also be referred to as a learning process.

[0086] Meanwhile, the generation of such candidate instructions can be performed continuously when a new primitive task is added or the stored code information in the plan pool is less than a preset number.

[0087] When a conversational content (or prompt) is received, the processor (120) can generate response information using the received conversational content. For example, the conversational content described above may include information corresponding to a user language command.

[0088] In this case, the processor (120) can use a large-scale language model to check whether a task corresponding to the user language command is included in the plan pool, and if a task corresponding to the user language command is included in the plan pool, response information can be generated using code information within the plan pool.

[0089] If a task corresponding to a user language command is not included in the plan pool, the processor (120) may generate code information corresponding to the user language command using a large-scale language model and information on multiple primitive tasks, as before. That is, code information may be generated using the existing method described in FIG. 4.

[0090] At this time, the processor (120) can obtain an execution result of the generated code information using the generated code information and check whether the obtained execution result corresponds to the user language command.

[0091] And, if the acquired execution result does not correspond to the user language command, the processor (120) may repeat the process of generating code information, and if the acquired execution result corresponds to the user language command, the generated code information may be included in the plan pool.

[0092] Meanwhile, although FIG. 2 illustrates the electronic device (100) as including only basic components (i.e., memory and processor), the electronic device (100) may include various components in addition to the aforementioned components. Such examples are described below with reference to FIG. 3.

[0093] FIG. 3 is a block diagram illustrating another configuration of an electronic device according to an embodiment of the present disclosure.

[0094] Referring to FIG. 3, the electronic device (100) may further include a communication device (130), memory (110), processor (120), display (150), operating input device (140), and speaker (170).

[0095] As for the memory (110), it has been previously described in Fig. 2, so a duplicate description will be omitted.

[0096] The communication device (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication device (130) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0097] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a connection before various other information can be transmitted and received.

[0098] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which lies between visible light and millimeter waves.

[0099] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0100] In addition, the communication device (130) may include at least one of wired communication modules that perform communication using a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, an optical fiber cable, or a UWB (Ultra Wide-Band) module. Such a communication device (130) may also be referred to as a transceiver.

[0101] A communication device (130) can receive conversation content from another device or transmit conversation content to another device. The conversation content may include a single user message or a message entered through multiple turns. The user message may be text directly typed by the user, a message converted into text through voice recognition from the user's spoken voice, or voice data corresponding to the user's spoken voice.

[0102] The communication device (130) can transmit response information corresponding to the conversation content to another device that transmitted the conversation content, or receive response information corresponding to the conversation content. This response information may include command information (i.e., code) for executing a specific service, content corresponding to information requested by a user, or guidance information regarding the execution results of a service requested by a user.

[0103] For example, if the conversation content includes a message requesting the user to launch a specific application, response information containing command information for launching the specific application may be received. Alternatively, if the conversation content includes a message requesting / reserving a specific service, such as a hotel reservation, taxi reservation, or ordering a specific item, information regarding the result of the order or reservation requested by the user may be received as response information. The above-described response information examples are merely examples, and various user requests and corresponding responses that may occur in the conversation system may be implemented.

[0104] The operating input device (140) can receive function selections and control commands for the corresponding functions from the user of the electronic device (100). The operating input device (140) may be a keyboard, mouse, touch pad, etc.

[0105] The display (150) displays a user interface window for selecting a function supported by the electronic device (100). Specifically, the display (150) may display a user interface window for selecting various functions provided by the electronic device (400). The display (150) may be a monitor such as an LCD (liquid crystal display), an OLED (Organic Light Emitting Diodes), etc., and may also be implemented as a touch screen capable of simultaneously performing some of the functions of the above-described operation input device (140).

[0106] The display (150) can display content (or code execution results) corresponding to response information or display current conversation history information.

[0107] A microphone (160) can receive a user's voice. Specifically, the microphone (160) may include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0108] When a user's voice is input through this microphone (160), the processor (120) can confirm the user's voice content through voice recognition and perform an action corresponding to the voice content.

[0109] Here, "voice recognition" involves the process of converting a user's voice into a form that an electronic device can process. For example, speech recognition involves converting the acoustic speech signal acquired by an electronic device into text, such as words or sentences, and may also be referred to as computer speech recognition or speech-to-text (STT).

[0110] The speaker (170) outputs sound. The speaker (170) can output response information corresponding to a received response message as sound.

[0111] For example, if the electronic device (100) is implemented as an AI speaker, the processor (120) can convert user speech input through a microphone into text. Then, the processor (120) can output text corresponding to the response information through the speaker (170) using the TTS function.

[0112] If the electronic device (100) is implemented as an AI speaker or a smartphone, etc., the processor (120) continuously monitors whether a voice of a preset trigger command has been input, and if it is determined that the input user speech voice includes a preset trigger command, the processor (120) can execute a conversation service as described above.

[0113] Although FIG. 5 further illustrates other components (e.g., a communication device, an operating input device, a display, a microphone, and a speaker) than those of FIG. 4, the implementation does not necessarily require all of the above-described components to be included, and the implementation may further include only some of the above-described components. For example, if the electronic device (100) is configured as an AI speaker, only a microphone and a speaker may be added.

[0114] As described above, the electronic device according to the present disclosure utilizes a pool of plans while utilizing a large-scale language model, enabling deterministic responses (i.e., outputting the same results each time a call is made) to user requests (or commands). Furthermore, by pre-generating various scenarios using the large-scale language model, execution time can be reduced during the process of utilizing the large-scale language model.

[0115] In addition, the above-described cases can be created using a large-scale language model without the user (or administrator) having to create a rule base, making it easier to create a code pool.

[0116] Figure 4 is a diagram for explaining the service provision operation using a large-scale language model.

[0117] Referring to Fig. 4, the flow of service provision operations using LLM is illustrated, and may include a service module (410), an LLM module (420), and an execution module (430). Each of these modules may be executed on one processor, or each module may be executed on different processors (or different devices).

[0118] First, the service module (410) is a module that manages the language system as a whole, and can control the LLM module (420) and the execution module (430) so that response information is generated when a language command is input from a user.

[0119] For this purpose, the service module (410) can manage data transmission between various components (or modules) within the conversation system.

[0120] The LLM module (420) is a large-scale language model, an artificial intelligence trained to understand and generate human language. This LLM module (420) can be trained to statistically generate the most appropriate output based on various natural language and user sentences. Accordingly, when a user request command (or prompt) is received from the service module (410), the LLM module (420) can output a task (or code) corresponding to the received request.

[0121] When a task like this is output, the service module (410) can pass the task to the execution module (430) to obtain an execution result corresponding to the generated code.

[0122] The execution module (430) can receive execution code and output a result corresponding to the received execution code. For example, if the execution code is an Internet search for a specific topic, the search result can be provided to the service module (410).

[0123] Using the configuration of each module, the operation of the conversation system is described below.

[0124] Meanwhile, the following description assumes that user commands and their corresponding results are output on a single device, but it is also possible to input user commands, etc. on a separate device and receive the input results through communication to perform the operation.

[0125] Meanwhile, there is an utterance from a user that includes a request for a specific function, and the electronic device can acquire the utterance through a microphone and obtain text information corresponding to the utterance through voice recognition.

[0126] Meanwhile, in the past, when such text information was acquired, a process such as natural language understanding was performed to understand the corresponding voice command, but when using the LLM according to the present disclosure, the natural language understanding process is not performed, and the corresponding text information is directly transferred to the LLM.

[0127] Meanwhile, in the above, LLM was described as being performed by receiving text information, but during implementation, it is also possible to use the signal form before speech recognition, and the user's language can be input in various formats, such as when text is input rather than the user's speech.

[0128] When such a user language is entered (1), the service module (410) can transmit the entered user language (or prompt) to the corresponding LLM module (420) (2).

[0129] The LLM module (420) can generate code corresponding to the response information using the input user language. However, this process takes time to generate for long tokens, requires complex user commands to be processed as a hierarchical task, and requires repeated text completion, which further increases the processing time.

[0130] Meanwhile, the above-described service module (410) receives a primary result (i.e., code) from the LLM (4), provides the code to the execution module (430), and can obtain an execution result for the code (5).

[0131] When such execution results are obtained, the service module (410) can provide the results to the LLM module (420) (6).

[0132] At this time, it is possible to check whether the execution result obtained by the LLM module (420) conforms to the user's command.

[0133] Meanwhile, if the result does not match the user's command, the above-described code generation process can be performed again.

[0134] If the user's command is met, the corresponding result can be provided as a response to the user, or the code that generates the response can be provided to the user (7, 8). In other words, if the electronic device performing the above-described operation is a device that directly receives user voice commands, etc., the response based on the service execution result can be provided directly to the user.

[0135] Alternatively, if the result is to be executed on an electronic device other than the electronic device including the service module described above, the code used for the final response described above may be output as a response.

[0136] Meanwhile, since LLM takes a significant amount of time in the 2 / 3 or 6 / 7 steps described above, the present disclosure predicts various commands corresponding to user commands in advance, creates corresponding codes in advance, and stores them in a plan pool, as described above. The operation of creating such a plan pool is described in detail in Figure 5.

[0137] In this disclosure, by preprocessing candidate situations for all cases in which various progress situations as described above are predicted, the time for generating the code described above is skipped in the above-described process, thereby reducing the time in the overall response time.

[0138] FIG. 5 is a diagram illustrating a process for generating plan information according to an embodiment of the present disclosure.

[0139] Referring to FIG. 5, the electronic device stores a plurality of primitive tasks (510). Here, a "primitive task" is the smallest unit of a function (or code) performed by the electronic device, and may include a name of the task, code corresponding to the task, and may describe a description of the task and operations in each mode. Examples of primitive tasks are described below in FIG. 6.

[0140] In this case, when multiple primitive tasks are stored, LLM (530) can generate candidate commands by combining multiple primitive tasks (540). Alternatively, it can generate candidate commands by combining a specific object (e.g., a frequently used object name) with a single primitive task.

[0141] Once a candidate instruction is generated in this manner, a filter (550) can be used to determine whether code information corresponding to the candidate instruction already exists. This filter (550) can be performed by a processor of an electronic device, or can be performed by the aforementioned LLM.

[0142] If there is no code information corresponding to the candidate command in the existing plan pool, code information (560) corresponding to the candidate command can be generated using LLM (530).

[0143] Accordingly, when new code information (560) is created, the code information can be added to the plan pool (520).

[0144] In this way, the present disclosure can respond to user requests more quickly by creating a plan pool by creating several possible candidate commands in advance and creating codes corresponding to the corresponding results in advance.

[0145] In other words, while previously users had to wait for LLM results to be received based on their requests, this disclosure allows users to immediately utilize pre-generated results. This reduces the delay caused by the time required for traditional generation. Furthermore, since responses are performed using pre-generated results, consistent responses are possible, rather than providing different responses to similar user requests, enabling deterministic responses.

[0146] Figure 6 is a diagram illustrating an example of information of a primitive task used in the present disclosure.

[0147] Referring to FIG. 6, this is a diagram illustrating an example of information for a call reception task among examples of primitive tasks according to the present disclosure.

[0148] The information may include a task name (610) corresponding to the information, a description thereof, an example, a response method for each mode, etc.

[0149] For example, as illustrated, receiving a phone call may proceed differently depending on whether the state is normal or driving, and so on, so a primitive task can describe various change operations in that task.

[0150] Meanwhile, such primitive tasks are related to the functions of electronic devices, and such primitive tasks are created by the electronic device (100) or the manufacturer, and the server (or LLM) can collect and use such primitive tasks externally.

[0151] Meanwhile, although the illustrated example describes how to answer a phone call in various modes, the implementation may include only one action.

[0152] FIG. 7 is a diagram illustrating an example of a plan code generated according to one embodiment of the present disclosure.

[0153] Referring to Figure 7, an example of code information for the task “Call Mom” is shown.

[0154] The code contains two primitive codes. For example, the first primitive code is for finding a specific person in a phone book, and the second primitive code is for making a phone call.

[0155] Accordingly, the plan code is composed of a phone number search code (710) using the first primitive code and a code (750) for making a call to the phone number searched using the second primitive code, as shown.

[0156] Meanwhile, the illustrated example not only includes the two codes but also includes information about exception conditions. For example, it includes cases where the mother's number is not stored in the phonebook (720) and cases where the mother's contact information exists but the contact information does not have a phone number registered (730, 740).

[0157] Such exception handling code is optional and can be omitted during implementation. For example, if the LLM code's operating environment has relatively low resources, the plan code can be generated with only the first and second primitive codes described above, without exception handling. If resources are high, the exception handling resources described above can also be generated.

[0158] In this way, the generated code information includes matters related to exception handling, which means that operations can be performed using code information that takes various situations into account, which has the effect of allowing exception handling, etc. during the operation process to be performed more quickly.

[0159] FIG. 8 is a flowchart for explaining the control operation of an electronic device according to an embodiment of the present disclosure.

[0160] Referring to Fig. 8, a plan pool including information on a plurality of primitive tasks and code information generated by combining the primitive tasks is stored (S810). For example, as described in Fig. 5, candidate tasks can be generated by combining primitive tasks in various ways using a large-scale language model, and it can be checked whether the generated candidate task is included in the plan pool. If the generated candidate task is not included in the plan pool, the generated candidate task is included in the plan pool. This process of generating candidate tasks (or updating the plan pool) can be automatically and repeatedly performed when a new primitive task is added or the number of stored code information in the plan pool is less than or equal to a preset number.

[0161] Information corresponding to the user's language command is then acquired (S820). For example, such information corresponding to the user's language command can be acquired as a result of voice recognition of the user's spoken voice input through the electronic device's microphone. Alternatively, text information directly input by the user or information transmitted from an external device can be utilized.

[0162] And, using a large-scale language model, it is checked whether a task corresponding to the user language command is included in the plan pool (S830).

[0163] If a task corresponding to the user language command is included in the plan pool, response information is generated using code information within the plan pool (S840). Here, the response information may be output as a voice prompt or displayed on the screen if the electronic device provides the response information to the user. Alternatively, if the response information described above is generated at the request of an external device, the response information described above may be transmitted to the external device.

[0164] At this time, the information transmitted to the external device may include content corresponding to the response information or may include program code that can be executed on the external device.

[0165] Here, content may consist of images, audio, or a combination thereof, and includes information that can be presented visually, audibly, or audiovisually to a user via an electronic device. For example, content may include audio content, music content, image content, video content, widget content, web pages, etc.

[0166] Meanwhile, if a task corresponding to a user language command is not included in the plan pool, code information corresponding to the user language command can be generated using a large-scale language model and information from multiple primitive tasks. This operation is described in detail in Figure 9.

[0167] As described above, the control method of an electronic device according to the present disclosure utilizes a plan pool created during the process of utilizing a large-scale language model, thereby enabling deterministic responses (i.e., outputting the same result each time a call is made) to user requests (or commands). Furthermore, by pre-generating various scenarios, the execution time during the process of utilizing a large-scale language model can be shortened.

[0168] FIG. 9 is a flowchart illustrating a method for generating response information according to one embodiment of the present disclosure.

[0169] Referring to Figure 9, when information corresponding to a user language command is acquired, a prompt corresponding to the information is generated (S905). For example, when a user's spoken voice is input through a microphone, a prompt can be generated as a result of voice recognition for the input voice.

[0170] Here, a 'prompt' is a dialogue that explains in natural language what action the generative model should take and allows it to output the desired result.

[0171] And, using LLM, it is checked whether a task corresponding to the user language command is included in the plan pool (S910).

[0172] If a task corresponding to a user language command is included in the plan pool, it can be confirmed that it can be processed immediately (S915). For example, if a task corresponding to a user language command is included in the plan pool, code information within the plan pool can be retrieved (S920).

[0173] And the brought code information can be executed (S950, S960, S970), and the final response can be generated using the executed result (S980) (S990).

[0174] If a task corresponding to the user language command is not included in the plan pool (i.e., not immediately processable) (S915-no), code information corresponding to the user language command can be generated using a large-scale language model and information on multiple primitive tasks (S930). This operation is similar to the generation of candidate tasks in the preceding Figure 5. However, the preceding candidate tasks are generated by LLM arbitrarily combining primitive tasks, whereas the operation at the current stage creates candidate tasks based on the user language command.

[0175] Once new code information is generated in this manner, execution operations can be performed using the code (S950). The results of this execution can be verified, and whether the verified results correspond to the user language commands can be verified (S960).

[0176] If the acquired execution result does not correspond to the user language command, the process of generating code information can be repeated (S970-No).

[0177] Conversely, if the acquired execution result corresponds to a user language command (S970-Yes), the generated code information can be included in the plan pool and response information based on the code information can be generated.

[0178] As described above, the control method of an electronic device according to the present disclosure utilizes a large-scale language model to create and utilize a plan pool, thereby enabling a deterministic response (i.e., outputting the same result each time it is called) to a user request (or command). Furthermore, for new requests not included in the plan pool, new code information is generated and utilized, enabling deterministic operation by utilizing the corresponding code in subsequent processes.

[0179] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0180] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device (e.g., an electronic device (300)) according to the disclosed embodiments, which is a device that can call instructions stored from the storage medium and operate according to the called instructions. When the above-described instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the above-described instructions. The instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0181] According to a temporary example, the methods according to the various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0182] Each component (e.g., a module or a program) according to various embodiments may be composed of one or more entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., a module or a program) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Claims

1. In electronic devices, A memory that stores a plan pool containing information about multiple primitive tasks, code information generated by combining primitive tasks, and a large language model (LLM); and When information corresponding to a user language command is obtained, a processor is included that generates response information corresponding to the user language command; The above processor, An electronic device that uses the large-scale language model to check whether a task corresponding to the user language command is included in the plan pool, and if the task corresponding to the user language command is included in the plan pool, generates response information using code information within the plan pool.

2. In paragraph 1, The above processor, An electronic device that generates code information corresponding to the user language command using the large-scale language model and the plurality of primitive task information, if a task corresponding to the user language command is not included in the plan pool.

3. In paragraph 2, The above processor, An electronic device that obtains an execution result of the generated code information using the generated code information and confirms whether the obtained execution result corresponds to the user language command.

4. In paragraph 3, The above processor, If the above acquired execution result does not correspond to the user language command, the process of generating code information is repeated, An electronic device that includes the generated code information in the plan pool when the acquired execution result corresponds to the user language command.

5. In paragraph 1, The above processor, Using the above large-scale language model, candidate tasks are generated by combining primitive tasks in various ways, Check whether the above generated candidate task is included in the above plan pool, An electronic device that includes the generated candidate task in the plan pool if the generated candidate task is not included in the plan pool.

6. In paragraph 5, The above processor, An electronic device that repeatedly generates the candidate tasks when a new primitive task is added or the number of stored code information in the plan pool is less than or equal to a preset number.

7. In paragraph 1, The above code information is, An electronic device comprising a first program code for performing a primitive task and a second program code for handling exceptions of the first program code.

8. In paragraph 1, Mike; and including speakers; The above processor, Recognize the user's spoken voice acquired through the above microphone, An electronic device that controls the speaker so that when response information corresponding to the above voice recognition result is generated, guidance information corresponding to the response information is output.

9. In paragraph 1, operating input device; and including display; The above processor, When text information is entered through the above-mentioned operating input device, An electronic device that controls the display so that a screen corresponding to the response information is displayed when response information corresponding to the input text information is generated.

10. In paragraph 1, Further comprising a communication device for performing communication with an external device; The above processor, An electronic device that controls the communication device to transmit a program code executable on the external device when information corresponding to the user language command is received.

11. In a control method in an electronic device, A step of storing a plan pool including information on multiple primitive tasks and code information generated by combining the primitive tasks; A step of obtaining information corresponding to a user language command; A step of using a large language model (LLM) to check whether a task corresponding to the user language command is included in the plan pool; and A control method comprising: a step of generating response information using code information within the plan pool if a task corresponding to the user language command is included in the plan pool; 12. In paragraph 11, A control method further comprising: a step of generating code information corresponding to the user language command using the large-scale language model and the plurality of primitive task information, if a task corresponding to the user language command is not included in the plan pool; 13. In paragraph 12, A step of obtaining an execution result of the generated code information using the generated code information; and A control method further comprising a step of checking whether the obtained execution result corresponds to the user language command.

14. In paragraph 13, If the acquired execution result does not correspond to the user language command, a step of repeating the process of generating new code information; and A control method further comprising a step of including the generated code information in the plan pool if the acquired execution result corresponds to the user language command.

15. In a non-transitory computer-readable recording medium including a program for executing a method of controlling an electronic device, The method of controlling the above electronic device is as follows: A step of storing a plan pool including information on multiple primitive tasks and code information generated by combining the primitive tasks; A step of obtaining information corresponding to a user language command; A step of using a large language model (LLM) to check whether a task corresponding to the user language command is included in the plan pool; and A computer-readable recording medium comprising: a step of generating response information using code information within the plan pool if a task corresponding to the user language command is included in the plan pool;

Images