Electronic device and control method thereof

Abstract

The electronic device includes: a memory storing a plurality of sample cues; a communication interface in communication with a server including a large language model; and at least one processor which acquires a user input from a user, selects a sample prompt corresponding to the user input from the plurality of stored sample prompts, acquires additional information corresponding to the user input, and transmits the additional information to the processor. And a processor configured to receive the user input, the sample cue, and the additional information, acquire an input cue based on the user input, the sample cue, and the additional information, send the input cue to the server using the communication interface, and provide response information corresponding to the input cue when the response information is received from the server via the communication interface.

Description

Technical Field

[0001] Devices and methods consistent with the present disclosure relate to an electronic device and a control method thereof, and more particularly, to an electronic device that provides response information corresponding to a user input based on a language model and a control method thereof. Background Art

[0002] An electronic device may use a language model to perform a voice assistant function. The voice assistant function may be a function of providing a response to a user input. Based on receiving the user input via text or voice, the voice assistant device may provide a response to the user input.

[0003] The language model may be trained by learning data. However, the language model may not be able to perform learning operations in real time, and therefore the language model may have difficulty reflecting information that changes in real time. For example, the time point of receiving user input may be in the present, while the learning time point may be in the past. Therefore, the language model that provides a response to the user input may generate output data based on past learning data.

[0004] Furthermore, even if learning data can be used to train a language model, the language model may not be able to provide response information based on the user input or the degree of learning. The information included in the user input may not be sufficient to generate a complete response. Furthermore, the scope of the learning data may be limited to a specific domain, and the language model may process user input in a domain where the model cannot provide an accurate response. In this case, the language model may not be able to provide an accurate response to the user.

[0005] Based on insufficient information included in the user input, the language model may request the user to repeat his or her utterance. However, the user may feel uncomfortable based on receiving a request to re-speak. In addition, the response information may have a lower amount of information or lower user satisfaction based on the user input alone. Summary of the invention

[0006] An electronic device and a control method thereof are provided that can provide response information based on a user input and a prompt including additional information.

[0007] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0008] According to one aspect of the present disclosure, an electronic device includes: a memory configured to store multiple sample prompts; a communication interface configured to communicate with a server including a large language model; and at least one processor configured to: obtain user input from a user, select a sample prompt corresponding to the user input from the multiple stored sample prompts, obtain additional information corresponding to the user input, obtain an input prompt based on the user input, the sample prompt and the additional information, and send the input prompt to the server using the communication interface; and provide the response information to the user based on response information corresponding to the input prompt received from the server through the communication interface.

[0009] The sample prompt may include at least one of description information, example information, and response structure information representing the sample prompt.

[0010] The at least one processor may be further configured to identify a task corresponding to the user input and select the sample prompt based on the identified task.

[0011] The server may be a first server, and the at least one processor may also be configured to: determine whether to use additional information corresponding to the user input, based on determining that the additional information is to be used, send a signal requesting the additional information to a second server using the communication interface, and receive the additional information from the second server using the communication interface.

[0012] The at least one processor may be further configured to obtain a first probability of generating a complete response based on the user input, and obtain the additional information based on the first probability being less than a threshold probability.

[0013] The at least one processor may be further configured to obtain a second probability of generating the complete response based on the user input and at least one piece of sample additional information among the multiple pieces of sample additional information, and obtain the additional information based on at least one piece of sample additional information based on the second probability being greater than or equal to the threshold probability.

[0014] The at least one processor may be further configured to obtain the input prompt by combining the sample prompt with the user input and the additional information.

[0015] The at least one processor may be further configured to obtain a user log corresponding to user activities within a threshold time, and select the sample prompt based on the user log and the user input.

[0016] The at least one processor may be further configured to obtain a user log corresponding to a user activity within a threshold time, and obtain the additional information based on the user log and the user input.

[0017] The user input may include at least one of audio information, text information, and image information.

[0018] According to one aspect of the present disclosure, a method for controlling an electronic device, the electronic device being configured to store a plurality of sample prompts and communicating with a server including a large language model, the method comprising: obtaining a user input of a user; selecting a sample prompt corresponding to the user input from the plurality of stored sample prompts; obtaining additional information corresponding to the user input; obtaining an input prompt based on the user input, the sample prompt and the additional information; sending the input prompt to the server; and providing the response information to the user based on response information corresponding to the input prompt received from the server.

[0019] The sample prompt may include at least one of description information representing the sample prompt, at least one example information, and response structure information.

[0020] The selecting of the sample prompt may include identifying a task corresponding to the user input, and selecting the sample prompt based on the identified task.

[0021] The server may be a first server, and the acquisition of the additional information may include: determining whether the additional information corresponding to the user input needs to be used, sending a signal requesting the additional information to a second server based on determining the additional information to be used, and receiving the additional information from the second server.

[0022] The acquiring of the additional information may include: acquiring a first probability of generating a complete response based on the user input, and acquiring the additional information based on the first probability being less than a threshold probability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which: Figure 1 is a view for explaining a system including an electronic device and at least one external server according to an embodiment; Figure 2 is a block diagram showing a configuration of an electronic device according to an embodiment; Figure 3 is a diagram showing a method according to an embodiment of the present invention. Figure 2 A block diagram of a specific configuration of an electronic device; Figure 4 is a view for explaining a plurality of modules for providing response information corresponding to a user input according to an embodiment; Figure 5 is a view for explaining a process of providing response information according to an embodiment; Figure 6 is a view for explaining a process of providing response information according to an embodiment; Figure 7 is a view for explaining a process of providing response information according to an embodiment; Figure 8 is a view for explaining a process of providing response information according to an embodiment; Fig. 9 is a view for explaining a plurality of sample prompts according to an embodiment; Fig.10 is a view for explaining categories of sample prompts according to an embodiment; Fig.11 is a view for explaining the operation of a task determination module according to an embodiment; Fig.12 is a view for explaining the operation of an additional information determination module according to an embodiment; Fig.13 is a view for explaining the operation of an additional information receiver module according to an embodiment; Fig.14 is a view for explaining the operation of an additional information determination module according to an embodiment; Fig.15 A view that explains the operation of obtaining additional information; Fig.16 is a view for explaining the operation of an additional information determination module according to an embodiment; Fig.17 is a flowchart for explaining an operation of acquiring response information according to an embodiment; Fig.18 is a flowchart for explaining an operation of acquiring response information using a server according to an embodiment; Fig.19 is a flow chart for explaining an embodiment in which sample prompts are divided into separate groups according to an embodiment; Fig. 20 is a flowchart for explaining an operation of acquiring additional information using a second server according to an embodiment; Fig.21 is a flowchart for explaining the operation of obtaining input prompts based on the probability of generating a complete response according to an embodiment; Fig. 22 is a flowchart for explaining an operation of determining additional information based on a probability of generating a complete response according to an embodiment; Fig.23 is a view for explaining the operation of a complete response determination module according to an embodiment; Fig.24 is a flowchart for explaining an operation of obtaining an input prompt using a user log according to an embodiment; Fig.25 is a flowchart for explaining an operation of acquiring additional information according to an embodiment; Fig.26 is a view for explaining a user interface (UI) for displaying information for acquiring an input hint according to an embodiment; Fig. 27 is a view for explaining a user interface (UI) for displaying an input prompt and response information according to an embodiment; Fig.28 is a flowchart for explaining a control method of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments are described in detail with reference to the accompanying drawings.

[0025] General terms can be used to describe the embodiments of the present disclosure in consideration of their functions in the present disclosure. However, these terms may change according to the intention of those skilled in the art or the emergence of judicial precedents, new technologies, etc. In addition, in specific cases, there may be terms arbitrarily selected by the applicant. In this case, the meaning of such terms is mentioned in detail in the corresponding description of the present disclosure. Therefore, the terms used in the present disclosure can be defined based on the meaning of the terms and the content throughout the present disclosure rather than the simple name of the terms.

[0026] As used herein, expressions such as “having”, “may have”, “including”, “may include”, etc. may indicate the existence of corresponding features (e.g., values, functions, operations, or components such as parts), and do not exclude the existence of additional features.

[0027] The expression “at least one of A or / and B” may mean “A or B” or “both A and B”.

[0028] The expressions "first", "second", etc. used herein may define various components regardless of the order or importance of the components. These expressions are only used to distinguish one component from another component, and do not limit the corresponding components.

[0029] Based on any component (e.g., the first component) being described as "coupled (operatively or communicatively) to / with another component (e.g., the second component)" or "connected to another component (e.g., the second component)", it should be understood that any component can be directly coupled to another component or can be coupled to another component through yet another component (e.g., the third component).

[0030] Singular terms may include their plurals, unless explicitly stated otherwise in the context. It should be understood that the terms "comprising", "consisting of", etc. used in this application specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof mentioned in the specification, and do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] As used herein, a "module" or component named with "~er" at the end can perform at least one function or operation, can be implemented by hardware or software, or by a combination of hardware and software. In addition, except for the "module" or component named with "~er" at the end that must be implemented by specific hardware, multiple "modules" or multiple components named with "~er" at the end can be integrated into at least one module implemented by at least one processor.

[0032] As used herein, a term such as "user" may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0033] Hereinafter, one or more embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0034] Figure 1 is a view for explaining a system including an electronic device 100 and at least one external server (e.g., the first server 200 and the second server 300).

[0035] Refer to Figure 1 , the system 1000 may include: the electronic device 100 and at least one of the first server 200 and the second server 300.

[0036] The electronic device 100 can be a device configured to obtain user input. The electronic device 100 can be a device configured to obtain user speech through a microphone. The electronic device 100 can be a device configured to obtain text information input by the user through an operation interface. The electronic device 100 can be a device configured to obtain an image through a camera. The electronic device 100 can provide response information corresponding to the user input. For example, the electronic device 100 can be a smart phone, a television (TV), a tablet personal computer (PC), a laptop computer, etc.

[0037] In addition, the electronic device 100 may transmit the user input to the first server 200. In addition, the electronic device 100 may receive response information corresponding to the user input from the first server 200. The electronic device 100 may provide the received response information to the user.

[0038] The first server 200 may be a server configured to generate response information. The first server 200 may include a large language understanding model. The first server 200 may use the large language understanding model to generate response information corresponding to the user input sent from the electronic device 100. The first server 200 may send the generated response information to the electronic device 100.

[0039] The second server 300 may be a server configured to provide additional information required to generate response information corresponding to the user input. In an embodiment, the additional information required to generate the response information may refer to, for example, additional information used to generate the response information or useful for generating the response information, or additional information determined to be used to generate the response information or determined to be useful for generating the response information. The electronic device 100 may request the additional information from the second server 300 and receive the additional information through the second server 300. The second server 300 may be a database server, a search engine, a calculator application, an image processing application, etc.

[0040] Figure 2 is a block diagram showing a configuration of an electronic device 100 according to various embodiments.

[0041] refer to Figure 2 , the electronic device 100 may include: at least one of a memory 110 , a communication interface 120 , and at least one processor 130 .

[0042] The memory 110 may store a plurality of sample prompts.

[0043] The communication interface 120 may communicate with the first server 200 including a large language model.

[0044] At least one processor 130 may be configured to obtain user input, identify, determine, or select a sample prompt corresponding to the user input from a plurality of stored sample prompts, obtain additional information corresponding to the user input, obtain an input prompt based on the user input, the sample prompt, and the additional information, send the input prompt to the first server 200 via the communication interface 120, and provide response information based on response information corresponding to the input prompt received from the first server 200 via the communication interface 120.

[0045] In an embodiment, the user input may include at least one of audio information, text information, and image information.

[0046] The user input may include audio information. The audio information may include audio data, or, for example, an audio signal. The at least one processor 130 may obtain the user input including the audio information. The at least one processor 130 may convert the audio data into text information using a speech recognition function. Fig.14 Example embodiments are described in which audio information is included in user input.

[0047] According to various embodiments, the voice recognition function may be performed by an external server. The at least one processor 130 may transmit audio data to the external server and receive text information corresponding to the audio data from the external server.

[0048] The user input may include text information. The text information may include character information.

[0049] The user input may include image information. The image information may include image data, or, for example, an image signal. The at least one processor 130 may identify an object included in the image information. In addition, response information corresponding to the user input may be obtained based on the object included in the image information. Figure 8 , Fig.14 and Fig.15 Example embodiments are described in which image information is included in user input.

[0050] At least one processor 130 may select one sample prompt corresponding to the user input from among a plurality of sample prompts stored in the memory 110 .

[0051] In an embodiment, the sample prompt may include at least one of the following: description information representing the sample prompt, at least one example information, and response structure information.

[0052] The description information may include text indicating the task or field for which the sample prompt will be used to process user input.

[0053] The sample information may include any user input and any response information corresponding to the user input. The sample information may include sample input and sample response.

[0054] The response structure information may include information about the format of displaying the response information. For example, the weather sample prompt may include structure information for generating a response of "1. Today's weather: and 2. Today's temperature: ", or a format structure, for example. The at least one processor 130 may obtain a response to the weather based on the response structure information, such as "1. Today's weather: sunny and 2. Today's temperature: 20 degrees".

[0055] The sample hint may be a sample of input data input to a large language model, or, for example, an example. In addition, the sample hint may indicate form data, style data, type data, or structure data underlying the input data.

[0056] A large language model may be a language model that can be used in a variety of tasks. In addition to user input, a large language model may use various types of information to provide appropriate response information to the user. A large language model may generate output data from a prompt that is a unit of input data.

[0057] The sample prompt may be input data for providing response information. The sample prompt may include various information for generating response information.

[0058] The sample prompt may be information used as input data for acquiring response information. The sample prompt may include at least one of description information representing the sample prompt and sample data related to the response information. The sample data may be data representing a structure of a plurality of detailed information included in the response information.

[0059] The sample prompt may indicate a data group including a plurality of information. Therefore, the sample prompt may be described as a sample group or an input group. In addition, the sample prompt may provide a form of response information. Therefore, the sample prompt may be described as response format information or response structure information. Fig. 9 and Fig.10 Example embodiments related to sample prompts are described.

[0060] The at least one processor 130 may acquire additional information corresponding to the user input. In addition to the user input, the at least one processor 130 may acquire additional information to provide the user with response information with high utilization.

[0061] According to various embodiments, additional information may be acquired by the electronic device 100. The at least one processor 130 may extract text information included in the user input, and acquire additional information related to the text information based on the extracted text information.

[0062] According to various embodiments, the additional information may be acquired by the first server 200. The at least one processor 130 may send a signal requesting the additional information to the first server 200. The signal requesting the additional information may include a user input. Therefore, the first server 200 may acquire the additional information based on the user input received from the electronic device 100. In addition, the first server 200 may send the additional information to the electronic device 100.

[0063] According to various embodiments, at least one processor 130 may generate an input prompt based on a sample prompt and a user input without acquiring additional information.

[0064] At least one processor 130 may obtain or generate an input prompt based on at least one of the user input, the sample prompt, and the additional information. The input prompt may be input data input into the large language model. Figures 5 to 8 Example embodiments for generating input prompts are described.

[0065] The large language model may be included in the first server 200. Therefore, the at least one processor 130 may transmit the generated input prompt to the first server 200. In addition, the first server 200 may generate response information based on the input prompt received from the at least one processor 130. The first server 200 may transmit the generated response information to the electronic device 100.

[0066] The at least one processor 130 may receive the response information from the first server 200. In addition, the at least one processor 130 may provide the response information to the user. According to various embodiments, the electronic device 100 may display the response information on the display 140. According to various embodiments, the electronic device 100 may output the response information through the speaker 170.

[0067] In an embodiment, at least one processor 130 may identify a task corresponding to the user input, and identify a sample prompt corresponding to the user input from a plurality of sample prompts based on the identified task.

[0068] At least one processor 130 may determine which task field the user input corresponds to for selecting a sample prompt. At least one processor 130 may select a task corresponding to the user input from a plurality of tasks stored in the memory 110. Fig.11 Provide details of its examples.

[0069] In an embodiment, at least one processor 130 may determine whether additional information corresponding to the user input is required, and based on determining that the additional information is required, send a signal requesting the additional information to the second server 300 through the communication interface 120, and receive the additional information from the second server 200 through the communication interface 120.

[0070] At least one processor 130 may determine whether response information can be generated based only on the user input. At least one processor 130 may determine whether a complete response corresponding to the user input can be generated through a complete response determination module. Fig.23 An example of a complete response determination module 2300 is described.

[0071] The at least one processor 130 may determine that additional information is necessary based on the inability to generate a complete response based solely on the user input. In addition, the at least one processor 130 may request additional information from the second server 300. The at least one processor 130 may send a control signal requesting additional information to the second server 300. The user input may be included in the sent control signal.

[0072] The second server 300 may acquire the additional information based on the control signal received from the electronic device 100. In addition, the second server 300 may transmit the acquired additional information to the electronic device 100 again.

[0073] The at least one processor 130 may receive the additional information from the second server 300. Furthermore, the at least one processor 130 may generate an input prompt based on the received additional information, the sample prompt, and the user input.

[0074] In an embodiment, at least one processor 130 may obtain a first probability of generating a complete response based on the user input, and obtain additional information based on the first probability being less than a threshold probability.

[0075] At least one processor 130 may obtain a probability or likelihood of generating a complete response based on the user input. A complete response may indicate a response having a complete form or a complete amount of information based on information included in the input data or the user input.

[0076] At least one processor 130 may obtain the probability of generating a complete response through the complete response generation module. At least one processor 130 may input input data to the complete response generation module. In addition, the complete response generation module may output the probability of generating a complete response based on the input data as output data.

[0077] For example, based on receiving a user input of "What time is it now?", at least one processor 130 may determine whether response information corresponding to the user input ("What time is it now?") can be generated. At least one processor 130 may determine that a complete response can be generated based on the electronic device 100 storing information about the current time. At least one processor 130 may determine that the probability value for generating a complete response is 80% or greater. However, at least one processor 130 may determine that a complete response cannot be generated based on the electronic device 100 not storing information about the current time. At least one processor 130 may determine that the probability value for generating a complete response is less than 10%.

[0078] The first probability value may refer to a probability value (as output data) based on that the input data input to the complete response generation module is a user input.

[0079] The second probability value may refer to a probability value (as output data) based on that the input data input to the complete response generation module is the user input and the sample additional information.

[0080] The at least one processor 130 may first obtain the first probability value using only the user input as input data. In addition, the at least one processor 130 may determine whether the first probability value is less than a threshold probability value.

[0081] At least one processor 130 may determine that a complete response can be generated based only on the user input based on the first probability value being greater than or equal to the threshold probability value. At least one processor 130 may generate response information based on the user input. At least one processor 130 may provide the generated response information to the user.

[0082] At least one processor 130 may determine that a complete response cannot be generated based solely on the user input based on the first probability value being less than a threshold probability value. At least one processor 130 may determine that response information cannot be provided to the user based solely on the user input based on the first probability value being less than a threshold probability value. At least one processor 130 may determine that additional information is required in addition to the user input. For example, at least one processor 130 may determine that additional information will be used in addition to the user input.

[0083] In addition to user input, at least one processor 130 may identify, determine, or select which additional information is necessary or to be used.

[0084] In an embodiment, at least one processor 130 may obtain a second probability of generating a complete response based on user input and at least one piece of sample additional information among multiple pieces of sample additional information, and identify additional information based on the at least one piece of sample additional information based on the second probability being greater than or equal to a threshold probability.

[0085] The electronic device 100 may store a plurality of sample additional information in the memory 110. The sample additional information may be input data for obtaining a probability of generating a complete response together with the user input. The sample additional information may be described or referred to as test information, test additional information, virtual information, or virtual additional information. The sample additional information may not be information for generating response information, and the sample additional information may be only information for obtaining a probability value for generating a complete response. The at least one processor 130 may obtain a probability value by inputting each of the pieces of sample additional information together with the user input into the complete response generation module.

[0086] There may be a problem of determining which of the pieces of sample additional information is first input into the complete response generation module together with the user input. The at least one processor 130 may calculate the correlation between each of the multiple pieces of sample additional information and the user input. In addition, the at least one processor 130 may select the sample additional information in the order of high correlation. In addition, the at least one processor 130 may input the user input and the selected sample additional information into the complete response generation module.

[0087] For example, the electronic device 100 may receive a user input of "What time is it now?" and may not store information about the current time. The complete response generation module may determine that the first probability value is 5%. In addition, at least one processor 130 may determine that additional information is necessary.

[0088] For example, at least one processor 130 may obtain input data including time information (e.g., random time information or past time information) and user input, and the time information may be sample additional information. In addition, at least one processor 130 may input sample additional information (e.g., random time information) and user input (e.g., "What time is it now?") as input data to the complete response generation module. The complete response generation module may determine that the second probability value is 90%. In addition to the user input, at least one processor 130 may determine that additional time information is required.

[0089] For example, at least one processor 130 may acquire time information as additional information. At least one processor 130 may acquire time information of the current time point as additional information instead of sample additional information. In addition, at least one processor 130 may generate response information based on the additional information (e.g., time information of the current time point) and user input.

[0090] According to various embodiments, the threshold probability to which the first probability value is compared may be the same as the threshold probability to which the second probability is compared.

[0091] According to various embodiments, the threshold probability compared with the first probability value may be different from the threshold probability compared with the second probability value. In this case, the different threshold probabilities may be referred to as a first threshold probability (or a first threshold probability value) and a second threshold probability (or a second threshold probability value). In addition to the user input, the second probability value may be calculated by further using sample additional information, thereby requesting a higher level of complete response. Therefore, the second threshold probability may have a greater value than the first threshold probability.

[0092] refer to Figure 21 to Figure 23 Details of examples of the first probability value and the second probability value are described.

[0093] In an embodiment, the at least one processor 130 may obtain an input prompt by combining a sample prompt with a user input and additional information.

[0094] The sample prompt may be form data input to the language model. The sample prompt may include information about the storage location of the user input and additional information. The at least one processor 130 may store the user input and the additional information in the sample prompt based on the information about the storage location included in the sample prompt. After performing the storage operation, the at least one processor 130 may obtain the sample prompt including the user input and the additional information. Figures 5 to 8 Describe the details of its example.

[0095] In an embodiment, at least one processor 130 may obtain a user log within a threshold time, and identify a sample prompt corresponding to the user input based on the user log and the user input.

[0096] The user log may include at least one of user input input from the electronic device 100, content information (e.g., audio, image, or text) provided based on the user input, instruction information executed based on the user input, and application information executed based on the user input. The user log may be described as log information.

[0097] The at least one processor 130 may select a sample prompt from the plurality of sample prompts based on the user log and the user input. The at least one processor 130 may select, identify or determine the most relevant sample prompt based on the user log and the user input.

[0098] In an embodiment, the at least one processor 130 may obtain a user log within a threshold time, and obtain additional information based on the user log and the user input.

[0099] The at least one processor 130 may use the user log and the user input to acquire additional information for generating the response information. The at least one processor 130 may transmit the user log and the user input to the second server 300 based on the additional information generated by the second server 300.

[0100] The second server 300 may generate additional information based on the received user log and user input, and may transmit the generated additional information to the electronic device 100.

[0101] At least one processor 130 may generate an input prompt based on the additional information received from the second server 300, the sample prompt, and the user input.

[0102] refer to Fig.11 and Fig.24 Provides a description of an example of a user log.

[0103] The electronic device 100 according to various embodiments can obtain input data input into a large language understanding model. In addition to user input, the input data can also include various information. The input data can be an input prompt including sample prompt information (description information, example information or answer format information), user input and additional information. In addition to user input, the input data can also include various information, so that the response information can have higher integrity. Therefore, the electronic device 100 can provide a high-level response service to the user.

[0104] According to various embodiments, in addition to the user input, the electronic device 100 may obtain the response information by further considering various information. Therefore, even if the user inputs insufficient data to generate a complete response, the electronic device 100 may generate the response information by further considering the sample prompt or additional information. Therefore, the response information may have a higher amount of information or a higher user satisfaction. In addition, even if the language model used to generate the response information is specialized for a specific category, the electronic device 100 may provide the user with advanced response information by further considering the sample prompt or additional information.

[0105] In the embodiment, the above description only shows and describes a simple configuration of the electronic device 100. However, various configurations may also be provided in its implementation. Figure 3 Provide a description of its examples.

[0106] Figure 3 It is shown Figure 2 A block diagram of a specific configuration of an electronic device 100.

[0107] refer to Figure 3 , the electronic device 100 may include at least one of a memory 110, a communication interface 120, at least one processor 130, a display 140, an operation interface 150, an input / output interface 160, a speaker 170, and a microphone 180. This description may omit redundant descriptions of the same operations as above.

[0108] The electronic device 100 according to various embodiments may include, for example, at least one of a smart phone, a tablet computer, a mobile phone, a desktop computer, a laptop computer, a personal digital assistant (PDA), and a portable multimedia player (PMP). In some embodiments, the electronic device 100 may include, for example, at least one of a television, a digital video disk (DVD) player, and a media box (e.g., Samsung HomeSync™, Apple TV™, or Google TV™).

[0109] The memory 110 may be implemented as an internal memory included in the processor 130, such as a read-only memory (ROM, for example, an electrically erasable programmable read-only memory (EEPROM)) or a random access memory (RAM), or implemented as a memory separate from the processor 130. In this case, based on the data storage purpose, the memory 110 may be implemented in the form of a memory embedded in the electronic device 100 or in the form of a memory detachable from the electronic device 100. For example, data for driving the electronic device 100 may be stored in a memory embedded in the electronic device 100, and data for an extended function of the electronic device 100 may be stored in a detachable memory in the electronic device 100.

[0110] In an embodiment, the memory embedded in the electronic device 100 may be implemented as at least one of a volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), a non-volatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM)), a mask ROM, a flash ROM, a flash memory (e.g., NAND flash memory or NOR flash memory), a hard disk drive, or a solid-state drive (SSD); and the memory removable from the electronic device 100 may be implemented as a memory card (e.g., compact flash (CF), secure digital (SD)), micro secure digital (Micro-SD), mini secure digital (mini-SD), extreme digital (xD), or multimedia card (MMC)) or an external memory (e.g., USB memory) that can be connected to a universal serial bus (USB) port, etc.

[0111] The communication interface 120 may be a component configured to communicate with various types of external devices using various types of communication methods. The communication interface 120 may include a wireless communication module and a wired communication module. Here, each communication module may be implemented in the form of at least one hardware chip.

[0112] The wireless communication module may be a module configured to communicate with an external device in a wireless manner. For example, the wireless communication module may include at least one of a Wireless Fidelity (Wi-Fi) module, a Bluetooth module, an infrared communication module, and other communication modules.

[0113] The Wi-Fi module and the Bluetooth module can perform communication in a Wi-Fi manner and a Bluetooth manner, respectively. Based on the use of the Wi-Fi module or the Bluetooth module, various connection information such as a service set identifier (SSID) or a session key can be first sent and received, the communication is connected using this connection information, and then various information is sent and received.

[0114] The infrared communication module may perform communication based on an Infrared Data Association (IrDA) technology that transmits data over a short distance in a wireless manner using infrared rays between visible light and millimeter waves.

[0115] In addition to the above-mentioned communication methods, other communication modules may include at least one communication chip, which is configured to perform communication based on various wireless communication standards such as zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G) and fifth generation (5G).

[0116] The wired communication module may be a module configured to communicate with an external device in a wired manner. For example, the wired communication module may include at least one of a local area network (LAN) module, an Ethernet module, a twisted pair, a coaxial cable, an optical fiber cable, and an ultra-wideband (UWB) module.

[0117] The processor 130 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) configured to process digital signals. However, the processor 130 is not limited thereto, and may include one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machine (arm) processor, or may be defined by these terms. In addition, the processor 130 may be implemented as a system on chip (SoC) or a large-scale integration (LSI) in which a processing algorithm is embedded, or may be implemented in the form of a field programmable gate array (FPGA). In addition, the processor 130 may perform various functions by executing computer executable instructions stored in a memory.

[0118] The display 140 may be implemented as any of various types of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display 140 may include a driving circuit, a backlight unit, etc., which may be implemented in the form of an amorphous silicon thin film transistor (a-si TFT), a low temperature polycrystalline silicon (LTPS) TFT, or an organic TFT (OTFT). In an embodiment, the display 140 may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional (3D) display, etc. In addition, the display 140 according to one or more embodiments of the present disclosure may include not only a display panel that outputs an image, but also a frame that accommodates the display panel. In particular, the frame may include a touch sensor that detects user interaction according to one or more embodiments of the present disclosure.

[0119] According to various embodiments, the electronic device 100 may include a display 140. For example, the electronic device 100 may directly display the acquired image or content on the display 140.

[0120] According to various embodiments, the electronic device 100 may not include the display 140. The electronic device 100 may be connected to an external display device, and may send an image or content stored in the electronic device 100 to the external display device. For example, the electronic device 100 may send an image or content to the external display device together with a control signal for controlling the image or content to be displayed on the external display device. Here, the external display device may be connected to the electronic device 100 through the communication interface 120 or the input / output interface 160. For example, the electronic device 100 may not include a display like a set-top box (STB). In addition, the electronic device 100 may include only a small display that may display only simple information such as text information. Here, the electronic device 100 may send an image or content to the external display device in a wired or wireless manner through the communication interface 120, or send an image or content to the external display device through the input / output interface 160.

[0121] The operation interface 150 may be implemented in a device such as a button, a touch pad, a mouse or a keyboard, or may be implemented in a touch screen that can perform an operation input function in addition to the above-mentioned display function. Here, the button may be any of various types of buttons, such as a mechanical button, a touch pad or a scroll wheel, which is located on any area of ​​the main body appearance of the electronic device 100, such as its front surface, side surface or rear surface.

[0122] The input / output interface 160 may be any one of a high-definition multimedia interface (HDMI), a mobile high-definition link (MHL), a universal serial bus (USB), a display port (DP), a thunderbolt interface, a video graphics array (VGA) port, a red, green, and blue (RGB) port, a D-subminiature (D-SUB), or a digital visual interface (DVI). The input / output interface 160 may receive an input of at least one of an audio and video signal or provide an output of at least one of an audio and video signal. In some implementation examples, the input / output interface 160 may include a port for inputting and outputting only an audio signal and a port for inputting and outputting only a video signal as separate ports thereof, or may be implemented as a single port for inputting and outputting both an audio signal and a video signal. In an embodiment, the electronic device 100 may send at least one of an audio and video signal to an external device (e.g., an external display device or an external speaker) through the input / output interface 160. For example, an output port included in the input / output interface 160 may be connected to an external device, and the electronic device 100 may send at least one of an audio and video signal to the external device through the output port.

[0123] Here, the input / output interface 160 may be connected to the communication interface. The input / output interface 160 may transmit information received from an external device to the communication interface, or transmit information received through the communication interface to the external device.

[0124] The speaker 170 may be a component that outputs not only various audio data but also various notification sounds or voice messages.

[0125] The microphone 180 may be a component that receives a user's voice or other sounds and converts them into audio data. The microphone 180 may receive the user's voice based on being activated. For example, the microphone 180 may be integrated with the electronic device 100 in the upper, front, or side directions of the electronic device 100. The microphone 180 may include various components, such as a microphone that collects the user's voice in an analog form, an amplifier circuit that amplifies the collected user's voice, an analog-to-digital (A / D) converter circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0126] Figure 4 is a view for explaining a plurality of modules for providing response information corresponding to a user input.

[0127] refer to Figure 4 The electronic device 100 may include at least one of a user input processing module 111, a user log storage module 112, a sample prompt determination module 113, an additional information acquisition module 114, and an input prompt generation module 115 or a response providing module 116.

[0128] The user input processing module 111 may be a module configured to receive a user input. The user input processing module 111 may include at least one of an audio processing module 111-1, a text processing module 111-2, and an image processing module 111-3.

[0129] The audio processing module 111-1 may be a module configured to process audio data acquired through a microphone. The audio data may be an audio analog signal or an audio digital signal. In an embodiment, the audio processing module 111-1 may be or include a speech processing module. The audio processing module may use a speech recognizer to convert an audio signal into text information. The audio processing module may determine whether the received audio data is speech data. Based on the audio data being speech data, the audio processing module may use a speech recognizer to convert the speech data into text data or, for example, text information.

[0130] The user input processing module 111 may determine the text information converted using the audio processing module as the user input.

[0131] The text processing module 111-2 may be a module configured to process input data acquired through a keyboard. The text processing module 111-2 may be a module configured to acquire text information input by a user. The user input processing module 111 may determine the text information as user input using the text processing module.

[0132] The image processing module may be a module configured to process image data acquired by a camera. The image processing module may be a module configured to convert or change image data into a form in which the image data can be used in another module.

[0133] According to various embodiments, the image processing module 111 - 3 may extract specific information from the image data using an optical character reader (OCR) function. The user input processing module 111 may determine the information extracted using the image processing module as a user input.

[0134] According to various embodiments, the image processing module 111 - 3 may change the resolution of the image data. The user input processing module 111 may determine the image data whose resolution is changed using the image processing module as the user input.

[0135] According to various embodiments, the image processing module 111 - 3 may determine the image data itself as the user input without converting or changing operations.

[0136] The user input processing module 111 may send the user input to at least one of the sample prompt determination module 113 , the additional information acquisition module 114 , and the input prompt generation module 115 .

[0137] The user log storage module 112 may be a module configured to store user logs related to user behavior. The user log may include at least one of user input input from the electronic device 100, content information (e.g., audio, image, or text) provided based on the user input, instruction information executed based on the user input, or application information executed based on the user input.

[0138] The user log storage module 112 may send the user log to at least one of the sample prompt determination module 113 and the additional information acquisition module 114 .

[0139] According to various embodiments, at least one of the sample prompt determination module 113 and the additional information acquisition module 114 may use a user log.

[0140] According to various embodiments, neither the sample prompt determination module 113 nor the additional information acquisition module 114 may use user logs.

[0141] The sample prompt determination module 113 may be a module configured to determine a sample prompt corresponding to a user input among a plurality of sample prompts. The sample prompt determination module 113 may include at least one of a plurality of sample prompt storage modules 113-1, a task determination module 113-2, and a sample prompt selection module 113-3.

[0142] The sample prompt may include at least one of description information indicating the sample prompt and sample data related to the response information. The sample data may be data indicating a structure of a plurality of detailed information included in the response information.

[0143] The multiple sample prompt storage module 113-1 may store predetermined sample prompts. According to various embodiments, each of the multiple sample prompts may correspond to a predetermined task. For example, a first sample prompt corresponding to a first task and a second sample prompt corresponding to a second task may be stored in the multiple sample prompt storage module 113-1. According to various embodiments, the multiple sample prompts may correspond to the first task or the second task.

[0144] The task determination module 113-2 may be a module configured to determine which task the user input is related to. For example, the task may be a dialogue task, a question task, a device control task, etc. In addition, the task determination module 113-2 may identify the task corresponding to the user input by considering the user log and the user input together.

[0145] The sample prompt selection module may be a module configured to select a sample prompt corresponding to the user input from a plurality of sample prompts. The sample prompt selection module 113-3 may select a sample prompt corresponding to the identified task based on the task determination module 113-2 identifying the task corresponding to the user input.

[0146] The sample prompt determination module 113 may select, identify or determine a sample prompt based on at least one of the user input received from the user input processing module 111 or the user log received from the user log storage module 112. The sample prompt determination module 113 may send the selected sample prompt to the input prompt generation module 115.

[0147] The additional information acquisition module 114 may be a module configured to acquire additional information corresponding to a user input. The additional information acquisition module 114 may include at least one of an additional information determination module 114-1, an additional information receiver module 114-2, and an additional information analysis module 114-3.

[0148] The additional information determination 114-1 module may be a module configured to determine whether additional information corresponding to the user input is required and determine whether any additional information is required. The additional information determination module 114-1 may determine whether a complete response can be generated based on the user input alone.

[0149] The additional information determination module 114-1 may determine that additional information is not necessary based on determining that a complete response can be generated based only on the user input. The additional information determination module 114-1 may determine that additional information is necessary based on determining that a complete response cannot be generated based only on the user input.

[0150] The additional information determination module 114-1 may determine what additional information is necessary to generate a complete response based on determining that additional information is necessary. The additional information determination module 114-1 may request the determined (or necessary) additional information from the additional information receiver module 114-2.

[0151] The additional information receiver module 114 - 2 may be a module configured to acquire additional information requested to the additional information determination module.

[0152] According to various embodiments, the additional information receiver module 114 - 2 may acquire additional information using information in the electronic device 100 .

[0153] According to various embodiments, the additional information receiver module 114-2 may acquire additional information through the second server 300. The additional information receiver module 114-2 may request the determined (or necessary) additional information from the second server 300. The second server 300 may acquire or generate the additional information requested by the additional information acquisition module 311 using the additional information acquisition module 114. In addition, the second server 300 may send the additional information to the additional information receiver module.

[0154] The additional information acquisition module 114 may acquire the additional information through the additional information receiver module.

[0155] The additional information analysis module 114-3 may be a module configured to determine whether the additional information obtained or received is data suitable for inclusion in the input prompt. The additional information obtained may be in a form that is not suitable for including the information in the input prompt. In addition, the additional information obtained may have a depth or level that makes it difficult for the information to be used to generate response information. Therefore, the additional information analysis module 114-3 may analyze the received additional information. As a result of the analysis, the additional information analysis module may obtain processed or detailed additional information.

[0156] The additional information acquisition module 114 may send the additional information or, for example, the processed additional information to the input prompt generation module 115 .

[0157] The input prompt generation module 115 may be a module configured to generate an input prompt based on at least one of a user input, a sample prompt, and additional information. The input prompt generation module 115 may include at least one of a module 115-1 for reflecting the user input to the sample prompt and a module 115-2 for reflecting the additional information to the sample prompt.

[0158] The module 115 - 1 for reflecting the user input to the sample prompt may be a module configured to add the user input or, for example, information corresponding to the user input to a specific sample prompt selected by the sample prompt determination module 113 .

[0159] The module 115 - 2 for reflecting additional information to the sample hint may be a module configured to add the additional information or, for example, processed additional information to a specific sample hint selected by the sample hint determination module 113 .

[0160] The input prompt generation module 115 may generate or acquire an input prompt including at least one of a user input and additional information in a specific sample prompt selected by the sample prompt determination module 113. In addition, the input prompt generation module 115 may transmit the generated input prompt to the first server 200.

[0161] The first server 200 may be a server configured to generate response information corresponding to the input data. The first server 200 may include a large language model 211. The first server 200 may use the large language model to generate response information corresponding to the input prompt. The first server 200 may send the generated response information to the response providing module 116.

[0162] The response providing module 116 may receive response information from the first server 200. The response providing module 116 may provide the received response information to the user. The response providing module 116 may include a user interface (UI) generating module 116-1. The response providing module 116 may generate a response UI based on the received response information. In addition, the response providing module 116 may provide the generated response UI to the user. The response UI may include at least one of audio, text, and image.

[0163] Figure 5 is a view for explaining a process of providing response information according to various embodiments.

[0164] refer to Figure 5 , the user input processing module 111 can receive user input ("How is the weather?"). The user input can be text information. Based on the fact that the user input is voice information, the user input processing module 111 can convert the voice information into text information.

[0165] The user input processing module 111 may send the user input (“How’s the weather?”) to at least one of the sample prompt determination module 113 , the additional information acquisition module 114 , or the input prompt generation module 115 .

[0166] The user log storage module 112 may obtain the user log (“Let’s meet at Gangnam Station, Seoul tomorrow”). In addition, the user log storage module 112 may send the user log (“Let’s meet at Gangnam Station, Seoul tomorrow”) to at least one of the sample prompt determination module 113, the additional information acquisition module 114, and the input prompt generation module 115.

[0167] The sample prompt determination module 113 may receive a user input (“How’s the weather?”) from the user input processing module 111. In addition, the sample prompt determination module 113 may receive a user log (“Let’s meet at Gangnam Station, Seoul tomorrow”) from the user log storage module 112. The sample prompt determination module 113 may identify, determine, or select a sample prompt based on at least one of the user input (“How’s the weather?”) and the user log (“Let’s meet at Gangnam Station, Seoul tomorrow”). The sample prompt determination module 113 may send the determined sample prompt to the input prompt generation module 115.

[0168] The additional information acquisition module 114 may receive a user input ("How is the weather?") from the user input processing module 111. In addition, the additional information acquisition module 114 may receive a user log ("Let's meet at Gangnam Station, Seoul tomorrow") from the user log storage module 112. The additional information acquisition module 114 may determine the additional information based on at least one of the user input ("How is the weather?") and the user log ("Let's meet at Gangnam Station, Seoul tomorrow").

[0169] The additional information acquisition module 114 may determine the additional information necessary for generating the response information corresponding to the user input. The additional information acquisition module 114 may determine the necessary additional information ("tomorrow's weather data"). In addition, the additional information acquisition module 114 may request the necessary additional information from the second server 300. For example, the additional information acquisition module 114 may send a control signal requesting the additional information to the user log storage module 112. The control signal may include information for requesting the additional information and the necessary additional information ("tomorrow's weather data").

[0170] The second server 300 may acquire weather data based on the additional information request received from the additional information acquisition module 114. The second server 300 may acquire necessary additional information ("tomorrow's weather data") from the received control signal. In addition, the second server 300 may send information ("today's weather data and tomorrow's weather data") corresponding to the necessary additional information ("tomorrow's weather data") to the additional information acquisition module 114. According to various embodiments, the second server 300 may send only the information requested by the additional information acquisition module 114. However, there may be no module in the second server 300 for selectively sending only the requested specific data. The second server 300 may send all weather-related data to the additional information acquisition module 114.

[0171] The additional information acquisition module 114 may receive information (“today's weather data and tomorrow's weather data”) from the second server 300. The received information (“today's weather data and tomorrow's weather data”) may be described as acquired additional information.

[0172] The additional information acquisition module 114 may acquire information corresponding to necessary additional information by analyzing the acquired additional information. The information acquired by the additional information acquisition module 114 may be described as analyzed additional information.

[0173] The additional information acquisition module 114 may acquire the final additional information (“tomorrow's weather data”) by analyzing the data transmitted by the second server 300 . In addition, the additional information acquisition module 114 may transmit the final additional information (“tomorrow's weather data”) to the input prompt generation module 115 .

[0174] The input prompt generation module 115 may receive user input ("How's the weather?") from the user input processing module 111. In addition, the input prompt generation module 115 may receive a user log ("Let's meet at Gangnam Station, Seoul tomorrow") from the user log storage module 112. In addition, the input prompt generation module 115 may receive a determined sample prompt from the sample prompt determination module 113. In addition, the input prompt generation module 115 may receive final additional information ("tomorrow's weather data") from the additional information acquisition module 114. The input prompt generation module 115 generates an input prompt based on at least one of the user input ("How's the weather?"), the user log ("Let's meet at Gangnam Station, Seoul tomorrow"), the determined sample prompt, and the additional information ("tomorrow's weather data"). The input prompt generation module 115 may send the generated input prompt to the first server 200.

[0175] The first server 200 may receive an input prompt from the input prompt generating module 115. In addition, the first server 200 may generate response information based on the received input prompt. The first server 200 may send the generated response information to the response providing module 116. The first server 200 may obtain at least one of the user input ("How is the weather?"), the user log ("Let's meet at Gangnam Station, Seoul tomorrow"), the determined sample prompt, and the additional information ("tomorrow's weather data") from the input prompt. In addition, the first server 200 may generate response information based on at least one of the user input ("How is the weather?"), the user log ("Let's meet at Gangnam Station, Seoul tomorrow"), the determined sample prompt, and the additional information ("tomorrow's weather data").

[0176] The response providing module 116 may receive response information from the first server 200. The response providing module 116 may provide the response information received from the first server 200 to the user.

[0177] Figure 6 is a view for explaining a process of providing response information according to various embodiments.

[0178] refer to Figure 6 , the user input processing module 111 may receive user input ("How much is 500 satoshi worth?"). The user input may be text information. Based on the user input being voice information, the user input processing module 111 may convert the voice information into text information.

[0179] The user input processing module 111 may send the user input (“How much is 500 satoshis worth?”) to at least one of the sample prompt determination module 113 , the additional information acquisition module 114 , and the input prompt generation module 115 .

[0180] The user log storage module 112 may obtain the user log (“Bitcoin application execution”). In addition, the user log storage module 112 may send the user log (“Bitcoin application execution”) to at least one of the sample prompt determination module 113, the additional information acquisition module 114, and the input prompt generation module 115.

[0181] The sample prompt determination module 113 may receive a user input (“How much is 500 satoshis worth?”) from the user input processing module 111. In addition, the sample prompt determination module 113 may receive a user log (“Bitcoin application execution”) from the user log storage module 112. The sample prompt determination module 113 may identify, determine, or select a sample prompt based on at least one of the user input (“How much is 500 satoshis worth?”) and the user log (“How much is 500 satoshis worth?”). The sample prompt determination module 113 may send the determined sample prompt to the input prompt generation module 115.

[0182] The additional information acquisition module 114 may receive a user input (“how much is 500 satoshis worth?”) from the user input processing module 111. In addition, the additional information acquisition module 114 may receive a user log (“Bitcoin application execution”) from the user log storage module 112. The additional information acquisition module 114 may acquire additional information based on at least one of the user input (“how much is 500 satoshis worth?”) and the user log (“Bitcoin application execution”).

[0183] The additional information acquisition module 114 may determine the additional information necessary for generating the response information corresponding to the user input. The additional information acquisition module 114 may determine the necessary additional information ("the price of 1 Bitcoin" and "the relationship between Bitcoin and 1 Satoshi"). In addition, the additional information acquisition module 114 may request the necessary additional information from the second server 300. For example, the additional information acquisition module 114 may send a control signal requesting the additional information to the user log storage module 112. The control signal may include information for requesting the additional information and the necessary additional information ("the price of 1 Bitcoin" and "the relationship between Bitcoin and 1 Satoshi").

[0184] The second server 300 may acquire “the price of 1 Bitcoin” and “the relationship between Bitcoin and 1 Satoshi” based on the additional information request received from the additional information acquisition module 114. The second server 300 may acquire the necessary additional information (“the price of 1 Bitcoin” and “the relationship between Bitcoin and 1 Satoshi”) from the received control signal. In addition, the second server 300 may send information (“1 BTC: $20000” and “1 Satoshi: 0.00000001 BTC”) corresponding to the necessary additional information (“the price of 1 Bitcoin” and “the relationship between Bitcoin and 1 Satoshi”) to the additional information acquisition module 114. The additional information acquisition module 114 may receive the information (“1 BTC: $20000” and “1 Satoshi: 0.00000001 BTC”) from the second server 300. The received information (“1 BTC: $20000” and “1 Satoshi: 0.00000001 BTC”) may be described as the acquired additional information or the final additional information.

[0185] According to various embodiments, the additional information acquisition module 114 may acquire the final additional information (“1 BTC: $20,000” and “1 satoshi: 0.00000001 BTC”) by analyzing the data sent by the second server 300. In addition, the additional information acquisition module 114 may send the final additional information (“1 BTC: $20,000” and “1 satoshi: 0.00000001 BTC”) to the input prompt generation module 115. For example, the additional information acquisition module 114 may receive Bitcoin-related web page information from the second server 300. In addition, the additional information acquisition module 114 may acquire the final additional information (“1 BTC: $20,000” and “1 satoshi: 0.00000001 BTC”) by analyzing the received web page information.

[0186] The input prompt generation module 115 may receive a user input (“How much is 500 satoshis worth?”) from the user input processing module 111. In addition, the input prompt generation module 115 may receive a user log (“Bitcoin application execution”) from the user log storage module 112. In addition, the input prompt generation module 115 may receive a determined sample prompt from the sample prompt determination module 113. In addition, the input prompt generation module 115 may receive final additional information (“the price of 1 Bitcoin” and “the relationship between Bitcoin and 1 Satoshi”) from the additional information acquisition module 114. The input prompt generation module 115 may generate an input prompt based on at least one of the user input (“How much is 500 satoshis worth?”), the user log (“Bitcoin application execution”), the determined sample prompt, and the additional information (“the price of 1 Bitcoin” and “the relationship between Bitcoin and 1 Satoshi”). The input prompt generation module 115 may send the generated input prompt to the first server 200.

[0187] The first server 200 may receive an input prompt from the input prompt generation module 115. In addition, the first server 200 may generate response information based on the received input prompt. The first server 200 may send the generated response information to the response providing module 116. The first server 200 may obtain at least one of the user input ("How much is 500 satoshis worth?"), the user log ("Bitcoin application execution"), the determined sample prompt, and the additional information ("The price of 1 Bitcoin" and "The relationship between Bitcoin and 1 Satoshi") from the input prompt. In addition, the first server 200 may generate response information based on at least one of the user input ("How much is 500 satoshis?"), the user log ("Bitcoin application execution"), the determined sample prompt, or the additional information ("The price of 1 Bitcoin" and "The relationship between Bitcoin and 1 Satoshi").

[0188] The response providing module 116 may receive response information from the first server 200. The response providing module 116 may provide the response information received from the first server 200 to the user.

[0189] Figure 7 is a view for explaining a process of providing response information according to various embodiments.

[0190] refer to Figure 7 , the user input processing module 111 may receive user input ("turn on the power at 7 pm"). The user input may be text information. Based on the user input being voice information, the user input processing module 111 may convert the voice information into text information.

[0191] The user input processing module 111 may send the user input (“turn on the power at 7 pm”) to at least one of the sample prompt determination module 113 , the additional information acquisition module 114 , and the input prompt generation module 115 .

[0192] The user log storage module 112 may acquire the user log (“It is too hot when I go home these days”). In addition, the user log storage module 112 may send the user log (“It is too hot when I go home these days”) to at least one of the sample prompt determination module 113, the additional information acquisition module 114, and the input prompt generation module 115.

[0193] The sample prompt determination module 113 may receive a user input (“turn on the power at 7pm”) from the user input processing module 111. In addition, the sample prompt determination module 113 may receive a user log (“it is too hot when I go home these days”) from the user log storage module 112. The sample prompt determination module 113 may identify, determine, or select a sample prompt based on at least one of the user input (“turn on the power at 7pm”) and the user log (“it is too hot when I go home these days”). The sample prompt determination module 113 may send the determined sample prompt to the input prompt generation module 115.

[0194] The additional information acquisition module 114 may receive a user input (“turn on the power at 7 pm”) from the user input processing module 111. In addition, the additional information acquisition module 114 may receive a user log (“it is too hot when I go home these days”) from the user log storage module 112. The additional information acquisition module 114 may acquire additional information based on at least one of the user input (“turn on the power at 7 pm”) and the user log (“it is too hot when I go home these days”).

[0195] The additional information acquisition module 114 may determine the additional information necessary for generating the response information corresponding to the user input. The additional information acquisition module 114 may determine the necessary additional information ("connected device" and "control code"). In addition, the additional information acquisition module 114 may request the necessary additional information from the second server 300. For example, the additional information acquisition module 114 may send a control signal requesting the additional information to the user log storage module 112. The control signal may include information for requesting the additional information and the necessary additional information ("connected device" and "control code").

[0196] The second server 300 may acquire the “connected device list” and the “control code list” based on the additional information request received from the additional information acquisition module 114. The second server 300 may acquire the necessary additional information (“connected device” and “control code”) from the received control signal. In addition, the second server 300 may send information (“connected device list” and “control code list”) corresponding to the necessary additional information (“connected device” and “control code”) to the additional information acquisition module 114. The additional information acquisition module 114 may receive information (“connected device list” and “control code list”) from the second server 300. The received information (“connected device list” and “control code list”) may be described as acquired additional information or final additional information.

[0197] According to various embodiments, the additional information acquisition module 114 may acquire the final additional information (“connected device list” and “control code list”) by analyzing the data transmitted by the second server 300. In addition, the additional information acquisition module 114 may send the final additional information (“connected device list” and “control code list”) to the input prompt generation module 115. For example, the additional information acquisition module 114 may receive management device information from the second server 300. In addition, the additional information acquisition module 114 may acquire the final additional information (“connected device list” and “control code list”) by analyzing the received management device information.

[0198] The input prompt generation module 115 may receive a user input (“turn on the power at 7pm”) from the user input processing module 111. In addition, the input prompt generation module 115 may receive a user log (“it’s too hot when I get home these days”) from the user log storage module 112. In addition, the input prompt generation module 115 may receive a determined sample prompt from the sample prompt determination module 113. In addition, the input prompt generation module 115 may receive final additional information (“a list of connected devices” and a “list of control codes”) from the additional information acquisition module 114. The input prompt generation module 115 may generate an input prompt based on at least one of the user input (“turn on the power at 7pm”), the user log (“it’s too hot when I get home these days”), the determined sample prompt, and the additional information (“a list of connected devices” and a “list of control codes”). The input prompt generation module 115 may send the generated input prompt to the first server 200.

[0199] The first server 200 may receive an input prompt from the input prompt generating module 115. In addition, the first server 200 may generate response information based on the received input prompt. The first server 200 may send the generated response information to the response providing module 116. The first server 200 may obtain at least one of the user input ("Turn on the power at 7pm"), the user log ("It's too hot when I get home these days"), the determined sample prompt, and the additional information ("List of connected devices" and "List of control codes") from the input prompt. In addition, the first server 200 may generate response information ("Instruction information for turning on the air conditioner") based on at least one of the user input ("Turn on the power at 7pm"), the user log ("It's too hot when I get home these days"), the determined sample prompt, and the additional information ("List of connected devices" and "List of control codes").

[0200] The response providing module 116 may receive response information ("instruction information for turning on the air conditioner") from the first server 200. The response providing module 116 may provide the user with the response information ("instruction information for turning on the air conditioner") received from the first server 200. According to various embodiments, the response providing module 116 may execute the reservation instruction to turn on the power of the air conditioner at a time point (7:00 p.m.) corresponding to the user input.

[0201] Figure 8 is a view for explaining a process of providing response information according to various embodiments.

[0202] refer to Figure 8, the user input processing module 111 may receive user input (“Can I wear this for mountain climbing?” and “image data”). The user input may include text information. Based on the user input being voice information, the user input processing module 111 may convert the voice information into text information. In addition, the user input may include image data (“synthesized image”).

[0203] The user input processing module 111 may send the user input (“Can I wear this for mountain climbing?” and “Image data”) to at least one of the sample prompt determination module 113 , the additional information acquisition module 114 , or the input prompt generation module 115 .

[0204] The user log storage module 112 may obtain the user log (“Let’s go hiking in the Alps”). In addition, the user log storage module 112 may send the user log (“Let’s go hiking in the Alps”) to at least one of the sample prompt determination module 113, the additional information acquisition module 114, and the input prompt generation module 115.

[0205] The sample prompt determination module 113 may receive the user input (“Can I wear this for mountain climbing?” and “image data”) from the user input processing module 111. In addition, the sample prompt determination module 113 may receive the user log (“Let’s go hiking in the Alps”) from the user log storage module 112. The sample prompt determination module 113 may identify, determine, or select a sample prompt based on at least one of the user input (“Can I wear this for mountain climbing?” and “image data”) and the user log (“Let’s go hiking in the Alps”). The sample prompt determination module 113 may send the determined sample prompt to the input prompt generation module 115.

[0206] The additional information acquisition module 114 may receive user input (“Can I wear this for mountain climbing?” and “image data”) from the user input processing module 111. In addition, the additional information acquisition module 114 may receive a user log (“Let’s go hiking in the Alps”) from the user log storage module 112. The additional information acquisition module 114 may acquire additional information based on at least one of the user input (“Can I wear this for mountain climbing?” and “image data”) and the user log (“Let’s go hiking in the Alps”).

[0207] The additional information acquisition module 114 may determine the necessary additional information to generate response information corresponding to the user input. The additional information acquisition module 114 may determine the necessary additional information ("mountain" and "image object"). In addition, the additional information acquisition module 114 may request the necessary additional information from the second server 300. For example, the additional information acquisition module 114 may send a control signal requesting additional information to the user log storage module 112. The control signal may include information for requesting additional information and the necessary additional information ("mountain" and "image object").

[0208] The second server 300 may acquire the image analysis information (“high heels”, “black”, “street”, and “female”) based on the additional information request received from the additional information acquisition module 114. The second server 300 may acquire the necessary additional information (“mountain” and “image object”) from the received control signal. In addition, the second server 300 may send information (“high heels”, “black”, “street”, and “female”) corresponding to the necessary additional information (“mountain” and “image object”) to the additional information acquisition module 114. The additional information acquisition module 114 may receive the information (“high heels”, “black”, “street”, “female”) from the second server 300. The received information (“high heels”, “black”, “street”, “female”) may be described as acquired additional information.

[0209] According to various embodiments, the additional information acquisition module 114 may acquire the final additional information ("high heels" and "female") by analyzing the data transmitted by the second server 300. In addition, the additional information acquisition module 114 may transmit the final additional information ("high heels" and "female") to the input prompt generation module 115. The additional information acquisition module 114 may select an object ("high heels" and "female") related to at least one of the user input and the user log from a plurality of objects ("high heels", "black", "street", and "female") included in the image. The additional information acquisition module 114 may determine the selected object ("high heels" and "female") as the final additional information.

[0210] The input prompt generation module 115 may receive user input (“Can I wear this for mountain climbing?” and “image data”) from the user input processing module 111. In addition, the input prompt generation module 115 may receive a user log (“Let’s go hiking in the Alps”) from the user log storage module 112. In addition, the input prompt generation module 115 may receive a determined sample prompt from the sample prompt determination module 113. In addition, the input prompt generation module 115 may receive final additional information (“high heels” and “female”) from the additional information acquisition module 114. The input prompt generation module 115 may generate an input prompt based on at least one of the user input (“Can I wear this for mountain climbing?” and “image data”), the user log (“Let’s go hiking in the Alps”), the determined sample prompt, or the additional information (“high heels” and “female”). The input prompt generation module 115 may send the generated input prompt to the first server 200.

[0211] The first server 200 may receive an input prompt from the input prompt generating module 115. In addition, the first server 200 may generate response information based on the received input prompt. The first server 200 may send the generated response information to the response providing module 116. The first server 200 may obtain at least one of the user input (“Can I wear this for mountain climbing?” and “image data”), the user log (“Let’s go hiking in the Alps”), the determined sample prompt, and the additional information (“high heels” and “female”) from the input prompt. In addition, the first server 200 may generate response information (“It’s dangerous! Can I recommend a place to go in high heels?”) based on at least one of the user input (“Can I wear this for mountain climbing?” and “image data”), the user log (“Let’s go hiking in the Alps”), the determined sample prompt, and the additional information (“high heels” and “female”).

[0212] The response providing module 116 may receive the response information (“It’s dangerous! Can I recommend a place to go wearing high heels?”) from the first server 200. The response providing module 116 may provide the user with the response information (“It’s dangerous! Can I recommend a place to go wearing high heels?”) received from the first server 200.

[0213] Fig. 9 is a view used to illustrate several sample prompts.

[0214] Fig. 9 The sample prompt 910 in may represent a weather prompt. The weather prompt may be a sample prompt selected based on user input related to the weather being received. The sample prompt 910 may include answer structure information for answers to weather, temperature, and precautions to be provided.

[0215] Fig. 9The sample prompt 920 in the example may represent a question answering prompt. The question answering prompt may be a sample prompt selected based on user input related to the question being received. The sample prompt 920 may include descriptive information related to the question answer and answer structure information.

[0216] Fig. 9 The sample prompt 930 in may represent a control code prompt. The control code prompt may be a sample prompt selected based on user input related to device control being received. The sample prompt 930 may include description information and answer structure information related to device control.

[0217] Fig. 9 The sample prompt 940 in may represent a chatbot prompt. The chatbot prompt may be a sample prompt selected based on user input related to the chatbot function being received. The sample prompt 940 may include description information and answer structure information related to the chatbot service.

[0218] Fig.10 is a view of the categories used to explain the sample prompts.

[0219] refer to Fig.10 In table 1010, multiple sample prompts may be assigned to a category. For example, the categories of the sample prompts may be weather, question answering, control code, and chatbot.

[0220] According to various embodiments, a sample prompt may exist in a category. For example, a sample prompt #02 may exist in the question answering category.

[0221] According to various embodiments, multiple sample prompts may exist in one category. For example, there may be sample prompts #01-1, #01-2, and #01-3 in the weather category. Each sample prompt may have a different answer structure. The sample prompt #01-1 may only provide weather as response information. The sample prompt #01-2 may provide weather and temperature as response information. The sample prompt #01-3 may provide weather, temperature, and humidity as response information.

[0222] As another example, the sample prompts of #04-1, #04-2, and #04-3 may exist in the chatbot category. Each sample prompt may have a different answer structure. The sample prompt of #01-1 may provide only text as response information. The sample prompt of #04-2 ​​may provide text and emoticons as response information. The sample prompt of #04-3 may provide control text, emoticons, and another application as response information.

[0223] Fig.11 is a view for explaining the operation of a task determination module according to various embodiments.

[0224] refer to Fig.11 In the embodiment 1110, the task determination module 113-2 may determine that the user input corresponds to a chatbot task based on receiving the user input ("It's really hot today"). The sample prompt determination module 113 may select a chatbot prompt from a plurality of sample prompts.

[0225] refer to Fig.11 In the embodiment 1120, the task determination module 113-2 may determine that the user input corresponds to the question answering task based on receiving the user input ("What's the temperature now?") and the user log ("It's really hot today" and "It's true"). The sample prompt determination module 113 may select a weather prompt from a plurality of sample prompts.

[0226] refer to Fig.11 In the embodiment 1130, the task determination module 113-2 may determine that the user input corresponds to the device control task based on receiving the user input ("Turn on the air conditioner?") and the user log ("It's really hot today," "Indeed," "What's the temperature now?" and "The temperature in Seoul is 33 degrees now"). The sample prompt determination module 113 may select a control code prompt from a plurality of sample prompts.

[0227] Fig.12 is a view for explaining the operation of the additional information determination module 114 - 1 according to various embodiments.

[0228] refer to Fig.12 In embodiment 1210, based on receiving user input ("How much are 500 bitcoins worth?"), the additional information determination module 114-1 may determine the bitcoin price ("coin market API", "application programming interface"), the relationship between bitcoins and satoshis (i.e., the search terms to be used in the search engine), and the bitcoin price*the relationship between bitcoins and satoshis*500 ("calculation formula") as necessary additional information.

[0229] refer to Fig.12 In embodiment 1220, based on receiving user input ("turn off the light closest to me"), the additional information determination module 114-1 may determine the user location ("ultra-wideband (UWB)"), the light list ("SmartThings API"), the distance between the user and the lights ("calculation formula"), and the sorting method ("the first result obtained by performing sorting in ascending order") as required additional information.

[0230] refer to Fig.12In embodiment 1230, based on receiving user input ("What's the weather like now?"), the additional information determination module 114-1 may determine the user location ("Global Positioning System (GPS)"), the current time, the weather at the user location, and the current time ("Weather API") as required additional information.

[0231] Fig.13 is a view for explaining an operation of an additional information receiver module according to various embodiments.

[0232] refer to Fig.13 , the additional information receiver module 1310 may acquire additional information based on at least one of a search engine server 1311, a database server 1312, an application program interface (API) server 1313, and a calculator application 1314. In an embodiment, the additional information receiver module 1310 may correspond to the additional information receiver module 114-2 discussed above.

[0233] The additional information receiver module 1310 may request additional information from an external server or an external application. In addition, the additional information receiver module 1310 may receive additional information from an external server or an external application.

[0234] The search engine server 1311 may acquire search results based on the keyword transmitted by the additional information receiver module 1310. The search engine server 1311 may transmit the search results to the additional information receiver module 1310.

[0235] The database server 1312 may acquire data corresponding to the additional information request transmitted by the additional information receiver module 1310. The database server 1312 may transmit the acquired data to the additional information receiver module 1310.

[0236] The API server 1313 may acquire data corresponding to the API information transmitted by the additional information receiver module 1310. The API server 1313 may transmit the acquired data to the additional information receiver module 1310.

[0237] The calculator application 1314 may acquire a calculation result corresponding to the calculation formula information transmitted by the additional information receiver module 1310. The calculator application 1314 may transmit the calculation result to the additional information receiver module 1310.

[0238] Fig.14 is a view for explaining the operation of an additional information determination module according to various embodiments.

[0239] refer to Fig.14In the embodiment 1410, based on receiving user input ("How much is Galaxy S22?"), the additional information determination module 114-1 can obtain requested information corresponding to the user input ("SELECT Price FROM GalaxyS22"). The requested information can indicate the required additional information. The additional information determination module can send the requested information to the database server. The additional information determination module 114-1 can obtain the additional information ("1 million Korean Won") from the database server 1312.

[0240] refer to Fig.14 In the embodiment 1420, based on receiving the user input ("If there are 5 packages of 6 candies each, how many candies are there in total?"), the additional information determination module 114-1 can obtain the requested information corresponding to the user input ("6*5"). The additional information determination module 114-1 can send the requested information to the calculator application. The additional information determination module 114-1 can obtain the additional information ("30") from the calculator application 1314.

[0241] refer to Fig.14 In the embodiment 1430, based on receiving the user input ("image data"), the additional information determination module 114-1 can obtain the requested information corresponding to the user input ("data representing the image"). The additional information determination module 114-1 can send the requested information to the image analyzer application 1431. The additional information determination module 114-1 can obtain the additional information ("apple object, red object, and the number of apple objects (3)") from the image analyzer application 1431.

[0242] refer to Fig.14 In the embodiment 1440, based on receiving the user input ("audio data: music"), the additional information determination module 114-1 can obtain the requested information corresponding to the user input ("data representing audio"). The additional information determination module 114-1 can send the requested information to the audio analyzer application 1441. The additional information determination module 114-1 can obtain the additional information ("music information: Hey Jude, Beatles") from the audio analyzer application 1441.

[0243] Fig.15 A view that explains an operation to obtain additional information.

[0244] refer to Fig.15 Based on receiving user input ("Can I wear this for hiking?"), user log ("Let's go hiking in the Alps"), and image data ("shoe image"), the additional information determination module 114-1 can obtain requested information 1510. The requested information 1510 can include a mountain object and an image object.

[0245] The additional information receiver module 114-2 may acquire additional information 1520 based on the requested information 1510. The additional information 1520 may include image analysis information (high-heeled shoe object, black object, street object, and female object).

[0246] The additional information analysis module 114-3 may analyze the received additional information 1520 and the user log. The additional information analysis module 114-3 may obtain the additional information (e.g., high-heeled shoe object and female object) related to the user input or the user log as the analysis result as the final additional information 1530. The additional information analysis module 114-3 may obtain the information of the Alps as the final additional information 1530 based on the user log ("Let's go hiking in the Alps").

[0247] Fig.16 is a view for explaining the operation of an additional information determination module according to various embodiments.

[0248] refer to Fig.16 , the electronic device 100 may obtain user input at operation S1610. The user input may be at least one of text information, image information, and audio information. The text information may be obtained through a keyboard. In addition, the electronic device 100 may use a text recognition technology in image information (e.g., an optical character reader (OCR)) to obtain text information. In addition, the electronic device 100 may use a text recognition technology in audio information (e.g., a voice recognition function) to obtain text information. The voice recognition function may be a function of converting an audio signal into text information.

[0249] In operation S1620, the electronic device 100 may select a sample prompt based on the user input. The prompt may be sample data of input data for providing response information. The prompt may include various information for generating response information. The prompt may include structural information of a response corresponding to the user input. The electronic device 100 may generate response information based on the structural information included in the prompt.

[0250] In operation S1630, the electronic device 100 may acquire additional information based on the user input. The electronic device 100 may acquire additional information related to the user input. The electronic device 100 may acquire various information related to the words included in the user input.

[0251] In operation S1640, the electronic device 100 may acquire an input prompt based on the user input, the sample prompt, and the additional information. The electronic device 100 may generate an input prompt by adding the user input and the additional information to the sample prompt.

[0252] In operation S1650, the electronic device 100 may provide response information based on the input prompt. The electronic device 100 may obtain the user input, the sample prompt, and the additional information included in the input prompt. The electronic device 100 may obtain the response information based on the user input, the sample prompt, and the additional information. The electronic device 100 may provide the obtained response information to the user.

[0253] although Fig.16 It is shown that operation S1630 is performed after operation S1620, but the embodiment is not limited thereto. For example, according to various embodiments, the electronic device 100 may first perform operation S1630, and then perform operation S1620. In addition, according to various embodiments, the electronic device 100 may perform operation S1620 and operation S1630 at the same time. The description of the order of the operation of selecting a sample prompt and the operation of obtaining additional information may be similarly applied to the following figures.

[0254] Fig.17 is a flowchart for explaining the operation of acquiring response information.

[0255] Fig.17 Operations S1710, S1720, S1730, S1740, and S1750 may correspond to Fig.16 Thus, the description may omit redundant descriptions thereof.

[0256] After acquiring the input hint, the electronic device 100 may transmit the input hint to the first server 200 in operation S1741. In operation S1742, the first server 200 may acquire response information based on the input hint received from the electronic device 100. In addition, the first server 200 may transmit the response information to the electronic device 100 in operation S1743.

[0257] The electronic device 100 may receive the response information from the first server 200. In addition, in operation S1750, the electronic device 100 may provide the response information to the user. According to various embodiments, the electronic device 100 may display the response information on the display 140. According to various embodiments, the electronic device 100 may output the response information through the speaker 170.

[0258] Fig.18 2 is a flowchart for explaining an operation of acquiring response information using the server 200.

[0259] Fig.18 Operations S1810, S1830, S1840, S1841, S1842, S1843, and S1850 may correspond to Fig.17Thus, the description may omit redundant descriptions thereof.

[0260] After acquiring the user input, in operation S1821, the electronic device 100 may identify a task corresponding to the user input from a plurality of predetermined tasks. Fig.11 Provides examples of operations related to this configuration.

[0261] In addition, in operation S1822, the electronic device 100 may select a sample prompt based on the identified task. The electronic device 100 may select a sample prompt corresponding to the identified task from a plurality of sample prompts stored in the memory 110. Subsequently, the electronic device 100 may perform operations S1830 to S1850.

[0262] Fig.19 is a flow chart for explaining an embodiment in which sample prompts are divided into separate groups.

[0263] Fig.19 Operations S1910, S1930, S1940, S1941, S1942, S1943, and S1950 may correspond to Fig.17 Thus, the description may omit redundant descriptions thereof.

[0264] After acquiring the user input, in operation S1921, the electronic device 100 may determine whether data other than text is included in the user input. The data other than text may indicate image data or audio data. The user input may include image data or audio data other than text data. The electronic device 100 may provide response information corresponding to the user input including image data or audio data.

[0265] The electronic device 100 may distinguish and process user inputs including only text data and user inputs including image data or audio data. The electronic device 100 may store sample prompts corresponding to user inputs including only text data as a first group. The first group may include a plurality of sample prompts, and the sample prompts included in the first group may be prompts for processing user inputs including only text data. The electronic device 100 may store sample prompts corresponding to user inputs including image data or audio data as a second group. The second group may include a plurality of sample prompts, and the sample prompts included in the second group may be prompts for processing user inputs including image data or audio data.

[0266] Based on the user input not including any data other than text (No in operation S1921), in operation S1922, the electronic device 100 may select a sample prompt corresponding to the user input from a plurality of sample prompts stored in the first group. Based on the fact that only text data is included in the user input, the electronic device 100 may select a sample prompt from the first group.

[0267] Based on the user input including data other than text (Yes in operation S1921), in operation S1923, the electronic device 100 may select a sample prompt corresponding to the user input from a plurality of sample prompts stored in the second group. Based on the image data or the image data included in the user input, the electronic device 100 may select a sample prompt from the second group. Subsequently, the electronic device 100 may perform operations S1930 to S1950.

[0268] Fig. 20 2 is a flowchart for explaining an operation of acquiring additional information using the second server 300 .

[0269] Fig. 20 Operations S2010, S2020, S2040, S2041, S2042, S2043, and S2050 may correspond to Fig.17 Thus, the description may omit redundant descriptions thereof.

[0270] After acquiring the user input, in operation S2031, the electronic device 100 may send the additional information request and the user input to the second server 300. The electronic device 100 may generate a control signal including information requesting the additional information and the user input. In addition, the electronic device 100 may send the generated control signal to the second server 300.

[0271] The second server 300 may receive an additional information request and a user input from the electronic device 100. Furthermore, in operation S2032, the second server 300 may generate additional information based on the user input. Furthermore, in operation S2033, the second server 300 may transmit the generated additional information to the electronic device 100.

[0272] In operation S2034, the electronic device 100 may acquire or receive additional information from the second server 300. Subsequently, the electronic device 100 may perform operations S2040 to S2050.

[0273] Fig.21 is a flow chart for explaining the operation of obtaining input prompts based on the probability of generating a complete response.

[0274] Fig.21Operations S2110, S2120, S2131, S2132, S2133, S2134, S2140, S2141, S2142, S2143, and S2150 may correspond to Fig. 20 Thus, the description may omit redundant descriptions thereof.

[0275] After acquiring the user input, in operation S2125, the electronic device 100 may acquire a first probability of generating a complete response based on the user input. A complete response may indicate a response that can accurately convey information corresponding to the user input. The electronic device 100 may acquire or calculate a first probability indicating the possibility of generating a complete response based on the user input.

[0276] In operation S2126, the electronic device 100 may determine whether the first probability is less than a threshold probability or a predetermined probability. Based on the first probability being less than the threshold probability (yes in operation S2126), the electronic device 100 may request additional information from the second server 300. Based on the first probability being less than the threshold probability, the electronic device 100 may determine that additional information is necessary. Subsequently, the electronic device 100 may perform operations S2131 to S2140.

[0277] Based on the first probability being not less than the threshold probability (No in operation S2126), in operation S2127, the electronic device 100 may obtain an input prompt based on the user input and the sample prompt. Based on the first probability being not less than the threshold probability, the electronic device 100 may generate an input prompt based only on the user input and the sample prompt without separately obtaining additional information.

[0278] Subsequently, the electronic device 100 may perform operations S2141 to S2150.

[0279] Fig. 22 is a flow chart for explaining an operation of determining additional information based on a probability of generating a complete response.

[0280] Fig. 22 Operations S2210, S2220, S2225, S2226, S2227, S2233, S2234, S2240, S2241, S2242, S2243, and S2250 may correspond to Fig.21 Thus, the description may omit redundant descriptions thereof.

[0281] Based on the first probability being determined or identified as being less than the threshold probability (yes in operation S2226), in operation S2228, the electronic device 100 may obtain a second probability of generating a complete response based on the sample additional information and the user input. The sample additional information may be described as virtual information or arbitrary information. The electronic device 100 may arbitrarily select any sample additional information from the multiple pieces of sample additional information stored in the memory 110. The electronic device 100 may obtain a second probability of generating a complete response based on the selected sample additional information and the user input.

[0282] According to various embodiments, the electronic device 100 may identify a keyword included in a user input and select sample additional information related to the identified keyword. Based on the presence of multiple pieces of sample additional information related to the keyword, the electronic device 100 may select the sample additional information in order of relevance. Fig.23 Additional description of the sample additional information.

[0283] The first probability obtained in step S2225 may be the probability of generating a complete response based only on the user input, and the second probability obtained in step S2228 may be the probability of generating a complete response based on the user input and the selected sample additional information.

[0284] In operation S2229, the electronic device 100 may determine whether the second probability is less than a threshold probability. According to various embodiments, the threshold probability of step S2226 and the threshold probability of step S2229 may be the same as each other. According to various embodiments, the threshold probability of step S2226 and the threshold probability of step S2229 may be different from each other. In this case, each threshold probability may be described as a first threshold probability or a second threshold probability.

[0285] Based on the second probability being less than the threshold probability (Yes in operation S2229), the electronic device 100 may reacquire the second probability of generating a complete response based on the new sample additional information and user input. The electronic device 100 may repeat operations S2228 to S2229 until the second probability is greater than or equal to the threshold probability.

[0286] Based on the second probability being not less than the threshold probability (No in operation S2229), in operation S2231, the electronic device 100 may send the sample additional information and the user input to the second server 300. The electronic device 100 may request the second server 300 for additional information corresponding to the sample additional information. The electronic device 100 may use the sample additional information to determine which information is needed to generate a complete response. Therefore, the electronic device 100 may obtain information of the same type as the sample additional information based on the current time point.

[0287] In operation S2232, the second server 300 may acquire additional information based on the sample additional information and the user input received from the electronic device 100. In addition, the second server 300 may transmit the additional information to the electronic device 100.

[0288] In operation S2234, the electronic device 100 may acquire additional information from the second server 300. In operation S2240, the electronic device 100 may acquire an input prompt based on the user input, the sample prompt, and the additional information.

[0289] Subsequently, the electronic device 100 may perform operations S2241 to S2250.

[0290] Fig.23 is a view for explaining the operation of a complete response determination module according to various embodiments.

[0291] refer to Fig.23 In the embodiment 2310, the complete response determination module 2300 may calculate the first probability as 10% based on receiving the user input ("What's the weather like?").

[0292] refer to Fig.23 In the embodiment 2320 of the present invention, the complete response determination module 2300 may calculate the second probability as 70% based on receiving the user input ("How is the weather?") and selecting the sample additional information ("Weather news"). For example, 80% may be a threshold probability associated with the second probability. The electronic device 100 may determine that it is difficult to generate a complete response based only on the user input ("How is the weather?") and the sample additional information ("Weather news"). The electronic device 100 may select new sample additional information.

[0293] refer to Fig.23 In the embodiment 2330 of the embodiment 2330, the complete response determination module 2300 may calculate the second probability as 90% based on receiving the user input ("How is the weather?") and selecting the sample additional information ("Weather news"). For example, 80% may be a threshold probability associated with the second probability. The electronic device 100 may determine that a complete response can be generated based on the user input ("How is the weather?") and the sample additional information ("Weather data").

[0294] In an embodiment, the sample additional information mentioned in embodiments 2320 and 2330 may not be data at the time point when the user input is received, but may be data at a past time point based on the time point when the user input is received. The sample additional information may be information used only to determine the probability of generating a complete response as indicated by the text itself, and may not be information used to actually generate response information.

[0295] Therefore, even if the second probability obtained based on the user input and the sample additional information is greater than or equal to the threshold probability, the electronic device 100 may request additional information. The electronic device 100 may determine which additional information to request based on the sample additional information. Based on the acquisition of additional information of the same type as the type of the sample additional information, the possibility of generating a complete response may be greater than or equal to the threshold probability. Therefore, the electronic device 100 may acquire additional information of the same type as the type of the sample additional information. The sample additional information may be information acquired before receiving the user input, and the additional information may be information acquired after receiving the user input.

[0296] Fig.24 is a flowchart for explaining the operation of obtaining input hints using a user log.

[0297] Fig.24 Operations S2410, S2440, S2441, S2442, S2443, and S2450 may correspond to Fig.17 Thus, the description may omit redundant descriptions thereof.

[0298] After acquiring the user input, the electronic device 100 may acquire a user log in operation S2415. The user log may indicate historical information showing that the user uses the electronic device 100. The user log may include at least one of a user session list, an application list used by the user, and a command list executed by the electronic device 100.

[0299] In operation S2421, the electronic device 100 may select a sample prompt based on a user log and a user input. In operation S2431, the electronic device 100 may acquire additional information based on the user log and the user input.

[0300] Subsequently, the electronic device 100 may perform operations S2440 to S2450.

[0301] Fig.25 is a flowchart for explaining the operation of acquiring additional information.

[0302] Fig.25 Operations S2510, S2520, S2540, S2541, S2542, S2543, and S2550 may correspond to Fig.17 Operations S1710, S1720, S1740, S1741, S1742, S1743 and S1750.

[0303] After obtaining the user input, in operation S2531, the electronic device 100 may determine whether time information is necessary for a complete response. Based on the time information being necessary for a complete response (yes in operation S2531), in operation S2532, the electronic device 100 may use a time information extractor to obtain time information. After obtaining the time information using the time information extractor, in operation S2533, the electronic device 100 may determine whether location information is necessary for a complete response.

[0304] Furthermore, based on the fact that the time information is not necessary for a complete response (No in operation S2531), in operation S2533, the electronic device 100 may determine whether the location information is necessary for a complete response.

[0305] Based on the location information being necessary for a complete response (Yes in operation S2533), in operation S2534, the electronic device 100 may use a location information extractor to acquire the location information. After acquiring the location information using the location information extractor, in operation S2535, the electronic device 100 may determine whether external system information is necessary for a complete response.

[0306] Furthermore, based on that the location information is not necessary for a complete response (No in operation S2533), in operation S2535, the electronic device 100 may determine whether external system information is necessary for a complete response.

[0307] Based on the fact that the external system information is necessary for a complete response (Yes in operation S2535), in operation S2536, the electronic device 100 may use an external system information extractor to obtain the external system information. After obtaining the external system information using the external system information extractor, in operation S2537, the electronic device 100 may determine whether the multi-mode information is necessary for a complete response.

[0308] Furthermore, based on that the information from the external system is not necessary for a complete response (No in operation S2535 ), in operation S2537 , the electronic device 100 may determine whether multimodal information is necessary for a complete response.

[0309] Based on the multimodal information being necessary for a complete response (Yes in operation S2537), in operation S2538, the electronic device 100 may acquire the multimodal information using the multimodal information extractor. After acquiring the multimodal information using the multimodal information extractor, in operation S2539, the electronic device 100 may acquire additional information.

[0310] Furthermore, based on that the multi-mode information is not necessary for a complete response (No in operation S2537), in operation S2539, the electronic device 100 may acquire additional information.

[0311] For example, the electronic device 100 may acquire the additional information based on at least one of time information, location information, information from an external system, and multimodal information.

[0312] Subsequently, the electronic device 100 may perform operations S2540 to S2550.

[0313] Fig.26 A view for explaining a user interface (UI) for displaying information for obtaining an input hint.

[0314] refer to Fig.26 , the electronic device 100 may provide the user with a screen 2610 for generating an input prompt. The screen 2610 may include at least one of a UI 2611 representing a voice recognition notification, a UI 2612 representing a user input, a UI 2613 representing a user log, a UI 2614 representing a sample prompt, a UI 2615 representing additional information, and a UI 2616 for receiving a user input to generate response information.

[0315] The electronic device 100 may receive a user command for re-inputting the user input through the UI 2612. In addition, the electronic device 100 may receive a user command for modifying the user log through the UI 2613. In addition, the electronic device 100 may receive a user command for modifying the additional information through the UI 2615.

[0316] Fig. 27 A view that interprets the user interface (UI) used to display input prompts and response information.

[0317] refer to Fig. 27 , the electronic device 100 may provide a screen 2710 related to response information to the user. The screen 2710 may include at least one of a UI 2711 representing a voice recognition notification, a UI 2712 representing an input prompt, and a UI 2713 representing response information.

[0318] In addition, the electronic device 100 may receive a user command for modifying the input prompt through the UI 2712. Based on the provided response information not including information desired by the user, the user may request new response information by modifying the input prompt.

[0319] Fig.28 is a flowchart for explaining a control method of the electronic device 100 according to various embodiments.

[0320] refer to Fig.28, a control method of an electronic device configured to store a plurality of sample prompts and communicate with a server including a large language model may include: obtaining user input, such as a user input of a user, at operation S2805; selecting a sample prompt corresponding to the user input from a plurality of stored sample prompts at operation S2810; obtaining additional information corresponding to the user input at operation S2815; obtaining an input prompt based on the user input, the sample prompt, and the additional information at operation S2820; sending the input prompt to the server at operation S2825; and providing response information based on response information corresponding to the input prompt received from the server at operation S2830. For example, the response information may be provided to the user.

[0321] In an embodiment, the sample prompt may include at least one of description information representing the sample prompt, at least one example information, and response structure information.

[0322] In an embodiment, the selection of the sample prompt in operation S2810 may include identifying a task corresponding to the user input, and selecting the sample prompt based on the identified task.

[0323] In an embodiment, the server may be a first server, and the acquisition of additional information in operation S2815 may include: determining whether additional information corresponding to the user input is to be used, sending a signal requesting additional information to a second server based on determining the additional information to be used, and receiving the additional information from the second server.

[0324] In an embodiment, the acquiring of the additional information in operation S2815 may include acquiring a first probability of generating a complete response based on a user input, and acquiring the additional information based on the first probability being less than a threshold probability.

[0325] In an embodiment, the acquisition of additional information in operation S2815 may include: acquiring a second probability of generating a complete response based on user input and at least one piece of sample additional information among multiple pieces of sample additional information; and acquiring additional information based on at least one piece of sample additional information based on the second probability being greater than or equal to a threshold probability.

[0326] In an embodiment, the acquiring of the input hint in operation S2820 may include acquiring the input hint by combining the sample hint with the user input and the additional information.

[0327] In an embodiment, the identifying of the sample prompt in operation S2810 may include acquiring a user log corresponding to a user activity within a threshold time, and selecting a sample prompt corresponding to a user input based on the user log and the user input.

[0328] In an embodiment, the acquiring of the additional information in operation S2815 may include acquiring a user log corresponding to a user activity within a threshold time, and acquiring the additional information based on the user log and a user input.

[0329] In an embodiment, the user input may include at least one of audio information, text information, and image information.

[0330] In an embodiment, Fig.28 The control method of the electronic device shown can be performed by Figure 2 or Figure 3 The present invention is performed by the electronic device 100 having the configuration of the present invention, and may also be performed by an electronic device having another configuration.

[0331] In an embodiment, the method according to various embodiments of the present disclosure described above may be implemented in the form of an application installable on an electronic device.

[0332] In addition, the methods according to the various embodiments of the present disclosure described above may be implemented only by software upgrade or hardware upgrade of the electronic device.

[0333] Furthermore, the various embodiments of the present disclosure described above may be performed by an embedded server included in the electronic device or an external server of at least one of the electronic device and the display apparatus.

[0334] In an embodiment, according to various embodiments of the present disclosure, the various embodiments described above may be implemented in software including instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). A machine may be a device that calls stored instructions from a storage medium, may be operated based on the called instructions, and may include an electronic device according to an embodiment of the present disclosure. Based on the instructions being executed by the processor, the processor may directly execute the functions corresponding to the instructions, or other components may execute the functions corresponding to the instructions under the control of the processor. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" indicates that the storage medium is tangible without including a signal, and does not distinguish whether the data is stored semi-permanently or temporarily in the storage medium.

[0335] In addition, according to one or more embodiments of the present disclosure, the method according to the various embodiments described above may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online through an application store (e.g., PlayStoreTM). Based on online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily provided in a storage medium such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0336] In addition, each component (e.g., module or program) in the various embodiments described above may include a single entity or multiple entities, and some of the corresponding subcomponents described above may be omitted, or other subcomponents may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity, and the functions performed by the respective corresponding components may be performed before being integrated in the same or similar manner. The operations performed by the modules, programs or other components according to the various embodiments may be performed in a sequential manner, in a parallel manner, in an iterative manner or in a heuristic manner, and at least some operations may be performed in a different order or omitted, or other operations may be added.

[0337] Although specific embodiments of the present disclosure are shown and described above, the present disclosure is not limited to the above specific embodiments, and various modifications may be made by those skilled in the art to which the present disclosure belongs without departing from the scope and spirit of the present disclosure disclosed in the appended claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. An electronic device, comprising: a memory configured to store a plurality of sample prompts; a communication interface configured to communicate with a server including a large language model; as well as At least one processor configured to: Get user input from the user, selecting a sample prompt corresponding to the user input from the plurality of stored sample prompts, obtaining additional information corresponding to the user input, obtaining an input prompt based on the user input, the sample prompt and the additional information, sending the input prompt to the server using the communication interface; as well as Based on receiving response information corresponding to the input prompt from the server through the communication interface, the response information is provided to the user.

2. The electronic device according to claim 1, wherein: The sample prompt includes at least one of description information, example information, and response structure information representing the sample prompt.

3. The electronic device according to claim 1, wherein: The at least one processor is further configured to: identifying a task corresponding to the user input, and The sample prompts are selected based on the identified task.

4. The electronic device according to claim 1, wherein: The server is a first server, and wherein the at least one processor is further configured to: determining whether to use the additional information corresponding to the user input, Based on determining that the additional information is to be used, sending a signal requesting the additional information to a second server using the communication interface, and The additional information is received from the second server using the communication interface.

5. The electronic device according to claim 1, wherein: The at least one processor is further configured to: obtaining a first probability of generating a complete response based on the user input, and The additional information is acquired based on that the first probability is less than a threshold probability.

6. The electronic device according to claim 5, wherein: The at least one processor is further configured to: obtaining a second probability of generating the complete response based on the user input and at least one piece of sample additional information among the plurality of pieces of sample additional information, and Based on the second probability being greater than or equal to the threshold probability, the additional information is acquired based on at least one piece of sample additional information.

7. The electronic device according to claim 1, wherein: The at least one processor is further configured to obtain the input prompt by combining the sample prompt with the user input and the additional information.

8. The electronic device according to claim 1, wherein: The at least one processor is further configured to: Obtain user logs corresponding to user activity within a threshold time, and The sample prompt is selected based on the user log and the user input.

9. The electronic device according to claim 1, wherein: The at least one processor is further configured to: Obtain user logs corresponding to user activity within a threshold time, and The additional information is acquired based on the user log and the user input.

10. The electronic device according to claim 1, wherein: The user input includes at least one of audio information, text information and image information.

11. A method of controlling an electronic device, the electronic device being configured to store a plurality of sample prompts and communicate with a server including a large language model, the method comprising: Get user input from the user; selecting a sample prompt corresponding to the user input from the plurality of stored sample prompts; Acquire additional information corresponding to the user input; Obtaining an input prompt based on the user input, the sample prompt, and the additional information; sending the input prompt to the server; as well as Based on receiving response information corresponding to the input prompt from the server, the response information is provided to the user.

12. The method of claim 11, wherein: The sample prompt includes at least one of description information representing the sample prompt, at least one example information, and response structure information.

13. The method of claim 11, wherein: The sample prompt selections include: identifying a task corresponding to the user input, and The sample prompts are selected based on the identified task.

14. The method of claim 11, wherein: The server is a first server, and wherein acquiring the additional information comprises: determining whether the additional information corresponding to the user input needs to be used, Based on determining that the additional information is to be used, sending a signal requesting the additional information to a second server, and The additional information is received from the second server.

15. The method of claim 11, wherein: The acquisition of the additional information includes: obtaining a first probability of generating a complete response based on the user input, and The additional information is acquired based on that the first probability is less than a threshold probability.

Images