Electronic device and its control method

Abstract

An electronic device is provided. The electronic device includes a display and a processor, and the processor is configured to control the electronic device to: obtain text corresponding to an input based on a machine reading comprehension (MRC) model, obtain a plurality of operation information by dividing the text into operation units of the electronic device, obtain sequence information of the plurality of operation information based on the plurality of operation information and the obtained text, and control the display to provide a guidance user interface (UI) so as to sequentially execute the plurality of operations based on the sequence information.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof. For example, the present disclosure relates to an electronic device using a machine reading comprehension (MRC) model and a control method thereof. Background Art

[0002] The development of electronic technology has led to the development and distribution of various types of electronic devices.

[0003] Not only have the types of electronic devices been diversified, but also the functions and operations that electronic devices can perform have been diversified.

[0004] However, due to the complexity of various functions, operations, etc. of electronic devices, there is a problem that older users who are not familiar with the latest electronic devices or users who are not good at operating electronic devices may have poor accessibility to various functions and operations.

[0005] Therefore, there is a need to provide a method for a user to flexibly and actively use an electronic device (e.g., a method for performing functions and operations, a control method, or an approach method) through a machine learning algorithm, rather than simply providing a method for performing various functions and operations and an approach method in a fixed and formal manner based on a manual of the electronic device. Summary of the Invention

[0006]

Technical Problem

[0007] Embodiments of the present disclosure provide an electronic device and a control method thereof, the electronic device being configured to provide a user interface (UI) for guiding a usage method based on a manual of the electronic device.

[0008]

Technical Solution

[0009] An electronic device according to an exemplary embodiment includes: a display and a processor configured to control the electronic device to: obtain text corresponding to an input based on a machine reading comprehension (MRC) model, obtain a plurality of operation information by dividing the text into operation units of the electronic device, obtain order information of the plurality of operation information based on the plurality of operation information and the obtained text, and control the display to provide a guidance UI so as to sequentially perform a plurality of operations based on the order information.

[0010] The operation unit may be based on a context that requires a user touch input.

[0011] The processor is configured to control the electronic device to: obtain current setting state information on the electronic device, identify at least one operation information among the plurality of operation information pre-executed in the electronic device based on the current setting state information, and obtain the order information based on the remaining operation information except the identified operation information.

[0012] The processor is configured to control the electronic device to: obtain current setting status information of the electronic device; based on the current setting status information, obtain additional operation information required to execute a function corresponding to the obtained text in addition to the plurality of operation information; and obtain sequence information based on the plurality of operation information and the additional operation information.

[0013] The processor is configured to control the electronic device to: based on the sequence information, provide a first guidance UI based on information related to a touch input required to execute first operation information among the plurality of operation information; based on the context of the electronic device changed based on a touch input corresponding to the first guidance UI, control the display to provide a second guidance UI based on information related to a touch input required to execute second operation information among the plurality of operation information based on the changed context and the sequence information.

[0014] The information related to the touch input may include at least one of a position of the touch input or a touch type.

[0015] The MRC model may include a question generation (QG) model configured to obtain a question and a question answer (QA) model configured to obtain an answer corresponding to the obtained question by performing natural language processing (NLP) based on a plurality of sentences included in a manual of the electronic device.

[0016] The electronic device may further include a communicator including a communication circuit, and the processor is configured to control the communicator to send the input to a server, and based on text received from the server corresponding to the input, obtain the plurality of operation information and the sequence information based on the received text, and the text may be obtained based on the manual of the electronic device and the MRC model.

[0017] The processor is configured to control the electronic device to provide at least one of visual feedback or sound feedback based on that at least one of a position or a touch type of the touch input does not correspond to recognition of the guidance UI.

[0018] The processor is configured to control the electronic device to: provide a UI for selecting any one of a manual mode or an automatic mode; and based on selecting the automatic mode, sequentially execute a plurality of operations by executing a touch input function corresponding to the guidance UI.

[0019] The processor is configured to control the electronic device to execute a touch input function corresponding to the guidance UI and provide visual feedback indicating execution of the touch input function.

[0020] According to an exemplary embodiment, a method of controlling an electronic device includes: obtaining text corresponding to an input based on a machine reading comprehension (MRC) model; obtaining a plurality of operation information by dividing the text into operation units of the electronic device; obtaining order information of the plurality of operation information based on the plurality of operation information and the obtained text; and providing a guidance user interface (UI) to sequentially perform the plurality of operations based on the order information.

[0021] The operation unit may be based on a context that requires a touch input.

[0022] The method may include obtaining current setting state information on the electronic device, and obtaining the order information may include identifying at least one operation information among the plurality of operation information pre-executed in the electronic device based on the current setting state information; and obtaining the order information based on the remaining operation information except the identified operation information.

[0023] The method may include obtaining current setting state information of the electronic device; and in addition to the plurality of operation information, obtaining additional operation information required to perform a function corresponding to the obtained text based on the current setting state information, and obtaining the order information may include obtaining the order information based on the plurality of operation information and the additional operation information.

[0024] Providing the guidance user interface may include, based on the order information, providing a first guidance UI based on information related to a touch input required to perform a first operation information among the plurality of operation information; and controlling the display to provide a second guidance UI based on information related to a touch input required to perform a second operation information among the plurality of operation information based on a changed context of the electronic device based on a touch input corresponding to the first guidance UI and the order information.

[0025] Information related to the touch input may include at least one of a position of the touch input or a touch type.

[0026] The MRC model may include a question generation (QG) model configured to obtain a question and a question answering (QA) model configured to obtain an answer corresponding to the obtained question by performing natural language processing (NLP) based on a plurality of sentences included in a manual of the electronic device.

[0027] Obtaining the text may include sending the input to a server, and the text may be obtained based on a manual of the electronic device and the MRC model.

[0028] The method may include providing at least one of visual feedback or sound feedback based on at least one of a position of the touch input or a touch type not corresponding to an identification of the guidance UI.

[0029]

Advantageous Effects

[0030] According to various exemplary embodiments, an intuitive guidance UI for a control method of an electronic device corresponding to a user input can be provided to a user, and a usage method considering a current setting state of the electronic device can be provided to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In conjunction with the accompanying drawings, from the following detailed description, the above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent, wherein:

[0032] Figure 1 is a block diagram showing an example configuration of an electronic device according to an embodiment;

[0033] Figure 2 is a block diagram showing an example configuration of the Figure 1 exemplary electronic device shown;

[0034] Figure 3 is a diagram showing an exemplary electronic device and a user input according to an embodiment;

[0035] Figure 4 is a diagram showing exemplary operation information and sequence information according to an embodiment;

[0036] Figure 5 is a diagram showing exemplary current setting information according to an embodiment;

[0037] Figure 6 is a diagram showing exemplary current setting state information according to another embodiment;

[0038] Figure 7 is a diagram showing exemplary additional operation information according to another embodiment;

[0039] Figure 8 is a diagram showing an exemplary guidance UI according to an embodiment;

[0040] Figure 9 is a diagram showing exemplary feedback according to an embodiment;

[0041] Figure 10 is a diagram illustrating an exemplary electronic device communicating with a server according to an embodiment; and

[0042] Figure 11 is a flowchart showing an exemplary method for controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0043] The present disclosure will be described in more detail below with reference to the accompanying drawings.

[0044] The terms used in this disclosure and the claims are general terms identified in consideration of the functions of various exemplary embodiments of this disclosure. However, these terms may vary according to the intention of those skilled in the relevant art, technical interpretations, the emergence of new technologies, etc. Unless there is a specific definition of a term, the term can be understood based on the overall content and the technical understanding of those skilled in the relevant art.

[0045] In this disclosure, expressions such as “have”, “may have”, “include”, or “may include” indicate the existence of corresponding features (e.g., components such as numbers, functions, operations, or parts), and do not exclude the existence of additional features.

[0046] The expression “at least one of A or / and B” should be understood to represent any one of “A” or “B” or “A and B”.

[0047] As used herein, terms such as “first”, “second”, etc. may represent various components regardless of order and / or importance, and may be used to distinguish one component from another, and do not limit the components.

[0048] In addition, the description in this disclosure that one element (e.g., the first element) is “(operatively or communicatively) coupled / coupled to” or “connected to” another element (e.g., the second element) should be interpreted to include the case where one element is directly coupled to another element, and the case where one element is coupled to another element through yet another intermediate element (e.g., the third element).

[0049] Unless otherwise stated, singular expressions include plural expressions. It should be understood that terms such as “include” or “consist of” are used herein to indicate the existence of features, quantities, steps, operations, elements, components, or combinations thereof, and do not exclude the existence or possibility of adding one or more other features, quantities, steps, operations, elements, components, or combinations thereof.

[0050] Terms such as “module”, “unit”, “part”, etc. may be used to refer to an element that performs at least one function or operation, and such an element may be implemented as hardware or software, or a combination of hardware and software. In addition, unless each of multiple “modules”, “units”, “parts”, etc. needs to be implemented in separate hardware, the components may be integrated in at least one module or chip and implemented in at least one processor (not shown).

[0051] In this disclosure, the term user may refer to a person who uses an electronic device or a device that uses an electronic device (e.g., an artificial intelligence (AI) electronic device).

[0052] Hereinafter, various exemplary embodiments of this disclosure will be described in more detail with reference to the accompanying drawings.

[0053] Figure 1 It is a block diagram showing an example configuration of an electronic device according to an embodiment.

[0054] According to various embodiments, the electronic device 100 may be implemented as a device having a display function, such as, but not limited to, a television, a smartphone, a tablet computer, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop computer, a smart watch, a head-mounted display (HMD), a near-eye display (NED), etc. The electronic device 100 may be implemented to include various types of displays 110 to provide a display function, such as, but not limited to, a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a micro LED, a quantum dot (QD) display panel, etc.

[0055] Various embodiments may be implemented by the electronic device 100 that does not include a display function. For example, the electronic device 100 may be implemented as various types of source devices that provide content to a display device, such as, but not limited to, a Blu-ray player, a digital versatile disc (DVD) player, a streaming content output device, a set-top box, etc. As another example, various types of household appliances, such as, but not limited to, a speaker, a refrigerator, a washing machine, an air conditioner, an air purifier, various Internet of Things (IoT), etc., may implement various embodiments. Hereinafter, for convenience of description, the electronic device 100 will be assumed to be a user terminal device including a display 110.

[0056] Referring to Figure 1 , the display device 100 includes a display 110 and a processor (e.g., including a processing circuit) 120.

[0057] The display 110 may provide various content screens that can be provided by the electronic device 100. The content screen may include, for example, but not limited to, an application execution screen, various contents, such as images, videos, texts, music, a graphical user interface (GUI) screen, etc.

[0058] The display 110 may display text and a guide UI corresponding to a user input based on a manual (or user guide) of the electronic device 100, which will be described in more detail below.

[0059] The processor 120 may include various processing circuits and may be implemented, for example but not limited to, with a digital signal processor (DSP), a microprocessor, and a timing controller (TCON) that process digital image signals, but is not limited thereto. The processor 120 may include, for example but not limited to, one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), a communication processor (CP), an advanced reduced instruction set (RISC) computer (ARM) processor, etc., or may be defined by corresponding terms. The processor 120 may be implemented in a system-on-chip (SoC) type or a large-scale integration (LSI) type with built-in processing algorithms, or in a field-programmable gate array (FPGA) type. The processor 120 may include various processing circuits and perform various functions by executing computer-executable instructions stored in the memory.

[0060] The processor 120 may obtain text corresponding to a user input based on a machine reading comprehension (MRC) model. For example, the MRC model may be a machine reading model that reads and interprets documents and manuals related to the electronic device 100 based on artificial intelligence (AI) algorithms. For example, the MRC model may use natural language processing (NLP) algorithms trained based on various types of deep learning to analyze documents, such as but not limited to, recurrent neural networks (RNNs) and convolutional neural networks (CNNs), etc. Documents, manuals, etc. related to the electronic device 100 are not limited to user manuals distributed by the manufacturer of the electronic device 100 and may include various types of data, and the various types of data include methods of operating or using multiple functions included in the electronic device 100.

[0061] The MRC model according to an embodiment may include a machine reading model applicable to natural language evaluation metrics, such as but not limited to General Language Understanding Evaluation (GLUE) or Stanford Question Answering Dataset (SCADY), etc. According to an embodiment, the MRC model may be a model that reads and interprets a document and finds text (e.g., an answer corresponding to a question) corresponding to a user input (e.g., a question) based on the document. In addition to RNNs and CNNs, the MRC model according to an embodiment may be, for example, a model trained using a transformer-based neural network.

[0062] The MRC model according to an embodiment may include a question generation (QG) model and a question answering (QA) model for obtaining an answer to the obtained question by performing NLP based on multiple sentences included in the manual of the electronic device 100.

[0063] For example, the QG mathematical model included in the MRC mathematical model can obtain the keywords (or topics) of the corresponding sentences based on the sentences included in the manual. The MRC mathematical model can obtain the best form of the questions among the multiple questions generated in the corresponding sentences. The QA model can identify the corresponding sentence as the answer corresponding to the obtained question.

[0064] The processor 120 according to an embodiment may perform preprocessing on a plurality of sentences included in the manual of the electronic device 100 to obtain a plurality of questions for each keyword and the answers corresponding to each of the plurality of questions. A more detailed description will be given below with reference to Figure 3 it.

[0065] Figure 3 is a diagram illustrating an example electronic device and user input according to an embodiment.

[0066] Refer to Figure 3 , the electronic device 100 according to an example embodiment may receive a user input. For example, the electronic device 100 may receive the voice uttered by the user via a microphone (not shown) provided in the electronic device 100. As another example, the electronic device 100 may receive a user input via various types of interfaces (e.g., a touchpad) provided in the electronic device 100. In the following description, for convenience of description, it is assumed that the voice uttered by the user received via the microphone is used as the user input.

[0067] As Figure 3 shown, if the user input "Turn on Wi-Fi and Bluetooth" is received as the user input, the processor 120 may obtain the text corresponding to the user input. According to an example, the processor 120 may identify the question based on the NLP algorithm regarding the user input. For example, the processor 120 may identify the keywords included in the user input and select one of the multiple questions for each keyword obtained from the QG model based on the identified keywords.

[0068] Refer to Figure 3 , the processor 120 may perform syntactic analysis on the user input. The processor 120 may then perform semantic analysis on the parsed data and identify the keywords included in the user input (e.g., Wi-Fi, Bluetooth, and power on). The processor 120 may then obtain the questions corresponding to the identified keywords among the multiple questions. For example, the processor 120 may obtain "How to turn on Wi-Fi?" and "How to turn on Bluetooth?"

[0069] The processor 120 according to an embodiment may obtain an answer corresponding to the obtained question. For example, the processor 120 may generate a question based on the MRC model from a sentence included in the manual of the electronic device 100, and obtain the sentence as an answer to the generated question. For example, the processor 120 may obtain a sentence such as "How to turn on Wi-Fi?" as an answer to the question. This embodiment is merely an example. In addition to multiple questions obtained by performing preprocessing on the manual and answers to each of the multiple questions, the processor 120 may also perform NLP on the user input to identify keywords and obtain an answer corresponding to the identified keywords. For example, if "Let me know how to install Application A" is received, the processor 120 may identify Application A and the installation method based on the NLP algorithm. The processor 120 may perform a search for a method of installing Application A to obtain an answer from the server. Meanwhile, for ease of description, the answer to the question corresponding to the user input is collectively referred to as the text corresponding to the user input.

[0070] The processor 120 according to an embodiment may output a question corresponding to the user input based on the MRC model. In one example, when "How to turn on Wi-Fi?" and "How to turn on Bluetooth?" are obtained according to the user input, the processor 120 may output a user interface (UI) for whether the user input is properly recognized, such as "Do you want to know how to turn on Wi-Fi and Bluetooth?" When a confirmation input from the user to the corresponding UI is received, the text (or answer) corresponding to the question may be obtained.

[0071] Return Figure 1 , the processor 120 according to an embodiment may obtain the text corresponding to the user input, perform NLP on the obtained text, and classify the text into the operation unit of the electronic device 100 to obtain multiple operation information. The processor 120 may then obtain the order information of the multiple operation information based on the multiple operation information and the obtained text. A more detailed description will be made below with reference to Figure 4 for a more detailed description thereof.

[0072] Figure 4 is a diagram showing example operation information and order information according to an embodiment.

[0073] Refer to Figure 4, when receiving a user input of "Please turn on Wi-Fi and Bluetooth", the processor 120 according to the embodiment can identify questions "How to connect to Wi-Fi?" and "How to connect to Bluetooth?" among multiple questions generated based on multiple sentences included in the manual of the electronic device 100. The processor 120 can then obtain a sentence corresponding to the identified question as the answer (or text) to the question. According to an example, the processor 120 can obtain the sentence "Press the connection button on the settings screen → Slide the Wi-Fi toggle button to the left" included in the manual of the electronic device 100. The processor 120 can obtain "Press the connection button on the settings screen → Slide the Bluetooth toggle button to the left" as the text corresponding to the user input.

[0074] The processor 120 can then perform NLP on the obtained text to distinguish it with the operation units of the electronic device 100. The operation units of the electronic device 100 can be units based on the context that requires user touch input. The operation units of the electronic device 100 can be based on whether user touch input is required, or the operation units of the electronic device 100 can be units based on whether the settings of the electronic device 100 change, whether the screen provided by the display 110 changes, or whether the state of the electronic device 100 changes according to the user's touch input.

[0075] For example, if "Press the connection button on the settings screen → Slide the Bluetooth toggle button to the left" is received as the text corresponding to the user input, the processor 120 can divide the text with the operation units of the electronic device 100 and obtain multiple operation information such as the settings button, connection button, Wi-Fi toggle button, etc.

[0076] The processor 120 can then obtain sequence information based on the multiple operation information and the corresponding text acquisition order. For example, the processor 120 can set an order for each of the multiple operation information based on the corresponding text. For example, the processor 120 can arrange the settings button → connection button → Wi-Fi toggle button in order based on the order in which multiple operations are included in the text. The processor 120 can obtain the sequence information of the multiple operations. The processor 120 according to the embodiment can assign Fn1, Fn2, Fn3, Fn4, etc. to each of the multiple operations based on the sequence information.

[0077] The processor 120 according to the embodiment can provide a guidance user interface (UI) 10 for sequentially performing multiple operations according to the sequence information.

[0078] Reference Figure 4, the processor 120 may obtain the setting button, the connection button, the Wi-Fi toggle button, and the Bluetooth toggle button as multiple operations based on the text corresponding to the user input "Please turn on Wi-Fi and Bluetooth". The processor 120 may then assign an order to each of the multiple operations to obtain order information. For example, the processor 120 may obtain the ordered order information in the order of the setting button → the connection button → the Wi-Fi toggle button → the Bluetooth toggle button.

[0079] As Figure 4 shown, the processor 120 may provide the first guidance UI through the display 110 based on the information related to the touch input required to perform the first operation information Fn1 in the multiple operation information according to the order information. For example, when the first operation information Fn1 is the setting button based on the order information, the processor 120 may provide the guidance UI 10 to trigger the touch input to the setting button. For example, the processor 120 may display the guidance UI 10 at the position of the setting button on the screen provided through the display 110 in a predetermined size and color.

[0080] When changing the context of the electronic device 100 based on the touch input corresponding to the first guidance UI, the processor 120 according to an embodiment may control the display 110 to provide the second guidance UI based on the information related to the touch input required to perform the second operation information Fn2 in the multiple operation information based on the changed context and the order information. The change in the context of the electronic device 100 may include at least one of receiving a touch input, changing the screen, or changing the settings of the electronic device 100.

[0081] Referring to Figure 4 , when the setting screen is displayed upon receiving the touch input to the first guidance UI, the processor 120 may provide the second guidance UI for guiding the touch input to the position of the connection button on the screen based on the order information and according to the second operation information Fn2. The information related to the touch input is not limited to the position of the touch input, and may include the touch type. For example, the processor 120 may display the guidance UI 10 to trigger a click touch input of the user at a specific position on the screen, and may provide the guidance UI 10 corresponding to various touch types, such as drag touch, tap touch, pinch touch, expand touch, etc. Its detailed description will be provided in Figure 8 it.

[0082] In addition to the guidance UI 10, the processor 120 according to an embodiment may provide the script 20. Referring to Figure 4, the processor 120 may provide a script 20 for each operation information based on the acquired text and its description on the screen area. For example, the processor 120 may provide a script 20 of "Press the settings button" with the guiding UI 10 to trigger the touch of the settings button. As another example, the processor 120 may provide a script 20 of "Press the connection button" corresponding to the second operation information Fn2 in the acquired text "Press the connection button on the settings screen → Slide the Wi-Fi toggle button to the left", and a second guiding UI for triggering the touch input required to execute the second operation information Fn2. As another example, the processor 120 may provide a script 20 of "Slide the Wi-Fi toggle button to the left" corresponding to the third operation information Fn3, and a third guiding UI for triggering the touch input required to execute the third operation information Fn3. As yet another example, the processor 120 may provide a script 20 of "Slide the Bluetooth toggle button to the left" corresponding to the fourth operation information Fn4, and a fourth guiding UI for triggering the touch input required to execute the fourth operation information Fn4.

[0083] The processor 120 according to an embodiment may output the script 20 provided together with the guiding UI on the screen as a sound signal. For example, the processor 120 may perform text-to-speech conversion (TTS) on the script 20 to obtain a sound signal corresponding to the script 20, and output the sound signal together with the guiding UI 10.

[0084] Return Figure 1 , the processor 120 according to an embodiment may acquire the current setting state information of the electronic device 100. The current setting state information may refer to, for example, the setting values of each of the multiple functions included in the electronic device 100 (e.g., whether the function is on / off), information about the applications installed in the electronic device 100 (e.g., application installation status, application version information, location of the icon for executing the application), the screen displayed through the display 110, etc. The processor 120 according to an embodiment may acquire the current setting state information at the system level, or may acquire the current setting state information through an application programming interface (API) call.

[0085] The processor 120 according to an embodiment may acquire order information by excluding some operation information from the multiple operation information obtained from the text based on the current setting state information. As another example, the processor 120 may acquire order information by adding other operation information in addition to the multiple operation information based on the current setting state information. This will be described in more detail below with reference to Figure 5 、 6 and 7.

[0086] Figure 5 is a diagram illustrating an example of current setting information according to an embodiment.

[0087] Reference Figure 5 According to an embodiment, the processor 120 may identify at least one pre-executed operation information among a plurality of operation information obtained by dividing text through an operation unit of the electronic device 100 based on the current setting status information.

[0088] For example, texts corresponding to user inputs (turning on Wi-Fi and Bluetooth), “Press the connection button on the setting screen → Slide the Wi-Fi toggle button to the left” and “Press the connection button on the setting screen → Slide the Bluetooth toggle button to the left” may be obtained.

[0089] The processor 120 may divide the obtained text into operation units to obtain the setting button, the connection button, the Wi-Fi toggle button, and the Bluetooth toggle button as a plurality of operation information. According to an embodiment, the processor 120 may identify the previously executed operation information among the plurality of operation information based on the current setting status information of the electronic device 100. As an example, if the screen provided through the display 110 is a setting screen, the processor 120 may identify that the operation of touching the setting button has been previously executed. In this example, the processor 120 may exclude the setting button from the plurality of operation information and then obtain order information based on the remaining operation information. As another example, when it is identified based on the current setting status information that the Wi-Fi function of the electronic device 100 is turned on, the processor 120 may exclude the Wi-Fi toggle button from the plurality of operation information and obtain order information based on the remaining operation information.

[0090] Even if the user input is in the same way “turn on Wi-Fi and Bluetooth”, reference Figure 5 the processor 120 may not provide a UI for guiding a touch input of the Wi-Fi toggle button for the operation information determined to be pre-executed, different from Figure 4 the example shown, and may provide a UI for guiding a touch input of the Bluetooth toggle button.

[0091] Figure 6 is a diagram showing the current setting status information according to another embodiment.

[0092] According to various embodiments, the processor 120 may identify a problem based on a user input and obtain text (e.g., an answer) corresponding to the problem. Reference Figure 6 If the user input is “Let me know how to use navigation”, the processor 120 may obtain, based on the MRC model, one question and an answer determined to be closest to the user among a plurality of questions with navigation as a keyword. Here, the keyword does not necessarily mean that the word should be included in the question or the answer. For example, the processor 120 may obtain one question corresponding to the user input and the answer to the question among a plurality of questions for each topic.

[0093] Reference Figure 6 , according to an embodiment, the processor 120 can obtain the text "Press the map application → Press the microphone button and let me know the destination" corresponding to "Let me know how to use the navigation" as a user input. Then, the processor 120 can obtain the sequence information for sequentially causing touch inputs to the map application Fn1 and the microphone button Fn2, and provide the guidance UI 10 and the script 20 based on the sequence information.

[0094] The processor 120 according to another embodiment can obtain the sequence information including additional operation information based on the current setting state information of the electronic device 100. The additional operation information can mean another operation information in addition to the multiple operation information obtained by dividing the text by the operation unit. This will be described in more detail below with reference to Figure 7 for a more detailed description.

[0095] Figure 7 is a diagram showing exemplary additional operation information according to another embodiment.

[0096] Reference Figure 7 , in addition to the multiple operation information based on the current setting state information, the processor 120 can obtain the additional operation information required to execute the function corresponding to the text. For example, the processor 120 can obtain the sequence information for sequentially causing touch inputs to the map application Fn1 and the microphone button Fn2. According to an example embodiment, the processor 120 can identify whether the map application is installed based on the current setting state information in order to execute "Please press the map application → Press the microphone button and let me know the destination", which is the text corresponding to the user input. As another example, the processor 120 can identify whether the electronic device 100 includes a microphone.

[0097] According to an embodiment, if it is identified based on the current setting state information that the map application is not installed on the electronic device 100, the processor 120 can obtain the operation information for installing the map application as additional operation information in addition to the multiple operation information (for example, the map application Fn1 and the microphone button Fn2). The processor 120 can then obtain the sequence information based on the multiple operation information and the additional operation information.

[0098] Even if the user input is "Let me know how to use the navigation" in the same manner, referring to Figure 7 , different from the example shown in Figure 6 , the processor 120 can additionally provide the guidance UI 10 for causing a touch input to press the store button and the guidance UI 10 for causing a touch input to install the map application.

[0099] As another example, in addition to the multiple operation information based on the current setting state information, the processor 120 according to an embodiment may acquire additional operation information required to perform a function corresponding to the text. In one example, the processor 120 may identify the required additional actions in the current setting state information such that the electronic device 100 operates corresponding to the user input. For example, if the user input is "Let me know how to use the navigation", the processor 120 may acquire the map application Fn1 and the microphone button Fn2 as operation information. The processor 120 may then identify whether the electronic device 100 is in a suitable state for executing the map application Fn1 based on the current setting state information of the electronic device 100. For example, if the global positioning system (GPS) function of the electronic device 100 is turned off, the processor 120 may acquire the operation Fn 1-1 (not shown) for turning on the GPS based on the MRC model.

[0100] The processor 120 according to an embodiment may identify sub-operations of each operation. As described above, the sub-operations corresponding to the operation Fn1 for executing the map application may include the operation Fn 1-1 for turning on the GPS function and the operation Fn 1-2 (not shown) for identifying the current location of the electronic device 100 through the map application, etc. The processor 120 may then identify at least one sub-operation among the multiple sub-operations as additional operation information based on the current setting state information of the electronic device 100. The processor 120 may then acquire sequence information based on the multiple operation information and the additional operation information. For example, if the GPS function is turned off, in addition to the operation Fn 1 for executing the map application and the operation Fn2 for uttering the destination based on the text corresponding to the user input "Let me know how to use the navigation", the processor 120 may acquire the operation Fn 1-1 for turning on the GPS function as additional operation information. The processor 120 may provide a guidance UI corresponding to each operation in the order of Fn1 → Fn 1-1 → Fn2.

[0101] As another example (not shown), the processor 120 according to the exemplary embodiment may identify additional operations that need to be performed before performing one of the multiple operations. For example, if the user input text is "Let me know how to use the dictionary function", the processor 120 may obtain pressing the search term Fn1, pressing the dictionary Fn 2 in the option list, and checking the search result Fn3 as the multiple operation information. The processor 120 may identify "Install dictionary application Fn 2-1" as an additional operation that needs to be performed in advance to perform pressing the dictionary Fn 2 in the option list. Then the order information may be adjusted such that the identified additional operation is placed in advance. For example, the guiding UI corresponding to each operation may be provided in the following order: Press the search term Fn1 → Install dictionary application Fn 2-1 → Press the dictionary Fn 2 in the option list → Check the search result Fn3 based on the operation information and the additional operation. The processor 120 according to the embodiment may not include the previous operation in the order information based on the current setting state information of the electronic device 100. For example, if it is identified based on the current setting state information that the dictionary application has been installed, the processor 120 may generate order information such that installing the dictionary application Fn 2-1 is not added before pressing the dictionary Fn 2 in the option list.

[0102] As yet another embodiment (not shown), the processor 120 according to the embodiment may perform some operations by itself in the background state without providing the guiding UI. For example, when the user input is "Let me know how to use the dictionary", the processor 120 may obtain pressing the search term Fn1 → Install dictionary application Fn2-1 → Press the dictionary Fn2 in the option list → Check the search result Fn3 as the order information based on the operation information and the additional operation based on the corresponding text. In this example, the processor 120 may install the dictionary application by itself in the background state by the processor 120 instead of providing the guiding UI corresponding to installing the dictionary application Fn 2-1, which is the previous operation. In this example, the guiding provided to the user may be in the order of Fn1 → Fn2 → Fn3.

[0103] Figure 8 FIG. is a diagram showing an exemplary guiding UI according to an embodiment.

[0104] The guiding UI according to the embodiment may guide the user to the position of the touch input and guide the touch type. For example, the processor 120 may identify the touch type required to perform any one of the multiple operation information obtained by performing NLP on the acquired text.

[0105] For example, as Figure 8As shown, the user input can be "How to magnify a photo?" and the processor 120 can obtain the text corresponding to the user input "Press the screen with two fingers and spread them apart to adjust the screen magnification". In this example, the processor 120 can provide the guidance UI 10 for guiding the extended touch input. As Figure 8 shown, the processor 120 can provide the guidance UI 10 including an animation effect as if the first object and the second object included in the guidance UI 10 are separated from each other.

[0106] As another example, when the user input is "How to magnify the screen?", the processor 120 can obtain the text corresponding to the user input "You can magnify the screen by quickly pressing the target screen area three times". In this example, the processor 120 can provide the guidance UI 10 including an animation effect as if three tap touch inputs are performed in one area of the screen. This is just an example and is not limited thereto.

[0107] The processor 120 according to an embodiment can provide a UI for selecting one of a manual mode or an automatic mode. When a touch input of the user is received according to the guidance UI 10, the manual mode can provide the guidance UI 10 of the operation information corresponding to the next sequence based on the sequence information. Even if no touch input of the user corresponding to the guidance UI 10 is received, the automatic mode can still be a mode for changing the context of the electronic device 100 according to the obtained text.

[0108] For example, when no touch input of the user is received in the automatic mode, the processor 120 can execute the touch input function corresponding to the guidance UI 10 and provide a visual feedback indicating that the touch input function is executed. For example, the processor 120 can change the context of the electronic device 100 and provide the guidance UI10 corresponding to the next operation information based on the sequence information, while providing an animation effect as if a touch input to the guidance UI 10 is received.

[0109] The embodiments are not necessarily limited thereto. For example, the processor 120 can sequentially change the context of the electronic device 100 based on the sequence information in the background without changing the screen provided through the display 110. For example, if the text corresponding to the user input is "Drag along the + direction on the adjustment bar to illuminate the screen", a UI (e.g., manual mode) for guiding the drag touch input can be provided, and the processor 120 can automatically adjust the brightness of the screen by itself in the background (e.g., automatic mode).

[0110] Return Figure 1, if at least one of the position or touch type of the user's touch input is recognized as not corresponding to the guidance UI 10, the processor 120 according to the embodiment may provide at least one of visual feedback or sound feedback. This will be described in more detail below with reference to Figure 9 for a more detailed description.

[0111] Figure 9 is a diagram showing an example feedback according to an embodiment.

[0112] When it is recognized that the position 30 of the user's touch input corresponds to the guidance UI 10, the processor 120 according to the embodiment may sequentially provide the guidance UI 10 corresponding to the next operation information based on the sequence information.

[0113] As another example, if it is recognized that the position 30 of the user's touch input does not correspond to the guidance UI 10, the processor 120 may provide feedback to cause the touch input to be performed again. Refer to Figure 9 , although according to the embodiment, the guidance UI 10 introduces a tap touch input to the display, when the position 30 of the user's touch input corresponds to sound and vibration, the processor 120 may provide at least one of visual feedback or sound feedback. The visual feedback may be feedback that changes at least one of the size or color of the guidance UI 10. However, this is only an example, and various types of visual feedback may be provided.

[0114] The sound feedback according to the example embodiment may include sound effects, beeps, etc. As another example, the sound feedback may be feedback output as a sound signal based on the script 20 corresponding to the guidance UI 10. For example, refer to Figure 9 , the processor 120 may provide sound feedback to output "You pressed the wrong button. Please press the display button again" as a sound signal.

[0115] Return Figure 1 , the electronic device 100 according to the example embodiment may further include a communicator (for example, including a communication circuit) (not shown), and the processor 120 may control the communicator to send the user input to the server. If text corresponding to the user input is received from the server, the processor 120 may obtain a plurality of operation information and sequence information based on the text. The text may be obtained based on the manual of the electronic device 100 and the MRC model. This will be described in detail below with reference to Figure 10 for a detailed description.

[0116] Figure 10 is a diagram illustrating an example electronic device communicating with a server according to an embodiment.

[0117] Refer to Figure 10, the electronic device 100 can communicate with the server 200. The processor 120 can send user input to the server 200 and receive text corresponding to the user input from the server 200. The server 200 can include a plurality of question-and-answer data sets according to an MRC model based on a document, manual, or instruction manual of the electronic device 100, but this is only an example, and the question and the answer corresponding to the question do not necessarily appear in pairs. For example, the server 200 can obtain keywords, topics, etc. by performing NLP on the user input, and obtain text (or answer) corresponding to the user input from the search results based on the obtained keywords, topics, etc.

[0118] Figure 2 is a block diagram showing an example configuration of an example electronic device according to an embodiment. Figure 1 of an example electronic device.

[0119] Referring Figure 2 , the electronic device 100 according to an embodiment may include a display 110, a processor (e.g., including processing circuitry) 120, a communicator (e.g., including communication circuitry) 130, a memory 140, a user interface (e.g., including user interface circuitry) 150, an input and output interface (e.g., including input and output circuitry) 160, a microphone 170, and a speaker 180. Configurations overlapping with Figure 1 will not be further described. Figure 2 of

[0120] The processor 120 may include various processing circuits and use various programs stored in the memory 140 to control the overall operation of the electronic device 100.

[0121] The processor 120 includes a random access memory (RAM) 121, a read only memory (ROM) 122, a main CPU 123, first to Nth interfaces 124-1 to 124-N, and a bus 135.

[0122] Here, the RAM 121, ROM 122, main CPU 123, first to Nth interfaces 124-1 to 124-N, etc. may be interconnected via a bus 125.

[0123] The ROM 122 stores a set of commands for booting the system, etc. When a power-on command is input and powered on, the main CPU 133 copies the O / S stored in the memory 140 to the RAM 121 according to the instructions stored in the ROM 132, executes the O / S to boot the system. When the boot is completed, the main CPU may copy various programs stored in the memory to the RAM 121, execute the programs copied to the RAM 121, and perform various operations.

[0124] The main CPU 123 accesses the memory 140 and performs booting using the operating system (OS) stored in the memory 140. In addition, the CPU 123 performs various operations using various programs and the like stored in the memory 140.

[0125] The first to nth interfaces 124-1 to 124-n are connected to the various elements described above. One of the interfaces can be a network interface connected to an external device through a network.

[0126] The processor 120 can perform a graphics processing function (video processing function). For example, the processor 120 can generate a screen including various objects such as icons, images, text, etc. using a calculator (not shown) and a renderer (not shown). Here, the calculator (not shown) can calculate the attribute values to be displayed by each object, such as coordinate values, shapes, sizes, and colors, based on the received control command according to the layout of the screen. The renderer (not shown) can generate a display screen of various layouts including the objects based on the attribute values calculated by the calculator (not shown). The processor 120 can perform various image processing on video data, such as decoding, scaling, noise filtering, frame rate conversion, resolution conversion, etc.

[0127] The processor 120 can perform the processing of audio data. For example, the processor 120 can perform various image processing on audio data, such as but not limited to decoding, amplification, noise filtering, etc.

[0128] The communicator 130 can include various communication circuits and is configured to communicate with various types of other external devices according to various communication types. The communication interface 130 includes various modules, and each module includes various communication circuits, such as but not limited to a Wi-Fi module 131, a Bluetooth module 132, an infrared communication module 133, a wireless communication module 134, a wired communication module 135, etc. Each communication module can be implemented in at least one hardware chip format.

[0129] The processor 120 can communicate with various external devices using the communicator 130. The external devices can include, for example, but not limited to display devices such as TVs, image processing devices such as set-top boxes, external servers, control devices such as remote controls, audio output devices such as Bluetooth speakers, lighting devices, smart vacuum cleaners, household appliances such as smart refrigerators, servers such as Internet of Things (IoT) home managers, etc.

[0130] The Wi-Fi module 131 and the Bluetooth module 132 perform communication using the Wi-Fi method and the Bluetooth method, respectively. When using the Wi-Fi module 131 or the Bluetooth module 132, various connection information such as a service set identifier (SSID) and a session key can be sent and received first, and communication information can be sent after the communication connection.

[0131] The infrared communication module 133 performs communication according to the Infrared Data Association (IrDA) technology, which uses infrared rays between visible light and millimeter waves to wirelessly transmit data to a local area.

[0132] In addition to the Wi-Fi module 131 and the Bluetooth module 132 described above, the wireless communication module 134 may refer to, for example, a module that performs communication according to various communication standards such as Zigbee, third generation (3G), Third Generation Partnership Project (3GPP), Long Term Evolution (LTE), LTE-Advanced (LTE-A), fourth generation (4G), fifth generation (5G), etc.

[0133] The communicator 130 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that performs communication using a paired cable, a coaxial cable, an optical fiber cable, etc.

[0134] According to an embodiment, the communicator 130 may use the same communication module (e.g., the Wi-Fi module) to communicate with external devices such as a remote controller and an external server.

[0135] According to another example, the communicator 130 may utilize different communication modules (e.g., the Wi-Fi module) to communicate with external devices such as a remote controller and an external server. For example, the communication interface 130 may use at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may use a Bluetooth (BT) module to communicate with an external device such as a remote controller. However, this is merely an example, and when communicating with multiple external devices or external servers, the communicator 130 may use at least one communication module among various communication modules.

[0136] According to an example, the electronic device 100 may further include a tuner and a demodulator.

[0137] The tuner (not shown) may receive an RF broadcast signal from the RF broadcast signals received through an antenna by tuning a user-selected channel or all pre-stored channels.

[0138] The demodulator (not shown) may receive and demodulate the digital intermediate frequency (DIF) signal converted by the tuner and perform channel decoding, etc.

[0139] The memory 140 may be implemented as an internal memory included in the processor 120, such as a ROM (e.g., electrically erasable programmable read-only memory (EEPROM)), a RAM, etc., or a memory separated from the processor 130. In this case, the memory 140 may be implemented as a type of memory embedded in the electronic device 100 or a type of memory detachable from the electronic device 100 according to the data storage purpose. For example, data for driving the electronic device 100 may be stored in the memory embedded in the electronic device 100, and data for expanding the electronic device 100 may be stored in the memory detachable from the electronic device 100. The memory embedded in the electronic device 100 may be a volatile memory, such as but not limited to dynamic random access memory (DRAM), static random access memory (SRAM), synchronous dynamic random access memory (SDRAM); or a non-volatile memory (e.g., one-time programmable ROM (OTPROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard disk drive, solid state drive (SSD), etc. In the case where the memory is detachably mounted to the electronic device 100, the memory may be implemented as, for example but not limited to, a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), multimedia card (MMC), etc.), an external memory connectable to a USB port (e.g., USB memory), etc.

[0140] The user interface 150 may include various user interface circuits and be implemented as a device such as but not limited to buttons, a touchpad, a mouse, and a keyboard, or be implemented as a touch screen capable of performing the above display function and also manipulating the input function. The buttons may be various types of buttons formed in any area of the body of the electronic device 100 (e.g., the front surface portion, the side surface portion, and the rear surface portion), such as mechanical buttons, touch pads, rollers, etc.

[0141] The input and output interface 160 may include various input / output circuits and be at least one interface such as but not limited to a high-definition multimedia interface (HDMI), a mobile high-definition link (MHL), a universal serial bus (USB), a display port (DP), Thunderbolt, a video graphics array (VGA) port, an RGB port, a D-subminiature (D-SUB), a digital visual interface (DVI), etc.

[0142] The input and output interface 160 may input and output at least one of a voice signal and a video signal.

[0143] According to an embodiment, the input and output interface 160 may include ports for inputting and outputting only voice signals and ports for inputting and outputting only video signals as separate ports, or may be implemented as one port for inputting and outputting voice signals and video signals.

[0144] The electronic device 100 may be implemented as a device that does not include a display and sends video signals to a separate display device.

[0145] The electronic device 100 may receive a user voice signal from an external device including a microphone. In this example, the received user voice signal may be a digital voice signal, but according to an embodiment, the signal may be an analog voice signal. For example, the electronic device 100 may receive a user voice signal through a wireless communication method such as Bluetooth or Wi-Fi. The external device may be implemented as a remote control device or a smartphone.

[0146] The electronic device 100 may send the corresponding voice signal to an external server for voice recognition of the voice signal received from the external device.

[0147] The communication module for communicating with the external device and the external server may be implemented separately. For example, communication with the external device may be performed through a Bluetooth module, and communication with the external server may be performed through an Ethernet modem or a Wi-Fi module.

[0148] The electronic device 100 according to an embodiment may send the received digital voice signal to a voice recognition server. In this case, the voice recognition server may convert the user input into text information using speech-to-text conversion (STT). The voice recognition server may send the text information to another server or electronic device to perform a search corresponding to the text information and, in some cases, perform a direct search.

[0149] The electronic device 100 according to another embodiment may directly apply a speech-to-text conversion (STT) function to the user input to convert it into text information and send the converted text information to an external server.

[0150] The electronic device 100 may further include a microphone 170. The microphone 170 may refer to, for example, an element that receives a user voice or other sound and converts the received sound into audio data.

[0151] The microphone 170 may receive a user's voice in an active state. For example, the microphone 170 may be integrally formed as an integral unit on the upper side, front side direction, side direction, etc. of the electronic device 100. The microphone may include various configurations, such as a microphone for collecting the user's voice in an analog format, an amplifier circuit for amplifying the collected user's voice, an audio-digital (A / D) conversion circuit for sampling the amplified user's voice to convert it into a digital signal, a filtering circuit for removing noise elements from the converted digital signal, etc.

[0152] According to another embodiment, the electronic device 100 may receive a user's voice from an external device (not shown) having a microphone. For example, the external device may be implemented as a remote control device (remote controller) including but not limited to a microphone. When the remote control device (remote controller) receives an analog voice signal of the user through the microphone, the remote control device (remote controller) may convert the analog voice signal into a digital voice signal. The remote control device (remote controller) may transmit the converted digital voice signal to the electronic device 100 using at least one of infrared, Wi-Fi, or Bluetooth communication. When receiving the digital voice signal from the external device, the electronic device 100 may perform voice recognition based on the received digital voice signal and perform a control operation based on the voice recognition result. According to another embodiment, the remote control device (remote controller) may perform a voice recognition operation on the digital voice signal and transmit information corresponding to the voice recognition result to the electronic device 100. The information corresponding to the voice recognition result may be at least one of the voice recognition result itself or a control command corresponding to the voice recognition result.

[0153] According to still another embodiment, the external device may be implemented as a smart phone including a microphone. In this example, the smart phone may use a remote control application for performing a pre-installed remote control function to remotely control the electronic device 100.

[0154] If an analog voice signal of the user is received through the microphone, the smart phone may convert the analog voice signal into a digital voice signal. In this example, the smart phone may perform voice recognition on the digital voice signal using a voice recognition application. Here, the voice recognition application may be the same as or different from the above-mentioned remote control application. When performing voice recognition on the digital voice signal, the smart phone may remotely control the electronic device 100 using the remote control application based on the voice recognition result. According to another embodiment, the smart phone may transmit the converted digital voice signal to the electronic device 100 using at least one of infrared, Wi-Fi, or Bluetooth communication methods. In this case, when receiving the digital voice signal from the external device, the electronic device 100 may perform voice recognition based on the received digital voice signal and perform a control operation based on the voice recognition result.

[0155] The speaker 180 may be configured to output various notification sounds or voice messages and various audio data processed by the input and output interface 160.

[0156] Figure 11 is a flowchart showing an example method for controlling an electronic device according to an embodiment.

[0157] In operation S1110, the method for controlling an electronic device according to an embodiment obtains text corresponding to a user input based on a machine reading comprehension model (MRC).

[0158] By dividing the text by the operation unit of the electronic device, a plurality of operation information is obtained in operation S1120.

[0159] In operation S1130, order information of the plurality of operation information is obtained based on the plurality of operation information and the obtained text.

[0160] In operation S1140, a guidance UI for sequentially performing the plurality of operations based on the order information is provided.

[0161] Here, the operation unit may refer to, for example, a unit based on the context that requires a user touch input.

[0162] The method for controlling an electronic device according to an example embodiment may include an operation of obtaining current setting state information of the electronic device, and the operation of obtaining order information in operation S1130 may include identifying at least one operation information previously executed in the electronic device among the plurality of operation information based on the current setting state information; and obtaining order information based on the remaining operation information other than the identified operation information.

[0163] According to an example embodiment, the method may include obtaining current setting state information of the electronic device; and obtaining additional operation information required to execute a function corresponding to the obtained text in addition to the plurality of operation information based on the current setting state information, and the step of obtaining order information in operation S1130 may include obtaining order information based on the plurality of operation information and the additional operation information.

[0164] The operation of providing the guidance UI in operation S1140 may include: providing a first guidance UI based on information related to a touch input required to execute the first operation information among the plurality of operation information according to the order information; and when the context of the electronic device is changed based on the touch input corresponding to the first guidance UI, providing a second guidance UI based on information related to a touch input required to execute the second operation information among the plurality of operation information based on the changed context and the order information.

[0165] The information related to the touch input may include at least one of a position of the touch input or a touch type.

[0166] The MRC model according to an embodiment may include a QG model for obtaining a question by performing NLP based on a plurality of sentences included in a manual of an electronic device, and a QA model for obtaining an answer corresponding to the obtained question.

[0167] The operation of obtaining text in operation S1110 may include sending a user input to a server, and may obtain the text based on the manual of the electronic device and the MRC model.

[0168] The method according to an embodiment may include providing at least one of visual feedback or sound feedback based on identifying that at least one of a position or a touch type of a user's touch input does not correspond to a guidance UI.

[0169] The method according to various example embodiments as described above may be implemented in an application format that can be installed in an electronic device.

[0170] The method according to various example embodiments as described above may be implemented through software upgrade and / or hardware upgrade of an electronic device.

[0171] Various example embodiments as described above may be executed by an embedded server provided in the electronic device or an external server of the electronic device.

[0172] According to the present disclosure, various example embodiments as described above may be implemented with software including instructions stored in a machine-readable storage medium readable by a machine (e.g., a computer). According to various example embodiments, a device may call instructions from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., Electronic Device A). When a processor executes the instructions, the processor may use other components to directly or under the control of the processor execute functions corresponding to the instructions. The instructions may include code generated by a compiler or code executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

[0173] According to various example embodiments of the present disclosure, a method may be provided as a computer program product. The computer program product may be exchanged as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online through an application store (e.g., PlayStore TM )). In the case of online distribution, at least a part of the computer program product may be temporarily or at least temporarily stored in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0174] Each element (e.g., module or program) according to various exemplary embodiments may include a single entity or multiple entities, and some sub-elements of the above-mentioned sub-elements may be omitted, and these elements may further be included in various embodiments. Alternatively or additionally, some elements (e.g., module or program) may be integrated into one entity to perform the same or similar functions as performed by each corresponding element before integration. According to various embodiments, the operations performed by a module, program, or other element may be executed sequentially, in parallel, repeatedly, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0175] Although various embodiments have been illustrated and described with reference to the accompanying drawings, the present disclosure is not limited to the exemplary embodiments or the drawings, and those of ordinary skill in the art will understand that various changes may be made in form and detail without departing from the spirit and scope including, for example, the following claims and their equivalents.

Claims

1. An electronic device, comprising: A display; and A processor configured to control the electronic device to: Receive a user input, Obtain a response text related to the user input based on a machine reading comprehension (MRC) model, Obtain information about a plurality of operations for controlling the electronic device from the response text, wherein, The multiple operations are required to complete the function corresponding to the response text, Obtain information about the current setting state of the electronic device, Based on the information about the current setting state of the electronic device, identify a first pre-execution operation among the multiple operations that is to be executed before receiving the user input, and obtain sequence information based on the remaining operations, Based on the information about the current setting state of the electronic device and the first pre-execution operation, identify at least one second operation among the multiple operations that is required to complete the function corresponding to the response text, wherein the second operation is an operation among the multiple operations that is after the first pre-execution operation based on the sequence information, and Control the display to provide a guidance user interface (UI) for sequentially executing the remaining operations including at least the second operation based on the sequence information, wherein the guidance UI includes a UI for executing the second operation based on information related to a touch input required for executing the second operation, and the UI for executing the second operation in the electronic device is different from the UI for executing the first pre-execution operation.

2. The electronic device according to claim 1, wherein, The processor is further configured to divide the response text into units of operations according to the context that requires a touch input.

3. The electronic device according to claim 1, wherein, The processor is further configured to control the electronic device to: In addition to the first pre-execution operation and the second operation, obtain additional operations required to complete the function.

4. The electronic device according to claim 3, wherein, The processor is further configured to control the electronic device to: Based on the context of the electronic device changed according to a touch input corresponding to the guidance UI, control the display to provide a next guidance UI based on information related to a touch input required for executing the additional operation.

5. The electronic device according to claim 4, wherein, The information related to the touch input includes at least one of a position or a type of the touch input.

6. The electronic device according to claim 1, wherein, The MRC model includes a question generation (QG) model configured to obtain a question and a question answering (QA) model configured to obtain an answer corresponding to the obtained question by performing natural language processing (NLP) based on multiple sentences included in a manual of the electronic device.

7. The electronic device according to claim 1, further comprising: A communicator including a communication circuit; Wherein the processor is further configured to control the electronic device to: Control the communicator to send the user input to a server, and Based on text received from the server corresponding to the user input, obtain the multiple operations based on the received text, and Wherein the received text is based on a manual of the electronic device and the MRC model.

8. The electronic device according to claim 1, wherein the processor is further configured to control the electronic device to provide at least one of visual feedback or sound feedback based on identifying that at least one of a position or a type of an input does not correspond to the guidance UI.

9. The electronic device according to claim 1, wherein, The processor is further configured to control the electronic device to: provide a UI for selecting any one of a manual mode or an automatic mode, and sequentially perform the plurality of operations by executing a touch input function corresponding to the guided UI based on selecting the automatic mode.

10. The electronic device according to claim 9, wherein, The processor is further configured to control the electronic device to execute a touch input function corresponding to the guided UI and provide visual feedback indicating the execution of the touch input function.

11. A method of controlling an electronic device, the method comprising: receiving a user input; obtaining, based on a machine reading comprehension MRC model, a response text related to the user input; obtaining, from the response text, information about a plurality of operations for controlling the electronic device, wherein the plurality of operations are required to complete a function corresponding to the response text; obtaining information about a current setting state of the electronic device; identifying, based on the information about the current setting state of the electronic device, a first pre-execution operation among the plurality of operations that is to be executed before receiving the user input, and obtaining sequence information based on the remaining operations; identifying, based on the information about the current setting state of the electronic device and the first pre-execution operation, at least one second operation among the plurality of operations that is required to complete the function corresponding to the response text, wherein the second operation is an operation among the plurality of operations that is after the first pre-execution operation based on the sequence information; and providing a guided user interface UI for sequentially performing the remaining operations including at least the second operation based on the sequence information, wherein the guided UI includes a UI for performing the second operation based on information related to a touch input required to perform the second operation, and the UI for performing the second operation in the electronic device is different from the UI for performing the first pre-execution operation.

12. The method according to claim 11, further comprising dividing the response text into units of operations according to a context that requires a touch input.

13. The method according to claim 11, further comprising: obtaining additional operations required to complete the function in addition to the pre-execution first operation and the second operation.

Images