Electronic device and control method thereof

Abstract

An electronic device and a control method thereof are provided. The electronic device includes a display and a processor configured to control the electronic device to receive a user input, acquire response text related to the user input, acquire information about a plurality of operations for controlling the electronic device from the response text, acquire information about a current setting state of the electronic device, and display the current setting state of the electronic device. Identifying a first pre-execution operation performed before receiving the user input among the plurality of operations based on the information about the current setting state of the electronic device, and acquiring order information based on the remaining operations, and executing the first pre-execution operation based on the information about the current setting state of the electronic device and the first pre-execution operation. Identifying at least one second operation required to complete a function corresponding to the response text among the plurality of operations, and controlling the display to provide a guide user interface (UI) that sequentially performs remaining operations including at least the second operation based on the order information.

Description

[0001] This application is a divisional application of the invention patent application with the application date of March 9, 2020, application number 202080007414.0, and invention name “Electronic device and its control method”. Technical Field

[0002] The present disclosure relates to an electronic device and a control method thereof. For example, the present disclosure relates to an electronic device and a control method thereof using a machine reading comprehension (MRC) model. Background Art

[0003] Advances in electronic technology have led to the development and distribution of various types of electronic devices.

[0004] Not only have the types of electronic devices diversified, but also the functions and operations that the electronic devices can perform have diversified.

[0005] However, since the various functions and operations of electronic devices are somewhat complex, there is a problem that the accessibility of various functions and operations may be poor for the silver (old) generation of users who are not familiar with the latest electronic devices or users who are not good at operating electronic devices.

[0006] Therefore, it is necessary to provide users with methods of using electronic devices flexibly and actively (for example, methods of executing functions and operations, control methods or approach methods) through machine learning algorithms, rather than simply providing execution methods and approach methods of various functions and operations in a fixed and formal manner based on the manual of the electronic device. Summary of the Invention

[0007]

Technical Issues

[0008] An embodiment of the present disclosure provides an electronic device for providing a user interface (UI) for guiding a usage method based on a manual of the electronic device, and a control method thereof.

[0009]

Technical Solution

[0010] According to an example embodiment, an electronic device includes: a display and a processor, the processor being configured to control the electronic device to: obtain text corresponding to an input based on a machine reading comprehension (MRC) model, obtain a plurality of operation information by dividing the text into operation units of the electronic device, obtain sequence information of the plurality of operation information based on the plurality of operation information and the obtained text, and control the display to provide a guide UI, thereby sequentially performing a plurality of operations based on the sequence information.

[0011] The operation unit may be based on a context requiring a user touch input.

[0012] The processor is configured to control the electronic device to: obtain current setting status information on the electronic device, identify at least one operation information among the multiple operation information pre-executed in the electronic device based on the current setting status information, and obtain the sequence information based on the remaining operation information other than the identified operation information.

[0013] The processor is configured to control the electronic device to: obtain current setting status information of the electronic device; based on the current setting status information, in addition to the multiple operation information, also obtain additional operation information required to perform the function corresponding to the obtained text; and obtain sequence information based on the multiple operation information and the additional operation information.

[0014] The processor is configured to control the electronic device to: provide a first guide UI based on information related to touch input required to execute first operation information among the multiple operation information based on the sequence information; and control the display to provide a second guide UI based on information related to touch input required to execute second operation information among the multiple operation information based on the changed context and sequence information based on the context of the electronic device that changes based on the touch input corresponding to the first guide UI.

[0015] The information related to the touch input may include at least one of a position of the touch input or a touch type.

[0016] The MRC model may include a question generation (QG) model configured to acquire a question and a question answering (QA) model configured to acquire an answer corresponding to the acquired question by performing natural language processing (NLP) based on a plurality of sentences included in a manual of the electronic device.

[0017] The electronic device may further include a communicator, which includes a communication circuit, and the processor is configured to control the communicator to send the input to a server, and based on receiving text corresponding to the input from the server, obtain the multiple operation information and the sequence information based on the received text, and the text can be obtained based on a manual and an MRC model of the electronic device.

[0018] The processor is configured to control the electronic device to provide at least one of visual feedback or sound feedback based on recognition that at least one of a position or a touch type of a touch input does not correspond to the guide UI.

[0019] The processor is configured to control the electronic device to: provide a UI for selecting any one of a manual mode or an automatic mode; and based on selecting the automatic mode, sequentially perform multiple operations by executing a touch input function corresponding to the guide UI.

[0020] The processor is configured to control the electronic device to execute a touch input function corresponding to the guide UI and provide visual feedback indicating that the touch input function is executed.

[0021] According to an example embodiment, a method for controlling an electronic device includes: acquiring text corresponding to an input based on a machine reading comprehension (MRC) model; acquiring a plurality of operation information by dividing the text into operation units of the electronic device; acquiring sequence information of the plurality of operation information based on the plurality of operation information and the acquired text; and providing a guiding user interface (UI) to sequentially perform the plurality of operations based on the sequence information.

[0022] The operating unit may be based on the context in which touch input is required.

[0023] The method may include obtaining current setting status information on the electronic device, and obtaining sequence information may include identifying at least one operation information among multiple operation information pre-executed in the electronic device based on the current setting status information; and obtaining sequence information based on the remaining operation information other than the identified operation information.

[0024] The method may include obtaining current setting status information of the electronic device; and obtaining additional operation information required to perform a function corresponding to the obtained text based on the current setting status information in addition to the multiple operation information, and obtaining sequence information may include obtaining sequence information based on the multiple operation information and the additional operation information.

[0025] Providing a guidance user interface may include providing a first guidance UI based on information related to the touch input required to execute the first operation information among the multiple operation information based on the sequence information; and based on the context of the electronic device that changes based on the touch input corresponding to the first guidance UI, controlling the display to provide a second guidance UI based on information related to the touch input required to execute the second operation information among the multiple operation information based on the changed context and sequence information.

[0026] The information related to the touch input may include at least one of a position of the touch input or a touch type.

[0027] The MRC model may include a question generation (QG) model configured to obtain a question and a question answering (QA) model configured to obtain an answer corresponding to the obtained question by performing natural language processing (NLP) based on a plurality of sentences included in a manual of the electronic device.

[0028] Retrieving the text may include sending the input to the server, and the text may be retrieved based on a manual of the electronic device and an MRC model.

[0029] The method may include providing at least one of visual feedback or audible feedback based on identification that at least one of the location of the touch input or the touch type does not correspond to the guidance UI.

[0030]

Beneficial effects

[0031] According to various exemplary embodiments, an intuitive guide UI for a control method of an electronic device corresponding to a user input may be provided to the user, and a use method considering a current setting state of the electronic device may be provided to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is a block diagram illustrating an example configuration of an electronic device according to an embodiment;

[0034] Figure 2 is a diagram showing a method according to an embodiment of the present invention. Figure 1 A block diagram of an example configuration of an example electronic device is shown;

[0035] Figure 3 is a diagram illustrating an example electronic device and user input according to an embodiment;

[0036] Figure 4 is a diagram illustrating example operation information and sequence information according to an embodiment;

[0037] Figure 5 is a diagram illustrating example current setting information according to an embodiment;

[0038] Figure 6 is a diagram illustrating example current setting state information according to another embodiment;

[0039] Figure 7 is a diagram showing exemplary additional operation information according to another embodiment;

[0040] Figure 8 is a diagram illustrating an example guide UI according to an embodiment;

[0041] Figure 9 is a diagram illustrating example feedback according to an embodiment;

[0042] Figure 10 is a diagram illustrating an example electronic device communicating with a server according to an embodiment; and

[0043] Figure 11 is a flowchart illustrating an example method for controlling an electronic device according to an embodiment. DETAILED DESCRIPTION

[0044] The present disclosure will be described in more detail below with reference to the accompanying drawings.

[0045] The terms used in this disclosure and claims are general terms identified in consideration of the functions of various exemplary embodiments of the present disclosure. However, these terms may vary depending on the intentions of those skilled in the relevant art, technical interpretations, the emergence of new technologies, etc. Unless otherwise specifically defined, the terms should be understood based on the overall context and the technical understanding of those skilled in the relevant art.

[0046] In the present disclosure, expressions such as “having”, “may have”, “including” or “may include” indicate the existence of corresponding features (e.g., components such as numbers, functions, operations or parts), and do not exclude the existence of additional features.

[0047] The expression "at least one of A or / and B" should be understood to mean any one of "A" or "B" or "A and B".

[0048] As used herein, the terms “first,” “second,” etc. may refer to various components regardless of order and / or importance, and may be used to distinguish one component from another and does not limit the components.

[0049] Furthermore, the description in the present disclosure that one element (e.g., a first element) and another element (e.g., a second element) are “(operably or communicatively) coupled / coupled to” or “connected to” another element (e.g., the second element) should be interpreted as including the case where one element is directly coupled to another element and the case where one element is coupled to another element via another intermediate element (e.g., a third element).

[0050] Unless otherwise specified, singular expressions include plural expressions. It should be understood that terms such as "comprising" or "consisting of..." are used herein to indicate the presence of a feature, quantity, step, operation, element, component, or combination thereof, and do not exclude the presence or possibility of adding one or more other features, quantities, steps, operations, elements, components, or combinations thereof.

[0051] Terms such as "module," "unit," and "component" may be used to refer to an element that performs at least one function or operation, and such an element may be implemented as hardware or software, or a combination of hardware and software. In addition, except when each of a plurality of "modules," "units," and "components" needs to be implemented in separate hardware, the components may be integrated into at least one module or chip and implemented in at least one processor (not shown).

[0052] In this disclosure, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence (AI) electronic device).

[0053] Hereinafter, various exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0054] Figure 1 is a block diagram illustrating an example configuration of an electronic device according to an embodiment.

[0055] The electronic device 100 according to various embodiments may be implemented as a device having a display function, such as, but not limited to, a television, a smartphone, a tablet computer, a portable multimedia player (PMP), a personal digital assistant (PDA), a laptop computer, a smartwatch, a head-mounted display (HMD), a near-eye display (NED), etc. The electronic device 100 may be implemented to include various types of displays 110 to provide a display function, such as, but not limited to, a liquid crystal display (LCD), an organic light emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a micro LED, a quantum dot (QD) display panel, etc.

[0056] Various embodiments can be implemented by an electronic device 100 that does not include a display function. For example, the electronic device 100 can be implemented as various types of source devices that provide content to a display device, such as, but not limited to, a Blu-ray player, a digital versatile disk (DVD) player, a streaming content output device, a set-top box, etc. As another example, various types of home appliances, such as, but not limited to, speakers, refrigerators, washing machines, air conditioners, air purifiers, various Internet of Things (IoT), etc., can implement various embodiments. Hereinafter, for ease of description, the electronic device 100 will be assumed to be a user terminal device including a display 110.

[0057] refer to Figure 1 , the display device 100 includes a display 110 and a processor (eg, including a processing circuit) 120 .

[0058] The display 110 may provide various content screens that may be provided by the electronic device 100. The content screens may include, for example, but not limited to, application execution screens, various contents such as images, videos, text, music, graphical user interface (GUI) screens, and the like.

[0059] The display 110 may display text corresponding to a user input and a guide UI based on a manual (or user guide) of the electronic device 100 , which will be described in more detail below.

[0060] The processor 120 may include various processing circuits and may be implemented, for example, but not limited to, a digital signal processor (DSP) for processing digital image signals, a microprocessor, and a time controller (TCON), but is not limited thereto. The processor 120 may include, for example, but not limited to, one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a graphics processing unit (GPU), a communication processor (CP), an advanced reduced instruction set (RISC) computer (ARM) processor, etc., or may be defined as a corresponding term. The processor 120 may be implemented in a system on chip (SoC) type or a large-scale integration (LSI) type with a built-in processing algorithm, or in a field programmable gate array (FPGA) type. The processor 120 may include various processing circuits and perform various functions by executing computer-executable instructions stored in a memory.

[0061] The processor 120 may acquire text corresponding to user input based on a machine reading comprehension (MRC) model. For example, the MRC model may be a machine reading model that reads and interprets documents and manuals related to the electronic device 100 based on an artificial intelligence (AI) algorithm. For example, the MRC model may analyze documents using a natural language processing (NLP) algorithm based on various types of deep learning training, such as, but not limited to, a recursive neural network (RNN) and a convolutional neural network (CNN). Documents, manuals, and the like associated with the electronic device 100 are not limited to user manuals distributed by the manufacturer of the electronic device 100, and may include various types of data, including methods for operating or using a plurality of functions included in the electronic device 100.

[0062] The MRC model according to an embodiment may include a machine reading model suitable for natural language evaluation indicators, such as, but not limited to, the General Language Understanding Evaluation (GLUE) or the Stanford Question Answering Dataset (SCADY). According to an embodiment, the MRC model may be a model that reads and interprets a document and finds a text corresponding to a user input (e.g., a question) based on the document (e.g., an answer to the question). In addition to RNN and CNN, the MRC model according to an embodiment may be, for example, a model trained using a neural network based on a transformer.

[0063] The MRC model according to an embodiment may include a question generation (QG) model and a question answering (QA) model for acquiring answers to acquired questions by performing NLP based on a plurality of sentences included in the manual of the electronic device 100 .

[0064] For example, the QG mathematical model included in the MRC mathematical model can obtain the keywords (or topics) of the corresponding sentences based on the sentences included in the manual. The MRC mathematical model can obtain the optimal form of the question from multiple questions generated in the corresponding sentences. The QA model can identify the corresponding sentence as the answer to the obtained question.

[0065] The processor 120 according to the embodiment may perform pre-processing on a plurality of sentences included in the manual of the electronic device 100 to obtain a plurality of questions for each keyword and an answer corresponding to each of the plurality of questions. Figure 3 Describe it in more detail.

[0066] Figure 3 is a diagram illustrating an example electronic device and user input according to an embodiment.

[0067] refer to Figure 3 , the electronic device 100 according to an example embodiment can receive user input. For example, the electronic device 100 can receive a voice uttered by a user via a microphone (not shown) provided in the electronic device 100. As another example, the electronic device 100 can receive user input via various types of interfaces (e.g., a touchpad) provided in the electronic device 100. In the following description, for convenience of description, it is assumed that a voice uttered by a user is received as user input via the microphone.

[0068] like Figure 3 As shown, if the user input "Turn on Wi-Fi and Bluetooth" is received as user input, the processor 120 may obtain text corresponding to the user input. According to an example, the processor 120 may identify questions based on an NLP algorithm regarding the user input. For example, the processor 120 may identify keywords included in the user input and select one of a plurality of questions for each keyword obtained from the QG model based on the identified keywords.

[0069] refer to Figure 3 , the processor 120 may perform syntactic analysis on the user input. The processor 120 may then perform semantic analysis on the parsed data and identify keywords (e.g., Wi-Fi, Bluetooth, and power on) included in the user input. The processor 120 may then obtain questions corresponding to the identified keywords from a plurality of questions. For example, the processor 120 may obtain “How do I turn on Wi-Fi?” and “How do I turn on Bluetooth?”

[0070] According to an embodiment, the processor 120 can obtain answers corresponding to the questions obtained. For example, the processor 120 can generate questions from sentences included in the manual of the electronic device 100 based on the MRC model, and obtain the sentences as answers to the generated questions. For example, the processor 120 can obtain the sentence "How do I turn on Wi-Fi?" as the answer to the question. This embodiment is just an example. In addition to the multiple questions obtained by pre-processing the manual and the answers to each of the multiple questions, the processor 120 can also perform NLP on the user input to identify keywords and obtain answers corresponding to the identified keywords. For example, if "Let me know how to install application A" is received, the processor 120 can identify application A and the installation method based on the NLP algorithm. The processor 120 can perform a search for methods to install application A to obtain answers from the server. At the same time, for ease of description, the answers to the questions input by the user are collectively referred to as text corresponding to the user input.

[0071] According to an embodiment, the processor 120 can output a question corresponding to the user input based on the MRC model. In one example, when the user inputs "How do I turn on Wi-Fi?" and "How do I turn on Bluetooth?", the processor 120 can output a user interface (UI) indicating whether the user input was properly recognized, such as "Do you want to know how to turn on Wi-Fi and Bluetooth?". Upon receiving a confirmation input from the user on the corresponding UI, the text (or answer) corresponding to the question can be obtained.

[0072] return Figure 1 , the processor 120 according to the embodiment can obtain text corresponding to the user input, perform NLP on the obtained text, and classify the text into the operation unit of the electronic device 100 to obtain a plurality of operation information. The processor 120 can then obtain sequence information of the plurality of operation information based on the plurality of operation information and the obtained text. Figure 4 Describe it in more detail.

[0073] Figure 4 is a diagram illustrating example operation information and sequence information according to an embodiment.

[0074] refer to Figure 4, when receiving the user input of “Please turn on Wi-Fi and Bluetooth”, the processor 120 according to an embodiment can identify the questions “How do I connect to Wi-Fi?” and “How do I connect to Bluetooth?” among the multiple questions generated based on the multiple sentences included in the manual of the electronic device 100. The processor 120 can then obtain the sentence corresponding to the identified question as the answer (or text) to the question. According to one example, the processor 120 can obtain the sentence “Press the connection button on the settings screen → slide the Wi-Fi switch button to the left” included in the manual of the electronic device 100. The processor 120 can obtain “Press the connection button on the settings screen → slide the Bluetooth switch button to the left” as the text corresponding to the user input.

[0075] The processor 120 may then perform NLP on the acquired text to distinguish it with the operating unit of the electronic device 100. The operating unit of the electronic device 100 may be a unit based on a context requiring a user touch input. The operating unit of the electronic device 100 may be based on whether a user's touch input is required, or the operating unit of the electronic device 100 may be a unit based on whether a setting of the electronic device 100 is changed, whether a screen provided by the display 110 is changed, or whether a state of the electronic device 100 is changed according to a user's touch input.

[0076] For example, if "Press the connection button on the settings screen → slide the Bluetooth switch button to the left" is received as text corresponding to user input, the processor 120 can divide the text according to the operation unit of the electronic device 100 and obtain multiple operation information, such as the settings button, the connection button, the Wi-Fi switch button, etc.

[0077] The processor 120 may then obtain sequence information based on the multiple operation information and the corresponding text. For example, the processor 120 may set an order for each of the multiple operation information based on the corresponding text. For example, the processor 120 may arrange the settings button → the connection button → the Wi-Fi switch button in order based on the order of the multiple operations included in the text. The processor 120 may obtain sequence information for the multiple operations. According to an embodiment, the processor 120 may assign Fn1, Fn2, Fn3, Fn4, etc. to each of the multiple operations based on the sequence information.

[0078] The processor 120 according to an embodiment may provide a guide user interface (UI) 10 for sequentially performing a plurality of operations according to sequence information.

[0079] refer to Figure 4, processor 120 may obtain a setup button, a connect button, a Wi-Fi toggle button, and a Bluetooth toggle button as multiple operations based on the text corresponding to the user input "Please turn on Wi-Fi and Bluetooth." Processor 120 may then assign a sequence to each of the multiple operations to obtain sequence information. For example, processor 120 may obtain ordered sequence information in the order of setup button → connect button → Wi-Fi toggle button → Bluetooth toggle button.

[0080] like Figure 4 As shown, the processor 120 may provide a first guidance UI through the display 110 based on information related to a touch input required to perform the first operation information Fn1 among the plurality of operation information according to the sequence information. For example, when the first operation information Fn1 is a setting button based on the sequence information, the processor 120 may provide the guidance UI 10 to induce a touch input to the setting button. For example, the processor 120 may display the guidance UI 10 in a predetermined size and color at the location of the setting button on the screen provided by the display 110.

[0081] When the context of the electronic device 100 is changed based on the touch input corresponding to the first guide UI, the processor 120 according to an embodiment may control the display 110 to provide a second guide UI based on information related to the touch input required to perform the second operation information Fn2 among the plurality of operation information based on the changed context and sequence information. The change in the context of the electronic device 100 may include at least one of receiving a touch input, changing a screen, or changing a setting of the electronic device 100.

[0082] refer to Figure 4 , when a touch input for the first guide UI is received and the setting screen is displayed, the processor 120 may provide a second guide UI for guiding the touch input to the position of the connection button on the screen according to the second operation information Fn2 based on the sequence information. The information related to the touch input is not limited to the position of the touch input, and may include the touch type. For example, the processor 120 may display a guide UI 10 for inducing a user's click touch input at a specific position on the screen, and may provide a guide UI 10 corresponding to various touch types, such as drag touch, tap touch, pinch touch, expand touch, etc. A detailed description thereof will be described in Figure 8 Available in.

[0083] In addition to the guide UI 10, the processor 120 according to an embodiment may provide a script 20. Figure 4Based on the acquired text and its description on the screen area, the processor 120 can provide a script 20 for each piece of operation information. For example, the processor 120 can provide a script 20 for "Press the Settings Button" with the guidance UI 10, which triggers a touch on the Settings button. As another example, the processor 120 can provide a script 20 for "Press the Connect Button" corresponding to the second operation information Fn2 in the acquired text "Press the Connect Button on the Settings Screen → Slide the Wi-Fi Switch Button Left," along with a second guidance UI for triggering the touch input required to execute the second operation information Fn2. As another example, the processor 120 can provide a script 20 for "Slide the Wi-Fi Switch Button Left" corresponding to the third operation information Fn3, along with a third guidance UI for triggering the touch input required to execute the third operation information Fn3. As yet another example, the processor 120 can provide a script 20 for "Slide the Bluetooth Switch Button Left" corresponding to the fourth operation information Fn4, along with a fourth guidance UI for triggering the touch input required to execute the fourth operation information Fn4.

[0084] The processor 120 according to an embodiment may output the script 20 provided along with the guide UI on the screen as a sound signal. For example, the processor 120 may perform text-to-speech conversion (TTS) on the script 20 to obtain a sound signal corresponding to the script 20 and output the sound signal along with the guide UI 10.

[0085] return Figure 1 , the processor 120 according to an embodiment may acquire current setting state information of the electronic device 100. The current setting state information may refer to, for example, a setting value of each of a plurality of functions included in the electronic device 100 (e.g., whether the function is turned on / off), information about applications installed in the electronic device 100 (e.g., application installation status, application version information, the position of an icon for executing the application), a screen displayed by the display 110, etc. The processor 120 according to an embodiment may acquire the current setting state information at a system level, or may acquire the current setting state information through an application programming interface (API) call.

[0086] The processor 120 according to the embodiment can obtain the sequence information by excluding some operation information from the multiple operation information obtained from the text based on the current setting state information. As another example, the processor 120 can obtain the sequence information by adding other operation information in addition to the multiple operation information based on the current setting state information. This will be referred to below. Figure 5 、 6 and 7 for a more detailed description.

[0087] Figure 5 is a diagram illustrating example current setting information according to an embodiment.

[0088] refer to Figure 5 , the processor 120 according to an embodiment may identify at least one pre-executed operation information among a plurality of operation information acquired by dividing a text by an operation unit of the electronic device 100 based on the current setting state information.

[0089] For example, text corresponding to user input (turn on Wi-Fi and Bluetooth) "Press the connect button on the settings screen → slide the Wi-Fi switch button to the left" and "Press the connect button on the settings screen → slide the Bluetooth switch button to the left" can be obtained.

[0090] The processor 120 can divide the acquired text into operation units to obtain a settings button, a connection button, a Wi-Fi switch button, and a Bluetooth switch button as multiple operation information. The processor 120 according to an embodiment can identify previously performed operation information among multiple operation information based on the current setting status information of the electronic device 100. As an example, if the screen provided by the display 110 is a settings screen, the processor 120 can identify that an operation of touching the settings button was previously performed. In this example, the processor 120 can exclude the settings button from the multiple operation information and then obtain sequence information based on the remaining operation information. As another example, when it is identified that the Wi-Fi function of the electronic device 100 is turned on based on the current setting status information, the processor 120 can exclude the Wi-Fi switch button from the multiple operation information and obtain sequence information based on the remaining operation information.

[0091] Even if the user input is in the same way "Turn on Wi-Fi and Bluetooth", reference Figure 5 , the processor 120 may also not provide a UI for guiding the touch input of the Wi-Fi switch button determined as the pre-executed operation information, unlike Figure 4 The example shown, and a UI for guiding the touch input of the Bluetooth switch button can be provided.

[0092] Figure 6 is a diagram illustrating current setting state information according to another embodiment.

[0093] The processor 120 according to various embodiments may identify a question based on user input and obtain text (eg, an answer) corresponding to the question. Figure 6 If the user input is "Let me know how to use navigation," processor 120 may retrieve a question and answer that is determined to be closest to the user from among multiple questions with navigation as a keyword based on the MRC model. Here, a keyword does not necessarily mean that the word should be included in the question or answer. For example, processor 120 may retrieve a question corresponding to the user input and an answer to the question from among multiple questions corresponding to each topic.

[0094] refer to Figure 6 , the processor 120 according to an embodiment may obtain text “Press the map application → Press the microphone button and let me know the destination” corresponding to “Let me know how to use navigation” as a user input. Then, the processor 120 may obtain sequence information for sequentially causing touch inputs of the map application Fn1 and the microphone button Fn2, and provide the guide UI 10 and the script 20 based on the sequence information.

[0095] According to another embodiment, the processor 120 may acquire sequence information including additional operation information based on the current setting state information of the electronic device 100. The additional operation information may mean another operation information in addition to the multiple operation information acquired by dividing the text into operation units. This will be referred to below. Figure 7 Describe in more detail.

[0096] Figure 7 is a diagram illustrating exemplary additional operation information according to another embodiment.

[0097] refer to Figure 7 In addition to the multiple operation information based on the current setting state information, the processor 120 may obtain additional operation information required to execute the function corresponding to the text. For example, the processor 120 may obtain sequence information for sequentially causing touch inputs of the map application Fn1 and the microphone button Fn2. According to an example embodiment, the processor 120 may identify whether the map application is installed based on the current setting state information in order to execute "Please press the map application → press the microphone button and let me know the destination", which is the text corresponding to the user input. As another example, the processor 120 may identify whether the electronic device 100 includes a microphone.

[0098] According to an embodiment, if it is recognized based on the current setting state information that the map application is not installed on the electronic device 100, the processor 120 may obtain operation information for installing the map application in addition to the plurality of operation information (e.g., the map application Fn1 and the microphone button Fn2) as additional operation information. The processor 120 may then obtain sequence information based on the plurality of operation information and the additional operation information.

[0099] Even if the user input is the same as "Let me know how to use navigation", refer to Figure 7 ,and Figure 6 Unlike the example shown in , the processor 120 may additionally provide a guide UI 10 for causing a touch input to press a shop button and a guide UI 10 for causing a touch input to install a map application.

[0100] As another example, in addition to a plurality of operation information based on the current setting state information, the processor 120 according to an embodiment may obtain additional operation information required to execute the function corresponding to the text. In one example, the processor 120 may identify the required additional actions in the current setting state information so that the electronic device 100 operates corresponding to the user input. For example, if the user input is "Let me know how to use navigation", the processor 120 may obtain the map application Fn1 and the microphone button Fn2 as operation information. The processor 120 may then identify whether the electronic device 100 is in a suitable state for executing the map application Fn1 based on the current setting state information of the electronic device 100. For example, if the global positioning system (GPS) function of the electronic device 100 is turned off, the processor 120 may obtain the operation Fn 1-1 (not shown) to turn on the GPS based on the MRC model.

[0101] According to an embodiment, the processor 120 can identify the sub-operations of each operation. As described above, the sub-operations corresponding to the operation Fn1 for executing the map application may include the operation Fn 1-1 for turning on the GPS function and the operation Fn 1-2 (not shown) for identifying the current location of the electronic device 100 through the map application. The processor 120 can then identify at least one of the multiple sub-operations as additional operation information based on the current setting state information of the electronic device 100. The processor 120 can then obtain sequence information based on the multiple operation information and the additional operation information. For example, if the GPS function is turned off, in addition to the operation Fn 1 for executing the map application based on the text corresponding to the user input "Let me know how to use navigation" and the operation Fn2 for eliciting the speech of the destination, the processor 120 can obtain the operation Fn 1-1 for turning on the GPS function as additional operation information. The processor 120 can provide a guidance UI corresponding to each operation in the order of Fn1 → Fn 1-1 → Fn2.

[0102] As another example (not shown), the processor 120 according to an example embodiment may identify additional operations that need to be performed before performing one of the multiple operations. For example, if the user input text is "Let me know how to use the dictionary function", the processor 120 may obtain pressing the search term Fn1, pressing the dictionary Fn 2 in the option list, and checking the search result Fn3 as multiple operation information. The processor 120 may identify "installing the dictionary application Fn 2-1" as an additional operation that needs to be performed in advance to perform pressing the dictionary Fn 2 in the option list. The sequence information may then be adjusted so that the identified additional operations are placed in advance. For example, the guide UI corresponding to each operation may be provided in the following order: pressing the search term Fn1 → installing the dictionary application Fn 2-1 → pressing the dictionary Fn 2 in the option list → checking the search result Fn3 based on the operation information and the additional operation. The processor 120 according to an embodiment may not include the previous operation in the sequence information based on the current setting state information of the electronic device 100. For example, if it is recognized based on the current setting state information that the dictionary application has been installed, the processor 120 may generate order information such that the installation dictionary application Fn 2-1 is not added before the dictionary Fn 2 in the option list is pressed.

[0103] As another embodiment (not shown), the processor 120 according to the embodiment can perform some operations by itself in the background state without providing a guidance UI. For example, when the user input is "Let me know how to use the dictionary", the processor 120 can obtain the following sequence information based on the operation information and the additional operation based on the text corresponding thereto: press the search term Fn1 → install the dictionary application Fn2-1 → press the dictionary Fn2 in the option list → check the search results Fn3. In this example, the processor 120 can install the dictionary application by itself in the background state instead of providing a guidance UI corresponding to the installation of the dictionary application Fn 2-1, which is a prior operation. In this example, the guidance provided to the user can be in the order of Fn1 → Fn2 → Fn3.

[0104] Figure 8 is a diagram illustrating an example guide UI according to an embodiment.

[0105] The guide UI according to an embodiment may guide the position of the user's touch input and guide the touch type. For example, the processor 120 may identify the touch type required to perform any one of the plurality of operation information acquired by performing NLP on the acquired text.

[0106] For example, Figure 8As shown, the user input may be “How to zoom in on a photo?” and the processor 120 may obtain text corresponding to the user input “press the screen with two fingers and spread them outward to adjust the screen magnification.” In this example, the processor 120 may provide a guide UI 10 for guiding the extended touch input. Figure 8 As shown, the processor 120 may provide the guide UI 10 including an animation effect as if the first object and the second object included in the guide UI 10 are spaced apart from each other.

[0107] As another example, when the user input is "How do I zoom in on the screen?", the processor 120 may obtain text corresponding to the user input "You can zoom in on the screen by quickly pressing the target screen area three times." In this example, the processor 120 may provide the guidance UI 10 including an animation effect, as if a three-tap touch input is performed on an area of ​​the screen. This is merely an example and is not limiting.

[0108] The processor 120 according to an embodiment may provide a UI for selecting one of the manual mode and the automatic mode. When a user's touch input is received according to the guide UI 10, the manual mode may provide the guide UI 10 corresponding to the operation information of the next sequence based on the sequence information. Even if a user's touch input corresponding to the guide UI 10 is not received, the automatic mode may still be a mode for changing the context of the electronic device 100 according to the acquired text.

[0109] For example, when no user's touch input is received in the automatic mode, the processor 120 may execute a touch input function corresponding to the guide UI 10 and provide visual feedback indicating that the touch input function is executed. For example, the processor 120 may change the context of the electronic device 100 and provide the guide UI 10 corresponding to the next operation information based on the sequence information while providing an animation effect as if a touch input to the guide UI 10 is received.

[0110] Embodiments are not necessarily limited thereto. For example, the processor 120 may sequentially change the context of the electronic device 100 based on the sequential information in the background without changing the screen provided by the display 110. For example, if the text corresponding to the user input is "Drag in the + direction on the adjustment bar to illuminate the screen", a UI for guiding the drag touch input (e.g., manual mode) may be provided, and the processor 120 may automatically adjust the brightness of the screen (e.g., automatic mode) using the background.

[0111] return Figure 1If it is recognized that at least one of the position or touch type of the user touch input does not correspond to the guide UI 10, the processor 120 according to the embodiment may provide at least one of visual feedback or sound feedback. This will be referred to below. Figure 9 Describe in more detail.

[0112] Figure 9 is a diagram illustrating example feedback according to an embodiment.

[0113] When recognizing that the position 30 of the user's touch input corresponds to the guide UI 10 , the processor 120 according to an embodiment may sequentially provide the guide UI 10 corresponding to the next operation information based on the sequence information.

[0114] As another example, if it is recognized that the position 30 of the user's touch input does not correspond to the guide UI 10, the processor 120 may provide feedback to cause the touch input to be performed again. Figure 9 Although, according to an embodiment, the guide UI 10 introduces a tap touch input to the display, when the position 30 of the user touch input corresponds to a sound and a vibration, the processor 120 may provide at least one of visual feedback or sound feedback. The visual feedback may be feedback that changes at least one of the size or color of the guide UI 10. However, this is merely an example, and various types of visual feedback may be provided.

[0115] The sound feedback according to the exemplary embodiment may include sound effects, beeps, etc. As another example, the sound feedback may be feedback output as a sound signal based on the script 20 corresponding to the guide UI 10. For example, referring to Figure 9 , the processor 120 can provide sound feedback to output “You pressed the wrong button, please press the displayed button again” as a sound signal.

[0116] return Figure 1 , the electronic device 100 according to an example embodiment may further include a communicator (e.g., including a communication circuit) (not shown), and the processor 120 may control the communicator to send the user input to the server. If a text corresponding to the user input is received from the server, the processor 120 may obtain a plurality of operation information and sequence information based on the text. The text may be obtained based on the manual of the electronic device 100 and the MRC model. Figure 10 This is described in detail.

[0117] Figure 10 is a diagram illustrating an example electronic device communicating with a server according to an embodiment.

[0118] refer to Figure 10, the electronic device 100 can communicate with the server 200. The processor 120 can send the user input to the server 200 and receive the text corresponding to the user input from the server 200. The server 200 may include multiple question-and-answer data sets based on the MRC model of the document, manual, or instruction manual of the electronic device 100, but this is merely an example, and the question and the answer corresponding to the question do not necessarily appear in pairs. For example, the server 200 can obtain keywords, topics, etc. by performing NLP on the user input, and obtain the text (or answer) corresponding to the user input from the search results based on the obtained keywords, topics, etc.

[0119] Figure 2 is a diagram showing a method according to an embodiment of the present invention. Figure 1 A block diagram of an example configuration of an example electronic device.

[0120] refer to Figure 2 The electronic device 100 according to the embodiment may include a display 110, a processor (e.g., including a processing circuit) 120, a communicator (e.g., including a communication circuit) 130, a memory 140, a user interface (e.g., including a user interface circuit) 150, an input and output interface (e.g., including an input and output circuit) 160, a microphone 170, and a speaker 180. The electronic device 100 will not be further described. Figure 1 Configuration overlap Figure 2 configuration.

[0121] The processor 120 may include various processing circuits and control the overall operation of the electronic device 100 using various programs stored in the memory 140 .

[0122] The processor 120 includes a random access memory (RAM) 121 , a read only memory (ROM) 122 , a main CPU 123 , first to Nth interfaces 124 - 1 to 124 -N, and a bus 135 .

[0123] Here, the RAM 121 , the ROM 122 , the main CPU 123 , the first to Nth interfaces 124 - 1 to 124 -N, etc. may be interconnected through the bus 125 .

[0124] ROM 122 stores a command set for booting the system, etc. When a power-on command is input and power is supplied, main CPU 133 copies the operating system (OS) stored in storage 140 to RAM 121 according to the instructions stored in ROM 132, executes the OS, and boots the system. When booting is complete, main CPU 133 can copy various programs stored in storage to RAM 121, execute the programs copied to RAM 121, and perform various operations.

[0125] The main CPU 123 accesses the memory 140 and performs booting using an operating system (OS) stored in the memory 140. In addition, the CPU 123 performs various operations using various programs and the like stored in the memory 140.

[0126] The first to nth interfaces 124-1 to 124-n are connected to the various elements described above. One of the interfaces may be a network interface connected to an external device through a network.

[0127] The processor 120 can perform graphics processing functions (video processing functions). For example, the processor 120 can use a calculator (not shown) and a renderer (not shown) to generate a screen including various objects such as icons, images, text, etc. Here, the calculator (not shown) can calculate the attribute values ​​to be displayed by each object, such as coordinate values, shape, size, and color, according to the layout of the screen based on the received control command. The renderer (not shown) can generate display screens of various layouts including objects based on the attribute values ​​calculated by the calculator (not shown). The processor 120 can perform various image processing on the video data, such as decoding, scaling, noise filtering, frame rate conversion, resolution conversion, etc.

[0128] The processor 120 may perform processing of the audio data. For example, the processor 120 may perform various image processing on the audio data, such as, but not limited to, decoding, amplification, noise filtering, etc.

[0129] The communicator 130 may include various communication circuits and may be configured to communicate with various types of other external devices according to various communication types. The communication interface 130 includes various modules, each of which includes various communication circuits, such as, but not limited to, a Wi-Fi module 131, a Bluetooth module 132, an infrared communication module 133, a wireless communication module 134, a wired communication module 135, etc. Each communication module may be implemented in at least one hardware chip format.

[0130] The processor 120 may communicate with various external devices using the communicator 130. The external devices may include, for example, but not limited to, display devices such as TVs, image processing devices such as set-top boxes, external servers, control devices such as remote controllers, audio output devices such as Bluetooth speakers, lighting devices, smart vacuum cleaners, home appliances such as smart refrigerators, servers such as Internet of Things (IoT) home managers, and the like.

[0131] The Wi-Fi module 131 and the Bluetooth module 132 perform communication using the Wi-Fi method and the Bluetooth method, respectively. When using the Wi-Fi module 131 or the Bluetooth module 132, various connection information such as a service set identifier (SSID) and a session key may be first transmitted and received, and communication information may be transmitted after the communication is connected.

[0132] The infrared communication module 133 performs communication according to an Infrared Data Association (IrDA) technology that wirelessly transmits data to a local area using infrared rays between visible light and millimeter waves.

[0133] In addition to the above-mentioned Wi-Fi module 131 and Bluetooth module 132, the wireless communication module 134 may refer to a module that performs communication according to various communication standards, such as Zigbee, third generation (3G), third generation partnership project (3GPP), long term evolution (LTE), advanced LTE (LTE-A), fourth generation (4G), fifth generation (5G), etc.

[0134] The communicator 130 may include at least one of a local area network (LAN) module, an Ethernet module, or a wired communication module that performs communication using a paired cable, a coaxial cable, an optical cable, or the like.

[0135] According to an embodiment, the communicator 130 may communicate with external devices such as a remote controller and an external server using the same communication module (eg, Wi-Fi module).

[0136] According to another example, the communicator 130 can use different communication modules (e.g., Wi-Fi modules) to communicate with external devices such as remote controls and external servers. For example, the communication interface 130 can use at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and can use a Bluetooth (BT) module to communicate with an external device such as a remote control. However, this is merely an example, and when communicating with multiple external devices or external servers, the communicator 130 can use at least one of the various communication modules.

[0137] According to an example, the electronic device 100 may further include a tuner and a demodulator.

[0138] The tuner (not shown) may receive an RF broadcast signal from among (RF) broadcast signals received through an antenna by tuning a user-selected channel or all pre-stored channels.

[0139] The demodulator (not shown) may receive and demodulate a digital intermediate frequency (DIF) signal converted by the tuner, and perform channel decoding and the like.

[0140] The memory 140 may be implemented as an internal memory included in the processor 120, such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)), a RAM, etc., or a memory separate from the processor 130. In this case, the memory 140 may be implemented as a memory type embedded in the electronic device 100 or a memory type detachable from the electronic device 100, depending on the data storage purpose. For example, data for driving the electronic device 100 may be stored in a memory embedded in the electronic device 100, and data for expanding the electronic device 100 may be stored in a memory detachable from the electronic device 100. The memory embedded in the electronic device 100 can be: a volatile memory, such as but not limited to a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM); or a non-volatile memory (such as a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (such as a NAND flash memory or a NOR flash memory), a hard disk drive, a solid-state drive (SSD), etc. In the case where the memory is detachably mounted to the electronic device 100, the memory can be implemented as, for example, but not limited to, a memory card (such as a compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), multimedia card (MMC), etc.), an external memory connectable to a USB port (such as a USB memory), etc.

[0141] The user interface 150 may include various user interface circuits and be implemented as devices such as, but not limited to, buttons, a touch panel, a mouse, and a keyboard, or as a touch screen capable of performing the above-mentioned display functions and also manipulating input functions. The buttons may be various types of buttons formed in any area (e.g., the front surface portion, the side surface portion, and the rear surface portion) of the main body of the electronic device 100, such as a mechanical button, a touch pad, a scroll wheel, etc.

[0142] The input and output interface 160 may include various input / output circuits and be, for example, but not limited to, at least one interface of High-Definition Multimedia Interface (HDMI), Mobile High-Definition Link (MHL), Universal Serial Bus (USB), Display Port (DP), Thunderbolt, Video Graphics Array (VGA) port, RGB port, D-Subminiature (D-SUB), Digital Visual Interface (DVI), etc.

[0143] The input and output interface 160 may input and output at least one of a voice signal and a video signal.

[0144] According to an embodiment, the input and output interface 160 may include a port for inputting and outputting only a voice signal and a port for inputting and outputting only a video signal as separate ports, or may be implemented as one port for inputting and outputting both a voice signal and a video signal.

[0145] The electronic apparatus 100 may be implemented as an apparatus that does not include a display, and transmit a video signal to a separate display device.

[0146] The electronic device 100 may receive a user voice signal from an external device including a microphone. In this example, the received user voice signal may be a digital voice signal, but in some embodiments, the signal may be an analog voice signal. For example, the electronic device 100 may receive the user voice signal via a wireless communication method such as Bluetooth or Wi-Fi. The external device may be implemented as a remote control device or a smartphone.

[0147] The electronic apparatus 100 may transmit a corresponding voice signal to an external server for voice recognition of the voice signal received from the external device.

[0148] The communication modules for communicating with external devices and external servers can be implemented separately. For example, communication with external devices can be performed through a Bluetooth module, and communication with external servers can be performed through an Ethernet modem or a Wi-Fi module.

[0149] The electronic device 100 according to an embodiment may send a received digital voice signal to a voice recognition server. In this case, the voice recognition server may convert the user input into text information using speech-to-text conversion (STT). The voice recognition server may send the text information to another server or electronic device to perform a search corresponding to the text information, and in some cases, perform a direct search.

[0150] The electronic device 100 according to another embodiment may directly apply a speech-to-text conversion (STT) function to a user input to convert into text information, and transmit the converted text information to an external server.

[0151] The electronic device 100 may further include a microphone 170. The microphone 170 may refer to, for example, an element that receives a user's voice or other sounds and converts the received sounds into audio data.

[0152] The microphone 170 can receive user voice in an active state. For example, the microphone 170 can be integrally formed as an integral unit on the upper side, front side, side side, etc. of the electronic device 100. The microphone can include various configurations, such as a microphone for collecting user voice in analog format, an amplifier circuit for amplifying the collected user voice, an audio-to-digital (A / D) conversion circuit for sampling the amplified user voice and converting it into a digital signal, a filter circuit for removing noise elements from the converted digital signal, etc.

[0153] According to another embodiment, the electronic device 100 can receive a user voice from an external device (not shown) having a microphone. For example, the external device can be implemented with a remote control device (remote controller) including but not limited to a microphone. When the remote control device (remote controller) receives an analog voice signal from the user through the microphone, the remote control device (remote controller) can convert the analog voice signal into a digital voice signal. The remote control device (remote controller) can use at least one of infrared, Wi-Fi or Bluetooth communication to send the converted digital voice signal to the electronic device 100. When a digital voice signal is received from the external device, the electronic device 100 can perform voice recognition based on the received digital voice signal and perform a control operation based on the voice recognition result. According to another embodiment, the remote control device (remote controller) can perform a voice recognition operation on the digital voice signal and send information corresponding to the voice recognition result to the electronic device 100. The information corresponding to the voice recognition result can be at least one of the voice recognition result itself or a control command corresponding to the voice recognition result.

[0154] According to another embodiment, the external device may be implemented as a smartphone including a microphone. In this example, the smartphone may remotely control the electronic apparatus 100 using a remote control application for executing a pre-installed remote control function.

[0155] If an analog voice signal of a user is received through a microphone, the smartphone can convert the analog voice signal into a digital voice signal. In this example, the smartphone can perform voice recognition on the digital voice signal using a voice recognition application. Here, the voice recognition application can be the same as or different from the above-mentioned remote control application. When voice recognition is performed on the digital voice signal, the smartphone can use the remote control application to remotely control the electronic device 100 based on the voice recognition result. According to another embodiment, the smartphone can use at least one of infrared, Wi-Fi, or Bluetooth communication methods to send the converted digital voice signal to the electronic device 100. In this case, when a digital voice signal is received from an external device, the electronic device 100 can perform voice recognition based on the received digital voice signal and perform a control operation based on the voice recognition result.

[0156] The speaker 180 may be configured to output various notification sounds or voice messages as well as various audio data processed by the input and output interface 160 .

[0157] Figure 11 is a flowchart illustrating an example method for controlling an electronic device according to an embodiment.

[0158] In operation S1110 , the method of controlling an electronic device according to an embodiment acquires a text corresponding to a user input based on a machine reading comprehension model (MRC).

[0159] By dividing the text by the operation unit of the electronic device, a plurality of operation information is acquired in operation S1120.

[0160] In operation S1130, sequence information of the plurality of operation information is acquired based on the plurality of operation information and the acquired text.

[0161] In operation S1140 , a guide UI for sequentially performing a plurality of operations based on the sequence information is provided.

[0162] Here, the operation unit may refer to, for example, a unit based on a context requiring a user's touch input.

[0163] According to an example embodiment, a method of controlling an electronic device may include an operation of acquiring current setting status information of the electronic device, and the operation of acquiring sequence information in operation S1130 may include identifying at least one operation information previously performed in the electronic device among multiple operation information based on the current setting status information; and acquiring sequence information based on the remaining operation information other than the identified operation information.

[0164] According to an example embodiment, the method may include obtaining current setting status information of the electronic device; and obtaining additional operation information required to perform a function corresponding to the obtained text in addition to multiple operation information based on the current setting status information, and the step of obtaining sequence information in operation S1130 may include obtaining sequence information based on multiple operation information and additional operation information.

[0165] The operation of providing a guide UI in operation S1140 may include: providing a first guide UI based on information related to the touch input required to execute the first operation information among multiple operation information according to the sequence information; and providing a second guide UI based on information related to the touch input required to execute the second operation information among the multiple operation information based on the changed context and the sequence information when the context of the electronic device is changed based on the touch input corresponding to the first guide UI.

[0166] The information related to the touch input may include at least one of a position of the touch input or a touch type.

[0167] The MRC model according to an embodiment may include a QG model for acquiring a question by performing NLP based on a plurality of sentences included in a manual of the electronic device and a QA model for acquiring an answer corresponding to the acquired question.

[0168] The operation of acquiring the text in operation S1110 may include sending a user input to a server, and the text may be acquired based on a manual of the electronic device and an MRC model.

[0169] The method according to the embodiment may include providing at least one of visual feedback or sound feedback based on recognizing that at least one of a position or a touch type of the user's touch input does not correspond to the guide UI.

[0170] The methods according to the various exemplary embodiments described above may be implemented in an application format that can be installed in an electronic device.

[0171] The methods according to the various exemplary embodiments described above may be implemented through software upgrade and / or hardware upgrade of the electronic device.

[0172] The various exemplary embodiments described above may be performed by an embedded server provided in an electronic device or an external server of the electronic device.

[0173] According to the present disclosure, the various example embodiments described above can be implemented with software including instructions stored in a machine-readable storage medium that is readable by a machine (e.g., a computer). According to various example embodiments, the device can call instructions from a storage medium and operate according to the called instructions, and can include an electronic device (e.g., electronic device A). When the processor executes the instructions, the processor can use other components to perform functions corresponding to the instructions directly or under the control of the processor. The instructions may include code generated by a compiler or code executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium.

[0174] According to various exemplary embodiments of the present disclosure, a method may be provided in the form of a computer program product. The computer program product may be exchanged as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)) or through an application store (e.g., PlayStore). TM ) Online distribution. In the case of online distribution, at least part of the computer program product may be temporarily or at least temporarily stored in a storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0175] Each element (e.g., module or program) according to various example embodiments may include a single entity or multiple entities, and some sub-elements of the above-mentioned sub-elements may be omitted, which may be further included in various embodiments. Alternatively or additionally, some elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each corresponding element before integration. According to various embodiments, the operations performed by modules, programs, or other elements may be performed sequentially, in parallel, repeatedly, or heuristically, or at least some operations may be performed in a different order, omitted, or other operations may be added.

[0176] Although various embodiments have been illustrated and described with reference to the drawings, the present disclosure is not limited to the exemplary embodiments or the drawings, and those skilled in the art will understand that various changes in form and details may be made without departing from the spirit and scope, including, for example, the following claims and their equivalents.

Claims

1. An electronic device comprising: Display; and a processor configured to control the electronic device to: Receive user input, Get the response text associated with the user input, acquiring information on a plurality of operations for controlling the electronic device from the response text, obtaining information about the current setting state of the electronic device, identifying a first pre-execution operation among the plurality of operations that is performed before receiving the user input based on information about a current setting state of the electronic device, and acquiring sequence information based on remaining operations, identifying, based on information about a current setting state of the electronic device and the first pre-execution operation, at least one second operation among the plurality of operations required to complete a function corresponding to the response text, and The display is controlled to provide a guide user interface (UI) for sequentially executing the remaining operations including at least the second operation based on the sequence information.

2. The electronic device according to claim 1, wherein The processor is further configured to divide the response text into units of operation based on a context requiring a touch input.

3. The electronic device according to claim 1, wherein The processor is further configured to control the electronic device to: In addition to the first pre-execution operation and the second operation, additional operations required to complete the function are also obtained.

4. The electronic device according to claim 3, wherein: The processor is further configured to control the electronic device to: Based on the context of the electronic device changed according to the touch input corresponding to the guide UI, the display is controlled to provide a next guide UI based on information related to the touch input required to perform the additional operation.

5. The electronic device according to claim 4, wherein: The information related to the touch input includes at least one of a position of the touch input or a touch type. The electronic device according to claim 1 , wherein: The processor is further configured to control the electronic device to: Based on the machine reading comprehension (MRC) model, obtaining the response text related to the user input, The MRC model includes a question generation (QG) model configured to acquire a question and a question answer (QA) model configured to acquire an answer corresponding to the acquired question by performing natural language processing (NLP) based on a plurality of sentences included in a manual of the electronic device.

7. The electronic device according to claim 1, further comprising: a communicator including a communication circuit; wherein the processor is further configured to control the electronic device to: controlling the communicator to send the user input to the server, and Based on receiving text corresponding to the user input from the server, obtaining the plurality of operations based on the received text, and The received text is based on a manual and an MRC model of the electronic device.

8. The electronic device according to claim 1, in, The guide UI includes a UI for performing the second operation based on information related to the touch input required to perform the second operation, the UI for performing the second operation being different from the UI for performing the first pre-performance operation in the electronic device. 9 . The electronic device of claim 8 , wherein the processor is further configured to control the electronic device to provide at least one of visual feedback or sound feedback based on recognizing that at least one of a position or a touch type of an input does not correspond to the guide UI.

10. The electronic device according to claim 8, wherein The processor is further configured to control the electronic device to: Provide a UI for selecting either manual mode or automatic mode, and Based on the selection of the automatic mode, the plurality of operations are sequentially performed by executing a touch input function corresponding to the guide UI. The electronic device according to claim 10 , wherein: The processor is further configured to control the electronic device to execute a touch input function corresponding to the guide UI and provide visual feedback indicating that the touch input function is executed.

12. The electronic device according to claim 1, in, The plurality of operations are required to complete the function corresponding to the response text, and The second operation is an operation subsequent to the first pre-execution operation among the plurality of operations based on the sequence information.

13. A method for controlling an electronic device, the method comprising: Receive user input; Obtaining response text related to the user input; acquiring information about a plurality of operations for controlling the electronic device from the response text; obtaining information about a current setting state of the electronic device; identifying, based on information about a current setting state of the electronic device, a first pre-execution operation among the plurality of operations that is performed before receiving the user input, and acquiring sequence information based on remaining operations; identifying, based on information about a current setting state of the electronic device and the first pre-execution operation, at least one second operation from among the plurality of operations required to complete a function corresponding to the response text; as well as A guide user interface (UI) for sequentially performing the remaining operations including at least the second operation is provided based on the sequence information. The method of claim 13 , further comprising dividing the response text by units of operation based on a context requiring a touch input.

15. The method according to claim 13, further comprising: In addition to the pre-execution first operation and the second operation, additional operations required to complete the function are also obtained.

Images