Electronic device for controlling plurality of assistant modules and control method therefor

Abstract

An electronic device is disclosed. This electronic device comprises: a memory for storing at least one instruction; and a processor connected to the memory to control the electronic device. The processor is configured, by executing the at least one instruction, to: when a user voice is acquired, input the user voice into at least one assistant module from among a plurality of assistant modules; when a first response corresponding to the user voice is acquired from the at least one assistant module, convert the first response into a first instruction of a type corresponding to operation control of the electronic device; and control the operation of the electronic device on the basis of the converted first instruction.

Description

Electronic device for controlling multiple assistant modules and method for controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for controlling a plurality of assistant modules and a method for controlling the same.

[0002] Recently, artificial intelligence systems are being utilized in a variety of fields. Unlike existing rule-based smart systems, AI systems are machines that learn, make decisions, and become intelligent on their own. As AI systems become more used, their recognition rates improve and their ability to understand user preferences more accurately is increasing. As a result, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

[0003] In particular, recently, assistant modules (or voice assistant modules, voice assistant systems, artificial intelligence secretary modules, artificial intelligence secretary systems) that can provide responses to user voices or control electronic devices based on user voices are being developed using artificial intelligence systems.

[0004] These assistant modules may include various models, such as a voice recognition module for recognizing the user's voice, a natural language understanding module for understanding the user's voice, and a Text to Speech (TTS) module for outputting the generated response in the form of voice data. In this case, the multiple models included in the assistant module may be implemented in the form of neural network models, but are not limited thereto.

[0005] An electronic device according to one embodiment of the present disclosure includes a memory storing at least one command and a processor connected to the memory to control the electronic device, wherein the processor, by executing the at least one command, when a user voice is obtained, inputs the user voice to at least one assistant module among a plurality of assistant modules, and when a first response corresponding to the user voice is obtained from the at least one assistant module, converts the first response into a first command corresponding to operation control of the electronic device, and controls an operation of the electronic device based on the converted first command.

[0006] A method for controlling an electronic device according to an embodiment of the present disclosure includes: when a user voice is acquired, inputting the user voice into at least one assistant module among a plurality of assistant modules; when a first response corresponding to the user voice is acquired from the at least one assistant module, converting the first response into a first command corresponding to operation control of the electronic device; and controlling an operation of the electronic device based on the converted first command.

[0007] A non-transitory computer-readable recording medium including a program for executing a control method of an electronic device according to an embodiment of the present disclosure, the control method comprises: when a user voice is acquired, inputting the user voice into at least one assistant module among a plurality of assistant modules; when a first response corresponding to the user voice is acquired from the at least one assistant module, converting the first response into a first command corresponding to operation control of the electronic device; and controlling an operation of the electronic device based on the converted first command.

[0008] FIG. 1 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0009] FIG. 2 is a block diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure.

[0010] FIG. 3 is a diagram illustrating the operation of a plurality of modules according to one or more embodiments of the present disclosure.

[0011] FIG. 4 is a diagram illustrating a conventional multi-assistant system according to one or more embodiments of the present disclosure.

[0012] FIG. 5 is a flowchart illustrating the operation of an electronic device when rule information does not include information for converting a response corresponding to a user voice into a type of command capable of controlling the operation of the electronic device, according to one or more embodiments of the present disclosure.

[0013] FIG. 6 is a flowchart illustrating a method by which an electronic device according to one or more embodiments of the present disclosure converts a response for controlling a first operation that the electronic device cannot perform into a command for controlling a second operation that the electronic device can perform.

[0014] FIG. 7 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0015] The present embodiments may be modified and have various embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope to specific embodiments, but should be understood to encompass various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0016] In describing the present disclosure, if it is determined that a specific description of a related known function or configuration may unnecessarily obscure the gist of the present disclosure, a detailed description thereof will be omitted.

[0017] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concepts of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to further faithfully and completely convey the technical concepts of the present disclosure to those skilled in the art.

[0018] The terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the scope of the rights. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0019] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0020] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to (1) including at least one A, (2) including at least one B, or (3) including both at least one A and at least one B.

[0021] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0022] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that said component may be directly coupled to said other component, or may be coupled via another component (e.g., a third component).

[0023] On the other hand, when it is said that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it can be understood that no other component (e.g., a third component) exists between said component and said other component.

[0024] The expression "configured to" used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0025] Instead, in some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0026] In the embodiments, a 'module' or 'part' performs at least one function or operation, and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of 'modules' or 'parts' may be integrated into at least one module and implemented as at least one processor, except for a 'module' or 'part' that needs to be implemented as a specific hardware.

[0027] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0028] Hereinafter, with reference to the attached drawings, embodiments according to the present disclosure will be described in detail so that a person having ordinary knowledge in the technical field to which the present disclosure pertains can easily implement the present disclosure.

[0029] FIG. 1 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0030] The electronic device (100) may include a memory (110), a communication interface (120), a display (130), a user interface (140), an input / output interface (150), a speaker (160), a microphone (170), and a processor (180). The electronic device (100) may omit some of the above components and may further include other components.

[0031] And, the electronic device (100) may be implemented as a TV, but this is only one example, and the electronic device (100) may be implemented in various forms such as a smartphone, a smart TV, a set-top box, a mobile phone, a server, a PDA (personal digital assistant), a laptop, a media player, an e-book reader, a digital broadcasting terminal, a navigation device, a kiosk, an MP3 player, a wearable device, a home appliance, and other mobile or non-mobile computing devices.

[0032] The memory (110) can store at least one instruction regarding the electronic device (100). The memory (110) can store an operating system (O / S) for driving the electronic device (100). In addition, the memory (110) can store various software programs or applications for operating the electronic device (100) according to various embodiments of the present disclosure. In addition, the memory (110) can include a semiconductor memory such as a flash memory or a magnetic storage medium such as a hard disk.

[0033] Specifically, the memory (110) can store various software modules for operating the electronic device (100) according to various embodiments of the present disclosure, and the processor (180) can control the operation of the electronic device (100) by executing various software modules stored in the memory (110). That is, the memory (110) is accessed by the processor (180), and data reading / recording / modifying / deleting / updating, etc. can be performed by the processor (180).

[0034] Meanwhile, in the present disclosure, the term memory (110) may be used to mean a memory (110), a ROM (not shown), a RAM (not shown) in a processor (180), or a memory card (not shown) (e.g., a micro SD card, a memory stick) mounted on an electronic device (100).

[0035] And, the communication interface (120) includes a circuitry and is a configuration capable of communicating with external devices and servers. The communication interface (120) can communicate with external devices or servers based on a wired or wireless communication method. The communication interface (120) may include a Bluetooth module (not shown), a Wi-Fi module (not shown), an IR (infrared) module, a LAN (Local Area Network) module, an Ethernet module, etc. Here, each communication module may be implemented in the form of at least one hardware chip. In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as Zigbee, USB (Universal Serial Bus), MIPI CSI (Mobile Industry Processor Interface Camera Serial Interface), 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc. However, this is only one embodiment, and the communication interface (120) can utilize at least one communication module among various communication modules.

[0036] The display (130) can be implemented as a variety of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (130) may also include a driving circuit, a backlight unit, etc., which can be implemented as a form such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (130) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display (three-dimensional dispaly), etc.

[0037] According to various embodiments, the electronic device (100) may not include a display (130). The electronic device (100) may be connected to an external display device and may transmit images or content stored in the electronic device (100) to the external display device. Specifically, the electronic device (100) may transmit images or content to the external display device together with a control signal for controlling the display of the images or content on the external display device. Here, the external display device may be connected to the electronic device (100) through a communication interface (120) or an input / output interface (150). For example, the electronic device (100) may not include a display, such as a set-top box (STB). In addition, the electronic device (100) may include only a small display capable of displaying simple information such as text information. Here, the electronic device (100) may transmit images or content to the external display device through the communication interface (120) wired or wirelessly, or through the input / output interface (150).

[0038] The user interface (140) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operation input function. Here, the button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0039] The input / output interface (150) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (150) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (150) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. Meanwhile, the electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (150). Specifically, an output port included in the input / output interface (150) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0040] Here, the input / output interface (150) can be connected to the communication interface (120). The input / output interface (150) can transmit information received from an external device to the communication interface (120) or transmit information received through the communication interface (120) to the external device.

[0041] The speaker (160) is a component that outputs an audio signal. In particular, the speaker (160) may include an audio output mixer, an audio signal processor, and an audio output module. The audio output mixer may synthesize a plurality of audio signals to be output into at least one audio signal. For example, the audio output mixer may synthesize an analog audio signal and another analog audio signal (e.g., an analog audio signal received from the outside) into at least one analog audio signal. The audio output module may include a speaker or an output terminal. According to various embodiments, the audio output module may include a plurality of speakers, and in this case, the audio output module may be arranged inside the main body, and sound emitted by covering at least a portion of the diaphragm of the audio output module may pass through a waveguide and be transmitted to the outside of the main body. The audio output module may include a plurality of audio output units, and the plurality of audio output units may be arranged symmetrically on the exterior of the main body, thereby radiating sound in all directions, that is, in a 360-degree omnidirectional manner. In addition, the speaker (160) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0042] The microphone (170) is a component for receiving a user's voice or other sounds and converting them into audio data. The microphone (170) can receive the user's voice in an activated state. For example, the microphone (170) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (170) can include various components such as a microphone for collecting the user's voice in analog form, an amplifier circuit for amplifying the collected user's voice, an A / D conversion circuit for sampling the amplified user's voice and converting it into a digital signal, and a filter circuit for removing noise components from the converted digital signal.

[0043] And, the electronic device (100) can obtain the user's voice through the microphone (170) included in the electronic device (100). Alternatively, the electronic device (100) can obtain the user's voice from an external device equipped with a microphone. Specifically, the microphone can be provided in a separate external device, such as a remote control, a smartphone, a speaker, etc., that transmits a signal to the electronic device (100). Here, the microphone provided in the external device can digitize an analog voice signal, and the electronic device (100) can perform a communication connection with the external device through a communication interface (120) to receive the digitized voice signal.

[0044] At this time, a remote control application may be installed on the external device. The external device may control the electronic device (100) and perform user voice recognition functions through the remote control application. At this time, the external device may communicate with the electronic device (100) and the server through multiple communication interfaces.

[0045] And, the external device on which the remote control application is installed may be a smartphone, but this is only one example, and the application may be implemented through various devices on which the application can be installed, such as an AI speaker.

[0046] In addition, the electronic device (100) can be connected to an external device via a communication interface (120) or an input / output interface (150) wired or wirelessly, and the electronic device (100) can receive an audio signal for a user's voice from the external device.

[0047] The processor (180) can control the overall operation and function of the electronic device (100). Specifically, the processor (180) is connected to the configuration of the electronic device (100) including the memory (110), and can control the overall operation of the electronic device (100) by executing at least one command stored in the memory (110) as described above.

[0048] The processor (180) may be implemented in various ways. For example, the processor (180) may be implemented as at least one of an application-specific integrated circuit (ASIC), a logic integrated circuit (Logic Integrated Circuit), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor (DSP). Meanwhile, the term "processor (180" in the present disclosure may be used to mean a central processing unit (CPU), a graphic processing unit (GPU), and a main processing unit (MPU).

[0049] In particular, the processor (180) may include one or more processors. Specifically, the one or more processors may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in a memory.

[0050] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. That is, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0051] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute program instructions for implementing a method according to one or more embodiments of the present disclosure.

[0052] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0053] In embodiments of the present disclosure, the processor (180) may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto.

[0054] The operation of the processor (180) for implementing various embodiments of the present disclosure may be implemented through a plurality of modules.

[0055] Specifically, data for a plurality of modules according to the present disclosure can be stored in a memory (110), and the processor (180) can access the memory (110) to load the data for the plurality of modules into a memory or buffer within the processor (180), and then implement various embodiments according to the present disclosure using the plurality of modules.

[0056] However, at least one of the plurality of modules according to the present disclosure may be implemented in hardware and included in the processor (180) in the form of a system on chip.

[0057] Alternatively, at least one of the plurality of modules according to the present disclosure may be implemented as a separate external device, and the electronic device (100) and each module may communicate and perform operations according to the present disclosure.

[0058] FIG. 2 is a block diagram illustrating a plurality of modules according to one or more embodiments of the present disclosure.

[0059] Referring to FIG. 2, the memory (110) may include a voice acquisition module (111), a plurality of assistant modules, a common command conversion module (113), a rule information management module (114), and a control module (115).

[0060] The voice acquisition module (111) can acquire a user's voice through a microphone (170) or from an external device equipped with a microphone. In the present disclosure, the user's voice may refer to a user's voice input.

[0061] At this time, the acquired user voice may be data on which a voice signal including the user voice has been preprocessed, but this is only one example, and the voice acquisition module (111) may preprocess a voice signal including the user voice.

[0062] Specifically, the voice acquisition module (111) can acquire an audio signal containing a user's voice spoken by the user. For example, the user's voice may be "Brighten the screen" or "Play music through application A."

[0063] In addition, the voice acquisition module (111) can detect a user voice section from the acquired audio signal. In addition, the voice acquisition module (111) can remove noise from the detected user voice section using a noise removal filter.

[0064] The multiple assistant modules may be modules that provide responses to user voice or control electronic devices based on user voice.

[0065] Meanwhile, in the present disclosure, the assistant module may mean a voice assistant module (or voice assistant system) that provides a response to a user's voice based on the user's voice or controls an electronic device (100) or an external device, but is not limited thereto.

[0066] In addition, the assistant module (112) included in the plurality of assistant modules may include a voice recognition module (112-1) for recognizing a user's voice, a natural language understanding module (112-2) for understanding the user's voice, and a response generation module (112-3) for generating a response to the user's voice. In this case, the plurality of modules (112-1, 112-2, 112-3) included in the assistant module (112) may be implemented in the form of a neural network model, but is not limited thereto. In addition, the assistant module (112) may omit some of the above modules, and may further include other modules.

[0067] The speech recognition module (112-1) can convert user speech in the form of audio data into text data. At this time, the speech recognition module (112-1) can include an acoustic model and a language model. The acoustic model can include information related to pronunciation, and the language model can include information about unit phoneme information and a combination of unit phoneme information. The speech recognition module (112-1) can convert the user speech into text data using the information about the voice-related information and the unit phoneme information. Information about the acoustic model and the language model can be stored, for example, in an automatic speech recognition database (ASR DB).

[0068] The natural language understanding module (112-2) can perform syntactic analysis or semantic analysis based on text data regarding the user's voice acquired through speech recognition to identify the domain and user intent of the user's voice. In this case, syntactic analysis can divide the user input into grammatical units (e.g., words, phrases, morphemes, etc.) and determine what grammatical elements each of the divided units has. Semantic analysis can be performed using semantic matching, rule matching, formula matching, etc.

[0069] At this time, the natural language understanding module (112-2) can obtain a natural language understanding result, a category of the user voice, an intention of the user voice, and a slot (or entity, parameter, etc.) for performing the intention of the user voice.

[0070] The response generation module (112-3) can obtain a response to the user's voice based on the user intent and slot acquired by the natural language understanding module (112-2). At this time, the response generation module (112-3) can provide a response to the user's voice based on a knowledge database. While the knowledge database may be included within the electronic device (100), this is merely an example and may be included in an external server.

[0071] The operations of the above-described voice recognition module (112-1), natural language understanding module (112-2), and response generation module (112-3) may be performed within the electronic device (100), but this is merely an example, and the operations performed by some of the above-described modules may be performed on an external server. In addition, the assistant module (112) may receive information acquired from the user's voice from the external server.

[0072] Specifically, when a user voice is input, the assistant module (112) according to one or more embodiments of the present disclosure can transmit the input user voice to an external server.

[0073] An external server can perform voice recognition on a user's voice. Furthermore, the external server can transmit the user's voice, converted into text data, to the assistant module (112). Furthermore, the assistant module (112) can input the user's voice, converted into text data, into a natural language understanding module.

[0074] Alternatively, an external server may perform natural language understanding and response generation operations on user voice acquired through voice recognition, and obtain response information for the user voice. Furthermore, the external server may transmit the response to the user voice to the assistant module (112).

[0075] When a response to the user voice is obtained, the common command conversion module (113) can convert the response to the user voice into a type of command that can control the operation of the electronic device (100).

[0076] Specifically, the common command conversion module (113) can convert a first response to a user voice acquired by the first assistant module into a first command of a type capable of controlling the operation of the electronic device (100), and can convert a second response to a user voice acquired by the second assistant module into a second command of a type capable of controlling the operation of the electronic device (100).

[0077] Here, the type that can control the electronic device (100) may mean the format, type, and extension of the command used to control the operation of the electronic device (100).

[0078] Here, the format of a command can refer to the method or rules for expressing or storing the command. Specifically, the format of a command can refer to its structure, layout, encoding method, etc. For example, the format of a command can be HTML, JSON, XML, CSV, etc.

[0079] Additionally, the format of a command can indicate its structure or type. For example, the format of a command can be text, image, or audio.

[0080] In addition, the extension of a command may refer to an attribute indicating the format or form of the command. At this time, the extension of the command may be information included in the file name of the data containing the command. For example, if the file name of the command is instruction1.html, the extension of the command may be html. At this time, the type of command capable of controlling the operation of the electronic device (100) may be implemented in a specific format, a specific form, or a specific extension.

[0081] For example, the format of the response obtained by the assistant module (112) may be HTML or JSON (JavaScript Object Notation). In this case, the format of the command that can control the electronic device (100) may be HTML. If the format of the response obtained by the assistant module (112) is JSON, the common command conversion module (113) may convert the JSON format response into an HTML format command. However, if the format of the response obtained by the assistant module (112) is HTML, the process of converting the obtained response into a type of command that can control the electronic device (100) may be omitted. In this case, the HTML format response may include URL information for the electronic device (100) to perform an operation corresponding to the user's voice.

[0082] At this time, the control module (115) can control the operation of the electronic device (100) using the acquired response.

[0083] Alternatively, a command in a format that can control an electronic device (100) may mean a command that the electronic device (100) can recognize and execute.

[0084] For example, the user voice may be “Turn on application A.”, and the response to the user voice may be “Turn.on.app(A).” At this time, the command required for the electronic device (100) to turn on application A may be “Turn.On.Application.A.” At this time, the common command conversion module (113) may convert “Turn.on.app(A)” into “Turn.On.Application.A.”

[0085] Alternatively, the type of command that can be controlled by the electronic device (100) may refer to a command that can be recognized by the A operating system installed in the electronic device (100). At this time, the assistant module (112) may be a module that operates based on the B operating system. Accordingly, the response generated by the assistant module (112) may be a response (or command) that can be recognized by the B operating system. In this case, the common command conversion module (113) may convert a response that can be recognized by the B operating system into a command that can be recognized by the A operating system.

[0086] And, the common command conversion module (113) can convert a response to a user's voice into a type of command that can control the electronic device (100) based on rule information stored in the memory (110).

[0087] Specifically, the memory (110) can store rule information for converting at least one response obtained from a plurality of assistant modules into a type of command capable of controlling the operation of the electronic device (100).

[0088] Here, the rule information may include information matching at least one response obtained from a plurality of assistant modules with at least one command capable of controlling the operation of the electronic device (i.e., information matching and storing at least one command).

[0089] And, based on the rule information, the common command conversion module (113) can convert the first response into a first command of a type capable of controlling the operation of the electronic device.

[0090] At this time, the converted command may be a command that is matched and stored in the response in the rule information.

[0091] Meanwhile, rule information stored in the memory (110) can be updated by the rule information management module (114). Specifically, the rule information management module (114) can request updated rule information from an external server.

[0092] Additionally, the rule information management module (114) can receive information for updating rule information from an external server.

[0093] And, when updated rule information is received from an external server, the rule information management module (114) can update the rule information stored in the memory (110) with the acquired rule information.

[0094] At this time, rule information can be stored in JSON format, but is not limited to this.

[0095] When rule information is updated, the common command conversion module (113) can convert the response into a command that can control the operation of the electronic device based on the updated rule information.

[0096] The control module (115) can control the operation of the electronic device (100) based on the converted command obtained by the common command conversion module (113).

[0097] Specifically, the control module (115) can control the operation of some components of the electronic device (100). For example, the control module (115) can control the display (130) to display a response screen for a user's voice, or control the speaker (160) to output a response voice for a user's voice.

[0098] Alternatively, the control module (115) can execute or control an application installed in the electronic device (100). For example, if the user's voice is "Play music with application A," the control module (115) can execute application A installed in the electronic device (100) and control application A to play music.

[0099] The operation of the above-described multiple modules will be explained with reference to Fig. 3.

[0100] FIG. 3 is a diagram illustrating the operation of a plurality of modules according to one or more embodiments of the present disclosure.

[0101] Referring to FIG. 3, the user voice acquisition module (111) can acquire the user voice (1). Specifically, the user voice acquisition module (111) can acquire the user voice (1) through a microphone (170) or acquire the user voice (1) from an external device equipped with a microphone.

[0102] When a user voice (1) is acquired, the processor (180) can input the user voice (1) to at least one assistant module among a plurality of assistant modules. At this time, the input user voice (1) may be a user voice that has undergone preprocessing.

[0103] And, at least one assistant module can output a response corresponding to the user's voice (1).

[0104] Specifically, the voice recognition module (112a-1, 112b-1, 112c-1) can convert user voice into text data.

[0105] And, the natural language understanding module (112a-2, 112b-2, 112c-2) can obtain a natural language understanding result, a category of the user voice, an intention of the user voice, and a slot (or entity, parameter, etc.) for performing the intention of the user voice based on text data for the user voice.

[0106] And, the response module (112a-3, 112b-3, 112c-3) can obtain a response to the user's voice based on the user's intention and slot.

[0107] And, if the acquired response is of a type that can control the operation of the electronic device (100), the control module (115) can control the operation of the electronic device (100) based on the acquired response.

[0108] Alternatively, the type of response obtained by at least one assistant module may be a type that does not allow the processor (180) to control the operation of the electronic device (100).

[0109] At this time, in order to control the operation of the electronic device (100) using the response obtained by at least one assistant module, the obtained response needs to be converted into a type of command capable of controlling the operation of the electronic device (100).

[0110] In the past, to implement a multi-assistant environment, each assistant module included in an electronic device had to include a command conversion module to convert the obtained response into a type of command that can control the operation of the electronic device.

[0111] FIG. 4 is a drawing for explaining a method for implementing a multi-assistant environment in a conventional electronic device.

[0112] Referring to FIG. 4, a conventional electronic device (10) may include a plurality of assistant modules (12a, 12b, 12c). At this time, each of the plurality of assistant modules (12a, 12b, 12c) must include a command conversion module (12a-1, 12b-1, 12c-1) for converting responses provided from each of the plurality of assistant modules (12a, 12b, 12c) into a type of command capable of controlling the operation of the electronic device.

[0113] At this time, when at least one assistant module among the modules included in the plurality of assistant modules (12a, 12b, 12c) is updated, there was an inconvenience that the command conversion module included in at least one assistant module also had to be updated in order to operate the updated assistant module in the electronic device (10).

[0114] Meanwhile, the electronic device (100) according to the present disclosure can implement a multi-assistant environment using a common command conversion module (113). In this case, the plurality of assistant modules may exclude the function of converting a response corresponding to a user's voice into a type of command capable of controlling the operation of the electronic device (100).

[0115] Accordingly, since each of the plurality of assistant modules does not need to include a command conversion module, the electronic device (100) according to the present disclosure can minimize the amount of memory required to implement a multi-assistant environment.

[0116] And, when at least one assistant module among a plurality of assistant modules is updated, the operation of updating each of the plurality of command conversion modules to operate the updated assistant module in the electronic device (100) may be omitted.

[0117] Accordingly, the code maintenance efficiency required to update a multi-assistant environment can be improved.

[0118] Referring again to FIG. 3, when a response corresponding to a user voice (1) is obtained from at least one assistant module, the common command conversion module (113) can convert the obtained response into a type of command capable of controlling the operation of the electronic device (100).

[0119] At this time, the common command conversion module (113) can convert a response corresponding to the user's voice into a type of command that can control the operation of the electronic device (100) based on the rule information (116).

[0120] At this time, rule information (116) can be updated. At this time, the rule information management module (114) can obtain updated rule information from an external server (200) and update the pre-stored rule information (116).

[0121] Specifically, when at least one assistant module among a plurality of assistant modules is updated, the rule information management module (114) can request updated rule information including information for converting a response obtained from the updated assistant module into a type of command capable of controlling the operation of the electronic device (100) from the external server (200). In addition, the rule information management module (114) can obtain updated rule information including information for converting a response obtained from the updated assistant module into a type of command capable of controlling the operation of the electronic device (100) from the external server (200).

[0122] Alternatively, the rule information management module (114) may request updated rule information from an external server (200) at preset time intervals. Accordingly, the rule information management module (114) may obtain updated rule information from the external server (200).

[0123] When rule information (116) is updated by rule information obtained from an external server (200), the common command conversion module (113) can convert a response obtained from at least one assistant module into a type of command capable of controlling the operation of the electronic device (100) based on the updated rule information.

[0124] And, when the response corresponding to the user voice (1) is converted into a type of command capable of controlling the electronic device (100), the common command conversion module (113) can store information about the history of the response corresponding to the user voice (1) being converted into a type of command capable of controlling the electronic device (100) in the history information (117).

[0125] At this time, the common command conversion module (113) can match at least one of the user voice (1), the response corresponding to the user voice (1), and the information on the command converted from the response corresponding to the user voice (1) and store it in the history information (117).

[0126] When a response corresponding to a user voice (1) is converted into a type of command capable of controlling the operation of an electronic device (100), the control module (115) can control the operation of the electronic device (100) based on the converted command.

[0127] Meanwhile, according to one or more embodiments of the present disclosure, the rule information (116) may not include information for converting a response corresponding to the user voice (1) into a type of command capable of controlling the operation of the electronic device (100).

[0128] If the rule information (116) does not include information for converting a response corresponding to the user voice (1) into a type of command capable of controlling the operation of the electronic device (100), the common command conversion module (113) can process the user voice (1) based on the history information (117).

[0129] FIG. 5 is a flowchart illustrating the operation of an electronic device (100) when rule information (116) does not include information for converting a response corresponding to a user voice into a type of command capable of controlling the operation of the electronic device (100), according to one or more embodiments of the present disclosure.

[0130] Referring to FIG. 5, the processor (180) can input the first user voice into the first assistant module (S510).

[0131] And, the processor (180) can obtain a first response corresponding to the first user voice from the first assistant module (S520).

[0132] When the first response is obtained, the processor (180) can identify whether the rule information (116) includes information that can convert the obtained first response into a type of command that can control the operation of the electronic device (100) (S530).

[0133] If the rule information (116) includes information that can convert the first response into a type of command that can control the operation of the electronic device (100) (S530-Y), the processor (180) can convert the first response into a type of first command that can control the operation of the electronic device (100) based on the rule information (116) (S540).

[0134] And, based on the first command, the processor (180) can control the operation of the electronic device (100) (S550).

[0135] And, if the rule information (116) does not include information that can convert the first response into a type of command that can control the operation of the electronic device (100) (S530-N), the processor (180) can input the first user voice to the second assistant module based on the history of the second user voice corresponding to the first user voice being processed by the second assistant module (S560).

[0136] Here, the second user voice corresponding to the first user voice may be a voice whose value indicating similarity with the first user voice is greater than a preset value.

[0137] Specifically, the memory (110) can store history information (117) including a history of at least one user voice or a response obtained from at least one user voice being converted into a type of command capable of controlling the electronic device (100).

[0138] That is, the memory (110) can store history information (117) including a history of at least one user voice being converted into a response and a command through at least one assistant module among a plurality of assistant modules.

[0139] In addition, the processor (180) can identify a second user voice having a value indicating a similarity with the first user voice greater than or equal to a preset value among at least one user voice stored in the history information (117). In this case, if multiple voices having a value indicating a similarity with the first user voice greater than or equal to the preset value among at least one user voice are identified, the processor (180) can identify the second user voice having the highest value indicating a similarity with the first user voice among the multiple voices.

[0140] And, the processor (180) can process a response to the first user voice based on the history of the second user voice being processed included in the history information (117).

[0141] Specifically, the processor (180) can identify the second assistant module into which the second user voice is input based on the history of processing the second user voice. Here, the history of processing the second user voice may include information about the second assistant module into which the second user voice is input.

[0142] When the second assistant module into which the second user voice is input is identified, the processor (180) can input the first user voice into the second assistant module.

[0143] And, the processor (180) can convert the second response obtained from the second assistant module into a second command of a type that can control the operation of the electronic device (100) (S570).

[0144] When the second response is converted into a second command, the processor (180) can control the operation of the electronic device (100) based on the second command.

[0145] That is, if the rule information (116) does not include information for converting the first response obtained from at least one assistant module into the first command, the processor (180) may input the first user voice to another assistant module based on the history of processing a second user voice having a value indicating a similarity with the first user voice greater than a preset value from at least one assistant module and another assistant module.

[0146] Meanwhile, according to one or more embodiments of the present disclosure, the response obtained from at least one assistant module may be a response for controlling a first operation of the electronic device (100).

[0147] At this time, the first operation may be an operation that the electronic device (100) cannot perform. For example, the first operation may be an operation for controlling a configuration that is not included in the electronic device (100). Alternatively, the first operation may be an operation for controlling an application that is not installed in the electronic device (100).

[0148] At this time, the electronic device (100) can convert a response for controlling the first operation into a command for controlling the second operation that the electronic device (100) can perform.

[0149] FIG. 6 is a flowchart illustrating a method for converting a response for controlling a first operation that the electronic device (100) cannot perform into a command for controlling a second operation that the electronic device (100) can perform, according to one or more embodiments of the present disclosure.

[0150] Referring to FIG. 6, the processor (180) can obtain a first response for controlling a first operation of the electronic device (100) (S610).

[0151] And, the processor (180) can identify whether the electronic device (100) can perform the first operation (S620).

[0152] For example, if the first operation is an operation for controlling a first application, the processor (180) can identify whether the first application is installed in the memory (110). Alternatively, if the first operation is an operation for controlling a specific configuration, the processor (180) can identify whether the specific configuration is installed or connected to the electronic device (100).

[0153] If the electronic device (100) can perform the first operation (S620-Y), the processor (180) can convert the first response into a first command of a type capable of controlling the first operation of the electronic device (100) (S630).

[0154] And, based on the first command, the processor (180) can control the electronic device (100) to perform the first operation (S640).

[0155] And, if the electronic device (100) cannot perform the first operation (S620-N), the processor (180) can convert the first response into a second command of a type capable of controlling the second operation of the electronic device (100) (S650). At this time, the second operation may be an operation that the electronic device (100) can perform.

[0156] Specifically, if the electronic device (100) cannot perform the first operation, the processor (180) can identify a second operation corresponding to the first operation. The second operation corresponding to the first operation control may be an operation for performing at least a part of the functions corresponding to the first operation control. The function corresponding to the first operation control may refer to a function realized by the electronic device (100) performing the first operation. Here, information on the second operation corresponding to the first operation may be stored in the memory (110).

[0157] For example, if the first action is an action controlling a first application, the second action corresponding to the first action may be an action controlling a second application corresponding to the first application. In this case, the second application may be an application capable of performing at least some of the functions of the first application. For example, if the first application is a music playback application, the second application may also be a music playback application.

[0158] Alternatively, if the first action is an action for controlling a first component that is not included in the electronic device (100), the second action may be an action for controlling a second component that is included in the electronic device (100). In this case, the second component may be a component that can perform at least some of the functions of the first component. For example, the first action may be an action for controlling a display to display “specific text.” And, the second action may be an action for controlling a speaker to output “specific text.”

[0159] And, when a second command of a type capable of controlling a second operation of the electronic device (100) is obtained, the processor (180) can control the second operation of the electronic device (100) based on the second command (S660).

[0160] Meanwhile, the assistant module into which the acquired user voice is input in the electronic device (100) according to the present disclosure may vary depending on user settings or circumstances.

[0161] According to one or more embodiments of the present disclosure, the processor (180) may input a user voice acquired by a preset assistant module among a plurality of assistant modules. In this case, the preset assistant module may be an assistant module set by the user.

[0162] According to one or more embodiments of the present disclosure, the processor (180) may input the user voice to an assistant module among a plurality of assistant modules corresponding to the wake-up word (or trigger word, wake word) included in the user voice.

[0163] Here, the wake-up word may be a word for loading or executing an assistant module to perform an action corresponding to the user's voice in the electronic device (100). For example, if the user's voice is "Hi, Bixby. Play A music playback application," the wake-up word included in the user's voice may be "Hi, Bixby."

[0164] Specifically, the memory (110) may include at least one wake-up engine. At this time, the at least one wake-up engine may be, but is not limited to, a neural network model capable of identifying a wake-up word included in a user's voice. For example, the first wake-up engine may be a model trained to identify a first wake-up word. Furthermore, the second wake-up engine may be a model trained to identify a second wake-up word.

[0165] When a user voice is input to at least one wake-up engine, at least one wake-up engine can identify whether the user voice contains a wake-up word.

[0166] When a wake-up word included in a user's voice is identified by at least one wake-up engine, the user's voice acquisition module (111) can input the user's voice to an assistant module corresponding to the identified wake-up word. At this time, information about the assistant module corresponding to the wake-up word may be stored in the memory (110).

[0167] According to one or more embodiments of the present disclosure, the processor (180) may input the user's voice to at least one assistant module based on a preset priority. Information regarding the priority of each of the plurality of assistant modules may be stored in the memory (110).

[0168] Specifically, the processor (180) can input the user voice acquired by the first assistant module having the highest priority (e.g., the first priority) among the plurality of assistant modules.

[0169] And, if the rule information does not include information for converting the first response obtained from the first assistant module into a command in a format for controlling the operation of the electronic device (100), the processor (180) may input the obtained user voice to a second assistant module having a lower priority (e.g., second priority) than the first assistant module.

[0170] According to one or more embodiments of the present disclosure, the processor (180) can input the acquired user voice to each of the plurality of assistant modules.

[0171] At this time, some of the multiple assistant modules may not be able to provide a response to the user's voice.

[0172] In addition, the processor (180) can store information about at least one assistant module that provides a response corresponding to the acquired user voice among a plurality of assistant modules in the memory (110).

[0173] And, when an additional user voice is acquired, the processor (180) can input the additional user voice to at least one assistant module that provides a response corresponding to the user voice.

[0174] According to one or more embodiments of the present disclosure, the processor (180) can input the user's voice to each of the plurality of assistant modules.

[0175] Additionally, the processor (180) can control the display (130) to display responses obtained from each of the plurality of assistant modules.

[0176] Alternatively, the processor (180) may control the speaker (160) to output responses obtained from each of the plurality of assistant modules.

[0177] Additionally, the processor (180) can obtain user input to select one of the displayed or outputted responses.

[0178] When user input is obtained to select one of the responses, the processor (180) can convert the selected response into a type of command capable of controlling the operation of the electronic device (100).

[0179] And, based on the converted command, the processor (180) can control the operation of the electronic device (100).

[0180] According to one or more embodiments of the present disclosure, the processor (180) can input a user voice to at least one assistant module among a plurality of assistant modules.

[0181] In addition, the processor (180) can control the operation of the electronic device (100) by using a response obtained from the assistant module that provides the response corresponding to the user's voice most quickly among at least one assistant module.

[0182] Specifically, the processor (180) can convert a response obtained from the assistant module that provides the fastest response corresponding to the user's voice into a command in a format that can control the operation of the electronic device (100).

[0183] And, based on the converted command, the processor (180) can control the operation of the electronic device (100).

[0184] The time it takes for each of the multiple assistant modules to provide a response may vary depending on the input user voice. According to the method described above, in a multi-assistant environment, when a user voice is acquired from a user, the electronic device (100) can perform an action corresponding to the user voice using the assistant module that can most quickly process a response to the input user voice. Accordingly, in a multi-assistant environment, the time it takes the electronic device (100) to perform an action corresponding to the user voice can be minimized.

[0185] FIG. 7 is a drawing for explaining a control method of an electronic device (100) according to one or more embodiments of the present disclosure.

[0186] When the user voice is acquired, the electronic device (100) can input the user voice to at least one assistant module among a plurality of assistant modules (S710).

[0187] And, the electronic device (100) can identify whether a response corresponding to user voice control is obtained from at least one first assistant module (S720).

[0188] If a first response corresponding to user voice control is not obtained from at least one first assistant module (S720-N), the electronic device (100) may additionally obtain an additional user voice and input the additional user voice into at least one assistant module to obtain a response to the additional user voice (S730). In this case, the electronic device (100) may also input the obtained user voice and the additional user voice together into at least one assistant module to obtain a response to the additional user voice.

[0189] When a first response corresponding to a user voice is obtained from at least one assistant module (S720-Y) or a response to an additional user voice is obtained, the electronic device (100) can convert the obtained response into a first command corresponding to the operation control of the electronic device (100) (S740). The command corresponding to the operation control of the electronic device (100) may be a command of a type capable of controlling the operation of the electronic device (100). That is, the command corresponding to the operation control of the electronic device (100) may be a command of a type capable of controlling the electronic device (100). Specifically, the memory (110) may store rule information (117) for converting at least one response obtained from a plurality of assistant modules into a command of a type capable of controlling the operation of the electronic device. At this time, the electronic device (100) may convert the obtained response into a first command of a type capable of controlling the operation of the electronic device (100) based on the rule information (117).

[0190] That is, the rule information (117) may include information in which at least one response obtained from multiple assistant modules and at least one command capable of controlling the operation of an electronic device are matched and stored. In addition, the converted first command may be a command matched and stored in the rule information (117) with the first response.

[0191] At this time, rule information (117) can be stored in JSON format.

[0192] Meanwhile, if the rule information does not include information for converting the response obtained into a first command, the electronic device (100) may input the user voice (or additional user voice) to another assistant module based on a history of processing a user voice having a similarity with the user voice (or additional user voice) that is equal to or greater than a preset value by at least one assistant module among a plurality of assistant modules and another assistant module. Then, the electronic device (100) may convert the response obtained from the other assistant module into a second command of a type capable of controlling the operation of the electronic device. Meanwhile, when at least one assistant module is updated, the electronic device (100) may receive information for updating the rule information (117) from an external server. Then, when the rule information (117) is updated, the electronic device (100) may convert the response into a first command of a type capable of controlling the operation of the electronic device (100) based on the updated rule information.

[0193] Meanwhile, the acquired response may be information for controlling the first application.

[0194] And, if there is no first application in the electronic device (100), the electronic device (100) can perform the second application of the electronic device (100).

[0195] For example, if there is no information about the first application in the memory (110) of the electronic device (100), the acquired response can be converted into a second command of a type capable of controlling the second application. Here, the information about the first application may be information for executing the first application. In addition, information about the second application (e.g., information for executing the second application) may be stored in the memory (110).

[0196] Alternatively, if the first application is not installed on the electronic device (100), the electronic device (100) can convert the acquired response into a second command capable of controlling a second application installed on the electronic device. Meanwhile, the acquired response may be information for controlling the first operation. In addition, if the electronic device (100) cannot perform the first operation, the electronic device (100) can convert the acquired response into a second command capable of controlling a second operation that the electronic device can perform.

[0197] Meanwhile, multiple assistant modules may lack the ability to convert the acquired response into a first command.

[0198] The electronic device (100) can input a user's voice to each of a plurality of assistant modules. In addition, the electronic device (100) can control the display (130) to display responses from each of the plurality of assistant modules.

[0199] Based on the converted first command, the electronic device (100) can control the operation of the electronic device (100) (S750).

[0200] Meanwhile, the terms "part" or "module" used in the present disclosure include units composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be an integrally composed component, a minimum unit performing one or more functions, or a portion thereof. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0201] Various embodiments of the present disclosure may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device, which is a device capable of calling instructions stored in the storage medium and operating according to the called instructions, may include an electronic device (100) according to the disclosed embodiments. When the instructions are executed by a processor, the processor may directly or under the control of the processor perform a function corresponding to the instructions using other components. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means that the storage medium does not contain signals and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0202] According to one or more embodiments, the methods according to the various embodiments disclosed herein may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0203] Each component (e.g., a module or a program) according to various embodiments may be composed of one or more entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., a module or a program) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by a module, program, or other component according to various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

Claims

1. In electronic devices, a memory storing at least one instruction; and A processor connected to said memory and controlling said electronic device, The processor, by executing at least one instruction, When a user voice is acquired, the user voice is input to at least one assistant module among a plurality of assistant modules, When a first response corresponding to the user voice is obtained from at least one assistant module, the first response is converted into a first command corresponding to the operation control of the electronic device, An electronic device that controls the operation of the electronic device based on the converted first command.

2. In paragraph 1, The command corresponding to the operation control of the electronic device is a type of command capable of controlling the operation of the electronic device, The above memory is, Store rule information for converting at least one response obtained from the plurality of assistant modules into a type of command capable of controlling the operation of the electronic device, The above processor, An electronic device that converts the first response into a first command of a type capable of controlling an operation of the electronic device based on the rule information.

3. In paragraph 2, The above rule information is, Contains information matching at least one response obtained from the plurality of assistant modules and at least one command capable of controlling the operation of the electronic device, An electronic device wherein the converted first command is a command that matches the first response in the rule information.

4. In paragraph 2, The above processor, If the rule information does not include information for converting the first response into the first command, the user voice having a similarity with the user voice greater than a preset value is input to the other assistant module based on the history of processing from at least one assistant module among the plurality of assistant modules and the other assistant module, An electronic device that converts a second response obtained from said other assistant module into a second command of a type capable of controlling the operation of said electronic device.

5. In paragraph 2, The above processor, When at least one of the above assistant modules is updated, information for updating the rule information is received from an external server, An electronic device that, when the above rule information is updated, converts the response into a first command of a type capable of controlling the operation of the electronic device based on the updated rule information.

6. In paragraph 2, The above rule information is stored in JSON format in an electronic device.

7. In paragraph 1, The above first response is information for controlling the first application, The above processor, An electronic device, wherein if the electronic device does not have a first application, the electronic device converts the first response into a second command of a type capable of controlling a second application of the electronic device.

8. In paragraph 1, The above first response is information for controlling the first operation, The above processor, An electronic device, wherein if the electronic device is unable to perform the first operation, the electronic device converts the first response into a second command capable of controlling a second operation that the electronic device is capable of performing.

9. In paragraph 1, The above multiple assistant modules are: An electronic device, excluding the function of converting the first response into a first command.

10. In paragraph 1, The above electronic device, display; including more; The above processor, Input the user voice to each of the plurality of assistant modules, An electronic device that controls the display to display responses from each of the plurality of assistant modules.

11. In a method for controlling an electronic device, When a user voice is acquired, a step of inputting the user voice to at least one assistant module among a plurality of assistant modules; When a first response corresponding to the user voice is obtained from at least one assistant module, a step of converting the first response into a first command corresponding to the operation control of the electronic device; and A control method comprising: a step of controlling the operation of the electronic device based on the converted first command.

12. In paragraph 11, The command corresponding to the operation control of the electronic device is a type of command capable of controlling the operation of the electronic device, The above electronic device, Store rule information for converting at least one response obtained from the plurality of assistant modules into a type of command capable of controlling the operation of the electronic device, The step of converting to the above first command is: A control method for converting the first response into a first command of a type capable of controlling the operation of the electronic device based on the above rule information.

13. In paragraph 12, The above rule information is, Contains information matching at least one response obtained from the plurality of assistant modules and at least one command capable of controlling the operation of the electronic device, A control method wherein the converted first command is a command that matches the first response in the rule information.

14. In paragraph 12, The above control method is, If the rule information does not include information for converting the first response into the first command, a step of inputting the user voice to the other assistant module based on a history of processing of a user voice having a similarity with the user voice greater than a preset value from at least one assistant module among the plurality of assistant modules; and A control method further comprising: a step of converting a second response obtained from said other assistant module into a second command of a type capable of controlling an operation of said electronic device.

15. A non-transitory computer-readable recording medium including a program for executing a method of controlling an electronic device, The above control method is, When a user voice is acquired, a step of inputting the user voice to at least one assistant module among a plurality of assistant modules; When a first response corresponding to the user voice is obtained from at least one assistant module, a step of converting the first response into a first command corresponding to the operation control of the electronic device; and A non-transitory computer-readable recording medium comprising: a step of controlling the operation of the electronic device based on the converted first command.

Images