Electronic device, method, and computer-readable storage medium for performing voice recognition in multi-device environment

Abstract

This electronic device may comprise: a memory for storing instructions; a communication circuit; and at least one processor. The instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: obtain signals for a voice including a wake-up word during a time interval from each of external electronic devices capable of recognizing the wake-up word; select first signals obtained from a first external electronic device by using some of the signals of each of the external electronic devices; transmit first voice information generated from some of the first signals of the first external electronic device; select second signals obtained from a second external electronic device by using the signals of each of the external electronic devices on the basis of the time interval which is greater than or equal to a reference length; and transmit second voice information generated from the second signals of the second external electronic device.

Description

Electronic device, method, and computer-readable storage medium for performing speech recognition in a multi-device environment

[0001] The following descriptions relate to electronic devices, methods, and computer-readable storage media for performing speech recognition in a multi-device environment.

[0002] An electronic device can acquire an external signal through a microphone. Alternatively, the electronic device can acquire the signal through an external electronic device. For example, the signal may include speech or voice uttered by a speaker or information about the voice. For example, the electronic device can recognize the speaker based on the signal.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] An electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include a communication circuit. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire signals for speech including a wake-up word from each of external electronic devices capable of recognizing the wake-up word during a time interval. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to select first signals acquired from a first external electronic device among the external electronic devices, using some of the signals from each of the external electronic devices, while the signals from each of the external electronic devices are acquired during the time interval. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit, to each of the external electronic devices, first voice information generated from some of the first signals of the first external electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to select, based on the time interval that is greater than or equal to a reference length, second signals obtained from a second external electronic device among the external electronic devices using the signals of each of the external electronic devices.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit, to each of the external electronic devices, second voice information generated from the second signals of the second external electronic device. The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0005] A non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by at least one processor of an electronic device including a communication circuit, cause the electronic device to obtain signals for speech including a wake-up word from each of external electronic devices capable of recognizing the wake-up word during a time interval. The non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to select, using some of the signals from each of the external electronic devices, first signals obtained from a first external electronic device among the external electronic devices while the signals from each of the external electronic devices are obtained during the time interval. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the first voice information generated from some of the first signals of the first external electronic device to be transmitted to each of the external electronic devices. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the second signals obtained from a second external electronic device among the external electronic devices to be selected using the signals of each of the external electronic devices based on the time interval that is greater than or equal to a reference length.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the second voice information generated from the second signals of the second external electronic devices to be transmitted to each of the external electronic devices. The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0006] A method performed by an electronic device may include acquiring signals for a voice including a wake-up word from each of external electronic devices capable of recognizing the wake-up word during a time interval. The method may include selecting first signals acquired from a first external electronic device among the external electronic devices using some of the signals of each of the external electronic devices while the signals of each of the external electronic devices are acquired during the time interval. The method may include transmitting first voice information generated from some of the first signals of the first external electronic device to each of the external electronic devices. The method may include selecting second signals acquired from a second external electronic device among the external electronic devices using the signals of each of the external electronic devices based on the time interval that is greater than or equal to a reference length. The method may include transmitting second voice information generated from the second signals of the second external electronic device to each of the external electronic devices. The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0007] An electronic device may include a memory for storing instructions and including one or more storage media. The electronic device may include a communication circuit. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain signals for voice including a wake-up word from each of external electronic devices capable of recognizing the wake-up word. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit, to each of the external electronic devices, first voice information generated from a portion of the first signals based on selecting first signals obtained from a first external electronic device among the external electronic devices using a portion of the signals from each of the external electronic devices. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to select the first signals and then, based on selecting second signals received from a second external electronic device among the external electronic devices using the signals of each of the external electronic devices, transmit second voice information generated from the second signals to each of the external electronic devices. The first voice information may cause at least some of the external electronic devices to display a visual object representing a portion of the voice recognized from the portion of the first signals.The second voice information may cause at least some of the external electronic devices to display a visual object representing the voice recognized from the second signals.

[0008] A method performed by an electronic device may include an operation in which the electronic device acquires signals for a voice including a wake-up word from each of external electronic devices capable of recognizing the wake-up word. The method may include an operation in which, based on selecting first signals acquired from a first external electronic device among the external electronic devices using some of the signals of each of the external electronic devices, first voice information generated from some of the first signals is transmitted to each of the external electronic devices. The method may include an operation in which, after selecting the first signals, second signals received from a second external electronic device among the external electronic devices using the signals of each of the external electronic devices are selected, second voice information generated from the second signals is transmitted to each of the external electronic devices. The first voice information may cause at least some of the external electronic devices to display a visual object representing a portion of the voice recognized from the portion of the first signals. The second voice information may cause at least some of the external electronic devices to display a visual object representing the voice recognized from the second signals.

[0009] A non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by at least one processor of an electronic device including a communication circuit, cause the electronic device to obtain signals for voice including the wake-up word from each of external electronic devices capable of recognizing the wake-up word. The non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to transmit, to each of the external electronic devices, first voice information generated from a portion of the first signals based on selecting first signals obtained from a first external electronic device among the external electronic devices using a portion of the signals from each of the external electronic devices. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to select the first signals and, based on selecting second signals received from a second external electronic device among the external electronic devices using the signals of each of the external electronic devices, transmit second voice information generated from the second signals to each of the external electronic devices. The first voice information may cause at least some of the external electronic devices to display a visual object representing a portion of the voice recognized from the portion of the first signals. The second voice information may cause at least some of the external electronic devices to display a visual object representing the voice recognized from the second signals.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0011] FIG. 2 is a block diagram illustrating an integrated intelligence system according to various embodiments.

[0012] FIG. 3 is a diagram showing the form in which relationship information between concepts and operations according to various embodiments is stored in a database.

[0013] FIG. 4 is a diagram illustrating a screen in which an electronic device according to various embodiments processes voice input received through an intelligent app.

[0014] Figure 5 illustrates an example of a method for performing speech recognition in a multiple device environment.

[0015] FIG. 6A illustrates an example of an electronic device that provides a voice recognition service using signals obtained from an external electronic device.

[0016] Figure 6b illustrates an example of a method for calculating scores for each external electronic device according to its distance from the speaker.

[0017] Figure 7 illustrates an example of a method for utilizing multiple automatic speech recognition (ASR) devices for external electronic devices in a multi-device environment.

[0018] FIGS. 8A to 8C illustrate examples of a method for performing voice recognition using a signal provided from an external electronic device selected from among external electronic devices in a multi-device environment.

[0019] Figures 9a to 9d illustrate examples of how external electronic devices within a multi-device environment display the results of voice recognition.

[0020] FIGS. 10A and 10B illustrate examples of a method for performing voice recognition according to the mode of external electronic devices in a multi-device environment.

[0021] FIG. 11 illustrates an example of an operational flow for a method of performing voice recognition using a signal provided from an external electronic device selected from among external electronic devices in a multi-device environment.

[0022] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0023] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0024] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0025] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0026] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0027] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] FIG. 2 is a block diagram illustrating an integrated intelligence system according to various embodiments.

[0049] Referring to FIG. 2, an integrated intelligent system of one embodiment may include an electronic device (101), an intelligent server (200) (e.g., server (108) of FIG. 1), and a service server (290) (e.g., server (108) of FIG. 1).

[0050] The electronic device (101) of one embodiment may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0051] According to the illustrated embodiment, the electronic device (101) may include an interface (177), an input module (150), an audio output module (155), a display module (160), a memory (130), or a processor (120). The above-listed components may be operatively or electrically connected to each other.

[0052] An interface (177) of one embodiment may be configured to connect to an external device and transmit and receive data. An input module (150) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. An audio output module (155) of one embodiment may output the electrical signal as sound (e.g., voice).

[0053] The display module (160) of one embodiment may be configured to display an image or video. The display module (160) of one embodiment may also display a graphical user interface (GUI) of a running app (or application program). The display module (160) of one embodiment may receive touch input via a touch sensor. For example, the display module (160) may receive text input via a touch sensor in an on-screen keyboard area displayed within the display module (160).

[0054] In one embodiment, the memory (130) may store a client module (151), a software development kit (SDK) (153), and a plurality of apps (146). The client module (151) and the SDK (153) may constitute a framework (or solution program) for performing general functions. In addition, the client module (151) or the SDK (153) may constitute a framework for processing user input (e.g., voice input, text input, touch input).

[0055] The plurality of apps (146) stored in the memory (130) of one embodiment may be programs for performing a specified function. According to one embodiment, the plurality of apps (146) may include a first app (146-1) and a second app (146-2). According to one embodiment, each of the plurality of apps (146) may include a plurality of operations for performing a specified function. For example, the apps may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (146) may be executed by the processor (120) to sequentially execute at least some of the plurality of operations.

[0056] The processor (120) of one embodiment can control the overall operation of the electronic device (101). For example, the processor (120) can be electrically connected to an interface (177), an input module (150), an audio output module (155), and a display module (160) to perform a specified operation.

[0057] The processor (120) of one embodiment may also execute a program stored in the memory (130) to perform a designated function. For example, the processor (120) may execute at least one of the client module (151) or the SDK (153) to perform the following operations for processing user input. The processor (120) may control the operations of multiple apps (146), for example, through the SDK (153). The following operations described as operations of the client module (151) or the SDK (153) may be operations executed by the processor (120).

[0058] The client module (151) of one embodiment can receive user input. For example, the client module (151) can receive a voice signal corresponding to a user utterance detected through the input module (150). Alternatively, the client module (151) can receive a touch input detected through the display module (160). Alternatively, the client module (151) can receive a text input detected through a keyboard or a visual keyboard. In addition, the client module (151) can receive various types of user input detected through an input module included in the electronic device (101) or an input module connected to the electronic device (101). The client module (151) can transmit the received user input to the intelligent server (200). The client module (151) can transmit status information of the electronic device (101) to the intelligent server (200) together with the received user input. The status information can be, for example, execution status information of an app.

[0059] The client module (151) of one embodiment can receive a result corresponding to the received user input. For example, the client module (151) can receive a result corresponding to the received voice input when the intelligent server (200) can produce a result corresponding to the received user input. The client module (151) can display the received result on the display module (160). In addition, the client module (151) can output the received result as audio through the audio output module (155).

[0060] In one embodiment, the client module (151) can receive a plan corresponding to the received user input. The client module (151) can display the results of executing multiple operations of the app according to the plan on the display module (160). For example, the client module (151) can sequentially display the results of executing multiple operations on the display and output audio through the audio output module (155). The electronic device (101) can, for example, display only some results of executing multiple operations (e.g., the result of the last operation) on the display module (160) and output audio through the audio output module (155).

[0061] According to one embodiment, the client module (151) may receive a request from the intelligent server (200) to obtain information necessary to produce a result corresponding to a user input. According to one embodiment, the client module (151) may transmit the necessary information to the intelligent server (200) in response to the request.

[0062] The client module (151) of one embodiment can transmit result information of executing multiple operations according to a plan to the intelligent server (200). The intelligent server (200) can use the result information to confirm that the received user input has been processed correctly.

[0063] The client module (151) of one embodiment may include a voice recognition module. According to one embodiment, the client module (151) may recognize voice inputs that perform limited functions through the voice recognition module. For example, the client module (151) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0064] An intelligent server (200) of one embodiment can receive information related to a user voice input from an electronic device (101) via a communication network. According to one embodiment, the intelligent server (200) can change data related to the received voice input into text data. According to one embodiment, the intelligent server (200) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0065] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the above or another AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0066] An intelligent server (200) of one embodiment may transmit a result according to a generated plan to an electronic device (101), or transmit the generated plan to the electronic device (101). According to one embodiment, the electronic device (101) may display a result according to the plan on a display module (160). According to one embodiment, the electronic device (101) may display a result of executing an operation according to the plan on a display module (160).

[0067] An intelligent server (200) of one embodiment may include a front end (210), a natural language platform (220), a capsule database (230), an execution engine (240), an end user interface (250), a management platform (260), a big data platform (270), or an analytic platform (280).

[0068] The front end (210) of one embodiment can receive user input from the electronic device (101). The front end (210) can transmit a response corresponding to the user input.

[0069] According to one embodiment, the natural language platform (220) may include an automatic speech recognition module (ASR module) (221), a natural language understanding module (NLU module) (223), a planner module (225), a natural language generator module (NLG module) (227), or a text to speech module (TTS module) (229).

[0070] The automatic speech recognition module (221) of one embodiment can convert voice input received from the electronic device (101) into text data. The natural language understanding module (223) of one embodiment can identify the user's intention using the text data of the voice input. For example, the natural language understanding module (223) can identify the user's intention by performing syntactic analysis or semantic analysis on user input in the form of text data. The natural language understanding module (223) of one embodiment can identify the meaning of words extracted from the user input using linguistic features (e.g., grammatical elements) of morphemes or phrases, and can determine the user's intention by matching the meaning of the identified words to the intent. The natural language understanding module (223) can obtain intent information corresponding to the user's utterance. The intent information may be information indicating the user's intent determined by interpreting text data. The intent information may include information indicating an action or function that the user intends to execute using the device.

[0071] The planner module (225) of one embodiment can generate a plan using the intent and parameters determined by the natural language understanding module (223). According to one embodiment, the planner module (225) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (225) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (225) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts of a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (225) can determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (225) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (225) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (225) can generate a plan including association information (e.g., ontology) between a plurality of actions and a plurality of concepts. The planner module (225) can generate the plan using information stored in a capsule database (230) in which a set of relationships between concepts and actions is stored.

[0072] The natural language generation module (227) of one embodiment can convert specified information into text format. The information converted into text format may be in the form of natural language utterances. The text-to-speech conversion module (229) of one embodiment can convert information in text format into information in speech format.

[0073] According to one embodiment, some or all of the functions of the natural language platform (220) may also be implemented in the electronic device (101).

[0074] The capsule database (230) may store information on the relationships between a plurality of concepts and actions corresponding to a plurality of domains. According to one embodiment, a capsule may include a plurality of action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (230) may store a plurality of capsules in the form of a concept action network (CAN). According to one embodiment, the plurality of capsules may be stored in a function registry included in the capsule database (230).

[0075] The capsule database (230) may include a strategy registry that stores strategy information required when determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a user input. According to one embodiment, the capsule database (230) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (230) may include a layout registry that stores layout information of information output through the electronic device (101). According to one embodiment, the capsule database (230) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (230) may include a dialog registry that stores information on dialogue (or interaction) with a user. The capsule database (230) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans. The developer tool may include a dialog editor for creating dialogues with the user.The developer tool may include a follow-up editor that activates follow-up goals and allows editing of follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions. In one embodiment, the capsule database (230) may also be implemented within the electronic device (101).

[0076] The execution engine (240) of one embodiment can produce a result using the generated plan. The end user interface (250) can transmit the produced result to the electronic device (101). Accordingly, the electronic device (101) can receive the result and provide the received result to the user. The management platform (260) of one embodiment can manage information used in the intelligent server (200). The big data platform (270) of one embodiment can collect user data. The analysis platform (280) of one embodiment can manage the quality of service (QoS) of the intelligent server (200). For example, the analysis platform (280) can manage the components and processing speed (or efficiency) of the intelligent server (200).

[0077] A service server (290) of one embodiment may include CP service A (291), CP service B (292), and CP service C (293). The service server (290) of one embodiment may provide a service (e.g., food ordering or hotel reservation) specified for the electronic device (101). According to one embodiment, the service server (290) may be a server operated by a third party. The service server (290) of one embodiment may provide information for generating a plan corresponding to the received user input to the intelligent server (200). The provided information may be stored in the capsule database (230). In addition, the service server (290) may provide result information according to the plan to the intelligent server (200).

[0078] In the integrated intelligence system of FIG. 2, the electronic device (101) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0079] In one embodiment, the electronic device (101) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the electronic device (101) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0080] In one embodiment, the electronic device (101) may perform a designated operation, either alone or in conjunction with the intelligent server and / or service server, based on the received voice input. For example, the electronic device (101) may execute an app corresponding to the received voice input and perform a designated operation through the executed app.

[0081] In one embodiment, when an electronic device (101) provides a service together with an intelligent server (200) and / or a service server (290), the electronic device (101) can detect a user's speech using the input module (150) and generate a signal (or voice data) corresponding to the detected user's speech. The electronic device (101) can transmit the voice data to the intelligent server (200) using the interface (177).

[0082] According to one embodiment, an intelligent server (200) may generate a plan for performing a task corresponding to a voice input received from an electronic device (101), or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to a user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0083] An electronic device (101) of one embodiment can receive the response using an interface (177). The electronic device (101) can output a voice signal generated within the electronic device (101) to the outside using the audio output module (155), or can output an image generated within the electronic device (101) to the outside using the display module (160).

[0084] FIG. 3 is a diagram showing the form in which relationship information between concepts and operations according to various embodiments is stored in a database.

[0085] The capsule database (e.g., the capsule database (230) of FIG. 2) of the intelligent server (e.g., the intelligent server (200) of FIG. 2) may store capsules in the form of a CAN (concept action network). The capsule database may store operations for processing tasks corresponding to a user's voice input and parameters necessary for the operations in the form of a CAN (concept action network).

[0086] The capsule database may store a plurality of capsules (capsule (A) (301), capsule (B) (304)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., capsule (A) (301)) may correspond to one domain (e.g., location (geo), application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (302), CP 2 (303), CP 3 (306), or CP 4 (305)) for performing a function for a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (310) and at least one concept (320) for performing a specified function.

[0087] The natural language platform (e.g., the natural language platform (220) of FIG. 2) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module of the natural language platform (e.g., the planner module (225) of FIG. 2) can generate a plan using capsules stored in a capsule database. For example, a plan (307) can be generated using actions (301-1, 301-3) and concepts (301-2, 301-4) of capsule A (301) and actions (304-1) and concepts (304-2) of capsule B (304).

[0088] FIG. 4 is a diagram illustrating a screen in which an electronic device according to various embodiments processes voice input received through an intelligent app.

[0089] The electronic device (101) can execute an intelligent app to process user input through an intelligent server (e.g., the intelligent server (200) of FIG. 2).

[0090] According to one embodiment, on the screen (410), when the electronic device (101) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the electronic device (101) may execute an intelligent app for processing the voice input. For example, the electronic device (101) may execute an intelligent app while the schedule app is running. According to one embodiment, the electronic device (101) may display an object (e.g., an icon) (411) corresponding to the intelligent app on a display module (e.g., the display module (160) of FIG. 1). According to one embodiment, the electronic device (101) may receive a voice input by a user's speech. For example, the electronic device (101) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the electronic device (101) can display a user interface (UI) (413) (e.g., an input window) of an intelligent app displaying text data of received voice input on a display module (e.g., the display module (160) of FIG. 1).

[0091] According to one embodiment, on the screen (420), the electronic device (101) may display a result corresponding to the received voice input on a display module (e.g., the display module (160) of FIG. 1). For example, the electronic device (101) may receive a plan corresponding to the received user input and display 'this week's schedule' on the display module (160) according to the plan.

[0092] Figure 5 illustrates an example of a method for performing speech recognition in a multiple device environment.

[0093] The multi-device environment may refer to an environment including external electronic devices capable of recognizing a wake-up word. For example, the multi-device environment may be referred to as a multi-device wake-up word (MDW) environment. For example, the multi-device environment may refer to an environment including external electronic devices within a specific space, external electronic devices having the same account information, or external electronic devices connected to a specific access point (AP). For example, the wake-up word may include a designated word or sentence used to designate a specific electronic device. The wake-up word may be used to execute a voice recognition function of the designated specific electronic device. For example, the wake-up word may be preset or set by a user.

[0094] Referring to FIG. 5, examples (500, 550) of a method in which an electronic device (101) is connected to external electronic devices (510) and the electronic device (101) performs voice recognition based on signals obtained from the external electronic devices (510) are illustrated. For example, the electronic device (101) may represent an example of the electronic device (101) of FIG. 1. For example, the electronic device (101) may be connected to the external electronic devices (510) and may include a server that manages the external electronic devices (510). However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may include a module or hub connected to the external electronic devices (510). In other words, the electronic device (101) may be implemented through hardware, software, or hardware and software.

[0095] For example, the external electronic devices (510) may represent electronic devices that can be set or recognized by the same call word. For example, the external electronic devices (510) may include external electronic device (511), external electronic device (512), external electronic device (513), and external electronic device (514). However, the embodiments of the present disclosure are not limited thereto. For example, the external electronic devices (510) may include three or fewer or five or more electronic devices.

[0096] Referring to examples (500, 550), the user (505) may utter a speech including the wake word. For example, the user (505) may utter the speech, "Wake word, send a text to Jane." While uttering the speech, the user (505) may move from the external electronic device (512) toward the external electronic device (514). However, the present disclosure is not limited thereto. For example, the user (505) may also utter the speech while stationary. For example, the speech may be divided into a plurality of chunks. For example, the plurality of chunks for the speech may include "Wake word, to Jane," "text," and "send." Although the above examples describe an example in which one chunk consists of one or more words, the present disclosure is not limited thereto. For example, a single syllable excluding the above-mentioned call word may be composed of a single chunk. For convenience of explanation, an example is described below where a single chunk is a single word excluding the above-mentioned call word.

[0097] Referring to examples (500, 550), the electronic device (101) may acquire signals from each of the external electronic devices (510) while the user (505) speaks the voice. For example, the electronic device (101) may acquire signals from each of the external electronic devices (510) during a time interval. For example, the length of the time interval may correspond to the length of time during which the voice is spoken. For example, the signals acquired from each of the external electronic devices (510) may be acquired periodically within the time interval. However, the embodiments of the present disclosure are not limited thereto. For example, the signals acquired from each of the external electronic devices (510) may also be acquired aperiodically within the time interval. For example, a signal obtained from each of the external electronic devices (510) in a first period within the time interval may include a portion of the speech (e.g., "Wake up word, Jane"), and a signal obtained from each of the external electronic devices (510) in a second period following the first period within the time interval may include another portion of the speech (e.g., "text"). In other words, while the user (505) speaks the speech, each of the external electronic devices (510) may obtain and process the speech in real time, and then periodically transmit a signal including at least a portion of the processed speech to the electronic device (101). Accordingly, even if the electronic device (101) periodically obtains the signals from each of the external electronic devices (510), the timing at which the signals are received by each of the external electronic devices (510) may not be consistent with each other.

[0098] For example, the electronic device (101) may receive first signals from an external electronic device (511) during the time interval. The electronic device (101) may receive second signals from an external electronic device (512) during the time interval. The electronic device (101) may receive third signals from an external electronic device (513) during the time interval. The electronic device (101) may receive fourth signals from an external electronic device (514). For example, the first signal received in a first period within the time interval may include a portion of the speech (e.g., "call word, Jane"). For example, the first signal received in a second period following the first period within the time interval may include another portion of the speech (e.g., "text"). For example, the second signal received in the first period within the time interval may include a portion of the speech (e.g., "Dear Jane"). For example, a second signal received in the second period within the time interval may include another portion of the speech (e.g., a “letter”).

[0099] For example, the signals (e.g., the first signals, the second signals, the third signals, and the fourth signals) obtained from each of the external electronic devices (510) during the time period may include the voice (or information about the voice) spoken by the user (505) in real time. In addition, for example, the signals (e.g., the first signals, the second signals, the third signals, and the fourth signals) obtained from each of the external electronic devices (510) may include a recognition result for the voice generated by each of the external electronic devices (510). For example, the recognition result may be included in a part of the signals. For example, the first recognition result recognized by the external electronic device (511) may be included in a part of the first signals (e.g., the first signal of the first period). For example, the second recognition result recognized by the external electronic device (512) may be included in a portion of the second signals (e.g., the second signal of the first cycle). For example, the third recognition result recognized by the external electronic device (513) may be included in a portion of the third signals (e.g., the third signal of the first cycle). For example, the fourth recognition result recognized by the external electronic device (514) may be included in a portion of the fourth signals (e.g., the fourth signal of the first cycle).

[0100] For example, the recognition result for the voice may include speech power. The speech power may be referred to as a sound power level or sound pressure level. For example, each of the external electronic devices (510) may identify the level of the voice spoken by the user (505). The volume or sound pressure level may be used to identify (or recognize) the electronic device that is located closest to the user (505) who spoke the signal (or voice signal) for the voice acquired through the microphone of each of the external electronic devices (510).

[0101] According to one embodiment, the electronic device (101) may perform voice recognition using the signals acquired from each of the external electronic devices (510). For example, the voice recognition may include identifying (or recognizing) the voice spoken by the user (505), generating a result of identifying the voice, and performing a designated function using the generated result (e.g., displaying text representing the voice using the result or instructing (or causing) the external electronic device to display text representing the voice using the result). For example, the electronic device (101) may select ASR (automatic speech recognition) to perform the voice recognition using the signals acquired from each of the external electronic devices (510). The ASR may be referred to as a module or service that performs the voice recognition. For example, the ASR may be implemented through hardware, software, or hardware and software. For example, the electronic device (101) may select the ASR for performing the voice recognition and allocate resources for the ASR. For example, the resources may include resources (e.g., memory, CPU resources, traffic) used by the electronic device (101) to perform the voice recognition through the ASR.

[0102] According to one embodiment, the electronic device (101) may select signals to be used for the voice recognition. For example, the electronic device (101) may select signals to be used for the voice recognition from among signals of external electronic devices (510) (e.g., the first signals, the second signals, the third signals, and the fourth signals). For example, the electronic device (101) selecting signals to be used for the voice recognition may refer to selecting (or recognizing) an external electronic device that provides signals to be used for the voice recognition from among the external electronic devices (510).

[0103] According to one embodiment, the electronic device (101) may select an external electronic device to be used for the voice recognition among the external electronic devices (510) by using a portion of the signals acquired from each of the external electronic devices (510) while the signals of each of the external electronic devices (510) are periodically acquired in the time interval. For example, the signals acquired from each of the external electronic devices (510) may correspond to the voice spoken by the user (505), and a portion of the signals (e.g., the signal of the first period) may correspond to a portion of the voice (e.g., "Wake up, Jane").

[0104] Referring to example (500), the electronic device (101) may select an external electronic device (511) from among external electronic devices (510) using a portion of the first signals, a portion of the second signals, a portion of the third signals, and a portion of the fourth signals. For example, the portion of the first signals may represent a signal in a section including the fastest timing (or the first signal of the first period) of the time section among the first signals acquired during the time section. For example, the portion of the second signals may represent a signal in a section including the fastest timing (or the second signal of the first period) of the time section among the second signals acquired during the time section. For example, the portion of the third signals may represent a signal in a section including the fastest timing (or the third signal of the first period) of the time section among the third signals acquired during the time section. For example, the part of the fourth signals may represent a signal of an interval including the fastest timing of the time interval (or the fourth signal of the first period) among the fourth signals acquired during the time interval.

[0105] According to one embodiment, the electronic device (101) may select the first signals acquired from the external electronic device (511) as signals to be used for the voice recognition based on the timing at which the signals are acquired from each of the external electronic devices (510). For example, the electronic device (101) may recognize the timing at which the signals are acquired from each of the external electronic devices (510). For example, the electronic device (101) may recognize the first timing at which the part of the first signals (e.g., the first signal of the first period) is acquired from the external electronic device (511). For example, the electronic device (101) may recognize the second timing at which the part of the second signals (e.g., the second signal of the first period) is acquired from the external electronic device (512). For example, the electronic device (101) can recognize a third timing at which the part of the third signals (e.g., the third signal of the first cycle) is acquired from the external electronic device (513). For example, the electronic device (101) can recognize a fourth timing at which the part of the fourth signals (e.g., the fourth signal of the first cycle) is acquired from the external electronic device (514). For example, the electronic device (101) can identify a first timing (or an earliest timing) among the first timing, the second timing, the third timing, and the fourth timing. For example, the electronic device (101) can identify the first timing among the first timing, the second timing, the third timing, and the fourth timing. In example (500), the electronic device (101) can select the external electronic device (511) among the external electronic devices (510) based on the identified first timing.

[0106] According to one embodiment, the electronic device (101) may select the first signals acquired from the external electronic device (511) as signals to be used for the voice recognition based on the recognition results of the signals from each of the external electronic devices (510). For example, the electronic device (101) may acquire a recognition result included in a part of the signals from each of the external electronic devices (510). For example, the electronic device (101) may acquire a first recognition result included in the part of the first signals (e.g., the first signal of the first period) from the external electronic device (511). For example, the electronic device (101) may acquire a second recognition result included in the part of the second signals (e.g., the second signal of the first period) from the external electronic device (512). For example, the electronic device (101) can obtain a third recognition result included in the part of the third signals (e.g., the third signal of the first period) from the external electronic device (513). For example, the electronic device (101) can obtain a fourth recognition result included in the part of the fourth signals (e.g., the fourth signal of the first period) from the external electronic device (514). For example, the electronic device (101) can identify a recognition result having a maximum value among the first recognition result, the second recognition result, the third recognition result, and the fourth recognition result. For example, the first recognition result can include a value calculated from feature values ​​for the voice recognized by the external electronic device (511). For example, the second recognition result can include a value calculated from feature values ​​for the voice recognized by the external electronic device (512). For example, the third recognition result may include a value calculated from feature values ​​for the voice recognized by the external electronic device (513).For example, the fourth recognition result may include a value calculated from feature values ​​for the voice recognized by the external electronic device (514). For example, the electronic device (101) may identify the first recognition result among the first recognition result, the second recognition result, the third recognition result, and the fourth recognition result. In example (500), the electronic device (101) may select the external electronic device (511) among the external electronic devices (510) based on the identified first recognition result.

[0107] According to one embodiment, the electronic device (101) may generate first voice information from a portion (e.g., a first signal of the first period) of the first signals acquired from the selected external electronic device (511). For example, the electronic device (101) may generate the first voice information from the portion of the first signals using ASR for the external electronic device (511). For example, the first voice information may include text representing the portion of the voice (e.g., “Call word, to Jane”) corresponding to the portion of the first signals. For example, the text may include “Call word, to Jane” excluding the call word from the portion of the voice. However, the embodiments of the present disclosure are not limited thereto. For example, the text may include “Call word, to Jane.”

[0108] According to one embodiment, the electronic device (101) may transmit the generated first voice information to each of the external electronic devices (510). At this time, the first voice information may cause at least some of the external electronic devices (510) to display a visual object representing a portion of the voice (e.g., "Call word, Jane"). Specific details related thereto are described below in FIGS. 9A to 9D .

[0109] According to one embodiment, the electronic device (101) may determine whether the time interval is greater than or equal to a reference length. For example, the electronic device (101) may determine whether the time interval during which the signals are acquired from each of the external electronic devices (510) is greater than or equal to the reference length. For example, the electronic device (101) may store the signals periodically acquired from each of the external electronic devices (510). For example, the electronic device (101) may determine whether the length of time (or the time interval) during which the periodically acquired signals are stored (or accumulated) is greater than or equal to the reference length. For example, if the time interval is less than the reference length, the electronic device (101) may perform voice recognition using another portion of the signals acquired from the selected external electronic device. For example, if the time interval is greater than or equal to the reference length, the electronic device (101) may perform voice recognition using all of the signals acquired from the selected external electronic device.

[0110] Referring to example (500), the electronic device (101) may perform the voice recognition using another part of the first signals obtained from the external electronic device (511) selected based on the timing or the recognition result when the time interval is less than the reference length. For example, the electronic device (101) may perform the voice recognition using another part of the first signals. In this case, the other part of the first signals (e.g., the first signal of the first period and the first signal of the second period) may include the part of the first signals (e.g., the first signal of the first period). In other words, the length of the other part of the first signals may be longer than the length of the part of the first signals.

[0111] Referring to example (550), the electronic device (101) may select an external electronic device (514) from among the external electronic devices (510) by using the signals obtained from each of the external electronic devices (510) when the time interval is longer than the reference length. For example, the electronic device (101) may perform the voice recognition by using all of the fourth signals obtained from the external electronic device (514).

[0112] According to one embodiment, the electronic device (101) can obtain values ​​for selecting an external electronic device (514) from among the external electronic devices (510). For example, the electronic device (101) can obtain a first value indicating a distance from each of the external electronic devices (510) to the user (505) and a second value indicating the quality of the voice. For example, the electronic device (101) can obtain the first value and the second value of each of the external electronic devices (510). In other words, the electronic device (101) can obtain the first value and the second value of the external electronic device (511), the first value and the second value of the external electronic device (512), the first value and the second value of the external electronic device (513), and the first value and the second value of the external electronic device (514). For example, the electronic device (101) can generate selection parameters for the external electronic devices (510) from the set of the first value and the second value of each of the external electronic devices (510). For example, each of the selection parameters can be calculated based on the first value, the second value, and a weight value. For example, the weight value can be determined according to the environment in which the voice is spoken. For example, the electronic device (101) can select the external electronic device (514) based on the selection parameter having the maximum value among the selection parameters.

[0113] As described above, the electronic device (101) may select an external electronic device (511) using some of the signals in example (500), and may select an external electronic device (514) using the signals in example (550). However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may select an external electronic device (511) using some of the signals, and may also select an external electronic device (511) using the signals. In other words, after selecting an external electronic device (511) based on the timing at which the signals are acquired or the recognition result, the electronic device (101) may re-select (or maintain the selection) an external electronic device (511) based on the quality of the signals.

[0114] According to one embodiment, the electronic device (101) may generate second voice information from the fourth signals acquired from the selected external electronic device (514). For example, the electronic device (101) may generate the second voice information from the fourth signals using ASR for the external electronic device (514). For example, the second voice information may include text representing the voice corresponding to the fourth signals (e.g., "Wake word, send a text to Jane"). For example, the text may include "To Jane" excluding the wake word from the voice. However, the embodiments of the present disclosure are not limited thereto. For example, the text may include "Wake word, send a text to Jane."

[0115] According to one embodiment, the electronic device (101) may transmit the generated second voice information to each of the external electronic devices (510). At this time, the second voice information may cause at least some of the external electronic devices (510) to display a visual object representing the voice (e.g., "Wake me up, send a text to Jane"). Specific details related to this are described below in FIGS. 9A to 9D .

[0116] Referring to FIG. 5, the electronic device (101) may select a temporary winner (e.g., the external electronic device (511)) using a portion of the signals (e.g., the signals of the first period) obtained from each of the external electronic devices (510), and may generate first voice information from a portion (e.g., the first signal of the first period) of the signals (e.g., the first signals) of the selected temporary winner. Hereinafter, in the present disclosure, the temporary winner may be referred to as the first external electronic device. Thereafter, the electronic device (101) may select a final winner (e.g., the external electronic device (514)) using the signals obtained from each of the external electronic devices (510), and may generate second voice information from all of the signals (e.g., the fourth signals) of the selected final winner. Hereinafter, in the present disclosure, the final winner may be referred to as the second external electronic device. The temporary winner and the final winner may represent electronic devices that provide signals that the electronic device (101) can use to perform the voice recognition. The temporary winner and the final winner may be referred to as winner devices.

[0117] As described above, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure can perform voice recognition more effectively by selecting a single winner device for voice recognition in the MDW environment. At this time, regardless of whether electronic devices are connected to a single AP, electronic devices are connected to multiple APs, or electronic devices have (or use) the same account, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure can select an optimal winner device in the MDW environment. Selecting the winner device may include performing a ranking on external electronic devices. The ranking may indicate determining an order among the external electronic devices. Accordingly, the electronic device, method, and computer-readable storage medium according to embodiments of the present disclosure can provide a seamless voice recognition service and obtain a voice recognition result (or ASR recognition result) with relatively high accuracy by utilizing double selection (e.g., selection of a temporary winner and selection of a final winner) in selecting a winner device.

[0118] As described above, voice recognition using dual selection according to embodiments of the present disclosure can be performed using signals acquired from a winner device selected by using the distance between each of the external electronic devices (510) and the user (505), unlike deep learning-based sound source localization that accurately finds the location or direction from which the sound source is generated. In addition, voice recognition using dual selection according to embodiments of the present disclosure can be performed using information about distance, quality, and environment, unlike a method that uses signals from nearby devices. In addition, voice recognition using dual selection according to embodiments of the present disclosure can be used for signals (or channels) transmitted from different types of electronic devices, unlike a method that selects a winner device through a neural network structure that uses signals (or channels) from external electronic devices as input, and can be performed using signals acquired from a winner device selected by using the distance between each of the external electronic devices (510) and the user (505). Additionally, voice recognition using dual selection according to embodiments of the present disclosure can be performed without additional procedures for generating the gain, unlike a method of selecting a winner device by comparing the corrected sound level using a distance-dependent gain.

[0119] FIG. 6A illustrates an example of an electronic device that provides a voice recognition service using signals obtained from an external electronic device.

[0120] The electronic device (101) of FIG. 6A may represent an example of the electronic device (101) of FIG. 5. For example, the electronic device (101) of FIG. 6A may include at least a portion of the electronic device (101) of FIG. 1. For example, the electronic device (101) may be referred to as a server that manages an external electronic device (601). For example, the external electronic device (601) of FIG. 6A may represent an example of at least a portion of the external electronic devices (510) of FIG. 5. For example, the external electronic device (601) may include devices such as a smartphone, a tablet PC, a TV, a robot vacuum cleaner, an air conditioner, and a refrigerator.

[0121] Referring to FIG. 6A, an example (600) of an electronic device (101) that provides the voice recognition service using signals acquired from an external electronic device (601) is illustrated. Although the example (600) illustrates an electronic device (101) that uses signals acquired from one external electronic device (601), this is merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. For example, as in the examples (500, 550) of FIG. 5, the electronic device (101) can establish connections with external electronic devices and acquire signals from each of the external electronic devices through the established connections. The electronic device (101) can provide the voice recognition service using signals acquired from each of the external electronic devices. For example, the voice recognition service may indicate that the electronic device (101) recognizes a voice represented by signals and provides a designated function based on the recognized voice.

[0122] Referring to example (600), a user (505) may utter a voice. For example, the voice may include a wake word. Alternatively, for example, the voice may include the wake word and a command. For example, the command may include a word or sentence for executing or requesting a mapped function, unlike the wake word for indicating a specific device. For example, the command may include "Send a text to Jane." The voice uttered by the user (505) may be input to an external electronic device (601). For example, the external electronic device (601) may obtain the voice through a microphone of the external electronic device (601). For example, the external electronic device (601) may obtain a voice signal including the voice using the microphone. The voice signal may include the voice, noise, or background sound.

[0123] According to one embodiment, the external electronic device (601) may perform feature extraction for the voice signal. For example, the external electronic device (601) may extract a plurality of feature values ​​for the voice signal. For example, before performing feature extraction for the voice signal, the external electronic device (601) may enhance the voice feature of the voice signal. For example, the operation of enhancing the voice feature may include at least one of the operations of removing noise from the voice signal, emphasizing the voice portion of the voice signal, and normalizing the volume of the voice portion. After enhancing the feature for the voice, the external electronic device (601) may obtain the plurality of feature values ​​for the voice signal. For example, the external electronic device (601) may obtain the plurality of feature values ​​based on a mel frequency cepstral coefficient (MFCC) conversion algorithm. The external electronic device (601) can obtain a spectrum by applying a fast-Fourier transform (FFT) for each frame to the voice signal. The external electronic device (601) can obtain the spectrum for the frequency domain by applying the FFT to the voice signal. The external electronic device (601) can obtain a mel spectrum by applying a mel filter bank to the spectrum. The external electronic device (601) can obtain the mel spectrum based on a mel scale that represents the relationship between the frequency domain and a low frequency band recognized by an actual human. The external electronic device (601) can obtain MFCCs by applying a cepstral analysis to the mel spectrum.The above MFCCs may be referred to as the feature values. For example, the external electronic device (601) may obtain the feature values, which are a portion of the total feature values, which are peaks obtained based on the cepstral analysis.

[0124] According to one embodiment, the external electronic device (601) may include a call word engine (605). For example, the external electronic device (601) may recognize the call word of the voice signal through the call word engine (605). For example, the external electronic device (601) may recognize the call word through the call word engine (605) based on the plurality of feature values. For example, the call word engine (605) may be implemented by hardware, software, or a combination of hardware and software for recognizing the call word. For example, the external electronic device (601) may identify the call word included in the voice signal using the call word engine (605). For example, the call word engine (605) may include a statistical model trained to identify (or recognize) the call word. For example, statistical models may include artificial neural network models, hidden Markov models (HMMs), Gaussian mixture models (GMMs), support vector machines (SVMs), and vector quantizers (VQs).

[0125] According to one embodiment, the external electronic device (601) may generate a recognition result while recognizing the call word through the call word engine (605). For example, the recognition result may include speech power, loudness, and sound pressure level. For example, the loudness may represent the sound power level of the voice signal (or call word). For example, the loudness may be referred to as speech power, loudness, or sound pressure level.

[0126] According to one embodiment, the electronic device (101) may include a frontend (FE) (611), an MDW manager (613), an MDW device selector (615), and an ASR (617). However, the embodiments of the present disclosure are not limited thereto.

[0127] According to one embodiment, the electronic device (101) may obtain the signals from the external electronic device (601) through the FE (611). For example, the FE (611) may represent a module of the electronic device (101) that first arrives at the electronic device (101) when the external electronic device (601) requests the voice recognition service. For example, the FE (611) may be used to distribute the signals to the MDW manager (613) and the ASR (617). However, the embodiments of the present disclosure are not limited thereto. Although not illustrated in FIG. 6A, the FE (611) may be used to distribute the signals to a natural language understanding (NLU) module for natural language processing or a text to speech (TTS) module for voice synthesis of the electronic device (101).

[0128] According to one embodiment, when triggered by a wake word, the electronic device (101) may, through the MDW manager (613), select an ASR (617) for the electronic device that most accurately recognized a voice including the wake word or the electronic device that most quickly transmitted a voice including the wake word. For example, the ASR (617) may be referred to as a speech recognition model. For example, the MDW manager (613) may select an ASR (617) associated with an external electronic device (601) that provided the signals, and request the ASR (617) to perform speech recognition on the signals. For example, the MDW manager (613) may store each of the signals acquired from the external electronic device (601). For example, the electronic device (101) may provide the stored signals to the MDW device selector (615) when the stored (or accumulated) signals are longer than a reference length. For example, the reference length may be set by the user, determined based on the environment in which the signals are provided, or based on the quality of the signals. As described above, the temporary winner may be selected through the MDW manager (613) of the electronic device (101).

[0129] According to one embodiment, the electronic device (101) may select an ASR (617) for speech recognition through the MDW device selector (615). For example, the MDW device selector (615) may include a trained machine learning model (ML model) to select an external electronic device (601) to perform speech recognition from signals acquired from each of the external electronic devices during the time period longer than the reference length, and to select an ASR (617) for the external electronic device (601). For example, the machine learning model may be used to generate at least one of a first value indicating a distance from the user (505) who uttered the speech to the external electronic device (601) and a second value indicating a quality of the speech. For example, the electronic device (101) may generate the first value and the second value through the MDW device selector (615). For example, the MDW device selector (615) can perform normalization on the feature values ​​of the signals. For example, the normalization can include cepstral mean and variance normalization (CMVN). For example, the MDW device selector (615) can generate the first value representing the relative distance between the user (505) and the external electronic device (601) using the normalized feature values. Furthermore, for example, the MDW device selector (615) can generate the second value representing the quality of the voice using the normalized feature values. For example, the electronic device (101) can select the ASR (617) using the first value and the second value generated by the MDW device selector (615). However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) can also select the ASR (617) using the first value, the second value, and the weight value.For example, the weight value may be determined based on the environment in which the voice is spoken. An example of a method for selecting an ASR (617) (or selecting an external electronic device (601) among external electronic devices) using the first value, the second value, and the weight value is described in detail in FIG. 6B below. Referring to the above, the final winner may be selected through the MDW device selector (615) of the electronic device (101).

[0130] According to one embodiment, the electronic device (101) can perform the speech recognition through the ASR (617). For example, the ASR (617) can be used to recognize text using the signals. For example, the ASR (617) can recognize the text corresponding to the speech included in the signals. Alternatively, for example, the ASR (617) can further use the type of the external electronic device (601) that provided the signals, the identifier of the external electronic device (601) that provided the signals, and the identifier of the user (505). Alternatively, for example, the ASR (617) can recognize the text from the speech using the user's personal information (or PLM (personal language model) information). For example, the user's personal information may include an app installed in the external electronic device (601), a contact list of the user (505) stored in the external electronic device (601), and a list of external electronic devices including the external electronic device (601). The user's personal information may be used to improve accuracy in recognizing the text from the voice. For example, the ASR (617) may use the personal information corresponding to the user (505) who uttered the voice, regardless of the external electronic device (601) providing the signals. In other words, even if the ASR (617) is selected according to the electronic device (e.g., the external electronic device (601)), the personal information within the ASR (617) may be used according to the user (505) rather than the electronic device.

[0131] Although not illustrated in FIG. 6A, the electronic device (101) may further include a PLM. For example, the PLM may represent a language model learned to distinguish and utilize the personal information. As described above, the PLM may be learned and utilized for a specific user (e.g., user (505)). In this case, the PLM may be utilized for at least one external electronic device. The at least one external electronic device may be utilized by the specific user.

[0132] According to one embodiment, the electronic device (101) can obtain signals for the voice from the external electronic device (601). For example, the electronic device (101) can obtain the signals for the voice spoken by the user (505) over a time period. For example, the time period may correspond to the length of time during which the user (505) speaks the voice. In other words, the electronic device (101) can obtain the signals for the voice in real time. For example, if the voice is "Wake word, send a text to Jane," the electronic device (101) can obtain signals representing each of a plurality of chunks divided into "Wake word," "To Jane," "Text," and "Send." In the above example, an example is described in which one chunk is one or more words, but the embodiments of the present disclosure are not limited thereto. For example, one syllable excluding the wake word may be composed of one chunk. For convenience of explanation, an example is described below where a chunk is a single word excluding the call word. For example, each of the signals can be acquired at each specified period.

[0133] According to one embodiment, the electronic device (101) can determine whether the time interval during which the signals of each of the external electronic devices, including the external electronic device (601), are acquired is longer than the reference length.

[0134] For example, if the time interval is less than the reference length, the electronic device (101) may select a temporary winner by using a portion of the signals of each of the external electronic devices while the signals of each of the external electronic devices are acquired during the time interval. For example, the temporary winner may be selected based on the timing at which the portion of the signals of each of the external electronic devices is acquired or a recognition result included in the portion.

[0135] For example, the electronic device (101) may select a final winner using the signals of each of the external electronic devices when the time interval is greater than or equal to the reference length. For example, the electronic device (101) may obtain the first value and the second value of each of the external electronic devices using the signals of each of the external electronic devices based on the time interval that is greater than or equal to the reference length. For example, the electronic device (101) may generate selection parameters for the external electronic devices from the first value and the second value of each of the external electronic devices. For example, the electronic device (101) may select the final winner based on a selection parameter having a maximum value among the selection parameters. Each of the selection parameters may be calculated based on the first value, the second value, and a weight value. The weight value may be determined based on the environment in which the voice is spoken.

[0136] According to one embodiment, when the electronic device (101) selects a winner device (e.g., the temporary winner or the final winner), the electronic device (101) may select an ASR for the winner device. For example, the electronic device (101) may include a plurality of ASRs, and the plurality of ASRs may correspond to the external electronic devices. For example, the electronic device (101) may select an ASR corresponding to the selected winner device. The electronic device (101) may perform voice recognition using the selected ASR. For example, the electronic device (101) may generate first voice information from some of the signals acquired from the temporary winner using the ASR selected according to the temporary winner. Alternatively, for example, the electronic device (101) may generate second voice information from signals acquired from the final winner using the ASR selected according to the final winner. According to one embodiment, the electronic device (101) may allocate resources to utilize the selected ASR. For example, the resources may include resources (e.g., memory, CPU resources, traffic) used by the electronic device (101) to perform the voice recognition through the selected ASR.

[0137] According to one embodiment, the electronic device (101) may provide voice information generated using ASR to each of the external electronic devices. For example, the electronic device (101) may transmit the first voice information related to the provisional winner to each of the external electronic devices. For example, the electronic device (101) may transmit the second voice information related to the final winner to each of the external electronic devices. For example, the first voice information may include a portion of the voice. For example, the second voice information may include the voice. Specific details of the electronic device (101) that performs voice recognition using double selection as described above may be referenced with reference to FIGS. 8A to 8C below.

[0138] Figure 6b illustrates an example of a method for calculating scores for each external electronic device according to its distance from the speaker.

[0139] FIG. 6B illustrates an example (650) of a model for calculating a score according to a distance from a speaker (e.g., a user (505) of FIG. 5) of an electronic device (101) to each of external electronic devices (e.g., external electronic devices (510) of FIG. 5). For example, the model may be referred to as a distance score calculation model. For example, the electronic device (101) may include the model of example (650). For example, the score according to the distance may be referred to as a value representing a distance from each of the external electronic devices to the speaker, a first value, a distance score, or a distance value.

[0140] Referring to example (650), the electronic device (101) can use signals obtained from each of the external electronic devices as inputs (660) of the model. For example, the signals obtained from each of the external electronic devices can represent signals for voices spoken by the speaker. For example, the electronic device (101) can use first signals (660-1) obtained from external electronic device #1, second signals (660-2) obtained from external electronic device #2, third signals (660-3) obtained from external electronic device #3, and Nth signals (660-N) obtained from external electronic device #N as inputs of the model. For example, the inputs (660) can include first signals (660-1), second signals (660-2), third signals (660-3), and Nth signals (660-N). For example, each of the first signals (660-1), the second signals (660-2), the third signals (660-3), and the Nth signals (660-N) may be provided to the electronic device (101) during a time interval from a corresponding external electronic device.

[0141] According to one embodiment, the electronic device (101) may input inputs (660) into the encoder (670) of the model. For example, the encoder (670) may perform processing on each of the inputs (660). At this time, a pooling layer may be applied to the time axis of the result tensor of the encoder (670). For example, the pooling layer may apply an attentive pooling scheme. By using the pooling layer, the length limitation for each of the inputs (660) may be removed. For example, since the parameters configuring the encoder (670) for the inputs (660) are shared, the size of the model may be maintained even if the number of inputs (660) changes. For example, the encoder (670) may extract an embedding vector. For example, the encoder (670) can extract the embedding vector by combining feature values ​​(e.g., local features) extracted from 1D depth-wise separable convolutions and context (e.g., global context) extracted from the squeeze and excitation layer.

[0142] For example, the electronic device (101) can obtain outputs (670-1, 670-2, 670-3 to 670-N) of the encoder (670). For example, the electronic device (101) can generate a first output (670-1) from the first signals (660-1), a second output (670-2) from the second signals (660-2), a third output (670-3) from the third signals (660-3), and an Nth output (670-N) from the Nth signals (660-N) using the encoder (670). For example, the electronic device (101) can generate a sequence (680) by combining outputs (670-1, 670-2, 670-3 to 670-N), and classify the sequence (680) according to the number of external electronic devices, thereby generating an output (690) of the model. For example, the output (690) of the model can include distance scores and ranking information. For example, the distance scores of the output (690) can include values ​​for the external electronic devices. For example, when the external electronic devices are three external electronic devices, the distance scores of the output (690) for the three external electronic devices can be [0.5 0.3 0.2]. For example, the distance scores can be 0.5, 0.3, and 0.2 in order of proximity to the speaker. Referring to the example, the sum of the distance scores can be 1. In other words, the distance score may be a value between 0 and 1. In this case, the ranking information may indicate an order according to the distance score. For example, the rank of an external electronic device with a distance score of 0.5 may be 1, the rank of an external electronic device with a distance score of 0.3 may be 2, and the rank of an external electronic device with a distance score of 0.2 may be 3.

[0143] Although not illustrated in FIG. 6B, according to one embodiment, the electronic device (101) may calculate a quality score representing the quality of the voice. For example, the quality score may be referred to as a value representing the quality of the voice calculated by each of the external electronic devices, a second value, a quality score, or a quality value. For example, the quality may include a signal to noise ratio (SNR), a perceptual evaluation of speech quality (PESQ), and an envelope variance (EV). For example, the electronic device (101) may calculate the quality (e.g., SNR) and normalize it based on a preset maximum value for the quality, thereby calculating the quality score. As in the above example, if the external electronic devices include three external electronic devices, and the calculated SNR for each of the three external electronic devices is [2.4 1.3 3.1] and the maximum value is 10.0, the quality score may be [0.24 0.13 0.31].

[0144] Additionally, although not illustrated in FIG. 6B, according to one embodiment, the electronic device (101) may generate selection parameters. The electronic device (101) may calculate the selection parameters for the external electronic devices. For example, each of the selection parameters may be calculated based on the following mathematical equation.

[0145]

[0146] The above mathematical formula 1 is merely an example to aid understanding and is not limited thereto, and can be modified, applied, or expanded in various ways.

[0147] For example, the above selection parameter is a selection parameter, the above distance score is the distance score (or the first value), the qualityscore may represent the quality score (or the second value), and α may represent a weight value.

[0148] Referring to the above mathematical formula, the electronic device (101) can calculate a selection parameter for each of the external electronic devices based on the first value and the second value of each of the external electronic devices. For example, the weight value (α) can be set so that the accuracy of ASR is maximized. Alternatively, for example, the weight value (α) can be determined based on the environment in which the voice is spoken. In other words, the weight value (α) can be determined based on various factors.

[0149] Referring to the above, the electronic device (101) may calculate the first value and the second value using signals obtained from each of the external electronic devices through an MDW device selector (e.g., the MDW device selector (615) of FIG. 6A), and may calculate the selection parameter based on the calculated first value and the second value. For example, the electronic device (101) may determine (or select) one of the external electronic devices as the final winner using the selection parameters for the external electronic devices.

[0150] Figure 7 illustrates an example of a method for utilizing multiple automatic speech recognition (ASR) devices for external electronic devices in a multi-device environment.

[0151] Referring to FIG. 7, an example (700) of a method for an electronic device (701) to perform speech recognition using multiple ASRs (721, 722, 723) in an MDW environment is illustrated. For example, the electronic device (701) may be connected to multiple external electronic devices (511, 512, 513). For example, the electronic device (701) may obtain signals from each of the external electronic devices (511, 512, 513). For example, the electronic device (701) may receive a first signal from the external electronic device (511) through FE1 (711). For example, the electronic device (701) may receive a second signal from the external electronic device (512) through FE2 (712). For example, the electronic device (701) can receive a third signal from an external electronic device (513) via FE3 (713). At this time, each of the first signal, the second signal, and the third signal can represent a signal for a voice spoken by a speaker.

[0152] For example, the electronic device (701) can perform voice recognition on the voice using ASR1 (721) for the first signal. At this time, the electronic device (701) can allocate a first resource for ASR1 (721). In addition, for example, the electronic device (701) can perform voice recognition on the voice using ASR2 (722) for the second signal. At this time, the electronic device (701) can allocate a second resource for ASR2 (722). For example, the electronic device (701) can perform voice recognition on the voice using ASR3 (723) for the third signal. At this time, the electronic device (701) can allocate a third resource for ASR3 (723).

[0153] As described above, the electronic device (701) may include ASRs (721, 722, 723) for external electronic devices (511, 512, 513), and may perform speech recognition on signals obtained from each of the external electronic devices (511, 512, 513) through separate ASRs. Accordingly, the electronic device (701) may allocate resources to each external electronic device in order to perform speech recognition on the speech. For example, the electronic device (701) may allocate three resources for three external electronic devices (511, 512, 513). However, according to the method described above, when the number of external electronic devices capable of recognizing the same speech (or wake word) increases, the number of resources that the electronic device (701) must allocate increases, and a lot of time may be required to process the speech (or wake word). In contrast, electronic devices, methods, and computer-readable storage media utilizing voice recognition using dual selection according to embodiments of the present disclosure can perform highly accurate voice recognition using fewer resources. An example of an electronic device (101) utilizing voice recognition using dual selection according to embodiments of the present disclosure is described in FIGS. 8A through 8C below.

[0154] FIGS. 8A to 8C illustrate examples of a method for performing voice recognition using a signal provided from an external electronic device selected from among external electronic devices in a multi-device environment.

[0155] The electronic device (101) of FIGS. 8A to 8C may represent an example of the electronic device (101) of FIG. 6A. For example, the content of each of the FEs (811, 812, and 813) may be substantially identically applied to the content of the FE (611) of FIG. 6A. For example, the content of the MDW manager (820) may be substantially identically applied to the content of the MDW manager (613) of FIG. 6A. In addition, for example, the content of the MDW device selector (830) may be substantially identically applied to the content of the MDW device selector (615) of FIG. 6A. In addition, for example, the content of each of the ASRs (841, 842) may be substantially identically applied to the content of the ASR (617) of FIG. 6A. For example, the electronic device (101) of FIGS. 8A to 8C may include at least a portion of the electronic device (101) of FIG. 1. For example, the electronic device (101) may be referred to as a server that manages external electronic devices (511, 512, 513).

[0156] For example, the external electronic devices (511, 512, 513) of FIGS. 8A to 8C may represent examples of the external electronic device (601) of FIG. 6A. For example, each of the external electronic devices (511, 512, 513) may include devices such as a smartphone, a tablet PC, a TV, a robot vacuum cleaner, an air conditioner, and a refrigerator. Although three external electronic devices (511, 512, 513) are illustrated in FIGS. 8A to 8C, embodiments of the present disclosure are not limited thereto. The external electronic devices (511, 512, 513) may include two or four or more electronic devices.

[0157] Referring to FIG. 8A, an example (801) of a method in which an electronic device (101) selects a first external electronic device (or a temporary winner) using some of the signals obtained from each of external electronic devices (511, 512, 513) and performs voice recognition using ASR1 (841) for the selected first external electronic device is illustrated.

[0158] Referring to example (801), the electronic device (101) can obtain signals from each of the external electronic devices (511, 512, 513). For example, the electronic device (101) can obtain a first signal (850-1) from the external electronic device (511) through FE1 (811). For example, the electronic device (101) can obtain a second signal (850-2) from the external electronic device (512) through FE2 (812). For example, the electronic device (101) can obtain a third signal (850-3) from the external electronic device (513) through FE3 (813). Each of the first signal (850-1), the second signal (850-2), and the third signal (850-3) can include a voice spoken by a speaker (or user). For example, the voice may include a wake word. For example, external electronic devices (511, 512, 513) may represent electronic devices that can recognize or are set to recognize the wake word. For example, the electronic device (101) may acquire each of the first signal (850-1), the second signal (850-2), and the third signal (850-3) within the first period of the time interval in which the speaker utters the voice. In other words, each of the first signal (850-1), the second signal (850-2), and the third signal (850-3) may represent a signal (or voice signal) including the voice transmitted in real time.

[0159] According to one embodiment, the electronic device (101) may select the first signal (850-1) from among the first signal (850-1), the second signal (850-2), and the third signal (850-3) through the MDW manager (820). For example, the electronic device (101) may select the first signal (850-1) based on the timing at which the signal is acquired or the recognition result included in the signal.

[0160] For example, the electronic device (101) can recognize a first timing at which a first signal (850-1) is acquired, a second timing at which a second signal (850-2) is acquired, and a third timing at which a third signal (850-3) is acquired. The electronic device (101) can recognize the first timing among the first timing, the second timing, and the third timing, and select the first signal (850-1) according to the first timing. The electronic device (101) selecting the first signal (850-1) can be substantially the same as selecting the external electronic device (511) as the temporary winner.

[0161] Alternatively, for example, the electronic device (101) may recognize a first recognition result included in the first signal (850-1), a second recognition result included in the second signal (850-2), and a third recognition result included in the third signal (850-3). For example, the first recognition result may include a value calculated from feature values ​​for the voice recognized by the external electronic device (511). For example, the second recognition result may include a value calculated from feature values ​​for the voice recognized by the external electronic device (512). For example, the third recognition result may include a value calculated from feature values ​​for the voice recognized by the external electronic device (513). The electronic device (101) may recognize the first recognition result having the maximum value among the first recognition result, the second recognition result, and the third recognition result, and select the first signal (850-1) accordingly. The electronic device (101) selecting the first signal (850-1) may be substantially the same as selecting the external electronic device (511) as the temporary winner.

[0162] According to one embodiment, the electronic device (101) can store the acquired signal. For example, the electronic device (101) can store the signal acquired from each of the external electronic devices (510).

[0163] According to one embodiment, in response to selecting the first signal (850-1), the electronic device (101) may select ASR1 (841) and allocate a first resource for ASR1 (841). For example, the electronic device (101) may transmit the first signal (850-1) to ASR1 (841). For example, the electronic device (101) may generate first voice information (856) through the first signal (850-1) using ASR1 (841). For example, the first voice information (856) may include a portion of a voice spoken by a speaker (or user). According to one embodiment, the electronic device (101) may transmit the generated first voice information (856) to each of the external electronic devices (511, 512, 513).

[0164] In FIG. 8A, an example (801) of generating and transmitting a single first voice information (856) is illustrated, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) can continuously generate and transmit voice information for first signals acquired within a time interval. For example, assume that the speaker utters the sentence “Wake me up, send a text to Jane,” and a portion of the voice uttered by the speaker (e.g., “Wake me up, send a text to Jane”) is continuously processed over time. In other words, “Wake me up, send a text to Jane” (or “Wake me up, text to Jane”) may represent a voice longer than a reference length, and the portion (e.g., “Wake me up, send a text to Jane”) may represent a voice shorter than the reference length.

[0165] For example, the electronic device (101) can obtain signals indicating "call word" and "to Jane" from each of the external electronic devices (511, 512, 513) within a time interval. For example, the electronic device (101) can receive a first signal including the "call word" from the external electronic device (511), and then receive a first signal indicating "to Jane" from the external electronic device (511). The time interval during which the first signal indicating the "call word" and the first signal indicating "to Jane" are obtained can be less than the reference length. When the temporary winner is the external electronic device (511), the electronic device (101) can generate first voice information using ASR1 (841) from the first signals indicating the "call word" and "to Jane". For example, the electronic device (101) may generate first voice information indicating a “call word” and then generate first voice information indicating “call word, to Jane.” The electronic device (101) may transmit the generated first voice information (e.g., “call word”, “call word, to Jane”) to each of the external electronic devices (511, 512, 513). Accordingly, each of the external electronic devices (511, 512, 513) may execute a designated function based on the received first voice information. For example, the designated function may include displaying text indicated by the received first voice information or outputting a sound representing the text. In other words, the first voice information may cause at least some of the external electronic devices (511, 512, 513) to display a visual object corresponding to the text or output a sound representing the text.

[0166] Referring to FIGS. 8b and 8c, examples (802) and (803) are shown of a method in which an electronic device (101) selects a second external electronic device (or final winner) using signals obtained from each of external electronic devices (511, 512, 513) and performs voice recognition using ASR2 (842) for the second external electronic device.

[0167] Referring to example (802) of FIG. 8B, the electronic device (101) can obtain signals from each of the external electronic devices (511, 512, 513). For example, the electronic device (101) can obtain a first signal (860-1) from the external electronic device (511) through FE1 (811). For example, the electronic device (101) can obtain a second signal (860-2) from the external electronic device (512) through FE2 (812). For example, the electronic device (101) can obtain a third signal (860-3) from the external electronic device (513) through FE3 (813). For example, each of the first signal (860-1), the second signal (860-2), and the third signal (860-3) acquired in the second period following the first period within the time interval may represent a continuous signal with respect to each of the first signal (850-1), the second signal (850-2), and the third signal (850-3) acquired in the first period within the time interval of example (801). According to one embodiment, the electronic device (101) may store the acquired signal. For example, the electronic device (101) may store the signal acquired from each of the external electronic devices (510).

[0168] According to one embodiment, the electronic device (101) can determine, through the MDW manager (820), whether the time interval in which the signals of each of the external electronic devices (511, 512, 513) are acquired is greater than or equal to the reference length. For example, the electronic device (101) can determine, through the MDW manager (820), whether the time interval in which the first signal (850-1) and the first signal (860-1) are stored is greater than or equal to the reference length. The electronic device (101) can determine, through the MDW manager (820), whether the time interval in which the second signal (850-2) and the second signal (860-2) are stored is greater than or equal to the reference length. The electronic device (101) can determine, through the MDW manager (820), whether the time interval in which the third signal (850-3) and the third signal (860-3) are stored is greater than or equal to the reference length.

[0169] According to one embodiment, the electronic device (101) may obtain the first value and the second value of each of the external electronic devices (511, 512, 513) based on the time interval that is greater than or equal to the reference length. For example, the electronic device (101) may calculate the first value and the second value of the external electronic device (511) using the stored first signals (850-1, 860-1) using the MDW device selector (830). For example, the electronic device (101) may calculate the first value and the second value of the external electronic device (512) using the stored second signals (850-2, 860-2) using the MDW device selector (830). For example, the electronic device (101) can use the MDW device selector (830) to calculate the first value and the second value of the external electronic device (513) using the stored third signals (850-3, 860-3).

[0170] According to one embodiment, the electronic device (101) may calculate selection parameters based on the first value and the second value of each of the external electronic devices (511, 512, 513). For example, the electronic device (101) may calculate the selection parameters for the external electronic devices (511, 512, 513). For example, each of the selection parameters may be calculated based on the first value, the second value, and a weight value. For example, the weight value may be determined based on the environment in which the voice is spoken.

[0171] In one embodiment, the electronic device (101) may select a second external electronic device (or final winner) based on the selection parameters. For example, the electronic device (101) may select, as the second external electronic device, an external electronic device corresponding to a selection parameter having a maximum value among the selection parameters.

[0172] According to one embodiment, the electronic device (101) may determine whether the first external electronic device and the second external electronic device are the same. For example, if the first external electronic device is the same as the second external electronic device, the electronic device (101) may generate second voice information from the accumulated first signals (e.g., stored first signals (850-1, 860-1)) based on ASR1 (841) using the first resource. Thereafter, the electronic device (101) may transmit the first voice information (856) generated from the first signal (850-1) and the second voice information generated from the first signal (860-1) to each of the external electronic devices (511, 512, 513). For example, the text identified by the first voice information (856) and the second voice information may be “Wake word, today’s weather.”

[0173] For example, the electronic device (101) may determine that the first external electronic device is different from the second external electronic device. For example, the electronic device (101) may select an external electronic device (512) different from the external electronic device (511) selected as the first external electronic device as the second external electronic device. In response to selecting the external electronic device (512) as the second external electronic device, the electronic device (101) may perform a connection (869) with ASR2 (842). For example, the electronic device (101) may transmit a second signal (850-2) and a second signal (860-2) greater than or equal to the reference length to ASR2 (842) through the connection (869). For example, the electronic device (101) can perform voice recognition based on ASR2 (842) from accumulated second signals (e.g., stored second signals (850-2, 860-2)) that are longer than the reference length.

[0174] According to one embodiment, the electronic device (101) may provide the first signal (860-1) to ASR1 (841) while providing the accumulated second signals (e.g., the stored second signals (850-2, 860-2)) that are longer than the reference length to ASR2 (842). In other words, the electronic device (101) may use ASR1 (841) and ASR2 (842) simultaneously. For example, the electronic device (101) may simultaneously allocate a first resource for ASR1 (841) and a second resource for ASR2 (842). This is because, since ASR2 (842) has not been previously used, it may take a relatively long time to process the accumulated second signals (e.g., the stored second signals (850-2, 860-2)) that are longer than the reference length based on ASR2 (842). Accordingly, the electronic device (101) may use ASR1 (841) together with ASR2 (842) to provide uninterrupted service.

[0175] Referring to example (802) of FIG. 8B, the electronic device (101) may allocate the first resource and the second resource to be used during a specified time interval from a timing corresponding to the reference length. Within the specified time interval, the electronic device (101) may generate first voice information (866) generated using ASR1 (841) and transmit the generated first voice information (866) to each of the external electronic devices (511, 512, 513). For example, the text identified by the first voice information (866) may be "Wake up, text to Jane." Thereafter, referring to example (803) of FIG. 8C, the electronic device (101) may remove the allocated first resource at a timing when the specified time interval has elapsed from the timing corresponding to the reference length.

[0176] Referring to example (803) of FIG. 8C, the electronic device (101) can obtain signals from each of the external electronic devices (511, 512, 513). For example, the electronic device (101) can obtain a first signal (870-1) from the external electronic device (511) through FE1 (811). For example, the electronic device (101) can obtain a second signal (870-2) from the external electronic device (512) through FE2 (812). For example, the electronic device (101) can obtain a third signal (870-3) from the external electronic device (513) through FE3 (813). For example, each of the first signal (870-1), the second signal (870-2), and the third signal (870-3) obtained in the third period following the second period within the time interval may represent a continuous signal with respect to each of the first signal (860-1), the second signal (860-2), and the third signal (860-3) obtained in the second period within the time interval of example (802).

[0177] According to one embodiment, the electronic device (101) may transmit the second signal (870-2) to ASR2 (842). For example, the electronic device (101) may generate second voice information (877) through the second signal (870-2) using ASR2 (842). According to one embodiment, the electronic device (101) may transmit the generated second voice information (877) to each of the external electronic devices (511, 512, 513). For example, the text identified by the generated second voice information (877) may be "Tell Me."

[0178] According to one embodiment, the electronic device (101) may combine the first voice information (866) and the second voice information (877) when the first external electronic device and the second external electronic device are different and the accumulated first voice information (e.g., “Wake word, text Jane”) and the second voice information (877) (e.g., “Send”) are not similar. For example, the electronic device (101) may combine the first voice information (866) and the second voice information (877) when the difference in the number of characters between the text identified by the first voice information (866) and the text identified by the second voice information (877) changes by a specified number or more. This is because, when the first voice information (866) and the second voice information (877) are not similar, the difference between the text displayed on the external electronic devices (511, 512, 513) based on the first voice information (866) and the text displayed on the external electronic devices (511, 512, 513) based on the second voice information (877) is unnatural.

[0179] According to one embodiment, the electronic device (101) may provide a sound source completion message to the MDW manager (820) in response to confirming the end of signals obtained from each of the external electronic devices (511, 512, 513). For example, the end may indicate the last period of the time interval. In other words, the end may indicate that the voice spoken by the speaker has ended. The MDW manager (820) may transmit the sound source completion message to an activated ASR (e.g., ASR2 (842)). Accordingly, the electronic device (101) may obtain final voice information from ASR2 (842), and the final voice information may be transmitted to each of the external electronic devices (511, 512, 513). For example, the text identified by the final voice information may include all of the voice spoken by the speaker.

[0180] In the following FIGS. 9A to 9D, examples of a method for displaying text on at least some of the external electronic devices (511, 512, 513) based on voice information (e.g., first voice information (856), first voice information (866), and second voice information (877)) are described.

[0181] Figures 9a to 9d illustrate examples of how external electronic devices within a multi-device environment display the results of voice recognition.

[0182] FIGS. 9A to 9D illustrate examples of how external electronic devices (510) connected to an electronic device (101) display the results of voice recognition based on voice information acquired from the electronic device (101). Although the electronic device (101) is not illustrated in FIGS. 9A to 9D , the external electronic devices (510) may be connected to the electronic device (101).

[0183] Referring to FIG. 9A, examples (900, 905) are shown in which text identified by a combination of first voice information generated based on signals of a first external electronic device (or provisional winner) selected by an electronic device (101) and second voice information generated based on signals of a second external electronic device (or final winner) different from the first external electronic device (or provisional winner) is displayed on external electronic devices (510).

[0184] Referring to example (900), the electronic device (101) may select the external electronic device (512) as the first external electronic device (911). For example, the electronic device (101) may obtain signals for a voice (e.g., “Wake up word, text Jane”) spoken by the user (505) within a first time interval from each of the external electronic devices (510). For example, the electronic device (101) may obtain first signals for the voice from the external electronic device (511) during the first time interval. The first time interval may represent a time interval less than a reference length. For example, the electronic device (101) may obtain second signals for the voice from the external electronic device (512) during the first time interval. For example, the electronic device (101) may obtain third signals for the voice from the external electronic device (513) during the first time interval. For example, the electronic device (101) may acquire fourth signals for the voice from the external electronic device (514) during the first time interval. For example, using some of the signals acquired during the first time interval (e.g., the first signals, the second signals, the third signals, and the fourth signals), the electronic device (101) may select the external electronic device (512) as the first external electronic device (911). At this time, the electronic device (101) may generate first voice information using the first signals of the first time interval. For example, the first voice information may be provided to each of the external electronic devices (510). For example, the first voice information may include text representing the voice (e.g., "Wake up, text Jane"). For example, the first voice information may cause each of the external electronic devices (510) to display the text (or visual object) representing the voice.In example (900), each of the external electronic devices (510) can display the text representing the voice. Additionally, although not shown in example (900), each of the external electronic devices (510) can also output a sound corresponding to the text representing the voice.

[0185] In example (905), the electronic device (101) may select the external electronic device (514) as the second external electronic device (912). For example, the electronic device (101) may obtain a signal for another voice (e.g., “Send”) uttered by the user (505) within a second time interval after the first time interval from each of the external electronic devices (510). For example, the sum of the entire first time interval and at least a portion of the second time interval may be greater than or equal to a reference length. For example, the electronic device (101) may obtain a first signal for the other voice from the external electronic device (511) during the second time interval. For example, the electronic device (101) may obtain a second signal for the other voice from the external electronic device (512) during the second time interval. For example, the electronic device (101) may obtain a third signal for the other voice from the external electronic device (513) during the second time interval. For example, the electronic device (101) may obtain a fourth signal for the other voice from the external electronic device (514) during the second time interval. For example, using the signals obtained during the first time interval and the signals obtained during the second time interval, the electronic device (101) may select the external electronic device (514) as the second external electronic device (912). At this time, the electronic device (101) may generate second voice information using the fourth signal of the second time interval. For example, the second voice information may include text representing the other voice (e.g., "Send me"). At this time, the electronic device (101) can combine the first voice information and the second voice information when the difference in the number of characters between the text representing the voice and the text representing the other voice changes by a specified number or more.Accordingly, the electronic device (101) can provide the combined voice information to each of the external electronic devices (510). Accordingly, text identified based on the combined voice information (e.g., “Wake me up, send a text to Jane”) can be displayed on each of the external electronic devices (510).

[0186] Referring to FIG. 9B, examples (920, 925) are shown in which text identified by first voice information generated based on signals of a first external electronic device (or provisional winner) selected by an electronic device (101) and text identified by second voice information generated based on signals of a second external electronic device (or final winner) different from the first external electronic device (or provisional winner) are displayed on the first external electronic device.

[0187] Referring to example (920), the electronic device (101) may select the external electronic device (512) as the first external electronic device (931) based on signals acquired from each of the external electronic devices (510) during a time period less than a reference length. The electronic device (101) may generate first voice information based on second signals acquired from the external electronic device (512) and transmit the first voice information to each of the external electronic devices (510). At this time, the first voice information may cause the external electronic device (512), which is the first external electronic device (931) among the external electronic devices (510), to display a text representing a voice (e.g., “Wake up, Jane”). At this time, the remaining external electronic devices (511, 513, 514) among the external electronic devices (510) excluding the external electronic device (512) may not display the text representing the voice.

[0188] Referring to example (925), the electronic device (101) may select the external electronic device (512) as the second external electronic device (932) based on signals obtained from each of the external electronic devices (510) during a time period longer than the reference length. In other words, the first external electronic device (931) and the second external electronic device (932) may be the same. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may generate second voice information based on second signals obtained from the external electronic device (512) and transmit the second voice information to each of the external electronic devices (510). In this case, the second voice information may cause the external electronic device (512), which is the second external electronic device (932) among the external electronic devices (510), to display a text representing the voice (e.g., "Wake me up, send a text to Jane"). At this time, among the external electronic devices (510), the remaining external electronic devices (511, 513, 514) excluding the external electronic device (512) may not display the text representing the voice. If the first external electronic device (931) and the second external electronic device (932) are different from each other, the external electronic device displaying the text representing the voice based on the second voice information may be changed.

[0189] Referring to FIG. 9c, an example (940) is shown in which text identified by first voice information generated based on signals of a first external electronic device (or provisional winner) selected by an electronic device (101) is displayed on the first external electronic device, and an example (945) is shown in which text identified by second voice information generated based on signals of a second external electronic device (or final winner) different from the first external electronic device (or provisional winner) is displayed on the first external electronic device and the second external electronic device.

[0190] Referring to example (940), the electronic device (101) may select the external electronic device (512) as the first external electronic device (951) based on signals acquired from each of the external electronic devices (510) during a time period less than a reference length. The electronic device (101) may generate first voice information based on second signals acquired from the external electronic device (512) and transmit the first voice information to each of the external electronic devices (510). At this time, the first voice information may cause the external electronic device (512), which is the first external electronic device (951) among the external electronic devices (510), to display a text representing a voice (e.g., “Wake up, Jane”). At this time, the remaining external electronic devices (511, 513, 514) among the external electronic devices (510) excluding the external electronic device (512) may not display the text representing the voice.

[0191] Referring to example (945), the electronic device (101) may select the external electronic device (514) as the second external electronic device (952) based on signals obtained from each of the external electronic devices (510) during a time period longer than the reference length. In other words, the first external electronic device (951) and the second external electronic device (952) may be different from each other. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may generate second voice information based on fourth signals obtained from the external electronic device (514) and transmit the second voice information to each of the external electronic devices (510). At this time, the second voice information may cause the external electronic device (512), which is the first external electronic device (951) among the external electronic devices (510), and the external electronic device (514), which is the second external electronic device (952), to display a text representing the voice (e.g., "Wake me up, send a text to Jane"). At this time, the remaining external electronic devices (511, 513) excluding the external electronic devices (512, 514) among the external electronic devices (510) may not display the text representing the voice. If the first external electronic device (951) and the second external electronic device (952) are the same, the external electronic device that displays the text representing the voice based on the second voice information may be one.

[0192] Referring to FIG. 9d, examples (960, 965) are shown in which text identified by first voice information generated based on signals of a first external electronic device (or provisional winner) selected by an electronic device (101) and text identified by second voice information generated based on signals of a second external electronic device (or final winner) different from the first external electronic device (or provisional winner) are displayed on the first external electronic device.

[0193] Referring to example (960), the electronic device (101) may select the external electronic device (512) as the first external electronic device (971) based on signals obtained from each of the external electronic devices (510) during a time interval less than a reference length. The electronic device (101) may generate first voice information based on second signals obtained from the external electronic devices (512) and transmit the first voice information to each of the external electronic devices (510). In this case, the first voice information may cause each of the external electronic devices (510) to display text representing the voice (e.g., "Wake me up, Jane").

[0194] Referring to example (965), the electronic device (101) may select the external electronic device (514) as the second external electronic device (972) based on signals obtained from each of the external electronic devices (510) during a time period longer than the reference length. In other words, the first external electronic device (971) and the second external electronic device (972) may be different from each other. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may generate second voice information based on the fourth signals obtained from the external electronic device (514) and transmit the second voice information to each of the external electronic devices (510). In this case, the second voice information may cause each of the external electronic devices (510) to display text representing the voice (e.g., "Wake me up, send a text to Jane").

[0195] According to one embodiment, the text representing the voice (e.g., "Send a text to Jane") indicated by the second voice information may have a visual effect. A portion (981) of the text representing a portion of the voice (e.g., "Send a text to Jane") may have a first visual effect. Conversely, a portion (982) of the text representing the remainder of the voice (e.g., "Send a text") may have a second visual effect. The first visual effect may be used to indicate that the portion (981) is generated based on a first signal received from a first external electronic device (971). The second visual effect may be used to indicate that the portion (982) is generated based on a second signal received from a second external electronic device (972).

[0196] In one embodiment, the second visual effect may be different from the first visual effect. For example, if the first visual effect is a first color applied to a portion (981), the second visual effect may be a second color applied to a portion (982). For example, if the first visual effect is a first background color displayed on a background portion of the portion (981), the second visual effect may be a second background color displayed on a background portion of the portion (982). For example, if the first visual effect is a first icon for indicating a portion (981), the second visual effect may be a second icon for indicating a portion (982). However, the embodiments of the present disclosure are not limited to the examples described above. In FIG. 9d, the first visual effect and the second visual effect are illustrated as being applied to text displayed on an external electronic device (514), but the embodiments of the present disclosure are not limited thereto. The first visual effect and the second visual effect may also be applied to text displayed on external electronic devices (511, 512, 513) of example (965).

[0197] Referring to FIGS. 9A to 9D , external electronic devices (511, 512, 513, and 514) can acquire a voice (e.g., “Wake me up, send a text to Jane”) spoken by a user (505) and transmit a signal for the acquired voice to the electronic device (101). The electronic device (101) can determine a provisional winner and a final winner based on signals received from each of the external electronic devices (511, 512, 513, and 514), perform voice recognition based on the determined winner, and transmit voice information as a result of the voice recognition to the external electronic devices (511, 512, 513, and 514). The voice information provided to the external electronic devices (511, 512, 513, 514) may cause at least some of the external electronic devices (511, 512, 513, 514) to display text representing the voice (e.g., "Wake me up, send a text to Jane"). After displaying the text representing the voice, the external electronic devices (511, 512, 513, 514) may execute a function according to the voice and then display another text in response to the voice. For example, in response to the voice requesting transmission of a text, a second external electronic device (or the final winner) among the external electronic devices (511, 512, 513, 514) may perform the function according to the voice (e.g., transmit a text). After this, at least some of the external electronic devices (511, 512, 513, 514) may display the other text in response to the voice. For example, an external electronic device selected as a first external electronic device (or a temporary winner) among the external electronic devices (511, 512, 513, 514) may display the other text. Alternatively, for example, an external electronic device selected as a second external electronic device (or a final winner) among the external electronic devices (511, 512, 513, 514) may display the other text.Alternatively, for example, an external electronic device located in the area closest to the user (505) at the timing selected as the second external electronic device (or final winner) among the external electronic devices (511, 512, 513, 514) may display the other text. The above example is merely an example for convenience of explanation, and embodiments of the present disclosure are not limited thereto.

[0198] FIGS. 10A and 10B illustrate examples of a method for performing voice recognition according to the mode of external electronic devices in a multi-device environment.

[0199] The above mode may include a use mode (or first mode) of an external electronic device that performs interaction by a user (505) and a standby mode (or second mode) of an external electronic device that does not perform the interaction by a user (505) but can recognize a call word.

[0200] Referring to FIG. 10A, an example (1000) of a method in which an electronic device (101) identifies external electronic devices (510) within the usage mode and performs voice recognition based on signals obtained from the identified external electronic devices is illustrated.

[0201] Referring to example (1000), a user (505) can use an external electronic device (511) and an external electronic device (512). For example, the electronic device (101) can recognize the external electronic device (511) and the external electronic device (512) within the use mode. At this time, the external electronic device (513) and the external electronic device (514) can be within the standby mode. When the user (505) utters a voice including a wake word, the electronic device (101) can obtain signals for the voice from each of the external electronic device (511) and the external electronic device (512). Thereafter, the electronic device (101) can perform voice recognition of FIGS. 8A to 8C based on the signals for the voice obtained from each of the external electronic device (511) and the external electronic device (512) within the use mode. As mentioned above, the user (505) may expect that a response to the voice will occur in the device currently in use (e.g., external electronic device (511) and external electronic device (512)) as compared to other devices (e.g., external electronic device (513) and external electronic device (514)).

[0202] Referring to FIG. 10b, an example (1050) of a method for identifying external electronic devices that change from the standby mode to the use mode when all external electronic devices (510) are in the standby mode is illustrated, and performing voice recognition based on signals obtained from the identified external electronic devices.

[0203] Referring to example (1050), all external electronic devices (510) may be in the standby mode. For example, the electronic device (101) may recognize the external electronic devices (510) in the standby mode. When the user (505) utters a voice including a wake word, the electronic device (101) may obtain signals for the voice from each of the external electronic devices (510) in the standby mode. Thereafter, the electronic device (101) may determine whether the external electronic devices (510) in the standby mode have changed from the standby mode to the use mode. For example, the electronic device (101) may recognize whether there is an external electronic device that has changed from the standby mode to the use mode within the reference change interval. For example, if there is an external electronic device that has changed from the standby mode to the use mode, the electronic device (101) may perform the voice recognition using signals obtained from the external electronic device that has changed to the use mode. Referring to examples (1055-1) and (1055-2), the electronic device (101) can recognize that the external electronic device (511) and the external electronic device (513) have changed from the standby mode to the use mode. Accordingly, the electronic device (101) can perform the voice recognition using signals obtained from the external electronic device (511) and signals obtained from the external electronic device (513). Unlike examples (1055-1, 1055-2), when there is no external electronic device that has changed from the standby mode to the use mode, the voice recognition can be performed using signals obtained from each of the external electronic devices (510) in the standby mode.

[0204] According to one embodiment, the electronic device (101) can set priorities for the standby mode and the use mode, and perform the voice recognition according to the set priorities. For example, if the electronic device (101) is set to perform the voice recognition only using external electronic devices within the use mode, the electronic device (101) can perform an operation such as example (1000) of FIG. 10A. Conversely, if the external electronic devices within the use mode are set as the first priority and the external electronic devices within the standby mode are set as the second priority for the electronic device (101), the electronic device (101) can perform an operation such as example (1050) of FIG. 10B.

[0205] FIG. 11 illustrates an example of an operational flow for a method of performing voice recognition using a signal provided from an external electronic device selected from among external electronic devices in a multi-device environment.

[0206] At least some of the methods of FIG. 11 may be performed by the electronic device (101) of FIG. 6A. For example, at least some of the methods may be controlled by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 1). In the embodiments described below, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0207] For example, the at least one processor of the electronic device (101) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0208] In operation (1110), the electronic device (101) may obtain signals for a voice including the call word during a time interval from each of external electronic devices capable of recognizing the call word. For example, the electronic device (101) may be connected to the external electronic devices capable of recognizing the call word using a communication circuit (e.g., the communication module (190) of FIG. 1). According to one embodiment, the external electronic devices may represent electronic devices for which the same call word is set or can be recognized. For example, the call word may include a designated word or sentence used to indicate a specific electronic device. The call word may be used to execute a voice recognition function of the indicated specific electronic device. For example, the call word may be preset or set by a user.

[0209] For example, the external electronic devices may represent devices within an MDW environment. For example, the MDW environment may represent an environment including external electronic devices within a specific space, external electronic devices having the same account information, or external electronic devices connected to a specific AP (access point).

[0210] According to one embodiment, the electronic device (101) may acquire signals from each of the external electronic devices while the user speaks. For example, the electronic device (101) may acquire signals from each of the external electronic devices during the time period during which the voice is spoken. For example, the signals acquired from each of the external electronic devices may be acquired periodically within the time period.

[0211] In operation (1120), the electronic device (101) may select first signals received from a first external electronic device (or a temporary winner) among the external electronic devices, using some of the signals of each of the external electronic devices while the signals of each of the external electronic devices are acquired in the time interval. For example, the electronic device (101) may select the first signals among the signals of each of the external electronic devices based on the timing at which some of the signals of each of the external electronic devices are acquired or the recognition results included in some of the signals. For example, the electronic device (101) may select a signal having the fastest timing (e.g., the first signals) among the timings for the external electronic devices, and select the external electronic device corresponding to the selected signal as the first external electronic device. Alternatively, for example, the electronic device (101) may select a signal including a recognition result having a maximum value among the recognition results for the external electronic devices, and select the external electronic device corresponding to the selected signal as the first external electronic device.

[0212] According to one embodiment, the electronic device (101) may select a first ASR (or a first voice recognition model) for the selected first external electronic device. For example, the electronic device (101) may allocate a first resource for the selected first ASR. For example, the electronic device (101) may generate first voice information using the first ASR from some of the first signals of the selected first external electronic device. For example, the first voice information may include a portion of the voice corresponding to the portion of the first signals. For example, the first voice information may include text representing a portion of the voice.

[0213] In operation (1130), the electronic device (101) may transmit the first voice information generated from the portion of the first signals of the first external electronic device to each of the external electronic devices. For example, the electronic device (101) may transmit the first voice information to each of the external electronic devices to cause the display of text representing a portion of the voice. At this time, the first voice information may cause at least some of the external electronic devices to display the text representing a portion of the voice. In addition, for example, the first voice information may cause the output of a sound corresponding to the text representing a portion of the voice.

[0214] In operation (1140), the electronic device (101) may select second signals acquired from a second external electronic device (or a final winner) among the external electronic devices, based on the time interval that is greater than or equal to a reference length, using the signals of each of the external electronic devices. For example, the electronic device (101) may determine whether the time interval is greater than or equal to the reference length. For example, if the time interval is less than the reference length, the electronic device (101) may repeat operations (1110) and (1120) using signals acquired during the time interval that is less than the reference length.

[0215] According to one embodiment, based on the time interval that is greater than or equal to the reference length, the electronic device (101) may select the second signals received from the second external electronic device using the signals of each of the external electronic devices. At this time, the signals may represent signals obtained in the time interval that is greater than or equal to the reference length and stored (or accumulated) in the electronic device (101). In the case of the time interval that is greater than or equal to the reference length, the electronic device (101) may obtain a first value and a second value from the signals of each of the external electronic devices. For example, the first value may represent a distance from each of the external electronic devices to a user. For example, the second value may represent a quality of the voice identified by the signals of each of the external electronic devices.

[0216] According to one embodiment, the electronic device (101) may generate selection parameters for the external electronic devices from a set of the first value and the second value of each of the external electronic devices. For example, each of the selection parameters may be calculated based on the first value, the second value, and a weight value. For example, the weight value may be determined based on the environment in which the voice is spoken. For example, the electronic device (101) may select the second external electronic device based on a selection parameter having a maximum value among the selection parameters.

[0217] According to one embodiment, the electronic device (101) may select a second ASR (or a second voice recognition model) for the selected second external electronic device. For example, the electronic device (101) may allocate a second resource for the selected second ASR. For example, the electronic device (101) may generate second voice information using the second ASR from second signals of the selected second external electronic device. For example, the second signals may represent signals acquired in the time interval longer than the reference length and stored (or accumulated) in the electronic device (101). For example, the second voice information may include all of the voice corresponding to the second signals. For example, the second voice information may include text representing all of the voice.

[0218] According to one embodiment, the electronic device (101) can determine whether the first external electronic device and the second external electronic device are the same. For example, if the first external electronic device is the same as the second external electronic device, the electronic device (101) can generate the second voice information from the second signals based on the first ASR using the first resource. At this time, the second signals can include the first signals. For example, the electronic device (101) can determine that the first external electronic device is different from the second external electronic device. If the first external electronic device is different from the second external electronic device, the electronic device (101) can perform a connection with the second ASR for the second external electronic device. For example, the electronic device (101) can transmit the second signals having a length greater than or equal to the reference length to the second ASR.

[0219] According to one embodiment, the electronic device (101) may provide the first signals having a length greater than or equal to the reference length to the first ASR while providing the second signals having a length greater than or equal to the reference length to the second ASR. The first signals having a length greater than or equal to the reference length may represent signals acquired from the first external electronic device in the time interval greater than or equal to the reference length and stored (or accumulated) in the electronic device (101). In other words, the electronic device (101) may utilize the first ASR and the second ASR simultaneously. For example, the electronic device (101) may simultaneously allocate the first resource for the first ASR and the second resource for the second ASR.

[0220] In operation (1150), the electronic device (101) may transmit the second voice information generated from the second signals of the second external electronic device to each of the external electronic devices. For example, the electronic device (101) may transmit the second voice information to each of the external electronic devices to cause the display of text representing the voice. At this time, the second voice information may cause at least some of the external electronic devices to display the text representing the voice. In addition, for example, the second voice information may cause the output of a sound corresponding to the text representing the voice.

[0221] Although not illustrated in FIG. 11, according to one embodiment, the electronic device (101) can recognize the mode of the external electronic devices. For example, the mode may include a use mode (or first mode) of the external electronic device that performs interaction by the user and a standby mode (or second mode) of the external electronic device that does not perform the interaction by the user but can recognize the wake word. The electronic device (101) may perform operations (1110) to (1150) using signals obtained from external electronic devices in the use mode among the external electronic devices. Alternatively, for example, when all of the external electronic devices are in the standby mode, the electronic device (101) may identify an external electronic device that is changed from the standby mode to the use mode within a reference change interval, and perform operations (1110) to (1150) using signals obtained from the identified external electronic device. At this time, if there is no external electronic device that changes from the standby mode to the use mode within the reference change interval, the electronic device (101) may perform operations (1110) to (1150) using signals obtained from each of the external electronic devices in the standby mode. For specific details related thereto, reference may be made to FIGS. 10A and 10B.

[0222] As described above, the electronic device (101) may include a memory (130) that stores instructions and includes one or more storage media. The electronic device (101) may include a communication circuit (190). The electronic device (101) may include at least one processor (120) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to acquire signals for speech including a wake-up word during a time interval from each of external electronic devices (510) capable of recognizing the wake-up word. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select first signals acquired from a first external electronic device among the external electronic devices (510) using some of the signals of each of the external electronic devices (510) while the signals of each of the external electronic devices (510) are acquired during the time interval. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to transmit first voice information generated from some of the first signals of the first external electronic device to each of the external electronic devices (510).The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select second signals acquired from a second external electronic device among the external electronic devices (510) using the signals of each of the external electronic devices (510) based on the time interval that is greater than or equal to a reference length. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to transmit second voice information generated from the second signals of the second external electronic device to each of the external electronic devices (510). The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0223] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to establish a connection with the external electronic devices (510) using the communication circuit (190). The external electronic devices (510) may be connected to an access point (AP) to which the electronic device is connected, or may have account information of the electronic device.

[0224] According to one embodiment, the signals obtained from each of the external electronic devices (510) may be obtained periodically within the time interval.

[0225] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to recognize a timing at which the portion of the signals from each of the external electronic devices (510) is acquired. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select the first signals received from the first external electronic device based on an initial timing among the timings of the external electronic devices (510).

[0226] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to obtain a recognition result for the voice included in the portion of the signals of each of the external electronic devices (510). The recognition result may include a value calculated from feature values ​​for the voice. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select the first signals received from the first external electronic device based on a recognition result having a maximum value among the recognition results of the external electronic devices (510).

[0227] According to one embodiment, the recognition result having the maximum value may include a value calculated from feature values ​​recognized by the first external electronic device.

[0228] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to allocate a first resource to be utilized in a first voice recognition model for the selected first external electronic device. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to generate, based on the first voice recognition model utilizing the first resource, the first voice information from the portion of the first signals. The first voice information may include first text recognized from the portion of the first signals and representing the portion of the voice.

[0229] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to recognize another portion of the first signals including the portion of the time interval less than the reference length. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to transmit third voice information generated from the other portion based on the first voice recognition model using the first resource, to each of the external electronic devices (510). The time length according to the other portion of the first signals may be longer than the time length according to the portion of the first signals.

[0230] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to obtain, from the signals of each of the external electronic devices (510), a first value indicating a distance from a user who uttered the voice and a second value indicating a quality of the voice, based on the time interval that is greater than or equal to the reference length. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to generate selection parameters for the external electronic devices (510) from the first value and the second value of each of the external electronic devices (510). The above instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select the second signals received from the second external electronic device based on a selection parameter having a maximum value among the selection parameters.

[0231] According to one embodiment, each of the selection parameters may be calculated based on the first value, the second value, and a weight value. The weight value may be determined based on the environment in which the voice is spoken.

[0232] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to determine whether the first external electronic device is identical to the second external electronic device. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to generate the second voice information from the second signals based on the first voice recognition model using the first resource when the first external electronic device is identical to the second external electronic device. When the first external electronic device is identical to the second external electronic device, the second signals of the time interval greater than or equal to the reference length may include the first signals of the time interval less than the reference length.

[0233] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to allocate a second resource to be used in a second voice recognition model for the selected second external electronic device, if the first external electronic device is different from the second external electronic device. The second voice recognition model may be different from the first voice recognition model. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to generate the second voice information from the second signals based on the second voice recognition model using the second resource. If the first external electronic device is different from the second external electronic device, the second signals may be distinct from the first signals.

[0234] According to one embodiment, when the first external electronic device is different from the second external electronic device and the difference between the part of the voice and the voice is greater than or equal to a specified number, the second voice information may be generated based on a combination of a voice recognition result of the first signals and a voice recognition result of the second signals.

[0235] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to allocate the first resource for the first voice recognition model and the second resource for the second voice recognition model to be used during a specified time interval from a timing corresponding to the reference length when the first external electronic device is different from the second external electronic device. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to cancel the allocated first resource at a timing when the specified time interval has elapsed from the timing corresponding to the reference length.

[0236] According to one embodiment, the first voice information may cause at least some of the external electronic devices (510) to display a visual object representing the portion of the voice. The second voice information may cause at least some of the external electronic devices (510) to display a visual object representing the voice.

[0237] In one embodiment, the first voice information may cause the first external electronic device to display the visual object representing the portion of the voice. The second voice information may cause each of the external electronic devices (510) to display the visual object representing the voice.

[0238] In one embodiment, the first voice information may cause the first external electronic device to display the visual object representing the portion of the voice. The second voice information may cause the second external electronic device to display the visual object representing the voice.

[0239] According to one embodiment, the external electronic devices (510) may include electronic devices that perform interaction by a user.

[0240] According to one embodiment, the external electronic devices (510) may include electronic devices in a ready mode. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to determine whether the external electronic devices (510) change from the ready mode to a different use mode within a reference change interval. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to obtain the signals from each of the external electronic devices (510) in the ready mode based on determining that the external electronic devices (510) do not change from the ready mode to the mode within the reference change interval.

[0241] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that, when individually or collectively executed by at least one processor (120) of an electronic device (101) including a communication circuit (190), cause the electronic device (101) to acquire signals for speech including a wake-up word from each of external electronic devices (510) capable of recognizing the wake-up word during a time interval. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to select first signals acquired from a first external electronic device among the external electronic devices (510) using some of the signals from each of the external electronic devices (510) while the signals from each of the external electronic devices (510) are acquired during the time interval. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the first voice information generated from some of the first signals of the first external electronic device to be transmitted to each of the external electronic devices (510). The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the second signals obtained from a second external electronic device among the external electronic devices (510) to be selected using the signals of each of the external electronic devices (510) based on the time interval that is greater than or equal to a reference length.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the second voice information generated from the second signals of the second external electronic device to be transmitted to each of the external electronic devices (510). The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0242] The method performed by the electronic device (101) as described above may include an operation of acquiring signals for a voice including a wake-up word from each of external electronic devices (510) capable of recognizing the wake-up word during a time interval. The method may include an operation of selecting first signals acquired from a first external electronic device among the external electronic devices (510) using some of the signals of each of the external electronic devices (510) while the signals of each of the external electronic devices (510) are acquired during the time interval. The method may include an operation of transmitting first voice information generated from some of the first signals of the first external electronic device to each of the external electronic devices (510). The method may include an operation of selecting second signals acquired from a second external electronic device among the external electronic devices (510) using the signals of each of the external electronic devices (510) based on the time interval that is greater than or equal to a reference length. The method may include an operation of transmitting second voice information generated from the second signals of the second external electronic device to each of the external electronic devices (510). The first voice information may include a portion of the voice recognized from the portion of the first signals. The second voice information may include the voice recognized from the second signals.

[0243] As described above, the electronic device (101) may include a memory (130) that stores instructions and includes one or more storage media. The electronic device (101) may include a communication circuit (190). The electronic device (101) may include at least one processor (120) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to obtain signals for a voice including a wake-up word from each of external electronic devices (510) capable of recognizing the wake-up word. The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to transmit, to each of the external electronic devices (510), first voice information generated from some of the first signals based on selecting first signals obtained from a first external electronic device among the external electronic devices (510) using some of the signals of each of the external electronic devices (510). The instructions, when individually or collectively executed by the at least one processor (120), may cause the electronic device (101) to select second signals received from a second external electronic device among the external electronic devices (510) using the signals of each of the external electronic devices (510) after selecting the first signals, and to transmit, to each of the external electronic devices (510), second voice information generated from the second signals.The first voice information may cause at least some of the external electronic devices (510) to display a visual object representing a portion of the voice recognized from the portion of the first signals. The second voice information may cause at least some of the external electronic devices (510) to display a visual object representing the voice recognized from the second signals.

[0244] The method performed by the electronic device (101) as described above may include an operation in which the electronic device (101) obtains signals for a voice including a wake-up word from each of external electronic devices (510) capable of recognizing the wake-up word. The method may include an operation in which, based on selecting first signals obtained from a first external electronic device among the external electronic devices (510) using some of the signals of each of the external electronic devices (510), first voice information generated from some of the first signals is transmitted to each of the external electronic devices (510). The method may include an operation of selecting second signals received from a second external electronic device among the external electronic devices (510) by using the signals of each of the external electronic devices (510) after selecting the first signals, and transmitting second voice information generated from the second signals to each of the external electronic devices (510). The first voice information may cause at least some of the external electronic devices (510) to display a visual object representing a portion of the voice recognized from the portion of the first signals. The second voice information may cause at least some of the external electronic devices (510) to display a visual object representing the voice recognized from the second signals.

[0245] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that, when individually or collectively executed by at least one processor (120) of an electronic device (101) including a communication circuit (190), cause the electronic device (101) to obtain signals for a voice including a wake-up word from each of external electronic devices (510) capable of recognizing the wake-up word. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to transmit, to each of the external electronic devices (510), first voice information generated from some of the first signals based on selecting first signals obtained from a first external electronic device among the external electronic devices (510) using some of the signals of each of the external electronic devices (510). The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to select the first signals and then, based on selecting second signals received from a second external electronic device among the external electronic devices (510) using the signals of each of the external electronic devices (510), transmit second voice information generated from the second signals to each of the external electronic devices (510). The first voice information may cause at least some of the external electronic devices (510) to display a visual object representing a portion of the voice recognized from the portion of the first signals.The second voice information may cause at least some of the external electronic devices (510) to display a visual object representing the voice recognized from the second signals.

[0246] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0247] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0248] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0249] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0250] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0251] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), A memory (130) storing instructions and including one or more storage media; a communication circuit (190); and At least one processor (120) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Acquire signals for voice including the wake-up word during a time interval from each of external electronic devices (510) capable of recognizing the wake-up word; While the signals of each of the external electronic devices (510) are acquired during the time interval, some of the signals of each of the external electronic devices (510) are used to select first signals acquired from a first external electronic device among the external electronic devices (510); Transmitting first voice information generated from some of the first signals of the first external electronic device to each of the external electronic devices (510); Based on the time interval that is greater than or equal to the reference length, using the signals of each of the external electronic devices (510), selecting second signals obtained from a second external electronic device among the external electronic devices (510); and Causes the second voice information generated from the second signals of the second external electronic device to be transmitted to each of the external electronic devices (510), The first voice information comprises a portion of the voice recognized from the portion of the first signals, and The second voice information includes the voice recognized from the second signals. Electronic devices (101).

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: By using the above communication circuit (190), a connection is established with the external electronic devices (510), The above external electronic devices (510) are connected to an AP (access point) to which the electronic device is connected, or have account information of the electronic device. Electronic devices (101).

3. In claim 1, The signals obtained from each of the above external electronic devices (510) are obtained periodically within the time interval. Electronic devices (101).

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Recognize the timing at which some of the signals of each of the external electronic devices (510) are acquired; and Causing to select the first signals obtained from the first external electronic device based on an initial timing among the timings of the external electronic devices (510). Electronic devices (101).

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Obtaining a recognition result for the voice included in the part of the signals of each of the external electronic devices (510), and the recognition result including a value calculated from feature values ​​for the voice; and Causing to select the first signals received from the first external electronic device based on the recognition result having the maximum value among the recognition results of the external electronic devices (510). Electronic devices (101).

6. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Allocating a first resource to be used in a first speech recognition model for the above-mentioned selected first external electronic device; and generating said first voice information from said part of said first signals based on said first voice recognition model using said first resource, The first voice information comprises a first text that is recognized from the part of the first signals and represents the part of the voice. Electronic devices (101).

7. In claim 6, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Recognizing another part of said first signals including said part of said time interval less than said reference length; and Causing to transmit third voice information generated from said other part based on said first voice recognition model using said first resource to each of said external electronic devices (510); The time length according to said other part of said first signals is longer than the time length according to said part of said first signals. Electronic devices (101).

8. In claim 6, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Based on the time interval longer than the reference length, a first value representing a distance from the user who uttered the voice and a second value representing the quality of the voice are obtained from the signals of each of the external electronic devices (510); Generating selection parameters for the external electronic devices (510) from the first value and the second value of each of the external electronic devices (510); and Causing to select the second signals received from the second external electronic device based on a selection parameter having a maximum value among the above selection parameters. Electronic devices (101).

9. In claim 8, Each of the above selection parameters is calculated based on the first value, the second value, and the weight value, and The above weight value is determined according to the environment in which the voice was spoken. Electronic devices (101).

10. In claim 8, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: determining whether the first external electronic device is identical to the second external electronic device; If the first external electronic device is identical to the second external electronic device, cause the second voice information to be generated from the second signals based on the first voice recognition model using the first resource, If the first external electronic device is identical to the second external electronic device, the second signals of the time interval greater than or equal to the reference length include the first signals of the time interval less than the reference length. Electronic devices (101).

11. In claim 10, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: If the first external electronic device is different from the second external electronic device, allocate a second resource to be used in a second voice recognition model for the selected second external electronic device, wherein the second voice recognition model is different from the first voice recognition model; and To cause the second voice information to be generated from the second signals based on the second voice recognition model using the second resource, If the first external electronic device is different from the second external electronic device, the second signals are distinct from the first signals. Electronic devices (101).

12. In claim 10, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: If the first external electronic device is different from the second external electronic device, allocate the first resource for the first speech recognition model and the second resource for the second speech recognition model to be used during a specified time interval from the timing corresponding to the reference length; and At a timing when the specified time interval has elapsed from the timing corresponding to the above reference length, causing the allocated first resource to be canceled. Electronic devices (101).

13. In claim 1, The first voice information causes at least some of the external electronic devices (510) to display a visual object representing the part of the voice, and The second voice information causes at least some of the external electronic devices (510) to display a visual object representing the voice. Electronic devices (101).

14. In claim 13, The first voice information causes the first external electronic device to display the visual object representing the part of the voice, and The second voice information causes each of the external electronic devices (510) to display the visual object representing the voice. Electronic devices (101).

15. In claim 1, The above external electronic devices (510) include electronic devices in a ready mode, The above instructions, when individually or collectively executed by the at least one processor (120), cause the electronic device (101) to: Determining whether the external electronic devices (510) change from the standby mode to a different usage mode within the reference change interval; and Based on determining that the external electronic devices (510) do not change from the standby mode to the mode within the above-mentioned reference change interval, causing the signals to be acquired from each of the external electronic devices (510) within the standby mode. Electronic devices (101).

Images