Electronic device for performing speech recognition and control method thereof

Abstract

The electronic device includes one or more processors that: while the electronic device is operating in a speech mode, input a user speech signal corresponding to a user speech received through a microphone into a speech recognition model to perform speech recognition; according to the result of voice recognition, acquiring environment information near the electronic device when the user voice is received; storing the acquired environment information in a memory; identifying, when the electronic device is operating in a learning mode, an external device for outputting a user voice learned by a user from among a plurality of external devices based on the environmental information among a plurality of pieces of environmental information stored in the memory; controlling a communication interface such that a command for controlling output of the user voice for learning is transmitted to the external device; and if a user voice signal for learning is received from the microphone, training the voice recognition model on the user voice signal for learning.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an electronic device and a control method thereof, and more particularly, to an electronic device that performs voice recognition in consideration of misrecognition situations and a control method thereof. Background Art

[0002] Various types of conventional electronic devices are developed and promoted, and more particularly, conventional electronic devices that can be controlled by using a user's voice may be distributed to houses to facilitate user's operation.

[0003] However, the conventional method is only configured to improve the recognition performance and recognition accuracy of speech recognition by training a speech recognition model using a large amount of data.

[0004] In particular, a speech recognition model is desired that takes into account an actual usage environment of an electronic device, a surrounding environment (eg, ambient noise) of the electronic device when a user voice is received, and trains the speech recognition model in consideration of the surrounding environment. Summary of the invention

[0005] An electronic device may include: a speaker; a microphone; a communication interface; a memory storing at least one instruction; and one or more processors connected to the speaker, the microphone, the communication interface and the memory. The one or more processors may be configured to: when the electronic device is operating in a speech recognition mode, perform speech recognition by inputting a user speech signal from the microphone into a speech recognition model, the user speech signal corresponding to the user speech received through the microphone; obtain environmental information around the electronic device when the user speech is received according to the result of the speech recognition; store the obtained environmental information in the memory; when the electronic device is operating in a learning mode, identify an external device for outputting a user speech for learning from a plurality of external devices based on the environmental information among the plurality of environmental information stored in the memory; control the communication interface to send a command for controlling the output of the user speech for learning to the external device; and train the speech recognition model for the received user speech signal for learning based on the user speech signal for learning received from the microphone, wherein the user speech signal for learning corresponds to the user speech for learning output by the external device and received by the microphone.

[0006] The one or more processors may also be configured to: obtain, in the voice recognition mode, information about a location where the user voice is uttered when the user voice signal is received, wherein the environmental information includes information about a location where the user voice is uttered; and identify, from the plurality of external devices, an external device for outputting a user voice for learning based on the information about a location where the user voice is uttered.

[0007] The one or more processors may also be configured to: obtain operation information of the external device when receiving the user voice signal in the voice recognition mode, wherein the environmental information includes the operation information; send a command for operating the external device corresponding to the operation information of the external device to the external device; and train the voice recognition model when the external device is operated according to the command for operating the external device and the user voice signal for learning is received.

[0008] The one or more processors may also be configured to: receive other noise signals corresponding to other noises that have occurred in the external device when the external device is operating according to a command for operating the external device, and train the speech recognition model based on the received user voice signal and the received other noise signals.

[0009] The one or more processors may also be configured to: in the learning mode, identify other noise signals corresponding to other noises around the electronic device when the user voice signal is received based on the environmental information, control the communication interface to send a command for controlling the external device to output the other noise to the external device, and during the period when the external device outputs the other noise according to the command for controlling the external device to output the other noise, receive the user voice signal and the other noise signal for learning, and train the voice recognition model based on the received user voice and the other noise for learning.

[0010] The electronic device may further include a sensor. The one or more processors may also be configured to: in the voice recognition mode, obtain object information located around the electronic device based on the sensing data of the sensor when receiving the user voice signal, wherein the environmental information includes the object information; based on receiving new sensing data from the sensor, identify whether the object information according to the new sensing data corresponds to the object information included in the environmental information; based on the object information according to the new sensing data corresponding to the object information included in the environmental information, enter the learning mode; and when operating in the learning mode, control the communication interface to send a command for the external device for controlling the output of the user voice for learning to the external device.

[0011] The one or more processors may be further configured to, when operating in the speech recognition mode, generate a text-to-speech (TTS) model based on the user speech received through the microphone, and obtain the user speech signal for learning based on the TTS model.

[0012] Each of the plurality of environmental information may include a confidence score regarding the result of the speech recognition, and based on the confidence score being greater than or equal to a threshold, include text corresponding to the result of the speech recognition.

[0013] The one or more processors may also be configured to: based on the electronic device entering the learning mode, identify any environmental information among the multiple environmental information stored in the memory whose confidence score is less than the threshold, obtain at least one of the multiple environmental information, wherein the at least one environmental information has the confidence score equal to or greater than the threshold and has a similarity with the environmental information equal to or greater than a threshold similarity; and obtain the user voice signal for learning based on the text included in the at least one environmental information among the multiple environmental information having the confidence score equal to or greater than the threshold.

[0014] The one or more processors may be further configured to increase repetition frequency information included in the environmental information based on the obtained environmental information corresponding to the environmental information stored in the memory.

[0015] The one or more processors may also be configured to: obtain at least one of the multiple environmental information based on the repetition number information included in the environmental information being equal to or greater than a threshold number, wherein the at least one environmental information has a confidence score of the result of the speech recognition equal to or greater than a threshold and has a similarity equal to or greater than a threshold similarity; obtain the user voice signal for learning based on the text included in the at least one environmental information, and wherein, as the repetition number information is increased, the threshold similarity is reduced.

[0016] The one or more processors may be further configured to cause the electronic device to enter a learning mode based on identifying a preset event.

[0017] A method for controlling an electronic device may include: when the electronic device is operating in a voice recognition mode, performing voice recognition by inputting a user voice signal from a microphone into a voice recognition model, the user voice signal corresponding to a received user voice received through the microphone; based on a result of the voice recognition, obtaining environmental information around the electronic device when the user voice is received; storing the obtained environmental information in a memory; when the electronic device is operating in a learning mode, identifying an external device for outputting a user voice for learning from a plurality of external devices based on the environmental information among a plurality of environmental information stored in the memory; sending a command for controlling the output of the user voice for learning to the external device; and training the voice recognition model for the received user voice for learning based on a user voice signal for learning received from the microphone, wherein the user voice signal for learning corresponds to the user voice for learning output by the external device and received by the microphone.

[0018] Acquiring the environmental information may include: acquiring information about a location where the user voice is emitted when the user voice signal is received in the voice recognition mode, wherein the environmental information includes information about the location where the user voice is emitted. Identifying the external device may include: identifying an external device for outputting the user voice for learning from the plurality of external devices based on the information about the location where the user voice is emitted.

[0019] Acquiring the environmental information may include: acquiring operation information of the external device when receiving the user voice signal in the voice recognition mode, wherein the environmental information includes the operation information. The sending may include: sending a command for operating the external device corresponding to the operation information of the external device to the external device. The training may include: training the voice recognition model when the external device is operated according to the command for operating the external device and the user voice signal for learning is received.

[0020] The method may further include receiving other noise signals corresponding to other noises that have occurred in the external device while the external device is operating according to a command for operating the external device, and training the speech recognition model based on the received user voice signal and the received other noise signals.

[0021] The method may also include: in the learning mode, identifying other noise signals corresponding to other noises around the electronic device when the user voice signal is received based on the environmental information, controlling the communication interface to send a command for controlling the external device to output the other noise to the external device, and during the period when the external device outputs the other noise according to the command for controlling the external device to output the other noise, receiving the user voice signal and the other noise signal for learning, and training the voice recognition model based on the received user voice and the other noise for learning.

[0022] The method may also include: in the voice recognition mode, obtaining object information located around the electronic device based on the sensing data of the sensor when receiving the user voice signal, wherein the environmental information includes the object information; based on receiving new sensing data from the sensor, identifying whether the object information according to the new sensing data corresponds to the object information included in the environmental information; based on the object information according to the new sensing data corresponding to the object information included in the environmental information, entering the learning mode; and when operating in the learning mode, controlling the communication interface to send a command for the external device for controlling the output of the user voice for learning to the external device.

[0023] The method may further include, when operating in the voice recognition mode, generating a text-to-speech (TTS) model based on the user voice received through the microphone, and obtaining the user voice signal for learning based on the TTS model.

[0024] Each of the plurality of environmental information may include a confidence score about the result of the speech recognition, and based on the confidence score being greater than or equal to a threshold, may include text corresponding to the result of the speech recognition. The method may also include: based on the electronic device entering the learning mode, identifying any environmental information of the plurality of environmental information stored in the memory whose confidence score is less than the threshold; obtaining at least one of the plurality of environmental information, wherein the at least one environmental information has the confidence score equal to or greater than the threshold and has a similarity with the environmental information equal to or greater than a threshold similarity; and obtaining the user voice signal for learning based on the text included in the at least one of the plurality of environmental information having the confidence score equal to or greater than the threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram illustrating an electronic device for receiving a user voice in a voice recognition mode according to one or more embodiments of the present disclosure.

[0026] Figure 2 is a diagram illustrating an external device for outputting a user voice for learning in a learning mode according to one or more embodiments of the present disclosure.

[0027] Figure 3 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0028] Figure 4 is a diagram illustrating a method for obtaining environmental information according to one or more embodiments of the present disclosure.

[0029] Figure 5 is a diagram illustrating a method for obtaining a user voice for learning based on environmental information according to one or more embodiments of the present disclosure.

[0030] Figure 6 is a diagram illustrating a method for outputting a user voice for learning according to one or more embodiments of the present disclosure.

[0031] Figure 7 is a diagram illustrating a method for training a speech recognition model according to one or more embodiments of the present disclosure.

[0032] Figure 8 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] The disclosed embodiments will be described in more detail with reference to the accompanying drawings.

[0034] The terms used herein and in the claims are general terms identified in view of the functions of the disclosed embodiments. However, these terms may change according to the intentions of technicians in the relevant fields, legal or technical interpretations, and the emergence of new technologies. In addition, in some cases, the applicant may select a term, in which case the term will be described in detail. Therefore, the terms used should be defined based on the meaning of the term and the content disclosed in the full text rather than the simple name of the term.

[0035] Expressions such as “have”, “may have”, “include”, “may include”, etc. indicate the existence of a corresponding number, function, operation, or component, and do not exclude the existence of additional features.

[0036] Expressions such as “at least one of A or B” and “at least one of A and B” should be understood to mean “A”, “B” or “A and B”.

[0037] As used herein, terms such as “first” and “second” may identify corresponding components regardless of order and / or importance, and are used to distinguish one component from another without limiting the components.

[0038] In addition, the description that one element (e.g., a first element) is “(operably or communicatively) coupled” to / is “(operably or communicatively) coupled to” or is “connected to” another element (e.g., the second element) should be interpreted to include both: the first element is directly coupled to the second element; and the first element is indirectly coupled to the second element via a third element.

[0039] Unless otherwise specified, singular expressions include plural expressions. It should be understood that terms such as "including" or "consisting of" are used herein to specify the presence of features, numbers, steps, operations, elements, components or combinations thereof, and do not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, elements, components or combinations thereof.

[0040] Terms such as "module", "unit" and "component" are used to indicate an element that performs at least one function or operation and can be implemented as hardware or software or a combination of hardware and software. Except when each of a plurality of "modules", "units", "components" etc. must be implemented in separate hardware, the components may be integrated in at least one module or chip and implemented in at least one processor.

[0041] In the following description, “user” may refer to a person using an electronic device or an artificial intelligence electronic device using an electronic device (e.g., an artificial intelligence electronic device).

[0042] The embodiments will be described in more detail with reference to the accompanying drawings.

[0043] Figure 1 is a diagram illustrating an electronic device for receiving a user voice in a voice recognition mode according to one or more embodiments of the present disclosure.

[0044] The electronic device 100 may operate in a speech recognition mode and a learning mode (or a learning data receiving mode or a speech recognition model learning mode).

[0045] The voice recognition mode is a mode for activating a configuration for voice recognition and performing a function corresponding to a user voice by recognizing a received user voice. For example, the electronic device 100 may perform voice recognition by inputting a user voice into a voice recognition model when operating in the voice recognition mode. Subsequently, the electronic device 100 may recognize a confidence score for the voice recognition.

[0046] According to one or more embodiments, when the confidence score of the voice recognition is greater than or equal to the threshold value (i.e., when the user voice is recognized), the electronic device 100 may perform a function corresponding to the user voice according to the voice recognition result. In addition, if the confidence score of the voice recognition is less than the threshold value (i.e., when the user voice is misrecognized), the electronic device 100 may ignore the voice recognition result.

[0047] In addition to the electronic device 100 at home, multiple external devices 210, 220, 230, 240 may be provided, and if noise according to the operation of each of the multiple external devices 210, 220, 230, 240 (e.g., buzzing sound, vibration sound (Brrr)) and pet noise (e.g., pet crying sound) are received together, the probability of misrecognition of the user's voice may increase.

[0048] According to one or more embodiments, when the confidence score regarding speech recognition is less than a threshold value, the electronic device 100 can not only obtain environmental information around the electronic device 100 when receiving the user voice by ignoring the speech recognition result, but also train the speech recognition model by reproducing the misrecognition environment of the user voice based on the obtained environmental information.

[0049] A speech recognition model trained in a reproduced misrecognition environment performs speech recognition on the user's speech taking into account the environment at home, and thus the probability of recognizing the user's speech (i.e., the probability that the confidence score of the speech recognition is greater than or equal to the threshold) may increase even when the user's speech and noise are received together.

[0050] Figure 2 is a diagram illustrating an external device for outputting a user voice for learning in a learning mode according to one or more embodiments of the present disclosure.

[0051] When a preset event is recognized or the current surrounding environment of the electronic device 100 is similar to the misrecognition environment of the user's voice, the electronic device 100 may enter the learning mode. However, this is merely an example, and the embodiment is not limited thereto. For example, the electronic device 100 may enter the learning mode according to a user command.

[0052] The learning mode may be a mode in which the electronic device 100 outputs a user voice for learning (or controls an external electronic device 200 at home to output a user voice for learning), and trains a voice recognition model by using the received user voice for learning as learning data.

[0053] According to one or more embodiments, the electronic device 100 may obtain a user voice for learning based on any one of a plurality of pieces of environmental information stored in the electronic device 100 when operating in the learning mode. Subsequently, the electronic device 100 may send a command for controlling the electronic device 100 itself to output a user voice for learning, or controlling an external device among a plurality of external devices 210, 220, 230, 240 located around the electronic device 100 to output a user voice for learning to any one of the external devices.

[0054] When a user voice for learning output by the electronic device 100 or the external device 200 is received, the electronic device 100 may train a voice recognition model based on the received user voice for learning.

[0055] Specifically, the electronic device 100 can identify the location where the user voice is emitted in an environment where the user voice is misrecognized based on any one of multiple environmental information, and identify any one of multiple external devices 210, 220, 230, and 240 for outputting the user voice for learning based on the location where the user voice is emitted.

[0056] refer to Figure 1 and Figure 2 , since the location where the user voice is emitted is adjacent to the location where the refrigerator 220 is set, the electronic device 100 according to one or more embodiments may send a command to the refrigerator 220 so that the refrigerator 220 emits the user voice for learning.

[0057] Here, any one of the environmental information may mean environmental information around the electronic device 100 obtained when the user voice is being received when the confidence score about the voice recognition result is less than a threshold value.

[0058] In addition, the electronic device 100 can operate each of the multiple external devices 210, 220, 23, and 240 to correspond to the operating state of each of the multiple external devices 210, 220, 23, and 240 when receiving the user voice based on any environmental information to reproduce the misrecognition environment.

[0059] Figure 3 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments of the present disclosure.

[0060] The electronic device 100 may include a speaker 110 , a microphone 120 , a communication interface 130 , a memory 140 , and one or more processors 150 .

[0061] The speaker 110 provided in the electronic device 100 may output a sound signal. The speaker 110 may include at least one speaker unit (or audio amplifier).

[0062] For example, the speaker 110 may be implemented with at least one full-range speaker designed to output sounds of almost all ranges of audible frequencies, or with a speaker array including a plurality of speaker units to reproduce different frequency bandwidths.

[0063] The microphone 120 provided in the electronic device 100 may receive a user voice and transmit the received user voice to the one or more processors 150. Subsequently, the one or more processors 150 may perform voice recognition by inputting the received user voice into a voice recognition model. For example, the one or more processors 150 may perform speech recognition to text (STT) on the user voice to perform voice recognition on the user voice.

[0064] The communication interface 130 provided in the electronic device 100 may receive various types of data and information in the communication with each of the plurality of external devices 210, 220, 230, 240. For example, the communication interface 130 may receive various types of data and information from an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a network hard disk) by using a communication method of AP-based Wi-Fi (wireless local area network), Bluetooth, Zigbee, wired / wireless local area network (LAN), WAN (wide area network), Ethernet, IEEE1394, High-Definition Multimedia Interface (HDMI), Universal Serial Bus (USB), Mobile High-Definition Link (MHL), Audio Engineering Society / European Broadcasting Union (AES / EBU), optical, coaxial communication.

[0065] Specifically, the communication interface 130 can receive operation information of each of the multiple external devices 210, 220, 230, and 240 under the control of one or more processors 150, and in the learning mode of the electronic device 100, under the control of one or more processors 150, send commands for controlling each of the multiple external devices 210, 220, 230, and 240 to perform corresponding operations based on the multiple operation information.

[0066] In addition, the communication interface 130 may transmit a command controlling output of a learning user voice to any one of the plurality of external devices 210 , 220 , 230 , 240 under the control of the one or more processors 150 in the learning mode of the electronic device 100 .

[0067] According to one or more embodiments, the memory 140 may store data. The memory 140 may be implemented as a memory embedded in the electronic device 100 or a memory detachable from the electronic device 100 according to the use of data storage.

[0068] For example, data for driving the electronic device 100 may be stored in a memory embedded in the electronic device 100, and data for upgrading the electronic device 100 may be stored in a memory detachable from the electronic device 100. The memory embedded in the electronic device 100 may be implemented as at least one of a volatile memory such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a synchronous dynamic random access memory (SDRAM) or a nonvolatile memory such as a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory such as a NAND flash memory or a NOR flash memory, a hard disk drive (HDD), or a solid state drive (SSD). In the case where the memory is detachably installed to the electronic device 100, the memory may be implemented as a memory card (such as compact flash (CF), secure digital (SD), micro secure digital (micro-SD), mini secure digital (mini-SD), extreme digital (xD), or multimedia card (MMC)) and an external memory (such as a universal serial bus (USB) memory connectable to a USB port).

[0069] According to one or more embodiments, the memory 140 may store at least one instruction for controlling the electronic device 100 or a computer program including the instruction.

[0070] One or more processors 150 control the overall operation of the electronic device 100 .

[0071] According to one or more embodiments, the processor 150 may be implemented with at least one of a digital signal processor (DSP), a microprocessor, and a time controller (TCON). The processor 150 is not limited thereto, and may include at least one or more of a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an advanced reduced instruction set computing (RISC) machine (ARM) processor, and an artificial intelligence (AI) processor, or may be defined as a corresponding term. The processor 150 may be implemented in a system on chip (SoC) type or a large scale integration (LSI) type or a field programmable gate array (FPGA) type in which a processing algorithm is built-in. The processor 150 may perform various functions by executing computer executable instructions stored in a memory.

[0072] The one or more processors 150 may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), an integrated many-core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors 150 may control one or any combination of other components of the electronic device and may perform operations related to communication or data processing. The one or more processors 150 may execute one or more programs or instructions stored in the memory. For example, the one or more processors 150 may execute a method by executing one or more instructions stored in the memory.

[0073] When the method includes multiple operations, the multiple operations may be performed by one processor or may be performed by multiple processors. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, all of the first operation, the second operation, and the third operation may be performed by the first processor, the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor), and the third operation may be performed by the second processor (e.g., an artificial intelligence dedicated processor).

[0074] One or more processors 150 may be implemented as a single-core processor including one core, or may be implemented as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When one or more processors 150 are implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include a processor internal memory, such as a cache memory and an on-chip memory, and a public cache shared by the multiple cores may be included in the multi-core processor. In addition, each of the multiple cores (or a portion of the multiple cores) included in the multi-core processor may independently read and execute program commands for implementing the method, and may read and execute program commands for implementing the method in association with all (or a portion) of the multiple cores.

[0075] When the method includes multiple operations, the multiple operations may be performed by one of the multiple cores included in the multi-core processor, or may be performed by multiple cores. For example, when the first operation, the second operation, and the third operation are performed by the method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first core included in the multi-core processor, the first operation and the second operation may be performed by the first core included in the multi-core processor, and the third operation may be performed by the second core included in the multi-core processor.

[0076] A processor may refer to a system on chip (SoC), a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor in which one or more processors and other electronic components are integrated, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the disclosed embodiments are not limited thereto.

[0077] According to one or more embodiments, the one or more processors 150 may perform voice recognition by inputting a user voice received through the microphone 120 to a voice recognition model.

[0078] The one or more processors 150 may obtain environmental information around the electronic device 100 during reception of the user's voice according to the voice recognition result.

[0079] Specific descriptions related to environmental information will refer to Figure 4 .

[0080] Figure 4 is a diagram illustrating a method for obtaining environmental information according to one or more embodiments of the present disclosure.

[0081] First, the electronic device 100 may include an environmental information storage module 10 for obtaining and storing environmental information, a learning user voice generation module 20 for generating a learning user voice, and a learning user voice output module 30 for controlling the electronic device 100 or an external device to output the generated learning user voice.

[0082] When the electronic device 100 operates in a voice recognition mode, the electronic device 100 according to one or more embodiments may perform voice recognition by inputting a user voice received through the microphone 120 to a voice recognition model.

[0083] The electronic device 100 may obtain the environment information around the electronic device 100 when receiving the user's voice according to the voice recognition result.

[0084] The environment information may include information about the environment around the electronic device 100 during the time when the user voice is emitted and received, the situation around the user who emitted the user voice.

[0085] Subsequently, the user voice generation module 20 for learning provided in the electronic device 100 can obtain the voice expected to be uttered by the user in the case of misrecognition of the user voice based on the environmental information as the user voice for learning. In addition, the user voice generation module 20 can obtain the noise generated in the case of misrecognition based on the environmental information.

[0086] The learning user voice output module 30 provided in the electronic device 100 may operate each device in the electronic device 100 or the external device in the same or similar manner as the misrecognition situation based on the environmental information, or may output the noise generated in the misrecognition situation through the electronic device 100 or the external device to reproduce the misrecognition situation. Subsequently, the learning user voice output module 30 may output the learning user voice obtained by the learning user voice generation module 20 in the reproduced misrecognition situation. In addition, if the current surrounding environment of the electronic device 100 is similar to the misrecognition situation, the learning user voice output module 30 may output the learning user voice obtained by the learning user voice generation module 20 through the electronic device 100 or the external device.

[0087] The electronic device 100 may train a voice recognition model by receiving a learning user voice output by the electronic device 100 or an external device.

[0088] In the following, reference Figure 4 , an example in which the environmental information storage module 10 obtains environmental information will be described.

[0089] For example, the environment information storage module 10 provided in the electronic device 100 may identify a confidence score regarding voice recognition based on a voice recognition result regarding a user voice.

[0090] As an example, if the confidence score of the speech recognition is less than the threshold value, the environment information storage module 10 may obtain the environment information around the electronic device 100 when the user's speech is being received as the misrecognition environment information, and store the misrecognition environment information. For example, when the confidence score value of the speech recognition is less than the threshold value (e.g., 0.5) in operation S410: Yes, in operation S420, the environment information storage module 10 may obtain the environment information when the user's speech is being received as the misrecognition environment information, and store it in the memory 140.

[0091] In addition, when the confidence score of the voice recognition is equal to or greater than the threshold, the environmental information storage module 10 can obtain the environmental information around the electronic device 100 when the user voice is being received as the recognition environmental information, and store the recognition environmental information. For example, when the confidence score value of the voice recognition is equal to or greater than the threshold (e.g., 0.5) in operation S410: No, in operation S430, the environmental information storage module 10 can obtain the environmental information when the user voice is being received as the recognition environmental information, and store the environmental information in the memory 140.

[0092] For example, the electronic device 100 may obtain a likelihood ratio of speech recognition, and may obtain a confidence score regarding speech recognition based on the obtained likelihood ratio.

[0093] For example, the electronic device 100 may obtain feature parameters by performing a series of noise removal, feature extraction, and normalization on the received user voice, obtain text with maximum likelihood for the feature parameters, and obtain a confidence score for voice recognition based on the likelihood ratio of the obtained text.

[0094] However, this is an example of the electronic device 100 obtaining the confidence score of the speech recognition, but is not limited thereto. The electronic device 100 may obtain the confidence score of the speech recognition by using various algorithms.

[0095] In the following, for the sake of ease of description, the environmental information obtained by the environmental information storage module 10 when the confidence score of speech recognition is less than a threshold may be referred to as misrecognized environmental information, and the environmental information obtained when the confidence score of speech recognition is greater than or equal to the threshold may be referred to as recognized environmental information.

[0096] According to one or more embodiments, the environmental information may include repetition number information, a confidence score, a time point (or date and time) when the user voice is received, a location where the user voice is emitted, a location where the electronic device 100 is located, operation information of the electronic device 100, user identification information, external devices located around it, operation information of the external devices, language information of the user voice (e.g., ko-KR, en-US), voice recognition results (e.g., text corresponding to the user voice), sound source information, and surrounding object information.

[0097] <Location where the user voice is emitted>

[0098] The environmental information storage module 10 can identify the user's location based on the intensity of the user's voice (e.g., the intensity of sound pressure in dB), the intensity of a signal received from the user's wearable device (not shown), and sensing data of sensors set in the electronic device 100 (e.g., a camera, a light detection and ranging (LiDAR) sensor).

[0099] For example, the environment information storage module 10 may identify the user's location based on the strength of the user's voice. For example, the environment information storage module 10 may identify that the user is located near (or adjacent to) the electronic device 100 when the strength of the user's voice is large, and identify that the user is located far away from the electronic device 100 when the strength of the user's voice is small.

[0100] The environmental information storage module 10 may communicate with the wearable device of the user, and may recognize that the stronger the strength of the signal received from the wearable device, the closer the user is to the electronic device 100 .

[0101] The environment information storage module 10 may also identify the user's location based on the sensing data of the sensor. For example, the environment information storage module 10 may identify the separation distance between the user and the electronic device 100 based on the sensing data, and identify the user's location based on the separation distance.

[0102] In addition, the environment information storage module 10 may include a map corresponding to the space in the network where the electronic device 100 is located, and may identify any one of the multiple subspaces included in the space as the location where the user voice is emitted based on the map and the identified user's location. Here, the subspace may include a space divided by the user, and an independent space surrounded by obstacles (e.g., walls, thresholds) or other entities (e.g., living room, kitchen, wardrobe).

[0103] For example, the environment information storage module 10 may identify any one of the plurality of subspaces where the user's voice is uttered based on the map and the identified location of the user.

[0104] Subsequently, as described below, one or more processors 150 may identify a device located in the same subspace as any one of the multiple external devices 210, 220, 230, 240, or one of the multiple external devices 210, 220, 230, 240, so as to identify a target device for outputting user voice, noise for learning.

[0105] For example, when the electronic device 100 and the user are located in the same subspace based on a map and the identified user's location, one or more processors 150 may identify the electronic device 100 or the electronic device 100 among multiple external devices 210, 220, 230, 240 as a target device for outputting user voice, noise for learning.

[0106] In another example, when the first external device 210 and the user are located in the same subspace based on the map and the recognized location of the user, the one or more processors 150 may identify the first external device 210 as a target device for outputting user voice, noise for learning.

[0107] <Location where electronic device is located and information about operation of electronic device>

[0108] According to one or more embodiments, the environment information storage module 10 may identify the location where the electronic device 100 receiving the user voice is located. For example, the environment information storage module 10 may identify any one of a plurality of subspaces included in a space in a house where the electronic device 100 is located. The location where the user voice is emitted and the location where the electronic device 100 receiving the user voice is located may be the same or different.

[0109] In addition, the environment information storage module 10 can obtain the operation information of the electronic device 100 at the time of receiving the user voice. The operation information of the electronic device 100 may include the operation mode of the electronic device 100 (e.g., standby mode, normal mode, low power mode, turbo mode) and the configuration information of the electronic device 100.

[0110] For example, when the electronic device 100 is implemented as a TV, the operation information of the electronic device 100 may include information on a channel (or content) being output by the electronic device 100, and volume (sound amplitude) information.

[0111] <User Identification Information>

[0112] The environment information storage module 10 may recognize user identification information based on feature information of the user's voice.

[0113] Here, the characteristic information of the user's voice may include speaking frequency information. The user's voice may have a unique speaking frequency according to the user's spoken language structure, age, gender, language, race. According to one or more embodiments, the environment information storage module 10 may identify user identification information corresponding to the characteristic information of the user's voice.

[0114] When the user identification information corresponding to the feature information of the user voice is identified, the environment information storage module 10 according to one or more embodiments may obtain the environment information including the identified user identification information. For example, if the environment information module 10 has received a user voice having feature information that is the same as / similar to the feature information of the currently received user voice, the environment information storage module 10 may identify the user identification information corresponding to the feature information of the previously received user voice.

[0115] As another example, when the user identification information corresponding to the feature information of the user voice is not recognized, the environment information storage module 10 may generate new user identification information and obtain the environment information including the new user identification information. For example, if there is no previously received user voice having feature information that is the same as / similar to the feature information of the currently received user voice, the environment information storage module 10 may generate new user identification information corresponding to the feature information of the currently received user voice.

[0116] <Nearby External Devices and Operation Information of External Devices>

[0117] The environment information storage module 10 may identify at least one external device 210 , . . . , 240 located around the electronic device 100 .

[0118] For example, when the first external device 210 can be operated through the Internet of Things (IoT), the environmental information storage module 10 can receive operation information of the first external device 210 at the time point of receiving the user's voice and information of the space where the first external device 210 is located by communicating with the first external device 210.

[0119] As another example, if the second external device 220 cannot be operated through the Internet of Things (IoT), the environmental information storage module 10 may receive operation information of the second external device 220 and the location of the second external device 220 at the time of receiving the user's voice based on sensing data of a sensor (e.g., a camera, a LiDAR sensor) set in the electronic device 100.

[0120] Here, the operation information may include whether the external device is operated, the operation mode of the external device (e.g., standby mode, low noise mode, normal mode, turbo mode), and setting information of the external device (e.g., sound volume information, vibration information).

[0121] <Language information and sound information of user speech>

[0122] According to one or more embodiments, the environment information storage module 10 can recognize the language information (e.g., ko-KR, en-US) of the user's voice, and store the sound source received by the microphone in the form of a file at the time of receiving the user's voice. Therefore, the sound source may include noise (e.g., voice noise or non-voice noise) in addition to the user's voice.

[0123] Here, speech noise may include noise generated by a person that is not in a language meaning transfer unit. For example, sneezing, hiccuping, breathing, snoring, laughing, crying, exclamation may be included in speech noise. Non-speech noise may refer to all kinds of noise except speech generated by a person. For example, noise according to the operation of an external device, vibration sound, music sound, telephone sound may be included in non-speech noise.

[0124] <Speech recognition result (text corresponding to the user's speech)>

[0125] The misrecognition environment information may not include text corresponding to the user voice, and the recognition environment information may also include text corresponding to the user voice.

[0126] For example, when the confidence score about speech recognition is greater than or equal to a threshold, the environment information storage module 10 can obtain text corresponding to the user's speech by using a speech recognition model (e.g., a speech-to-text (STT) model). Subsequently, the environment information storage module 10 can store the recognition environment information including the obtained text in the memory 140.

[0127] However, the embodiment is not limited thereto, and the misrecognition environment information may include some text in the text corresponding to the user voice. For example, the misrecognition environment information may include text corresponding to some words (or some word segments) whose confidence scores are greater than or equal to a threshold value among a plurality of words (or word segments) recognized in the entire user voice (i.e., included in the user voice).

[0128] For example, the environmental information storage module 10 can obtain the confidence score of each of the multiple words included in the user's voice, obtain texts corresponding to some words (or some word segments) among the multiple words (or some word segments) whose confidence scores are greater than or equal to a threshold, and can ignore the remaining words (or remaining word segments) whose confidence scores are less than the threshold.

[0129] <Information about surrounding objects>

[0130] According to one or more embodiments, the environment information storage module 10 may obtain object information located around the electronic device 100 at the time of receiving the user voice based on the sensing data of the sensor provided in the electronic device 100. Here, the object information may include whether a window is opened, whether a door is opened, whether a person exists, and whether a pet exists.

[0131] However, this is merely exemplary, and the object may include various types of objects that generate voice noise or non-voice noise in addition to the above-mentioned window, door, person, or pet.

[0132] <Information about the number of repetitions>

[0133] According to one or more embodiments, the environmental information storage module 10 can obtain environmental information including the time point (or date and time) of receiving the user voice, the location where the user voice is uttered, the location where the electronic device 100 is located, the operation information of the electronic device 100, user identification information, an external device located nearby, the operation information of the external device, the language information of the user voice (for example, ko-KR, en-US), the user voice, and the surrounding object information, and if the obtained environmental information is the same or similar to the environmental information stored in the memory 140 to a threshold value or more, the repetition number information of the environmental information stored in the memory 140 can be increased.

[0134] For example, if environmental information identical or similar to the currently obtained environmental information has been previously obtained, the environmental information storage module 10 may increase the number of repetitions information included in the environmental information previously obtained and stored in the memory 140 .

[0135] As another example, if no environmental information stored in the memory 140 that is equal to or similar to the obtained environmental information to a threshold value is identified, the environmental information storage module 10 may generate new environmental information from the obtained environmental information and store the information in the memory 140. For example, if environmental information that is the same as or similar to the currently obtained environmental information has not been previously obtained, the environmental information storage module 10 may generate the currently obtained environmental information as new environmental information.

[0136] Hereinafter, a method for generating a learning user voice based on environmental information stored in the memory 140 by the learning user voice generation module will be described.

[0137] Figure 5 is a diagram illustrating a method for obtaining a user voice for learning based on environmental information according to one or more embodiments of the present disclosure.

[0138] refer to Figure 5In operation S510 , the learning user voice generation module 20 may obtain the learning user voice based on the environmental information stored in the memory 140 .

[0139] Hereinafter, various embodiments of generating a learning text based on environmental information by the learning user voice generation module 20 will be described.

[0140] <Generate learning text by using the time point of receiving user speech>

[0141] For example, the user voice generation module 20 for learning may generate a learning text based on the time at which the user voice included in the environmental information is received.

[0142] For example, if the time point of receiving the user voice is 12 o'clock in the afternoon, the user voice generation module 20 for learning can generate texts that are frequently spoken in the corresponding time period, such as "Please lower the volume" or "Please turn off the TV" as learning texts, and when the time point of receiving the user voice is 6 o'clock in the morning, the learning user voice generation module 20 can generate the sentence "Please pay attention to today's schedule" that is frequently spoken in the corresponding time period as learning texts.

[0143] <Generate learning text by using the location where the user's voice is emitted>

[0144] For example, the learning user voice generation module 20 may obtain the learning text based on the place where the user voice included in the environmental information is uttered.

[0145] For example, when the place where the user voice is uttered is the living room, the user voice generation module 20 for learning can generate "Please turn on the TV" (which is a sentence frequently spoken in the living room) as a learning text, and when the place where the user voice is uttered is the kitchen, the learning user voice generation module 20 can generate "Please tell me the recipe" (which is a sentence frequently spoken in the kitchen) as a learning text.

[0146] <Generating a learning text using operation information of an electronic device and a location where the electronic device is located>

[0147] For example, the learning user voice generation module 20 may acquire the learning text based on the location where the electronic device 100 is located included in the environmental information.

[0148] For example, when the place where the electronic device 100 is located is a dressing room, the learning user voice generation module 20 may generate a learning text of “Please turn on the dehumidification mode” which is a sentence frequently spoken in a dressing room.

[0149] The learning user voice generation module 20 may obtain a learning text based on operation information of the electronic device 100 included in the environmental information.

[0150] For example, if the channel information being output by the electronic device 100 is a sports channel based on the operation information of the electronic device 100, the user voice generation module 20 for learning can generate "Please notify the current score" (which is a sentence frequently spoken when watching sports channels) as a learning text.

[0151] As another example, if the electronic device 100 is in standby mode based on operation information of the electronic device 100, the user voice generation module 20 for learning may generate a sentence such as “Please turn on the TV” that is frequently uttered in the standby mode of the electronic device 100 as a learning text.

[0152] <Generating a learning text by using an external device located nearby and operation information of the external device>

[0153] For example, the learning user voice generation module 20 may obtain the learning text based on operation information of an external device located near the electronic device 100 included in the environmental information.

[0154] For example, if the user voice generation module 20 for learning is in the cooling mode of the external device based on the operation information of the external device, the user voice generation module 20 for learning can generate a learning text of "Please increase the cooling temperature" (which is a sentence frequently spoken in the cooling mode of the external device).

[0155] As another example, if the external device is in turbo mode based on the operation information of the external device, the user voice generation module 20 for learning may generate "Please operate in low noise mode" (which is a sentence frequently spoken in the turbo mode of the external device) as a learning text.

[0156] In the above example, the user voice generation module 20 for learning may consider the time point (or date and time) of the user voice included in the receiving environment information, the location where the user voice is uttered, the location where the electronic device 100 is located, the operation information of the electronic device 100, the external device located nearby, and the operation information of the external device to generate the learning text, or may consider the information comprehensively to generate the learning text.

[0157] On the other hand, the user voice generation module 20 for learning according to one or more embodiments may recognize frequently spoken sentences based on text corresponding to the user voice included in the recognition environment information.

[0158] <Generate learning text by using speech recognition results>

[0159] For example, the learning user voice generation module 20 may obtain a learning text based on a voice recognition result included in the environmental information.

[0160] For example, the user voice generation module 20 for learning may identify recognized environmental information having a similarity greater than or equal to a threshold value among a plurality of pieces of environmental information including both misrecognized environmental information and recognized environmental information.

[0161] Then, the learning user voice generating module 20 may recognize the voice recognition result (ie, the text corresponding to the user voice) included in the recognized recognition environment information as the learning text.

[0162] For example, the user voice generation module 20 for learning can obtain a feature vector corresponding to each of multiple environmental information, and can identify at least one recognized environmental information from the multiple environmental information based on the multiple feature vectors, which has a similarity equal to or greater than a threshold with the misrecognized environmental information.

[0163] Here, the user voice generation module 20 for learning may obtain a feature vector corresponding to each of the multiple pieces of environmental information in the vector space, and may identify at least one piece of recognized environmental information having a similarity equal to or greater than a threshold value with any piece of misrecognized recognized environmental information by using a function indicating a value where the distance between the multiple feature vectors is 0 to 1. However, this is merely an example, and the learning user voice generation module 20 may use various similarity recognition functions and similarity recognition algorithms.

[0164] The learning user voice generating module 20 may obtain a learning text based on a voice recognition result included in the recognized at least one piece of recognition environment information.

[0165] According to one example, the environmental information may include repetition number information, and the learning user voice generation module 20 may reduce the threshold as the repetition number information included in the misrecognized environmental information increases, and may reduce the threshold when at least one piece of recognized environmental information having a similarity equal to or greater than the reduced threshold is identified.

[0166] For example, the user speech generation module 20 for learning may identify recognition environment information within a threshold distance (e.g., 0.7) from the misrecognized environment information in the vector space (i.e., recognition environment information having a similarity greater than or equal to a threshold value compared to the misrecognized environment information) as recognition environment information having a similarity with the corresponding misrecognized environment information having a threshold value or greater.

[0167] In addition, the learning user voice generation module 20 can reduce the threshold distance (i.e., reduce the threshold similarity) as the number of repetitions included in the corresponding misrecognized environmental information increases, and can identify the recognized environmental information within the reduced threshold distance (e.g., 0.5) in the vector space as recognized environmental information having a similarity equal to or greater than the threshold.

[0168] For example, when the number of repetitions included in the misrecognition environment information is greater than or equal to a threshold number of times, the learning user voice generation module 20 may obtain at least one piece of recognition environment information from multiple pieces of environment information that has a confidence score about the voice recognition result greater than or equal to a threshold (i.e., recognition environment information) and has a similarity greater than or equal to a threshold similarity with the misrecognition environment information, and obtain the learning user voice based on the text included in the at least one piece of recognition environment information. The learning user voice generation module 20 may reduce the threshold similarity as the number of repetitions included in the misrecognition environment information increases, and obtain at least one piece of recognition environment information from the multiple pieces of environment information based on the reduced threshold similarity.

[0169] The learning user voice generating module 20 may obtain a learning text based on some words included in the misrecognition environmental information.

[0170] For example, the misrecognition environment information includes text corresponding to some words (or some word segments) among multiple words (or word segments) recognized in the entire user voice whose confidence scores are greater than or equal to a threshold, and the learning user voice generation module 20 can obtain the learning text by using some words.

[0171] For example, if some of the words included in the misrecognized environmental information are "dehumidification", the learning user voice generation module 20 can add the remaining words (such as "please turn on the mode") to "dehumidification", and can obtain "Please turn on the dehumidification mode" as the learning text.

[0172] As another example, if part of the words included in the misrecognized environmental information is "today's schedule", the user voice generation module 20 for learning can obtain, for example, "Please let me know today's schedule" or "What is today's schedule" as a learning text by adding, for example, "Please let me know", "What is".

[0173] <Using feature information of user voice to generate user voice for learning corresponding to learning text>

[0174] In operation S520, the learning user voice generation module 20 may generate a learning user voice corresponding to the learning text by using a TTS model based on feature information of the user voice included in the environmental information.

[0175] For example, the learning user voice generation module 20 may generate a learning user voice corresponding to a learning text by using a TTS model generated based on feature information of the user voice, so that the learning user voice may have a speaking frequency similar to that of the user voice.

[0176] For example, when it is recognized that the user's gender is male and the age is middle-aged based on feature information of the user's voice, the TTS model may generate a learning user voice corresponding to the learning text by using the voice of a middle-aged male.

[0177] After S520, when the electronic device 100 operates in the learning mode, the user voice generation module 20 for learning may reproduce the noise generated in the surrounding environment of the electronic device 100 when receiving the user voice based on the misrecognition environment information (ie, reproduce the misrecognition situation).

[0178] <Identifying Noise by Using Location Where Electronic Device is Located and Operation Information of Electronic Device>

[0179] First, the user voice generation module 20 for learning can control the electronic device 100 based on the operation information of the electronic device 100 included in the environmental information so that the electronic device 100 operates the same or similar to the time during which the user voice is received. Since the electronic device 100 operates in the same or similar manner as the time when the user voice is received, the noise according to the operation of the electronic device 100 can be reproduced.

[0180] For example, if the electronic device 100 outputs content to the sound volume 11 at the time of receiving the user voice, the learning user voice generation module 20 may control the electronic device 100 so that the electronic device 100 outputs content to the sound volume 11 .

[0181] As another example, if the electronic device 100 is operating in the low noise mode at the time when the user voice is received, the learning user voice generation module 20 may control the electronic device 100 to operate in the low noise mode.

[0182] <Reproduction of External Device Located Nearby and Operation Information of External Device>

[0183] The user voice generation module 20 for learning according to one or more embodiments may send a command for controlling an external device to an external device based on operation information of the external device included in environmental information, so that the external device operates in the same or similar manner as receiving the user voice.

[0184] Since the external device operates according to the command in the same or similar manner as when the user's voice is received, noise according to the operation of the external device may be reproduced.

[0185] For example, if the external device operates in the dehumidification mode at the time when the user voice is received, the learning user voice generation module 20 may transmit a command for controlling the external device to operate in the dehumidification mode to the external device.

[0186] As another example, if the external device operates in the standby mode at the time of receiving the user voice, the learning user voice generation module 20 may transmit a command for controlling the external device to operate in the standby mode to the external device.

[0187] <Reproducing noise by using a sound source>

[0188] According to one or more embodiments, the user voice generation module 20 for learning can obtain noise information (for example, including voice noise or non-voice noise) at the time of receiving the user voice based on the sound source included in the stored environmental information, and can control the electronic device 100 or the external device to output noise in operation S530.

[0189] For example, since the electronic device 100 or the external device outputs noise obtained from a sound source, the noise generated in the surrounding environment at the time when the user voice is received may be reproduced.

[0190] The one or more processors 150 may identify a target device that outputs a learning user voice and noise among the electronic device 100 and a plurality of external devices 210, ..., 240 located around the electronic device 100. Then, the one or more processors 150 may send a command for controlling the target device to output the learning user voice and noise to the target device. Figure 6 Provide a detailed description thereof.

[0191] Figure 6 is a diagram illustrating a method for outputting a user voice for learning according to one or more embodiments of the present disclosure.

[0192] refer to Figure 6 In operation S510, the user voice output module 30 for learning may detect a preset event and identify whether it can be operated in the learning mode. For example, when a preset event is detected, the learning user voice output module 30 may enter the learning mode. When the electronic device 100 enters the learning mode, the learning user voice output module 30 may Figure 5 The operation of S530 reproduces the noise generated in the surrounding environment of the electronic device 100 when receiving the user voice based on the misrecognition environment information.

[0193] Here, the preset event can be one of the following events: an event of entering a time zone set by the user, an event of entering a time zone corresponding to the time point of receiving the user voice included in the pre-stored environmental information, an event of the user being detected as going out, an event of an object located around the electronic device 100 corresponding to the object information included in the environmental information obtained at the time of receiving the user voice.

[0194] For example, when entering an event set by the user at 1 pm is detected, the learning user voice output module 30 may determine to operate in the learning mode.

[0195] In addition, when an event of entering 3 p.m. on Friday is detected, the learning user voice output module 30 may determine to operate in the learning mode, wherein 3 p.m. on Friday is a time zone corresponding to a time point of receiving a user voice included in preset environmental information, wherein the number of times information included in the pre-stored environmental information is greater than or equal to a threshold number of times (e.g., 5 times). In addition, when it is detected that the user voice included in the pre-stored environmental information is greater than or equal to a threshold number of times (e.g., five times), the user voice output module 30 for learning may be determined to operate in the learning mode.

[0196] Furthermore, when an event that no user is at home is detected based on the sensing data of the sensor, the user voice output module 30 for learning may be determined to operate in the learning mode.

[0197] Furthermore, when the current state of an object located around the electronic device 100 corresponds to object information included in pre-stored environmental information, the learning user voice output module 30 may determine to operate in the learning mode.

[0198] For example, when the window located around the electronic device 100 is open and the door is closed based on the sensing data of the sensor, the object information included in the pre-stored environmental information is in an open state, and when the door is closed, the learning user voice output module 30 can determine to operate in the learning mode.

[0199] It is also possible to determine to operate in the learning mode in a case where the current user is detected from the user terminal as going out, traveling, or going to work.

[0200] Subsequently, in operation S520, the user voice output module 30 for learning may identify a target device for comparing a location where the user voice is uttered and a location where the electronic device 100 is located when the electronic device 100 is operating in the learning mode, or comparing a location where the user voice is uttered and a location where an external device is located, to identify a target device for outputting the learning user voice.

[0201] For example, if the place where the user voice is uttered is the same as the place where the electronic device 100 is located based on pre-stored environmental information, the user voice output module 30 for learning may recognize the electronic device 100 as a target device.

[0202] In another example, if the place where the user voice is uttered and the place where the external device is located are the same place based on pre-stored environmental information, the user voice output module 30 for learning may recognize the external device as a target device.

[0203] In addition, the user voice output module 30 for learning can identify any one of the multiple external devices as a target device based on the location where the user voice is emitted and the location where each of the multiple external devices is located based on pre-stored environmental information.

[0204] Here, the location where any one of the external devices identified as the target device is located may be the same location as the location where the user voice is uttered, or a location that is most adjacent to the location where the user voice is uttered.

[0205] In addition, the learning user voice output module 30 can identify a robot device (e.g., a robot cleaner) that can be moved by using a driver among multiple external devices as a target device. According to one or more embodiments, the user voice output module 30 for learning can send a command for moving the user voice to the place where it is emitted to the robot device. According to one or more embodiments, the robot device that receives the command can output the user voice for learning after moving to the place where the user voice is emitted. Here, the driver may include wheels for moving (or driving) the robot device and wheel drive motors for rotating the wheels.

[0206] Subsequently, in operation S530, the learning user voice output module 30 may transmit a command for controlling output of the learning user voice and noise to the target device.

[0207] However, this is an example, and the learning user voice output module 30 may transmit a command for controlling output of the learning user voice to the target device, and control the electronic device 100 to output noise through the speaker 110 of the electronic device 100. The learning user voice output module 30 may transmit a command for controlling output of the user voice for learning to the target device, and may transmit a command for controlling output noise to an external device.

[0208] Subsequently, in operation S540, the learning user voice output module 30 may receive the learning user voice and noise output by the target device, and the one or more processors 150 may train a voice recognition model by using the received learning user voice and noise.

[0209] Figure 5 The user voice generation module 20 for learning may obtain a plurality of user voices for learning, rather than a single learning user voice. The learning user voice output module 30 may send a command for controlling the target device to sequentially output a plurality of user voices for learning to the target device.

[0210] According to one or more embodiments, the one or more processors 150 may obtain a confidence score for speech recognition by inputting a plurality of learning user voices received through the microphone 120 into the trained speech recognition model. Subsequently, when the obtained confidence score is equal to or greater than a threshold, the one or more processors 150 may adopt the trained speech recognition model and perform speech recognition using the trained speech recognition model in the speech recognition mode.

[0211] The detailed description will refer to Figure 7 .

[0212] Figure 7 is a diagram illustrating a method for training a speech recognition model according to one or more embodiments of the present disclosure.

[0213] In addition to recognizing a preset event, when the current surrounding environment of the electronic device 100 is similar to a misrecognition environment for the user's voice, the electronic device 100 may enter a learning mode.

[0214] refer to Figure 7 , the learning user voice output module 30 may obtain current surrounding environment information of the electronic device 100 by using sensing data of the sensor.

[0215] Subsequently, when any one misrecognized environmental information similar to the current surrounding environmental information is identified from among a plurality of misrecognized environmental information, the learning user voice output module 30 may enter a learning mode.

[0216] For example, the user voice output module 30 for learning can obtain current surrounding environment information including object information located around the electronic device 100 (e.g., whether a window is open, whether a door is open, whether a person exists, whether a pet exists) based on the sensing data.

[0217] Subsequently, the learning user voice output module 30 may obtain any one misrecognition environment information based on the object information included in each of the plurality of misrecognition environment information and the object information included in the current surrounding environment information.

[0218] Subsequently, the user voice output module 30 for learning may enter a learning mode, and may output a user voice for learning through any one of the plurality of external devices or the electronic device 100 (i.e., through a target device) based on any one of the misrecognition environment information. In addition, the user voice output module 30 for learning may reproduce the noise generated in the surrounding environment of the electronic device 100 when any one of the misrecognition environment information is obtained by using the electronic device 100 and at least one of the plurality of external devices.

[0219] Subsequently, the voice recognition model may receive noise output by at least one of a plurality of external devices and a user voice for learning output by the target device, and perform learning based on the received user voice for learning and the received noise.

[0220] The voice recognition model may perform learning by using the user voice received for learning in the reproduced misrecognition situation, so that learning may be performed using a customized voice recognition model for the indoor environment in which the electronic device 100 is located.

[0221] For ease of description, operations of the environment information storage module 10 , the learned user voice generation module 20 , and the learned user voice output module 30 are described separately, but this is an example and may be implemented by operations of one or more processors 150 .

[0222] Functions related to the artificial intelligence model including the speech recognition model are operated by one or more processors 150 and memory 140 of the electronic device 100 .

[0223] The processor may be composed of one or more processors. The one or more processors may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU), but is not limited to the aforementioned examples of processors.

[0224] The CPU is a general-purpose processor that can perform AI calculations in addition to general-purpose calculations, and can efficiently execute complex programs through a multi-layer cache structure. The CPU is advantageous for serial processing that can organically associate the previous calculation result with the next calculation result through sequential calculations. In addition to the case of the CPU described above, the general-purpose processor is not limited to the aforementioned example.

[0225] The GPU is a processor for batch operations such as floating point calculations for graphics processing, and can perform large-capacity operations (batch operations) in parallel by integrating large-capacity cores. In particular, compared with the CPU, the GPU may be advantageous in parallel processing such as convolution calculations. In addition, the GPU can be used as a co-processor to supplement the functions of the CPU. Except when the processor is designated as the GPU described above, the processor for batch operations is not limited to the above examples.

[0226] NPU is a processor dedicated to AI operations using AI neural networks, and each layer that makes up the AI ​​neural network can be implemented as hardware (e.g., crystalline silicon). Since the NPU is designed to be specialized according to the company's required specifications, the degree of freedom is lower than that of the CPU or GPU, but it can effectively process artificial intelligence operations for requesting companies. Processors dedicated to artificial intelligence computing can be implemented in various forms such as tensor processing units (TPUs), intelligent processing units (IPUs), and visual processing units (VPUs). The artificial intelligence processor is not limited to the above examples, except for the case where it is designated as the NPU described above.

[0227] One or more processors may also be implemented using a system on a chip (SoC). In addition to one or more processors, the SoC may also include a memory, a processor, and a network interface, such as a bus for data communication between the processor and the memory.

[0228] When a plurality of processors are included in the SoC included in the electronic device 100, the electronic device 100 may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model) by using some of the plurality of processors. For example, the electronic device 100 may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator dedicated to artificial intelligence operations (such as convolution operations, matrix multiplication operations) among the plurality of processors. However, this is merely an embodiment, and operations related to artificial intelligence may be processed by using a CPU and a general-purpose processor.

[0229] In addition, the electronic device 100 can perform operations related to functions related to artificial intelligence by using multiple cores (e.g., dual cores, quad cores) included in one processor. In particular, the electronic device 100 can perform artificial intelligence operations such as convolution operations and matrix multiplication operations in parallel using the multiple cores included in the processor.

[0230] One or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0231] Here, providing by learning means making an AI model of predefined operating rules or desired features by applying a learning algorithm to a plurality of learning data. The learning may be performed in the device itself that executes the AI ​​according to one or more embodiments and / or may be implemented by a separate server / system.

[0232] The AI ​​model may consist of multiple neural network layers. Each layer has at least one weight value, and the layer operation is performed by the calculation of the previous layer and the operation of multiple weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0233] A learning algorithm is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to enable the target device to perform determination or prediction by itself. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0234] Figure 8 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments of the present disclosure.

[0235] The control method of the electronic device may include, at operation S810, performing voice recognition by inputting a received user voice into a voice recognition model when the electronic device operates in a voice recognition mode.

[0236] The method may include, at operation S820, obtaining, according to the voice recognition result, environmental information around the electronic device when receiving the user voice.

[0237] While the electronic device is operating in the learning mode, the method may include, at operation S830, identifying an external device for outputting a user voice for learning among a plurality of external devices located around the electronic device based on any one of a plurality of environmental information stored in a memory.

[0238] The method may include transmitting a command for controlling output of a user voice for learning to an external device at operation S840.

[0239] Based on receiving the user voice for learning output by the external device, the method may include, at operation S850, learning a voice recognition model based on the received user voice for learning.

[0240] Obtaining environmental information in operation S820 may include: obtaining information about a location where a user voice is emitted when receiving a user voice in a voice recognition mode, and storing environmental information including information about a location where the user voice is emitted in a memory, and identifying an external device in operation S830 may include: in a learning mode, identifying an external device for outputting a user voice for learning among a plurality of external devices based on information about a location where a user voice is emitted included in one of the environmental information.

[0241] The obtaining of environmental information in operation S820 may include: obtaining, in a voice recognition mode, operation information of the external device when receiving a user voice, and the sending in operation S840 may include: in a learning mode, sending to the external device a command for operating the external device to correspond to the operation information of the external device included in the one environmental information, and the learning in operation S850 may include: when the external device is operating according to the command, receiving noise generated in the external device and a user voice output by the external device for learning, and learning a voice recognition model based on the received user voice and the received noise.

[0242] The identifying of the external device in operation S830 may include: in a learning mode, identifying the noise around the electronic device when receiving a user voice based on the one environmental information, and the sending in operation S840 may include: sending a command for controlling the external device to output noise to the external device, and the learning in operation S850 may include: during the period when the external device outputs noise according to the command, receiving a user voice for learning and noise output by the external device, and learning a voice recognition model based on the received user voice and noise.

[0243] Obtaining environmental information in operation S820 may include: obtaining object information located around the electronic device based on sensing data of the sensor when receiving a user voice in a voice recognition mode, and storing environmental information including the object information, and the control method may include: based on receiving new sensing data from the sensor, identifying whether the object information according to the new sensing data corresponds to the object information included in each of multiple environmental information, and entering a learning mode based on the object information according to the new sensing data corresponding to the object information included in the one environmental information.

[0244] The method may further include, when operating in the voice recognition mode, generating a text-to-speech (TTS) model based on a user voice received through a microphone, and obtaining the user voice for learning based on the TTS model.

[0245] Each of the plurality of environmental information may include a confidence score regarding the speech recognition result, and based on the confidence score being greater than or equal to a threshold, include text corresponding to the speech recognition result.

[0246] The method may further include increasing repetition frequency information included in one of the environmental information based on the obtained environmental information corresponding to one of the environmental information.

[0247] Obtaining environmental information in operation S820 may include: obtaining at least one piece of environmental information among multiple pieces of environmental information having a confidence score of a speech recognition result equal to or greater than a threshold and a similarity equal to or greater than a threshold similarity based on repetition number information included in one piece of environmental information being equal to or greater than a threshold number, and identifying an external device in operation S830 may include: obtaining a user voice for learning based on text included in at least one piece of environmental information.

[0248] Various embodiments are applicable not only to electronic devices but also to all types of electronic devices capable of voice recognition.

[0249] The various example embodiments described above can be implemented in a recordable medium using software, hardware, or a combination of software and hardware, and the recordable medium can be read by a computer or a device similar to a computer. In some cases, the embodiments described herein can be implemented by the processor itself. According to software implementation, embodiments such as the processes and functions described herein can be implemented with separate software modules. Each module in the above-mentioned software modules can perform one or more of the functions and operations described herein.

[0250] The computer instructions for executing the processing operations of the electronic device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in this non-transitory computer-readable medium, when executed by the processor of the specific device described above, cause the specific device to execute the processing operations of the electronic device 100 according to the various embodiments described above.

[0251] A non-transitory computer-readable medium may refer to, for example, a medium that stores data and can be read by a device, such as a register, a cache, a memory. For example, the various applications, instructions, or programs described above may be stored in a non-transitory computer-readable medium (e.g., a compact disc (CD), a digital versatile disc (DVD), a hard disk, a Blu-ray disc, a universal serial bus (USB), a memory card, a read-only memory (ROM)), and may be provided.

[0252] Although exemplary embodiments have been shown and described, the disclosed embodiments are not limited to the specific embodiments described above. It will be appreciated by those skilled in the art that various changes in form and details may be made therein without departing from the true spirit and full scope of the disclosed embodiments (including the appended claims and their equivalents).

Claims

1. An electronic device, comprising: speaker; microphone; Communication interface; a memory storing at least one instruction; as well as one or more processors connected to the speaker, the microphone, the communication interface and the memory, Wherein, the one or more processors are configured to: When the electronic device operates in a voice recognition mode, voice recognition is performed by inputting a user voice signal from the microphone into a voice recognition model, the user voice signal corresponding to the user voice received through the microphone, According to the result of the voice recognition, obtaining the environment information around the electronic device when receiving the user voice, storing the obtained environmental information in the memory, identifying, while the electronic device is operating in a learning mode, an external device for outputting a user voice for learning from among a plurality of external devices based on the environmental information among a plurality of environmental information stored in the memory, The communication interface is controlled to transmit a command for controlling output of a user voice for learning to the external device.

2. The electronic device according to claim 1, wherein: The one or more processors train the speech recognition model based on the user voice signal for learning received from the microphone, wherein the user voice signal for learning corresponds to the user voice for learning output by the external device and received by the microphone.

3. The electronic device according to claim 1, wherein: The one or more processors are further configured to: obtaining, in the voice recognition mode, information about a location where the user voice is emitted when the user voice signal is received, wherein the environmental information includes information about a location where the user voice is emitted, and Based on the information about the place where the user voice is uttered, an external device for outputting the user voice for learning is identified from among the plurality of external devices.

4. The electronic device as claimed in claim 2, wherein: The one or more processors are further configured to: obtaining, in the voice recognition mode, operation information of the external device when receiving the user voice signal, wherein the environmental information includes the operation information, sending a command for operating the external device corresponding to the operation information of the external device to the external device, and When the external device is operated according to a command for operating the external device and the user voice signal for learning is received, the voice recognition model is trained.

5. The electronic device as claimed in claim 4, wherein: The one or more processors are further configured to: receiving other noise signals corresponding to other noises that have occurred in the external device while the external device is operating according to a command for operating the external device, and The speech recognition model is trained based on the received user speech signal and other received noise signals.

6. The electronic device as claimed in claim 1, wherein: The one or more processors are further configured to: in the learning mode, identifying other noise signals corresponding to other noises around the electronic device when the user voice signal is received based on the environmental information, controlling the communication interface to send a command for controlling the external device to output the other noise to the external device, and During the period when the external device outputs the other noise according to a command for the external device for controlling the external device to output the other noise, a user voice signal and the other noise signal for learning are received, and the voice recognition model is trained based on the received user voice and the other noise for learning.

7. The electronic device as claimed in claim 1, further comprising: sensor, Wherein, the one or more processors are further configured to: In the voice recognition mode, object information located around the electronic device is obtained based on the sensing data of the sensor when the user voice signal is received, wherein the environmental information includes the object information, based on receiving new sensing data from the sensor, identifying whether the object information according to the new sensing data corresponds to the object information included in the environmental information, entering the learning mode based on the object information according to the new sensing data corresponding to the object information included in the environmental information, and When operating in the learning mode, the communication interface is controlled to transmit a command for the external device for controlling output of a user voice for learning to the external device.

8. The electronic device as claimed in claim 2, wherein: The one or more processors are further configured to: When operating in the speech recognition mode, generating a text-to-speech (TTS) model based on user speech received via the microphone, and The user voice signal for learning is obtained based on the TTS model.

9. The electronic device as claimed in claim 2, wherein: Each of the plurality of environmental information includes a confidence score regarding the result of the speech recognition, and includes text corresponding to the result of the speech recognition based on the confidence score being greater than or equal to a threshold.

10. The electronic device according to claim 9, wherein: The one or more processors are further configured to: Based on the electronic device entering the learning mode, identifying any environmental information among the plurality of environmental information stored in the memory whose confidence score is less than the threshold value, obtaining at least one piece of environmental information from the plurality of environmental information, wherein the at least one piece of environmental information has the confidence score equal to or greater than the threshold and has a similarity with the environmental information equal to or greater than a threshold similarity, and The user voice signal for learning is obtained based on text included in the at least one environmental information among the plurality of environmental information having the confidence score equal to or greater than the threshold.

11. The electronic device according to claim 2, wherein: The one or more processors are further configured to increase repetition frequency information included in the environmental information based on the obtained environmental information corresponding to the environmental information stored in the memory.

12. The electronic device according to claim 11, wherein: The one or more processors are further configured to: Based on the repetition number information included in the environmental information being equal to or greater than a threshold number, obtaining at least one environmental information among the plurality of environmental information, wherein the at least one environmental information has a confidence score of the result of the speech recognition equal to or greater than a threshold and has a similarity equal to or greater than a threshold similarity, obtaining the user voice signal for learning based on the text included in the at least one environmental information, and Herein, as the repetition number information is increased, the threshold similarity is reduced.

13. The electronic device according to claim 1, wherein: The one or more processors are further configured to cause the electronic device to enter a learning mode based on identifying a preset event.

14. A method for controlling an electronic device, the method comprising: performing speech recognition by inputting a user speech signal from a microphone into a speech recognition model when the electronic device is operated in a speech recognition mode, the user speech signal corresponding to a received user speech received through the microphone; According to the result of the speech recognition, obtaining the environment information around the electronic device when receiving the user speech; The obtained environmental information is stored in the memory, identifying, while the electronic device is operating in a learning mode, an external device for outputting a user voice for learning from among a plurality of external devices based on the environmental information among a plurality of environmental information stored in the memory; A command for controlling output of the user voice for learning is transmitted to the external device.

15. The method of claim 14, further comprising: Based on a user voice signal for learning received from the microphone, the voice recognition model is trained for the received user voice for learning, wherein the user voice signal for learning corresponds to the user voice for learning output by the external device and received by the microphone.

Images