Terminal device and control method thereof

Abstract

A terminal device is provided. The terminal device includes a communication interface and a processor. The processor is configured to: receive performance information of one or more other terminal devices from each of one or more other terminal devices; identify an edge device for determining a terminal device to perform speech recognition based on the performance information received from each of the one or more other terminal devices; receive information associated with reception quality from one or more other terminal devices that receive sound waves including a trigger word based on the terminal device being identified as an edge device; determine a terminal device that acquires sound waves for speech recognition based on the received information associated with reception quality; and send a command to the determined terminal device to send the acquired sound waves for speech recognition to an external speech recognition device.

Description

Technical Field

[0001] The present disclosure relates to a terminal device and a control method thereof. More specifically, the present disclosure relates to a terminal device for determining a device for acquiring a sound wave generated by a user for recognizing a voice from among a plurality of terminal devices and a control method thereof. Background Art

[0002] In recent years, artificial intelligence (AI) systems that realize human intelligence have been applied to various fields. Unlike traditional rule-based intelligent systems, AI systems are systems that learn, judge, and become intelligent. As the application of AI systems improves the recognition rate, understanding or prediction of user preferences can be performed more accurately. Therefore, traditional rule-based intelligent systems are gradually replaced by AI systems based on deep learning.

[0003] AI technology consists of machine learning (e.g., deep learning) and basic techniques that utilize machine learning.

[0004] Machine learning is an algorithmic technology that can classify or learn the characteristics of input data without the need for additional information or intervention. The basic technology is a technology that uses machine learning algorithms (such as deep learning) to simulate the recognition and judgment functions of the human brain. Machine learning relies on many technical fields such as language understanding, visual understanding, reasoning, prediction, knowledge representation, motion control, etc.

[0005] Various fields where AI technology is applied include, for example, language understanding, visual understanding, reasoning prediction, knowledge representation, and motion control. Language understanding is a technology for recognizing, applying and / or processing human language or characters, and includes natural language processing, machine translation, dialogue systems, question and answer, speech recognition or synthesis, etc. Visual understanding is a technology for recognizing and processing objects as human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning prediction is a technology for judging and logically reasoning and predicting information, including knowledge-based and probability-based reasoning, optimization prediction, preference-based planning and recommendation. Knowledge representation is a technology for automatically processing human experience information into knowledge data, including knowledge construction (data generation or classification) and knowledge management (data utilization). Motion control is a technology for controlling the automatic driving of vehicles and the movement of robots, including motion control (navigation, collision, driving), operation control (behavior control), etc. Summary of the invention

[0006] [Technical issues]

[0007] At the same time, in recent years, the use of AI agents (such as Bixby TM 、Assistant TMAlexa TM However, when there are multiple devices including artificial intelligence agents set up in a small geographical area, confusion may arise so that multiple devices receiving the user's voice provide responses.

[0008] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above is applicable as prior art with respect to the present disclosure.

[0009] [Technical solution]

[0010] Aspects of the present disclosure will at least solve the above problems and / or disadvantages and provide at least the following advantages. Therefore, one aspect of the present disclosure will provide an electronic device and a control method thereof, which considers at least one of performance information and voice reception quality information to determine a device for acquiring a sound wave generated by a user for recognizing a voice from a plurality of terminal devices.

[0011] Another aspect of the present disclosure provides a communication interface and a processor including a communication circuit. The processor is configured to: receive performance information of one or more other terminal devices from each of one or more other terminal devices through a communication interface; based on the performance information received from each of the one or more other terminal devices, identify an edge device for determining a terminal device to perform speech recognition from the terminal device and one or more other terminal devices; based on the terminal device being identified as an edge device, receive information associated with reception quality from one or more other terminal devices that receive sound waves including a trigger word among the one or more other terminal devices that send performance information; based on the received information associated with reception quality, determine a terminal device that acquires sound waves for speech recognition from one or more other terminal devices that receive sound waves; and send a command to the determined terminal device to send the acquired sound waves for speech recognition to an external speech recognition device.

[0012] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0013] The performance information received from each of the one or more other terminal devices includes: a first score obtained based on the performance of the one or more other terminal devices, and the information associated with the reception quality received from the one or more other terminal devices receiving the sound waves may include: a second score representing the reception quality associated with the sound waves received by the one or more other terminal devices.

[0014] The processor may determine a terminal device having the highest second score from among one or more other terminal devices that receive the sound waves as a terminal device that acquires the sound waves for speech recognition.

[0015] According to one aspect of the present disclosure, a terminal device is provided, which may further include a microphone and a processor. The processor: based on receiving a sound wave including a trigger word at the microphone, obtains a second score of the terminal device based on at least part of the sound wave; determines the terminal device with the highest second score among the terminal device and one or more other terminal devices receiving the sound wave as the terminal device for obtaining the sound wave for speech recognition; and based on the terminal device being determined as the terminal device for obtaining the sound wave, obtains part of the sound wave received after the trigger word; and sends the obtained part of the sound wave to an external speech recognition device.

[0016] The information related to the reception quality may be acquired based on at least one of a signal-to-noise ratio (SNR), a volume of a received sound wave, a rate of change of the volume of the received sound wave, a sound pressure, and a distance from a source of the sound wave.

[0017] The performance information may be acquired based on at least one of the following: the number of other terminal devices connected, the number of types of available communication networks, a connection method with a server, a power supply method, processing performance, storage performance, and location movement status.

[0018] The processor may change a terminal device having a highest second score among one or more other terminal devices receiving sound waves to a terminal device for acquiring sound waves for speech recognition based on the second score of the determined terminal device being less than a preset value.

[0019] At least one of the first score and the second score received from each of the one or more other terminal devices may include information of at least one of the first score and the second score of a terminal device that communicates with the one or more other terminal devices and does not communicate with the terminal device.

[0020] According to another aspect of the present disclosure, a terminal device is provided. The terminal device may further include a memory, the memory including a speech recognition module. The processor may perform speech recognition on the sound wave using the speech recognition module based on receiving the sound wave from the determined terminal device, and send the result of the speech recognition to an external server; and the external server may determine the terminal device to perform an operation corresponding to the speech recognition result from the terminal device and one or more other terminal devices based on the speech recognition result, and send an operation execution command corresponding to the speech recognition result to the terminal device to perform the operation.

[0021] According to another aspect of the present disclosure, a terminal device is provided. The terminal device may further include a microphone and a processor. The processor may: determine, based on the first score of the terminal device being less than the first score of one or more other terminal devices, another terminal device with the highest first score as an edge device from one or more other terminal devices; based on receiving a sound wave including a trigger word at the microphone, obtain information associated with reception quality based on at least part of the sound wave; and send the obtained information related to reception quality to the determined edge device.

[0022] The processor may not send the second score to the determined edge device based on the second score of the terminal device being less than a preset value.

[0023] According to one aspect of the present disclosure, a method for controlling a terminal device is provided. The method includes: receiving information related to performance information of one or more other terminal devices from each of one or more other terminal devices; based on the performance information received from each of the one or more other terminal devices, identifying an edge device for determining a terminal device to perform speech recognition from the terminal device and one or more other terminal devices; based on the terminal device being identified as an edge device, receiving information associated with reception quality from one or more other terminal devices that receive sound waves including a trigger word among the one or more other terminal devices that send performance information; based on the received information associated with reception quality, determining a terminal device that acquires sound waves for speech recognition from one or more other terminal devices that receive sound waves; and sending a command to the determined terminal device to send the acquired sound waves for speech recognition to an external speech recognition device.

[0024] The performance information received from each of one or more other terminal devices may include: a first score obtained based on the performance of the other terminal devices; and the information associated with the reception quality received from one or more other terminal devices that receive sound waves may include: a second score representing the reception quality associated with the sound waves received by the one or more other terminal devices.

[0025] Determining the terminal device may include determining a terminal device having a highest second score from one or more other terminal devices that receive the sound waves as the terminal device that acquires the sound waves for speech recognition.

[0026] Determining a terminal device may include: based on receiving a sound wave including a trigger word at a microphone, obtaining a second score of the terminal device based on at least a portion of the sound wave; determining a terminal device with a highest second score among the terminal device and one or more other terminal devices that receive the sound wave as a terminal device that obtains sound waves for speech recognition; based on the terminal device being determined as a terminal device that obtains sound waves, obtaining a portion of the sound wave received after the trigger word, and sending the obtained portion of the sound wave to an external speech recognition device.

[0027] The information related to the reception quality may be acquired based on at least one of the following: SNR, volume of the received sound wave, a rate of change of volume of the received sound wave, sound pressure, and a distance from a source of the sound wave.

[0028] The performance information may be acquired based on at least one of the following: the number of other terminal devices connected, the number of types of available communication networks, a connection method with a server, a power supply method, processing performance, storage performance, and location movement status.

[0029] According to one aspect of the present disclosure, a control method is provided, the method comprising: based on a determined second score of a terminal device being less than a preset value, changing a terminal device with a highest second score among one or more other terminal devices receiving sound waves to a terminal device for acquiring sound waves for speech recognition.

[0030] At least one of the first score and the second score received from each of the one or more other terminal devices may include information of at least one of the first score and the second score of a terminal device that communicates with the one or more other terminal devices and does not communicate with the terminal device.

[0031] According to one aspect of the present disclosure, a control method is provided. The method includes: based on receiving a sound wave from a determined terminal device, using a speech recognition module included in a memory to perform speech recognition on the received sound wave, and sending the result of the speech recognition to an external server; and the external server can determine a terminal device to perform an operation corresponding to the speech recognition result from the terminal device and one or more other terminal devices based on the speech recognition result, and send an operation execution command corresponding to the speech recognition result to the terminal device to perform the operation.

[0032] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description, which discloses various embodiments of the present disclosure in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A speech recognition system according to an embodiment of the present disclosure is shown;

[0035] Figure 2 is a block diagram showing a configuration of a terminal device according to an embodiment of the present disclosure;

[0036] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 A block diagram of the detailed configuration of the terminal device;

[0037] Figure 4 A method for sharing performance information among multiple terminal devices according to an embodiment of the present disclosure is shown;

[0038] Figure 5 A method for sharing performance information among multiple terminal devices according to an embodiment of the present disclosure is shown;

[0039] Figure 6 A method for sharing performance information among multiple terminal devices according to an embodiment of the present disclosure is shown;

[0040] Figure 7 An embodiment of determining a terminal device to perform speech recognition from a terminal device receiving a speech including a trigger word or a trigger phrase according to an embodiment of the present disclosure is shown;

[0041] Figure 8 Various embodiments of performing speech recognition by a server according to an embodiment of the present disclosure are shown;

[0042] Fig. 9 Various embodiments of performing speech recognition by a server according to an embodiment of the present disclosure are shown;

[0043] Fig.10 An implementation of performing speech recognition by a terminal device determined by an edge device according to an implementation of the present disclosure is shown;

[0044] Fig.11 An embodiment of performing speech recognition by an edge device according to an embodiment of the present disclosure is shown;

[0045] Fig.12 An embodiment of performing speech recognition by an edge device according to an embodiment of the present disclosure is shown;

[0046] Fig.13 An embodiment of changing a device for receiving sound waves for voice recognition when a user utters a voice while moving according to an embodiment of the present disclosure is shown;

[0047] Fig.14is a flowchart illustrating a control method of a terminal device according to an embodiment of the present disclosure; and

[0048] Fig.15 is a sequence diagram showing a specific operation of a speech recognition system according to an embodiment of the present disclosure.

[0049] Throughout the drawings, like reference numerals will be understood to refer to like parts, components and structures. DETAILED DESCRIPTION

[0050] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. The description includes various specific details to assist in understanding, but these are considered to be exemplary only. Therefore, it will be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0051] The terms and words used in the following description and claims are not limited to the bibliographical meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be clear to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and not for the purpose of limiting the disclosure as defined by the appended claims and their equivalents.

[0052] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of such surfaces.

[0053] Since the embodiments may have a variety of modifications and several examples, certain embodiments will be illustrated in the drawings and described in detail in the description thereof. However, the scope of the embodiments need not be limited to a particular embodiment form. Instead, modifications, equivalent forms, and forms included in the disclosed concepts and technical scope of this specification may be adopted. While describing the embodiments, if it is determined that a specific description of known technology makes the subject matter of the present disclosure unclear, the specific description is omitted.

[0054] In the present disclosure, relational terms such as first and second, etc. may be used to distinguish one entity from another entity without necessarily implying any actual relationship or order between the entities. In embodiments of the present disclosure, relational terms such as first and second, etc. may be used to distinguish one entity from another entity without necessarily implying any actual relationship or order between the entities.

[0055] The terms "include", "comprising", "configured to", etc. in this specification are used to indicate the existence of features, numbers, operations, elements, parts or their combinations, and should not exclude the possibility of combining or adding one or more features, numbers, operations, elements, parts or their combinations.

[0056] According to the implementation mode, a "module" or "unit" performs at least one function or operation and can be implemented as hardware or software, or a combination of hardware and software. In addition, in addition to a "module" or "unit" that should be implemented in specific hardware, multiple "modules" or multiple "units" can be integrated into at least one module and can be implemented as at least one processor.

[0057] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure can be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly show the present disclosure in the accompanying drawings, parts not related to the description are omitted, and similar reference numerals are assigned to similar parts throughout the specification.

[0058] Hereinafter, embodiments of the present disclosure will be further described with reference to the accompanying drawings.

[0059] Figure 1 A speech recognition system according to an embodiment of the present disclosure is shown.

[0060] Reference Figure 1 , the speech recognition system 1000 according to an embodiment of the present disclosure may include a plurality of terminal devices 100-1 to 100-5. For example, the first terminal device 100-1 (eg, smart TV), the second terminal device 100-2 (eg, smart speaker) and the fifth terminal device 100-5 (eg, mobile device) may perform direct communication with each other. That is, the first terminal device 100-1 may be configured to send a message to the second terminal device 100-2 and the fifth terminal device 100-5, and / or receive a message from the second terminal device 100-2 and the fifth terminal device 100-5; the second terminal device 100-2 may be configured to send a message to the first terminal device 100-1 and the fifth terminal device 100-5, and / or receive a message from the first terminal device 100-1 and the fifth terminal device 100-5; and the fifth terminal device 100-5 may be configured to send a message to the first terminal device 100-1 and the second terminal device 100-2, and / or receive a message from the first terminal device 100-1 and the second terminal device 100-2. Similarly, the second terminal device 100-2, the third terminal device 100-3 (e.g., a home appliance) and the fourth terminal device 100-4 (e.g., a smart watch) may perform direct communication with each other.

[0061] Although the reference Figure 1Direct communication is described, but a plurality of terminal devices 100-1 to 100-5 or Figure 1 One or more other terminals not shown in the figure but included in the speech recognition system 1000 may not establish direct communication when communicating with one or more other terminal devices in the speech recognition system 1000. For example, one or more of the multiple terminal devices 100-1 to 100-5 may perform indirect communication with another terminal device of the speech recognition system 1000, so that any communication can be sent between the two terminal devices via the third terminal device. For example, if the fifth terminal device 100-5 intends to communicate with the third terminal device 100-3, the fifth terminal device 100-5 may send a message to the second terminal device 100-2, and the second terminal device 100-2 may send a message received from the fifth terminal device 100-5 to the third terminal device 100-3.

[0062] The terminal devices 100-1 to 100-5 may have a configuration separate from and / or different from that of the server, and may refer to devices directly used by the user 10. For example, the terminal devices may be various electronic devices, such as televisions (TVs), mobile phones, smart phones, personal digital assistants (PDAs), notebook personal computers (PCs), desktops, tablet PCs, e-books and electronic photo frames, information kiosks, refrigerators, washing machines, smart speakers, smart watches, air conditioners, audio, digital video disc (DVD) players, microwave ovens, air purifiers, door locks, kettles, closed-circuit televisions (CCTVs), fire alarm equipment, home appliances, home systems or environmental conditioning equipment (e.g., heating, ventilation, humidifiers, sewage pumps, water filtration, etc.), etc.

[0063] like Figure 1 As shown, the user 10 may make a voice (e.g., generate a sound wave) with the intention of triggering a response from one or more terminal devices 100-1 to 100-5, so that one or more terminal devices 100-1 to 100-5 perform a corresponding function based on the generated sound wave (①); and at least one terminal device 100-2 to 100-5 among the multiple terminal devices 100- and 100-5 may receive the sound wave (②). For example, the sound wave generated by the user voice received by at least one terminal device 100-2 and 100-5 may include a trigger word or a trigger phrase for activating a function.

[0064] If at least one of the multiple terminal devices 100-1 to 100-5 of the speech recognition system 1000 receives the generated sound wave, the terminal device receiving the generated sound wave may forward the sound wave to a suitable or intended terminal device suitable for performing an operation or function corresponding to the sound wave (③). Figure 1As shown, the second terminal device 100-2 can be a terminal device that directly receives the sound wave, or it can be a terminal device that intends to perform an operation corresponding to the sound wave based on a trigger word or trigger phrase corresponding to the sound wave. Optionally, the second terminal device 100-2 and the fifth terminal device 100-5 can receive the sound wave, and one or more other terminal devices (for example, terminal devices 100-1, 100-3, and 100-4) can be terminal devices that intend to perform an operation corresponding to the sound wave. In this way, the terminal device that receives the sound wave can forward the sound wave or information associated with the sound wave to the intended terminal device, so that the intended terminal device can perform a corresponding operation or function based on the sound wave.

[0065] A plurality of terminal devices 100-1 to 100-5 may be arranged in close proximity (e.g., within a predetermined space). Here, the predetermined space may be a home space such as a "living room," a "main room," a "kitchen," etc. If different users in different spaces simultaneously generate sound waves including a trigger word or a trigger phrase, one terminal device in each space may receive the sound waves generated by the different users.

[0066] An exemplary process of determining which terminal device among the plurality of terminal devices 100 - 1 to 100 - 5 of the voice recognition system 1000 may perform an operation or function based on sound waves received from one or more users will be described in detail with reference to the accompanying drawings.

[0067] Figure 2 is a block diagram showing a configuration of a terminal device according to an embodiment of the present disclosure.

[0068] refer to Figure 2 , the terminal device 100 may include a communication interface 110 and a processor 120 .

[0069] The communication interface 110 may be configured to communicate with various types of external devices according to various types of communication methods. The external device communicating with the terminal device 100 may be a server and / or one or more other terminal devices.

[0070] The communication of the communication interface 110 with the external device may include communication via a third device (e.g., a repeater, a hub, an access point, a server, or a gateway). The communication interface 110 may be configured to communicate using wireless communication, including cellular communication using at least one of, for example, Long Term Evolution (LTE), Advanced LTE (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), and Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM), etc. According to one embodiment, additionally or alternatively, the wireless communication may include, for example, at least one of Wireless Fidelity (Wi-Fi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, Near Field Communication (NFC), Magnetic Secure Transmission, Radio Frequency (RF), or Body Area Network (BAN). The communication interface 110 may be configured to communicate using wired communication. Wired communication may include, for example, at least one of Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), Power Line Communication, or Plain Old Telephone Service (POTS). The network performing wireless or wired communication may include at least one of a telecommunication network (eg, a computer network (eg, a local area network (LAN) or a wide area network (WAN))), the Internet, or a telephone network.

[0071] The processor 120 may be configured to control the operation of the terminal device 100 .

[0072] According to one embodiment, the processor 120 may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON). However, the present disclosure is not limited thereto, and may include or be defined as a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), and an advanced reduced instruction set computer (RISC) machine (ARM) processor. The processor 120 may be implemented as a system on chip (SoC), a large-scale integrated circuit (LSI) with a built-in processing algorithm, or a field programmable gate array (FPGA).

[0073] In an exemplary embodiment, the processor 120 may share performance information with one or more other terminal devices connected through the communication interface 110. For example, the processor 120 may send the performance information of the terminal device 100 to one or more other terminal devices through the communication interface 110, or may receive the performance information of each of the one or more other terminal devices from the one or more other terminal devices.

[0074] In this case, the performance information sent to or received from one or more other terminal devices can be used to identify an edge device for determining a terminal device for acquiring voice or sound waves for voice recognition from multiple terminal devices. The edge device can be referred to as a main terminal device, an edge computing device, etc. The multiple terminal devices may include the terminal device 100 and one or more other terminal devices.

[0075] The performance information may include a score obtained based on the performance of each terminal device (hereinafter referred to as a first score or a performance score (P-score)).

[0076] For example, the performance-based score is a score obtained by scoring according to a plurality of criteria, and examples of the plurality of criteria may be shown in Table 1.

[0077] [Table 1]

[0078]

[0079]

[0080] Referring to Table 1, the processor 120 may obtain a performance-based score based on various criteria such as a networking standard, a performance standard, a stability standard, a user interface standard, and the like.

[0081] Specifically, the performance information is used to determine the edge device and may include whether the device is conducive to data processing. For example, the networking criteria may include the number of neighboring devices, the operable networking IF, and the server connection via wired communication or wireless communication, etc. For example, when there are a large number of other connected terminal devices, a large number of operable networking methods, and the communication between the server and the terminal device 100 is a wired connection, the terminal device 100 may obtain a high score.

[0082] Alternatively or additionally, the performance criteria may include power supply (e.g., battery vs. DC), processing power or computing power (processor, memory), etc. In an exemplary embodiment, the terminal device 100 may obtain a higher score when power is supplied directly to the terminal device 100 (e.g., via a DC connection) without being powered from a battery. In another exemplary embodiment, the terminal device 100 may obtain a higher score based on the amount of power consumed during processing or during interaction with memory.

[0083] Alternatively or additionally, the performance criteria may include stability criteria, such as mobility factors (e.g., the degree of device location change), stability factors, stabilization time, etc. When the location changes or moves less frequently and the stabilization time is longer, the terminal device 100 may be assigned a higher score.

[0084] Alternatively or additionally, the user interface criteria may include relatively assistive criteria, and may score high when a touch screen is provided or voice output is available.

[0085] The processor 120 may evaluate the performance of the terminal device 100 according to one or more of the plurality of criteria and calculate the performance as a score. As shown in the following equation 1, the processor 120 may obtain a score by summing the scores reflecting the weight according to the importance of each of the plurality of criteria based on the performance information of the terminal device 100.

[0086] P i＝ ∑ k w k C k Equation 1

[0087] Here, P i is the first score of terminal device i, w k is the standard weight, and C k It can be the score for this criterion.

[0088] For example, since the above performance criteria are relatively important, the weight is high; and the importance of the user interface criteria is relatively low, so the weight may be low. In addition, if the item is not applied, the weight may be zero.

[0089] As described above, the processor 120 may send the performance information of the terminal device 100 obtained based on various criteria to one or more other terminal devices. Specifically, the processor 120 may send the first score based on the performance information to one or more other terminal devices.

[0090] At the same time, the processor 120 may receive performance information of one or more other terminal devices from one or more other terminal devices. In an exemplary embodiment, the terminal device 100 may receive the first score of one or more other terminal devices calculated by one or more other terminal devices; or receive information related to the performance of one or more other terminal devices, and the processor 120 may calculate the first score of one or more other terminal devices.

[0091] The performance information received from one or more other terminal devices may include performance information associated with the terminal device connected to the one or more other terminal devices. Therefore, the processor 120 may not only obtain the performance information of the 1-hop neighbor device directly connected to the terminal device 100, but also obtain the performance information of the 2-hop or more hop neighbor devices connected only to the one or more other terminal devices (1-hop neighbor devices), and identify the edge device including the 2-hop or more hop neighbor devices. Figures 4 to 6 The operation of sharing performance information with one or more other terminal devices is described in detail.

[0092] The processor 120 may identify an edge device using the obtained performance information of the terminal device 100 and the received performance information of one or more other terminal devices. Specifically, the processor 120 may identify a terminal device having the highest first score as an edge device based on the performance information.

[0093] In an exemplary embodiment, the edge device may be a local terminal device rather than a server. As the amount of data increases dramatically due to the popularity of Internet (Internet of Things) devices, cloud computing has faced limitations. The edge device as determined in the exemplary embodiment of this article may be a device that can implement edge computing technology to overcome the limitations introduced by the use of a server. In an exemplary embodiment, the edge device may be a device that can perform all or part of the computing tasks performed in a server according to the prior art.

[0094] In this embodiment, the edge device identified may determine a device to perform voice recognition from a plurality of terminal devices. Performing voice recognition may include at least one of the following operations: activating a microphone to receive sound waves, sending information associated with the received sound waves to a server for performing voice recognition, and performing voice recognition by the device receiving the sound waves via the microphone.

[0095] Based on the performance information, when the terminal device 100 is identified as an edge device, the processor 120 may receive information related to the voice reception quality from one or more other terminal devices. For example, the one or more other terminal devices that send information related to the sound wave reception quality may be at least one terminal device among the one or more other terminal devices that has received a sound wave including a trigger word or a trigger phrase. In an exemplary embodiment, the trigger word or trigger phrase may be a call word or phrase for activating an artificial intelligence (AI) auxiliary function of the terminal device 100, and may be referred to as a wake-up word such as "Bixby", "Siri", "Okay Google", and "Alexa".

[0096] In an exemplary embodiment, information associated with the quality of sound wave reception may be used to determine a device that receives the sound waves generated by a user (e.g., as input at a receiving device) with the best quality. For example, the information associated with the quality of sound wave reception may include a score (hereinafter referred to as a second score or a quality score (Q-score)) obtained based on at least one of the following: information related to the performance of a microphone in the terminal device 100, a signal-to-noise ratio (SNR), a volume associated with the received sound waves, a rate of change of the volume of the received sound waves, sound pressure, and information associated with the distance between the user generating the sound waves and the terminal device 100.

[0097] In an exemplary embodiment, information associated with the distance between the user generating the sound wave and the terminal device 100 may be obtained by at least one sensor (eg, sensor 160 ) provided in the terminal device 100 .

[0098] According to this embodiment, the terminal device 100 may receive scores calculated by one or more other terminal devices, receive sound wave signals received by one or more other terminal devices, and the processor 120 may obtain second scores of the one or more other terminal devices.

[0099] In an exemplary embodiment, the performance information received from one or more other terminal devices may include information related to the acoustic wave reception quality of the terminal device connected to the one or more other terminal devices. Therefore, the processor 120 may obtain information related to the acoustic wave reception quality of a terminal device (e.g., a 2-hop or more hop neighbor device) that is only connected to one or more other terminal devices and not connected to the terminal device 100.

[0100] The processor 120 may determine a device to perform voice recognition based on information about the sound wave reception quality of each of at least one terminal device that has received a sound wave including a trigger word or a trigger phrase. Specifically, the processor 120 may determine a device to perform voice recognition using a second score of at least one terminal device that has received a sound wave including a trigger word or a trigger phrase.

[0101] Specifically, the processor 120 may determine a terminal device having a highest second score among at least one terminal device that has received the sound wave as a device that performs voice recognition.

[0102] At the same time, the processor 120 may determine a device on which voice recognition is to be performed, which includes not only information on the quality of sound wave reception but also performance information of at least one terminal device receiving the sound wave. Then, the processor 120 may send a voice recognition command to the determined terminal device.

[0103] Specifically, as shown in the following Equation 2, the processor 120 may obtain a B-score according to the sum of the first score and the second score reflecting the degree of importance based on the weight to determine the device most suitable for performing voice recognition.

[0104] B i =W 1 P i+ w 2 Q i Equation 2

[0105] Here, P i is the first score of terminal device i, w 1 is the weight of the first score, Q i is the second score of terminal device i, and w2 is the weight of the second score.

[0106] For example, for a score used to determine the device most suitable for performing speech recognition, the quality of the received sound wave is important. According to an embodiment, when the terminal device receiving the sound wave directly performs speech recognition without sending the received sound wave to the server, the performance of the terminal device may be considered together with the first score. When the received sound wave is sent to the server, the quality of the received sound wave is important. 1 Can be very low or zero.

[0107] In another embodiment, when information associated with the sound wave reception quality is received from a plurality of terminal devices that have received the sound wave, if the difference between the second scores or B-scores between the plurality of terminal devices is within a predetermined value, the processor 120 may determine the terminal device that has sent the information related to the received sound wave reception quality as a speech recognition device. In this way, the real-time characteristic in the speech recognition process can be ensured.

[0108] Meanwhile, when the terminal device 100 is an edge device, the processor 120 may determine the terminal device with the highest B-score as the terminal device that performs voice recognition. Then, the processor 120 may send a voice recognition command to the determined terminal device.

[0109] In an exemplary embodiment, when the terminal device 100 as an edge device is a device capable of analyzing sound waves to discern user intentions, the processor 120 may receive a sound wave signal input from a device receiving voice. For example, voice recognition may include analysis of user intentions. The processor 120 may generate an operation execution command through voice recognition, and may determine a terminal device to perform an operation corresponding to the input sound wave. The processor 120 may send the generated operation execution command to the terminal device that performs the operation corresponding to the sound wave.

[0110] In one embodiment, the processor 120 may determine the device that performs intent analysis on the sound wave generated by the user based on the length of the sound wave. For example, if the length of the sound wave is short, the number of words may be small or may include words corresponding to a specific terminal device, and the processor 120 may directly perform intent analysis on the sound wave generated by the user. The words corresponding to the specific terminal device may be terms mainly used for the terminal devices included in the speech recognition system 1000. For example, "Please increase / lower the temperature", "temperature" is a term mainly used in environmental rules such as heating, air conditioning, etc. If it is determined that "temperature" is included in the sound wave, the processor 120 may directly analyze the intention of the user who generates the sound wave, generate a corresponding control command, and send the generated control command to the terminal device as an air conditioner.

[0111] If the length of the sound wave is long or the number of words associated with the sound wave is large, the processor 120 may send the received voice signal to an external server to analyze the sound wave generated by the user. For example, the processor 120 may send a voice signal (e.g., a sound wave) input or received at an input device to a server, or only receive length information of a voice signal input from an input device, and send information about whether to send to a server or terminal device 100 to a device receiving the voice.

[0112] As described above, even if multiple terminal devices simultaneously receive sound waves generated by a user by identifying an edge device among multiple terminal devices and determining a device for performing voice recognition by the identified edge device, the operation corresponding to the sound waves generated by the user may be performed in only one terminal device.

[0113] Figure 3 The present invention is shown in the embodiment of the present invention. Figure 2 A block diagram of the detailed configuration of the terminal device.

[0114] refer to Figure 3 , the terminal device 100 may include a communication interface 110 , a processor 120 , a microphone 130 , a memory 140 , a display 150 , a sensor 160 , and a speaker 170 .

[0115] Some operations of the communication interface 110 and the processor 120 are similar to Figure 2 The operations are the same as those of , and repeated descriptions will be omitted.

[0116] The microphone 130 may receive sound waves generated by a user. The microphone 130 generates (or converts) voice or sound waves received from the outside into electrical signals under the control of the processor 120. The electrical signals generated by the microphone 130 may be converted by the processor 120 and stored in the memory 140, or may be sent to an external device for performing voice recognition through the communication interface 110.

[0117] In an exemplary embodiment, when a sound wave including a trigger word or a trigger phrase is received via the microphone 130, the terminal device performing voice recognition may be determined to be the terminal device 100. The edge device determining the device to perform voice recognition may be the terminal device 100, or may not be the terminal device 100.

[0118] For example, if the terminal device 100 is an edge device, and when the terminal device 100 is determined to be a device that recognizes sound waves based on at least one of the first score and the second score, the processor 120 may perform speech recognition on the received sound waves after recognizing the sound waves including a trigger word or a trigger phrase.

[0119] In an exemplary embodiment, the processor 120 may send an electrical signal associated with the received sound wave to an external server for performing voice recognition after recognizing a trigger word or a trigger phrase. For example, the external server may perform voice recognition, determine a terminal device for generating an operation execution command and performing an operation based on the voice recognition result, and send the operation execution command to the determined terminal device.

[0120] If the terminal device 100 as an edge device is a device capable of performing voice recognition, the processor 120 may perform voice recognition on the received sound waves after recognizing the trigger word or trigger phrase. The received sound waves may be received by the terminal device 100 through the microphone 130, or may be electrical signals associated with sound waves provided by one or more other terminal devices.

[0121] The processor 120 may send the speech recognition result to an external server. Based on the speech recognition result, the external server may send an operation execution command corresponding to the speech recognition result to a terminal device among multiple terminal devices that performs the operation corresponding to the speech recognition result. According to an embodiment of the present disclosure, the processor 120 may not send the speech recognition result to an external server, but directly send the operation execution command corresponding to the speech recognition result to a terminal device among multiple terminal devices that performs the operation corresponding to the speech recognition result.

[0122] At the same time, if the first score of one or more other terminal devices is higher than the first score of the terminal device 100, the processor 120 may determine the terminal device with the highest first score among the one or more other terminal devices as an edge device. In an exemplary embodiment, when a sound wave including a trigger word or a trigger phrase is received via the microphone 130, the processor 120 may obtain information related to the reception quality of the sound wave based on at least a portion of the received sound wave. For example, the processor 120 may obtain information related to the reception quality based on a sound wave including a trigger word or a trigger phrase, a sound wave including a trigger word or a trigger phrase received at a predetermined time after the trigger word or the trigger phrase, and a sound wave not including a trigger word or a trigger phrase received at a predetermined time after the trigger word or the trigger phrase. The information related to the reception quality may include a second score. Then, the processor 120 may control the communication interface 110 to send the information related to the obtained reception quality to the edge device.

[0123] At this time, if the obtained reception quality is less than the preset standard, the processor 120 may discard the information related to the obtained reception quality without sending the information to the edge device. Specifically, if the obtained second score is less than a predetermined value, the processor 120 may discard the obtained second score without sending the second score to the edge device.

[0124] The memory 140 may store various programs and data required for the operation of the terminal device 100. For example, at least one instruction may be stored in the memory 140. The processor 120 may perform the above operations by executing the instructions stored in the memory 140. The memory 140 may be implemented as a nonvolatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0125] The learned AI model may be stored in the memory 140. In an exemplary embodiment, the learned AI model may be learned to recognize the received sound wave and convert the sound wave into text, or determine the user's intention based on the sound wave. In this way, if the learned AI model for speech recognition is stored in the memory 140, the processor 120 may perform speech recognition using the stored AI model without sending an electrical signal associated with the received sound wave to an external server for performing speech recognition.

[0126] The display 150 may display various information under the control of the processor 120. In an exemplary embodiment, the display 150 may be implemented as a touch screen together with a touch panel. According to various embodiments, the display 150 may not be provided in the terminal device 100. For example, when the terminal device 100 is a smart speaker, etc., the display 150 may not be provided.

[0127] The sensor 160 may sense various status information of the terminal device 100 or the user. For example, the sensor 160 may include a motion sensor (e.g., a gyroscope sensor, an acceleration sensor, etc.) capable of sensing motion information of the terminal device 100, a sensor (e.g., a global positioning system sensor) capable of sensing location information, a sensor (e.g., a temperature sensor, a humidity sensor, an air pressure sensor, an ultrasonic sensor, an infrared sensor, etc.) capable of sensing environmental information, and a sensor (e.g., a blood pressure sensor, a blood sugar sensor, a pulse rate sensor, etc.) capable of sensing biological user information of the terminal device 100, etc. In addition, the sensor 160 may also include an image sensor (e.g., a camera) for photographing or capturing an image of the environment outside the terminal device 100.

[0128] The processor 120 may determine whether the user moves according to the distance and distance change between the user and the terminal device 100 based on various sensing values ​​of the sensor 160. In an exemplary embodiment, the processor 120 may acquire information related to voice reception quality based on the distance between the user and the terminal device 100.

[0129] The speaker 170 may have a configuration for outputting various notification sounds or voice messages as well as various audio data, and the audio processor performs various processing tasks on the audio data such as decoding, amplification, noise filtering, etc. In particular, the speaker 170 may output a response to the sound waves generated by the user as a voice message in a natural language format. The configuration for outputting audio may be implemented as a speaker, but may also be implemented as an output terminal capable of outputting audio data.

[0130] Figure 4 , Figure 5 and Figure 6 A method for sharing performance information among multiple terminal devices according to an embodiment of the present disclosure is shown.

[0131] Specifically, Figure 4 A method of sharing performance information when a new device is connected to the voice recognition system is shown. Here, the TV 400-1, the speaker 400-2, and the refrigerator 400-3 are already connected to the network and can be in a state of sharing information with each other.

[0132] refer to Figure 4 , when the new terminal device 400-4 accesses the network, in operation S401, the new terminal device 400-4 may send an information request signal to at least one of the plurality of terminal devices connected to the network. In an exemplary embodiment, the new terminal device 400-4 may send the information request signal in a multicast method. The new terminal device 400-4 may send the performance information including the P-score associated with the new terminal device 400-4 to other terminal devices together with the information request signal, or send the performance information to other terminal devices alone.

[0133] In an exemplary embodiment, in operations S402 and S403, the speaker 400-2 and the refrigerator 400-3 of the terminal devices receiving the request from the new terminal device 400-4 may transmit own performance information including the P-score to the new terminal device 400-4.

[0134] In operation S404, the new terminal device 400-4 may request a device list 410 from at least one of the plurality of terminal devices connected to the network. In an exemplary embodiment, the device list 410 may be a list record of other terminal devices capable of communication. The new terminal device 400-4 may request a device list 410 from all one or more other terminal devices that send performance information; or may request a device list 410 from one or more other terminal devices with the highest performance based on the performance information received from the one or more other terminal devices. Although it has been described above that the device list 410 is requested after receiving the performance information, in an exemplary embodiment, the device list 410 may be requested at the same time.

[0135] In operation S405, the new terminal device 400-4 may receive a device list 410 of the speaker from the speaker 400-2 as another terminal device. For example, the device list 410 of the speaker may include information associated with one or more other terminal devices that can communicate with the speaker 400-2. Therefore, the device list 410 may include information associated with one or more other terminal devices that do not communicate with the new terminal device 400-4.

[0136] As an implementation method, Figure 4 In the device list 410 of the speaker, information associated with the TV 400-1 may be included, which communicates with the speaker 400-2 but not with the new terminal device 400-4. Therefore, the new terminal device 400-4 can obtain information associated with one or more other terminal devices that do not directly communicate with each other.

[0137] The device list 410 may include information related to ID, name, address, performance information (P-score), direct communication (1-hop), and the like.

[0138] Therefore, if Figure 5 As shown, each of the plurality of terminal devices 400-1 to 400-4 connected to the network may share information of one or more other terminal devices. Each of the plurality of terminal devices 400-1 to 400-4 may identify an edge device among the plurality of terminal devices 400-1 to 400-4 based on the shared performance information.

[0139] Reference Figure 5 , each of the plurality of terminal devices 400 - 1 to 400 - 4 may identify the TV 400 - 1 having the highest P-score as an edge device.

[0140] In an exemplary embodiment, if a predetermined condition is satisfied, each of the plurality of terminal devices 400-1 to 400-4 may update the shared performance information. For example, each of the plurality of terminal devices 400-1 to 400-4 may update the shared performance information according to a predetermined period, or update the shared performance information when a new terminal device 400-4 requests performance information. In an exemplary embodiment, each terminal device may identify a new edge device based on the updated performance information. For example, each terminal device may identify a new edge device each time the performance information is updated, and if there is a terminal device having a higher P-score than the terminal device identified as the current edge device, the terminal device having a higher P-score may be identified as a new edge device.

[0141] Alternatively, if Figure 6As shown, when the new terminal device 400-4 connected to the network cannot communicate with the voice recognition system, the shared performance information can be updated. For example, each of the plurality of terminal devices 400-1 to 400-3 can delete information about the new terminal device 400-4 that cannot communicate with the voice recognition system from the device list 410.

[0142] When an edge device is unable to communicate with the speech recognition system, each terminal device can identify a new edge device based on the shared performance information.

[0143] Figure 4 , Figure 5 and Figure 6 Only an embodiment in which an edge device is identified based on performance information of a plurality of terminal devices is shown, but in various embodiments, an edge device may be identified through an input of a user selecting an operation of the edge device.

[0144] Considering the disabled state of the identified edge device, the lower-level candidate edge device may be selected together with the edge device when the edge device is identified. In an exemplary embodiment, the lower-level candidate edge device may be a terminal with the second highest score based on the performance information, or may be identified by a user's selection. Therefore, when the identified edge device is in a disabled state such as power failure or failure, the determined lower-level candidate edge device may be automatically changed to an edge device without performing the edge device identification process again.

[0145] Figure 7 An embodiment of the present disclosure is shown for determining a terminal device for performing speech recognition in a terminal device that receives a speech including a trigger word or a trigger phrase.

[0146] first, Figure 7 An embodiment is shown in which, based on performance information shared among multiple terminal devices 700-1 to 700-5 (e.g., TV 700-1, refrigerator 700-2, smart watch 700-3, mobile device 700-4, and smart speaker 700-5) connected to a network, TV 700-1 with the highest P-score can be identified as an edge device.

[0147] Reference Figure 7 When the user 10 says the trigger phrase "Hi, Bixby", the smart watch 700-3, the mobile device 700-4 and the smart speaker 700-5 in the multiple terminal devices 700-1 to 700-5 can receive the sound waves generated by the user including the trigger phrase.

[0148] In an exemplary embodiment, each of the smart watch 700-3, the mobile device 700-4, and the smart speaker 700-5 that have received the sound wave including the trigger phrase may acquire information related to the reception quality of the sound wave based on at least one of microphone performance, the volume of the received sound wave, the rate of change of the volume of the received sound wave, the sound pressure, and the distance prediction from the user generating the sound wave. In an exemplary embodiment, the information associated with the reception quality may include a Q-score indicating the reception quality of the sound wave.

[0149] Each terminal device that has received the sound wave including the trigger phrase may transmit information related to the obtained reception quality to the TV 700-1 as an edge device. If the obtained voice reception quality is lower than a predetermined level, the terminal device that has received the sound wave including the trigger word may discard or ignore the information associated with the reception quality without transmitting the information to the edge device. For example, if the predetermined reference value of the Q-score of the reception quality is 15, then Figure 7 As shown, the smart watch 700 - 3 whose acquired Q-score is less than 15 may delete the information associated with the reception quality without transmitting the reception quality information to the TV 700 - 1 which is an edge device.

[0150] The TV 700-1 that receives the reception quality information from the mobile device 700-4 and the smart speaker 700-5 as one or more other terminal devices may determine a device for performing voice recognition based on the performance information of the mobile device 700-4 and the smart speaker 700-5 and the reception quality information. In an exemplary embodiment, performing voice recognition may mean sending an electrical signal corresponding to the received sound wave to a server for voice recognition operation, or performing voice recognition operation by the terminal itself.

[0151] In an exemplary embodiment, TV 700-1 as an edge device may use at least one of a P-score as a performance information score and a Q-score as a reception quality score to determine a device to perform voice recognition. The edge device may reflect a weight to each of the P-score and the Q-score and then add them together, and the weight may vary according to the embodiment. For example, if voice recognition is performed by a device that receives sound waves generated by a user, performance information is also important. Therefore, the weight of the P-score may be increased. If voice recognition is performed in an external server, only the reception quality may be important, and the weight of the P-score may be lower.

[0152] The edge device that determines the device on which voice recognition is to be performed based on the performance information and the reception quality may send a voice recognition execution command to the determined terminal device.

[0153] For example, if the weights of the P-score and the Q-score are 1 respectively, the TV 700-1 may send a voice recognition execution command to the mobile device 700-4 having the highest sum of the P-score and the Q-score. The mobile device 700-4 receiving the voice recognition execution command may perform voice recognition on a portion of the sound wave following the trigger word or trigger phrase. For example, the mobile device 700-4 may send a sound wave including the phrase "Please turn on the song on the speaker" provided after receiving the sound wave including the trigger phrase "Hi, Bixby" to a server for voice recognition, or perform voice recognition by itself.

[0154] In the above description, according to the present embodiment, the TV 700-1 as an edge device may recognize a sound wave including a trigger word or a trigger phrase, and the TV 700-1 as an edge device may be determined as a device that performs voice recognition. Figure 8 , Fig. 9 , Fig.10 , Fig.11 and Fig.12 Various implementations of the speech recognition subject are described in detail.

[0155] Meanwhile, according to the present embodiment, information related to reception quality may be sent to the edge device at a predetermined cycle. For example, when the user 10 generates sound waves while moving, a plurality of terminal devices receiving the sound waves of the user 10 may send information related to reception quality associated with the received sound waves to the edge device at a predetermined cycle. Therefore, during the duration of the sound waves, the device for performing voice recognition may be changed.

[0156] Specifically, when the reception quality of a terminal device determined as a device for performing voice recognition decreases, the edge device may change the terminal device for performing voice recognition. For example, a plurality of terminal devices that receive sound waves generated by the user 10 may send information related to the reception quality to the edge device at a predetermined period, or if the second score of a terminal device determined as a device for performing voice recognition decreases to less than a predetermined value, the edge device may request information related to the reception quality from a plurality of terminal devices that receive electrical signals associated with the sound waves generated by the user 10, and receive information related to the reception quality of a plurality of terminal devices that receive the voice of the user 10. At this time, the edge device may change the terminal device having the highest second score among the plurality of terminal devices to the device for performing voice recognition.

[0157] In other words, the sound waves generated before the terminal device that performs voice recognition is changed are used to perform voice recognition by the terminal device before the change; and the part of the sound waves received after the change can be recognized by the changed terminal device. For example, the terminal device used to perform voice recognition before and after the change can send the electrical signal associated with the received sound wave to the server or edge device for voice recognition, respectively. According to another embodiment, if the terminal device used to perform voice recognition before and after the change can send the received voice to the edge device, respectively, the edge device can splice the received partial sound waves into one sound wave, send the spliced ​​one sound wave to the server for voice recognition, or the edge device can directly perform voice recognition. In an exemplary embodiment, considering the terminal device that has sent the electrical signal corresponding to the received sound wave, the edge device can send information associated with the device that performs the operation corresponding to the voice recognition to the server.

[0158] As described above, even if sound waves generated by a user are received at multiple terminal devices, only one device performs voice recognition, which can reduce unnecessary resource consumption and prevent conflicts between devices when performing operations related to the user's voice command.

[0159] Figure 8 and Fig. 9 is a view for describing various embodiments of performing voice recognition by a server according to an embodiment of the present disclosure.

[0160] Figure 8 and Fig. 9 It is shown that according to the sharing of performance information of a plurality of terminal devices 800-1 to 800-3 connected to the network, TV 800-1 having the highest P-score is identified as an edge device, and mobile device 800-2 is determined by TV 800-1 as a device for performing voice recognition.

[0161] Reference Figure 8 , the mobile device 800-2 that receives part of the sound wave after the trigger word or trigger phrase "turn on the music on the speaker" can send the electrical signal corresponding to the received sound wave to the server 200 (①) for performing voice recognition.

[0162] The server 200 may perform voice recognition on the received electrical signal corresponding to the sound wave to determine the terminal device that generates the operation execution command and executes the operation command corresponding to the sound wave (②). For example, as a result of voice recognition, the server 200 may determine the smart speaker 800-3 as the terminal device that generates the music reproduction command and executes the operation command. The server 200 may send the generated operation execution command to the smart speaker 800-3 (③).

[0163] The smart speaker 800-3 that receives the operation command from the server 200 may perform an operation corresponding to the received command (④). For example, the smart speaker 800-3 may reproduce music based on the received reproduction command.

[0164] When the user 10 has generated a sound wave including an operation subject, if the user 10 further says "Please turn on the music", the server 200 may determine the terminal device to execute the operation command based on various criteria. For example, the terminal device closest to the user 10 or the terminal device with the best audio output performance may be determined as the device to execute the operation command, and the operation command may be sent.

[0165] like Fig. 9 As shown, when multiple terminal devices are determined as terminal devices for executing an operation command, the device for executing the final operation can be determined according to the user's selection.

[0166] Reference Fig. 9 , the server 200 may perform voice recognition (②) on the electric signal associated with the sound wave (①) generated by the user 10 received from the mobile device 800-2. As a result of performing the voice recognition, if it is determined that the voice recognition is performed by a plurality of terminal devices, the server 200 may send an operation execution command (③) to the determined plurality of terminal devices 800-2 and 800-3.

[0167] In this case, the user 10 may be requested to select one of the plurality of terminal devices 800-2 and 800-3 (④). For example, the terminal device requesting the user 10 to select a device may be at least one of the plurality of terminal devices 800-2 and 800-3.

[0168] If the user 10 selects a device according to the selection request (⑤), the selected device may execute the operation command corresponding to the sound wave generated by the user (⑥). For example, when the user 10 selects the smart speaker 800-3 among the mobile device 800-2 and the smart speaker 800-3 that receive the operation execution command, the selected smart speaker 800-3 may reproduce music according to the received operation command.

[0169] In the above description, it is described that the server 200 sends the operation execution command to the determined multiple terminal devices 800-2 and 800-3. However, in an optional embodiment, if it is determined that multiple terminal devices perform voice recognition, the server 200 may send a command for a terminal device selection request to the device 800-2 that has sent a voice signal, and if the device 800-2 that has sent an electrical signal associated with the received sound wave sends a response to the user's selection to the server 200, the server 200 may be configured to send the operation execution command to the selected terminal device.

[0170] In the above, an embodiment in which a voice recognition operation on a received voice is performed by an external server is described. Hereinafter, an embodiment in which voice recognition is performed by a terminal device will be described.

[0171] Fig.10 It is a view for describing an implementation in which a terminal device determined by an edge device performs voice recognition according to an implementation of the present disclosure.

[0172] Fig.10 The embodiment is shown after TV 800-1 having the highest P-score is identified as an edge device based on sharing of performance information of a plurality of terminal devices 800-1 to 800-3 connected to a network, and mobile device 800-2 is determined by TV 800-1 as a device for performing voice recognition.

[0173] Reference Fig.10 , the mobile device 800-2 that receives the sound wave corresponding to the phrase "Please turn on the music on the speaker" can perform voice recognition (①) on the received sound wave, which is the sound wave received after the user 10 says the trigger word or trigger phrase.

[0174] Specifically, the mobile device 800-2 may perform voice recognition on the received sound waves, and determine the terminal device that generates the operation execution command corresponding to the sound waves and executes the operation command. For example, the mobile device 800-2 may generate a music reproduction command as a result of voice recognition according to the intention of the user 10, and determine the smart speaker 800-3 as the terminal device that executes the operation command. Then, the mobile device 800-2 may send the generated operation execution command to the smart speaker 800-3 (②).

[0175] The smart speaker 800-3 that receives the operation command from the mobile device 800-2 may perform an operation corresponding to the received command (③). For example, the smart speaker 800-3 may reproduce music based on the received reproduction command.

[0176] In the above description, the user 10 has generated a sound wave including a subject that performs an operation, but if the user generates a sound wave corresponding to the phrase "Please turn on the music", the mobile device 800-2 can determine the terminal device to execute the command based on various criteria. For example, the terminal device with the closest distance to the user 10 or the terminal device with the best audio output performance can be determined as the device that performs the operation command, and the operation command can be sent. Therefore, the mobile device 800-2 that has performed voice recognition can be determined as the device that performs the operation command. In this case, the mobile device 800-2 can directly provide a response corresponding to the voice recognition result to the user 10.

[0177] Meanwhile, if a plurality of terminal devices are determined as terminal devices to execute the operation command, the mobile device 800 - 2 may request the user 10 to select a device and determine a device to perform the final operation.

[0178] Fig.11 and Fig.12 An embodiment in which speech recognition is performed by an edge device according to an embodiment of the present disclosure is shown.

[0179] first, Fig.11 and Fig.12 The embodiment is shown after TV 800-1 having the highest P-score is identified as an edge device based on sharing of performance information of a plurality of terminal devices 800-1 to 800-3 connected to a network, and mobile device 800-2 is determined by TV 800-1 as a device for performing voice recognition.

[0180] Reference Fig.11 , after the user 10 says the trigger word or trigger phrase, the mobile device 800-2, which has received the sound wave "Please turn on the music on the speaker" generated by the user, can send the received voice signal to the TV 800-1 (①) which is an edge device for performing voice recognition.

[0181] TV 800-1 as an edge device may perform voice recognition of the received sound waves to determine the terminal device that generates an operation command corresponding to the sound waves and executes the operation command (②). For example, TV 800-1 as an edge device may generate a music reproduction command as a result of voice recognition according to the speech intention of user 10, and determine the smart speaker 800-3 as the terminal device that executes the operation command. Then, TV 800-1 as an edge device may send the generated operation execution command to the smart speaker 800-3 (③).

[0182] The smart speaker 800-3 receiving the operation command from the TV 800-1 as an edge device may perform an operation corresponding to the received command (④). For example, the smart speaker 800-3 may reproduce music based on the received reproduction command.

[0183] In the above description, the user 10 has generated a sound wave including the subject of the operation, but when the user 10 generates a sound wave associated with the phrase "turn on the music", the TV 800-1 as an edge device can determine the terminal device to execute the operation command. For example, the terminal device closest to the user 10 or the terminal device with the best audio output performance can be determined as the device to execute the operation command, and the operation command can be sent. Therefore, the TV 800-1 as an edge device can be determined as the device to execute the operation command. In this case, the TV 800-1 as an edge device can directly provide a response corresponding to the voice recognition result to the user 10.

[0184] When a plurality of terminal devices are determined as terminal devices to execute the operation command, the mobile device 800 - 2 may request the user 10 to select a device, and determine a device to perform the final operation according to the selection of the user 10 .

[0185] like Fig.12 As shown, when the terminal device 800-4 determined by the TV 800-1 as an edge device to execute the operation command corresponding to the sound waves generated by the user 10 is not a terminal device that directly communicates with the TV 800-1 as an edge device, the operation execution command can be sent to the terminal device 800-3 connected to both the edge device TV 800-1 and the terminal device 800-4 that executes the operation command.

[0186] This operation can be achieved by sharing the information of each terminal device. When the mobile device 800-2 performing voice recognition determines the terminal device to execute the operation command, Fig.12 The operation can be applied to a case where a terminal device that does not directly communicate with the mobile device 800-2 is determined as a device that executes the operation execution command.

[0187] Fig.13 An embodiment of changing a device receiving a sound wave for voice recognition when the first user 10 emits the sound wave while moving according to an embodiment of the present disclosure is shown.

[0188] Reference Fig.13 , when the first user 10 generates a sound wave including a trigger word or a trigger phrase (e.g., a wake-up word or a wake-up phrase) (①) and the first terminal device 1300-1 receives the sound wave generated by the user, the first terminal device 1300-1 may be activated according to the reception of the sound wave including the trigger word, and the first terminal device 1300-1 may identify the speaker of the received sound wave (e.g., the user generating the sound wave) (②). In an exemplary embodiment, the first terminal device 1300-1 may be a terminal device that has received the voice of the first user 10, or a selected terminal device that has acquired the sound wave for voice recognition among a plurality of terminal devices that have received the sound wave generated by the user.

[0189] Thereafter, when the sound quality of the sound wave received by the first terminal device 1300-1 decreases, the first terminal device 1300-1 may send a wake-up request to the peripheral devices 1300-2 to 1300-4. In an exemplary embodiment, due to the movement of the first user 10, the sound quality of the sound wave received at the first terminal device 1300-1 may decrease.

[0190] Then, the first terminal device 1300-1 may send information associated with the recognized sound waves together with the activation request (③). Therefore, even if the sound waves generated by the second user 20 are recognized in the activated second terminal device 1300-2, the second terminal device 1300-2 may not perform the voice recognition operation for the second user 20.

[0191] In an exemplary embodiment, after the peripheral devices 1300 - 2 to 1300 - 4 are activated, the peripheral devices may transmit a control command to minimize operation noise to at least some of the terminal devices among the peripheral devices to minimize noise of the surrounding environment.

[0192] For example, when the noise of the surrounding environment is equal to or greater than a preset value, the activated third terminal device 1300-3 may request a low noise operation mode (④) from the fourth terminal device 1300-3 and the fifth terminal device 1300-5 having a large operation noise. Therefore, the fourth terminal device 1300-4 (e.g., air conditioner) may be changed to a silent or still air mode, and the fifth terminal device 1300-5 (e.g., robot cleaner) may be changed to a low noise cleaning mode (⑤).

[0193] When the noise of the surrounding environment is less than a preset value, the activated third terminal device 1300 - 3 may omit the operation of requesting the low noise operation mode from other terminal devices.

[0194] Thereafter, when the first user 10 generates a sound wave associated with the phrase "news briefing" (⑥), among the first terminal device 1300-1 (a smart speaker capable of performing news briefing operations) and the third terminal device 1300-3 (smart TV) among the terminal devices 1300-1 to 1300-4 that receive the sound wave generated by the first user 10, the smart TV 1300-3 with good reception quality can perform operations corresponding to the sound wave and voice recognition of the first user 10.

[0195] When the first user 10 generates a sound wave associated with the phrase “turn on still air mode” (⑦), among the terminal devices 1300-1 to 1300-4 that receive the sound wave generated by the first user 10, the fourth terminal device 1300-4 capable of operating in still air mode can perform an operation corresponding to voice recognition based on the sound wave generated by the first user 10.

[0196] When the first user 10 generates sound waves while moving, when the reception quality of the sound waves received by the fourth terminal device 1300-4 activated by the first terminal device 1300-1 is less than a preset value, the fourth terminal device 1300-4 may send information related to the identified sound and / or the identified user together with the activation request to the peripheral terminal device.

[0197] As described above, by receiving a trigger word or a trigger phrase and requesting the activated terminal device to activate the peripheral terminal device according to the movement of the first user, the first user 10 can control various devices with voice commands without repeatedly saying the trigger word or trigger phrase, thereby improving the convenience of user recognition performance and voice recognition performance.

[0198] If the first user 10 speaks a trigger word for deactivating the voice recognition function of the terminal device or does not generate sound waves within a predetermined time, the terminal device may terminate the voice recognition function. For example, if the first user 10 speaks a trigger word or phrase for deactivating the voice recognition function such as "Goodbye, Bixby" or does not generate sound waves within a predetermined time, the terminal may end the voice recognition function.

[0199] Fig.14 is a flowchart describing a control method of a terminal device according to an embodiment of the present disclosure.

[0200] Reference Fig.14 In operation S1401, the terminal device may receive performance information of one or more other terminal devices from each of the one or more other terminal devices. In an exemplary embodiment, the received performance information may include information about a terminal device that communicates with another terminal device but does not communicate with the terminal device.

[0201] In operation S1402, the terminal device may identify an edge device based on the received performance information. For example, the edge device may be a device used to determine a device to perform speech recognition among multiple terminal devices. The edge device may be determined in consideration of processing performance, computing performance, mobile status, etc.

[0202] If the terminal device is identified as an edge device according to the above criteria in operation S1403-Y, then in operation S1404, the terminal device may receive information related to reception quality from other terminal devices that receive a signal including a trigger word.

[0203] In operation S1405, the terminal device may determine a terminal device for acquiring a sound wave generated by a user for speech recognition based on performance information of other terminal devices and information related to reception quality. In an exemplary embodiment, the terminal device for performing speech recognition may be a device for acquiring a sound wave generated by a user for speech recognition. The terminal device may send a speech recognition execution command to one or more other terminal devices determined to perform speech recognition.

[0204] In operation S1403-N, when the terminal device is not an edge device, if the terminal device receives a sound wave including a trigger word or a trigger phrase, in operation S1406, information associated with the reception quality of the sound wave may be sent to the terminal device determined as an edge device. For example, information associated with the voice reception quality may be obtained based on the received sound wave generated by the user. In an exemplary embodiment, if the voice reception quality is lower than a preset level, the terminal device may discard the information associated with the reception quality of the sound wave without sending the information associated with the reception quality of the sound wave to the edge device.

[0205] Thereafter, when a voice recognition command is received from the edge device, the terminal device may perform voice recognition on the sound wave following the sound wave following the trigger word or trigger phrase. For example, voice recognition may refer to an operation of sending a received user-generated sound wave to a server for voice recognition, or an operation of the terminal device performing voice recognition to determine a user intent associated with the sound wave.

[0206] As described above, even if multiple terminal devices have received the sound waves generated by the user, voice recognition can be performed by a single device determined by the edge device, thereby reducing unnecessary resource consumption and preventing conflicts between devices when performing operations on commands included in the sound waves generated by the user.

[0207] Fig.15 The operation of the speech recognition system according to an embodiment of the present disclosure is illustrated.

[0208] Reference Fig.15 , the speech recognition system 1000 of the present disclosure may include a plurality of terminal devices 100-1 to 100-4 and a server 200. In an exemplary embodiment, the server 200 may include a configuration for determining the user's intention through speech recognition. If the terminal device can perform speech recognition, the server 200 may be omitted.

[0209] In operation S1501, a plurality of terminal devices 100-1 to 100-4 in the speech recognition system 1000 may share performance information and determine edge devices. Fig.15 In the example, the first terminal device 100 - 1 is an edge device.

[0210] Thereafter, when at least one of the plurality of terminal devices 100-1 to 100-4 receives a sound wave including a trigger word or a trigger phrase generated by a user, at least one terminal device that has received the sound wave including the trigger word may send reception quality information associated with the received sound wave to the edge device 100-1. In an exemplary embodiment, the reception quality information may be a Q-score obtained by at least one terminal device that has received the sound wave including the trigger word or the trigger phrase, and may be the received sound wave. In an exemplary embodiment, a score for the reception quality of each terminal device may be calculated by the edge device.

[0211] For example, when in operation S1502, the second terminal device 100-2 and the third terminal device 100-3 among multiple terminal devices 100-1 to 100-4 receive sound waves generated by the user including a trigger word or a trigger phrase, in operations S1503 and S1504, the second terminal device 100-2 and the third terminal device 100-3 may send reception quality information to the edge device 100-1.

[0212] In operation S1505, the first terminal device 100-1 as an edge device may determine a voice recognition device based on information related to the received voice reception quality. For example, the voice recognition device may be a device that acquires sound waves generated by a user for voice recognition. The first terminal device 100-1 as an edge device may send a voice recognition command to the terminal device determined to perform voice recognition. For example, if the second terminal device 100-2 is determined as a terminal device for performing voice recognition, in operation S1506, the first terminal device 100-1 may send a voice recognition command to the second terminal device.

[0213] Even after the second terminal device 100-2 and the third terminal device 100-3 receive the sound wave including the trigger word or the trigger phrase in operation S1507, only the second terminal device 100-2 that receives the voice recognition command may transmit the received sound wave to the server 200 in operation S1508. Fig.15 , although operations S1502 and S1507, which are respectively the first sound wave receiving operation and the second sound wave receiving operation, are shown as separate operations, in actual implementation, the operations may be continuous operations, and operation S1507 may be performed simultaneously with operations S1503 to S1506. For example, operations S1502 and S1507 may be a single sound wave generated by a user.

[0214] In operation S1509, the server 200 may perform voice recognition on the received sound wave and may determine a device to perform an operation corresponding to the sound wave. For example, the server 200 may generate an operation command corresponding to the sound wave based on the voice recognition result and determine a terminal device to perform the operation command.

[0215] The server 200 may send the generated operation execution command to the terminal device determined to execute the operation command. For example, if the terminal device determined to execute the operation command is the fourth terminal device 100-4, then in operation S1510, the server 200 may send the operation execution command to the fourth terminal device 100-4. In operation S1511, the fourth terminal device 100-4 that receives the operation execution command may execute the operation corresponding to the command.

[0216] As described above, even if multiple terminal devices have received the sound waves generated by the user, voice recognition can be performed by one device determined by the edge device, and an operation execution command can be sent only to one terminal device based on the result of the voice recognition, thereby reducing unnecessary resource consumption and preventing conflicts between devices when executing operations on commands included in the sound waves generated by the user.

[0217] At the same time, the various embodiments described above can be implemented in a recording medium, which can be read by a computer or a similar device using software, hardware or a combination thereof. According to hardware implementation, the embodiments described in this disclosure can be implemented using at least one of an application specific integrated circuit (ASIC), a DSP, a digital signal processing device (DSPD), a programmable logic device (PLD), an FPGA, a processor, a controller, a microcontroller, a microprocessor and an electrical unit for performing other functions. In some cases, the embodiments described herein can be implemented by the processor itself. According to software implementation, the embodiments such as the processes and functions described herein can be implemented with separate software modules. Each of the software modules can perform one or more of the functions and operations described herein.

[0218] Meanwhile, the methods according to the various embodiments disclosed above may be stored in a non-transitory readable medium, which may be used in various devices.

[0219] Non-volatile computer-readable media refers to media (such as registers, caches, memories, etc.) that store data for a short period of time, but store data semi-permanently and can be read by a device. Specific examples of non-transitory computer-readable media include compact discs (CDs), DVDs, hard disks, Blu-ray discs, USBs, memory cards, read-only memories (ROMs), etc.

[0220] According to one embodiment, the method according to various embodiments of the present disclosure may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be published in the form of a machine-readable storage medium (e.g., a CD-ROM) or distributed through an application store (e.g., PlayStore). TM) Online publishing. In the case of online publishing, at least a part of the computer program product may be temporarily or at least temporarily stored in a storage medium (such as a memory in a manufacturer's server, an application store's server, or a relay server).

[0221] While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. A terminal device, comprising: a communication interface, including a communication circuit; as well as Processor, configured as: receiving, via the communication interface, performance information of the one or more other terminal devices from each of the one or more other terminal devices, identifying an edge device from the terminal device and the one or more other terminal devices based on the performance information received from each of the one or more other terminal devices, the edge device being used to determine a terminal device on which speech recognition is to be performed, receiving information associated with reception quality from one or more other terminal devices that receive the sound wave including the trigger word among the one or more other terminal devices that transmit the performance information based on the terminal device being identified as the edge device, determining, based on the received information associated with the reception quality, a terminal device on which speech recognition is to be performed from the one or more other terminal devices that received the sound wave, and A command is sent to the determined terminal device to send the sound waves for voice recognition acquired by the determined terminal device to an external voice recognition device.

2. The terminal device according to claim 1, in, The performance information received from each of the one or more other terminal devices includes: a first score obtained based on the performance of the one or more other terminal devices, and The information associated with the reception quality received from the one or more other terminal devices that receive the sound waves includes: a second score representing the reception quality associated with the sound waves received by the one or more other terminal devices.

3. The terminal device according to claim 2, wherein: The processor is further configured to determine a terminal device having a highest second score among the one or more other terminal devices that receive the sound wave as a terminal device for acquiring the sound wave for speech recognition.

4. The terminal device according to claim 2, further comprising: microphone, Wherein, the processor is further configured to: Based on receiving the sound wave including the trigger word at the microphone, obtaining the second score of the terminal device based on at least a portion of the sound wave; determining a terminal device having a highest second score among the terminal device and the one or more other terminal devices that receive the sound wave as a terminal device that acquires the sound wave for speech recognition; and Acquiring a portion of the sound wave received after the trigger word based on the terminal device being determined as the terminal device that acquires the sound wave; and The acquired portion of the sound wave is sent to the external speech recognition device.

5. The terminal device according to claim 2, wherein: The information related to the reception quality is acquired based on at least one of: a signal-to-noise ratio SNR, a volume of the received sound wave, a rate of change of the volume of the received sound wave, a sound pressure, and a distance from a source of the sound wave.

6. The terminal device according to claim 2, wherein: The performance information is acquired based on at least one of the following: the number of other terminal devices connected, the number of types of available communication networks, a connection method with a server, a power supply method, processing performance, storage performance, and a location movement status.

7. The terminal device according to claim 2, wherein: The processor is also configured to: based on the second score of the determined terminal device being less than a preset value, change the terminal device with the highest second score among the one or more other terminal devices that receive the sound wave to the terminal device that obtains the sound wave for speech recognition.

8. The terminal device according to claim 2, wherein: At least one of the first score or the second score received from each of the one or more other terminal devices includes at least one of the first score or the second score information about terminal devices that communicate with the one or more other terminal devices and do not communicate with the terminal device.

9. The terminal device according to claim 1, further comprising: memory, including a speech recognition module, Wherein, the processor is further configured to: performing speech recognition on the sound wave using the speech recognition module based on receiving the sound wave from the determined terminal device, and Sending speech recognition results to an external server, and Wherein, the external server is configured as follows: Based on the speech recognition result, determining a terminal device from the terminal device and the one or more other terminal devices that is to perform an operation corresponding to the speech recognition result, and An operation execution command corresponding to the voice recognition result is sent to the determined terminal device to execute the operation.

10. The terminal device according to claim 2, further comprising: microphone, Wherein, the processor is further configured to: Based on the first score of the terminal device being less than the first score of the one or more other terminal devices, determining the one or more other terminal devices with the highest first score among the one or more other terminal devices as edge devices, Based on receiving the sound wave including the trigger word at the microphone, obtaining information associated with reception quality based on at least a portion of the sound wave, and The acquired information related to the reception quality is sent to the determined edge device.

11. The terminal device according to claim 10, wherein: The processor is further configured to: based on the second score of the terminal device being less than a preset value, not send the second score to the determined edge device.

12. A method for controlling a terminal device, the method comprising: receiving, from each of the one or more other terminal devices, information relating to performance information of the one or more other terminal devices; identifying an edge device from the terminal device and the one or more other terminal devices based on the performance information received from each of the one or more other terminal devices, the edge device being used to determine an edge device of a terminal device on which speech recognition is to be performed; receiving, based on the terminal device being identified as the edge device, information associated with reception quality from one or more other terminal devices that receive the sound wave including the trigger word among the one or more other terminal devices that transmit the performance information; determining, based on the received information associated with the reception quality, a terminal device on which speech recognition is to be performed from the one or more other terminal devices that received the sound wave; as well as A command is sent to the determined terminal device to send the sound waves for voice recognition acquired by the determined terminal device to an external voice recognition device.

13. The method according to claim 12, in, The performance information received from each of the one or more other terminal devices includes: a first score obtained based on the performance of the one or more other terminal devices, and The information associated with the reception quality received from the one or more other terminal devices that receive the sound waves includes: a second score representing the reception quality associated with the sound waves received by the one or more other terminal devices.

14. The method according to claim 13, wherein: Determining the terminal device includes: determining a terminal device having a highest second score among the one or more other terminal devices that receive the sound wave as a terminal device that acquires the sound wave for speech recognition.

15. The method according to claim 13, wherein: Determining that the terminal device comprises: Based on receiving the sound wave including the trigger word at the microphone of the terminal device, obtaining the second score of the terminal device based on at least a part of the sound wave; determining the terminal device having the highest second score among the terminal device and the one or more other terminal devices receiving the sound wave as the terminal device for acquiring the sound wave for speech recognition, Based on the terminal device being determined as the terminal device that acquires the sound wave, acquiring a portion of the sound wave received after the trigger word, and The acquired portion of the sound wave is sent to the external speech recognition device.

Images