Electronic device for processing voice signal, operating method thereof, and storage medium
The electronic device improves voice recognition accuracy by detecting a call word, outputting a notification sound, and removing overlapping notification sounds from the voice signal using a reference signal, addressing noise interference and enhancing user interaction.
Patent Information
- Application Number
- PCT/KR2024/005887
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-27
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-07
AI Technical Summary
Current electronic devices face challenges in providing high-quality voice recognition services due to noise interference and notification sounds overlapping with user voice signals, leading to reduced voice recognition accuracy.
An electronic device equipped with a processor and memory that detects a call word, outputs a preset notification sound, and removes the notification sound signal from the voice signal using a reference signal to improve voice recognition accuracy by preprocessing the signal before recognition.
The solution enhances voice recognition performance by isolating user voice from notification sounds, thereby improving the accuracy of speech recognition and reducing unintended operations.
Smart Images

Figure KR2024005887_07112024_PF_FP_ABST
Abstract
Description
Electronic device for processing voice signals, method of operation thereof and storage medium
[0001] One embodiment disclosed in this document relates to an electronic device for processing a voice signal, a method of operating the same, and a storage medium.
[0002] The variety of services and additional features offered through electronic devices such as smartphones is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated. Recently, various types of intelligence services for electronic devices have been offered. One such intelligence service, voice recognition, allows users to control electronic devices through voice recognition, providing them with a variety of services.
[0003] To provide voice recognition services, electronic devices can be equipped with technologies that recognize and analyze human language (e.g., automatic speech recognition, natural language understanding, natural language generation, machine translation, conversational systems, question-and-answering, and speech recognition / synthesis). To provide high-quality voice recognition services to consumers, voice recognition performance must be improved. To this end, research and development are underway to improve voice recognition performance by removing noise from the input voice signal or adjusting the signal amplitude during the preprocessing stage.
[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.
[0005] According to one embodiment, an electronic device may include a microphone, at least one processor, and a memory. According to one embodiment, instructions stored in the memory may be configured such that, when executed by the at least one processor, the electronic device receives a first voice signal through the microphone, detects a wake word from the first voice signal, outputs a preset notification sound in response to the wake word detection, and removes a wake sound signal from a second voice signal received through the microphone based on a position of the detected wake sound signal in the second voice signal, wherein the wake sound signal corresponds to the output preset notification sound.
[0006] According to one embodiment, a method for processing a voice signal in an electronic device includes receiving a first voice signal through a microphone of the electronic device, detecting a wake word from the first voice signal, outputting a preset notification sound in response to detection of the wake word, and removing a notification sound signal from a second voice signal received through the microphone based on a position of the detected notification sound signal in the second voice signal, wherein the notification sound signal corresponds to the output preset notification sound.
[0007] According to one embodiment, a non-transitory storage medium storing instructions, wherein the instructions, when executed by at least one processor of an electronic device, are configured to cause the electronic device to perform at least one operation, the at least one operation including: receiving a first voice signal through a microphone of the electronic device, detecting a wake word from the first voice signal, outputting a preset notification sound in response to detection of the wake word, and removing a notification sound signal from a second voice signal received through the microphone based on a position of the detected notification sound signal in the second voice signal, wherein the notification sound signal corresponds to the output preset notification sound.
[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.
[0009] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.
[0010] Figure 3 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to one embodiment.
[0011] FIG. 4 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app according to one embodiment.
[0012] Figure 5a is an internal block diagram of an electronic device according to one embodiment.
[0013] FIG. 5b is a detailed block diagram of an electronic device for processing a voice signal according to one embodiment.
[0014] FIG. 6 is an example diagram showing input signals in cases where the notification sound and the user voice do not overlap and overlap according to one embodiment.
[0015] Figure 7 is an example diagram showing signals corresponding to an original notification sound and a recorded notification sound according to one embodiment.
[0016] Figure 8 is a flowchart of operations in an electronic device for processing a voice signal according to one embodiment.
[0017] Figure 9 is a detailed operation flowchart in an electronic device according to one embodiment.
[0018] Fig. 10 is an exemplary diagram showing a spectrogram of a reference signal for removing a notification sound according to one embodiment.
[0019] Fig. 11 is an example diagram showing a spectrogram of a reference signal for notification sound search according to one embodiment.
[0020] FIG. 12 is an exemplary diagram illustrating a method for removing a notification sound from an input voice signal according to one embodiment.
[0021] FIG. 13 is an exemplary diagram showing an input signal before / after removing a notification sound when the notification sound and the user voice do not overlap according to one embodiment, and an input signal before / after removing a notification sound when the notification sound and the user voice overlap.
[0022] FIG. 14 is a diagram for explaining a preprocessing operation for removing a notification sound before voice recognition according to one embodiment.
[0023] FIG. 15 is a flowchart illustrating an operation of an electronic device for generating and storing a reference signal according to an embodiment.
[0024] FIG. 16 is an example screen diagram providing guide information for storing (or registering) a reference signal according to one embodiment.
[0025] FIG. 17 is a diagram illustrating a method for managing notification sounds for a specific speaker according to one embodiment.
[0026] FIG. 18 is a diagram for explaining a preprocessing operation for removing a notification sound for a specific speaker before voice recognition according to one embodiment.
[0027] Figure 19 is an example screen showing the results of voice recognition before and after removing the notification sound according to one embodiment.
[0028] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.
[0029] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100), according to one embodiment. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).
[0030] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.
[0031] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.
[0032] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).
[0033] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).
[0034] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).
[0035] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.
[0036] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.
[0037] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).
[0038] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
[0039] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
[0040] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).
[0041] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
[0042] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.
[0043] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).
[0044] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.
[0045] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).
[0046] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.
[0047] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).
[0048] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.
[0049] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).
[0050] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.
[0051] FIG. 2 is a block diagram illustrating an integrated intelligence system according to one embodiment.
[0052] Referring to FIG. 2, an integrated intelligent system of one embodiment may include a user terminal (290) (e.g., electronic device (101) of FIG. 1), an intelligent server (200) (e.g., server (108) of FIG. 1), and a service server (300) (e.g., server (108) of FIG. 1).
[0053] The user terminal (290) of one embodiment may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.
[0054] According to the illustrated embodiment, the user terminal (290) may include a communication interface (291) (e.g., the interface (177) of FIG. 1), a microphone (295) (e.g., the input module (150) of FIG. 1), a speaker (294) (e.g., the audio output module (155) of FIG. 1), a display (293) (e.g., the display module (160) of FIG. 1), a memory (299) (e.g., the memory (130) of FIG. 1), or a processor (292) (e.g., the processor (120) of FIG. 1). The above-listed components may be operatively or electrically connected to each other.
[0055] The communication interface (291) of one embodiment may be configured to connect to an external device and transmit and receive data. The microphone (295) of one embodiment may receive sound (e.g., user speech) and convert it into an electrical signal. The speaker (294) of one embodiment may output the electrical signal as sound (e.g., voice). The display (293) of one embodiment may be configured to display an image or video. The display (293) of one embodiment may also display a graphical user interface (GUI) of a running app (or application program). The display (293) of one embodiment may receive a touch input via a touch sensor. For example, the display (293) may receive a text input via a touch sensor in an on-screen keyboard area displayed within the display (293).
[0056] In one embodiment, the memory (299) may store a client module (298), a software development kit (SDK) (297), and a plurality of apps (296). The client module (298) and the SDK (297) may constitute a framework (or solution program) for performing general functions. In addition, the client module (298) or the SDK (297) may constitute a framework for processing user input (e.g., voice input, text input, touch input).
[0057] The plurality of apps (296) stored in the memory (299) of one embodiment may be programs for performing a specified function. According to one embodiment, the plurality of apps (296) may include a first app (296_1) and a second app (296_2). According to one embodiment, each of the plurality of apps (296) may include a plurality of operations for performing a specified function. For example, the apps may include an alarm app, a message app, and / or a schedule app. According to one embodiment, the plurality of apps (296) may be executed by the processor (292) to sequentially execute at least some of the plurality of operations.
[0058] The processor (292) of one embodiment can control the overall operation of the user terminal (290). For example, the processor (292) can be electrically connected to a communication interface (291), a microphone (295), a speaker (294), and a display (293) to perform a designated operation.
[0059] The processor (292) of one embodiment may also execute a program stored in the memory (299) to perform a designated function. For example, the processor (292) may execute at least one of the client module (298) or the SDK (297) to perform the following operations for processing voice input. The processor (292) may control the operations of multiple apps (296), for example, through the SDK (297). The following operations described as operations of the client module (298) or the SDK (297) may be operations executed by the processor (292).
[0060] The client module (298) of one embodiment can receive user input. For example, the client module (298) can receive a voice signal corresponding to a user utterance detected through a microphone (295). Alternatively, the client module (298) can receive a touch input detected through a display (293). Alternatively, the client module (298) can receive a text input detected through a keyboard or a screen keyboard. In addition, the client module (298) can receive various forms of user input detected through an input module included in the user terminal (290) or an input module connected to the user terminal (290). The client module (298) can transmit the received user input to the intelligent server (200). The client module (298) can transmit status information of the user terminal (290) together with the received user input to the intelligent server (200). The status information can be, for example, execution status information of an app.
[0061] The client module (298) of one embodiment can receive a result corresponding to the received user input. For example, the client module (298) can receive a result corresponding to the received user input if the intelligent server (200) can produce a result corresponding to the received user input. The client module (298) can display the received result on the display (293). In addition, the client module (298) can output the received result as audio through the speaker (294).
[0062] In one embodiment, the client module (298) may receive a plan corresponding to the received user input. The client module (298) may display the results of executing multiple operations of the app according to the plan on the display (293). For example, the client module (298) may sequentially display the results of executing multiple operations on the display (293) and output audio through the speaker (294). The user terminal (290) may, for example, display only some results of executing multiple operations (e.g., the result of the last operation) on the display (293).
[0063] In one embodiment, the client module (298) may receive a request from the intelligent server (200) to obtain information necessary to produce a result corresponding to a user input. In one embodiment, the client module (298) may transmit the necessary information to the intelligent server (200) in response to the request.
[0064] The client module (298) of one embodiment can transmit result information of executing multiple operations according to a plan to the intelligent server (200). The intelligent server (200) can use the result information to confirm that the received user input has been processed correctly.
[0065] The client module (298) of one embodiment may include a voice recognition module. According to one embodiment, the client module (298) may recognize voice inputs to perform limited functions through the voice recognition module. For example, the client module (298) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").
[0066] An intelligent server (200) of one embodiment may receive information related to a user voice input from a user terminal (290) via a communication network. According to one embodiment, the intelligent server (200) may convert data related to the received voice input into text data. According to one embodiment, the intelligent server (200) may generate a plan for performing a task corresponding to the user voice input based on the text data.
[0067] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the foregoing or another AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI system may select at least one plan from a plurality of predefined plans.
[0068] An intelligent server (200) of one embodiment may transmit results according to a generated plan to a user terminal (290), or transmit the generated plan to the user terminal (290). According to one embodiment, the user terminal (290) may display results according to the plan on a display (293). According to one embodiment, the user terminal (290) may display results of executing an operation according to the plan on a display (293).
[0069] An intelligent server (200) of one embodiment may include a front end (210), a natural language platform (220), a capsule database (230), an execution engine (240), an end user interface (250), a management platform (260), a big data platform (270), or an analytic platform (280).
[0070] The front end (210) of one embodiment can receive user input from a user terminal (290). The front end (210) can transmit a response corresponding to the user input.
[0071] According to one embodiment, the natural language platform (220) may include an automatic speech recognition module (ASR module) (221), a natural language understanding module (NLU module) (223), a planner module (225), a natural language generator module (NLG module) (227), or a text to speech module (TTS module) (229).
[0072] The automatic speech recognition module (221) of one embodiment can convert voice input received from the user terminal (290) into text data. The natural language understanding module (223) of one embodiment can identify the user's intention using the text data of the voice input. For example, the natural language understanding module (223) can identify the user's intention by performing syntactic analysis or semantic analysis on user input in the form of text data. The natural language understanding module (223) of one embodiment can identify the meaning of words extracted from voice input using linguistic features (e.g., grammatical elements) of morphemes or phrases, and can determine the user's intention by matching the meaning of the identified words to the intent. The natural language understanding module (223) can obtain intent information corresponding to the user's utterance. The intent information may be information indicating the user's intent determined by interpreting text data. The intent information may include information indicating an action or function that the user intends to execute using the device.
[0073] The planner module (225) of one embodiment can generate a plan using the intent and parameters determined by the natural language understanding module (223). According to one embodiment, the planner module (225) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (225) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (225) can determine parameters necessary to execute the determined plurality of operations or result values output by the execution of the plurality of operations. The parameters and the result values can be defined as concepts of a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (225) can determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (225) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (225) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (225) can generate a plan including association information (e.g., ontology) between a plurality of actions and a plurality of concepts. The planner module (225) can generate the plan using information stored in a capsule database (230) in which a set of relationships between concepts and actions is stored.
[0074] The natural language generation module (227) of one embodiment can convert specified information into text format. The information converted into text format may be in the form of natural language utterances. The text-to-speech conversion module (229) of one embodiment can convert information in text format into information in speech format.
[0075] According to one embodiment, some or all of the functions of the natural language platform (220) may also be implemented in the user terminal (290).
[0076] The capsule database (230) may store information on the relationships between a plurality of concepts and actions corresponding to a plurality of domains. According to one embodiment, a capsule may include a plurality of action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (230) may store a plurality of capsules in the form of a concept action network (CAN). According to one embodiment, the plurality of capsules may be stored in a function registry included in the capsule database (230).
[0077] The capsule database (230) may include a strategy registry that stores strategy information required when determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a user input. According to one embodiment, the capsule database (230) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (230) may include a layout registry that stores layout information of information output through the user terminal (290). According to one embodiment, the capsule database (230) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (230) may include a dialog registry that stores information on dialogue (or interaction) with a user. The capsule database (230) may update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for creating and registering a strategy that determines a plan. The developer tool may include a dialog editor for creating a dialogue with a user.The developer tool may include a follow-up editor that activates follow-up goals and allows editing of follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions. In one embodiment, the capsule database (230) may also be implemented within the user terminal (290).
[0078] The execution engine (240) of one embodiment can produce a result using the generated plan. The end user interface (250) can transmit the produced result to the user terminal (290). Accordingly, the user terminal (290) can receive the result and provide the received result to the user. The management platform (260) of one embodiment can manage information used in the intelligent server (200). The big data platform (270) of one embodiment can collect user data. The analysis platform (280) of one embodiment can manage the quality of service (QoS) of the intelligent server (200). For example, the analysis platform (280) can manage the components and processing speed (or efficiency) of the intelligent server (200).
[0079] In one embodiment, the service server (300) may provide a designated service (e.g., food ordering or hotel reservation) to the user terminal (290). According to one embodiment, the service server (300) may be a server operated by a third party. In one embodiment, the service server (300) may provide information for generating a plan corresponding to the received user input to the intelligent server (200). The provided information may be stored in the capsule database (230). In addition, the service server (300) may provide result information according to the plan to the intelligent server (200).
[0080] In the integrated intelligence system (10) described above, the user terminal (290) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.
[0081] In one embodiment, the user terminal (290) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (290) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.
[0082] In one embodiment, the user terminal (290) may perform a designated action based on the received voice input, either alone or together with the intelligent server (200) and / or the service server (300). For example, the user terminal (290) may execute an app corresponding to the received voice input and perform a designated action through the executed app.
[0083] In one embodiment, when a user terminal (290) provides a service together with an intelligent server (200) and / or a service server (300), the user terminal (290) can detect a user's speech using the microphone (295) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal (290) can transmit the voice data to the intelligent server (200) using a communication interface (291).
[0084] According to one embodiment, an intelligent server (200) may generate a plan for performing a task corresponding to a voice input received from a user terminal (290), or a result of performing an operation according to the plan, in response to a voice input. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations, or result values output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.
[0085] In one embodiment, the user terminal (290) can receive the response using the communication interface (291). The user terminal (290) can output a voice signal generated within the user terminal (290) to the outside using the speaker (294), or can output an image generated within the user terminal (290) to the outside using the display (293).
[0086] FIG. 3 is a drawing (400) showing a form in which relationship information between concepts and actions according to one embodiment is stored in a database.
[0087] The capsule database (e.g., the capsule database (230) of FIG. 2) of the intelligent server (200) (e.g., the intelligent server (200) of FIG. 2) may store capsules in the form of a CAN (concept action network). The capsule database may store operations for processing tasks corresponding to a user's voice input and parameters necessary for the operations in the form of a CAN (concept action network).
[0088] The capsule database may store a plurality of capsules (capsule (A) (401), capsule (B) (404)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., capsule (A) (401)) may correspond to one domain (e.g., location (geo), application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (402) or CP 2 (403)) for performing a function for a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (410) and at least one concept (420) for performing a specified function.
[0089] The above natural language platform (e.g., the natural language platform (220) of FIG. 2) can generate a plan for performing a task corresponding to a received voice input using capsules stored in a capsule database. For example, the planner module of the natural language platform (e.g., the planner module (225) of FIG. 2) can generate a plan using capsules stored in a capsule database. For example, a plan (407) can be generated using actions (4011, 4013) and concepts (4012, 4014) of capsule A (401) and actions (4041) and concepts (4042) of capsule B (404).
[0090] FIG. 4 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app according to one embodiment.
[0091] The user terminal (290) can execute an intelligent app to process user input through an intelligent server (e.g., the intelligent server (200) of FIG. 2).
[0092] According to one embodiment, on screen 310, when the user terminal (290) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the user terminal (290) may execute an intelligent app for processing the voice input. For example, the user terminal (290) may execute the intelligent app while the schedule app is running. According to one embodiment, the user terminal (290) may display an object (e.g., an icon) (311) corresponding to the intelligent app on the display (293) (e.g., the display module (160) of FIG. 1, the display (293) of FIG. 2). According to one embodiment, the user terminal (290) may receive a voice input by a user's speech. For example, the user terminal (290) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the user terminal (290) can display a user interface (UI) (313) (e.g., an input window) of an intelligent app on which text data of a received voice input is displayed.
[0093] According to one embodiment, on screen 320, the user terminal (290) may display a result corresponding to the received voice input on the display. For example, the user terminal (290) may receive a plan corresponding to the received user input and display "This Week's Schedule" on the display according to the plan.
[0094] In the detailed description below, reference numerals in the drawings may be used interchangeably or omitted for components that can be easily understood through the preceding embodiments, and their detailed descriptions may also be omitted. The electronic device (101) according to one embodiment disclosed in this document may be implemented by selectively combining components of different embodiments, and components of one embodiment may be replaced by components of another embodiment. For example, it should be noted that the present invention is not limited to specific drawings or embodiments.
[0095] Meanwhile, voice recognition technology recognizes human speech and identifies linguistic meaning from the recognized speech. Electronic devices providing voice recognition services can receive control commands via the user's voice. To execute their functions based on the user's voice, the electronic device can recognize the user's speech.
[0096] In this case, the electronic device can recognize not only the voice signal of the user's speech but also various sounds generated in the surrounding environment. To improve the accuracy of voice recognition, methods for removing noise included in the voice signal input in the preprocessing stage or adjusting the signal size are being researched and developed. In addition, there is a technology called WoV (wake on voice) that notifies the electronic device when the user will initiate voice recognition operation. For example, when the user utters a specific wake-up command (or wake word, wakeup word, predetermined word, or first command - these terms can be used in place of a specific wake-up command) (e.g., "hi bixby"), the voice recognition function of the electronic device is activated in response to the specific wake-up command. For example, wake-up word recognition is performed on a low-power processor that is separate from the processor that performs the voice recognition function. For example, wake-up word recognition is currently performed on a processor operating in low-power mode, and after the wake-up word is detected, it is performed on a processor operating in high-performance mode. Additionally, the call word recognition can be performed when the processor is operating in low power mode, and after the call word is detected, the processor can switch to high performance mode and perform the operation after the call word detection.
[0097] However, when performing voice recognition for a section where the user does not speak after the wake word, unexpected text may be output due to surrounding noise. For example, if the user accidentally speaks the wake word or the electronic device incorrectly detects the wake word, the user may not attempt to provide additional speech (e.g., instructions, commands, etc.) to the electronic device, but the voice recognition function may be activated and detect background conversation (e.g., the user speaking to another person or object). This may unnecessarily drain the battery of the electronic device or cause the user to perform unintended / unwanted actions. Therefore, the electronic device needs to perform voice recognition by focusing on the voice of the user who spoke the wake word after the wake word. The electronic device may output (or play) a preset notification sound (hereinafter, “prompt,” “warning,” or “sound indication”) to induce (e.g., prompt or request) the user to speak after recognizing the wake word. A preset notification sound may be a sound played back through the speaker of an electronic device to prompt the user to speak further, rather than a voice actually spoken by the user. For example, a preset notification sound may be a pre-acquired special voice or a voice synthesized using text-to-speech (TTS) from a text pre-specified by the user (hereinafter referred to as a "TTS voice"). A notification sound output (or played back) through the speaker of an electronic device after a wake-up word is input through a microphone on the electronic device may be referred to as a verbal listening sound (VLS).
[0098] In an electronic device, if a voice signal corresponding to the user's speech is not input within a certain period of time after recognizing a wake-up word, a notification sound may be output to prompt the user to speak again. If a voice signal corresponding to the user's speech is input after the notification sound has been output, the electronic device may output a voice recognition result by removing the patterns corresponding to the wake-up word and the notification sound. However, if the voice signal corresponding to the user's speech is input with at least a partial overlap with the notification sound, the overlap between the notification sound and the user's speech may cause the voice signal itself to be distorted, which may result in a voice recognition error. Since the notification sound can affect voice recognition in this way, it is necessary to remove the notification sound in a preprocessing stage before voice recognition.
[0099] In one embodiment, an electronic device, a method of operation thereof, and a storage medium are provided for processing a voice signal including a notification sound in a preprocessing step to further improve a voice recognition rate.
[0100] Figure 5a is an internal block diagram of an electronic device according to one embodiment.
[0101] Referring to FIG. 5A, an electronic device (101) according to one embodiment (e.g., electronic device (101) of FIG. 1, user terminal (290) of FIG. 2) includes at least one processor (520) (e.g., processor (120) of FIG. 1, processor (292) of FIG. 2) and a microphone (550) (e.g., input module (150) of FIG. 1, microphone (295) of FIG. 2). According to one embodiment, the electronic device (101) further includes a memory (530) (e.g., the memory (130) of FIG. 1, the memory (299) of FIG. 2), a display (560) (e.g., the display module (160) of FIG. 1, the display (293) of FIG. 2), a speaker (555) (e.g., the audio output module (155) of FIG. 1, the speaker (294) of FIG. 2), and / or a communication module (590) (e.g., the communication module (190) of FIG. 1, the communication interface (291) of FIG. 2). Here, not all components illustrated in FIG. 5A are essential components of the electronic device (101), and the electronic device (101) may be implemented with more or fewer components than the components illustrated in FIG. 5A.
[0102] The processor (520) can control the overall operation of an electronic device (101) equipped with a voice recognition function. For example, the processor (520) can be implemented as an application processor (AP).
[0103] The processor (520) can receive a voice signal (e.g., a signal representing speech) spoken by a user through a microphone (550). The processor (520) can perform voice recognition on the user's voice (e.g., the user's voice is included in the voice signal (i.e., can be obtained from the voice signal)) and output the voice recognition result through, for example, a display (560) or a speaker (555). The received voice signal may include not only the voice according to the user's speech but also noise (e.g., environmental or surrounding noise). For example, the received voice signal includes a start command (or a wake word) for voice recognition. The processor (520) can recognize the wake word included in the received voice signal. In one embodiment, the call word recognition includes speaker recognition (e.g., speaker verification, speaker identification; i.e., the process of recognizing the call word includes verifying or identifying the speaker, for example, to check whether the speaker is authorized or authorized to use the electronic device or voice recognition function).
[0104] In addition, when a predetermined notification sound is output (or played) through the speaker (555) to induce the user's speech after the call word is recognized, the processor (520) may receive (e.g., detect, obtain, acquire, or identify) a voice signal including the predetermined notification sound together with the user's speech through the microphone (550) (e.g., the voice signal includes a first component corresponding to the user's speech and a second component corresponding to the notification sound). Here, the predetermined notification sound may be guide information for inducing the user's additional speech in response to the passage of a predetermined time (or waiting time) when no voice signal is input within a predetermined time (or waiting time) after the call word. Since the call word recognition may include speaker recognition, a specific speaker may be identified through speaker recognition. According to one embodiment, the waiting time may be set differently for each different speaker (i.e., the waiting time may be set for each of a plurality of users / speakers, and the waiting times of two or more of these users may be different from each other). Accordingly, the processor (520) can determine a waiting time corresponding to the identified specific speaker, and can control to output a notification sound after the determined waiting time. The predetermined notification sound may include a sound effect and / or the user's voice. A sound effect such as 'ding dong' and / or a short word such as 'yes' can be used as the notification sound. In addition, the notification sound may be replaced with a general or arbitrary speaker's voice, i.e., a different tone, rather than the user's voice. In addition, although a short word such as 'yes' has been described as an example of the notification sound, the content (or text) of the notification sound may be variously defined in advance, and thus may not be limited thereto. For example, the voice recognition system may support multiple wake words, and may output a notification sound with a different tone and content depending on each wake word.
[0105] Likewise, the speech recognition rate may be reduced by factors other than the user's speech, such as noise as well as notification sounds (e.g., the speech signal includes a third component corresponding to ambient / environmental noise).
[0106] Accordingly, the processor (520) may perform preprocessing on the voice signal received through the microphone (550) to increase the accuracy of voice recognition. Preprocessing is a series of signal processing performed before voice recognition, and may include an operation of removing (or filtering) signals other than the user's speech from the voice signal. According to one embodiment, the processor (520) may perform preprocessing to remove a predetermined notification sound corresponding to a signal other than the user's speech from the received voice signal. Herein and hereinafter, 'removing' the predetermined notification sound from the received voice signal may be replaced with various terms meaning that only the voice signal excluding the notification sound remains, such as filtering (e.g., spectral subtraction, Wiener filter, MMSE, Kalman filter), deletion, separation, distinction, or extraction.
[0107] According to one embodiment, the processor (520) removes a signal corresponding to a predetermined notification sound (e.g., a second component) from the received voice signal by using a reference signal pre-stored in the memory (530) in an operation of removing (or filtering) a signal other than the user's speech in the preprocessing operation. Here, the predetermined notification sound is intended to induce additional speech from the user when a voice signal according to the user's speech is not input within a certain period of time after the call sound, and may be a voice recorded (or stored) by the user or a predetermined special voice. In order to more clearly identify the notification sound included in the voice signal when the voice signal is input, the predetermined notification sound may be obtained in advance, and the notification sound or related information may be obtained in the storage (or registration) step. For example, in the storage (or registration) step, a feature or identification information (e.g., ID) for the notification sound is mapped to the notification sound and stored and managed. By storing the characteristics or identification information of the notification sound together when saving the notification sound, it is possible to identify the notification sound even if there are multiple notification sounds (or types). Here, the predetermined notification sound used to remove the notification sound from the voice signal may be referred to as a reference signal, notification sound information, or VLS data. In the following description, the reference signal should be understood to represent the predetermined notification sound used to remove (or filter) the notification sound played from the voice signal in the electronic device (101).
[0108] The operations performed by the processor (520) to remove a notification sound from a voice signal will be described below. According to one embodiment, the processor (520) obtains a voice signal from which a notification sound has been removed through a preprocessing operation.
[0109] According to one embodiment, the call word recognition operation may be performed by an electronic device (101) (e.g., the electronic device (101) of FIG. 1, the user terminal (290) of FIG. 2), and the following description will exemplify a case where the notification sound removal operation is performed by the electronic device (101), but it is to be understood that the notification sound removal operation may also be performed by a server (e.g., the intelligent server (200) and / or service server (300) of FIG. 2).
[0110] The processor (520) performs speech recognition on the acquired speech signal. According to one embodiment, speech recognition processing for a user's speech may partially include automatic speech recognition (ASR) and / or natural language understanding (NLU) processing. For example, an ASR module (or ASR engine) may be used to convert the user's speech into text, and an NLU module (or NLU engine) may extract the meaning of the user's speech from the recognition result of the ASR module.
[0111] According to one embodiment, the voice recognition processing is processed in a voice recognition module (or voice recognition engine) stored in the memory (530) of the electronic device (101), or in a server (e.g., the intelligent server (200) and / or the service server (300) of FIG. 2). Accordingly, the processor (520) may transmit the voice spoken by the user to the server via the communication module (590) or transmit the voice recognition result to the server. For example, the processor (520) may transmit a voice signal with the notification sound removed to the server or transmit the voice recognition result to the server.
[0112] As described above, the operation of the processor (520) to increase the voice recognition rate can be broadly divided into an operation of storing (or registering) a reference signal or related information (or feature) (e.g., spectrogram, MFCC (mel-frequency cepstral coefficients)) used to remove the notification sound, and a filtering operation of removing the notification sound using the stored reference signal.
[0113] According to one embodiment, the processor (520) performs an operation of storing (or registering) the reference signal and a filtering operation for removing the notification sound, but the operation may be performed by a plurality of processors. In addition, the processor (520) may be connected to modules for processing voice signals and modules for voice recognition, as illustrated in FIG. 5B. FIG. 5B is a detailed block diagram of an electronic device for processing voice signals according to one embodiment.
[0114] Referring to FIG. 5b, the call word recognition unit (or recognition module) (521) can perform call word recognition, which is a command that instructs the start of voice recognition from a voice signal received through a microphone (550). If the call word is recognized, the call word recognition unit (521) can induce the voice recognition unit (524) to perform voice recognition on a voice signal input after the call word. The voice signal transmitted to the voice recognition unit (524) may be preprocessed and transmitted for more accurate voice recognition. Examples of the preprocessing operation may include an adaptive gain control (AGC) and / or noise suppression (NS) operation. Meanwhile, a command according to the call word recognized by the call word recognition unit (521) may be implemented in a processor (e.g., DSP, audio codec) different from the voice recognition unit (524), or may be implemented in a processor (e.g., AP).
[0115] The notification sound filtering unit (or filtering module) (523) can remove a notification sound from a voice signal input after a call word by using the stored notification sound (e.g., VLS) data (531). Here, removing (e.g., filtering) a notification sound from a voice signal input after a call word should be understood as removing a notification sound signal corresponding to a preset notification sound output through a speaker (555) from the voice signal when the preset notification sound is received together with a voice signal through a microphone (550) (e.g., when the voice signal includes a current notification sound output through the speaker (555) and a user's voice, at least some overlap exists between the two). The notification sound (e.g., VLS) data (531) may be a reference signal used for notification sound filtering. For example, the notification sound (e.g., VLS) data (531) is a waveform signal of a reference signal or a feature vector (e.g., spectrogram, MFCC (mel-frequency cepstral coefficients)) representing the waveform in the form of a frequency. According to one embodiment, the notification sound (e.g., VLS) data (531) is obtained in the same manner as the feature vector used in the voice recognition unit (524). For example, the reference signal is a signal recorded when the notification sound is output through the speaker (555) or a signal stored by further processing the recorded signal.
[0116] For example, if a wake word is detected in the first voice signal of the first section, the output of the notification sound is determined based on whether the user makes additional speech within a certain period of time (or a designated waiting time) (e.g., about 1 to 5 seconds). If no voice signal is detected in the voice signal input for a certain period of time, this may correspond, for example, to a case where there is no additional speech from the user within a certain period of time. Accordingly, if there is no additional speech, a notification sound is output to induce the user to make additional speech. Accordingly, the second voice signal of the second section following the first section may include a section in which the user does not make speech (e.g., a silent section) and a notification sound to induce the user to make additional speech. According to one embodiment, the first section and the second section may not overlap and may be spaced apart by a certain time interval. The notification sound filtering unit (523) may perform an operation of removing a notification sound from the second voice signal of the second section, and this will be described in detail with reference to FIG. 6.
[0117] FIG. 6 is an example diagram showing input signals in cases where the notification sound and the user voice do not overlap and in cases where they overlap, according to one embodiment.
[0118] Fig. 6(a) illustrates the waveform of a voice signal input through a microphone (e.g., microphone (550) of Figs. 5a and 5b), and exemplifies a case where the portion (600a) corresponding to the notification sound after the call word and the portion (610a) of the user's voice do not overlap. Fig. 6(b) illustrates a portion (600b) where the notification sound after the call word and the user's voice overlap. In the waveform of the voice signal in Fig. 6, the x-axis may represent time, and the y-axis may represent frequency.
[0119] After a certain period of time has passed since the wake-up call, a predetermined notification sound is output through a speaker (e.g., speaker (555) of FIGS. 5A and 5B) and input together with a voice signal according to the user's speech through a microphone (550). However, although the electronic device (101) can know that the notification sound has been output and what kind of voice (e.g., TTS voice type) the notification sound was output with, it cannot know exactly how much delay the notification sound output through the speaker (555) is before it is received through the microphone (550). Therefore, the exact location (e.g., a specific point in time or period) of the notification sound within the received (or recorded) voice signal may not be known.
[0120] If a voice signal according to the user's additional speech is input after the notification sound is output as in Fig. 6(a), the electronic device (101) can output a voice recognition result by removing the pattern corresponding to the wake word and the notification sound. However, if a voice signal according to the user's additional speech is input with at least a portion overlapping with the notification sound as illustrated in Fig. 6(b), the voice signal itself may be distorted due to the overlapping of the notification sound and the user's speech, which may cause a voice recognition error.
[0121] In addition, even if a high-quality original notification sound (e.g., VLS) (700a) is output (or played) through the speaker (555) as illustrated in FIG. 7, the notification sound (700b) received (or recorded) through the microphone (550) may be distorted for various reasons. Here, FIG. 7 is an exemplary diagram showing signals corresponding to the original notification sound and the recorded notification sound according to one embodiment. For example, the signal characteristics of the notification sound (700b) may vary depending on the reproducibility of the speaker (555), the sound reception frequency characteristics of the microphone (550), or the sampling rate set during recording. If voice recognition is performed on a voice signal including a notification sound, the voice recognition rate may be relatively lowered.
[0122] In addition, when a voice signal is input according to the user's additional speech after the notification sound is output as in Fig. 6(a), the electronic device (101) can output a voice recognition result by removing a pattern corresponding to the notification sound, but due to the signal difference between the original notification sound (700a) and the recorded notification sound (700b), it may be difficult to accurately remove the notification sound included in the voice signal even if the location of the notification sound is estimated.
[0123] Accordingly, in one embodiment, the electronic device (101) can use a reference signal to identify the location of the notification sound in the voice signal following the call word and to accurately remove the notification sound included in the voice signal.
[0124] Specifically, the notification sound filtering unit (523) can use the reference signal from the reference signal generating unit (522) to find the location of the notification sound from the second voice signal of the second section and to remove the notification sound at the corresponding location. The reference signal generated from the reference signal generating unit (522) is a signal used for removing (or filtering) the notification sound, and may be a notification sound that is generated in advance using TTS selected by the user and stored (or registered) in the notification sound (e.g., VLS) data (or data module, database) (531).
[0125] The reference signal generation unit (522) may convert a notification text (e.g., a predetermined notification sound is a specific voice acquired in advance, or a voice synthesized from a pre-specified text by the user through TTS) into a notification sound based on a voice synthesis model, and output (or play) the converted notification sound through a speaker (555). In one embodiment, the voice synthesis model may be selected by user input from among a plurality of voice synthesis models, or may be newly generated by user input. The reference signal generation unit (522) may obtain a reference signal corresponding to the output notification sound input through a microphone (550), and store the obtained reference signal in a memory (530) (e.g., notification sound (e.g., VLS) data (531)).
[0126] The processor (520) can control operations and signals between the aforementioned components (or modules). According to one embodiment, the processor (520) controls each component (521, 522, 523, 524, 531) that performs an operation to remove the notification sound from a voice signal including the notification sound, or performs at least a part of the operation of each component (521, 522, 523, 524, 531). The processor (520) can determine whether to remove (or filter) the notification sound after recognizing a call word, and control voice recognition to be performed by providing a voice signal with the notification sound removed (or filtered) to the voice recognition unit (ASR) (524).
[0127] The speech recognition unit (ASR) (524) analyzes the input speech signal with the notification sound removed to generate a feature vector, and may generate text corresponding to the speech signal or perform a specified function based on the text through various statistical algorithms (e.g., hidden Markov model (HMM), artificial neural net). Here, some operations of the speech recognition unit (ASR) (524) may be performed in an external server (e.g., the intelligent server (200) and / or the service server (300) of FIG. 2). In one embodiment, the notification sound removal is processed on the feature vector (or in the domain) and transmitted to the speech recognition unit (ASR) (524) without an operation of generating a redundant feature vector.
[0128] Meanwhile, an operation for removing (or filtering) a notification sound according to one embodiment will be described later with reference to FIGS. 10 to 12, and an operation for generating a reference signal used for removing a notification sound will be described in detail later with reference to FIGS. 15 and 16.
[0129] Meanwhile, at least some of the above-described operations of the electronic device (101) may be performed by the server of FIG. 2 (e.g., the intelligent server (200) and / or service server (300) of FIG. 2).
[0130] According to one embodiment, the remaining operations except for outputting a notification sound and receiving a voice signal in the electronic device (101) are performed in the server. For example, the server identifies the location of the notification sound within the voice signal based on the stored reference signal, removes the notification sound from the voice signal based on the identified location, and then performs voice recognition. Accordingly, at least some of the remaining operations except for receiving a voice signal in the electronic device (101) may be implemented to be performed in the electronic device (101) or the server, respectively.
[0131] According to one embodiment, the electronic device (101) includes a microphone (295, 550), at least one processor (120, 292, 520), and a memory (130, 299, 530). According to one embodiment, the memory may store instructions, which, when executed by the at least one processor, may be configured to cause the electronic device to receive a first voice signal through the microphone during a first period, detect a wake word from the first voice signal, output a preset wake word in response to the wake word detection, detect a wake sound signal corresponding to the output wake sound from a second voice signal received through the microphone during a second period, and remove the wake sound signal from the second voice signal of the second period based on a position of the detected wake sound signal. According to one embodiment, the instructions may be configured to cause the electronic device to perform voice recognition on the second voice signal of the second period from which the wake sound signal has been removed.
[0132] According to one embodiment, the instructions may be configured to cause the electronic device to determine a notification sound search range and identify a location of the notification sound signal by comparing at least a portion of the second voice signal of the second section with a pre-stored reference signal in the notification sound search range.
[0133] According to one embodiment, the electronic device (101) further includes a speaker (294, 555). The instructions may be configured to cause the electronic device to convert a notification text into a notification sound based on one of a plurality of voice synthesis models, output the converted notification sound through the speaker, obtain a reference signal corresponding to the output notification sound input through the microphone, and store the obtained reference signal in the memory.
[0134] According to one embodiment, the instructions may be configured to cause the electronic device to remove the notification sound signal from the second voice signal using the reference signal, the reference signal corresponding to a specified section length of the reference signal from the position of the notification sound signal.
[0135] According to one embodiment, the instructions may be configured to cause the electronic device to obtain a first feature vector in a frequency domain corresponding to the reference signal, compare the first feature vector with a second feature vector in a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range, and identify a location of the notification sound signal based on a result of the comparison.
[0136] According to one embodiment, the instructions may be configured to cause the electronic device to determine a portion of the entire frequency range corresponding to the reference signal that includes a user's voice characteristic, and to identify a location of the notification sound signal using a first feature vector of the portion of the frequency range.
[0137] According to one embodiment, the instructions may be configured to cause the electronic device to compare the first feature vector and the second feature vector and identify a location where a mean squared error (MSE) is less than or equal to a threshold value as the location of the notification sound signal.
[0138] According to one embodiment, the instructions may be configured to cause the electronic device to identify a similarity between the first feature vector and a second feature vector in a region corresponding to the partial region among the entire frequency domain corresponding to at least a portion of the second voice signal, and to identify a location having a similarity greater than a threshold value as a location of the notification sound signal.
[0139] According to one embodiment, the instructions may be configured to cause the electronic device to normalize the first feature vector and the second feature vector, compare the normalized first and second feature vectors to identify a location of the notification sound signal, and remove the notification sound signal using a signal obtained by normalizing the reference signal based on the location of the notification sound signal.
[0140] According to one embodiment, the instructions may be configured to cause the electronic device to determine the notification sound search range based on the timing of outputting the notification sound.
[0141] According to one embodiment, the instructions may be configured to cause the electronic device to perform voice recognition on the second voice signal of the second section from which the notification sound signal has been removed to generate text data, and to perform a specified action based on the text data.
[0142] FIG. 8 is a flowchart illustrating an operation for processing a voice signal in an electronic device according to an embodiment.
[0143] Referring to FIG. 8, the operating method may include operations 805 to 830. Each operation of the operating method of FIG. 8 may be performed by an electronic device (e.g., the electronic device (101) of FIGS. 1 to 5A), or at least one processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (520) of FIGS. 5A and 5B). In various embodiments, at least one of operations 805 to 830 is omitted, the order of some operations is changed, another operation is added, and at least two operations are performed in parallel. In addition, in the following description, each operation of the operating method of FIG. 8 is described as being performed in the electronic device (101), but operations 805 to 815 may be performed in a client module (e.g., the client module (298) of FIG. 2), and operations 820 to 830 may be performed in an automatic speech recognition module (ASR module) (e.g., the automatic speech recognition module (221) of FIG. 2). For example, when a 'wake word' (e.g., 'Hi Bixby') and a 'command' (e.g., '~do it') are spoken without a pause, the entire input voice signal may be transmitted to and processed by the automatic speech recognition module so that the voice signal can be processed without separation of the first section and the second section.
[0144] In operation 805, the electronic device (101) receives a first voice signal through the microphone (295, 550) during a first period. In operation 810, the electronic device (101) detects a wake-up word from the first voice signal. According to one embodiment, the wake-up word may be a wake-up command indicating when to start a voice recognition operation. For example, the wake-up word includes 'Hi Bixby.'
[0145] In operation 815, the electronic device (101) outputs a preset notification sound in response to the detection of the call word. The electronic device (101) may output the preset notification sound after a predetermined waiting period after the detection of the call word. For example, the electronic device (101) outputs the notification sound after a predetermined waiting period regardless of whether there is additional utterance. According to one embodiment, the electronic device (101) may identify whether there is additional utterance from the user within a predetermined period after the detection of the call word, and output the notification sound based on the identification result. For example, voice recognition is performed using a second voice signal received for a predetermined period of time (or a designated waiting period) (e.g., about 2.5 seconds) after the first section. For example, the predetermined period of time may be set within a range of about 1 to 5 seconds.
[0146] As described above, in response to the passage of a certain period of time or as a result of performing voice recognition, if no additional speech from the user is detected within a certain period of time, the electronic device (101) may output (or play) a preset notification sound through the speaker (555). The preset notification sound is intended to induce or prompt the user to make additional speech, and the user can recognize through the output notification sound that the voice recognition function has been called and that a voice command must be input by speech.
[0147] In one embodiment, the waiting time for the notification sound to be output after the detection of a wake word varies depending on user information. The user information may be a profile of the user using the electronic device (101), such as information about the user, such as age and gender. For example, the waiting time may be set longer for older users. The user information may be determined based on the speaker recognition results performed upon the detection of the wake word.
[0148] In one embodiment, the notification sound output varies depending on user information (e.g., age, gender) and the input situation. For example, if the user is an elderly person, the electronic device (101) outputs a notification sound saying "Yes, I'm listening." Otherwise, it outputs a notification sound saying "Yes." For example, the number of times the notification sound is output may be two or more, and the input situation may be the order in which the notification sound is output. The first notification sound may be "Yes," and the second notification sound may be "I'm listening." In this way, the reference signal used for notification sound removal may be changed depending on the notification sounds output differently.
[0149] In operation 820, the electronic device (101) detects a notification sound signal corresponding to the output notification sound from a second voice signal received through the microphone during the second period. According to one embodiment, the electronic device (101) performs a preprocessing operation on the second voice signal during the second period following the first period. For example, since the electronic device (101) outputs (or plays) the preset notification sound through the speaker (555) according to the passage of the predetermined time, the electronic device (101) receives a second voice signal including a notification sound signal corresponding to the output notification sound through the microphone (550). According to one embodiment, when a voice signal by a user's speech is input, the electronic device (101) processes the voice signal by dividing it into frames of preset time units (e.g., each frame corresponds to a preset number of one or more time units). Therefore, the first period and the second period may be predetermined times (e.g., set time periods or durations) in consideration of the frame unit. For example, the first section and the second section are sections each containing the same number of frames.
[0150] In operation 825, the electronic device (101) removes the notification sound signal from the second voice signal of the second section based on the location of the detected notification sound signal.
[0151] In operation 830, the electronic device (101) performs voice recognition on the second voice signal of the second section from which the notification sound signal has been removed. Here, the second voice signal of the second section from which the notification sound signal has been removed may be a signal obtained by removing (or separating, filtering, etc.) a signal corresponding to an actual notification sound using a reference signal from the position of the notification sound signal. Accordingly, the second voice signal of the second section, which is distinguished from the notification sound, may be a signal from which the notification sound signal has been removed, which is a different signal from the second voice signal received through the microphone during the second section (or a third signal, the voice signal is a second signal, and the signal including the wake word is a first signal).
[0152] Here, since the notification sound is output in response to the passage of a certain period of time (or standby time), the position of the notification sound signal corresponding to the output notification sound within the received second voice signal may be the same as the certain period of time (or standby time), but may correspond to a time after the certain period of time due to the signal difference between the notification sound output through the speaker (555) and the notification sound received through the microphone (550) (i.e., there will be a delay between the output of the notification sound through the speaker (555) and the reception / detection of the output notification sound by the microphone (550). Accordingly, the electronic device (101) can estimate the position of the notification sound signal within the received second voice signal based on the certain period of time (or standby time), and can identify the actual position of the notification sound signal within the received second voice signal based on the certain period of time (or standby time).
[0153] According to one embodiment, the electronic device (101) finds and removes the position of the notification sound signal using a reference signal. The electronic device (101) determines a notification sound search range based on the predetermined time (or standby time), and compares at least a part of the second voice signal of the second section with a pre-stored reference signal in the notification sound search range, thereby identifying the position (or location) of the notification sound signal (i.e., the second voice signal). In one embodiment, the position of the notification sound signal, which serves as a reference (or starting position) for performing voice recognition on the second voice signal of the second section, is regarded (or estimated) to be the same as the predetermined time (or standby time). Therefore, the electronic device (101) can determine a notification sound search range starting from a predetermined time (e.g., about 2.5 seconds) in the second voice signal of the second section on the time axis to identify (or find) the actual position of the notification sound signal. For example, since the electronic device (101) can know whether a notification sound is output and the time at which the notification sound is output, the electronic device can estimate the position of the notification sound signal in the second voice signal of the second section based on a certain time (or standby time) at which the notification sound is output, and identify the actual position of the notification sound signal in the second voice signal through notification sound search.
[0154] Since the electronic device (101) can know in advance the length of a predetermined notification sound, such as a sound effect or a short word, i.e., the designated section length of the reference signal, if the actual position of the notification sound signal, i.e., the position where the notification sound signal starts, in the second voice signal of the second section, the electronic device (101) can remove a signal corresponding to the section length of the notification sound from the starting position of the notification sound signal (i.e., a signal having a predetermined length or section length starting from the starting position is removed from the second voice signal). Accordingly, the electronic device (101) can remove a signal corresponding to the designated section length of the reference signal from the position of the identified notification sound signal in the second voice signal using the reference signal. Here, when removing the notification sound signal from the second voice signal, rather than removing all signals corresponding to the section length of the reference signal from the second voice signal, only the second voice signal from which only signals having a form substantially identical to the reference signal are removed remains (i.e., a signal substantially corresponding to the second voice signal excluding the reference signal remains).
[0155] According to one embodiment (e.g., in the processor in operation 820 or operation 825), the electronic device (101) obtains a first feature vector of a frequency domain (i.e., a frequency band, a frequency range, or a frequency width) corresponding to the reference signal. The electronic device (101) may compare the first feature vector with a second feature vector of a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range. The electronic device (101) may identify a location of the notification sound signal based on the comparison result. According to one embodiment, the location of the notification sound signal corresponds to a location where a mean squared error (MSE) is less than or equal to a threshold value by comparing the first feature vector and the second feature vector. In addition, the location of the notification sound signal may correspond to a location where a similarity between two feature vectors is greater than or equal to a threshold value by determining a similarity between the two feature vectors.
[0156] According to one embodiment, the electronic device (101) converts a wave file representing a voice signal in a wave form into a spectrogram representing the voice signal in a frequency form. For example, the first feature vector in the frequency domain corresponding to the reference signal is an MFCC (mel-frequency cepstral coefficients) value representing frequency characteristics. As another example, the first feature vector is a feature vector used in speech recognition (ASR) performed thereafter.
[0157] According to one embodiment, the operation of obtaining a first feature vector in a frequency domain corresponding to the reference signal includes the operation of determining a portion of the entire frequency domain corresponding to the reference signal that includes a user's voice characteristic, and the operation of obtaining a first feature vector in the portion of the frequency domain.
[0158] According to one embodiment, the electronic device (101) performs voice recognition on the second voice signal of the second section from which the notification sound signal has been removed to generate text data. The electronic device (101) can perform a specified action based on the text data.
[0159] Meanwhile, in the above-described case, the operation of removing the notification sound (or notification sound signal) has been examined, but the registration step of the reference signal used to remove the notification sound may be as follows. In the registration step, the electronic device (101) may convert the notification text into the notification sound based on any one of the plurality of voice synthesis models. The electronic device (101) may output the converted notification sound through the speaker (555) of the electronic device (101). The electronic device (101) may obtain a reference signal corresponding to the output notification sound input through the microphone (550), and may store (or register) the obtained reference signal in the memory (530).
[0160] Figure 9 is a detailed operation flowchart in an electronic device according to one embodiment.
[0161] Referring to FIG. 9, the operating method may include operations 905 to 930. Each operation of the operating method of FIG. 9 may be performed by an electronic device (e.g., the electronic device (101) of FIGS. 1 to 5A), at least one processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (520) of FIGS. 5A and 5B). In various embodiments, at least one of operations 905 to 930 may be omitted, the order of some operations may be changed, another operation may be added, and at least two operations may be performed in parallel. To facilitate understanding of the description in FIG. 9, the description will be made with reference to FIGS. 10 to 13.
[0162] In addition, in the following description, each operation of the operating method of FIG. 9 is described as being performed in the electronic device (101), but operation 905 may be performed in a client module (e.g., client module (298) of FIG. 2) (or DSP wake-up module), operations 910 and 915 may be performed in the client module, or operations 920 to 930 may be performed in an automatic speech recognition module (ASR module) (e.g., automatic speech recognition module (221) of FIG. 2).
[0163] In operation 905, the electronic device (101) detects a wake word from the first voice signal of the first section. If the wake word is detected, in operation 910, the electronic device (101) can identify whether output of a preset notification sound is required to induce a voice input. For example, if no additional user speech is detected within a certain period of time after detection of the wake word, it is identified that output of a notification sound is required to induce additional user speech.
[0164] Accordingly, when output of a preset notification sound is required, the electronic device (101) outputs a preset notification sound in operation 915. For example, the electronic device (101) outputs a preset notification sound through the speaker (555) in response to identifying that a user who has uttered a wake word does not make additional utterances within a certain period of time. At this time, the notification sound output through the speaker (555) can be received together with the voice according to the user's utterance through the microphone (550).
[0165] In operation 920, the electronic device (101) removes a notification sound signal corresponding to the output notification sound from the second voice signal of the second section. Here, the notification sound signal is a signal corresponding to the notification sound output through the speaker (555) and can be received together with the voice signal through the microphone (550). According to one embodiment, the electronic device (101) identifies the location of the notification sound signal within the second voice signal to remove the notification sound signal from the second voice signal of the second section.
[0166] Here, in order to specifically examine the operation of removing (or filtering) a notification sound (or notification sound signal), reference will be made to FIGS. 10 to 12. FIG. 10 is an exemplary diagram showing a spectrogram of a reference signal for removing a notification sound according to an embodiment, FIG. 11 is an exemplary diagram showing a spectrogram of a reference signal for searching for a notification sound according to an embodiment, and FIG. 12 is an exemplary diagram explaining a method for removing a notification sound from an input voice signal according to an embodiment.
[0167] Fig. 10 illustrates a spectrogram for the entire frequency band of a reference signal used for removing a notification sound from a voice signal. Here, the reference signal may be, for example, a preset notification sound received (or recorded) through a microphone by playing a TTS sound source selected by the user. This reference signal may be downloaded from a server and may also be stored in a separate memory space within the electronic device (101). In this case, the reference signal may be a signal for a notification sound section of a sound effect such as "ding dong" and / or a short word such as "yes", excluding a silent section for a certain period of time after a wake word.
[0168] In order to reduce the amount of computation by using the features being computed in the decoder of the speech recognition unit (ASR) (e.g., the speech recognition unit (524) of FIG. 5b) in the computation of the processor (e.g., the processor (520) of FIGS. 5a and 5b) (or the notification sound filtering unit (523) of FIG. 5b), a reference signal represented in the form of a wave (e.g., used or considered instead) can be represented as a spectrogram as in FIG. 10. For example, referring to FIG. 12, when a voice signal (y) is input in a waveform as shown in 1200a, if both the reference signal and the voice signal are represented as spectrograms, they are as shown in 1200b.
[0169] Meanwhile, FIG. 10 illustrates the magnitude of the entire frequency band of the reference signal, which can be used to calculate the magnitude (mean) value for each band. For example, the processor (520) can extract the entire spectrogram standardized to a value between [0, 1]. In this way, the processor (520) can normalize the spectrogram of the reference signal to have a constant size. The processor (520) can determine a portion of the entire frequency range corresponding to the reference signal that includes the user's voice characteristics. To this end, a certain number of bands (e.g., N) in the order of the largest standardized values among the entire frequency bands can be determined as bands of interest, and an index for each band can be extracted.
[0170] In Fig. 11, the spectrogram of the reference signal for searching for a notification sound is composed of some bands among the entire frequency bands of Fig. 10 in which frequency characteristics are significantly displayed, and these some areas may represent the user's voice characteristics (for example, the processor (520) identifies a band having a frequency characteristic that is greater than or equal to a predetermined value). Since the voice frequency characteristics of a speaker have a harmonic band, the band representing the user's voice characteristics can be used to search for the location of the notification sound signal within the second voice signal of the second section.
[0171] Accordingly, the processor (520) can use the feature vector of the partial region (or partial band) to search for the position of the notification sound signal within the voice signal. At this time, the processor (520) can search for the position of the notification sound signal using not only the first feature vector of the reference signal but also the second feature vector of the harmonic band corresponding to a partial region of the entire frequency domain corresponding to at least a portion of the second voice signal. That is, the processor (520) can search for the position of the notification sound signal in the second voice signal using the first feature vector of the reference signal and the region corresponding to the determined partial region of the entire frequency domain, wherein the partial region itself corresponds to at least a portion of the second voice signal as described above. If, as a result of the comparison between the first feature vector and the second feature vector, referring to 1200c of FIG. 12, a position where the mean squared error (MSE) is less than or equal to a threshold value, for example, a position where the MSE shows a minimum value, as indicated by the drawing number 1210c, can be identified as the position of the notification sound signal. At this time, when the location (1210c) of the notification sound signal is identified, a signal corresponding to the specified section length (1220c) of the reference signal can be removed from the location (1210c) of the identified notification sound signal using the reference signal of FIG. 11.
[0172] Additionally, the processor (520) may identify the similarity between the first feature vector and the second feature vector based on the comparison results, and may identify a location having a similarity greater than a threshold value as the location of the notification sound signal. In the above description, the case of identifying the location of the notification sound signal within the voice signal using MSE or similarity was described as an example, but the method of determining the location is not limited thereto, and the location may be determined in various ways.
[0173] According to one embodiment, in order to search for the location of a notification sound signal, a first feature vector of a portion of a frequency domain corresponding to the reference signal of FIG. 11, which includes a user's voice characteristic, and a second feature vector of a frequency domain corresponding to at least a portion of the second voice signal may be used to perform windowing using a notification sound search range. That is, the first feature vector and the second feature vector may be used to perform windowing using a notification sound search range. Here, the notification sound search range may be determined based on the output time of the notification sound. For example, when the output time of the notification sound is n0, a search is performed in the range [n0-Δ1, n0+Δ2].
[0174] At this time, in order to compare the two feature vectors, the first feature vector and the second feature vector may be normalized, and the normalized first and second feature vectors may be compared to identify the location of the notification sound signal. The processor (520) may remove the notification sound signal from the second voice signal using a signal obtained by normalizing the reference signal based on the location of the notification sound signal. For example, as illustrated in 1200d of FIG. 12, a signal in a section corresponding to 1220d is removed, thereby obtaining a voice signal (y_new) from which the notification sound signal has been removed, as illustrated in 1200e of FIG. 12.
[0175] In operation 925, the electronic device (101) performs voice recognition on the second voice signal of the second section from which the notification sound signal has been removed.
[0176] On the other hand, if the output of the preset notification sound is not required, the electronic device (101) performs voice recognition on the second voice signal of the second section in operation 930. For example, if the output of the notification sound is not required, since the notification sound is not output, the second voice signal of the second section does not include the notification sound, and therefore, the preprocessing operation for removing the notification sound is omitted, and then voice recognition can be performed on the second voice signal of the second section in the postprocessing operation. It will be appreciated that in various embodiments, operation 930 is omitted in order to focus more on operations 915 and 920. In addition, in some embodiments, operation 925 may be omitted in order to focus more on operations 915 and 920, where explicit inclusion of a voice recognition operation is not required to explain operation 920.
[0177] FIG. 13 is an exemplary diagram showing an input signal before / after removing a notification sound when the notification sound and the user voice do not overlap according to one embodiment, and an input signal before / after removing a notification sound when the notification sound and the user voice overlap.
[0178] Figures 13(a) and 13(b) illustrate a case where the notification sound (or notification sound signal) is removed while the portion (1310) corresponding to the notification sound after the call word and the user voice portion do not overlap as in Figure 13(a). The voice signal from which the portion (1320) corresponding to the notification sound is removed can be obtained as illustrated in Figure 13(b).
[0179] On the other hand, FIGS. 13(c) and 13(d) illustrate a case where the notification sound is removed in a state where the portion (1330) corresponding to the notification sound and the user voice overlap as in FIG. 13(c) after the call word. A voice signal from which the portion (1340) corresponding to the notification sound is removed can be obtained, for example, as illustrated in FIG. 13(d) according to the example disclosed herein.
[0180] FIG. 14 is a diagram for explaining a preprocessing operation for removing a notification sound before voice recognition according to one embodiment.
[0181] Referring to FIG. 14, a notification sound (e.g., VLS) filter (1430) may operate continuously or selectively as needed. Here, the notification sound filtering operation may be performed in a processor (e.g., processor (520) of FIGS. 5A and 5B) or a notification sound filtering unit (e.g., notification sound filtering unit (523) of FIG. 5B).
[0182] According to one embodiment, when a user utters (e.g., speaks), a wake-up word recognition is performed on a voice signal in a first section, and in operation 1410, it is determined whether to output a notification sound in response to the wake-up word recognition. After the wake-up word is recognized, a voice command input according to the user's additional utterance (e.g., the processor (520)) waits, and this can be defined as a second section. If it is determined that no voice input according to the additional utterance has been made for a certain period of time (e.g., about 2.5 seconds), a notification sound (e.g., VLS) is output. At this time, since the notification sound filtered according to the voice style may also vary, the client (or processor (520)) may provide notification sound-related information (e.g., VLS waveform, played TTS voice ID (identification), or VLS characteristic information).
[0183] At this time, since a voice command may be input from a user through a microphone (e.g., microphone (550) of FIGS. 5 a and 5b) even while monitoring whether a notification sound is output, notification sound removal or voice recognition (ASR) may be performed on a voice signal input through the microphone (550). If a notification sound is output, a flag may be transmitted from the client (or processor (520)) to the notification sound (e.g., VLS) filter (1430) to indicate that the notification sound has been played.
[0184] In one embodiment, the notification sound varies depending on the wake word. For example, a voice assistant system supports multiple wake words and outputs a wake sound with a different tone and content (or text) depending on each wake word. In this case, when recognizing the input wake word, the wake word ID can be determined, and the type of wake sound to be output can be determined based on the wake word ID.
[0185] If the notification sound is not output, a flag is passed from the client (or processor (520)) to the notification sound (e.g., VLS) filter (1430) to inform that the notification sound was not played, so that the notification sound (e.g., VLS) filter (1430) operation can be omitted.
[0186] FIG. 15 is a flowchart illustrating an operation of an electronic device for generating and storing a reference signal according to an embodiment.
[0187] Referring to FIG. 15, the operating method may include operations 1505 to 1530. Each operation of the operating method of FIG. 15 may be performed by an electronic device (e.g., the electronic device (101) of FIGS. 1 to 5A), at least one processor of the electronic device (e.g., the processor (120) of FIG. 1, the processor (520) of FIGS. 5A and 5B). In various embodiments, at least one of operations 1505 to 1530 may be omitted, the order of some operations may be changed, another operation may be added, and at least two operations may be performed in parallel. To help understand the description of FIG. 15, reference will be made to FIG. 16. Here, FIG. 16 is an example screen diagram providing guide information for storing (or registering) a reference signal according to one embodiment.
[0188] In operation 1505, the electronic device (101) performs an operation of selecting a voice synthesis model. In operation 1510, the electronic device (101) converts a notification text into a notification sound. In operation 1515, the electronic device (101) activates a microphone (550) and outputs a notification sound through a speaker (555) in operation 1520. In operation 1525, the electronic device (101) acquires a reference signal corresponding to the notification sound through the activated microphone (550). In operation 1530, the electronic device (101) stores (or registers) the acquired reference signal in a memory. In one embodiment, the electronic device (101) converts the acquired reference signal into a feature vector and stores it in the memory.
[0189] A method for processing a voice signal according to an embodiment can be broadly divided into a step of generating and storing (or registering) a reference signal, and a step of processing an input voice signal to remove (or filter) a notification sound (or notification sound signal). For example, a reference signal may be generated in advance in the registration step, and the reference signal may be used to remove a notification sound signal in the filtering step.
[0190] In order to generate a reference signal, a user interface (UI) (1600) for selecting a voice (TTS: text to speech) of a voice assistant may be provided to the user as illustrated in FIG. 16. The UI (1600) may display an object for selecting a language setting and a voice style, for example, a voice (1610) of each voice assistant. According to one embodiment, when the voice assistant voice is selected, a sample voice is output, and simultaneously, a sound (or voice signal) corresponding to the sample voice is input through a microphone (550). Through the input sound, the electronic device (101) may generate and store a reference signal for a notification sound. According to one embodiment, in order to distinguish voice styles, the electronic device (101) stores and manages the reference signal using identification information (e.g., ID) corresponding to each voice assistant.
[0191] According to one embodiment, the user interface (1600) further displays an object (1620) for obtaining a reference signal. For example, when the object (1620) for voice learning is selected by the user, the electronic device (101) converts a notification sound phrase (or content) for inducing additional speech from the user into a waveform based on a voice synthesis model (e.g., a TTS model) corresponding to the previously selected voice assistant voice. Then, while the microphone (550) is activated, the notification sound phrase of the waveform may be output through the speaker (555), and the notification sound input as feedback through the microphone (550) may be stored as a reference signal. Here, the reference signal may be stored in the form of a waveform, and may also be characterized and stored using an algorithm substantially identical to a feature vector used in voice recognition. Therefore, the storage form of the reference signal may vary and may not be limited thereto.
[0192] FIG. 17 is a diagram illustrating a method for managing notification sounds for a specific speaker according to one embodiment.
[0193] Referring to FIG. 17, the personalized text to speech (PTTS) (1710) of the client may represent something specialized for the speaker's voice. The voice to be used for the voice assistant may be a personalized TTS (TTS) trained based on personally collected voice data. When adaptive training is performed using user voice data to generate a PTTS, a notification sound generated based on a PTTS model generated as a result of the training may be played through a speaker (e.g., speaker (555) of FIGS. 5A and 5B) in the electronic device (101). In response to the playback of the notification sound, a reference signal corresponding to the notification sound may be generated using a signal received (or recorded) through a microphone (e.g., microphone (550) of FIGS. 5A and 5B) of the electronic device (101). The generated reference signal (or recorded VLS) (1720) may be uploaded to the cloud (1715) with the user's consent. Even if the user does not agree, features extracted from a pre-created reference signal as a basic PTTS model used for user voice training among multiple PTTS models can be stored in PDSS (1740) instead.
[0194] Thereafter, the notification sound features (1730) extracted from the reference signal (or recorded VLS) (1720) generated within the cloud (1715) can be stored in the PDSS (1740) and managed as notification sound features (1750) for PTTS. Accordingly, it may be possible to provide PTTS specialized for the user's voice.
[0195] FIG. 18 is a diagram for explaining a preprocessing operation for removing a notification sound for a specific speaker before voice recognition according to one embodiment.
[0196] Referring to FIG. 18, when a PTTS model generated in response to a user's speech is used as a voice of a voice assistant (1810), the notification sound features (1850) for the PTTS stored in a server, e.g., PDSS, can be retrieved and used for feature extraction (1420) in the notification sound (e.g., VLS) filter (1430) operation performed before voice recognition.
[0197] Figure 19 is an example screen showing the results of voice recognition before and after removing the notification sound according to one embodiment.
[0198] FIG. 19(a) illustrates a case where, after the wake word “Hi Bixby” and a period of silence for a certain period of time, an additional user utterance (1910) is input to the electronic device (101), and a notification sound (1920) that induces the user’s additional utterance is output from the electronic device (101). However, before the notification sound is removed, the electronic device (101) illustrates a result in which voice recognition fails. That is, FIG. 19(a) illustrates a case in which the additional utterance “Today’s weather!” is received by the electronic device (101) (e.g., microphone (550)) that at least partially overlaps with the electronic device (e.g., microphone (550)) that receives the output notification sound; and if the notification sound is not removed (or filtered), voice recognition fails (e.g., because the second voice signal includes an output notification sound that partially overlaps with the additional utterance).
[0199] On the other hand, FIG. 19(b) illustrates a case where, after the wake word 'Hi Bixby' and a period of silence for a certain period of time, the user's additional utterance (1910) is input to the electronic device (101), and a notification sound (1920) inducing the user's additional utterance is output from the electronic device (101). According to one embodiment, even if the notification sound (1920) inducing the user's additional utterance and the user's additional utterance (1910) overlap at least partially, the notification sound signal included in the voice signal can be accurately removed using a reference signal generated in advance based on the position of the notification sound signal, thereby further improving voice recognition performance. Accordingly, the result of performing voice recognition on the voice signal from which the notification sound signal has been removed can be displayed as text on the screen (1930) of the electronic device (101), and a designated action based on the text can be performed.
[0200] According to one embodiment, it can have lower computational complexity compared to continuously applying an adaptive echo canceller (AEC) algorithm in a preprocessing step before speech recognition.
[0201] According to one embodiment, by adaptively removing a notification sound depending on whether a notification sound is generated in the preprocessing step, speech recognition can be efficiently performed using only a voice signal according to the user's speech.
[0202] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.
[0203] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.
[0204] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0205] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.
[0206] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.
[0207] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0208] According to one embodiment, in a non-transitory storage medium storing instructions, the instructions, when executed by at least one processor (120, 520) of an electronic device (101), are configured to cause the electronic device to perform at least one operation, wherein the at least one operation may include at least one of: receiving a first voice signal through a microphone (295, 550) during a first period; detecting a wake word from the first voice signal; outputting a preset wake word in response to detection of the wake word; detecting a wake sound signal corresponding to the output wake sound from a second voice signal received through the microphone during a second period; removing the wake sound signal from the second voice signal of the second period based on a position of the detected wake sound signal; or performing voice recognition on the second voice signal of the second period from which the wake sound signal has been removed.
[0209] According to one example, an electronic device is provided, including a microphone; at least one processor; and a memory storing instructions, wherein the instructions, when executed by the at least one processor, are configured to cause the electronic device to receive a first voice signal through the microphone during a first period, detect a wake word from the first voice signal, output a preset wake word in response to the wake word detection, detect a wake sound signal corresponding to the output wake sound from a second voice signal received through the microphone during a second period, remove the wake sound signal from the second voice signal of the second period based on a position of the detected wake sound signal, and perform voice recognition on the second voice signal of the second period from which the wake sound signal has been removed.
[0210] According to one example, the instructions may be configured to cause the electronic device to determine a notification sound search range and identify a location of the notification sound signal by comparing at least a portion of the second voice signal of the second section with a pre-stored reference signal in the notification sound search range.
[0211] According to one example, the electronic device further includes a speaker, and the instructions may be configured to cause the electronic device to convert a notification text into a notification sound based on one of a plurality of voice synthesis models, output the converted notification sound through the speaker, obtain a reference signal corresponding to the output notification sound input through the microphone, and store the obtained reference signal in the memory.
[0212] According to one example, the instructions may be configured to cause the electronic device to remove the notification sound signal from the second voice signal using the reference signal, a signal corresponding to a specified section length of the reference signal from a position of the notification sound signal.
[0213] According to one example, the instructions may be configured to cause the electronic device to obtain a first feature vector in a frequency domain corresponding to the reference signal, compare the first feature vector with a second feature vector in a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range, and identify a location of the notification sound signal based on a result of the comparison.
[0214] According to one example, the instructions may be configured to cause the electronic device to determine a portion of the entire frequency range corresponding to the reference signal that includes a user's voice characteristic, and to identify a location of the notification sound signal using a first feature vector of the portion of the frequency range.
[0215] According to one example, the instructions may be configured to cause the electronic device to compare the first feature vector and the second feature vector and identify a location where a mean squared error (MSE) is less than or equal to a threshold value as the location of the notification sound signal.
[0216] According to one example, the instructions may be configured to cause the electronic device to identify a similarity between the first feature vector and a second feature vector of a region corresponding to the partial region among the entire frequency domain corresponding to at least a portion of the second voice signal, and to identify a location having a similarity greater than a threshold value as a location of the notification sound signal.
[0217] According to one example, the instructions may be configured to cause the electronic device to normalize the first feature vector and the second feature vector, compare the normalized first and second feature vectors to identify a location of the notification sound signal, and remove the notification sound signal using a signal obtained by normalizing the reference signal based on the location of the notification sound signal.
[0218] In one example, the instructions may be configured to cause the electronic device to determine the notification sound search range based on the timing of the notification sound output.
[0219] According to one example, the instructions may be configured to cause the electronic device to perform voice recognition on the second voice signal of the second section from which the notification sound signal has been removed to generate text data, and to perform a specified action based on the text data.
[0220] According to one example, a method for processing a voice signal by an electronic device includes receiving a first voice signal through a microphone of the electronic device during a first period, detecting a wake word from the first voice signal, outputting a preset wake word in response to detection of the wake word, detecting a wake sound signal corresponding to the output wake sound from a second voice signal received through the microphone during a second period, removing the wake sound signal from a second voice signal of the second period based on a position of the detected wake sound signal, and performing voice recognition on the second voice signal of the second period from which the wake sound signal has been removed.
[0221] According to one example, the method further includes an operation of determining a notification sound search range and an operation of identifying a location of the notification sound signal by comparing at least a portion of the second voice signal of the second section with a pre-stored reference signal in the notification sound search range.
[0222] According to one example, the operation of removing the notification sound signal from the second voice signal of the second section includes the operation of removing the notification sound signal using the reference signal, using a signal corresponding to a specified section length of the reference signal from the position of the identified notification sound in the second voice signal.
[0223] According to one example, the method further includes an operation of obtaining a first feature vector in a frequency domain corresponding to the reference signal, an operation of comparing the first feature vector with a second feature vector in a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range, and an operation of identifying a location of the notification sound signal based on a result of the comparison.
[0224] According to one example, the operation of obtaining a first feature vector in a frequency domain corresponding to the reference signal includes the operation of determining a portion of the entire frequency domain corresponding to the reference signal that includes a user's voice characteristic, and the operation of obtaining a first feature vector in the portion of the frequency domain.
[0225] According to an example, the location of the notification sound signal corresponds to a location where the mean squared error (MSE) is less than or equal to a threshold value or a location where the similarity is greater than or equal to a threshold value by comparing the first feature vector and the second feature vector.
[0226] According to one example, the method further includes an operation of converting a notification text into a notification sound based on one of a plurality of voice synthesis models, an operation of outputting the converted notification sound through a speaker of the electronic device, an operation of obtaining a reference signal corresponding to the output notification sound input through the microphone, and an operation of storing the obtained reference signal in a memory of the electronic device.
[0227] According to one example, the method further includes an operation of generating text data by performing voice recognition on a second voice signal of the second section from which the notification sound signal has been removed, and an operation of performing a specified operation based on the text data.
[0228] It will also be understood that the present disclosure includes examples according to the following numbered paragraphs:
[0229] Paragraph 1. An electronic device comprises: a microphone; at least one processor; and a memory storing instructions, wherein the instructions, when executed by the at least one processor, are configured to cause the electronic device to receive a first voice signal through the microphone, detect a wake word from the first voice signal, output a preset notification sound in response to the wake word detection, and remove a wake sound signal from a second voice signal received through the microphone based on a position of the detected wake sound signal in the second voice signal, wherein the wake sound signal corresponds to the output preset notification sound.
[0230] Paragraph 2. In the above paragraph 1, the instructions are configured to cause the electronic device to remove the notification sound signal from the second voice signal based on a reference signal stored in the electronic device.
[0231] Paragraph 3. In the above paragraph 2, the instructions are set so that the electronic device determines a notification sound search range and identifies a location of the notification sound signal by comparing at least a portion of the reference signal and the second voice signal in the notification sound search range.
[0232] Paragraph 4. In the above paragraph 3, the instructions are set so that the electronic device determines the search range of the notification sound based on the timing of outputting the notification sound.
[0233] Paragraph 5. An electronic device according to paragraphs 2 to 4, further comprising a speaker, wherein the instructions are configured to cause the electronic device to obtain the preset notification sound by converting a notification text based on one of a plurality of voice synthesis models, output the converted notification sound through the speaker, receive the notification sound output through the speaker through the microphone, obtain a signal corresponding to the received notification sound, and store the obtained signal in the memory as the reference signal.
[0234] Paragraph 6. An electronic device according to paragraphs 2 to 5, wherein the instructions are configured to cause the electronic device to remove a section of a signal corresponding to a specified section length of the reference signal from a position of the notification sound signal in the second voice signal using the reference signal.
[0235] Paragraph 7. An electronic device according to paragraph 3 or 4, wherein the instructions are configured to cause the electronic device to obtain a first feature vector in a frequency domain corresponding to the reference signal, compare the first feature vector with a second feature vector in a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range, and identify a location of the notification sound signal based on a result of the comparison.
[0236] Paragraph 8. In the above paragraph 7, the instructions are set so that the electronic device compares the first feature vector and the second feature vector and identifies a location where the mean squared error (MSE) is less than or equal to a threshold value as the location of the notification sound signal.
[0237] Paragraph 9. An electronic device according to paragraphs 2 to 6, wherein the instructions are configured to cause the electronic device to determine a portion of the entire frequency range corresponding to the reference signal that includes a voice feature, and to identify a location of the notification sound signal using a first feature vector of the portion of the frequency range.
[0238] Paragraph 10. In the above paragraph 9, the instructions are set so that the electronic device identifies a similarity between the first feature vector and a second feature vector of a region corresponding to the partial region among the entire frequency region corresponding to at least a portion of the second voice signal, and identifies a location having a similarity greater than a threshold value as a location of the notification sound signal.
[0239] Paragraph 11. An electronic device according to paragraph 7, 8, or 10, wherein the instructions are configured to cause the electronic device to normalize the first feature vector and the second feature vector, identify the location of the notification sound signal by comparing the normalized first and second feature vectors, and remove the notification sound signal using a signal obtained by normalizing the reference signal based on the location of the notification sound signal.
[0240] Paragraph 12. An electronic device according to any one of paragraphs 1 to 11, wherein the instructions are configured to cause the electronic device to perform voice recognition on the second voice signal from which the notification sound signal has been removed, wherein the first voice signal is received during a first period and the second voice signal is received during a second period.
[0241] Paragraph 13. An electronic device according to paragraph 12, wherein the instructions are configured to cause the electronic device to perform voice recognition on the second voice signal from which the notification sound signal has been removed to generate text data, and to perform a specified action based on the text data.
[0242] Paragraph 14. A method for processing a voice signal in an electronic device comprises: receiving a first voice signal through a microphone of the electronic device; detecting a wake word from the first voice signal; outputting a preset notification sound in response to detection of the wake word; and removing a notification sound signal from a second voice signal received through the microphone based on a position of the detected notification sound signal in the second voice signal, wherein the notification sound signal may correspond to the output preset notification sound.
[0243] Paragraph 15. The method according to paragraph 14, wherein the operation of removing the notification sound signal includes an operation of removing the notification sound signal from the second voice signal based on a reference signal stored in the electronic device.
[0244] Paragraph 16. The method of paragraph 15, further comprising: an operation of determining a search range for the notification sound; an operation of identifying a location of the notification sound signal by comparing at least a portion of the reference signal and the second voice signal in the search range for the notification sound.
[0245] Paragraph 17. The method according to paragraph 16, wherein the operation of determining the notification sound search range includes an operation of determining the notification sound search range based on the timing of outputting the notification sound.
[0246] Paragraph 18. A method according to any one of paragraphs 15 to 17, wherein the electronic device further comprises a speaker, and comprises: an operation of obtaining the preset notification sound by converting a notification text based on one of a plurality of voice synthesis models; an operation of outputting the converted notification sound through the speaker; an operation of receiving the notification sound output through the speaker through the microphone; an operation of obtaining a signal corresponding to the received notification sound; and an operation of storing the obtained signal in the memory as the reference signal.
[0247] Paragraph 19. The method according to any one of paragraphs 15 to 18, wherein the operation of removing the notification sound signal includes an operation of removing a section of a signal corresponding to a specified section length of the reference signal from a position of the notification sound signal in the second voice signal using the reference signal.
[0248] Paragraph 20. The method further comprising, in any one of paragraphs 16 or 17, an operation of obtaining a first feature vector in a frequency domain corresponding to the reference signal; an operation of comparing the first feature vector with a second feature vector in a frequency domain corresponding to at least a portion of the second voice signal in the notification sound search range; and an operation of identifying a location of the notification sound signal based on a result of the comparison.
[0249] Paragraph 21. The method of paragraph 20, further comprising an operation of comparing the first feature vector and the second feature vector to identify a location where the mean squared error (MSE) is less than or equal to a threshold value as the location of the notification sound signal.
[0250] Paragraph 22. A method according to any one of paragraphs 15 to 19, further comprising: determining a portion of the entire frequency range corresponding to the reference signal that includes a voice feature; and identifying a location of the notification sound signal using a first feature vector of the portion of the frequency range.
[0251] Paragraph 23. The method of paragraph 22, further comprising: an operation of identifying a similarity between the first feature vector and a second feature vector of a region corresponding to the partial region among the entire frequency domain corresponding to at least a portion of the second voice signal; and an operation of identifying a location having a similarity greater than a threshold value as a location of the notification sound signal.
[0252] Paragraph 24. The method according to any one of paragraphs 20, 21 or 23, wherein the operation of removing the notification sound signal comprises: an operation of normalizing the first feature vector and the second feature vector; an operation of identifying the location of the notification sound signal by comparing the normalized first and second feature vectors; and an operation of removing the notification sound signal using a signal obtained by normalizing the reference signal based on the location of the notification sound signal.
[0253] Paragraph 25. A method according to any one of paragraphs 14 to 24, wherein the first voice signal is received during a first period, the second voice signal is received during a second period, and the method comprises an operation of performing voice recognition on the second voice signal from which the notification sound signal has been removed.
[0254] Paragraph 26. The method of paragraph 25, further comprising: performing voice recognition on the second voice signal from which the notification sound signal has been removed to generate text data; and performing a specified action based on the text data.
[0255] Paragraph 27. In a non-transitory storage medium storing instructions, the instructions, when executed by at least one processor of an electronic device, are configured to cause the electronic device to perform at least one operation, the at least one operation including: receiving a first voice signal through a microphone; detecting a wake word from the first voice signal; outputting a preset notification sound in response to detection of the wake word; and removing the wake sound signal from a second voice signal received through the microphone based on a position of the detected wake sound signal in the second voice signal, wherein the wake sound signal corresponds to the output preset notification sound.
Claims
1. In an electronic device (101), Mike (295, 550); At least one processor (120, 292, 520); and A memory (130, 299, 530) storing the instructions, wherein the instructions, when executed by the at least one processor, cause the electronic device to: Receive a first voice signal through the above microphone, Detecting a call word from the above first voice signal, In response to the above call word detection, a preset notification sound is output, An electronic device, wherein the notification sound signal is set to be removed from the second voice signal received through the microphone based on the location of the detected notification sound signal in the second voice signal, wherein the notification sound signal corresponds to the output preset notification sound.
2. In the first paragraph, the instructions cause the electronic device to: An electronic device configured to remove the notification sound signal from the second voice signal based on a reference signal stored in the electronic device.
3. In the second paragraph, the instructions cause the electronic device to: Determine the search range for the notification sound, An electronic device configured to identify a location of the alert sound signal by comparing at least a portion of the reference signal and the second voice signal within the alert sound search range.
4. In the third paragraph, the instructions cause the electronic device to: An electronic device set to determine a search range for the notification sound based on the timing of output of the notification sound.
5. In either of paragraph 2 or paragraph 4, further comprising a speaker (294, 555), The above instructions cause the electronic device to: By converting the notification text based on one of the multiple voice synthesis models, the preset notification sound is obtained, The above converted notification sound is output through the above speaker, The above notification sound output through the above speaker is received through the above microphone, Obtain a signal corresponding to the above received notification sound, An electronic device set to store the acquired signal in the memory as the reference signal.
6. In any one of paragraphs 2 to 5, the instructions cause the electronic device to: An electronic device configured to remove a section of a signal corresponding to a specified section length of the reference signal from a position of the notification sound signal in the second voice signal using the reference signal.
7. In the third or fourth paragraph, at least one processor, Obtain a first feature vector in the frequency domain corresponding to the above reference signal, In the above notification sound search range, the first feature vector is compared with a second feature vector in a frequency domain corresponding to at least a part of the second voice signal, An electronic device configured to identify the location of the notification sound signal based on the results of the comparison.
8. In the 7th paragraph, the instructions cause the electronic device to: An electronic device configured to compare the first feature vector and the second feature vector to identify a location where the mean squared error (MSE) is less than or equal to a threshold value as the location of the notification sound signal.
9. In any one of paragraphs 2 to 6, the instructions cause the electronic device to: Determine a portion of the entire frequency range corresponding to the above reference signal that includes voice features, An electronic device configured to identify a location of the notification sound signal by using a first feature vector of the above-described part of the region.
10. In the 9th paragraph, the instructions cause the electronic device to: Identifying the similarity between the first feature vector and the second feature vector of a region corresponding to the partial region among the entire frequency domain corresponding to at least a portion of the second voice signal, An electronic device configured to identify a location having a similarity greater than a threshold value as the location of the alarm sound signal.
11. In any one of paragraphs 7, 8 or 10, the instructions cause the electronic device to: Normalize the first feature vector and the second feature vector, The location of the notification sound signal is identified by comparing the normalized first and second feature vectors, An electronic device configured to remove the notification sound signal by using a signal obtained by normalizing the reference signal based on the location of the notification sound signal.
12. In any one of paragraphs 1 to 11, The first voice signal is received during the first period, and the second voice signal is received during the second period. The above instructions cause the electronic device to: An electronic device configured to perform voice recognition on the second voice signal from which the above-described notification sound signal has been removed.
13. In the 12th, the instructions cause the electronic device to: An electronic device configured to perform voice recognition on the second voice signal from which the notification sound signal has been removed to generate text data, and to perform a specified action based on the text data.
14. A method for processing a voice signal in an electronic device (101), An operation of receiving a first voice signal through a microphone (295, 550) of the electronic device; An operation of detecting a call word from the first voice signal; An action of outputting a preset notification sound in response to detection of the above call word; and A method for processing a voice signal, comprising: removing a notification sound signal from a second voice signal received through the microphone based on a position of the detected notification sound signal in the second voice signal, wherein the notification sound signal corresponds to the output preset notification sound.
15. In a non-transitory storage medium storing instructions, the instructions are set to cause the electronic device (101) to perform at least one operation when executed by at least one processor (120, 520), the at least one operation being: An act of receiving a first voice signal through a microphone (295, 550); An operation of detecting a call word from the first voice signal; An action of outputting a preset notification sound in response to detection of the above call word; and A storage medium comprising an action of removing the notification sound signal from the second voice signal received through the microphone based on the position of the detected notification sound signal in the second voice signal, wherein the notification sound signal corresponds to the output preset notification sound.
Citation Information
Patent Citations
Reporting device with alarm sound canceling function
JP2003296854A
Voice control device, operation method and computer program
JP2022033258A
Voice recognition method using recognition device resource efficiently
KR1020000042364A
Traffic information analysis method and apparatus for building digital twin
KR1020240158689A
Method and Apparatus for Treating Hypertension through Respiratory Training
KR102318827B1
