Sound recognition robot and control method of that robot

Abstract

The present invention relates to a sound recognition robot, and comprises: a microphone that detects a sound; a normalization unit that normalizes a Mel spectrogram generated from the sound; an artificial intelligence unit including a plurality of sound recognition models that have been trained with pieces of training data to which probability values for recognizing a sound as a specific sound signal respectively correspond in different noise environments; a feature information conversion unit that converts the normalized Mel spectrogram into a conversion method corresponding to each of the plurality of sound recognition models to generate feature information corresponding to each of the plurality of sound recognition models; and a control unit that selects any one of the plurality of sound recognition models, generates feature information from the Mel spectrogram in a conversion method corresponding to the selected sound recognition model, and controls a sound detected through any one sound recognition model to be recognized as a specific sound signal.

Description

Acoustic recognition robot and control method thereof

[0001] The present invention relates to a robot that recognizes sound and performs a function according to the recognized sound, and a method for controlling the robot.

[0002] With the advancement of robotics technology, robots are now detecting ambient sounds, including human voice signals, and providing various functions based on the detected sounds. For these audio recognition robots, the ability to extract meaningful acoustic signals from the sounds detected in their surroundings and accurately recognize these signals can be crucial criteria for determining their performance.

[0003] Meanwhile, various technologies are being applied to recognize acoustic signals more accurately. In particular, with the recent development of artificial intelligence technology, technologies for recognizing acoustic signals are emerging using acoustic recognition models with a DNN (Deep Neural Network) structure, an artificial neural network that includes multiple hidden layers between the input and output layers. In addition, deep learning technology, which applies weights learned using augmented data from various acoustic environments to multiple hidden layers included in the acoustic recognition model, is being further applied, thereby improving the accuracy of acoustic signals recognized from sounds detected from the robot's surroundings.

[0004] The acoustic recognition model to which this deep learning technology is applied extracts feature information from the received acoustic signal, and calculates the probability that the extracted feature information is one of multiple different preset acoustic signals through hidden layers to which the learned weights are applied. Then, based on the calculated probabilities, it has a process of recognizing the received acoustic signal as the acoustic signal with the highest probability. Therefore, if the noise level is high compared to the acoustic signal (low Signal to Noise Ratio, SNR) or the reverberation time (RT) is long, it is difficult to accurately extract feature information of the acoustic signal from the detected sound, and the accuracy of acoustic recognition may decrease.

[0005] Therefore, in cases where the SNR is low or the RT is large, methods to remove the detected noise and background noise have been studied, and as part of this study, PCEN (per Channel Energy Normalization) was introduced, which divides the Mel frequency spectrogram (hereinafter referred to as Mel spectrogram) into each channel by frequency band, and converts each divided channel into a recursive estimation of the expected value by assuming local weak-sense stationarity.

[0006] This PCEN transform method is proposed as an alternative to the log Mel scaling method of the conventional Mel Spectrogram, which amplifies the audio signal by a log value according to the Mel frequency, for the purpose of improving the robustness against channel distortion. It combines dynamic range compression (DRC) and adaptive gain control (AGC) with temporal integration. Here, dynamic range compression reduces the range of foreground loudness from the audio signal, and the adaptive gain control suppresses fixed background noise, thereby preserving and enhancing the local contrast near the onset and offset of the foreground loudness while suppressing amplitude fluctuations due to background noise.

[0007] However, this PCEN transform method can improve the accuracy of sound recognition more than the log-Mel transform method in situations where the SNR is low and the RT is high. However, it has the problem that the number of parameters increases and the amount of computation increases because it performs additional operations such as performance and adaptive gain control for each channel, that is, frequency. Moreover, in general noisy situations with high SNR or low RT, the PCEN transform method distinguishes channels according to frequency and suppresses some sounds as background noise, which may actually lower the accuracy of sound recognition compared to the existing log-Mel transform method.

[0008] The present invention aims to solve the above-mentioned problems and other problems, and provides a sound recognition robot and a control method thereof that can perform more accurate sound recognition not only in an extremely noisy environment with low SNR and high RT, but also in a normal noise environment with high SNR and low RT.

[0009] In order to achieve the above or other purposes, according to one aspect of the present invention, a sound recognition robot according to an embodiment of the present invention is characterized by including a microphone for detecting sound, a normalization unit for normalizing a Mel spectrogram generated from the detected sound, an artificial intelligence unit including a plurality of sound recognition models, each of which is learned with learning data corresponding to different noise environments and having probability values ​​for recognizing the detected sound as a specific sound signal, a feature information conversion unit for converting the normalized Mel spectrogram using different conversion methods corresponding to each of the plurality of sound recognition models to generate feature information corresponding to each of the plurality of sound recognition models, and a control unit for selecting any one of the plurality of sound recognition models, generating feature information from the normalized Mel spectrogram using a conversion method corresponding to the selected sound recognition model, and controlling the feature information conversion unit and the artificial intelligence unit to recognize the detected sound as a specific sound signal through any one of the selected sound recognition models, and controlling the operation of the sound recognition robot according to the recognition result.

[0010] In one embodiment, the feature information conversion unit is characterized in that it converts the normalized Mel spectrogram using a PCEN (Per Channel Energy Normalization) conversion method to generate PCEN feature information in which adaptive gain control (AGC) is performed for each channel according to Mel frequency, or converts the normalized Mel spectrogram using a Log Mel conversion method to generate Log Mel feature information scaled by a log exponent according to Mel frequency.

[0011] In one embodiment, the artificial intelligence unit includes a PCEN acoustic recognition model that recognizes the detected sound as a specific acoustic signal based on the PCEN feature information, and a log-MEL acoustic recognition model that recognizes the detected sound as a specific acoustic signal based on the log-MEL feature information, wherein the PCEN acoustic recognition model is a deep neural network (DNN) model in which weights of hidden layers are learned through learning data converted into the PCEN feature information, and the log-MEL acoustic recognition model is characterized in that the weights of the hidden layers are learned through learning data converted into the log-MEL feature information.

[0012] In one embodiment, the learning data converted into the PCEN feature information are learning data formed in a high-noise environment in which there is a signal to noise ratio (SNR) below a preset level and a reverberation time (RT) above a predetermined value, and the learning data converted into the log-mel feature information are learning data formed in a low-noise environment in which there is a signal to noise ratio (SNR) above the preset level and a reverberation time (RT) below the predetermined value.

[0013] In one embodiment, the control unit is characterized in that it selects one of the plurality of acoustic recognition models based on at least one of the operation mode of the acoustic recognition robot and the noise environment of the acoustic recognition robot, and controls the feature information conversion unit to convert the normalized mel spectrogram into feature information according to the selected acoustic recognition model.

[0014] In one embodiment, the control unit is characterized in that, when the operation mode of the robot is a preset operation mode, the control unit controls the feature information conversion unit to generate the PCEN feature information by converting the normalized Mel spectrogram, and controls the artificial intelligence unit so that the PCEN acoustic recognition model recognizes an acoustic signal based on the PCEN feature information; and, when the operation mode of the robot is not a preset operation mode, the control unit controls the feature information conversion unit to generate the log Mel feature information by converting the normalized Mel spectrogram, and controls the artificial intelligence unit so that the log Mel acoustic recognition model recognizes an acoustic signal based on the log Mel feature information.

[0015] In one embodiment, the preset operation mode is an operation mode in which the acoustic recognition robot generates noise above a preset level, and when the acoustic recognition robot is a robot vacuum cleaner, the operation mode is a turbo operation mode or an intensive cleaning mode in which the RPM (Revolutions Per Minute) of the vacuum motor providing vacuum suction power is above a certain speed.

[0016] In one embodiment, the control unit is characterized in that, when the sound recognition robot is driven according to the preset operation mode, the control unit converts the normalized mel spectrogram according to the first conversion method to generate feature information, and inputs the converted normalized mel spectrogram into a first sound recognition model among the plurality of sound recognition models to perform first sound signal recognition, and when an sound signal corresponding to the sound is recognized as a result of the first sound recognition, the control unit changes the operation mode of the sound recognition robot, and converts the normalized mel spectrogram into a second conversion method different from the first conversion method according to the changed operation mode to generate feature information, and performs second sound signal recognition through the second sound recognition model different from the first sound recognition model.

[0017] In one embodiment, when the acoustic recognition robot is a robot vacuum cleaner, the preset operation mode is the turbo operation mode or the intensive cleaning mode, and the changed operation mode is an operation mode in which noise resulting from the operation of the acoustic recognition robot is below the preset level among operation modes other than the turbo operation mode or the intensive operation mode.

[0018] In one embodiment, the control unit controls the sound recognition robot to be driven according to the result of the secondary sound signal recognition, and, when the recognized sound signal is any one of a preset pet sound, a preset sound according to a user's gesture, and a drive sound of a specific device that has been registered, controls the sound recognition robot to perform a preset operation according to the recognized sound signal.

[0019] In one embodiment, the sound according to the user's gesture includes at least one of a clapping sound, a finger snap sound, and a whistling sound, and the operating sound of the specific device is a sound generated when a preset device is operated, and includes at least one of a telephone ringtone registered by the user and an operating noise of a hair dryer.

[0020] In one embodiment, the control unit is characterized in that, when the driving sound of the specific device is detected, the control unit controls the sound recognition robot to perform a first function related to the driving of the specific device while the driving sound of the specific device is detected through the first or second sound recognition, and when the driving sound of the specific device is lost as a result of the sound recognition performed while performing the first function, the control unit controls the sound recognition robot to perform a second function related to the termination of the driving of the specific device.

[0021] In one embodiment, the sound recognition robot further includes an sound output unit that outputs an sound signal, and the control unit controls the sound output unit to change the operation mode and utter a message requesting a user's confirmation when the recognized sound signal is a preset trigger word, and performs sound recognition on a sound detected in response to the uttered message in a conversion manner with an sound recognition model according to the changed operation mode.

[0022] In one embodiment, a first threshold value of a probability value for recognizing a sound detected by the microphone as a specific sound signal in the first sound signal recognition is lower than a second threshold value of a probability value for recognizing a sound detected by the microphone as a specific sound signal in the second sound signal recognition.

[0023] In one embodiment, the learning data is formed so that a learning sound source to be learned is placed after noise of a preset size, and the noise of the preset size is characterized in that it is noise filled in front of the learning sound source in a zero padding manner by the difference between the length of the preset learning data and the length of the learning sound source.

[0024] In order to achieve the above or other purposes, according to one aspect of the present invention, a method for controlling a sound recognition robot according to an embodiment of the present invention comprises the steps of: controlling the operation of the sound recognition robot so as to be driven in an operation mode according to a user's input; generating a normalized Mel spectrogram from sound detected from a microphone of the sound recognition robot; selecting one sound recognition model according to the operation mode of the sound recognition robot from among a plurality of sound recognition models learned with learning data corresponding to different noise environments, each of which has probability values ​​for recognizing the detected sound as a specific sound signal; converting the normalized Mel spectrogram using one of the different conversion methods corresponding to the selected sound recognition model from among the different conversion methods corresponding to each of the plurality of sound recognition models to generate feature information; inputting the generated feature information into the selected sound recognition model to recognize the detected sound as a specific sound signal; and controlling the sound recognition robot so as to perform a function corresponding to the recognized sound signal according to a result of the sound signal recognition.

[0025] In one embodiment, the step of selecting one of the acoustic recognition models is characterized in that it is a step of selecting one of the acoustic recognition models from among a plurality of acoustic recognition models depending on whether the operation mode of the acoustic recognition robot is a preset operation mode and whether the noise level around the acoustic recognition robot is greater than or equal to the preset level.

[0026] In one embodiment, the preset operation mode is an operation mode in which the acoustic recognition robot generates noise above a preset level, and when the acoustic recognition robot is a robot vacuum cleaner, the operation mode is a turbo operation mode or an intensive cleaning mode in which the RPM of a vacuum motor providing vacuum suction power is above a certain speed.

[0027] In one embodiment, the step of selecting any one of the acoustic recognition models is characterized in that, when the operation mode of the acoustic recognition robot is the preset operation mode or the noise level around the acoustic recognition robot is equal to or greater than the preset level, a PCEN acoustic recognition model that recognizes an acoustic signal from feature information obtained by converting the normalized Mel spectrogram using a PCEN (Per Channel Energy Normalization) transformation method is selected from among the plurality of acoustic recognition models, and when the operation mode of the acoustic recognition robot is not the preset operation mode and the noise level around the acoustic recognition robot is less than the preset level, a Log Mel acoustic recognition model that recognizes an acoustic signal from feature information obtained by converting the normalized Mel spectrogram using a Log Mel transformation method is selected from among the plurality of acoustic recognition models.

[0028] In one embodiment, the step of recognizing the detected sound as a specific sound signal is characterized in that the step further includes: a step of changing the operation mode of the sound recognition robot to an operation mode other than the preset operation mode when the detected sound is recognized as a specific sound signal through the selected sound recognition model when the operation mode of the sound recognition robot is the preset operation mode; a step of changing the selected sound recognition model according to the changed operation mode; a step of generating feature information of a different method by converting the normalized mel spectrogram using a different conversion method corresponding to the changed sound recognition model; and a step of inputting the feature information of the different method into the changed sound recognition model to recognize the detected sound as a specific sound signal.

[0029] The effects of the acoustic recognition robot and the control method thereof according to the present invention are described as follows.

[0030] According to at least one of the embodiments of the present invention, the present invention can determine whether the current situation is an extremely bad noise state or a normal noise state based on at least one of the operation mode of the robot and the result of detecting the level of ambient sound, and can convert a Mel spectrogram into feature information (Mel feature) according to any one of different feature information conversion methods depending on the determined noise state. In addition, by inputting the converted feature information into a pre-trained acoustic recognition model according to the determined noise state to perform acoustic recognition, there is an effect that more accurate acoustic recognition can be performed.

[0031] Furthermore, the present invention can change the robot's operating mode based on the recognized acoustic signal when a preset acoustic signal is recognized, and can transform the feature information conversion method and the acoustic recognition model for recognizing the acoustic signal based on the changed operating mode. Accordingly, the present invention has the effect of enabling the robot to more accurately recognize the user's acoustic signal by changing the operating mode based on the recognized acoustic signal in an extremely noisy environment.

[0032] FIG. 1 is a block diagram illustrating the configuration of an acoustic recognition robot according to an embodiment of the present invention.

[0033] FIG. 2 is an exemplary diagram showing an example of learning data for learning each acoustic recognition model of an acoustic recognition robot according to an embodiment of the present invention.

[0034] FIG. 3 is a flowchart illustrating an operation process of recognizing an acoustic signal and performing a function according to the recognized acoustic signal in an acoustic recognition robot according to an embodiment of the present invention.

[0035] FIG. 4 is a conceptual diagram for explaining the process of generating a Mel Spectrogram in an acoustic recognition robot according to an embodiment of the present invention.

[0036] FIG. 5 is a flowchart illustrating in more detail the operation process of recognizing an acoustic signal using one acoustic recognition model based on the operation mode and ambient noise in an acoustic recognition robot according to an embodiment of the present invention.

[0037] FIG. 6 is a flowchart illustrating an operation process in which an acoustic recognition robot according to an embodiment of the present invention changes an operation mode based on an acoustic recognition result and re-recognizes an acoustic signal.

[0038] FIG. 7 is a flowchart illustrating an operation process in which an acoustic recognition robot according to an embodiment of the present invention changes an operation mode based on an acoustic recognition result.

[0039] FIG. 8 is a flowchart illustrating an operation process in which an acoustic recognition robot according to an embodiment of the present invention recognizes an acoustic signal by changing the operation mode depending on whether a preset trigger word is recognized.

[0040] FIG. 9 is a flowchart illustrating an operation process in which an acoustic recognition robot according to an embodiment of the present invention determines the operation state of a preset device and performs different operations according to the determined operation state.

[0041] It should be noted that the technical terms used herein are used merely to describe specific embodiments and are not intended to limit the present invention. Furthermore, singular expressions used herein include plural expressions unless the context clearly dictates otherwise. The suffixes "module" and "part" used in the following description for components are assigned or used interchangeably solely for the convenience of writing the specification, and do not in themselves have distinct meanings or roles.

[0042] In this specification, the terms “comprises” or “includes” should not be construed to necessarily include all of the components or steps described in the specification, and some of the components or steps may not be included, or additional components or steps may be included.

[0043] In addition, when describing the technology disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the technology disclosed in this specification, the detailed description is omitted.

[0044] In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. In addition, not only each embodiment described below, but also a combination of embodiments may correspond to the spirit and technical scope of the present invention as modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

[0045] FIG. 1 is a block diagram illustrating the configuration of an acoustic recognition robot according to an embodiment of the present invention. FIG. 2 is an exemplary diagram illustrating examples of learning data for training each acoustic recognition model of an acoustic recognition robot according to an embodiment of the present invention.

[0046] First, referring to FIG. 1, the sound recognition robot (10) according to an embodiment of the present invention may be configured to include a control unit (100), a wireless communication unit (110) controlled by the control unit (100), an input unit (120), a sensing unit (130), a driving unit (140), an output unit (150), a normalization unit (160), a feature information conversion unit (170), an artificial intelligence unit (180), and a memory (190). The components illustrated in FIG. 1 are not essential for implementing the sound recognition robot (10), and thus the sound recognition robot (10) described in this specification may have more or fewer components than the components listed above.

[0047] More specifically, among the above components, the communication unit (110) may include one or more modules that enable wireless communication between the acoustic recognition robot (10) and a wireless communication system, between the acoustic recognition robot (10) and at least one peripheral device, or between the acoustic recognition robot (10) and an external server.

[0048] This communication unit (110) may include at least one of a wireless Internet module (111), a short-range communication module (112), and a location information module (113).

[0049] The wireless Internet module (111) refers to a module for wireless Internet access, and can be built into or externally installed in the sound recognition robot (10). The wireless Internet module (111) is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0050] The short-range communication module (112) is for short-range communication and can support short-range communication using at least one of Bluetooth, RFID, infrared communication, UWB, ZigBee, NFC, Wi-Fi, Wi-Fi Direct, and Wireless USB technologies. This short-range communication module (112) can support wireless communication between the sound recognition robot (10) and a wireless communication system, between the sound recognition robot (10) and a peripheral device, or between the sound recognition robot (10) and a network where an external server is located through a short-range wireless communication network (Wireless Area Networks).

[0051] The location information module (113) is a module for obtaining the location (or current location) of the acoustic recognition robot (10), and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if the acoustic recognition robot (10) utilizes a GPS module, it can obtain the location of the acoustic recognition robot (10) using signals transmitted from GPS satellites. As another example, if the acoustic recognition robot (10) utilizes a Wi-Fi module, it can obtain the location of the acoustic recognition robot (10) based on information from a wireless AP (Wireless Access Point) that transmits or receives wireless signals with the Wi-Fi module.

[0052] As needed, the location information module (113) may perform the function of any one of the other modules of the communication unit (110) to obtain data regarding the location of the acoustic recognition robot (10) as a substitute or additionally. The location information module (113) is a module used to obtain the location (or current location) of the acoustic recognition robot (10), and is not limited to a module that directly calculates or obtains the location of the acoustic recognition robot (10).

[0053] And the input unit (120) may include a camera (121) or a video input unit for inputting a video signal, a microphone (122, hereinafter referred to as a microphone) for inputting an audio signal, or an audio input unit (hereinafter referred to as a microphone). In addition, it may include a user input unit (123, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected from the input unit (120) may be recognized through a preset algorithm, for example, an artificial intelligence model, and may be processed by a user's control command.

[0054] Meanwhile, the camera (121) and microphone (122) can collect images and audio signals from around the sound recognition robot (10), respectively. In this case, the camera (121) may be equipped with one or more image sensors and can process image frames, such as still images or moving images, obtained by the image sensors. The processed image frames can then be stored in the memory (190).

[0055] Additionally, the microphone (122) can sample external acoustic signals at preset time intervals and process them into acoustic data. The processed acoustic data can be converted into data for acoustic recognition in the acoustic recognition robot (10), for example, into a frame-by-frame Mel Spectrogram. The converted Mel Spectrogram can then be used as input for an artificial intelligence model for acoustic recognition.

[0056] Here, various noise removal algorithms may be implemented in the microphone (122) to remove noise generated during the process of receiving external acoustic signals. The microphone (122) may be designed with a beamforming structure having directionality, so as to be formed to further improve the reception rate of acoustic information in a specific direction. In this case, the specific direction may be the front of the robot in which the acoustic recognition robot (10) is implemented.

[0057] And the sensing unit (130) may include one or more sensors for sensing at least one of information within the sound recognition robot (10) and information about the surrounding environment surrounding the sound recognition robot (10). For example, the sensing unit (130) may include at least one of a proximity sensor (131), an illumination sensor (132), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), an ultrasonic sensor, an optical sensor (e.g., a camera (see 121)), a microphone (see 122), an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). The acoustic recognition robot (10) disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0058] Meanwhile, the driving unit (140) may be a component for performing the function of an acoustic recognition robot (10) or a device equipped with an acoustic recognition robot (10).

[0059] For example, the above-mentioned acoustic recognition robot (10) may be a robot for performing a specific function. And the above-mentioned driving unit (140) may be a driving unit for performing a specific function of the robot.

[0060] For example, the acoustic recognition robot (10) may be a robot (e.g., a vacuum cleaner) that performs a cleaning function. In this case, the driving unit (140) may include at least one component for performing the cleaning function. For example, the driving unit (140) may include a suction unit for vacuum-sucking dust and a vacuum motor for providing the vacuum suction force. In addition, the driving unit (140) may include at least one wheel for movement and at least one motor for controlling the rotation of the wheel and at least one actuator. In addition, the driving unit (140) may include a brush and at least one motor for controlling the rotation of the brush.

[0061] Meanwhile, if the acoustic recognition robot (10) is a robot for performing a patrol function, the driving unit (140) may include a component for the patrol function. For example, the driving unit (140) may include a moving unit (e.g., a wheel, a motor, an actuator) for moving the acoustic recognition robot (10), i.e., the robot, and may include a lighting unit including at least one light for supporting the patrol function.

[0062] In this way, if the acoustic recognition robot (10) is a robot that performs a specific function, the driving unit (140) may be implemented as an integral part of the acoustic recognition robot (10). Alternatively, the driving unit (140) may be formed to be detachable from the acoustic recognition robot (10), and if the mounted driving unit is changed, the acoustic recognition robot (10) may operate as a robot that performs different functions depending on the recognized acoustic signal.

[0063] And the output unit (150) is for generating output related to vision or hearing, and may include a display unit (151) and an audio output unit (152).

[0064] The display unit (151) can display (output) various information (hereinafter referred to as image information) related to the operation of the sound recognition robot (10) and information related to the operation of the driving unit (140). For example, the display unit (151) can display information on a recognized sound signal, an execution screen of a currently executing function, or UI (User Interface) or GUI (Graphical User Interface) information according to the displayed information.

[0065] In addition, the audio output unit (152) can output various audio data according to the control of the control unit (100). For example, the audio output unit (152) can output information about a recognized audio signal in the form of a human voice or output audio data related to a currently executing function (e.g., voice alarm, siren, etc.) according to the control of the control unit (100).

[0066] Meanwhile, the normalization unit (160) can collect sound data generated by sampling sound signals from a microphone (122) to extract a Mel Spectrum vector, and can collect the extracted Mel Spectrum vectors to generate a Mel Spectrogram. To this end, the normalization unit (160) can divide the detected voice data into short time units and perform a Fourier transform (STFT, Short Time Fourier Transform). Then, the frequency of the Fourier-transformed short time unit sound signal can be converted into a Mel signal by using the following logarithmic function (Mel scale function).

[0067] [Mel scale function: Mel(f)]

[0068]

[0069] Here, f is the frequency of the acoustic signal.

[0070] Here, the Mel frequency (Mel(f)) is the subjective perception of an acoustic signal according to a frequency recognizable to humans, and can be a unit expressed linearly by tuning a nonlinear frequency unit according to a frequency recognizable to humans.

[0071] And the normalization unit (160) can convert the data of the sound signal converted into the Mel frequency into a spectral density function to generate a Mel spectrum vector. And the normalization unit (160) can collect the Mel spectrogram vector to generate a Mel spectrogram. In this case, the Mel spectrogram is a spectrogram that divides the frequency of the sound signal into a preset number of Mel units to reflect the audible frequency of a human, and can represent the spectrum change of the sound signal according to the change in time.

[0072] Here, when a Mel Spectrogram is generated, the normalization unit (160) can detect the Mel Spectrogram vector having the maximum value among each Mel Spectrogram vector constituting the Mel Spectrogram. Then, the Mel Spectrogram can be normalized based on the detected maximum value. For example, the normalization unit (160) can normalize the Mel Spectrogram by dividing the Mel Spectrogram by the square of the detected maximum value.

[0073] And the feature information conversion unit (170) can convert the normalized Mel Spectrogram into feature information for inputting into an artificial intelligence model for sound recognition, i.e., a sound recognition model. Here, the feature information conversion unit (170) can convert the Mel Spectrogram into different feature information specialized for different sound recognition models under the control of the control unit (100).

[0074] For example, the feature information conversion unit (170) can convert the Mel Spectrogram into feature information (hereinafter referred to as log Mel feature information) by using a log scaling method that scales the Mel Spectrogram using a log function. In this case, the Mel Spectrogram can be scaled linearly within a preset frequency band according to the Mel frequency, and according to a log function in other frequency bands. In other words, the Mel Spectrogram can be converted into feature information (hereinafter referred to as log Mel feature information) by using a log Mel transformation method that scales the Mel Spectrogram into a log value according to the Mel frequency.

[0075] Alternatively, the feature information conversion unit (170) may divide the Mel Spectrogram (Mel Spectrogram) into channels for each frequency band. Then, the Mel Spectrogram may be converted into feature information (PCEN feature information) by using a PCEN conversion method that performs adaptive gain control (AGC) through recursive estimation for each divided channel.

[0076] Here, the feature information conversion unit (170) may be configured to include a log-Mel conversion unit (172) that converts the Mel spectrogram using the log-Mel conversion method, and a PCEN conversion unit (171) that converts the Mel spectrogram using the PCEN conversion method. In this case, the feature information conversion unit (170) may convert the Mel spectrogram normalized in the normalization unit (160) through either the log-Mel conversion unit (172) or the PCEN conversion unit (171) under the control of the control unit (100).

[0077] Meanwhile, the artificial intelligence unit (180) performs the role of processing information based on artificial intelligence technology, and may include one or more modules that perform at least one of information learning, information inference, information perception, and natural language processing.

[0078] The artificial intelligence unit (180) can use machine learning technology to determine what kind of acoustic signal the sound received through the microphone (122) is. To this end, the artificial intelligence unit (180) can use a deep neural network (DNN) model, i.e., an acoustic recognition model, based on data learned using the machine learning technology.

[0079] Here, learning can be achieved through the aforementioned machine learning technology. This machine learning technology, based on at least one algorithm, collects and learns large amounts of information, and uses the learned information to make judgments and predictions. Information learning involves identifying the characteristics, rules, and judgment criteria of the information, quantifying the relationships between information, and using these quantified patterns to predict new data.

[0080] Meanwhile, the probability vector inferred through the deep neural network may be a probability value indicating the probability that the received sound is a specific acoustic signal. In other words, the probability vectors for each of the multiple different acoustic signals inferred based on the deep learning model may indicate probability values ​​that the received sound may be each of the different acoustic signals. In addition, the probability values ​​may be learned from the provided learning data through the machine learning technology.

[0081] Here, the aforementioned deep neural network is a type of artificial neural network (ANN) and may include multiple layers (hidden layers) between the input layer and the output layer. Furthermore, the layers may be interconnected, and may have a structure in which data is transferred between the interconnected layers.

[0082] Meanwhile, the artificial intelligence unit (180) may include a plurality of acoustic recognition models trained with a sufficient number of learning data to infer probability vectors of each of a plurality of different acoustic signals set in advance for the acoustic data.

[0083] Here, the training data for each of the above acoustic recognition models may be training data formed in different acoustic environments, for example, environments with different SNR (Signal To Noise) and RT (Reverberation Time) values. In this case, the different acoustic recognition models provided in the artificial intelligence unit (180) may be acoustic recognition models trained using training data formed in different acoustic environments.

[0084] For example, among the plurality of acoustic recognition models, a first acoustic recognition model may be a model learned through training data having a low SNR and a high RT value. And, among the plurality of acoustic recognition models, a second acoustic recognition model may be a model learned through training data having a high SNR and a low RT value.

[0085] In this case, the training data used to train the different acoustic recognition models may be mel spectrograms transformed using different feature information transformation methods. In other words, the different acoustic recognition models may be models specialized in different feature information transformation methods, trained based on feature information obtained by transforming the normalized mel spectrograms using different methods.

[0086] For example, the first acoustic recognition model may be an acoustic recognition model in which probability values ​​corresponding to each acoustic signal for the received sound are learned based on feature information generated by converting a mel spectrogram generated from an acoustic signal (training data) having a low SNR and a high RT value according to the PCEN conversion method. In other words, the first acoustic recognition model may be an acoustic recognition model in which learning is performed through feature information (PCEN feature information) obtained by converting a mel spectrogram generated from training data according to the PCEN conversion, and may be an acoustic recognition model (PCEN acoustic recognition model (181)) specialized in the PCEN conversion method.

[0087] On the other hand, the second acoustic recognition model may be an acoustic recognition model in which probability values ​​corresponding to each acoustic signal for the received sound are learned based on feature information generated by converting a Mel spectrogram generated from an acoustic signal (training data) having a high SNR and a low RT value according to the log Mel transformation method. That is, the second acoustic recognition model is an acoustic recognition model in which learning is performed through feature information (log Mel feature information) obtained by converting a Mel spectrogram generated from training data according to log Mel scaling, and may be an acoustic recognition model (PCEN acoustic recognition model (181)) specialized in the PCEN transformation method.

[0088] In this way, the artificial intelligence unit (180) may be equipped with different acoustic recognition models specialized to convert augmented learning data in different acoustic environments according to different feature information conversion methods and to recognize acoustic signals according to the converted feature information. In addition, when the Mel Spectrogram normalized in the normalization unit (160) is converted according to a specific feature information conversion method under the control of the control unit (100) to generate feature information, the generated feature information may be input to any one of the plurality of acoustic recognition models. In addition, the acoustic recognition result produced from the acoustic recognition model into which the feature information has been input may be transmitted to the control unit (100) as an acoustic signal recognition result.

[0089] Meanwhile, the above learning data may be learning data in which the sound source to be learned is placed after zero padding.

[0090] For example, in training an acoustic recognition model, if the length of the sound source to be trained is shorter than the length of the training data generated to the minimum length required for acoustic recognition, zero padding may be performed, in which the remaining area excluding the sound source to be trained is filled with noise of a preset size, i.e., a zero value. In this case, the training data of the acoustic recognition model according to an embodiment of the present invention may be training data in which zero padding is performed in front of the sound source to be trained.

[0091] FIG. 2 is an exemplary diagram showing an example of learning data for learning each acoustic recognition model of an acoustic recognition robot according to an embodiment of the present invention.

[0092] First, referring to (a) of Fig. 2, (a) of Fig. 2 shows the length (200) of the sound source (201) to be learned, and (b) of Fig. 2 shows the length of learning data having the minimum length (210) required for sound recognition.

[0093] In this case, the learning data according to the embodiment of the present invention may be filled with zero values ​​in advance of the sound source (201) to be learned, which is equal to the difference (220) between the length of the sound source (201) to be learned and the length (210) of the learning data. That is, zero values ​​in the amount of the difference (220) between the length (210) of the learning data and the length (200) of the sound source to be learned may be further filled in front of the sound source (201) to be learned, so that learning data having a length corresponding to the length (210) of the learning data may be generated.

[0094] In this way, in the learning data according to an embodiment of the present invention, zero padding can be performed so that zero values ​​are filled in front of the sound source (201) by a difference (220) corresponding to the difference between the length (210) of the learning data and the length (200) of the sound source so that the end of the length (200) of the sound source (201) matches the end of the length (210) of the learning data. Therefore, the learning data of the present invention can simulate the acoustic signal of a real-time system in which noise exists before the actual sound source since the sound source to be learned is placed after the zero values. Accordingly, the acoustic recognition response speed of the learning data can be further improved.

[0095] Meanwhile, the memory (190) can store data that supports the functions of the acoustic recognition robot (10). The memory (190) can store a plurality of application programs (or applications) running on the acoustic recognition robot (10), data for the operation of the acoustic recognition robot (10), commands, and data for the operation of the artificial intelligence unit (180) (e.g., hidden layers of each acoustic recognition model and weight information of each hidden layer, information for updating or learning, etc.).

[0096] In addition, the memory (190) may store data obtained by sampling the sound signal received from the microphone (122), i.e., sound data. In addition, a Mel Spectrogram generated by the normalization unit (160) based on the stored sound data, data required for normalization of the Mel Spectrogram, and information on the Mel Spectrogram normalized by the normalization unit (160) may be stored. In addition, the memory (190) may store results output from the artificial intelligence unit (180).

[0097] Meanwhile, the control unit (100) controls each connected component and can control the overall operation of the acoustic recognition robot (10).

[0098] First, the control unit (100) can sample sound received through the microphone (122). Then, the control unit (100) can perform STFT on a preset number of samples to detect the frequency of the sound signal corresponding to the preset number of samples, and convert the detected frequency into a Mel frequency. Then, the control unit (100) can control the normalization unit (160) to convert the sound signals converted into the Mel frequency into a spectral density function to generate a Mel spectrum vector, and then compile the generated Mel spectrogram vectors to generate a Mel spectrogram. Then, among the Mel spectrum vectors constituting the Mel spectrogram, normalization can be performed based on the Mel spectrum vector having the maximum value.

[0099] And the control unit (100) can detect at least one of the operation state and surrounding situation of the acoustic recognition robot (10). And, based on at least one of the detected operation state and surrounding situation, the feature information conversion unit (170) can be controlled to convert the normalized Mel Spectrogram through any one of a plurality of feature information conversion methods. And, when the Mel Spectrogram is converted into feature information, any one of the detected operation state and surrounding situation can be selected, and the artificial intelligence unit (180) can be controlled to recognize an acoustic signal using the selected acoustic recognition model. And, when an acoustic signal is recognized through the selected acoustic recognition model, the control unit (100) can control the driving unit (140) to perform a function corresponding to the recognized acoustic signal.

[0100] Here, the control unit (100) can control the normalization unit (160), the feature information conversion unit (170), and the artificial intelligence unit (180) to continuously generate a normalized Mel Spectrogram from sound received through the microphone (122), convert the generated Mel Spectrogram into feature information, and continue the operation of recognizing the sound signal from the converted feature information even while the driving unit (140) is controlled in a specific operation mode according to a recognized sound signal or an input input through the user input unit (123).

[0101] And as a result of the continuous acoustic recognition, if a designated acoustic signal is recognized, the operation mode of the acoustic recognition robot (10) can be changed. In this case, if the operation mode is changed, the control unit (100) can change both the feature information conversion method for converting the Mel Spectrogram into feature information according to the changed operation mode and the acoustic recognition model for recognizing the acoustic signal according to the converted feature information. And acoustic recognition can be performed based on the changed feature information conversion method and the changed acoustic recognition model. Therefore, if a preset acoustic signal is recognized while driving in an operation mode that generates a lot of noise, such as a turbo operation mode, the control unit (100) changes the operation mode to an operation mode that generates less noise and recognizes the acoustic signal again, thereby improving the accuracy of acoustic recognition.

[0102] FIG. 3 is a flowchart illustrating an operational process of recognizing an acoustic signal and performing a function based on the recognized acoustic signal in an acoustic recognition robot according to an embodiment of the present invention. FIG. 4 is a conceptual diagram illustrating a process of generating a Mel Spectrogram in an acoustic recognition robot according to an embodiment of the present invention.

[0103] First, referring to FIG. 3, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention can first sample acoustic signals detected through a microphone (122) at preset time intervals (S300). In this case, each of the sampled acoustic signals can be discretized into different acoustic data.

[0104] And the control unit (100) can sequentially collect a preset number of sound data through buffer handling (S302). And the collected sound data can be converted into data in the frequency domain through a Fourier transform (STFT) operation on the collected sound data, and the converted frequency domain sound data can be converted into Mel frequency through a Mel scale function. And the sound data converted into Mel frequency can be converted into a spectral density function to extract a Mel spectrum vector (S304).

[0105] Here, the above-described number of sound data may be sound data corresponding to one frame. Accordingly, in step S304, a mel spectrum vector corresponding to the sound data of the one frame may be generated.

[0106] And the control unit (100) can sequentially collect a preset number of Mel spectrum vectors. Then, the Mel spectrum vectors collected in the preset number can be stored as a Mel spectrogram representing the change in frequency (Mel frequency) over time corresponding to the preset number of Mel spectrum vectors (S306).

[0107] FIG. 4 is a conceptual diagram for explaining the process of generating the Mel Spectrogram in an acoustic recognition robot (10) according to an embodiment of the present invention.

[0108] Referring to FIG. 4, as shown in the acoustic signal detection result (400), an acoustic signal can be detected by a microphone (122). Then, the control unit (100) can sample the detected acoustic signal at preset time intervals and discretize the sampled acoustic signals into acoustic data. The buffered data (410) of FIG. 4 shows an example in which such discretized acoustic data are listed in chronological order.

[0109] In this state, the control unit (100) can collect a preset number of sound data into one frame of sound data. Here, one frame of sound data can be composed of 320 sound data. Therefore, the control unit (100) can collect 320 sound data into one frame of sound data. Therefore, the control unit (100) can perform a Fourier transform (STFT) on the sound data of each of the 320 collected sound frames and convert them into Mel frequency through the Mel scale function. Then, the sound data converted into Mel frequency can be converted into a spectral density function to generate Mel spectrum vectors (451, 452).

[0110] Accordingly, a first Mel spectrum vector (451) can be generated from sound samples sampled from 1 to 320. And a second Mel spectrum data (452) can be generated from sound samples sampled from 321 to 640. And when this process is sequentially repeated to generate a preset number of Mel spectrum vectors, the control unit (100) can generate one Mel spectrogram by collecting the preset number of Mel spectrum vectors in the time order of the sound data. In this case, the control unit (100) can collect 96 Mel spectrum vectors and generate one Mel spectrogram.

[0111] The Mel spectrogram (460) of Fig. 4 may be generated by collecting 96 Mel spectrum vectors in this way. In this case, the Mel spectrogram (460) may be a spectrogram showing the change in Mel frequency over a time corresponding to 320 pieces of sound data X 96 Mel spectrum vectors, i.e. 30720 pieces of sound data.

[0112] However, in the case of a typical audio buffer, there may be a case where the unit is 256, which is 2 to the power of 8. In this case, as shown in the audio data (320) collected in units of 256 in FIG. 4, audio data can be stored in units of 256. Accordingly, the control unit (100) can collect 320 audio data (330) through buffer handling using a median buffer, thereby generating one mel spectrum vector for each of the 320 audio data.

[0113] Meanwhile, another important factor to consider in real-time acoustic recognition is the potential for amplitude changes in the acoustic signal. Acoustic signals generated by living organisms, such as dogs or cats, can experience real-time changes in amplitude because the sound source can move. Furthermore, the acoustic recognition robot (10) according to an embodiment of the present invention may be configured as a mobile robot or may be configured on a mobile robot capable of movement. Therefore, it is difficult to incorporate all of these amplitude changes into model learning through data augmentation.

[0114] Accordingly, the control unit (100) can perform normalization on the mel spectrogram based on the maximum value of the mel spectrogram when a mel spectrogram is generated by collecting a preset number of mel spectrum vectors (S308).

[0115] For example, the Mel spectrogram Mel(f) can be defined as the result of the weighted average spectrum as shown in [Mathematical Formula 1] below.

[0116]

[0117] Here, S(f) represents the spectrogram and Γ(j) represents the map filter.

[0118] In this case, the above [Mathematical Formula 1] can be expressed as a proportional formula for the Mel spectrogram as shown in [Mathematical Formula 2] below.

[0119]

[0120] Here, x is the Mel spectrum vector of the input audio signal.

[0121] Therefore, when performing a single inference, if one feature, that is, the maximum value of the Mel Spectrogram among the frames constituting one Mel Spectrogram, is stored, the Mel Spectrogram can be normalized just before each inference. For example, when the current frame of the audio signal is the nth frame, the vector (m(n)) composed of the maximum value of each frame constituting the Mel Spectrogram, that is, the maximum value of the Mel Spectrogram vector, can be defined as in [Mathematical Formula 3] below.

[0122]

[0123] Here, m(n) is the maximum absolute value of the nth frame, and N is the number of frames of the audio signal used for one inference.

[0124] In this case, normalization of the Mel spectrogram can be performed as in [Mathematical Formula 4] below.

[0125]

[0126] Here, Mel(f) is the Mel spectrogram, max(m) is the Mel spectrogram value of the frame with the maximum absolute value of the Mel spectrogram among the frames of the audio signal used for a single inference, and ε is a divergence prevention value to prevent the value of the Mel spectrogram from diverging to infinity when the denominator becomes 0.

[0127] Meanwhile, when the Mel Spectrogram is normalized in step S208, the control unit (100) can convert the normalized Mel Spectrogram into feature information to be input into an acoustic recognition model using any one of a plurality of different feature information conversion methods. Then, an acoustic signal can be recognized from the feature information through any one of the plurality of acoustic recognition models learned according to the feature information conversion method (S310). In this case, the control unit (100) can determine the feature information conversion method and the acoustic recognition model to convert the feature information based on at least one of the current operating state of the acoustic recognition robot (10) and the surrounding situation of the acoustic recognition model.

[0128] Here, the operating state of the acoustic recognition robot (10) can be determined according to the operating mode of the acoustic recognition robot (10). In addition, the surrounding situation of the acoustic recognition robot (10) can be determined according to the size of noise detected in the surroundings of the acoustic recognition robot (10). In this case, the more detailed operation process of step S310, which determines the feature information conversion method and the acoustic recognition model for converting the feature information according to the operating mode of the acoustic recognition robot (10) and the size of the surrounding noise, and recognizes the received acoustic signal, will be examined in more detail with reference to FIG. 5 below.

[0129] Meanwhile, based on the result of the acoustic signal recognition in step S310, the control unit (100) can check whether a recognizable acoustic signal has been received (S312).

[0130] For example, if the acoustic recognition result of step S310 indicates that an acoustic signal is not recognized, the control unit (100) may determine that a recognizable acoustic signal has not been received. Then, the control unit (100) may proceed to step S300 again, detect and sample an acoustic signal, and perform the process from step S304 to step S310 again to generate and normalize a mel spectrogram from the sampled acoustic signal, i.e., acoustic data, and convert the normalized mel spectrogram into feature information to recognize the acoustic signal. In addition, it may be checked in step S312 whether a recognizable acoustic signal has been received.

[0131] Meanwhile, if, as a result of the check in step S312, a recognizable acoustic signal is received, i.e., if the received acoustic signal is recognized, the control unit (100) can determine the function of the acoustic recognition robot (10) according to the recognized acoustic signal. Then, the driving unit (140) can be controlled so that the acoustic recognition robot (10) is driven according to the determined function of the acoustic recognition robot (10) (S314).

[0132] And the control unit (100) controls the driving unit (140) to perform a function according to the recognized acoustic signal, and then proceeds to step S300 again to detect and sample the acoustic signal. And then, while performing the subsequent steps, it can proceed to step S312 to check whether a recognizable acoustic signal has been received. Accordingly, the acoustic recognition robot (10) according to an embodiment of the present invention can receive an acoustic signal and control the microphone (122), the normalization unit (160), the feature information conversion unit (170), and the artificial intelligence unit (180) to generate and normalize a mel spectrogram from the received acoustic signal not only before performing the function according to acoustic recognition but also while performing the function according to the recognized acoustic signal, as shown in FIG. 3. Therefore, the acoustic recognition robot (10) according to an embodiment of the present invention can maintain a state in which acoustic recognition is always performed.

[0133] Meanwhile, according to the above-described description, it has been mentioned that the control unit (100) of the acoustic recognition robot (10) according to the embodiment of the present invention can determine the acoustic recognition model and the feature information conversion method for converting the feature information according to the operation mode and the size of the surrounding noise.

[0134] FIG. 5 is a flowchart illustrating in more detail the operation process of recognizing an acoustic signal using one acoustic recognition model based on the operation mode and ambient noise in an acoustic recognition robot (10) according to an embodiment of the present invention.

[0135] Referring to FIG. 5, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention can select an acoustic recognition model suitable for the current noise environment of the acoustic recognition robot (10) when the Mel Spectrogram is normalized in step S308 of FIG. 3, and a conversion method for converting the normalized Mel Spectrogram into feature information for inputting into the acoustic recognition model suitable for the current noise environment.

[0136] For example, the above-described acoustic recognition robot (10) can transform the normalized Mel spectrogram in different ways depending on the noise environment around the robot. For example, the above-described acoustic recognition robot (10) can transform the normalized Mel spectrogram into feature information according to a conversion method according to per-channel energy normalization (PCEN) that performs adaptive gain control (AGC) for each channel according to Mel frequency, or a log Mel transformation method that scales the Mel spectrogram using a logarithmic scaling method that uses a logarithmic function.

[0137] Here, the PCEN conversion method can have a higher recognition rate in an extremely noisy environment, that is, an environment with low SNR and high RT value (hereinafter referred to as a high-noise environment), by suppressing the acoustic signal of a channel with a high noise content through adaptive gain control (AGC) for each channel. However, the PCEN conversion method has a problem in that, if the channel containing the noise contains an acoustic signal component, the included acoustic signal component may be suppressed together. Therefore, in the case of the PCEN conversion method, the recognition rate of the acoustic signal through the acoustic recognition model can be improved in an extremely noisy environment, that is, an environment with low SNR and high RT value (hereinafter referred to as a high-noise environment). However, in an environment with high SNR and low RT value, the amount of computation increases due to unnecessary gain control, and the recognition rate of the acoustic signal may actually decrease due to the suppressed acoustic signal component.

[0138] On the other hand, the log-Mel transform method converts the normalized Mel spectrogram into feature information by scaling it according to the log value corresponding to the Mel frequency, and thus can perform scaling for the entire frequency of the normalized Mel spectrogram. Therefore, in the case of the high-noise environment, not only the acoustic signal but also the noise is scaled, so the recognition rate of the acoustic signal in the high-noise environment may be reduced. However, conversely, in a low-noise environment with a high SNR and a low RT value, the acoustic signal included in the entire frequency can all be amplified, so that the loss of the acoustic signal components may be small. Therefore, in the case of a low-noise environment, when feature information converted by the log-Mel transform method rather than the PCEN transform method is input, the recognition rate of the acoustic signal in the acoustic recognition model may be higher.

[0139] Meanwhile, as described above, the acoustic recognition robot (10) may be equipped with different acoustic recognition models learned based on the feature information converted in the different ways. Here, the different acoustic recognition models have the same learning method and method of generating probability vectors for acoustic signal recognition based on the input feature information. However, the learning data used for learning may be different deep neural network (DNN) models.

[0140] For example, the above-described acoustic recognition robot (10) may be equipped with an acoustic recognition model (hereinafter referred to as a PCEN acoustic recognition model (181)) that is sufficiently trained to recognize acoustic signals according to feature information obtained by converting normalized mel spectrograms generated from learning data formed in a high-noise environment with low SNR and high RT values, i.e., learning data including noise above a certain level in the learning sound source, using the PCEN transformation method. In addition, the above-described acoustic recognition robot (10) may be equipped with an acoustic recognition model (hereinafter referred to as a log-Mel acoustic recognition model (182)) that is sufficiently trained to recognize acoustic signals according to feature information obtained by converting normalized mel spectrograms generated from learning data formed in a low-noise environment with high SNR and low RT values, i.e., learning data including noise below a certain level in the learning sound source, using the log-Mel transformation method.

[0141] In this case, the PCEN acoustic recognition model (181) may be a learning model specialized to recognize an acoustic signal according to feature information converted by the PCEN conversion method. Therefore, it may have a high recognition rate for feature information converted by the PCEN conversion method. In addition, the Log-Mel acoustic recognition model (182) may be a learning model specialized to recognize an acoustic signal according to feature information converted by the Log-Mel conversion method. Therefore, it may have a high recognition rate for feature information converted by the Log-Mel conversion method.

[0142] Accordingly, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention selects one of different feature information conversion methods and different acoustic recognition models according to the surrounding noise environment, and recognizes the acoustic signal through the selected feature information conversion method and acoustic recognition model, thereby further improving the recognition rate of the acoustic signal.

[0143] Meanwhile, in order to detect the noise environment around the acoustic recognition robot (10), the control unit (100) can first detect the operation mode of the acoustic recognition robot (10). Then, it can check whether the current operation mode of the acoustic recognition robot (10) is a preset operation mode (S502).

[0144] For example, the preset operation mode may be an operation mode preset by the user or during the manufacturing of the acoustic recognition robot (10), and may be an operation mode in which a loud noise of a certain level or greater is generated when the acoustic recognition robot (10) operates. For example, if the acoustic recognition robot (10) is a cleaning robot, the preset operation mode may be a turbo operation mode or an intensive cleaning mode in which the RPM of the vibration motor is driven at a certain speed or greater to generate a stronger suction force than in the general cleaning mode. In this case, if the acoustic recognition robot (10) is driven in the turbo operation mode or the intensive cleaning mode, a high-noise environment may be formed due to the noise generated by the robot itself, regardless of the noise around the robot.

[0145] As a result of the check in step S502, if the current operation mode of the acoustic recognition robot (10) is the preset mode, the control unit (100) can determine that the surroundings of the current acoustic recognition robot (10) are a high-noise environment. Accordingly, the control unit (100) can convert the normalized Mel Spectrogram in step S308 of FIG. 3 into PCEN conversion method, and convert it into feature information according to the PCEN conversion method, i.e., PCEN feature information (S506).

[0146] In the above step S506, when the normalized Mel Spectrogram is converted into PCEN feature information, the control unit (100) can input the PCEN feature information into the PCEN acoustic recognition model (181), which is an acoustic recognition model specialized in PCEN feature information among the provided acoustic recognition models. Then, acoustic recognition can be performed according to the input PCEN feature information through the PCEN acoustic recognition model (181) (S508). Then, the performed acoustic recognition result can be transmitted to the control unit (100).

[0147] Then, the control unit (100) can check whether a recognizable acoustic signal has been received in step S312 of FIG. 3 based on the received acoustic signal recognition result. Then, depending on the check result, the control unit (100) can proceed to step S314 and control the driving unit (140) to perform a function according to the recognized acoustic signal.

[0148] On the other hand, if the current operation mode of the acoustic recognition robot (10) is not the preset mode as a result of the check in step S502, the control unit (100) can detect the noise level around the acoustic recognition robot (10). Then, it can check whether the detected noise level is greater than the preset standard level (S504).

[0149] Here, the preset noise level may be a level greater than the noise generated by the acoustic recognition robot (10) when the acoustic recognition robot (10) is not operating in the preset operation mode, i.e., in the normal operation mode. For example, the control unit (100) may determine the noise level to be a value greater than the preset noise value corresponding to the current operation mode of the robot by a predefined threshold noise level when the acoustic recognition robot (10) is not operating in the preset operation mode.

[0150] Here, in the case where there are multiple normal operation modes, the control unit (100) can determine a reference size for measuring the noise environment by reflecting the critical noise size to each noise value set for each operation mode. Accordingly, if the noise values ​​are set differently for each operation mode, the reference size for measuring the noise environment around the acoustic recognition robot (10) may also vary depending on the current operation mode of the acoustic recognition robot (10).

[0151] If the noise level around the sound recognition robot (10) is greater than the reference level as a result of the check in step S504, the control unit (100) can proceed to step S506 to convert the normalized Mel Spectrogram into PCEN feature information. Then, the control unit can proceed to step S508 to perform sound recognition using the PCEN sound recognition model (181).

[0152] However, if the noise level around the acoustic recognition robot (10) is less than the reference level as a result of the check in step S504, that is, if the noise level around the acoustic recognition robot (10) is less than the reference level while the operation mode of the acoustic recognition robot (10) is not a preset operation mode, the control unit (100) can determine that the current area around the acoustic recognition robot (10) is a low-noise environment. Accordingly, the control unit (100) can convert the normalized Mel Spectrogram in step S308 of FIG. 3 into feature information according to the Log Mel transformation method, that is, Log Mel feature information (S510).

[0153] In the above step S510, when the normalized Mel spectrogram is converted into log Mel feature information, the control unit (100) can input the log Mel feature information into a log Mel acoustic recognition model (182), which is an acoustic recognition model specialized in log Mel feature information among the provided acoustic recognition models. Then, acoustic recognition can be performed according to the input log Mel feature information through the log Mel acoustic recognition model (182) (S512). Then, the performed acoustic recognition result can be transmitted to the control unit (100).

[0154] Then, the control unit (100) can check whether a recognizable acoustic signal has been received based on the result of the received acoustic signal recognition (step S312), and control the driving unit (140) to perform a function according to the recognized acoustic signal according to the check result (step S314).

[0155] That is, the acoustic recognition robot (10) according to an embodiment of the present invention can perform acoustic recognition according to the log-mel method when the robot's operating mode is a normal operating mode, that is, an operating mode in which little noise is generated, such as a general operating mode, and can perform acoustic recognition according to the PCEN method when the robot's operating mode is an operating mode in which loud noise is generated, such as a turbo operating mode or an intensive operating mode. In addition, when the ambient noise is low, the acoustic signal is recognized according to the log-mel acoustic recognition model learned by the log-mel feature information, thereby improving the acoustic signal recognition rate in a low-noise environment. On the other hand, when the ambient noise is high, the acoustic signal is recognized according to the PCEN acoustic recognition model learned by the PCEN feature information, thereby improving the acoustic signal recognition rate in a high-noise environment.

[0156] Therefore, the sound recognition robot (10) according to an embodiment of the present invention recognizes sound using a more specialized feature information conversion method and sound recognition model depending on the operating mode and noise environment, so that sound recognition can be continuously performed not only when the noise around the robot is small but also when the noise is large. In addition, sound recognition can be performed even when the operating state of the robot (10) is a turbo operation mode or intensive operation mode that causes large noise generation.

[0157] However, in environments with high noise levels like this, even when utilizing the PCEN method, the accuracy of acoustic recognition can deteriorate due to the noise. Therefore, some acoustic recognition robots restrict control using acoustic recognition, such as user voice commands, in environments where acoustic recognition is difficult, such as when operating in a mode that generates loud noises, such as turbo mode. This prevents inaccurate acoustic recognition.

[0158] In order to solve this problem, the sound recognition robot (10) according to an embodiment of the present invention, when operating in an operation mode that causes the generation of a large amount of noise, for example, a turbo operation mode, primarily recognizes an acoustic signal in the PCEN method (PCEN feature information - PCEN acoustic recognition model), and when the acoustic signal is recognized, changes the operation mode to a general operation mode that causes less noise and recognizes an acoustic signal in the log-mel method (log-mel feature information - log-mel acoustic recognition model).

[0159] In this case, the threshold for recognizing the feature information converted from the Mel spectrogram by the PCEN method as a specific acoustic signal may be a threshold lower than the threshold for recognizing the feature information converted from the Mel spectrogram by the Log Mel method as a specific acoustic signal. That is, in case of severe noise generation, the threshold is first lowered by the PCEN method to detect whether a recognizable acoustic signal is generated, and when a recognizable acoustic signal is detected, the operation mode is changed to perform acoustic recognition once again by the Log Mel method, thereby increasing the accuracy of acoustic recognition. Through this, the acoustic recognition robot (10) according to the embodiment of the present invention can control the robot using an acoustic signal even when it operates in an operating state in which a loud noise is generated or in a situation in which ambient noise is loud.

[0160] FIG. 6 is a flowchart illustrating an operation process in which an acoustic recognition robot (10) according to an embodiment of the present invention changes its operation mode based on the acoustic recognition result and re-recognizes an acoustic signal.

[0161] Referring to FIG. 6, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention can start operation according to a user's input, etc. In this case, the control unit (100) can control the driving unit (140) so that the acoustic recognition robot (10) operates according to a pre-designated operation mode (S600). Then, the control unit (100) can sample the acoustic signal detected through the microphone (122) at a preset time interval while operating according to the pre-designated operation mode (S602).

[0162] And the control unit (100) can collect sampled sound signals, i.e., sound data, through buffer handling and perform Fourier operation and Mel scale function on the collected sound data to convert them into Mel frequencies. And, it can extract a Mel spectrum vector from the sound data converted into Mel frequencies, collect the extracted Mel spectrum vectors to generate a Mel spectrogram, and perform normalization on the generated Mel spectrogram (S604). Here, the step S604 may be a process corresponding to steps S302 to S308 of the above-mentioned FIG. 3.

[0163] Then, the control unit (100) can convert the normalized Mel Spectrogram in step S604 into a feature information conversion method according to the current operation mode of the acoustic recognition robot (10) (S606). Then, the control unit (100) can input the converted feature information into one acoustic recognition model according to the current operation mode of the acoustic recognition robot (10) to recognize an acoustic signal (S608).

[0164] Therefore, if the current operating mode of the acoustic recognition robot (10) is an operating mode that generates noise above a certain level, such as a turbo operating mode or an intensive operating mode (hereinafter referred to as a turbo operating mode), the control unit (100) can convert the normalized Mel Spectrogram into the PCEN conversion method in step S606. Then, in step S608, the PCEN feature information converted according to the PCEN conversion method can be input into the PCEN acoustic recognition model trained according to the PCEN feature information to recognize the acoustic signal. Acoustic recognition performed according to the PCEN conversion method in this way will be referred to as acoustic recognition using the PCEN method hereinafter.

[0165] On the other hand, if the current operation mode of the acoustic recognition robot (10) is a normal operation mode, i.e., a general operation mode, rather than a turbo operation mode or an intensive operation mode, the control unit (100) can, in step S606, convert the normalized Mel spectrogram using the log-Mel transformation method. Then, in step S608, the log-Mel feature information converted according to the log-Mel transformation method can be input into the log-Mel acoustic recognition model learned according to the log-Mel feature information to recognize the acoustic signal. Acoustic recognition performed according to the log-Mel transformation method in this way will be referred to as acoustic recognition using the log-Mel method hereinafter.

[0166] Therefore, if the current operation mode is the turbo operation mode as described above, the control unit (100) can convert the normalized Mel Spectrogram into the PCEN conversion method in steps S606 and S608 and perform sound recognition through the PCEN sound recognition model. In addition, the control unit (100) can check whether a recognizable sound signal has been detected as a result of the sound recognition in step S608 (S610).

[0167] In the above step S610, the acoustic signals that the acoustic recognition robot (10) can recognize may be diverse. For example, the acoustic signals that can be recognized in the above step S610 may be not only human voices, but also sounds of preset objects or sounds of various pets such as dogs or cats. Alternatively, sounds that are generated by preset user gestures, such as clapping or finger snapping, or sounds generated when preset devices operate, such as phone ringtones, may also be acoustic signals that can be recognized in the above step S610.

[0168] And if the result of the check in step S610 is that a recognizable acoustic signal is not received, the control unit (100) can proceed to step S602 again to detect and sample the acoustic signal. And the process in steps S604 to S608 can be performed again to check whether a recognizable acoustic signal has been detected as a result of the acoustic recognition.

[0169] Meanwhile, if a recognizable acoustic signal is detected as a result of the check in step S610, the control unit (100) can check whether the current operation mode is the turbo operation mode (S612). If the current operation mode is the turbo operation mode, the operation mode can be changed to the preset general operation mode (S616).

[0170] Here, the above-described normal operation mode may be any one of the operation modes in which the noise generated during operation is below a certain level, other than the turbo operation mode.

[0171] When the operation mode is changed in the above step S616 and the turbo operation mode is released, the control unit (100) can change the feature information conversion method and the acoustic recognition model according to the changed operation mode to the feature information conversion method and the acoustic recognition model corresponding to the currently changed operation mode, i.e., the general operation mode. That is, the control unit (100) can control the feature information conversion unit (170) to change the method of converting the normalized Mel spectrogram to the log Mel transformation method, and control the artificial intelligence unit (180) to change the acoustic recognition model that performs acoustic recognition to the log Mel acoustic recognition model.

[0172] Then, the control unit (100) can proceed to step S602 again to detect and sample the acoustic signal. Then, it can proceed to step S604 to generate a normalized Mel Spectrogram according to the sampled acoustic data. Then, it can proceed to step S606 to convert the normalized Mel Spectrogram into a feature information conversion method according to the currently changed operation mode of the acoustic recognition robot (10). Then, it can proceed to step S608 to perform acoustic recognition using an acoustic recognition model according to the currently changed operation mode of the acoustic recognition robot (10) using the converted feature information.

[0173] In this case, since the operation mode of the sound recognition robot (10) has been changed to the general operation mode, the control unit (100) can convert the normalized Mel spectrogram into log-Mel feature information through the log-Mel transformation method in step S606, and input the log-Mel feature information into the log-Mel acoustic recognition model in step S608 to perform acoustic recognition. Then, the process proceeds to step S610 to check whether a recognizable acoustic signal has been detected as a result of the acoustic recognition of the log-Mel acoustic recognition model.

[0174] And if the control unit (100) recognizes an acoustic signal as a result of the acoustic recognition of the log-mel acoustic recognition model, it can proceed to step S612 to check whether the current operation mode is the turbo operation mode. In this case, since the current operation mode has been changed to the general operation mode by step S616, the control unit (100) can determine a function according to the recognized acoustic signal. And it can control the driving unit (140) to perform the determined function (S616).

[0175] In this way, when the sound recognition robot (10) according to an embodiment of the present invention operates in turbo operation mode, the control unit (100) can first perform sound recognition in the PCEN method suitable for sound recognition in a high-noise environment. Therefore, when the sound recognition robot (10) is operated according to the turbo operation mode, the control unit (100) can check whether various sound signals such as a human voice, the sound of a preset object, the sound of a pet such as a dog or cat, applause, and a telephone ringtone are detected in the PCEN method. In addition, when an sound signal is detected, the operation mode can be changed to a general operation mode that creates a low-noise environment, and sound recognition can be performed again in the log-mel method suitable for sound recognition in the low-noise environment.

[0176] And if the sound recognition result performed again in the log-mel method is a case where a human voice, a preset object sound, a pet sound such as a dog or cat, a clap, or a telephone ringtone is recognized, the control unit (100) can perform a function according to a scenario corresponding to the recognized sound signal. For example, if a human voice is received, a function according to the received voice command can be performed. Alternatively, if the sound of a pet such as a dog or cat is detected, the control unit (100) can move in the direction where the sound was detected and perform a designated function (e.g., user voice output) or capture an image in the direction where the sound was detected and transmit it to a designated terminal (e.g., user terminal). Alternatively, if an sound signal such as a clap or a telephone ringtone is detected, the driving unit (140) can be temporarily stopped so as not to generate noise, or can be operated in an operation mode that generates less noise, for example, a silence operation mode.

[0177] Meanwhile, as described above, when the acoustic recognition robot (10) according to an embodiment of the present invention primarily recognizes an acoustic signal while operating in a turbo operation mode that generates a large amount of noise, it can change its operation mode to a normal operation mode that generates a small amount of noise and secondarily recognize the acoustic signal. In addition, it can perform a function corresponding to the second acoustic recognition result.

[0178] Accordingly, the first acoustic signal recognition may be for detecting whether a recognizable acoustic signal is detected, and the second acoustic signal recognition may be for recognizing an acoustic signal to perform an actual function. Accordingly, the control unit (100) may set different threshold values ​​for the probability values ​​for recognizing feature information as a specific acoustic signal for the first acoustic signal recognition and the second acoustic signal recognition.

[0179] In this case, the threshold value of the probability value used for the recognition of the first acoustic signal may be lower than the threshold value of the probability value used for the recognition of the second acoustic signal. In this case, the accuracy of the recognition of the first acoustic signal may be reduced, but it is possible to detect whether an acoustic signal that can be recognized more easily and quickly has been received.

[0180] In contrast, the threshold value of the probability value used for secondary acoustic signal recognition may be higher than the threshold value of the probability value used for primary acoustic signal recognition. Therefore, the acoustic signal corresponding to the feature information resulting from secondary acoustic signal recognition can be more accurately identified. Furthermore, by performing a function based on the secondary acoustic recognition result, the recognition rate of the acoustic signal can be further improved.

[0181] Meanwhile, the acoustic recognition robot (10) according to an embodiment of the present invention can change its operating mode when a preset acoustic signal is recognized. In this case, when the operating mode is changed, the method of recognizing the acoustic signal can also be changed.

[0182] FIG. 7 is a flowchart illustrating an operation process in which an acoustic recognition robot (10) according to an embodiment of the present invention changes an operation mode based on an acoustic recognition result.

[0183] Referring to FIG. 7, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention can start operation according to a user's input, etc. Then, the control unit (100) can control the driving unit (140) to operate the acoustic recognition robot (10) according to a pre-designated operation mode (S700). In addition, the control unit (100) can sample the acoustic signal detected through the microphone (122) at a preset time interval while operating according to the pre-designated operation mode (S702).

[0184] And the control unit (100) can collect sampled sound signals, i.e., sound data, through buffer handling and perform Fourier operation and Mel scale function on the collected sound data to convert them into Mel frequencies. And, it can extract a Mel spectrum vector from the sound data converted into Mel frequencies, collect the extracted Mel spectrum vectors to generate a Mel spectrogram, and perform normalization on the generated Mel spectrogram (S704). Here, the step S704 may be a process corresponding to steps S302 to S308 of the above-mentioned FIG. 3.

[0185] Then, the control unit (100) can convert the normalized Mel Spectrogram in step S704 into feature information conversion method according to the current operation mode of the acoustic recognition robot (10) (S706). Therefore, if the current operation mode of the acoustic recognition robot (10) is the turbo operation mode, the control unit (100) can convert the normalized Mel Spectrogram into PCEN feature information in step S706. However, if the current operation mode of the acoustic recognition robot (10) is not the turbo operation mode, the control unit (100) can convert the normalized Mel Spectrogram into log Mel feature information in step S706.

[0186] And the control unit (100) can input the converted feature information into one of the sound recognition models according to the current operation mode of the sound recognition robot (10) to recognize the sound signal (S708). Therefore, if the current operation mode of the sound recognition robot (10) is the turbo operation mode, the control unit (100) can input the PCEN feature information into the PCEN sound recognition model to perform sound recognition in step S708. On the other hand, if the current operation mode of the sound recognition robot (10) is not the turbo operation mode, the control unit (100) can input the log-mel feature information into the log-mel sound recognition model to perform sound recognition in step S706.

[0187] Then, the control unit (100) can check whether a preset sound signal has been recognized a certain number of times or more based on the sound recognition result of step S708. Here, the preset sound signal may be an sound signal generated through a preset user gesture, such as clapping or finger snapping. Or, it may be an sound signal generated by the user, such as a whistle. Or, it may be an sound signal generated when a preset device is operated, such as a phone ringtone.

[0188] As a result of the recognition in step S710, if a preset acoustic signal is recognized, the control unit (100) can check whether the acoustic signal has been recognized continuously for a certain period of time or longer (S710). Therefore, if as a result of the recognition in step S708, the number of times or the time for which the acoustic signal has been recognized is insufficient, the control unit (100) can proceed to step S702 again to sample the acoustic signal. Then, the control unit proceeds to step S704 to generate and normalize a mel spectrogram from the sampled acoustic data, and through steps S706 and S708, the acoustic signal can be recognized again using an acoustic recognition method according to the current operating state of the acoustic recognition robot (10).

[0189] Meanwhile, if the recognition result of the step S710 is that the preset acoustic signal has been continuously recognized for a certain period of time or longer, the control unit (100) can change the operation mode of the acoustic recognition robot (10) (S712). Accordingly, if the acoustic recognition robot (10) is operating in turbo operation mode, the operation mode of the acoustic recognition robot (10) can be changed to the general operation mode through the continuous acoustic signal for a certain period of time or longer or the preset number of times. When the operation mode is changed, the control unit (100) can change the feature information conversion method according to the changed operation mode (S714) and change the acoustic recognition model (S716).

[0190] Therefore, when the operation mode is changed from turbo operation mode to normal operation mode, the control unit (100) of the acoustic recognition robot (10) can change the feature information conversion method from the PCEN conversion method to the log-MEL conversion method. In addition, the acoustic recognition model can be changed from the PCEN acoustic recognition model to the log-MEL acoustic recognition model. Then, the control unit (100) can proceed to step S702 again to sample the acoustic signal and perform the subsequent steps.

[0191] Meanwhile, the above description describes an example of changing from turbo operation mode to normal operation mode based on whether the preset sound signal is detected a certain number of times or for a certain period of time, but it is of course also possible to change the operation mode from normal operation mode to turbo operation mode.

[0192] In this way, when the turbo operation mode is changed in step S712, the control unit (100) can convert the feature information conversion method from the log-MEL conversion method to the PCEN conversion method in step S714, and can convert the sound recognition model from the log-MEL sound recognition model to the PCEN sound recognition model in step S716. Then, the control unit (100) can proceed to step S702 again to convert the normalized Mel spectrogram generated from the sampling data of the sound signal detected from the microphone (122) into feature information according to the PCEN method and perform sound recognition.

[0193] According to the above-described operation process, the acoustic recognition robot (10) according to the embodiment of the present invention can, when a preset activation word is recognized while operating in turbo operation mode, switch to the normal operation mode and confirm the user's activation word. Then, a function according to the confirmed user activation word can be performed.

[0194] FIG. 8 is a flowchart illustrating an operation process in which an acoustic recognition robot (10) according to an embodiment of the present invention recognizes an acoustic signal by changing the operation mode depending on whether a preset activation word is recognized.

[0195] Referring to FIG. 8, the control unit (100) of the acoustic recognition robot (10) can control the driving unit (140) to operate in turbo operation mode according to user input, etc. (S800). In addition, the acoustic signal detected through the microphone (122) can be sampled at preset time intervals (S802).

[0196] And the control unit (100) can collect sampled sound signals, i.e., sound data, through buffer handling and perform Fourier operation and Mel scale function on the collected sound data to convert them into Mel frequencies. And, it can extract a Mel spectrum vector from the sound data converted into Mel frequencies, collect the extracted Mel spectrum vectors to generate a Mel spectrogram, and perform normalization on the generated Mel spectrogram (S804). Here, the step S804 may be a process corresponding to steps S302 to S308 of the above-mentioned FIG. 3.

[0197] Then, the control unit (100) can convert the normalized Mel Spectrogram in the step S804 into PCEN feature information according to the turbo operation mode, which is the current operation mode of the sound recognition robot (10) (S806). Then, the control unit (100) can input the PCEN feature information into the PCEN sound recognition model to perform sound recognition (S808). Then, based on the sound signal recognition result according to the PCEN sound recognition model, it can be checked whether an sound signal corresponding to a preset operation has been received (S810).

[0198] If the result of the check in step S810 indicates that the acoustic signal corresponding to the preset trigger word is not recognized, the control unit (100) may proceed to step S802 to detect and sample the acoustic signal. Then, the process from step S802 to step S808 may be performed to perform acoustic signal recognition. Then, the control unit may proceed to step S810 to check whether the acoustic signal corresponding to the preset trigger word has been recognized.

[0199] On the other hand, if the result of the check in step S810 is that an acoustic signal corresponding to a preset trigger word is recognized, the control unit (100) can change the operation mode to an operation mode that generates less noise than the turbo operation mode, for example, a general operation mode in which the RPM (Revolutions Per Minute) of the vacuum motor is lower than the turbo operation mode (S812). Here, when the operation mode of the acoustic recognition robot (10) is changed, the control unit (100) can change the feature information conversion method for converting the normalized Mel Spectrogram into feature information and the acoustic recognition model used for recognizing the acoustic signal according to the changed operation mode. Therefore, the control unit (100) can select the log Mel conversion method and the log Mel acoustic recognition model as the feature information conversion method and the acoustic recognition model for recognizing the acoustic signal, respectively.

[0200] And in order to clearly understand the user's intention, the sound output unit (152) can be controlled to utter a message such as 'Yes? Did you call?' to confirm the recognized sound signal (S814). And the control unit (100) can sample the sound signal detected through the microphone (122) at a preset time interval (S816), generate a Mel Spectrogram for the sampled sound data, and perform normalization on the generated Mel Spectrogram (S818). Here, the step S818 may be a process corresponding to steps S302 to S308 of the above-mentioned FIG. 3.

[0201] Then, the control unit (100) can convert the normalized Mel spectrogram into a feature information conversion method according to the currently changed operation mode of the acoustic recognition robot (10) (S820). Therefore, the control unit (100) can generate log Mel feature information by converting the normalized Mel spectrogram into a log Mel transformation method. Then, the control unit (100) can input the log Mel feature information into an acoustic recognition model according to the currently changed operation mode of the acoustic recognition robot (10), that is, a log Mel acoustic recognition model, to perform acoustic recognition. Then, it can be checked whether a preset user's activation word has been detected from the acoustic recognition result according to the log Mel acoustic recognition model (S822). Then, if the user's activation word is recognized again as a result of the activation word recognition in step S822, the control unit (100) can control the driving unit (140) to perform a function according to the recognized user activation word.

[0202] Therefore, in the step S810, when the user utters the trigger word again in response to the message uttered based on the recognition of the trigger word, the control unit (100) can detect the trigger word uttered again by changing to the normal operation mode, which generates less noise than the turbo operation mode. In this case, by performing acoustic recognition using the log-mel method, which is more effective than the PCEN method, in a low-noise mode such as the normal operation mode, the trigger word intended by the user can be recognized more accurately.

[0203] Meanwhile, the acoustic recognition robot (10) according to an embodiment of the present invention may perform different functions depending on the operating state of the preset device when an acoustic signal related to the operation of the preset device is received.

[0204] FIG. 9 is a flowchart illustrating an operation process in which an acoustic recognition robot (10) according to an embodiment of the present invention determines the operating state of a preset device from the sound of the preset device and performs different operations according to the determined operating state.

[0205] Referring to FIG. 9, the control unit (100) of the acoustic recognition robot (10) according to an embodiment of the present invention can control the driving unit (140) to operate in a specific operation mode according to a user's input (S900). In addition, the control unit (100) can sample the acoustic signal detected through the microphone (122) at preset time intervals (S902).

[0206] And the control unit (100) can collect sampled sound signals, i.e., sound data, through buffer handling and perform Fourier operation and Mel scale function on the collected sound data to convert them into Mel frequencies. And, it can extract a Mel spectrum vector from the sound data converted into Mel frequencies, collect the extracted Mel spectrum vectors to generate a Mel spectrogram, and perform normalization on the generated Mel spectrogram (S904). Here, the step S904 may be a process corresponding to steps S302 to S308 of the above-mentioned FIG. 3.

[0207] Then, the control unit (100) can generate feature information by converting the normalized Mel Spectrogram in step S904 into a feature information conversion method according to the current operation mode of the acoustic recognition robot (10). Then, the generated feature information can be input into an acoustic recognition model according to the current operation mode of the acoustic recognition robot to perform acoustic signal recognition (S906).

[0208] In this case, if the current operation mode of the acoustic recognition robot (10) is a preset operation mode, for example, a turbo operation mode, the control unit (100) can convert the normalized Mel spectrogram into PCEN feature information in step S906. Then, acoustic recognition can be performed through the PCEN acoustic recognition model. However, if the current operation mode of the acoustic recognition robot (10) is not the preset operation mode, the control unit (100) can convert the normalized Mel spectrogram into log Mel feature information in step S906. Then, the log Mel feature information can be input into the log Mel acoustic recognition model to perform acoustic recognition.

[0209] And the control unit (100) can check whether the sound generated when the preset device is operated, i.e., the driving sound, has been recognized as a result of the sound recognition in step S906 (S908).

[0210] For example, the above-described preset device may be a hair dryer. And the driving sound may be a noise generated when the hair dryer is operated. In this case, the control unit (100) checks whether the driving sound of the hair dryer is detected in step S908, and if the driving sound of the hair dryer is not detected, the control unit (100) may proceed to step S902 again. And the sound recognition process from step S902 to step S906 may be performed again.

[0211] On the other hand, if the recognized sound signal as a result of the check in step S908 is the operating sound of a hair dryer, the control unit (100) can control the driving unit (140) to move the hair dryer, i.e., the preset device, to a registered location (S910). Here, the location of the preset device can be designated in advance by a user or the like on a map of an area where the sound recognition robot (10) operates. In this case, if the sound recognition robot (10) is a robot vacuum cleaner, the map may be a map of a cleaning area searched according to the operation of the robot vacuum cleaner.

[0212] When moving to a location where the operating sound of the preset device is detected through the above step S910, the control unit (100) can perform a first function corresponding to the state in which the preset device is operating (S912). For example, if the preset device is a hair dryer and the sound recognition robot (10) is a robot vacuum cleaner, the first function may be a function of cleaning the area around the location where the hair dryer is registered. For example, the area around the location where the hair dryer is registered may be the room where the hair dryer is registered.

[0213] And in a state where the above first function is performed, the control unit (100) can perform acoustic signal detection and sampling for acoustic signal recognition (S914).

[0214] And the control unit (100) can recognize the acoustic signal detected in the state of performing the first function. To this end, the control unit (100) generates a normalized Mel Spectrogram for the acoustic data sampled in the step S914 in the state of performing the first function (S916), generates feature information in a manner according to the current operation mode of the acoustic recognition robot (10), and recognizes the acoustic signal from the generated feature information (S918). In this case, if the operation mode of the acoustic recognition robot (10) is a preset operation mode, the acoustic signal can be recognized in the PCEN manner, and if it is not the preset operation mode, the acoustic signal can be recognized in the Log Mel manner.

[0215] Meanwhile, the control unit (100), while performing the first function, can check whether the operating sound of the preset device has been lost as a result of the sound recognition in step S918 (S920). That is, the control unit (100) can check whether the operating sound of the preset device is no longer detected as a result of the sound recognition in step S918.

[0216] And if the result of the check in step S920 is that the driving sound of the preset device is continuously recognized, the control unit (100) can proceed to step S914 again, and perform the process of steps S914 to S918 again, sampling sound data while performing the first function and recognizing a sound signal from the sampled sound data. And then proceed to step S920 again to check whether the driving sound of the preset device has been lost. And if the driving sound of the preset device has not been lost, that is, if the driving sound of the preset device is recognized, the sound signal can be continuously recognized while performing the first function.

[0217] Meanwhile, if the operating sound of the preset device is lost as a result of the check in step S920, i.e., if the operating sound of the preset device is no longer recognized, the control unit (100) determines that the operation of the preset device has ended and can perform a second function corresponding to the end of the operation of the preset device (S922).

[0218] For example, if the above-described preset device is a hair dryer and the sound recognition robot (10) is a robot vacuum cleaner, the second function may be a function of cleaning the location where the hair dryer is registered, rather than the area around the hair dryer. Accordingly, the area where cleaning is performed according to the second function may be narrower than the area where cleaning is performed according to the first function.

[0219] When the second function is performed, the control unit (100) can check whether the second function has been completed (S924). For example, if the second function is a function of cleaning the location where the hair dryer is registered, the control unit (100) can check whether the sound recognition robot (10), i.e., the robot cleaner, has cleaned the location where the hair dryer is registered a specified number of times. If the check result of step S924 indicates that the second function has been completed, the control unit (100) can proceed to step S900 and continue to perform the operation that was performed before the operating sound of the preset device was detected.

[0220] Meanwhile, in the above description, the present invention is described as a robot that executes a specific function corresponding to a recognized acoustic signal. However, the present invention may also be an acoustic signal recognition device that detects sound, extracts feature information from the detected sound, and recognizes a meaningful acoustic signal included in the sound based on the extracted feature information.

[0221] In this case, the acoustic signal recognition device may be configured to be detachably attached to the robot body, including the driving unit (140). Furthermore, when mounted on the robot, the device may be configured to transmit an acoustic signal recognized from the detected sound to the robot's control unit. The robot's control unit may then execute a function corresponding to the acoustic signal received from the acoustic recognition device according to an embodiment of the present invention.

[0222] The present invention described above can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc., and also includes media implemented in the form of carrier waves (e.g., transmission via the Internet). In addition, the computer may include a control unit (100) of the acoustic recognition robot (10). Therefore, the detailed description above should not be construed as limiting in all respects, but should be considered as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are intended to be included in the scope of the present invention.

Claims

1. In sound recognition robots, A microphone that detects sound; A normalization unit that normalizes the Mel spectrogram generated from the above-detected sound; An artificial intelligence unit including a plurality of sound recognition models, each of which is trained with learning data corresponding to different noise environments, and each of which has probability values ​​for recognizing the detected sound as a specific sound signal; A feature information conversion unit that converts the normalized Mel Spectrogram into different conversion methods corresponding to each of the plurality of sound recognition models, thereby generating feature information corresponding to each of the plurality of sound recognition models; and An acoustic recognition robot characterized in that it comprises a control unit for selecting any one of the plurality of acoustic recognition models, generating feature information from the normalized Mel Spectrogram in a conversion method corresponding to the selected acoustic recognition model, and controlling the feature information conversion unit and the artificial intelligence unit to recognize the detected sound as a specific acoustic signal through the selected one of the acoustic recognition models, and controlling the operation of the acoustic recognition robot according to the recognition result.

2. In the first paragraph, the feature information conversion unit, An acoustic recognition robot characterized in that the normalized Mel spectrogram is converted into PCEN (Per Channel Energy Normalization) conversion method to generate PCEN feature information in which adaptive gain control (AGC) is performed for each channel according to Mel frequency, or converted into Log Mel conversion method to generate Log Mel feature information scaled by a logarithmic exponent according to Mel frequency.

3. In the second paragraph, the artificial intelligence unit, It includes a PCEN sound recognition model that recognizes the detected sound as a specific sound signal based on the PCEN feature information, and a log-mel sound recognition model that recognizes the detected sound as a specific sound signal based on the log-mel feature information. The above PCEN sound recognition model is, It is a deep neural network (DNN) model in which the weights of hidden layers are learned through learning data converted into the above PCEN feature information. The above log-mel sound recognition model is, An acoustic recognition robot characterized in that the weights of the hidden layers are learned through learning data converted into the above log-mel feature information, and the deep neural network model.

4. In paragraph 3, The learning data converted into the above PCEN feature information are: These are learning data formed in a high-noise environment where the SNR (Signal to Noise Ratio) is below a preset level and the RT (Reverberation Time) is above a certain value. The learning data converted into the above log-mel feature information are An acoustic recognition robot characterized in that the learning data is formed in a low-noise environment in which an SNR exceeding the above-mentioned preset level and an RT below the above-mentioned predetermined value exist.

5. In the third paragraph, the control unit, An acoustic recognition robot characterized in that it selects one of the plurality of acoustic recognition models based on at least one of the operation mode of the acoustic recognition robot and the noise environment of the acoustic recognition robot, and controls the feature information conversion unit to convert the normalized Mel Spectrogram into feature information according to the selected acoustic recognition model.

6. In paragraph 5, the control unit, When the operation mode of the robot is a preset operation mode, the feature information conversion unit is controlled to convert the normalized Mel spectrogram to generate the PCEN feature information, and the artificial intelligence unit is controlled to allow the PCEN sound recognition model to recognize an acoustic signal based on the PCEN feature information. An acoustic recognition robot characterized in that, when the operation mode of the robot is not a preset operation mode, the feature information conversion unit is controlled to convert the normalized Mel spectrogram to generate the log Mel feature information, and the artificial intelligence unit is controlled so that the log Mel acoustic recognition model recognizes an acoustic signal based on the log Mel feature information.

7. In the 6th paragraph, the preset operation mode is: The above sound recognition robot is in an operating mode that generates noise above a preset level, An acoustic recognition robot characterized in that, when the above acoustic recognition robot is a robot vacuum cleaner, the RPM (Revolutions Per Minute) of the vacuum motor providing vacuum suction power is in a turbo operation mode or intensive cleaning mode at a certain speed or higher.

8. In paragraph 7, the control unit, When the above-mentioned acoustic recognition robot operates according to the above-mentioned preset operation mode, the normalized Mel Spectrogram is converted according to the first conversion method to generate feature information, and the feature information is input into the first acoustic recognition model among the plurality of acoustic recognition models to perform primary acoustic signal recognition. An acoustic recognition robot characterized in that, when an acoustic signal corresponding to the sound is recognized as a result of the first acoustic recognition, the operation mode of the acoustic recognition robot is changed, and the normalized Mel Spectrogram is converted into a second conversion method different from the first conversion method according to the changed operation mode to generate feature information, and secondary acoustic signal recognition is performed through a second acoustic recognition model different from the first acoustic recognition model.

9. In paragraph 8, If the above-mentioned sound recognition robot is a robot vacuum cleaner, the above-mentioned preset operation mode is: The above turbo operation mode or the above intensive cleaning mode, The above changed operation mode is, An acoustic recognition robot characterized in that the operating mode is one in which the noise resulting from the operation of the acoustic recognition robot is below the preset level among operating modes other than the turbo operating mode or the intensive operating mode.

10. In paragraph 8, the control unit, Control the sound recognition robot to be driven according to the result of the above secondary sound signal recognition, An acoustic recognition robot characterized in that, when the recognized acoustic signal is any one of a preset pet sound, a preset user gesture sound, and a driving sound of a specific registered device, the acoustic recognition robot is controlled to perform a preset action according to the recognized acoustic signal.

11. In paragraph 10, The sound according to the user's gesture is: Includes at least one of the following sounds: clapping, finger snapping, or whistling; The operating sound of the above specific device is, An acoustic recognition robot characterized in that it comprises at least one of a telephone ringtone registered by a user and an operating noise of a hair dryer, which is generated when a preset device is operated.

12. In the 11th paragraph, the control unit, When the driving sound of the specific device is detected, the sound recognition robot is controlled to perform a first function related to the driving of the specific device while the driving sound of the specific device is detected through the first or second sound recognition, An acoustic recognition robot characterized in that, when the driving sound of the specific device is lost as a result of acoustic recognition performed while performing the first function, the acoustic recognition robot is controlled to perform a second function related to the termination of the operation of the specific device.

13. In paragraph 8, The above sound recognition robot, Further comprising an audio output section for outputting an audio signal, The above control unit, If the recognized sound signal is a preset trigger word, the sound output unit is controlled to change the operation mode and output a message requesting the user's confirmation. An acoustic recognition robot characterized in that it performs acoustic recognition on sound detected in response to the spoken message using an acoustic recognition model and conversion method according to the changed operation mode.

14. In paragraph 8, In the above first acoustic signal recognition, the first threshold value of the probability value for recognizing the sound detected by the microphone as a specific acoustic signal is An acoustic recognition robot characterized in that the probability value for recognizing the sound detected by the microphone as a specific acoustic signal in the above secondary acoustic signal recognition is lower than the second threshold value.

15. In paragraph 1, The above learning data is, The learning sound source to be learned is formed so that it is placed after the noise of a preset size. The noise of the above preset size is An acoustic recognition robot characterized in that the noise is filled in front of the learning sound source in a zero padding manner equal to the difference between the length of the preset learning data and the length of the learning sound source.

16. In a control method of an acoustic recognition robot, A step of controlling the operation of the acoustic recognition robot so that it is operated in an operation mode according to a user's input; A step of generating a normalized Mel spectrogram from sound detected from a microphone of the above sound recognition robot; A step of selecting one acoustic recognition model according to the operation mode of the acoustic recognition robot among a plurality of acoustic recognition models trained with learning data corresponding to different noise environments, each having probability values ​​for recognizing the detected sound as a specific acoustic signal; A step of generating feature information by converting the normalized mel spectrogram using one of the different conversion methods corresponding to the selected acoustic recognition model among the different conversion methods corresponding to the plurality of acoustic recognition models; A step of inputting the generated feature information into the selected acoustic recognition model to recognize the detected sound as a specific acoustic signal; A method for controlling an acoustic recognition robot, characterized in that it comprises a step of controlling the acoustic recognition robot to perform a function corresponding to the recognized acoustic signal according to the result of the acoustic signal recognition.

17. In paragraph 16, The step of selecting one of the above acoustic recognition models is: A control method for an acoustic recognition robot, characterized in that the method comprises a step of selecting one acoustic recognition model among a plurality of acoustic recognition models depending on whether the operation mode of the acoustic recognition robot is a preset operation mode and whether the noise level around the acoustic recognition robot is greater than or equal to a preset level.

18. In the 17th paragraph, the preset operation mode is: The above sound recognition robot is in an operating mode that generates noise above a preset level, A control method for an acoustic recognition robot characterized in that, when the acoustic recognition robot is a robot vacuum cleaner, the RPM of the vacuum motor providing vacuum suction power is in a turbo operation mode or intensive cleaning mode at a certain speed or higher.

19. In paragraph 18, The step of selecting one of the above acoustic recognition models is: When the operation mode of the above-mentioned acoustic recognition robot is the above-mentioned preset operation mode or the noise level around the above-mentioned acoustic recognition robot is greater than the preset level, a PCEN acoustic recognition model that recognizes an acoustic signal from feature information converted from the normalized Mel Spectrogram using the PCEN (Per Channel Energy Normalization) conversion method is selected from among the plurality of acoustic recognition models, A control method for an acoustic recognition robot, characterized in that, when the operation mode of the acoustic recognition robot is not the preset operation mode and the noise level around the acoustic recognition robot is less than the preset level, a log-mel acoustic recognition model that recognizes an acoustic signal from feature information obtained by converting the normalized Mel spectrogram using a Log Mel transformation method among the plurality of acoustic recognition models is selected.

20. In paragraph 18, The step of recognizing the above-detected sound as a specific sound signal is: When the operation mode of the acoustic recognition robot is the preset operation mode, if the detected sound is recognized as a specific acoustic signal through the selected acoustic recognition model, a step of changing the operation mode of the acoustic recognition robot to an operation mode other than the preset operation mode; A step of changing the selected acoustic recognition model according to the changed operation mode; A step of generating feature information in a different manner by transforming the normalized Mel Spectrogram in a different transformation method corresponding to the changed acoustic recognition model; and, A control method for an acoustic recognition robot, characterized in that it further includes a step of inputting the characteristic information of the different method into the changed acoustic recognition model to recognize the detected sound as a specific acoustic signal.

Images