Electronic device and method for recognizing sound

By performing random FFT and jump-length transformation on the input sound, the problem of low sound recognition accuracy in the prior art is solved, accurate recognition and distinction of different types of sounds is achieved, and the neural network model is improved. Recognition accuracy.

CN118984993APending Publication Date: 2024-11-19SAMSUNG ELECTRONICS CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202380027685.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-09-27
Filing Date
2023-01-12
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The prior art is difficult to effectively identify and distinguish different types of sounds, especially in complex environments, and existing methods cannot generate multiple feature maps with different features from the same sound data to train neural network models.

Method used

By performing a fast Fourier transform of random FFT numbers and random jump lengths on the input sound, a 2D feature map with time axis and frequency axis is generated, and using these feature maps as training data to train a neural network model that recognizes sounds, generating Multiple feature maps containing different features to improve recognition accuracy.

Benefits of technology

It realizes accurate identification and distinction of different types of sounds, improves the recognition accuracy of neural network models, can generate a large amount of training data from a small amount of sound data, and enhances the performance of sound recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984993A_ABST
    Figure CN118984993A_ABST
Patent Text Reader

Abstract

An example of a sound recognition method may include the step of sampling an input sound based on a pre-configured sampling rate. The sound recognition method may include performing an FFT transform on a sampled input sound based on at least one of a random FFT number and a random hop length, and generating a two-dimensional feature map including a time axis and a frequency axis from the FFT-transformed input sound. The sound recognition method may include training a neural network model for recognizing a sound by using a plurality of generated two-dimensional feature maps including a first two-dimensional feature map and an nth two-dimensional feature map as training data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for recognizing a sound type from an input sound. Background Art

[0002] The electronic device may process an electronic signal and use the processed electronic signal to perform various functions. For example, the electronic device may receive an analog sound signal and convert the analog sound signal into a digital signal. The electronic device may process a digital signal and send the processed digital signal to an external device, or convert the processed digital signal into an analog signal and output the analog signal. In addition, the electronic device may identify the type of sound from the sound signal converted into a digital signal. When the sound is speech, the electronic device may identify the meaning of the speech and generate text based on the meaning of the recognized speech. The electronic device may execute a control command based on the meaning of the recognized speech, or output the generated text in a visual or acoustic manner. Summary of the invention

[0003] Technical Solution

[0004] A sound recognition method according to an embodiment of the present disclosure may include sampling an input sound based on a preset sampling rate. The sound recognition method may include performing a fast Fourier transform (FFT) on the sampled input sound based on at least one of a random FFT number and a random hop length, and generating a two-dimensional (2D) feature map having a time axis and a frequency axis from the sampled input sound on which the FFT is performed. Generating a 2D feature map may include transforming the sampled input sound into a first FFT data based on at least one of a first FFT number in the random FFT number and a first hop length in the random hop length, and generating a first 2D feature map including a first feature from the first FFT data. Generating a 2D feature map may include transforming the sampled input sound into an nth FFT data based on at least one of an nth FFT number in the random FFT number and an nth hop length in the random hop length, and generating an nth 2D feature map including an nth feature from the nth FFT data. The sound recognition method may include using a plurality of 2D feature maps including a first 2D feature map and an nth 2D feature map as training data to train a neural network model for recognizing sound.

[0005] An electronic device according to an embodiment of the present disclosure may include an input interface and at least one processor. The at least one processor may be configured to sample a sound input through the input interface based on a preset sampling rate. The at least one processor may be configured to perform FFT on the sampled input sound based on at least one of a random FFT number or a random hop length, and generate a 2D feature map having a time axis and a frequency axis from the sampled input sound on which FFT is performed. The at least one processor may be configured to transform the sampled input sound into a first FFT data based on at least one of a first FFT number in a random FFT number and a first hop length in a random hop length, and generate a first 2D feature map including a first feature from the first FFT data. The at least one processor may be configured to transform the sampled input sound into an nth FFT data based on at least one of an nth FFT number in a random FFT number and an nth hop length in a random hop length, and generate an nth 2D feature map including an nth feature from the nth FFT data. The at least one processor may be configured to train a neural network model for recognizing sound using a plurality of 2D feature maps including a first 2D feature map and an nth 2D feature map as training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram for describing a configuration of an example electronic device according to various embodiments;

[0007] Figure 2 is a detailed block diagram for describing a configuration of an example electronic device according to various embodiments;

[0008] Figure 3 is a diagram for describing an example process of sampling an input sound according to various embodiments;

[0009] Figure 4 , Figure 5 and Figure 6 is a diagram for describing an example process of generating a two-dimensional (2D) feature map according to various embodiments;

[0010] Figure 7a , Figure 7b , Figure 7c , Figure 7d , Figure 7e , Figure 7f and Figure 7g is a diagram for describing example masking patterns according to various embodiments;

[0011] Figure 8 is a diagram for describing an example process of recognizing a sound type based on a 2D feature map according to various embodiments;

[0012] Fig. 9is a diagram for describing example personalized voice recognition according to various embodiments;

[0013] Fig.10 is a flow chart of an example sound recognition method according to various embodiments;

[0014] Fig.11 is a diagram for describing example voice recognition performance according to various embodiments. DETAILED DESCRIPTION

[0015] Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings. The embodiments set forth herein may be modified in various ways. Specific embodiments are shown in the accompanying drawings and may be described in detail in specific embodiments. However, the embodiments shown in the accompanying drawings are only intended to help understand the various embodiments. Therefore, the technical scope of the present disclosure is not limited to the embodiments shown in the accompanying drawings, and should be understood to include all equivalents or substitutes within the spirit and scope of the present disclosure.

[0016] Terms including ordinal numbers such as first and second may be used to describe various components, but these components are not limited by these terms. These terms are used only to distinguish one component from another.

[0017] It should be understood that when used in this article, the term "comprising" specifies the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, but does not exclude the presence or addition of one or more features, integers, steps, operations, elements, components, or combinations thereof. When a component is referred to as being "coupled to" or "connected to" another component, it should be understood that the component may be directly coupled to or connected to the other component, but other components may be interposed therebetween. Conversely, when a component is referred to as being "directly coupled to" or "directly connected to" another component, it should be understood that there are no intervening components.

[0018] Meanwhile, the term "module" or "unit" when used to describe a component should be understood to perform at least one function or operation. In addition, a "module" or "unit" may perform a function or operation by hardware, software, or a combination of hardware and software. Multiple "modules" or multiple "units" may be integrated into at least one module, excluding a "module" or "unit" that should be executed by specific hardware or a processor. As used herein, a singular expression is intended to include the plural form as well, unless the context clearly dictates otherwise.

[0019] In the following description of the present disclosure, the order of operations should be understood without restriction, unless the preceding operation should be performed logically and temporally prior to the subsequent operation. That is, except in such exceptional circumstances, even when the operation described as a subsequent operation is performed before the operation described as a previous operation, the essence of the present disclosure will not change, and the scope of the present disclosure should be defined regardless of the order of the operations. As used herein, for example, the expression "A or B" means not only selectively referring to A or B, but also including A and B. The term "comprising" used herein should be understood to include other components in addition to the listed components.

[0020] In the present disclosure, only the necessary components required for describing the present disclosure are described, and components irrelevant to the essence of the present disclosure are not described. In addition, the term "comprising" should not be interpreted as an exclusive meaning of only including the components described herein, but should be interpreted as a non-exclusive meaning of including other components.

[0021] In the following description of the present disclosure, when it is determined that the related known functions or components obscure the subject matter of the present disclosure due to unnecessary details, the related known functions or components will be briefly described or not described. In addition, the embodiments can be implemented or operated independently, but can also be implemented or operated in combination.

[0022] Figure 1 is a block diagram for describing a configuration of an example electronic device according to various embodiments.

[0023] Reference Figure 1 , the electronic device 100 includes an input interface 110 (e.g., including an interface circuit) and a processor 120 (e.g., including a processing circuit). The electronic device 100 can recognize sounds by training a neural network model that recognizes (or identifies) sounds based on training data and / or using the trained neural network model. Examples of the electronic device 100 may include servers, clouds, smart phones, tablet PCs, navigation devices, board PCs, wearable devices, digital TVs, desktop computers, laptops, home appliances, Internet of Things (IoT) devices, self-service terminals, and the like. For example, when the electronic device 100 is a server or a cloud, the electronic device 100 may train a neural network model to recognize sounds. When the electronic device 100 is a smart phone, a tablet PC, and the like, for example, the electronic device 100 may store a trained sound recognition artificial intelligence (AI) neural network and use the stored sound recognition AI neural network to recognize sounds. Of course, the present disclosure is not limited in this regard.

[0024] A control command may be received from a user through the input interface 110. In addition, a sound may be received through the input interface 110. For example, the input interface 110 configured to receive a command from a user may be implemented as a keyboard, a button, a keypad, a touch screen, etc. The input interface 110 configured to receive a sound may be implemented as a microphone. When the input interface 110 is implemented as a microphone, the sound generated in the surrounding environment may be received through the input interface 110. Optionally, the input interface 110 may include a communication interface (e.g., including a communication circuit) to communicate with an external device. Sound data may be received from an external device through the input interface 110 including the communication interface. For example, the communication interface may establish communication with an external device through at least one of a communication method such as Wi-Fi, Wi-Fi Direct, Bluetooth, Zigbee, a third generation mobile communication technology (3G), a third generation partnership project (3GPP), or a long term evolution (LTE). The communication interface may be referred to as, for example, a communication device, a communicator, a communication module, a transceiver, etc. Optionally, the input interface 110 may be implemented as an input / output port. In this case, the input / output port may be connected to an external device to receive sound data from the external device. The input interface 110 may be referred to as, for example, an input device, an inputter, an input module, and the like.

[0025] The processor 120 may control components of the electronic device 100. The processor 120 may control the input interface 110 to receive a sound. The processor 120 may perform a fast Fourier transform (FFT) on an input sound sampled based on a preset sampling rate based on at least one of a random FFT number or a random hop length, and generate a two-dimensional (2D) feature map having a time axis and a frequency axis from the sampled input sound on which the FFT is performed.

[0026] For example, the processor 120 may perform FFT on the sampled input sound based on the first FFT number and generate a 2D feature map. The FFT number may be a frequency resolution. When the frequency resolution is high (or when the FFT number is large), the resolution of the frequency axis and the size (data amount) of the frequency components of the sound data may increase. The first FFT number obtained by the transformation based on the first FFT number may be a frequency resolution. f FFT data may include f The FFT number is related to the resolution and data volume of the frequency axis. f FFT data generation f The 2D feature map may include the first f In addition, the processor 120 may perform FFT on the sampled input sound based on the nth FFT number and generate a 2D feature map. The nth FFT obtained by the transformation based on the nth FFT number f The FFT data may include f The resolution and amount of data on the frequency axis are related to the number of FFTs. fFFT data generated by nth f The 2D feature map may include the nth f Here, n may be a natural number greater than or equal to 2, and the first FFT number and the nth FFT number may be different from each other. For example, the FFT number may be 2 n , such as 4, 8, 16, 32, 64, 128, 256, 1024, 2048, etc. That is, the FFT number can be a random number.

[0027] As described above, the FFT number may be related to the resolution of the frequency components of the sound data. An untrained neural network model cannot determine that the same data with different resolutions are identical to each other. This is because even the data, components, features, etc. included in the same partial area of ​​the same data may vary depending on the resolution. Therefore, the neural network model may recognize the same data with different resolutions as different types of data, and may be trained to recognize the same data with different resolutions as the same data. That is, the electronic device 100 of the present disclosure may generate a plurality of 2D feature maps including different features from each other from the same sound data according to the FFT number. The plurality of 2D feature maps including different features from each other may be used as different types of training data for training a neural network model for recognizing sound.

[0028] Optionally, the processor 120 may perform FFT on the sampled input sound based on the random jump length and generate a 2D feature map. The jump length may be understood as the number of segments of the sound data to be jumped (skipped, ignored or skipped) among all the segments of the sampled sound data. The first segment obtained by the transformation based on the first jump length may be represented by a random jump length. h The FFT data may include the sound data components on the time axis according to the first hop length. h FFT data generation h The 2D feature map may include the first h In addition, the processor 120 may perform FFT on the sampled input sound based on the nth hop length and generate a 2D feature map. h The FFT data may include h The sound data component on the time axis is skipped. h FFT data generated by nth h The 2D feature map may include the nth h Here, n may be a natural number greater than or equal to 2, and the first hop length and the nth hop length may be different from each other. For example, the hop length may be 2 n , such as 4, 8, 16, 32, 64, 128, 256, 1024, 2048, etc. In other words, the jump length can be a random value.

[0029] The electronic device 100 of the present disclosure can generate multiple 2D feature maps including different features from the same sound data according to the hop length. The multiple 2D feature maps including different features can be used as different types of training data for training a neural network model for recognizing sound.

[0030] Optionally, the processor 120 may perform FFT on the sampled input sound based on the random FFT number and the random hop length, and generate a 2D feature map. The first feature map obtained by the transformation based on the first FFT number and the first hop length fh The FFT data may include a resolution on the frequency axis associated with the first FFT number and a sound data component on the time axis according to the first jump length. fh FFT data generation fh The 2D feature map may include the first fh In addition, the processor 120 may perform FFT on the sampled input sound based on the nth FFT number and the nth hop length, and generate a 2D feature map. The nth FFT obtained by the transformation based on the nth FFT number and the nth hop length fh The FFT data may include the resolution of the frequency axis associated with the nth FFT number and the sound data components on the time axis according to the nth jump length. fh FFT data generated by nth fh The 2D feature map may include the nth fh Features. The FFT number and hop length can be random values ​​and can be independent of each other. For example, the FFT number can be 1024 and the hop length can be 512.

[0031] A method of performing FFT based on at least one of various FFT numbers (eg, random FFT numbers) and various hop lengths (eg, random hop lengths) may be referred to as a random FFT method.

[0032] The electronic device 100 of the present disclosure can generate multiple 2D feature maps including different features from the same sound data according to random FFT numbers and / or random hop lengths. Multiple 2D feature maps including different features can be used as different types of training data for training a neural network model for recognizing sound.

[0033] In order to generate a 2D feature map, the processor 120 may generate a plurality of local feature maps and connect the local feature maps to finally generate a 2D feature map.

[0034] For example, the processor 120 may perform FFT on a sound portion of a first time corresponding to a preset number of samples to generate a first local feature map. The processor 120 may perform FFT on a sound portion of a second time that is continuous with the first time and corresponds to a preset number of samples to generate a second local feature map. In addition, the processor 120 may perform FFT on a sound portion of an nth time that is continuous with the (n-1)th time and corresponds to a preset number of samples to generate an nth local feature map. The processor 120 may connect the first local feature map, the second local feature map, and the nth local feature map to generate a 2D feature map.

[0035] As described above, the processor 120 may generate different feature maps based on the same sound data. For example, the processor 120 may generate different feature maps with different resolutions of the frequency axis by changing the FFT number. The FFT number may be related to the size of the representation of the data on the frequency axis. In addition, the processor 120 may generate a second 2D feature map by jumping once according to the number of samples corresponding to the jump length. The processor 120 may generate a third 2D feature map by jumping twice according to the number of samples corresponding to the jump length. The processor 120 may generate an mth 2D feature map by jumping (m-1) times according to the number of samples corresponding to the jump length.

[0036] The processor 120 can use the generated 2D feature map as training data to train a neural network model for recognizing sound. A large amount of training data is required to train a neural network model. The electronic device 100 can use multiple 2D feature maps generated by jumping the same sound data multiple times as multiple pieces of training data.

[0037] Optionally, the processor 120 may generate multiple pieces of training data by masking the generated 2D feature map in various ways. For example, the processor 120 may mask a preset area of ​​the 2D feature map. Examples of masked preset areas may include an area of ​​a geometric shape that is long on the time axis, an area of ​​a geometric shape that is long on the frequency axis, a diagonal area with a preset width, an area with a grid pattern, etc. The electronic device 100 may mask the same 2D feature map in various forms and use the masked results as multiple pieces of training data.

[0038] Functions related to artificial intelligence according to the present disclosure may be performed by the processor 120 and the memory of the electronic device 100 .

[0039] The electronic device 100 may include one or more processors 120. In this case, the one or more processors 120 may include, but are not limited to, at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a neural processing unit (NPU). For example, when the electronic device 100 includes a plurality of processors 120, the main processor may perform a main operation, and the auxiliary processor may perform an operation that assists the operation of the main processor. Optionally, one of the plurality of processors 120 may perform a first function, and another may perform a second function. Optionally, the plurality of processors 120 may process control sequences in parallel.

[0040] The CPU is a general-purpose processor capable of performing artificial intelligence operations as well as general operations, and can efficiently execute complex programs through a multi-layer cache structure. The CPU is advantageous for a serial processing method in which a previous calculation result and a subsequent calculation result can be organically connected to each other through sequential calculations. The general-purpose processor is not limited to the above examples unless it is specified as the above-mentioned CPU.

[0041] A GPU is a processor configured to perform a large number of calculations (such as floating-point operations for graphics processing), in which a plurality of cores are integrated on a large scale to perform large-scale calculations in parallel. In particular, a GPU may be superior to a CPU in terms of parallel processing methods such as convolution operations. Alternatively, a GPU may be used as an auxiliary processor to supplement the functions of a CPU. A processor configured to perform a large number of calculations is not limited to the above examples unless it is specified as the above-mentioned GPU.

[0042] NPU is a processor dedicated to AI operations using AI neural networks, and each layer of the AI ​​neural network can be implemented by hardware (e.g., silicon). In this case, the NPU is designed to be specialized according to, for example, the specifications required by the company, so its degree of freedom is lower than that of the CPU or GPU, but the NPU is able to effectively handle, for example, artificial intelligence operations required by the company. Processors dedicated to AI intelligent operations can be implemented in various forms, such as tensor processing units (TPUs), intelligent processing units (IPUs), and visual processing units (VPUs). The AI ​​processor is not limited to the above examples unless it is specified as the above-mentioned NPU.

[0043] One or more processors may be implemented as a system on chip (SoC). In this case, in addition to one or more processors, the SoC may also include a memory and a network interface, such as a bus for data communication between the processor and the memory.

[0044] When the SoC included in the electronic device includes a plurality of processors, the electronic device may use some of the plurality of processors to perform AI-related operations (e.g., operations related to training an AI model or reasoning). For example, the electronic device may use at least one of a GPU, an NPU, a VPU, a TPU, or a hardware accelerator dedicated to AI-related operations (e.g., convolution operations and matrix multiplication operations) among the plurality of processors to perform AI-related operations. However, the above description is only an example, and a general-purpose processor such as a CPU may be used to process AI-related operations.

[0045] In addition, the electronic device can use multiple cores (eg, dual cores, quad cores, etc.) included in one processor to perform AI-related operations. Specifically, the electronic device can use multiple cores included in the processor to perform AI-related operations such as convolution operations and matrix multiplication operations in parallel.

[0046] One or more processors control the input data to be processed according to predefined operating rules stored in the memory or using an AI model. The predefined operating rules or AI model may be provided by learning.

[0047] Here, the predefined operation rules of the AI ​​model provided by learning can be understood as, for example, predefined operation rules or AI corresponding to the desired features obtained by applying the learning algorithm to multiple training data. Such learning can be performed by the device for executing AI according to the present disclosure or by a separate server / system.

[0048] The artificial intelligence model may include multiple neural network layers. At least one weight value is assigned to at least one layer, and the operation of at least one layer is performed by the operation result of the previous layer and at least one defined operation. Examples of neural networks disclosed in the present invention include convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q networks, and transformers, but are not limited thereto unless otherwise specified.

[0049] A learning algorithm is a method of using multiple pieces of training data to train a target device (e.g., a robot) so that the target device can make decisions or predictions on its own. Examples of learning algorithms may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited thereto unless otherwise specified.

[0050] As described above, the 2D feature graph may include a time axis and a frequency axis. The resolution of the frequency axis of the 2D feature graph may be determined by a preset number of samples. The resolution of the time axis of the 2D feature graph may be determined by a random jump length. The feature graph will be described in detail below.

[0051] The processor 120 may use a trained neural network model to recognize a sound. For example, the processor 120 may receive a sound (or sound data) through the input interface 110. The processor 120 may generate a 2D feature map from the input sound in a manner similar to the above process. The processor 120 may use a trained neural network model to recognize a sound from the generated 2D feature map. That is, the processor 120 may recognize the type of sound and the place where the sound has appeared. For example, the processor 120 may recognize the type of sound, such as a human voice, a baby's cry, a car sound, a dog barking, the sound of the wind, etc., and may recognize the place where the sound has appeared, such as an urban area, a forest, or a beach.

[0052] Figure 2 is a detailed block diagram for describing a configuration of an example electronic device according to various embodiments.

[0053] Reference Figure 2 , the electronic device 100 may include an input interface 110, a processor 120, a camera 130, a microphone 140, a speaker 150, a display 160, a memory 170, and a sensor 180. The input interface 110 refers to Figure 1 As described above, a detailed description thereof will not be repeated.

[0054] The camera 130 may capture the surroundings of the electronic device 100. Optionally, the camera 130 may capture the user's facial expressions, movements, gazes, etc. The processor 120 may perform a control operation based on the captured information of the surroundings or information about the user. Examples of the camera 130 may include a charge coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. Examples of the camera 130 may include an RGB camera and a depth camera.

[0055] The sound of the surrounding environment may be received through the microphone 140. In addition, the user's voice may be received through the microphone 140. The processor 120 may recognize the type of the input sound based on the input sound. Alternatively, the processor 120 may recognize a control command based on the user's input voice and perform a control operation corresponding to the recognized control command.

[0056] The speaker 150 outputs a sound signal obtained by performing signal processing. For example, the speaker 150 may output information about the recognized sound type in the form of voice. In addition, the speaker 150 may output a user's input command, information related to the state or operation of the electronic device 100, etc. in the form of voice or notification sound.

[0057] The display 160 may display information visually. For example, the display 160 may display information about the recognized sound type. For example, the display 160 may be implemented as a liquid crystal display (LCD), an organic light emitting diode (OLED), a touch screen, etc. When the display 160 is implemented as a touch screen, the electronic device 100 may receive a control command through the touch screen.

[0058] The memory 170 may store data, algorithms, etc. for executing the functions of the electronic device 100, and store programs, commands, etc. to be executed by the electronic device 100. For example, the memory 170 may store a trained neural network model. The algorithms, data, etc. stored in the memory 170 may be loaded into the processor 120 under the control of the processor 120 to perform data processing thereon. That is, the trained neural network model may be loaded into the processor 120 to recognize the type of input sound. For example, the memory 170 may be implemented as a memory type such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), a solid-state drive (SSD), or a memory card.

[0059] The sensor 180 may detect information related to the user or the surrounding environment. The processor 120 may perform a control operation based on the detected information. Examples of the sensor 180 may include an image sensor, a tracking sensor, an angle sensor, an acceleration sensor, a gravity sensor, a gyroscope sensor, a geomagnetic sensor, a direction sensor, a motion recognition sensor, a proximity sensor, a voltmeter, an ammeter, a barometer, a hygrometer, a thermometer, an illumination sensor, a thermal sensor, a touch sensor, an infrared sensor, an ultrasonic sensor, etc.

[0060] The electronic device 100 may include all or some of the above-mentioned components. In addition to the above-mentioned components, the electronic device 100 may further include other components for performing various functions.

[0061] The process of generating 2D feature maps is described in detail below.

[0062] Figure 3 is a diagram illustrating an example process of sampling an input sound according to various embodiments.

[0063] Figure 3 The waveform 11 of the sound is shown. The sound input through the microphone may be an analog signal. The electronic device 100 may convert the analog signal into a digital signal to process the input sound. The electronic device 100 may sample the input sound based on a preset sampling rate. For example, when the sampling rate is 16kHz, the electronic device 100 may sample 16,000 pieces of data per second. That is, the electronic device 100 may sample one piece of data at an interval of 62.5us. Figure 3As shown, the electronic device 100 may sample a plurality of pieces of data including first data d1 , fifth data d5 , and nth data dn.

[0064] The electronic device 100 may perform FFT on the sampled data and generate a 2D feature map based on the sampled data on which the FFT is performed.

[0065] Figure 4 , Figure 5 and Figure 6 is a diagram for describing a process of generating a 2D feature map according to various embodiments.

[0066] Figure 4 The sample data and 2D feature map are shown. Figure 3 As described, the electronic device 100 may sample the input sound at a preset sampling rate to obtain a plurality of data including the first data d1, the fifth data d5, and the nth data dn. In addition, the electronic device 100 may perform FFT on the sampled sound data to generate a 2D feature map. The 2D feature map may be a spectrum map of the sound data. The X-axis of the 2D feature map may be a time component, and the Y-axis thereof may be a frequency component. That is, the 2D feature map may represent a frequency component f including the sound data according to time t.

[0067] In a 2D feature map, the resolution of the frequency axis can be determined by a random FFT number. For example, when the sampling rate is 16kHz, according to the Nyquist theorem, the data obtained by performing FFT can be expressed as 8kHz, which is 0.5 times the sampling rate. When the FFT number is 512, the electronic device 100 can process the frequency band of 8kHz by dividing it into 257 (512 / 2+1) regions. When the FFT number is 2048, the electronic device 100 can process the frequency band by dividing it into 1025 (2048 / 2+1) regions. As the FFT number increases, the resolution of the frequency axis can increase, and the size of the frequency components of the sound data can increase. Therefore, the electronic device 100 can determine the amount of data on the frequency axis of the 2D feature map according to the FFT number, and change the resolution of the frequency axis by changing the FFT number.

[0068] The electronic device 100 may perform FFT by dividing the sampled sound data into a plurality of regions to generate a local feature map, and thereafter, may generate an entire 2D feature map by connecting the generated local feature maps.

[0069] exist Figure 4, i may represent a preset number of samples. For example, when the preset number of samples is 1024, the electronic device 100 may perform FFT on 1024 pieces of sound data to generate a first local feature map 21. When the sampling rate is 16kHz, the sampling time interval is 62.5us, and therefore t1 is approximately 75.2ms. That is, in the above example, the electronic device 100 may generate a first local feature map 21 based on 1024 pieces of sound data from a time period of 0 to 75.2ms. Thereafter, the electronic device 100 may generate a second local feature map 22 based on 1024 pieces of sound data from a time period of 75.2ms to 150.4ms. The electronic device 100 may connect the second local feature map 22 to the first local feature map 21. In the present disclosure, the term "connection" may be understood as, for example, sequentially arranging the generated feature maps. The electronic device 100 may repeat the above process to generate the nth local feature map corresponding to the sound data until the nth time, and connect the nth local feature map to the (n-1)th local feature map. That is, the electronic device 100 may generate the entire 2D feature map by connecting the first local feature map 21 , . . . , and the nth local feature map to each other.

[0070] In this case, when the FFT number is 512, the electronic device 100 can process the frequency band of 8kHz by dividing it into 257 areas (or 257 resolutions). The electronic device 100 generates a first 2D feature map including a first feature through the above process. In addition, the electronic device 100 can generate other 2D feature maps through the same process while changing the FFT number. For example, the electronic device 100 may change the FFT number to 1024. When the FFT number is 1024, the electronic device 100 can process the frequency band by dividing it into 513 areas (or 513 resolutions). The electronic device 100 may generate a local feature map based on the 513 resolutions, and connect the generated local feature maps to generate a second 2D feature map including a second feature. The electronic device 100 may change the FFT number to 2048. When the FFT number is 2048, the electronic device 100 may process the frequency band by dividing the frequency band of 8 kHz into 1025 regions (or 1025 resolutions). The electronic device 100 may generate a local feature map based on the 1025 resolutions, and connect the generated local feature maps to generate a third 2D feature map including a third feature. The above resolutions are just examples of describing the resolution difference according to the FFT number and should not be understood as actual resolutions.

[0071] The generated 2D feature map can be used as training data for a neural network model for sound recognition. The electronic device 100 can use the 2D feature map of sound data whose result value is known to train a neural network model for sound recognition. Optionally, the electronic device 100 can use a neural network model trained based on the generated 2D feature map to recognize the sound. The electronic device 100 can use a neural network model trained based on the 2D feature map of sound data to recognize the sound. For example, the recognition of the sound can be understood as recognizing the type of sound and the place where the sound has appeared.

[0072] Figure 5 Sampled data and 2D feature maps are shown. Figure 5 The sample data shown is similar to Figure 4 The sample data shown is the same. That is, Figure 5 The sound data and Figure 4 The sound data is the same.

[0073] As mentioned above, refer to Figure 4 , the electronic device 100 may perform FFT on the sampled sound data by dividing the sampled sound data into a plurality of regions and generating a local feature map. Thereafter, the entire 2D feature map may be generated by connecting the generated partial feature maps.

[0074] However, Figure 5 The process of generating a 2D feature map by applying jump length is shown. Figure 5 In the example, when the preset number of samples i is 1024 and the jump length j is 512, the electronic device 100 may skip 512 sound data and generate the first local feature map 23 from 1024 sound data starting from the 513th sound data. Since the sampling time interval is 62.5us, the first local feature map 23 may be generated based on the 1024 sound data in the time period from 32ms to 107.2ms. Figure 4 As described above, the electronic device 100 may generate the second local feature map 24 based on 1024 pieces of data in the time period of 107.2ms to 182.4ms. The electronic device 100 may connect the second local feature map 24 to the first local feature map 23. That is, the electronic device 100 may generate the entire 2D feature map by connecting the first local feature map 23, ..., and the nth local feature map to each other. Figure 5 The generated 2D feature map is Figure 4 The generated 2D feature map is based on the same sound data, but may include Figure 4 The features of the 2D feature map are different features.

[0075] The electronic device 100 may change the jump length j. When the jump length j is 1024, the electronic device 100 may jump 1024 pieces of sound data and generate a second 2D feature map starting from the 1025th piece of sound data. When the jump length j is 2048, the electronic device 100 may jump 2048 pieces of sound data and generate a third 2D feature map starting from the 2049th piece of sound data. That is, the electronic device 100 may generate a plurality of 2D feature maps containing different features from the same input sound data according to a random jump length 2n. In addition, the electronic device 100 may generate a plurality of 2D feature maps by applying all random FFT numbers and random jump lengths. Therefore, the electronic device 100 may generate various types of training data for training a neural network model based on the same sound data.

[0076] Optionally, the electronic device 100 may generate other 2D feature maps according to the number of jumps based on the jump length j. For example, the electronic device 100 may generate a feature map starting from the 1025th sound data by jumping twice with the jump length j. Therefore, the electronic device 100 may generate m 2D feature maps by jumping the sample of the sound data m times with the jump length j.

[0077] In a 2D feature map, the resolution of the time axis may be determined by the jump length and / or the number of jumps. For example, the number of sampled input data may be 4096, and the preset number i of samples for generating a local feature map may be 1024. The electronic device 100 may generate four local feature maps from the sampled input data. When the jump length is 1024, the electronic device 100 may generate three local feature maps. When the jump length is 2048, the electronic device 100 may generate two local feature maps. When the jump length is 1024 and the jump is performed once, the electronic device 100 may generate three local feature maps. When the jump length is 1024 and the jump is performed twice, the electronic device 100 may generate two local feature maps. Therefore, the electronic device 100 may change the resolution of the time axis for the same sampled input data according to the jump length and / or the number of jumps. That is, the electronic device 100 may change the resolution of the time axis of the 2D feature map based on the jump length and / or the number of jumps.

[0078] In addition, the last local feature map may include a portion of the sound data according to the number of sampled input data, a preset number of samples, and a jump length.

[0079] Figure 6 is a diagram for describing an example process of generating a final local feature map according to various embodiments.

[0080] Figure 6A 2D feature map 30 obtained by connecting local feature maps to each other is shown. The last piece of data of the sampled input data may be data sampled at time point tn. Therefore, no data may be input after time point tn of the last local feature map. When data is not input, the electronic device 100 may fill (e.g., fill with zeros) a specific value to the area after time point tn. For example, the electronic device 100 may fill zeros to the area of ​​the last local feature map after time point tn.

[0081] Alternatively, if Figure 6 As shown, the electronic device 100 may copy data of a specific region of the input data. For example, when the last feature map lacks 128 pieces of data, the electronic device 100 may connect 64 pieces of initial data and 64 pieces of data immediately before the last local feature map to the region after time point tn. In this case, the electronic device may connect the data of the copied region to each other in reverse order (reverse direction). Figure 6 As shown, the electronic device 100 may connect the first data to the 64th data in reverse order starting from the 64th data, and connect the 10176th data to the 10240th data in reverse order starting from the 10240th data.

[0082] In addition, the electronic device 100 may mask the generated 2D feature map in various patterns to generate various types of training data.

[0083] Figure 7a , Figure 7b , Figure 7c , Figure 7d , Figure 7e , Figure 7f and Figure 7g is a diagram for describing a masking pattern according to an embodiment. Figure 7a , Figure 7b , Figure 7c , Figure 7d , Figure 7e , Figure 7f and Figure 7g .

[0084] Figure 7a As described above, the electronic device 100 may generate a 2D feature map 30 by generating local feature maps for some regions of sound data and connecting the generated local feature maps to each other. The X axis of the 2D feature map 30 may be a time component, and the Y axis thereof may be a frequency component.

[0085] The electronic device 100 can generate various types of training data by masking a preset area of ​​the 2D feature map. For example, Figure 7bThe 2D feature map 30 shows that a specific frequency component is masked (1) over the entire time region. Figure 7c The 2D feature map 30 is shown masked (1) in a diagonal shape from the upper left end of the 2D feature map 30 to the lower right end thereof. Figure 7d A 2D feature map 30 is shown masked in a diagonal shape (1) from the left region to the upper right end. Figure 7e A 2D feature map 30 is shown masked (1) in a trapezoidal shape. Figure 7f A 2D feature map 30 is shown which is masked (1) in a rectangular shape from a specific time point to a last time point on the time axis.

[0086] Included in Figure 7b , Figure 7c , Figure 7d , Figure 7e and Figure 7f The data in the masked area shown in may be 0. However, masking for changing values ​​may be applied to a 2D feature map. Figure 7g Shown includes Figure 7b The result of masking 1 is similar to the result of masking 1 in 2D feature map 30. However, Figure 7b The data in the masked area is 0, and Figure 7g The data of the masked area may include specific values. Figure 7b , Figure 7c , Figure 7d , Figure 7e , Figure 7f and Figure 7g The masking area is only an example, and the embodiment is not limited thereto. That is, examples of the masking area may include an area of ​​a geometric shape that is long on the time axis, an area of ​​a geometric shape that is long on the frequency axis, a diagonal area with a preset width, an area with a grid pattern, an area where data values ​​change, etc.

[0087] The electronic device 100 may perform masking so that the characteristics of the input sound data can be maintained. The electronic device 100 may mask an area of ​​the 2D feature map at a preset first ratio, wherein, in the area, the amount (or area) of the effective sound data is greater than the preset first level. In addition, the electronic device 100 may mask an area of ​​the 2D feature map at a preset second ratio, wherein, in the area, the amount of effective sound data is equal to or less than a predetermined second level. The preset second ratio may be higher than the preset first ratio. For example, the electronic device 100 may set the masking area of ​​the 2D feature map to be less than 50%. The electronic device 100 may mask an area of ​​the 2D feature map to 10% or more, wherein, in the area, the area of ​​effective sound data is empty for one second or longer or the amount of effective sound data is less than 1kHz. Alternatively, the electronic device 100 may mask an area of ​​the 2D feature map to be less than 10%, wherein, in the area, the area of ​​effective sound data lasts for 0.5 seconds or longer, or the amount of effective sound data is 1kHz or more. The electronic device 100 may variably mask the 2D feature map based on the amount (or area) of effective sound data to generate various 2D feature maps including (or maintaining) the features of the sound data.

[0088] Since the example electronic device 100 of the present disclosure is capable of generating various types of training data from one piece of sound data, a large amount of training data can be generated from a small amount of sound data, and the accuracy of a neural network model that recognizes sound can be improved.

[0089] Figure 8 is a diagram for describing an example process of recognizing a type of sound based on a 2D feature map according to an embodiment.

[0090] Figure 8 A 2D feature graph 30 including a first sound feature 3b and a second sound feature 5b is shown. The X-axis of the 2D feature graph 30 may be a time component, and the Y-axis thereof may be a frequency component. Therefore, the first sound feature 3b and the second sound feature 5b may be features associated with sounds generated in the same time zone. However, the first sound feature 3b may include a higher frequency component than the frequency component of the second sound feature 5b.

[0091] For example, when the first sound feature 3b and the second sound feature 5b are features related to dog barking, the first sound feature 3b may be a feature of the barking of a puppy 3a, and the second sound feature 5b may be a feature of the barking of a large dog 5a. Alternatively, when the first sound feature 3b and the second sound feature 5b are related to voice, the first sound feature 3b may be a feature of the voice of a child or a woman, and the second sound feature 5b may be a feature of the voice of an adult or a man.

[0092] The electronic device 100 of the present disclosure is a sound recognition device. Recognizing sound is different from recognizing speech. That is, for example, sound recognition is understood as recognizing the type of sound or the place where the sound has appeared, and speech recognition is understood as also including recognizing the meaning of speech.

[0093] For example, when a man says "turn on the light", the spoken content "turn on the light" is recognized by voice recognition, but the man's voice is recognized instead of the spoken content by sound recognition. That is, in voice recognition, the sound of wind, children crying, dogs barking, etc. are ignored as noise or meaningless sounds, but in sound recognition, the type of each sound or the surrounding environment can be recognized.

[0094] Therefore, various types of training data as described above cannot be generated by speech recognition. For example, sound recognition allows the characteristics of sound to be recognized, and thus the type of sound data can be recognized from a feature map generated by performing jumps according to the jump length or a feature map to which various forms of masking are applied. However, speech recognition allows the meaning of speech to be recognized, and thus the exact meaning of speech cannot be recognized by jumping sound data or applying masking.

[0095] In addition, the electronic device 100 may recognize a personalized sound. Personalized sound recognition may be understood as recognizing an object that generates a sound.

[0096] Fig. 9 is a diagram for describing example personalized voice recognition according to various embodiments.

[0097] Reference Fig. 9 , the barking sound of the first puppy 3a may be input to the electronic device 100. The electronic device 100 may perform FFT on the input barking sound of the first puppy 3a to generate a 2D feature map. The electronic device 100 may identify the first feature vector 3c from the input 2D feature map using the trained sound recognition neural network model and recognize the sound. Next, the electronic device 100 may store the identified first feature vector 3c.

[0098] Thereafter, the electronic device 100 may receive the dog barking sound and identify the second feature vector 7c in the same manner using the trained sound recognition neural network model. The electronic device 100 may compare the identified second feature vector 7c with the stored first feature vector 3c. For example, the electronic device 100 may use a method such as cosine similarity, Euclidean similarity, or vector similarity to determine the similarity between vectors. As a result of the comparison, the electronic device 100 may determine that the second feature vector 7c is similar to the first feature vector 3c. The electronic device 100 may identify the received dog barking sound as the barking sound of the first puppy 3a.

[0099] Various examples of voice recognition have been described above. A flow chart of a voice recognition method will be described below.

[0100] Fig.10 is a flow chart of an example voice recognition method according to various embodiments.

[0101] Reference Fig.10 , the electronic device receives an input sound and samples the input sound based on a preset sampling rate (S1010). The electronic device generates a 2D feature map (having a time axis and a frequency axis) from the sampled input sound (S1020). The electronic device may perform FFT on the sampled input sound based on at least one of a random FFT or a random jump length, and generate a 2D feature map having a time axis and a frequency axis from the sampled input sound on which the FFT is performed.

[0102] For example, the electronic device may transform the sampled input sound into the first FFT data based on at least one of the first FFT number in the random FFT number and the first jump length in the random jump length, and generate a first 2D feature map including the first feature from the first FFT data. In addition, the electronic device may transform the sampled input sound into the nth FFT data based on at least one of the nth FFT number in the random FFT number or the nth jump length in the random jump length, and generate an nth 2D feature map including the nth feature from the nth FFT data.

[0103] The electronic device may perform FFT on a sound portion of a first time corresponding to a preset number of samples to generate a first local feature map. In addition, the electronic device may generate a second local feature map by performing FFT on a sound portion of a second time that is continuous with the first time and corresponds to a preset number of samples. The electronic device may generate an nth local feature map by repeatedly performing the above process (i.e., performing FFT on a sound portion of an nth time that is continuous with the (n-1)th time and corresponds to a preset number of samples). The electronic device may connect the first local feature map, the second local feature map, and the nth local feature map to generate a 2D feature map.

[0104] In S1030, the electronic device may train a neural network model for recognizing sound using a plurality of 2D feature maps including the first 2D feature map and the nth 2D feature map as training data.

[0105] For example, the electronic device may determine the amount of data on the frequency axis based on a random FFT number. In addition, the electronic device may change the resolution of the frequency axis of the 2D feature map at the random FFT number. The FFT number may be the frequency resolution. The electronic device may generate various 2D feature maps from the same input sound data based on various FFT numbers, and use the various 2D feature maps as multiple training data to train the neural network model.

[0106] Optionally, the electronic device can generate multiple 2D feature maps by jumping the number of samples corresponding to the random jump length m times, and use the multiple 2D feature maps as multiple training data to train the neural network model. The electronic device can change the resolution of the time axis of the 2D feature map based on the random jump length.

[0107] In addition, the electronic device may mask a preset area of ​​the 2D feature map. For example, the electronic device may mask an area in the generated 2D feature map where the amount of sound data is greater than a preset first level at a preset first ratio, and mask an area in the 2D feature map where the amount of sound data is equal to or less than a preset second level at a preset second ratio. The preset second ratio may be higher than the preset first ratio. Examples of masked preset areas may include an area of ​​a long geometric shape on the time axis, an area of ​​a long geometric shape on the frequency axis, a diagonal area with a preset width, an area with a grid pattern, and the like. The electronic device may mask different areas of a plurality of 2D feature maps, and use a plurality of 2D feature maps with different areas masked as a plurality of training data to train a neural network model for recognizing sound.

[0108] The electronic device may recognize a sound using a trained neural network model. For example, the electronic device may receive a sound to be recognized. The electronic device may generate a 2D feature map to be recognized from the input sound. The electronic device may recognize the sound by inputting the generated 2D feature map into the trained neural network model.

[0109] The electronic device can identify the type of sound and the place where the sound has been generated. For example, the electronic device can identify the type of sound (e.g., the sound of waves or the sound of a vehicle horn, etc.) and the place where the sound has appeared (e.g., urban area, indoors, or beach).

[0110] In addition, the electronic device can use the trained neural network model to recognize the object generating the sound (personalized sound recognition).

[0111] For example, in the sound recognition process described above, the electronic device may recognize a first feature vector of an input sound. When another sound to be recognized is input, the electronic device may recognize a second feature vector of the other sound. The electronic device may compare the first feature vector and the second feature vector with each other to recognize whether the input sound and the other sound to be recognized are generated from the same object (or the same source or sound source).

[0112] Fig.11 is a diagram for describing example voice recognition performance according to various embodiments.

[0113] Fig.11 The classification results on the DCASE2018 and 2019 datasets are shown by different data augmentation methods applied to various CNN architectures. Fig.11The box plot results of Speaugment are shown, where each of our mask random transformations uses Speaugment and Masking RandAugment. We observe that Method 5 according to the present disclosure is effective regardless of the CNN architecture and dataset. Random masking transformation outperforms SpecAugment not only when comparing the median of each test case, but also when comparing the quartiles of each test case. Comparing the two datasets, the improvement is higher in DCASE2019 than in DCASE2018 because the dataset size of DCASE2019 is larger than that of DCASE2018. This suggests that the improvements produced by Method 5 according to the present disclosure can be further improved with larger dataset sizes and diversity.

[0114] The effects of the present disclosure are not limited thereto, and other effects not described herein will be apparent to those of ordinary skill in the art based on the above description.

[0115] The sound recognition method according to the above-described various embodiments may be provided by a computer program product. The computer program product may be a software (S / W) program or may include a non-transitory computer-readable medium storing the S / W program.

[0116] A non-transitory computer-readable medium may be understood, for example, as a medium that stores data semi-permanently and can be read by a machine, rather than a medium that stores data for a short period of time, such as a register, a cache, or a memory. Specifically, various applications or programs as described above may be provided by being stored in a non-transitory computer-readable recording medium such as a CD, a DVD, a hard disk, a Blu-ray disc, a USB, a memory card, a ROM, etc.

[0117] Although the present disclosure has been shown and described with reference to various exemplary embodiments, it will be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. It will be further understood by those skilled in the art that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure (including the appended claims and their equivalents). It will also be understood that any embodiment described herein may be used in combination with any other embodiment described herein.

Claims

1. A method for voice recognition, include: Sampling the input sound based on a preset sampling rate; Performing a fast Fourier transform (FFT) on the sampled input sound based on at least one of a random FFT number or a random jump length, and generating a two-dimensional 2D feature map having a time axis and a frequency axis from the sampled input sound on which the FFT is performed, wherein generating the 2D feature map includes: transforming the sampled input sound into first FFT data based on at least one of a first FFT number in the random FFT numbers or a first jump length in the random jump lengths, generating a first 2D feature map including a first feature from the first FFT data, transforming the sampled input sound into nth FFT data based on at least one of an nth FFT number in the random FFT numbers and an nth jump length in the random jump lengths, and generating an nth 2D feature map including an nth feature from the nth FFT data, wherein n is greater than 1; A neural network model for recognizing sound is trained using a plurality of 2D feature maps including the first 2D feature map and the nth 2D feature map as training data.

2. The voice recognition method according to claim 1, in, Generating a 2D feature map also includes: Perform FFT on the sound part of the first time corresponding to a preset number of samples to generate a first local feature map, perform FFT on the sound part of the second time continuous with the first time and corresponding to the preset number of samples to generate a second local feature map, perform FFT on the sound part of the nth time continuous with the (n-1)th time and corresponding to the preset number of samples to generate an nth local feature map, and connect the first local feature map, the second local feature map and the nth local feature map to each other.

3. The voice recognition method according to claim 1, in, Generating a 2D feature map also includes: determining the amount of data on the frequency axis according to the random FFT number.

4. The voice recognition method according to claim 1, in, Generating a 2D feature map further includes: changing a resolution of a frequency axis of the 2D feature map based on the random FFT number.

5. The voice recognition method according to claim 1, in, Generating a 2D feature map further includes: generating a plurality of 2D feature maps including mutually different features by jumping a number of samples corresponding to the random jump length m times, and Training the neural network model includes: using the multiple 2D feature maps as multiple training data to train the neural network model.

6. The voice recognition method according to claim 5, in, Generating a 2D feature map includes: changing a resolution of a time axis of the 2D feature map based on the random jump length.

7. The voice recognition method according to claim 1, further comprising: include: Mask a preset region of the 2D feature map.

8. The voice recognition method according to claim 7, in, The preset regions for masking 2D feature maps include: The effective sound data amount in the 2D feature map generated by masking with a preset first ratio is greater than the area of ​​the preset first level, and the effective sound data amount in the 2D feature map generated by masking with a preset second ratio is equal to or less than the area of ​​the preset second level, wherein the preset second ratio is higher than the preset first ratio.

9. The voice recognition method according to claim 7, in, Preset areas for masking include: At least one of an area of ​​a geometric shape long on a time axis, an area of ​​a geometric shape long on a frequency axis, a diagonal area having a preset width, or a preset area having a grid pattern.

10. The voice recognition method according to claim 7, in, Masking the preset area of ​​the 2D feature map includes: masking different areas of the plurality of 2D feature maps; and Training the neural network model includes: using the multiple 2D feature maps with different areas masked as multiple training data to train the neural network model for recognizing sound.

11. The voice recognition method according to claim 1, further comprising: include: receiving a first sound to be recognized; Generating a 2D feature map to be recognized from the received first sound based on a preset FFT number and a preset hop length; The sound is recognized by inputting the generated 2D feature map into the trained neural network model.

12. The voice recognition method according to claim 11, in, Identifying the sound includes identifying at least one of a type of the sound or a location where the sound is generated.

13. The voice recognition method according to claim 11, further comprising: include: identifying a first feature vector of a received first sound to be identified; When a second sound is input, identifying a second feature vector of the second sound to be identified; The first feature vector and the second feature vector are compared to each other to identify whether the second sound is generated from the same source as the first sound.

14. An electronic device, include: Input interface; at least one processor, wherein the at least one processor is configured to: Sampling the sound input through the input interface based on a preset sampling rate; performing a fast Fourier transform (FFT) on the sampled input sound based on at least one of a random FFT number or a random hop length, and generating a two-dimensional 2D feature map having a time axis and a frequency axis from the sampled input sound on which the FFT is performed; transforming the sampled input sound into first FFT data based on at least one of a first FFT number in the random FFT numbers or a first jump length in the random jump lengths, generating a first 2D feature map including a first feature from the first FFT data, transforming the sampled input sound into nth FFT data based on at least one of an nth FFT number in the random FFT numbers and an nth jump length in the random jump lengths, and generating an nth 2D feature map including an nth feature from the nth FFT data, wherein n is greater than 1; A neural network model for recognizing sound is trained using a plurality of 2D feature maps including the first 2D feature map and the nth 2D feature map as training data.

15. The electronic device as claimed in claim 14, in, The at least one processor is also configured to: perform FFT on the sound portion of the first time corresponding to a preset number of samples to generate a first local feature map, perform FFT on the sound portion of the second time continuous with the first time and corresponding to the preset number of samples to generate a second local feature map, perform FFT on the sound portion of the nth time continuous with the (n-1)th time and corresponding to the preset number of samples to generate an nth local feature map, and connect the first local feature map, the second local feature map and the nth local feature map to each other to generate the 2D feature map.

Citation Information

Patent Citations

  • Structural health monitoring data distortion detection method, system and device and storage medium

    CN112434624A

  • Voice recognition device and method

    CN113454717A

  • Motion sound false judgment method and device based on time-frequency graph and convolutional neural network

    CN113870896A

  • Service quality call route selecting system using adviser and speech recognition engine and its method

    CN1882015A

  • Acoustic signal processing apparatus, acoustic signal processing method, acoustic signal processing program, and computer readable record medium which records acoustic signal processing program

    JP2006340391A