A method and apparatus for detecting sound event using artificial intelligence

Abstract

A method performed by an electronic device using artificial intelligence according to an embodiment of the present invention comprises: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data; and outputting a second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, wherein the first artificial intelligence network that has been trained may be trained based on the plurality of first data and a third data output from a second artificial intelligence network.

Description

Technology Field

[0001] The present disclosure describes a method and apparatus for detecting acoustic events (or events) using artificial intelligence. Specifically, it describes a method for detecting the location where a specific sound source occurs by utilizing voice event detection technology through the training of artificial intelligence. Background Technology

[0003] The present disclosure relates to an apparatus and method for detecting a specific sound source (or acoustic event) in relation to a voice signal. Acoustic event detection technology is a technology for detecting a specific sound (e.g., a ringtone, a dog barking) at each time interval. In particular, it has the advantage of being able to detect and utilize only specific sounds among the various sounds occurring in real life according to the user's selection. Acoustic event detection technology using artificial intelligence is performed by preprocessing general acoustic data, extracting features of each acoustic event from the preprocessed acoustic data, and training an algorithm model with the corresponding features.

[0004] However, to build a system for AI-based acoustic event detection technology, acoustic data with labeled answers is required every hour. To label the acoustic data used for training, the person conducting the training must manually label each sound, which has the disadvantage of being time-consuming and costly. Therefore, there is a need for a method that can effectively train data even in situations where labeled information is unavailable or insufficient.

[0005] In particular, semi-supervised learning methods have been predominantly used in acoustic event detection technology. Semi-supervised learning is a learning method primarily used for image data, utilizing techniques such as mean teacher and fix-match. Based on data augmentation techniques, semi-supervised learning employs a method where the neural network's learning output remains identical to the original image's output even when slight deformation or disturbance is applied to the image. However, while the x and y axes in image data are defined as position, in audio spectrograms, they are defined as time and frequency, respectively. These differences present problems, specifically that the meanings of the x and y axes differ, and that the corresponding correct answer changes when disturbance is applied to the time axis of the data. In other words, when applying learning methods used for image data to audio data, a method is required to maintain consistency because the correct answer can change if disturbance is applied in the time domain. Therefore, a consistency learning method utilizing data augmentation techniques can be employed in semi-supervised learning, and the following proposes a method for using a consistency learning method based on resolution. The problem to be solved

[0006] The present disclosure aims to provide a method and apparatus for improving acoustic event detection capabilities by deriving various resolutions through a data preprocessing method.

[0007] The present disclosure aims to provide a method for improving acoustic event detection capabilities by setting different augmentation levels for unlabeled data. means of solving the problem

[0009] A method performed by an electronic device using artificial intelligence according to an embodiment of the present invention comprises: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data; and outputting a second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, wherein the first artificial intelligence network that has been trained may be trained based on the plurality of first data and a third data output from a second artificial intelligence network.

[0010] In one embodiment, the method may further include the step of determining a loss between the second data and the third data; and the step of determining a parameter that minimizes the loss.

[0011] In one embodiment, if the voice data includes a label, the first artificial intelligence network may further include a step of determining a loss between the label and the second data.

[0012] In one embodiment, if the voice data does not include a label, the method may further include the step of determining a plurality of fourth data representing a plurality of resolutions based on the sampling rate of the voice data, and the step of determining a loss between the fifth data output from a third artificial intelligence network and the second data using the plurality of fourth data as inputs.

[0013] In one embodiment, the step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data may include: a step of extracting at least one of the sampling rate and a window size based on a log-Gaussian distribution; and a step of determining the plurality of first data by applying at least one of the extracted sampling rate and a window size to the voice data.

[0014] In one embodiment, the method may further include a step of detecting specific voice data from the voice data based on the second data.

[0015] In one embodiment, the first artificial intelligence network is learned from the second artificial intelligence network through a knowledge distillation process, the first artificial intelligence network is a student network, and the second artificial intelligence network may be a teacher network.

[0016] An electronic device according to an embodiment of the present invention comprises: a memory; a modem; and a processor connected to the modem and the memory, wherein the processor is configured to receive voice data, determine a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data, and output second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, and the first artificial intelligence network that has been trained can be trained based on the plurality of first data and third data output from the second artificial intelligence network.

[0017] A program stored in a medium for performing a direction estimation method through an artificial intelligence algorithm executable by a processor according to an embodiment of the present invention, comprising the steps of: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data; and outputting a second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, wherein the first artificial intelligence network that has been trained can be trained based on the plurality of first data and the third data output from the second artificial intelligence network. Effects of the invention

[0019] According to one embodiment of the present disclosure, various resolutions can be derived through a data preprocessing method to mitigate the overfitting problem and improve the acoustic event detection function. Brief explanation of the drawing

[0021] A brief description of each drawing is provided to help to more fully understand the drawings cited in the detailed description of the present invention. FIG. 1 is a conceptual diagram illustrating the basic principles of an artificial intelligence structure according to one embodiment of the present disclosure. FIG. 2 is a diagram showing the structure of an artificial intelligence network according to one embodiment of the present disclosure. FIG. 3 is a diagram showing the structure of an artificial intelligence network according to another embodiment of the present disclosure. FIG. 4 shows a spectrogram generated according to a data augmentation method according to one embodiment of the present disclosure. FIG. 5 shows a spectrum analysis according to window size according to one embodiment of the present disclosure. FIG. 6 shows a probability distribution usable for data augmentation according to one embodiment of the present disclosure. FIG. 7 is a diagram showing the structure of an artificial neural network according to one embodiment of the present invention. FIG. 8 is a diagram showing verification loss according to the type of algorithm according to various embodiments of the present disclosure. FIG. 9 is a table showing the structure of a data set according to one embodiment of the present disclosure. FIG. 10 is a table showing performance according to various algorithm methods according to various embodiments of the present disclosure. FIG. 11 is a block diagram of an electronic device for detecting voice events according to one embodiment of the present disclosure. FIG. 12 is a flowchart illustrating a voice event detection method according to an embodiment of the present invention. Specific details for implementing the invention

[0022] The technical concept of the present invention is subject to various modifications and may have various embodiments. Specific embodiments are illustrated in the drawings and described in detail through the detailed description. However, this is not intended to limit the technical concept of the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the scope of the technical concept of the present invention.

[0023] In describing the technical concept of the present invention, detailed descriptions of related prior art are omitted if it is determined that such descriptions may unnecessarily obscure the essence of the invention. Furthermore, numbers used in the description of this specification (e.g., First, Second, etc.) are merely identification symbols to distinguish one component from another.

[0024] In addition, when a component is described in this specification as being "connected" or "connected" to another component, it should be understood that the component may be directly connected to or directly connected to the other component, but unless otherwise specifically stated, it may also be connected or connected through another component in between.

[0025] In addition, terms such as “~part,” “~device,” “~device,” and “~module” described in this specification refer to a unit that processes at least one function or operation, and may be implemented as hardware or software or a combination of hardware and software such as a processor, microprocessor, microcontroller, CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerate Processor Unit), DSP (Drive Signal Processor), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array), and may also be implemented in a form combined with memory that stores data necessary for processing at least one function or operation.

[0026] Furthermore, it is intended to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Additionally, each component described below may additionally perform some or all of the functions of other components in addition to the primary function it is responsible for, and it is obvious that some of the primary functions of each component may be exclusively performed by other components.

[0027] In describing the embodiments of the present disclosure, specific descriptions of related functions or configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the present disclosure. Furthermore, terms used below are defined in consideration of their functions within the present disclosure, and these definitions may vary depending on the intent or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0028] For the same reason, some components in the attached drawings may be exaggerated, omitted, or schematically depicted. Additionally, the size of each component does not entirely reflect its actual size. Identical or corresponding components in each drawing have been assigned the same reference number.

[0029] The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below but may be implemented in various different forms. The embodiments are provided merely to make the description of the present disclosure complete and to fully inform those skilled in the art of the scope of the invention to which the embodiments of the present disclosure belong, and the scope of the claims of the present disclosure is defined only by the scope of the claims.

[0030] At this point, it will be understood that each block of the drawings showing the process flow diagram and the combinations of the process flow diagrams can be executed by computer program instructions. Since these computer program instructions can be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create means to perform the functions described in the flow diagram block(s). Since these computer program instructions can also be stored in computer-available or computer-readable memory that can be directed toward the computer or other programmable data processing equipment to implement the function in a specific way, the instructions stored in computer-available or computer-readable memory can also produce a manufactured item containing the means of instruction to perform the function described in the flow diagram block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the flowchart block(s).

[0031] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0032] As used in this disclosure, the term “unit or part” refers to a software or hardware component, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), and the “part” may be configured to perform specific roles. However, the “part” is not limited to software or hardware. The “part” may be configured to reside in an addressable storage medium or to execute one or more processors. Thus, by example, the “part” includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and “parts” may be combined into a smaller number of components and “parts” or further separated into additional components and “parts.” In addition, the components and 'parts' may be implemented to utilize one or more CPUs within the device or secure multimedia card. Also, in the embodiments, 'parts' may include one or more processors and / or devices.

[0033] Hereinafter, embodiments according to the technical concept of the present invention will be described in detail in turn.

[0035] FIG. 1 is a conceptual diagram illustrating the basic principles of an artificial intelligence structure according to one embodiment of the present disclosure.

[0036] Referring to Figure 1, the basic principle of how learning is performed in an artificial intelligence structure is illustrated.

[0037] Artificial intelligence (AI) technology refers to techniques designed to solve cognitive problems primarily associated with human intelligence, such as learning, problem-solving, and perception. AI can be trained through machine learning (ML) and deep learning (DL). Machine learning is primarily used in techniques for pattern recognition and learning, representing algorithms that learn from recorded data to predict future data. It represents a technology that learns autonomously from data rather than relying on predefined rules or patterns. On the other hand, deep learning is a subfield of machine learning that differs in that it processes data based on Artificial Neural Networks (ANN). Because deep learning utilizes artificial neural networks, it can handle more complex and sophisticated computations than machine learning. Types of algorithms for deep learning include Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), Artificial Neural Networks (ANN), and Recurrent Neural Networks (RNN).

[0038] Referring to FIG. 1, the artificial intelligence structure can be represented as an artificial intelligence module (110). The artificial intelligence module (110) receives predetermined input data (105), performs learning through a method predetermined in the module, and outputs output data (115) for the learning result. According to one embodiment, the input data (105) may include predetermined data (e.g., picture, sound, etc.), voice signals, and input sequences. The output data (115) may include output sequences, enhanced voice signals, voice event information (e.g., specific voices), etc.

[0040] FIG. 2 is a diagram showing the structure of an artificial intelligence network according to one embodiment of the present disclosure.

[0041] The artificial intelligence algorithm used in FIG. 2 may be one of the types of the artificial intelligence module (110) of FIG. 1. FIG. 2 may be a diagram showing an artificial intelligence algorithm in the case where the input voice data includes a label.

[0042] Referring to Fig. 2, the structure of the artificial intelligence system may include voice data and an artificial intelligence algorithm structure.

[0043] According to one embodiment, voice data may refer to voice data including a label or voice data not including a label.

[0044] According to one embodiment, the structure of an artificial intelligence algorithm may be divided into steps of data preprocessing, artificial intelligence learning, and output data verification. Data preprocessing in the present disclosure may utilize a data augmentation technique. A data augmentation technique is a method of augmenting data by varying the sampling rate of an audio signal and the window size, which is a manipulation variable of the short-time Fourier transform. Data augmentation techniques may include strong augmentation and weak augmentation techniques. A strong augmentation technique may apply a sampling rate and a window size by randomly extracting them from a log-Gaussian distribution. For example, the mean of the log-Gaussian distribution becomes the sampling rate and window size values ​​of a reference spectrogram (i.e., an un-augmented spectrogram), and the standard deviation can be set to 1.25. Values ​​randomly extracted from the log-Gaussian distribution may be limited to 1 / 2 or 2 times to prevent excessive distortion. Weak augmentation refers to a method where the window size is the same as the reference spectrogram, but the sampling rate is extracted and used in the same way as strong augmentation. For example, the reference spectrogram uses a sampling rate of 16 kHz and a window size of 128 ms.

[0045] Here, a spectrogram is a two-dimensional graph that visually represents frequency components and is used to analyze audio signals. In particular, a log-mel spectrogram is obtained by converting a spectrogram in the time-frequency domain to a Mel scale and log-processing it.

[0046] The artificial intelligence system of FIG. 2 may be composed of a student network (220) and a teacher network (225) consisting of an Exponential Moving Average (EMA) of student network parameters that is updated at every learning step. The technique used here is the knowledge distillation technique, in which the teacher network, which is generally a large and complex model in deep learning, transfers information to the student network, which is a small and simple model. The goal of knowledge distillation is to maintain the performance of the teacher network in the student model to the maximum while reducing the size and computational complexity of the model. In the knowledge distillation technique, the teacher network is trained first, and the student network is trained based on the result data from the teacher network to compare their performance.

[0047] Referring to FIG. 2, a strong augmentation technique (215) may be applied to voice data (210) including a label (240). By changing at least one of the sampling rate and window size according to the strong augmentation technique, data having various resolutions can be extracted.

[0048] Data augmented based on strong augmentation techniques (215) is used as input information for the student network (220) and the teacher network (225), respectively, and a consistency loss (230) can be calculated between the result data according to the student network (220) and the result data according to the teacher network (225). The calculated consistency loss (230) can be used again to optimize the student network (220). The consistency loss can be calculated as the mean squared error.

[0049] Additionally, a supervised loss (235) can be calculated between the label (240) and the result data obtained from the student network (220) in the voice data (210) containing the label (240). The calculated supervised loss (235) can also be used to optimize the student network (220) together with the consistency loss (230). The supervised loss can be calculated as binary cross-entropy.

[0050] According to an embodiment of the present disclosure, the student network (220) can be trained as an artificial intelligence network with performance similar to that of the teacher network (225) using a lightweight model. Through the artificial intelligence algorithm according to the present disclosure, specific voice data (or voice events) can be detected in voice data.

[0052] FIG. 3 is a diagram showing the structure of an artificial intelligence network according to another embodiment of the present disclosure.

[0053] The artificial intelligence algorithm used in FIG. 3 may be one of the types of the artificial intelligence module (110) of FIG. 1. FIG. 3 may be a diagram showing an artificial intelligence algorithm in the case where the input voice data does not include a label.

[0054] Referring to Fig. 3, the structure of the artificial intelligence system may include voice data and an artificial intelligence algorithm structure.

[0055] According to one embodiment, voice data may refer to voice data including a label or voice data not including a label.

[0056] The artificial intelligence system of Fig. 3 may be composed of a student network (320) and a teacher network (325) consisting of an exponential moving average (EMA) of student network parameters that is updated at every learning step.

[0057] Referring to FIG. 3, a strong augmentation technique (315) can be applied to voice data (310) that does not contain a label.

[0058] Data augmented based on a strong augmentation technique (315) is used as input information for the student network (320) and the teacher network (325), respectively, and a consistency loss (330) can be calculated between the result data according to the student network (320) and the result data according to the teacher network (325). The calculated consistency loss (330) can then be used to optimize the student network (320).

[0059] Additionally, the student network (320) value can be used to perform the role of a label in voice data (310) that does not contain a label. By determining the value in the student network (320), a fixed student network (345) can be established, and output data can be extracted using the data (340) augmented by a weak augmentation technique on the voice data (310) as input information. Depending on the weak augmentation technique, data with various resolutions can be extracted by fixing the window size and changing the sampling rate. Here, the meaning of a fixed network refers to a network that does not calculate a gradient for the input augmented data.

[0060] A supervised loss (335) can be calculated between the output data of the fixed student network (345) and the result data obtained from the student network (320). The calculated supervised loss (335) can also be used to optimize the student network (320) together with the consistency loss (330).

[0061] According to an embodiment of the present disclosure, the student network (320) can be trained as an artificial intelligence network with performance similar to that of the teacher network (325) using a lightweight model.

[0062] FIG. 4 shows a spectrogram generated according to a data augmentation method according to one embodiment of the present disclosure.

[0063] Figure 4 may show an example of data augmented according to the data augmentation method of Figures 2 and 3.

[0064] Referring to FIG. 4, the resolution of the log-mel spectrogram is adjusted by changing the sampling rate when the window size is the same (410), and the resolution of the log-mel spectrogram is adjusted by fixing the sampling rate and changing only the window size (420).

[0065] When the window size is the same, the resolution of the log-mel spectrogram is adjusted by changing the sampling rate (410). The window size is fixed at 64ms, and then the reference log-mel spectrogram (410b) is set to 16kHz to perform downsampling (410a) and upsampling (410c). It can be confirmed that there is a difference in the maximum frequency limit depending on the sampling rate.

[0066] In the case where the resolution of the log-mel spectrogram is adjusted by fixing the sampling rate and changing only the window size (420), the sampling rate is fixed at 16 kHz, and then the reference log-mel spectrogram (420b) is set to 64 ms to perform downsampling (420a) and upsampling (420c). Depending on the window size, there is a difference in frequency resolution or time resolution.

[0068] FIG. 5 shows a spectrum analysis according to window size according to one embodiment of the present disclosure.

[0069] Referring to FIG. 5, an analysis of the output spectrum is shown when the data is augmented by varying the window size according to one embodiment of the present disclosure. Spectral results are shown when the range of variation of the window size is reduced (510) and when the range of variation of the window size is increased (530), centered around the reference case (520). By determining at which point the difference between the spectra becomes severe when the window size is divided by or multiplied by the r value, the sound onset point can be identified. In the case where the range of variation of the window size is reduced (510), the r value is set to 1.25, and as a result, the difference between the spectra becomes severe in one frame, so it can be recognized that sound occurred in that part. In the case of the reference case (520), the r value is set to 1.5, and the difference becomes severe in two frames, so it can be recognized that sound occurred during two frames. In the case where the range of variation of the window size is increased (530), the r value is set to 2, so the frame where the difference becomes severe appears in three frames, and it can be confirmed when the sound occurred during three frames.

[0070] When observing the results, the onset and offset of a sound source can be accurately detected through the embodiments of the present disclosure. At the point where the sound source originates, the spectrum patterns exhibit different patterns, while in the intervals outside the point where the sound source originates, similar patterns are observed. In other words, it can be confirmed that the data augmentation technique is effective in detecting the onset and offset by causing disturbances at the origin and offset points of the sound source.

[0072] FIG. 6 shows a probability distribution usable for data augmentation according to one embodiment of the present disclosure.

[0073] Referring to FIG. 6, various probability distributions can be used to extract the sampling rate and window size. The types of probability distributions include the log-uniform distribution (610), the log-Gaussian distribution (620), and the log-beta distribution (630). The log-uniform distribution (610) indicates that it has the same probability regardless of changes in the sampling rate and window size, while the log-Gaussian distribution (620) has the highest probability when the reference spectrogram is 16 kHz and 128 ms, and tends to have a decreasing probability towards the end. Conversely to the log-Gaussian distribution (620), the log-beta distribution (630) has the lowest probability when the reference spectrogram is 16 kHz and 128 ms, and tends to have a decreasing probability towards the end.

[0075] FIG. 7 is a diagram showing the structure of an artificial neural network according to one embodiment of the present invention.

[0076] The structure of the artificial neural network in Fig. 7 may be the same as the structure of the student network and teacher network in Figs. 2 to 5.

[0077] Referring to FIG. 7, data (710) augmented using a data augmentation technique according to one embodiment of the present disclosure can be input to a CNN (720). For example, the CNN (720) can be a ResNet with block count = {3, 3, 3, 1}, stack = {2, 2, 2, 2}, CNN channels = {8, 16, 32, 64}), a BCResNet (initial channel count = 50), or a FUSE (initial channel count = 32, block count = 3). The CNN (720) can be replaced with an EfficientNetv2. The data output from the CNN (720) can be input to an RNN (730). For example, the RNN (730) can be one-sided gated recurrent units (GRU, layer count = 2, hidden size = 10).

[0079] FIG. 8 is a diagram showing verification loss according to the type of algorithm according to various embodiments of the present disclosure.

[0080] Figure 8 shows the change in validation loss according to the number of epochs using a total of six artificial intelligence algorithm methods. In Figure 8, the x-axis represents the number of epochs, and the y-axis represents the validation loss.

[0081] Referring to Fig. 8, it can be seen that the FixMatch (860) method and Unsupervised Data Augmentation (UDA) (850), which are frequently used in semi-supervised learning, do not show a trend of steadily decreasing loss as the number of epochs increases, but rather show a pattern of decreasing up to 50 times and then gradually increasing again.

[0082] Next, it can be seen that the rising trend of losses in ICT (840) and SCT (830) is smaller than that of FixMatch (860) or UDA (850), but still decreases with increasing epochs and then rises again.

[0083] Next, it can be seen that MT (820), one of the semi-supervised learning methods, has a loss that decreases up to a certain epoch, and then maintains a relatively small increase.

[0084] Finally, in the case of the RCT (Resolution Consistency Training) (810) method, which is one of the methods proposed in this disclosure, as the epoch increases, the loss decreases steadily up to a certain point and then does not rise again, and the loss is maintained, showing a very small range of loss.

[0085] Therefore, it can be seen that learning is best and performance is relatively superior as one moves from FixMatch (860) to the RCT (810) method.

[0087] FIG. 9 is a table showing the structure of a data set according to one embodiment of the present disclosure. FIG. 10 is a table showing performance according to various algorithm methods according to various embodiments of the present disclosure.

[0088] FIG. 9 illustrates a dataset used for training and experimenting with models according to one embodiment of the present disclosure. The dataset used for model building is the DESED database, which is frequently used in acoustic event detection experiments. The DESED database consists of audio clips that are 10 seconds long and may be 44.1 kHz or 16 kHz.

[0089] Referring to FIG. 9, audio data can be classified into three types. It consists of data with labels at each time level (frame-level or strongly labeled data) (905), data with labels at each clip level (clip-level or weakly labeled data) (910), and data without labels (unlabeled) (915). Additionally, it may consist of actual recorded data and data in which a sound source is synthesized with background sound.

[0090] Referring to FIG. 9, the frame-level (905) data may contain 3,470 actual recorded sounds in the train area and 1,168 in the Eval. area, and 10,000 theoretical sounds. For the clip-level (910), there may be 1,578 actual recorded sounds in the train area. The unlabeled (915) data may contain 14,412 actual recorded sounds in the train area.

[0091] FIG. 10 may show the results of an experiment to verify the performance of voice event detection according to various embodiments of the present disclosure.

[0092] For the six algorithm methods shown in Fig. 8, the event-f1 (1040), intersection-f1 (1050), PSDS1 (1060), and PSDS2 (1070) methods were evaluated as performance evaluation methods. Here, the event-f1 (1040) and PSDS1 (1060) methods are mainly used to test the performance of detecting onset and offset, while the intersection-f1 (1050) and PSDS2 (1070) methods are mainly used to test the performance of detecting the presence or absence of sound.

[0093] Referring to FIG. 10, it can be seen that MT (1005) showed the best performance in PSDS2 (1070), but was not effective in EVENT-F1 (1040) or PSDS1 (1060). It can be seen that the RCT (1030) method showed the most effective performance in all tests except PSDS2 (1070). In particular, it can be seen that it showed superior performance compared to other algorithm methods in event-f1 (1040) and PSDS1 (1060), which are indicators of voice event detection performance that detect onset and offset.

[0095] FIG. 11 is a block diagram of an electronic device for detecting voice events according to one embodiment of the present disclosure.

[0096] Referring to FIG. 11, the electronic device (1110) may include a modem (MODEM, 1120), a memory (MEMORY, 1140), and a processor (PROCESSOR, 1130).

[0097] The modem (1120) may be a communication modem that is electrically connected to other electronic devices to enable mutual communication. In particular, the modem (1120) may receive data input and transmit it to a processor (1130), and the processor (130) may store the input data value in memory (1140). Additionally, the system may transmit the output data value to another electronic device by an artificial intelligence algorithm learned from the system.

[0098] The memory (1140) is a configuration in which various information and program instructions for the operation of the electronic device (1110) are stored, and may be a storage device such as a hard disk or a solid state drive (SSD). In particular, the memory (1140) may store one or more data input values ​​input from the modem (1120) under the control of the processor (1130). Additionally, the memory (1140) may store program instructions such as an artificial intelligence algorithm for direction estimation that can be executed by the processor (1130). Additionally, the memory (1140) may store programs (or program instructions) such as artificial intelligence algorithms learned through the curriculum learning method, deep supervision method, or deeply supervised curriculum learning described in the present disclosure.

[0099] The processor (1130) is composed of at least one processor and can learn an artificial intelligence algorithm related to voice event detection using data and program instructions stored in memory (1140) and calculate data using the same. The processor (1130) can control and calculate all artificial intelligence algorithms described in FIGS. 1 to 7. The processor (1130) can then perform operations for the method described in FIG. 12.

[0101] FIG. 12 is a flowchart illustrating a voice event detection method according to an embodiment of the present invention.

[0102] With reference to FIG. 12, the learning operation of the artificial intelligence algorithm of the electronic device and the voice event detection method described with reference to FIG. 1 to FIG. 11 will be summarized and explained below. Each operation is not an operation that must be necessarily included in a series of processes, and only some of them may be configured and operated depending on the situation.

[0103] In step S1210, voice data may be received to estimate voice events through artificial intelligence. The voice data may be utilized as input information to train the artificial intelligence. The voice data may include labels that can be interpreted as correct answers in the artificial intelligence training. The artificial intelligence may be trained in different ways depending on whether labels are included or not.

[0104] In step S1220, a plurality of first data representing a plurality of resolutions can be determined based on at least one of a sampling rate and a window size of the received voice data. When a strong augmentation technique is applied among the data augmentation techniques, the data can be augmented by changing both the sampling rate and the window size, and when a weak augmentation technique is applied, data augmentation can be performed based on at least one of the sampling rate and the window size. The sampling rate and the window size can be extracted based on a log-Gaussian distribution.

[0105] In step S1230, a first artificial intelligence network that has been trained can output second data using a plurality of first data as input information. The second data may include information regarding onset and offset. Based on the second data, specific voice data can be detected from the received voice data.

[0106] In step S1240, the previously trained first artificial intelligence network can be trained based on the plurality of first data and the third data output from the second artificial intelligence network. It is obvious that the first artificial intelligence network is not trained only in the above manner, but can be trained through various methods described in FIGS. 2 and 3. Here, the first artificial intelligence network is a student network used in the knowledge distillation technique, and the second artificial intelligence network may be a teacher network.

[0108] Although the technical concept of the present invention has been described in detail with reference to various embodiments, the technical concept of the present invention is not limited to the above embodiments, and various modifications and changes are possible by those skilled in the art within the scope of the technical concept of the present invention. Explanation of the symbols

[0110] 110: Artificial Intelligence Module 1110 : Electronic device 1120 : Modem 1130 : Processor 1140: Memory

Claims

Claim 1 A method performed by an electronic device using artificial intelligence, comprising: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on a sampling rate and a window size of the voice data; outputting a second data from a first artificial intelligence network that has been trained using the plurality of first data as input information; if the voice data does not include a label, determining a plurality of fourth data representing a plurality of resolutions based on the sampling rate of the voice data; and determining a loss between a fifth data output from a third artificial intelligence network and the second data using the plurality of fourth data as input, wherein the first artificial intelligence network that has been trained is trained based on the plurality of first data and the third data output from the second artificial intelligence network, and the third artificial intelligence network is set based on the parameter values ​​of the first artificial intelligence network, but is fixed so that the parameters are not updated with respect to the input of the fourth data. Claim 2 A method according to claim 1, further comprising: a step of determining a loss between the second data and the third data; and a step of determining a parameter that minimizes the loss. Claim 3 A method according to claim 1, wherein, when the voice data includes a label, the first artificial intelligence network further comprises the step of determining a loss between the label and the second data. Claim 4 delete Claim 5 A method according to claim 1, wherein the step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data comprises: a step of extracting at least one of the sampling rate and a window size based on a log-Gaussian distribution; and a step of determining the plurality of first data by applying at least one of the extracted sampling rate and a window size to the voice data. Claim 6 A method according to claim 1, further comprising the step of detecting specific voice data from the voice data based on the second data. Claim 7 A method according to claim 1, wherein the first artificial intelligence network is learned from the second artificial intelligence network through a knowledge distillation process, the first artificial intelligence network is a student network, and the second artificial intelligence network is a teacher network. Claim 8 An electronic device comprises: a memory; a modem; and a processor connected to the modem and the memory, wherein the processor is configured to receive voice data, determine a plurality of first data representing a plurality of resolutions based on a sampling rate and a window size of the voice data, and output second data from a pre-trained first artificial intelligence network using the plurality of first data as input information, wherein if the voice data does not include a label, determine a plurality of fourth data representing a plurality of resolutions based on the sampling rate of the voice data, and determine a loss between the second data and a fifth data output from a third artificial intelligence network using the plurality of fourth data as input, wherein the pre-trained first artificial intelligence network is trained based on the plurality of first data and the third data output from the second artificial intelligence network, and the third artificial intelligence network is set based on the parameter values ​​of the first artificial intelligence network, but is fixed so that the parameters are not updated for the input of the fourth data. Electronic device. Claim 9 In claim 8, the processor is further configured to determine the loss between the second data and the third data, and to determine a parameter that minimizes the loss. Claim 10 In claim 8, the processor is an electronic device further configured such that, when the voice data includes a label, the first artificial intelligence network determines the loss between the label and the second data. Claim 11 delete Claim 12 In claim 8, the processor is configured to: extract at least one of the sampling rate and window size based on a log-Gaussian distribution, and apply at least one of the extracted sampling rate and window size to the voice data to determine the plurality of first data. Claim 13 In paragraph 8, the processor is: an electronic device further configured to detect specific voice data from the voice data based on the second data. Claim 14 An electronic device according to claim 8, wherein the first artificial intelligence network is learned from the second artificial intelligence network through a knowledge distillation process, the first artificial intelligence network is a student network, and the second artificial intelligence network is a teacher network. Claim 15 A program stored in a medium for performing an acoustic event detection method through an artificial intelligence algorithm executable by a processor, comprising: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on a sampling rate and a window size of the voice data; outputting a second data from a first artificial intelligence network that has been trained using the plurality of first data as input information; if the voice data does not include a label, determining a plurality of fourth data representing a plurality of resolutions based on the sampling rate of the voice data; and determining a loss between a fifth data output from a third artificial intelligence network and the second data using the plurality of fourth data as input, wherein the first artificial intelligence network that has been trained is trained based on the plurality of first data and the third data output from the second artificial intelligence network, and the third artificial intelligence network is set based on the parameter values ​​of the first artificial intelligence network, but is fixed so that the parameters are not updated with respect to the input of the fourth data.

Images