Method for detecting acoustic event using artificial intelligence, and device therefor

The method addresses the inefficiencies in acoustic event detection by preprocessing audio data to derive various resolutions and applying semi-supervised learning techniques, enhancing the detection of specific sound events with improved accuracy and resolution.

WO2024228508A1PCT designated stage expired Publication Date: 2024-11-07INDUSTRY UNIVERSITY COOPERATION FOUNDATION HANYANG UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2024/005434
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-03
Filing Date
2024-04-22
Publication Date
2024-11-07

Smart Images

  • Figure KR2024005434_07112024_PF_FP_ABST
    Figure KR2024005434_07112024_PF_FP_ABST
Patent Text Reader

Abstract

A method performed by an electronic device using artificial intelligence, according to an embodiment of the present invention, comprises the steps of: receiving voice data; with respect to the voice data, determining a plurality of pieces of first data representing a plurality of resolutions on the basis of at least one of a sampling rate and a window size; and outputting second data in a first artificial intelligence network that has been pre-trained using the plurality of pieces of first data as input information, wherein the pre-trained first artificial intelligence network may be trained on the basis of the plurality of pieces of first data and third data outputted in a second artificial intelligence network.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for acoustic event detection using artificial intelligence

[0001] The present disclosure describes a method and device for detecting acoustic events (or incidents) using artificial intelligence. Specifically, it describes a method for training artificial intelligence to utilize audio event detection technology to detect the location where a specific sound source originates.

[0002] The present disclosure relates to a device and method for detecting a specific sound source (or acoustic event) in a speech signal. Acoustic event detection technology is a technology for detecting specific sounds (e.g., a bell sound, a dog barking) at each time point. In particular, it has the advantage of being able to detect and utilize only specific sounds selected by the user among various sounds occurring in real life. Acoustic event detection technology using artificial intelligence performs preprocessing on general acoustic data, extracts the characteristics of each acoustic event from the preprocessed acoustic data, and trains an algorithm model on these characteristics.

[0003] However, building a system for AI-based acoustic event detection requires audio data labeled with correct answers every hour. Labeling audio data used for training requires the trainee to manually label each sound, which is time-consuming and expensive. Therefore, a method capable of effectively training data even in situations where labeling information is absent or insufficient is needed.

[0004] In particular, semi-supervised learning has been the dominant approach in acoustic event detection technology. Semi-supervised learning is primarily used for image data, with methods such as mean teacher and FixMatch being widely used. Semi-supervised learning, based on data augmentation techniques, ensures that the neural network's learning output remains the same as the original image, even if the image is slightly distorted or perturbed. However, while the x and y axes in image data are defined by position, in audio spectrograms, the x and y axes are defined by time and frequency, respectively. This difference creates problems: the meaning of the x and y axes differ, and the corresponding correct answer value changes when the time axis of the data is disturbed. Therefore, when applying learning methods used for image data to audio data, a method is needed to maintain consistency, as perturbations in the time domain can alter the correct answer value. Therefore, consistent learning methods utilizing data augmentation techniques can be used in semi-supervised learning. Below, we propose a method that utilizes consistency learning based on resolution.

[0005] The present disclosure provides a method and device for improving an acoustic event detection function by deriving various resolutions through a data preprocessing method.

[0006] In this disclosure, we aim to provide a method for improving acoustic event detection capability by setting different degrees of enhancement for unlabeled data.

[0007] A method performed by an electronic device using artificial intelligence according to an embodiment of the present invention comprises: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data; and outputting second data from a first artificial intelligence network trained using the plurality of first data as input information, wherein the first artificial intelligence network trained can be trained based on the plurality of first data and third data output from the second artificial intelligence network.

[0008] In one embodiment, the method may further include: determining a loss between the second data and the third data; and determining a parameter that minimizes the loss.

[0009] In one embodiment, if the speech data includes a label, the first artificial intelligence network may further include a step of determining a loss between the label and the second data.

[0010] In one embodiment, if the voice data does not include a label, the method may further include a step of determining a plurality of fourth data representing a plurality of resolutions based on a sampling rate of the voice data, and a step of determining a loss between fifth data output from a third artificial intelligence network using the plurality of fourth data as input and the second data.

[0011] In one embodiment, the step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the speech data may include: a step of extracting at least one of the sampling rate and the window size based on a log-Gaussian distribution; and a step of applying at least one of the extracted sampling rate and the window size to the speech data to determine the plurality of first data.

[0012] In one embodiment, the method may further include a step of detecting specific voice data from the voice data based on the second data.

[0013] In one embodiment, the first artificial intelligence network may be trained from the second artificial intelligence network through a knowledge distillation process, the first artificial intelligence network may be a student network, and the second artificial intelligence network may be a teacher network.

[0014] In an embodiment of the present invention, an electronic device comprises: a memory; a modem; and a processor connected to the modem and the memory, wherein the processor is configured to: receive voice data, determine a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data, and output second data from a first artificial intelligence network trained using the plurality of first data as input information, and the first artificial intelligence network trained can be trained based on the plurality of first data and third data output from the second artificial intelligence network.

[0015] A program stored in a medium for performing a direction estimation method through an artificial intelligence algorithm executable by a processor according to an embodiment of the present invention, comprising: receiving voice data; determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data; and outputting second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, wherein the first artificial intelligence network that has been trained can be trained based on the plurality of first data and third data output from the second artificial intelligence network.

[0016] According to one embodiment of the present disclosure, the overfitting problem can be alleviated and the acoustic event detection function can be improved by deriving various resolutions through a data preprocessing method.

[0017] To more fully understand the drawings cited in the detailed description of the present disclosure, a brief description of each drawing is provided.

[0018] Figure 1 is a conceptual diagram illustrating the basic principles of an artificial intelligence structure according to one embodiment of the present disclosure.

[0019] FIG. 2 is a diagram showing the structure of an artificial intelligence network according to one embodiment of the present disclosure.

[0020] FIG. 3 is a diagram showing the structure of an artificial intelligence network according to another embodiment of the present disclosure.

[0021] FIG. 4 illustrates a spectrogram generated according to a data augmentation method according to one embodiment of the present disclosure.

[0022] FIG. 5 illustrates a spectrum analysis according to a window size according to one embodiment of the present disclosure.

[0023] FIG. 6 illustrates a probability distribution usable for data augmentation according to one embodiment of the present disclosure.

[0024] FIG. 7 is a diagram showing the structure of an artificial neural network according to one embodiment of the present disclosure.

[0025] FIG. 8 is a diagram showing verification loss according to the type of algorithm according to various embodiments of the present disclosure.

[0026] FIG. 9 is a table showing the structure of a data set according to one embodiment of the present disclosure.

[0027] FIG. 10 is a table showing performance according to various algorithm methods according to various embodiments of the present disclosure.

[0028] FIG. 11 is a block diagram of an electronic device for voice event detection according to one embodiment of the present disclosure.

[0029] FIG. 12 is a flowchart illustrating a voice event detection method according to one embodiment of the present disclosure.

[0030] The technical concept of the present invention is susceptible to various modifications and various embodiments. Specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technical concept of the present invention to specific embodiments, and it should be understood that all modifications, equivalents, and alternatives fall within the scope of the technical concept of the present invention.

[0031] When explaining the technical concepts of the present invention, detailed descriptions of related known technologies will be omitted if they are deemed to unnecessarily obscure the gist of the present invention. Furthermore, numbers (e.g., "first," "second," etc.) used throughout the description of this specification are merely identifiers used to distinguish one component from another.

[0032] Additionally, in this specification, when a component is referred to as being “connected” or “connected” to another component, it should be understood that the component may be directly connected or connected to the other component, but may also be connected or connected via another component in between, unless there is a specific description to the contrary.

[0033] In addition, terms such as “~part”, “~device”, “~sub-subject”, and “~module” described in this specification mean a unit that processes at least one function or operation, which may be implemented by hardware such as a processor, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), an accelerate processor unit (APU), a drive signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a combination of hardware and software, and may also be implemented in a form combined with a memory that stores data necessary for processing at least one function or operation.

[0034] And it is to be clarified that the division of components in this specification is only a division based on the main function of each component. In other words, two or more components described below may be combined into one component, or one component may be further divided into two or more components with more detailed functions. In addition to its own main function, each component described below may additionally perform some or all of the functions of other components, and of course, some of the main functions of each component may be exclusively performed by other components.

[0035] In describing embodiments of the present disclosure, detailed descriptions of related functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the disclosure. Furthermore, the terms described below are defined based on the functions of the present disclosure and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.

[0036] For the same reason, some components in the attached drawings may be exaggerated, omitted, or schematically depicted. Furthermore, the dimensions of each component do not entirely reflect its actual size. Identical or corresponding components in each drawing are assigned the same reference numbers.

[0037] The advantages and features of the present disclosure, and methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The embodiments are provided solely to ensure that the description of the present disclosure is complete and to fully inform those skilled in the art of the scope of the invention, and the claimed scope of the present disclosure is defined solely by the scope of the claims.

[0038] At this time, it will be understood that each block of the drawings showing the processing flowchart and the combination of the processing flowchart drawings can be performed by computer program instructions. These computer program instructions can be installed in a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions executed through the processor of the computer or other programmable data processing equipment create a means for performing the functions described in the flowchart block(s). These computer program instructions can also be stored in a computer-available or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement the functions in a specific manner, so that the instructions stored in the computer-available or computer-readable memory can also produce a manufactured item that includes an instruction means for performing the functions described in the flowchart block(s). Since the computer program instructions may be installed on a computer or other programmable data processing device, a series of operational steps may be performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for performing the functions described in the flowchart block(s) may also provide steps for performing the functions described in the flowchart block(s).

[0039] Additionally, each block may represent a module, segment, or portion of code that contains one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative implementation examples, the functions described in the blocks may occur out of order. For example, two blocks depicted in succession may actually be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on their respective functions.

[0040] The term 'unit or part' used in this disclosure means a software or hardware component such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the 'unit' may be configured to perform specific roles. However, the 'unit' is not limited to software or hardware. The 'unit' may be configured to reside in an addressable storage medium and may be configured to execute one or more processors. Thus, as an example, the 'unit' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functionality provided within the components and 'units' may be combined into a smaller number of components and 'units' or further separated into additional components and 'units'. Additionally, the components and '~parts' may be implemented to activate one or more CPUs within a device or secure multimedia card. In addition, in an embodiment, the '~parts' may include one or more processors and / or devices.

[0041] Hereinafter, embodiments according to the technical idea of ​​the present invention will be described in detail one by one.

[0042]

[0043] Figure 1 is a conceptual diagram illustrating the basic principles of an artificial intelligence structure according to one embodiment of the present disclosure.

[0044] Referring to Figure 1, the basic principles by which learning is performed in an artificial intelligence structure are illustrated.

[0045] Artificial intelligence (AI) technology primarily addresses cognitive problems associated with human intelligence, such as learning, problem solving, and perception. AI can be learned through machine learning (ML) and deep learning (DL). Machine learning primarily involves techniques for pattern recognition and learning, and refers to algorithms that learn from recorded data and predict future data based on that data. It refers to a technology that learns from data without relying on predefined rules or patterns. Deep learning, on the other hand, is a branch of machine learning that differs from machine learning in that it processes data based on artificial neural networks (ANNs). Because deep learning utilizes artificial neural networks, it can handle more complex and sophisticated calculations than machine learning. Algorithms for deep learning include convolutional neural networks (CNNs), deep neural networks (DNNs), artificial neural networks (ANNs), and recurrent neural networks (RNNs).

[0046] Referring to FIG. 1, the artificial intelligence structure can be represented by an artificial intelligence module (110). The artificial intelligence module (110) receives predetermined input data (105), performs learning through a method predetermined in the module, and outputs output data (115) regarding the learning result. According to one embodiment, the input data (105) may include predetermined data (e.g., pictures, sounds, etc.), voice signals, and input sequences. The output data (115) may include an output sequence, an enhanced voice signal, voice event information (e.g., a specific voice), etc.

[0047]

[0048] FIG. 2 is a diagram showing the structure of an artificial intelligence network according to one embodiment of the present disclosure.

[0049] The artificial intelligence algorithm used in Fig. 2 may be one of the types of the artificial intelligence module (110) of Fig. 1. Fig. 2 may be a drawing showing an artificial intelligence algorithm when input voice data includes a label.

[0050] Referring to FIG. 2, the structure of the artificial intelligence system may include voice data and an artificial intelligence algorithm structure.

[0051] According to one embodiment, the speech data may refer to speech data including a label or speech data not including a label.

[0052] According to one embodiment, the artificial intelligence algorithm structure may be divided into stages such as data preprocessing, artificial intelligence learning, and output data verification. Data preprocessing in the present disclosure may use a data augmentation technique. The data augmentation technique is a method of augmenting data by varying the sampling rate of the audio signal and the window size, which is an operating variable of the short-time Fourier transform. The data augmentation technique may include a strong augmentation technique and a weak augmentation technique. The strong augmentation technique may be applied by randomly extracting the sampling rate and window size from a log-Gaussian distribution. For example, the mean of the log-Gaussian distribution may be the sampling rate and window size of the reference spectrogram (i.e., the non-augmented spectrogram), and the standard deviation may be set to 1.25. The values ​​randomly extracted from the log-Gaussian distribution may be limited to 1 / 2 or 2 to prevent excessive distortion. Weak augmentation uses the same window size as the reference spectrogram, but extracts and uses the same sampling rate as the strong augmentation technique. For example, the reference spectrogram uses a sampling rate of 16 kHz and a window size of 128 ms.

[0053] Here, a spectrogram is a two-dimensional graph that visually represents frequency components used to analyze audio signals. In particular, a log-Mel spectrogram is a spectrogram converted to a MEL scale in the time-frequency domain and processed logarithmically.

[0054] The artificial intelligence system of FIG. 2 may be composed of a student network (220) and a teacher network (225) formed by an exponential moving average (EMA) of student network parameters that are updated at each learning step. The technique used here is a knowledge distillation technique, which is a form in which the teacher network, which is generally a large and complex model in deep learning, transfers information to the student network, which is a small and simple model. The goal of knowledge distillation is to maximize the performance of the teacher network in the student model while reducing the size and computational complexity of the model. In the knowledge distillation technique, the teacher network is trained first, and then the student network is trained based on the resulting data from the teacher network, and their performances are compared.

[0055] Referring to FIG. 2, a strong enhancement technique (215) can be applied to voice data (210) including a label (240). According to the strong enhancement technique, data having various resolutions can be extracted by changing at least one of the sampling rate and window size.

[0056] Augmented data (215) based on a strong augmentation technique is used as input information for the student network (220) and the teacher network (225), respectively, and consistency loss (230) can be calculated between the result data according to the student network (220) and the result data according to the teacher network (225). The calculated consistency loss (230) can be used again to optimize the student network (220). The consistency loss can be calculated as the mean square error.

[0057] Additionally, a supervised loss (235) based on supervised learning can be calculated between the label (240) and the result data obtained from the student network (220) in the speech data (210) including the label (240). The calculated supervised loss (235) can also be used to optimize the student network (220) along with the consistency loss (230). The supervised loss can be calculated using binary cross entropy.

[0058] According to an embodiment of the present disclosure, the student network (220) can be trained as an artificial intelligence network with similar performance to the teacher network (225) using a lightweight model. The artificial intelligence algorithm according to the present disclosure can detect specific speech data (or speech events) from speech data.

[0059]

[0060] FIG. 3 is a diagram showing the structure of an artificial intelligence network according to another embodiment of the present disclosure.

[0061] The artificial intelligence algorithm used in Fig. 3 may be one of the types of the artificial intelligence module (110) of Fig. 1. Fig. 3 may be a diagram showing an artificial intelligence algorithm when input voice data does not include a label.

[0062] Referring to FIG. 3, the structure of the artificial intelligence system may include voice data and an artificial intelligence algorithm structure.

[0063] According to one embodiment, the speech data may refer to speech data including a label or speech data not including a label.

[0064] The artificial intelligence system of FIG. 3 may be composed of a student network (320) and a teacher network (325) composed of an exponential moving average (EMA) of student network parameters that are updated at each learning step.

[0065] Referring to FIG. 3, a strong augmentation technique (315) can be applied to speech data (310) that does not include a label.

[0066] Augmented data (315) based on a strong augmentation technique is used as input information for the student network (320) and the teacher network (325), respectively, and consistency loss (330) can be calculated between the result data according to the student network (320) and the result data according to the teacher network (325). The calculated consistency loss (330) can be used again to optimize the student network (320).

[0067] In addition, the student network (320) value can be used in speech data (310) that does not include a label to perform a role similar to a label. The fixed student network (345) can be set by determining the value in the student network (320), and the data (340) augmented by a weak augmentation technique on the speech data (310) can be used as input information to extract output data. According to the weak augmentation technique, the window size can be fixed and the sampling rate can be changed to extract data with various resolutions. Here, the fixed network refers to a network that does not calculate a gradient for the input augmented data.

[0068] A supervised loss (335) based on supervised learning can be calculated between the output data of the fixed student network (345) and the result data obtained from the student network (320). The calculated supervised loss (335) can also be used to optimize the student network (320) along with the consistency loss (330).

[0069] According to an embodiment of the present disclosure, the student network (320) can be trained as an artificial intelligence network with similar performance to the teacher network (325) as a lightweight model.

[0070] FIG. 4 illustrates a spectrogram generated according to a data augmentation method according to one embodiment of the present disclosure.

[0071] FIG. 4 may illustrate an example of data augmented according to the data augmentation method of FIGS. 2 and 3.

[0072] Referring to FIG. 4, the case where the resolution of the log-mel spectrogram is adjusted by changing the sampling rate when the window size is the same (410) and the case where the resolution of the log-mel spectrogram is adjusted by fixing the sampling rate and changing only the window size (420) are shown.

[0073] When the resolution of the log-mel spectrogram was adjusted by changing the sampling rate while the window size was the same (410), the window size was fixed to 64 ms, and downsampling (410a) and upsampling (410c) were performed by setting the reference log-mel spectrogram (410b) to 16 kHz. It can be confirmed that there is a difference in the maximum frequency limit depending on the sampling rate.

[0074] In the case where the resolution of the log-mel spectrogram was adjusted by fixing the sampling rate and changing only the window size (420), the sampling rate was fixed to 16 kHz, and downsampling (420a) and upsampling (420c) were performed by setting the reference log-mel spectrogram (420b) to 64 ms. Depending on the window size, a difference occurred in the frequency resolution or time resolution.

[0075]

[0076] FIG. 5 illustrates a spectrum analysis according to a window size according to one embodiment of the present disclosure.

[0077] Referring to FIG. 5, an analysis of the output spectrum is illustrated when data is augmented by varying the window size according to one embodiment of the present disclosure. The spectrum results are illustrated when the window size variation is reduced (510) and increased (530) with respect to the reference case (520). In the case of dividing and multiplying the window size by the r value, the point at which the difference between the spectra becomes severe can be used to identify the point of sound generation (onset). In the case of reducing the window size variation (510), the r value is set to 1.25, and as a result, the difference between the spectra becomes severe in one frame, so that it can be recognized that sound has occurred at that part. In the case of the reference case (520), the r value is set to 1.5, and the difference becomes severe in two frames, so that it can be recognized that sound has occurred during two frames. In the case of increasing the window size variation (530), the r value is set to 2, so that the frames in which the difference becomes severe appear as three frames, and it can be identified when the sound has occurred during the three frames.

[0078] The results demonstrate that the present disclosure can accurately detect the onset and offset of a sound source. Spectral patterns differ at the point where the sound source originates, while similar patterns are observed in sections outside the source area. This demonstrates the effectiveness of data augmentation techniques in detecting onset and offset by inducing disturbances at the origin and end points of the sound source.

[0079]

[0080] FIG. 6 illustrates a probability distribution usable for data augmentation according to one embodiment of the present disclosure.

[0081] Referring to FIG. 6, various probability distributions can be used to extract the sampling rate and window size. The types of probability distributions include the log-uniform distribution (610), the log-Gaussian distribution (620), and the log-beta distribution (630). The log-uniform distribution (610) indicates that it has the same probability regardless of changes in the sampling rate and window size, and the log-Gaussian distribution (620) has the highest probability when the reference spectrogram is 16 kHz and 128 ms and has a tendency to have a probability that gradually decreases toward the end. The log-beta distribution (630) has the lowest probability when the reference spectrogram is 16 kHz and 128 ms and has a tendency to have a probability that gradually increases toward the end, in contrast to the log-Gaussian distribution (620).

[0082]

[0083] Figure 7 is a diagram showing the structure of an artificial neural network according to one embodiment of the present invention.

[0084] The structure of the artificial neural network of Fig. 7 may be the same as the structure of the student network and teacher network of Figs. 2 to 5.

[0085] Referring to FIG. 7, data (710) augmented using a data augmentation technique according to an embodiment of the present disclosure can be input to a CNN (720). For example, the CNN (720) can be a ResNet (number of blocks = {3, 3, 3, 1}, stack = {2, 2, 2, 2}, CNN channels = {8, 16, 32, 64}), BCResNet (initial number of channels = 50), FUSE (initial number of channels = 32, number of blocks = 3). The CNN (720) can be used instead of an EfficientNetv2. The data output from the CNN (720) can be input to an RNN (730). For example, the RNN (730) can be one-sided gated recurrent units (GRU, number of layers = 2, hidden size = 10).

[0086]

[0087] FIG. 8 is a diagram showing verification loss according to the type of algorithm according to various embodiments of the present disclosure.

[0088] Figure 8 illustrates the variation in validation loss according to the number of epoch trials using a total of six artificial intelligence algorithm methods. In Figure 8, the x-axis represents the number of epoch trials, and the y-axis represents the validation loss.

[0089] Referring to Figure 8, it can be seen that the FixMatch (860) method and Unsupervised Data Augmentation (UDA) (850), which are frequently used in semi-supervised learning, do not show a trend of steady decrease in loss as the number of epochs increases, but rather show a pattern of decreasing up to 50 times and then gradually increasing again.

[0090] Next, it can be seen that ICT (840) and SCT (830) show a smaller increase in loss than FixMatch (860) or UDA (850), but still show a pattern of decreasing and then increasing again as the epoch increases.

[0091] Next, it can be confirmed that MT (820), one of the semi-supervised learning methods, has a relatively small increase after the loss has decreased for a certain number of epochs.

[0092] Lastly, in the case of the RCT (Resolution Consistency Training) (810) method, which is one of the methods proposed in this disclosure, it can be confirmed that as the epoch increases, the loss steadily decreases to a certain section and then does not increase again, but is maintained, showing that the range of the loss is very small.

[0093] Therefore, it can be confirmed that learning is the best and performance is relatively excellent as you move from FixMatch (860) to RCT (810).

[0094]

[0095] FIG. 9 is a table showing the structure of a data set according to one embodiment of the present disclosure. FIG. 10 is a table showing performance according to various algorithmic methods according to various embodiments of the present disclosure.

[0096] Figure 9 illustrates a dataset used for training and experimentation of models according to one embodiment of the present disclosure. The dataset used for model building is the DESED database, which is frequently used in acoustic event detection experiments. The DESED database consists of 10-second audio clips, which can be at 44.1 kHz or 16 kHz.

[0097] Referring to Figure 9, audio data can be divided into three types: data with labels on an hourly basis (frame-level or strongly labeled data) (905), data with labels on a clip-level basis (clip-level or weakly labeled data) (910), and data without labels (unlabeled) (915). Furthermore, the data may be composed of actual recorded data and data synthesized from sound sources with background sound.

[0098] Referring to Figure 9, frame-level (905) data can contain 3,470 train regions and 1,168 Eval regions of actual recorded sounds, and 10,000 theoretical sounds. Clip-level (910) data can contain 1,578 train regions of actual recorded sounds. Unlabeled (915) data can contain 14,412 train regions of actual recorded sounds.

[0099] FIG. 10 may represent results of an experiment to verify the performance of voice event detection according to various embodiments of the present disclosure.

[0100] The performance evaluation methods for the six algorithm methods shown in Fig. 8 were event-f1 (1040), intersection-f1 (1050), PSDS1 (1060), and PSDS2 (1070), respectively. Here, the event-f1 (1040) and PSDS1 (1060) methods are mainly used to test the performance of detecting onset and offset, and the intersection-f1 (1050) and PSDS2 (1070) methods are mainly used to test the performance of detecting the presence or absence of sound.

[0101] Referring to Fig. 10, MT (1005) showed the best performance in PSDS2 (1070), but was not effective in EVENT-F1 (1040) or PSDS1 (1060). In all tests except PSDS2 (1070), RCT (1030) showed the most effective performance. In particular, it was confirmed that RCT (1030) showed superior performance compared to other algorithms in event-f1 (1040) and PSDS1 (1060), which are indicators of the performance of voice event detection that detects onset and offset.

[0102]

[0103] FIG. 11 is a block diagram of an electronic device for voice event detection according to one embodiment of the present disclosure.

[0104] Referring to FIG. 11, the electronic device (1110) may include a modem (MODEM, 1120), a memory (MEMORY, 1140), and a processor (PROCESSOR, 1130).

[0105] The modem (1120) may be a communication modem that is electrically connected to other electronic devices to enable mutual communication. In particular, the modem (1120) may receive data input and transmit it to the processor (1130), and the processor (130) may store the input data values ​​in the memory (1140). In addition, the modem (1120) may transmit data values ​​output by an artificial intelligence algorithm learned by the system to other electronic devices.

[0106] The memory (1140) is a configuration in which various information and program commands for the operation of the electronic device (1110) are stored, and may be a storage device such as a hard disk, an SSD (Solid State Drive), etc. In particular, the memory (1140) may store one or more data input values ​​input from the modem (1120) under the control of the processor (1130). In addition, the memory (1140) may store program commands such as an artificial intelligence algorithm for direction estimation executable by the processor (1130). In addition, the memory (1140) may store a program (or program command) such as an artificial intelligence algorithm learned through the curriculum learning method, deep supervision method, and deeply supervised curriculum learning described in the present disclosure.

[0107] The processor (1130) comprises at least one processor and can learn artificial intelligence algorithms related to voice event detection using data and program instructions stored in the memory (1140) and use them to calculate data. The processor (1130) can control and calculate all artificial intelligence algorithms described in FIGS. 1 to 7. The processor (1130) can then perform operations according to the method described in FIG. 12.

[0108]

[0109] Fig. 12 is a flowchart for explaining a voice event detection method according to one embodiment of the present invention.

[0110] Referring to FIG. 12 below, the learning operations and voice event detection methods of the artificial intelligence algorithm of the electronic device described with reference to FIGS. 1 to 11 are summarized and described. Each operation is not necessarily included in a series of steps, and only some of them may be configured and operated depending on the situation.

[0111] In step S1210, voice data may be received to estimate voice events using artificial intelligence. The voice data may be used as input information to train the artificial intelligence. The voice data may include labels that can be interpreted as correct answers during AI training. The artificial intelligence may be trained in different ways depending on whether the data includes labels or not.

[0112] In step S1220, a plurality of first data representing a plurality of resolutions may be determined based on at least one of a sampling rate and a window size of the received voice data. If a strong data augmentation technique is applied, data augmentation may be performed by changing both the sampling rate and the window size, and if a weak data augmentation technique is applied, data augmentation may be performed based on at least one of the sampling rate and the window size. The sampling rate and the window size may be extracted based on a log-Gaussian distribution.

[0113] In step S1230, a first artificial intelligence network trained using multiple first data as input information can output second data. The second data can include information regarding onset and offset. Based on the second data, specific voice data can be detected from the received voice data.

[0114] In step S1240, the first AI network, which has been previously trained, may be trained based on the plurality of first data and third data output from the second AI network. It should be noted that the first AI network is not trained solely through the aforementioned method, and may be trained through various methods as described in FIGS. 2 and 3. Here, the first AI network may be a student network used in the knowledge distillation technique, and the second AI network may be a teacher network.

[0115]

[0116] Above, the technical idea of ​​the present invention has been described in detail with reference to various embodiments, but the technical idea of ​​the present invention is not limited to the above embodiments, and various modifications and changes are possible by a person having ordinary knowledge in the art within the scope of the technical idea of ​​the present invention.

Claims

1. A method performed by an electronic device using artificial intelligence, Step of receiving voice data; A step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the above voice data; and It includes a step of outputting second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, A method in which the above-mentioned first artificial intelligence network is trained based on the plurality of first data and third data output from the second artificial intelligence network.

2. In paragraph 1, A step of determining a loss between the second data and the third data; and A method further comprising the step of determining a parameter that minimizes the above loss.

3. In paragraph 1, If the above voice data includes a label, A method wherein the first artificial intelligence network further comprises a step of determining a loss between the label and the second data.

4. In paragraph 1, If the above voice data does not contain a label, A step of determining a plurality of fourth data representing a plurality of resolutions based on the sampling rate of the above voice data, A method further comprising a step of determining a loss between the fifth data output from the third artificial intelligence network and the second data by using the plurality of fourth data as input.

5. In paragraph 1, The step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the above voice data is: A step of extracting at least one of the above sampling rate and window size based on a log-Gaussian distribution; and, A method comprising a step of determining the plurality of first data by applying at least one of the extracted sampling rate and window size to the voice data.

6. In paragraph 1, A method further comprising a step of detecting specific voice data from the voice data based on the second data.

7. In paragraph 1, The above first artificial intelligence network is learned from the second artificial intelligence network through a knowledge distillation process, The above first artificial intelligence network is a student network, and The above second artificial intelligence network is a teacher network.

8. In electronic devices, memory; modem; and including a processor connected to the modem and the memory; The above processor: Receive voice data, determine a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the voice data, and It is configured to output second data from a first artificial intelligence network that has been trained using the plurality of first data as input information, An electronic device in which the first artificial intelligence network learned above is learned based on the plurality of first data and third data output from the second artificial intelligence network.

9. In paragraph 8, the processor: Determine the loss between the second data and the third data, and An electronic device further configured to determine a parameter that minimizes the above loss.

10. In paragraph 8, the processor: If the above voice data includes a label, An electronic device wherein the first artificial intelligence network is further configured to determine a loss between the label and the second data.

11. In paragraph 8, the processor: If the above voice data does not contain a label, A method further configured to determine a plurality of fourth data representing a plurality of resolutions based on a sampling rate of the above voice data, and to determine a loss between fifth data output from a third artificial intelligence network and the second data using the plurality of fourth data as input.

12. In paragraph 8, the processor: At least one of the above sampling rate and window size is extracted based on a log-Gaussian distribution, An electronic device configured to determine the plurality of first data by applying at least one of the extracted sampling rate and window size to the voice data.

13. In paragraph 8, the processor: An electronic device further configured to detect specific voice data from the voice data based on the second data.

14. In paragraph 8, The above first artificial intelligence network is learned from the second artificial intelligence network through a knowledge distillation process, The above first artificial intelligence network is a student network, and The above second artificial intelligence network is an electronic device that is a teacher network.

15. A program stored in a medium for performing a direction estimation method through an artificial intelligence algorithm executable by a processor, Step of receiving voice data; A step of determining a plurality of first data representing a plurality of resolutions based on at least one of a sampling rate and a window size of the above voice data; and A step of outputting second data from a first artificial intelligence network that has been trained using the plurality of first data as input information is performed, The above-mentioned first artificial intelligence network is a program that learns based on the plurality of first data and third data output from the second artificial intelligence network.

Citation Information

Patent Citations

  • Speech synthesis methods, systems, devices and storage media

    CN113409756B

  • Composition for preventing or treating of obesity comprising peptides derived from Neuronal growth regulator 1(NEGR1)

    KR102861386B1

  • Audio processing method and apparatus, computer device, and storage medium

    US20220084532A1

  • KR20190108804A