Method, apparatus and program for improving recognition accuracy of acoustic data
By configuring acoustic frames and applying post-processing corrections, the method improves the accuracy of acoustic event recognition in complex environments, reducing error detection and enhancing reliability.
Patent Information
- Application Number
- JP2024526923
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-22
- Filing Date
- 2022-11-11
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2042-11-11
AI Technical Summary
Existing acoustic event recognition technologies face challenges in accurately distinguishing between ambient noise and incident sounds, especially in complex environments, leading to low reliability and high error rates, particularly when multiple events occur simultaneously.
The method involves configuring acoustic frames based on acoustic data, processing them through an acoustic recognition model, performing critical value analysis and time series analysis to identify recognized frames, and applying transformations to improve accuracy by reducing false alarms and misrecognitions.
This approach enhances the recognition accuracy of acoustic events by minimizing error detection probabilities and improving the reliability of sound detection in real-life environments.
Smart Images

Figure 2024542144000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for improving the recognition rate of acoustic data, and more particularly to a technique for improving the recognition rate through post-processing correction of acoustic data. [Background technology]
[0002] Hearing-impaired people who cannot hear sounds at all or cannot distinguish sounds well have many difficulties in daily life because they have difficulty hearing sounds and judging the situation, and they are unable to immediately respond because they cannot recognize dangerous situations in indoor and outdoor environments using sound information. In situations where the hearing-impaired, pedestrians wearing earphones, elderly people, and others have no or limited hearing, the sounds generated around the user may be blocked. In addition, in situations where it is difficult for the user to detect sounds, such as when sleeping, the user may encounter dangerous situations or have an accident because they cannot recognize the surrounding situation. Meanwhile, there is an increasing need to develop technology that can detect and recognize acoustic events in such environments. Technology that can detect and recognize acoustic events is a technology that can be applied to various fields such as recognition of real-life environment context, recognition of dangerous situations, recognition of media content, and situation analysis in wired communication, and is therefore being continuously researched. For audio event recognition technology, the main research is on extracting various feature values such as MFCC, energy, spectral flux, zero crossing rate from audio signals and verifying excellent features, and on Gaussian mixture model or rule-based classification methods, and recently, deep learning-based machine learning methods have been researched to improve on the above methods. However, such methods have limitations in that the accuracy of audio detection is guaranteed at low signal-to-noise ratios and it is difficult to distinguish between surrounding noise and the audio of an event. That is, in a real-life environment containing various ambient noises, it may be difficult to detect a sound event with high reliability. Specifically, in order to detect a valid sound event, it is necessary to determine whether a sound event has occurred from sound data acquired in a time series (i.e., continuously), and it is also necessary to recognize what event class has occurred, so it may be difficult to ensure a high reliability. In addition, when two or more events occur simultaneously, it is necessary to solve not only a single event (monophonic) but also a polyphonic event recognition problem, which may further reduce the sound event recognition rate. In addition, the reason why the recognition rate is low when detecting acoustic events from acoustic data acquired in real life is because there is a probability of false alarms, that is, the probability of determining that an event exists when an acoustic event does not occur, or that the event does not exist when an event does occur.
[0003] Therefore, reducing the probability of erroneous detection in response to time-sequentially acquired acoustic data may enable detection of acoustic events with increased confidence in real-life environments. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Republic of Korea Registered Patent 10-2014-0143069 Summary of the Invention [Problem to be solved by the invention]
[0005] The problem to be solved by the present invention is to solve the above-mentioned problems, and to provide an acoustic data recognition environment with improved accuracy through post-processing correction related to acoustic data. The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems, a method for improving recognition accuracy of audio data according to various embodiments of the present invention is disclosed, which includes the steps of: constructing one or more audio frames based on audio data, processing each of the one or more audio frames as an input of an audio recognition model to output a predicted value corresponding to each audio frame, identifying one or more recognition audio frames through a threshold value analysis based on the predicted value corresponding to each audio frame, identifying a transformed audio frame through a time series analysis based on the one or more recognition audio frames, and performing a transformation on the predicted value corresponding to the transformed audio frame.
[0007] In an alternative embodiment, the step of constructing one or more audio frames based on the audio data may include a step of dividing the audio data to have a size of a predetermined first time unit to construct the one or more audio frames. In an alternative embodiment, the start time of each of the one or more audio frames may be determined to have a difference of a magnitude of a second time unit from the start time of each of the adjacent audio frames. In an alternative embodiment, the predicted value may include one or more predicted item information and predicted numerical information corresponding to each of the one or more predicted item information, and the critical value analysis may be an analysis for identifying the one or more recognized audio frames by determining whether each of the one or more predicted numerical information corresponding to each of the predicted item information is equal to or greater than a predetermined critical value corresponding to each of the predicted item information. In an alternative embodiment, the step of identifying a transformed audio frame through the time series analysis may include a step of identifying predicted item information corresponding to each of the one or more recognition audio frames, a step of determining whether the identified predicted item information is repeated a predetermined number of times or more during a predetermined reference time, and a step of identifying the transformed audio frame based on the determination result. In an alternative embodiment, the method may include a step of identifying a correlation between each of the recognition audio frames based on prediction item information corresponding to each of the one or more recognition audio frames, and a step of determining whether or not to adjust a critical value and a critical number of times corresponding to each of the one or more audio frames based on the correlation. In an alternative embodiment, the transformation of the prediction value may include at least one of a noise transformation that transforms the output of the acoustic recognition model based on the transformed acoustic frame into unrecognized items, and an acoustic item transformation that transforms predicted item information associated with the transformed acoustic frame into calibrated predicted item information. In an alternative embodiment, the proofreading prediction item information may be determined based on a correlation of the prediction item information.
[0008] According to another embodiment of the present invention, there is disclosed an apparatus for performing a method for improving accuracy of recognition of audio data, the apparatus including a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, the processor executing the one or more instructions to perform the method for improving accuracy of recognition of audio data. According to yet another embodiment of the present invention, there is disclosed a computer program stored on a computer readable recording medium, the computer program being capable of performing the method for improving recognition accuracy of acoustic data as described above when combined with a computer, which is hardware. Other specific details of the invention are included in the detailed description and the drawings. Effect of the Invention
[0009] Various embodiments of the present invention can provide an effect of improving the recognition accuracy of audio data by correcting the audio data. The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. [Brief description of the drawings]
[0010] [Figure 1] 1 is a diagram illustrating a system for performing a method for improving recognition accuracy of audio data according to an embodiment of the present invention. [Diagram 2] 2 is a hardware configuration diagram of a server for improving the recognition accuracy of acoustic data in accordance with an embodiment of the present invention. FIG. [Diagram 3] 1 shows a flow chart illustrating an exemplary method for improving recognition accuracy of acoustic data in accordance with one embodiment of the present invention. [Figure 4] 1 illustrates an example diagram for explaining a process of constructing one or more acoustic frames based on acoustic data in accordance with an embodiment of the present invention; [Diagram 5] 1 is an exemplary diagram illustrating a process in which an acoustic recognition model according to an embodiment of the present invention outputs a predicted value based on an acoustic frame; [Figure 6] 1 shows a flow chart illustrating an exemplary critical value analysis process related to one embodiment of the present invention. [Figure 7] 1 shows a flow chart illustrating an exemplary process for time series analysis in accordance with one embodiment of the present invention. [Figure 8] 1 illustrates an exemplary table for explaining an acoustic data correction process in accordance with one embodiment of the present invention. [Figure 9] 1 is an exemplary diagram for explaining a process of correcting acoustic data in accordance with an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Various embodiments will now be described with reference to the drawings. Various descriptions are provided herein to provide an understanding of the present invention. However, it will be apparent that such embodiments may be practiced without such specific descriptions. As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or software implementations. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and / or thread of execution. A component may be localized within one computer. A component may be distributed among two or more computers. Also, such components may execute from various computer readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, via signals comprising one or more data packets (e.g., data from one component interacting with other components in a local system, a distributed system, and / or data transmitted over a network such as the Internet to other systems via signals). Additionally, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X utilizes A or B" is intended to mean one of the natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, then "X utilizes A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated listed items. Additionally, the terms "comprise" and / or "comprising" should be understood to mean the presence of the indicated feature and / or component. However, the terms "comprise" and / or "comprising" should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Additionally, in the present specification and claims, the singular should generally be construed to mean "one or more," unless otherwise specified or clear from the context as indicating the singular form.
[0012] Those skilled in the art should additionally appreciate that the various exemplary logical blocks, configurations, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the various exemplary components, blocks, configurations, means, logic, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is embodied in hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in various ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
[0013] The description of the embodiments presented is provided to enable one of ordinary skill in the art to make and practice the present invention. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The generic principles defined herein may be applied to other embodiments without departing from the scope of the present invention, and the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the broadest scope consistent with the principles and novel features presented herein. In this specification, a computer means any type of hardware device including at least one processor, and may also be understood to include software configurations that operate on the corresponding hardware device according to the embodiment. For example, a computer may be understood to include, but is not limited to, a smartphone, a tablet PC, a desktop, a notebook computer, and user clients and applications that run on each device.
[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Although each step described in this specification is described as being performed by a computer, the subject of each step is not limited to this, and depending on the embodiment, at least some of each step may be performed by different devices. Here, the method for improving the recognition accuracy of sound data according to various embodiments of the present invention may relate to a method for correcting sound data so as to improve the recognition rate of the sound data. The correction of the sound data may mean, for example, post-processing correction related to the sound data. That is, when time-series sound data is acquired, the present invention may perform post-processing correction on the sound data to improve the accuracy in the sound data recognition process. In the embodiment, the improvement of the recognition accuracy of the sound data may mean improving the recognition accuracy of detecting a specific event in the sound data.
[0015] Meanwhile, in order to detect or recognize a specific event in acoustic data with high accuracy, it may be important to reduce the probability of erroneous detection, which may refer to the probability of determining that an acoustic event exists when it does not occur, or determining that an event does not exist when an event does occur. According to an embodiment, the method for improving the accuracy of recognition of audio data may divide audio data into a plurality of audio frames each having a certain time unit in order to minimize the probability of error detection of the audio data, and improve the accuracy of the recognition of the audio data through audio recognition corresponding to each divided audio frame. In this case, each audio frame may have at least a part of an overlapping section with other audio frames. That is, the present invention may subdivide audio data, which is time-series information, into a plurality of audio frames in a certain time unit, perform an analysis on each audio frame, and perform a transformation on at least some of the plurality of audio frames as a result of the analysis. For example, only when a specific audio (e.g., a siren sound) is recognized over at least two or more audio frames among the plurality of audio frames, it may be determined that the specific audio has been recognized. In other words, when a specific audio (e.g., a siren sound) is recognized only in a specific audio frame among the plurality of audio frames (i.e., the specific audio is not recognized in audio frames adjacent to the specific audio frame), it may be determined that the specific audio has not been recognized, and a transformation related to the corresponding audio frame may be performed. Here, conversion related to an audio frame may mean, for example, converting a sound recognized in relation to a specific audio frame (e.g., a siren sound) into an unrecognized sound, or converting the sound into another sound (e.g., a sound not related to the recognition), since the sound is an erroneous recognition sound. That is, a sound recognized in only one frame can be removed as an error, and a sound recognized continuously in relation to the frame can be determined as a normal recognition sound.
[0016] In summary, the present invention divides audio data into frames, and determines that sounds that are not continuously recognized in each frame are misrecognized sounds and performs post-processing correction to improve the accuracy of recognition of the entire audio data. A more detailed description of the method for improving the accuracy of recognition of audio data will be provided below.
[0017] Fig. 1 is a diagram illustrating a system for performing a method for improving the recognition accuracy of acoustic data according to an embodiment of the present invention. As illustrated in Fig. 1, the system for performing the method for improving the recognition accuracy of acoustic data according to an embodiment of the present invention may include a server 100 for improving the recognition accuracy of acoustic data, a user terminal 200, and an external server 300. Here, the system for performing the method for improving the recognition accuracy of acoustic data illustrated in Fig. 1 is according to one embodiment, and its components are not limited to the embodiment illustrated in Fig. 1, and may be added, changed, or deleted as necessary.
[0018] In one embodiment, the server 100 for improving the recognition accuracy of sound data may determine whether a specific event has occurred based on the sound data. Specifically, the server 100 for improving the recognition accuracy of sound data may acquire sound data related to real life and determine whether a specific event has occurred through analysis of the acquired sound data. In one embodiment, the specific event may be related to the occurrence of security, safety, or danger, for example, an alarm sound, a child's crying, a glass breaking sound, a tire puncture sound, etc. The specific description of the sound related to the specific event described above is merely an example, and the present invention is not limited thereto.
[0019] According to the embodiment, since the sound data acquired in real life includes various ambient noises, it may be difficult to detect a sound event with high reliability. Accordingly, when the server 100 for improving the recognition accuracy of sound data of the present invention receives the sound data, the server 100 for improving the recognition accuracy of sound data of the present invention may perform post-processing correction on the corresponding sound data. Here, the post-processing correction may mean a correction for reducing the probability of error detection in the process of recognizing the sound data. For example, the post-processing correction may include converting a sound (e.g., a sound of breaking glass) recognized in a part of the sound data into an unrecognized one (i.e., processing it with noise) or converting a recognized sound result into a different sound. That is, the server 100 for improving the recognition accuracy of sound data may obtain time-series sound data related to real life and ensure improved recognition accuracy through post-processing correction on the obtained sound data. According to the embodiment, the server 100 for improving the recognition accuracy of sound data may include any server implemented by an API (Application Programming Interface). For example, the user terminal 200 may acquire sound data and transmit it to the server 100 through the API. For example, the server 100 may acquire sound data from the user terminal 200 and determine that an emergency alarm sound (e.g., a siren sound) has occurred through analysis of the sound data. In the embodiment, the server 100 for improving the recognition accuracy of sound data may perform analysis of the sound data through a sound recognition model (e.g., an artificial intelligence model).
[0020] In one embodiment, an acoustic recognition model (e.g., an artificial intelligence model) may be comprised of one or more network functions, which may be comprised of a collection of interconnected computational units that may generally be referred to as "nodes." Such "nodes" may also be referred to as "neurons." One or more network functions may be comprised of at least one or more nodes. The nodes (or neurons) that make up one or more network functions may be interconnected by one or more "links." In an artificial intelligence model, one or more nodes connected through links can form a relationship between input nodes and output nodes relative to each other. The concepts of input nodes and output nodes are relative, and any node that is in an output node relationship with one node can be in an input node relationship with another node, and vice versa. As mentioned above, the input node to output node relationship can be generated around links. One or more output nodes can be connected to one input node through links, and vice versa. In the relationship between an input node and an output node connected through a link, the value of the output node may be determined based on data input to the input node. Here, the node connecting the input node and the output node may have a weight. The weight may be variable and may be changed by a user or an algorithm so that the artificial intelligence model performs a desired function. For example, when one or more input nodes are connected to one output node through respective links, the output node may determine the output node value based on the value input to the input node connected to the output node and the weight set for the link corresponding to each input node.
[0021] As described above, in an AI model, one or more nodes are connected to each other through one or more links to form a relationship between input nodes and output nodes within the AI model. The characteristics of the AI model can be determined by the number of nodes and links, the correlation between the nodes and links, and the weights assigned to each link within the AI model. For example, if there are two AI models that have the same number of nodes and links but different weights between the links, the two AI models can be recognized as different from each other. Some of the nodes constituting the artificial intelligence model may constitute a layer based on the distance from the first input node. For example, a set of nodes whose distance from the first input node is n may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be traversed to reach the corresponding node from the first input node. However, such a definition of a layer is arbitrary for the purpose of explanation, and the order of layers in the artificial intelligence model may be defined in a manner different from that described above. For example, the layer of a node may be defined by the distance from the final output node.
[0022] The first input node may refer to one or more nodes to which data is directly input without passing through a link in relation to other nodes in the artificial intelligence model. Or, in the artificial intelligence model network, it may refer to a node that does not have other input nodes connected by a link in the node-to-node relationship based on the link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes in the artificial intelligence model. Also, the hidden node may refer to a node that constitutes the artificial intelligence model, rather than the first input node and the last output node. The artificial intelligence model according to an embodiment of the present invention may be an artificial intelligence model in which the nodes of the input layer may be more than the nodes of the hidden layer close to the output layer, and the number of nodes decreases as the input layer progresses to the hidden layer.
[0023] An artificial intelligence model may include one or more hidden layers. Hidden nodes in a hidden layer may receive the output of a previous layer and the output of surrounding hidden nodes as input. The number of hidden nodes in each hidden layer may be the same or different. The number of nodes in an input layer may be determined based on the number of data fields of the input data, and may be the same or different from the number of hidden nodes. Input data input to the input layer may be operated by hidden nodes in the hidden layer and may be output by a fully connected layer (FCL), which is an output layer.
[0024] In various embodiments, the artificial intelligence model may be subjected to supervised learning using a plurality of acoustic data and feature information corresponding to each acoustic data as learning data, but is not limited thereto and various learning methods may be applied. Here, supervised learning is a method of generating learning data by labeling specific data and information related to the specific data, and learning using the generated learning data, or a method of labeling two pieces of data having a causal relationship to generate learning data, and learning through the generated learning data.
[0025] In one embodiment, the server 100 for improving the recognition accuracy of acoustic data may determine whether to stop the learning using verification data when the learning of one or more network functions is performed for a predetermined epoch or more. The predetermined epoch may be a part of the entire learning target epoch. The verification data may be composed of at least a part of the labeled learning data. That is, the server 100 for improving the recognition accuracy of acoustic data performs learning of an artificial intelligence model through the learning data, and after the learning of the artificial intelligence model is repeated for a predetermined epoch or more, the server 100 for improving the recognition accuracy of acoustic data may determine whether the learning effect of the artificial intelligence model is equal to or higher than a predetermined level using the verification data. For example, when performing learning with a target number of repeated learning times of 10 times using 100 pieces of learning data, the server 100 for improving the recognition accuracy of acoustic data may perform 3 repeated learning times using 10 pieces of verification data after performing 10 repeated learning times, which are the predetermined epochs, and if the change in the output of the artificial intelligence model during the three repeated learning times is equal to or lower than a predetermined level, the server 100 may determine that further learning is meaningless and terminate the learning.
[0026] That is, the validation data may be used to determine the completion of learning based on whether the effect of the learning per epoch is above or below a certain level in the iterative learning of the artificial intelligence model. The numbers of learning data, validation data, and the number of iterations described above are merely examples, and the present invention is not limited thereto. The server 100 for improving the recognition accuracy of acoustic data may generate an artificial intelligence model by testing the performance of one or more network functions using test data and determining whether or not to activate one or more network functions. The test data may be used to verify the performance of the artificial intelligence model and may be composed of at least a portion of the learning data. For example, 70% of the learning data may be used for learning the artificial intelligence model (i.e., learning to adjust weights to output result values similar to the labels), and 30% may be used as test data for verifying the performance of the artificial intelligence model. The server 100 for improving the recognition accuracy of acoustic data may input the test data to the artificial intelligence model that has completed learning, measure an error, and determine whether or not to activate the artificial intelligence model depending on whether the error is equal to or exceeds a predetermined performance.
[0027] The server 100 for improving the accuracy of recognizing acoustic data verifies the performance of an artificial intelligence model that has completed training by using test data for the artificial intelligence model, and if the performance of the artificial intelligence model that has completed training is above a predetermined standard, the server 100 can activate the artificial intelligence model for use in other applications. In addition, the server 100 for improving the recognition accuracy of acoustic data may deactivate and discard the AI model if the performance of the AI model that has completed learning is below a predetermined standard. For example, the server 100 for improving the recognition accuracy of acoustic data may determine the performance of the AI model generated based on factors such as accuracy, precision, and recall. The above-mentioned performance evaluation criteria are merely examples, and the present invention is not limited thereto. The server 100 for improving the recognition accuracy of acoustic data may generate a plurality of AI models by independently learning each AI model, and may evaluate the performance and use only AI models with a certain level of performance or higher. However, the present invention is not limited thereto.
[0028] Throughout this specification, the terms computation model, neural network, network function, and neural network may be used interchangeably (hereinafter, the term neural network will be used interchangeably). The data structure may include a neural network. The data structure including a neural network may be stored on a computer-readable medium. The data structure including a neural network may also include data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for training the neural network. The data structure including a neural network may include any of the components disclosed above. That is, the data structure including a neural network may include all or any combination of data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation functions associated with each node or layer of the neural network, and loss functions for training the neural network. In addition to the above-mentioned configurations, the data structure including a neural network may include any other information that determines the characteristics of the neural network. The data structure may also include any form of data used or generated in the computation process of the neural network, and is not limited to the above. The computer-readable medium may include a computer-readable recording medium and / or a computer-readable transmission medium. A neural network may be composed of a collection of interconnected computational units that may generally be referred to as nodes. Such nodes may also be referred to as neurons. A neural network is composed of at least one or more nodes.
[0029] According to an embodiment of the present invention, the server 100 for improving the recognition accuracy of acoustic data may be a server that provides a cloud computing service. More specifically, the server 100 for improving the recognition accuracy of acoustic data may be a server that provides a cloud computing service, which is a type of Internet-based computing and processes information not on a user's computer but on another computer connected to the Internet. The cloud computing service may be a service that stores data on the Internet and allows a user to use the data or program he or she needs through an Internet connection anytime and anywhere without installing it on his or her computer, and allows the data stored on the Internet to be easily shared and transmitted with simple operations and clicks. In addition, the cloud computing service may be a service that not only simply stores data on a server on the Internet, but also allows a user to perform a desired task using a function of an application program provided on the web without installing a separate program, and allows multiple people to work while sharing a document at the same time. In addition, the cloud computing service may be embodied in at least one form of Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS), a virtual machine-based cloud server, and a container-based cloud server. That is, the server 100 for improving the recognition accuracy of acoustic data according to the present invention may be embodied in the form of at least one of the above-mentioned cloud computing services. The above-mentioned specific descriptions of the cloud computing services are merely examples, and the present invention may include any platform that constructs a cloud computing environment.
[0030] In various embodiments, a server 100 for improving the recognition accuracy of acoustic data may be connected to a user terminal 200 via a network, and may not only generate and provide an acoustic recognition model for analyzing acoustic data, but also provide information (e.g., acoustic event information) obtained by analyzing the acoustic data through the acoustic recognition model to the user terminal. Here, the network may refer to a connection structure that allows information exchange between multiple nodes such as terminals and servers, etc. For example, the network may include a local area network (LAN), a wide area network (WAN), the Internet (WWW), wired / wireless data communication networks, telephone networks, wired / wireless television communication networks, etc.
[0031] In addition, the wireless data communication network includes, but is not limited to, 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), 5GPP (5th Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc.
[0032] In one embodiment, the user terminal 200 may be connected to a server 100 for improving the accuracy of recognition of acoustic data via a network, and may provide acoustic data to the server 100 for improving the accuracy of recognition of acoustic data, and may receive information related to the occurrence of various events (e.g., the occurrence of an alarm sound, a child's crying, the sound of glass breaking, a flat tire, etc.) in response to the provided acoustic data. Here, the user terminal 200 is a wireless communication device that is guaranteed to be portable and mobile, and may include, but is not limited to, all kinds of handheld-based wireless communication devices such as navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), personal handyphone system (PHS), personal digital assistant (PDA), international mobile telecommunication (IMT)-2000, code division multiple access (CDMA)-2000, W-code division multiple access (W-CDMA), wireless broadband internet (Wibro) terminal, smartphone, smartpad, tablet PC, etc. For example, the user terminal 200 may be installed in a specific area to perform sensing related to the specific area. For example, the user terminal 200 may be installed in a vehicle and acquire sound data generated while the vehicle is parked or running. The above description of specific locations or places where the user terminal is provided is merely an example, and the present invention is not limited thereto.
[0033] In one embodiment, the external server 300 may be connected to the server 100 for improving the recognition accuracy of acoustic data through a network, and may provide various information / data necessary for the server 100 for improving the recognition accuracy of acoustic data to analyze acoustic data using an artificial intelligence model, or may receive, store, and manage result data derived by performing the acoustic data analysis using the artificial intelligence model. For example, the external server 300 may be, but is not limited to, a storage server separately provided outside the server 100 for improving the recognition accuracy of acoustic data. Hereinafter, a hardware configuration of the server 100 for improving the recognition accuracy of acoustic data will be described with reference to FIG. 2.
[0034] FIG. 2 is a hardware configuration diagram of a server for improving the recognition accuracy of acoustic data according to an embodiment of the present invention. Referring to Fig. 2, a server 100 for improving recognition accuracy of acoustic data according to an embodiment of the present invention (hereinafter, "server 100") may include one or more processors 110, a memory 120 for loading a computer program 151 executed by the processor 110, a bus 130, a communication interface 140, and a storage 150 for storing the computer program 151. Here, Fig. 2 illustrates only components related to the embodiment of the present invention. Therefore, a person skilled in the art to which the present invention belongs will understand that other general components may be included in addition to the components illustrated in Fig. 2. The processor 110 controls the overall operation of each component of the server 100. The processor 110 may include a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), a graphic processing unit (GPU), or any other type of processor commonly known in the technical field of the present invention. The processor 110 can read a computer program stored in the memory 120 and perform data processing for an artificial intelligence model according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 110 can perform calculations for neural network learning. The processor 110 can perform calculations for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from the input data, calculating errors, and updating weights of the neural network using backpropagation.
[0035] In addition, at least one of the processor 110, a CPU, a GPGPU, and a TPU, may process the learning of a network function. For example, the CPU and the GPGPU may both process the learning of a network function and data classification using the network function. In addition, in one embodiment of the present invention, the processors of a plurality of computing devices may be used together to process the learning of a network function and data classification using the network function. In addition, the computer program executed in the computing device according to one embodiment of the present invention may be a CPU, GPGPU, or TPU executable program. As used herein, the term network function may be used interchangeably with the term artificial neural network or neural network. As used herein, the network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks.
[0036] The processor 110 may read a computer program stored in the memory 120 to provide an acoustic recognition model according to an embodiment of the present invention. According to an embodiment of the present invention, the processor 110 may perform calculations for training the acoustic recognition model. According to one embodiment of the present invention, the processor 110 may generally process the overall operation of the server 100. The processor 110 may process signals, data, information, etc. input or output through the components detailed above, or may run application programs stored in the memory 120, thereby providing or processing appropriate information or functions to a user or a user terminal.
[0037] Furthermore, the processor 110 may execute operations for at least one application or program for executing a method according to an embodiment of the present invention, and the server 100 may include one or more processors. In various embodiments, the processor 110 may further include a random access memory (RAM, not shown) and a read-only memory (ROM, not shown) for temporarily and / or permanently storing signals (or data) processed within the processor 110. The processor 110 may also be embodied in the form of a system on chip (SoC) including at least one of a graphics processing unit, a RAM, and a ROM. The memory 120 stores various data, instructions and / or information. The memory 120 can load a computer program 151 from the storage 150 to perform the methods / operations according to various embodiments of the present invention. When the computer program 151 is loaded into the memory 120, the processor 110 can execute one or more instructions constituting the computer program 151 to perform the methods / operations. The memory 120 can be embodied as a volatile memory such as a RAM, although the scope of the present invention is not limited in this respect.
[0038] The bus 130 provides a communication function between the components of the server 100. The bus 130 may be implemented as various types of buses such as an address bus, a data bus, and a control bus. The communication interface 140 supports wired / wireless Internet communication of the server 100. The communication interface 140 may also support various communication methods other than Internet communication. To this end, the communication interface 140 may be configured to include a communication module that is widely known in the technical field of the present invention. In some embodiments, the communication interface 140 may be omitted.
[0039] The storage 150 may non-temporarily store a computer program 151. When a process for improving the recognition accuracy of acoustic data is performed through the server 100, the storage 150 may store various information required to provide the process for improving the recognition accuracy of acoustic data. Storage 150 may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or any form of computer-readable recording medium widely known in the technical field to which the present invention belongs. The computer program 151, when loaded into the memory 120, may include one or more instructions that cause the processor 110 to perform the methods / operations according to various embodiments of the present invention. That is, the processor 110 may execute the one or more instructions to perform the methods / operations according to various embodiments of the present invention.
[0040] In one embodiment, the computer program 151 may include one or more instructions for performing a method for improving recognition accuracy of acoustic data, including the steps of: constructing one or more acoustic frames based on acoustic data; processing each of the one or more acoustic frames as an input for an acoustic recognition model to output a predicted value corresponding to each acoustic frame; identifying one or more recognition acoustic frames through a threshold value analysis based on the predicted value corresponding to each acoustic frame; identifying a transformed acoustic frame through a time series analysis based on the one or more recognition acoustic frames; and performing a transformation on the predicted value corresponding to the transformed acoustic frame.
[0041] The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in a software module executed by hardware, or in a combination thereof. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer readable recording medium commonly known in the art to which the present invention pertains. The components of the present invention may be embodied as a program (or application) and stored on a medium to be executed in combination with a computer, which is hardware. The components of the present invention may be implemented as software programming or software elements, and similarly, embodiments include various algorithms embodied as a combination of data structures, processes, routines or other programming constructs, and may be embodied in programming or scripting languages such as C, C++, Java, assembler, etc. Functional aspects may be embodied as algorithms executed by one or more processors. Hereinafter, a method for improving the recognition accuracy of acoustic data performed by the server 100 will be described with reference to Figures 3 to 9.
[0042] 3 is a flow chart showing an example of a method for improving the recognition accuracy of audio data according to an embodiment of the present invention. The order of the steps shown in FIG. 3 may be changed as necessary, and at least one step may be omitted or added. That is, the following steps are merely an embodiment of the present invention, and the scope of the present invention is not limited thereto. According to an embodiment of the present invention, the server 100 may acquire acoustic data. The acoustic data may include information related to sounds acquired in real life. The acquisition of acoustic data according to an embodiment of the present invention may involve receiving or loading acoustic data stored in the memory 120. Furthermore, the acquisition of acoustic data may involve receiving or loading data from another storage medium, another computing device, or a separate processing module within the same computing device based on wired / wireless communication means.
[0043] According to an embodiment, the acoustic data may be acquired through a user terminal 200 associated with a user. For example, the user terminal 200 associated with a user may include any kind of handheld-based wireless communication device such as a smartphone, a smartpad, a tablet PC, etc., or an electronic device (e.g., a device capable of receiving acoustic data through a microphone) provided in a specific space (e.g., a user's residential space), etc. According to an embodiment of the present invention, the server 100 may generate one or more audio frames based on the audio data (S100). The one or more audio frames may be obtained by dividing the audio data, which is time-series information, into a plurality of frames based on a specific time unit. Specifically, the server 100 may generate one or more audio frames by dividing the audio data to have a size of a predetermined first time unit. For example, if the first audio data is audio data acquired corresponding to a time of one minute, the server 100 may configure 30 audio frames by setting the first time unit to two seconds and dividing the first audio data. The above-mentioned specific numerical values related to the first time unit and the one or more audio frames are merely examples, and the present invention is not limited thereto.
[0044] According to an embodiment, the server 100 may configure one or more audio frames such that each of the one or more audio frames at least partially overlaps with each other. To explain in detail with reference to FIG. 4, the start time of each of the one or more audio frames may be determined to have the size of the second time unit 400b with the start time of each of the adjacent audio frames. According to an embodiment, the size of the second time unit 400b may be determined to be smaller than the size of the first time unit 400a. That is, the server 100 may generate one or more audio frames (i.e., a first audio frame 411, a second audio frame 412, a third audio frame 413, etc.) 410 having the same first time unit 400a as shown in FIG. In this case, each audio frame may be configured to differ from each of the adjacent audio frames by the size of the second time unit 400b, which is smaller than the size of the first time unit 400a. Accordingly, each audio frame may at least partially overlap with each of the adjacent audio frames. As a specific example, if the sound data 400 is related to sound acquired for 10 seconds, the first time unit 400a may be set to 2 seconds, and the second time unit 400b may be set to 1 second, which is less than the first time unit 400a. In this case, the first sound frame 411 may be related to sound acquired for 0 to 2 seconds, the second sound frame 412 may be related to sound acquired for 1 to 3 seconds, and the third sound frame 413 may be related to sound acquired for 2 to 4 seconds. The specific numerical values related to the total time, the first time unit, and the second time unit of the sound data described above are merely examples, and the present invention is not limited thereto. That is, one or more audio frames are configured so that the start point of each audio frame has a difference in size between the start point of each adjacent audio frame and the start point of the second time unit 400b that is smaller than the size of the first time unit 400a, so that at least a portion of each audio frame can have an overlapping section.
[0045] According to an embodiment of the present invention, the server 100 may process one or more audio frames as inputs to an audio recognition model and output a prediction value corresponding to each audio frame (S200). According to an embodiment, the server 100 can train an autoencoder through an unsupervised learning method. Specifically, the server 100 can train a dimension reduction network function (e.g., an encoder) and a dimension restoration network function (e.g., a decoder) that configure an autoencoder to output output data similar to input data. In more detail, only core feature data (or features) of audio data input in the encoding process through the dimension reduction network function can be learned through a hidden layer, and the remaining information can be lost. In this case, the output data of the hidden layer in the decoding process through the dimension restoration network function can be an approximation of the input data (i.e., audio data) rather than a perfect copy value. That is, the server 100 can train an autoencoder by adjusting weights so that the output data and the input data are as identical as possible.
[0046] An autoencoder may be a type of neural network for outputting output data similar to input data. An autoencoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to an output layer (symmetric to the input layer). The number of input layers and output layers may correspond to the number of items of input data remaining after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may be structured to decrease as it moves away from the input layer. The number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and the decoder) may be maintained at a certain number or more (e.g., more than half of the input layer) because if the number is too small, a sufficient amount of information may not be transmitted.
[0047] The server 100 may use a training data set including a plurality of training data tagged with object information as an input of a trained dimension reduction network to match and store the output feature data for each object with the tagged object information. Specifically, the server 100 may use a dimension reduction network function to obtain feature data of a first object for training data included in the first training data subset by using a first training data subset tagged with first sound discrimination information (e.g., the sound of breaking glass) as an input of the dimension reduction network function. The obtained feature data may be expressed as a vector. In this case, the feature data output corresponding to each of the plurality of training data included in the first training data subset is an output through the training data related to the first sound, and therefore may be located relatively close to each other in the vector space. The server 100 may match and store the feature data related to the first sound represented as a vector with the first sound discrimination information (i.e., the sound of breaking glass). In the case of a trained autoencoder dimensionality reduction network function, the dimensionality restoration network function can be trained to extract features well that allow the input data to be well restored.
[0048] Also, for example, a plurality of training data included in each of the first training data subsets tagged with the second sound discrimination information (e.g., the sound of a siren) may be converted into feature data (i.e., features) through a dimensionality reduction network function and displayed in a vector space. In this case, since the feature data is an output through the training data related to the second sound discrimination information (i.e., the sound of a siren), it may be located relatively close to each other in the vector space. In this case, the feature data corresponding to the second sound discrimination information may be displayed in a vector space different from the feature data corresponding to the first sound discrimination information (e.g., the sound of glass breaking).
[0049] In the embodiment, the server 100 may configure the acoustic recognition model 500 by including a dimension reduction network function in the trained autoencoder. That is, when an acoustic frame is input, the acoustic recognition model 500 configured by including the dimension reduction network function generated through the above-mentioned training process may extract feature information (i.e., features) corresponding to the acoustic frame through an operation using the dimension reduction network function for the corresponding acoustic frame. In this case, the acoustic recognition model 500 can evaluate the similarity of the acoustic style by comparing the distance in the vector space between the area in which the feature corresponding to the acoustic frame is displayed and the object-specific feature data, and can output a predicted value corresponding to the acoustic data based on the similarity evaluation. In one embodiment, the predicted value can include one or more predicted item information and predicted numerical information corresponding to each of the one or more predicted item information.
[0050] Specifically, the acoustic recognition model 500 can output feature information (i.e., features) by computing the acoustic frame using a dimension reduction network function. In this case, the acoustic recognition model can include one or more predicted item information corresponding to the acoustic frame and predicted numerical information corresponding to each predicted item information based on the position between the feature information output corresponding to the acoustic frame and the feature data for each acoustic identification information previously recorded on the vector space through learning. The one or more pieces of predicted item information are information related to what kind of sound is involved, and may include, for example, the sound of glass breaking, a tire going flat, an emergency siren going off, a puppy barking, rain falling, etc. Such predicted item information may be generated based on feature information output corresponding to the sound frame and sound identification information located close to the position in the vector space. For example, the sound recognition model may configure one or more pieces of predicted item information through sound identification information matched to feature information located close to the first feature information output corresponding to the first sound frame. The specific description related to the one or more pieces of predicted item information described above is merely exemplary, and the present invention is not limited thereto.
[0051] The predicted numerical information corresponding to each prediction item information may be information about a numerical value predicted corresponding to each prediction item information. For example, the acoustic recognition model may configure one or more prediction item information through acoustic identification information matched with feature information located close to the first feature information output corresponding to the first acoustic frame. In this case, the closer the position of the first feature information is to the feature information corresponding to each acoustic identification information, the higher the predicted numerical information may be output, and the farther the position of the feature information corresponding to each acoustic identification information is from the first feature information, the lower the predicted numerical information may be output. 5, the acoustic recognition model 500 may output predicted item information 610 related to "sound of a siren", "scream", "sound of glass breaking" and "other sounds" in response to the first acoustic frame 411. In addition, the acoustic recognition model 500 may output predicted numerical information 620 of "1", "95", "3" and "2" in response to each predicted item information 610. That is, the acoustic recognition model 500 may output predicted values 600 in which the probability related to the sound of a siren is 1, the probability related to the scream is 95, the probability related to the sound of glass breaking is "3", and the probability related to other sounds is "2" in response to the first acoustic frame 411. The above descriptions of specific values for each of the predicted item information and predicted numerical information are merely examples, and the present invention is not limited thereto. That is, the server 100 can output a predicted value corresponding to one or more acoustic frames constructed based on acoustic data through the acoustic recognition model 500. For example, the acoustic recognition model 500 can output a first predicted value corresponding to the first acoustic frame 411, a second predicted value corresponding to the second acoustic frame 412, and a third predicted value corresponding to the third acoustic frame 413.
[0052] According to an embodiment of the present invention, the server 100 can identify one or more recognition audio frames through a threshold value analysis based on a prediction value corresponding to each audio frame (S300). Here, the threshold value analysis can refer to an analysis that determines whether one or more prediction numerical information corresponding to each audio frame is equal to or greater than a predetermined threshold value corresponding to each prediction item information, thereby identifying one or more recognition audio frames. A detailed description of a method for identifying one or more recognition audio frames through the threshold value analysis will be given below with reference to FIG. 6. In one embodiment, the server 100 may identify one or more predicted numerical information corresponding to each of the one or more acoustic frames (S310). As a specific example, the one or more acoustic frames may include a first acoustic frame and a second acoustic frame. The server 100 may process each acoustic frame as an input of the acoustic recognition model 500 and output a predicted value corresponding to each acoustic frame. Here, the predicted value may include one or more predicted item information and predicted numerical information corresponding to each predicted item information. Accordingly, the server 100 may identify predicted numerical information corresponding to each acoustic frame through a predicted value output by the acoustic recognition model corresponding to each acoustic frame.
[0053] For example, the server 100 can identify that the predicted numerical information corresponding to the "sound of breaking glass" and the "crying of a child" are "82" and "5" based on the predicted value corresponding to the first sound frame 411. Also, for example, the server 100 may identify that the predicted numerical information corresponding to the "sound of breaking glass" and the "sound of a siren" are "50" and "12", respectively, through the predicted value corresponding to the second sound frame 412. The specific numerical descriptions of the predicted numerical information described above are merely examples, and the present invention is not limited thereto. Also, the server 100 may identify a predetermined threshold value corresponding to one or more pieces of predicted item information (S320). In one embodiment, a threshold value may be preset corresponding to each piece of predicted item information. The threshold value may refer to a threshold value for identifying an audio recognition result having a certain level of accuracy or higher. For example, if the predicted numerical information corresponding to a first audio frame is equal to or higher than the threshold value, it may mean that the audio recognition result of the first audio frame is at a reliable level. As another example, if the predicted numerical information corresponding to a second audio frame is lower than the threshold value, it may mean that the audio recognition result of the second audio frame is somewhat lacking in accuracy. The above-mentioned specific descriptions of each audio frame are merely examples, and the present invention is not limited thereto.
[0054] The threshold value may be set differently for each prediction item. According to an embodiment, the threshold value for each prediction item may be determined in advance in response to the difficulty of sound recognition. For example, the more difficult the sound is to recognize, the lower the threshold value may be set, and the easier the sound is to recognize, the higher the threshold value may be set. The determination as to whether or not the sound is easy to recognize may be based on, for example, a distribution map of feature information included in each sound identification information in a vector space. In an embodiment, when feature information outputted in response to a specific sound identification information is widely distributed, the sound may be difficult to recognize, and when the feature information is densely distributed, the sound may be easy to recognize. That is, a threshold value may be set in response to each of one or more prediction item information. As a specific example, the threshold value for an explosion sound that is relatively easy to recognize may be 90, and the threshold value for a child's crying sound that is difficult to recognize may be 60. The specific description of the threshold values preset in association with each sound described above is merely exemplary, and the present invention is not limited thereto. The server 100 can identify one or more recognition audio frames by determining whether each of the predicted numerical information is equal to or greater than a predetermined threshold value (S330). Specifically, the server 100 can output a predicted value corresponding to each audio frame. In this case, the predicted value corresponding to each audio frame can include predicted item information and predicted numerical information.
[0055] As a specific example, the server 100 can identify, through a predicted value corresponding to the first sound frame 411, that the predicted numerical information corresponding to the “sound of breaking glass” and the “sound of a child crying” is “82” and “5”, respectively, and can identify, through a predicted value corresponding to the second sound frame 412, that the predicted numerical information corresponding to the “sound of breaking glass” and the “sound of a siren” is “50” and “12”, respectively.
[0056] In addition, the server 100 may identify a predetermined threshold value for each of the prediction item information (i.e., the sound of glass breaking, the child crying, and the sound of a siren) corresponding to each sound frame. For example, the predetermined threshold values corresponding to the sound of glass breaking, the child crying, and the sound of a siren may be identified as 80, 60, and 90, respectively. The server 100 can identify one or more recognition sound frames by comparing the predicted numerical information corresponding to each sound frame with the corresponding threshold value. Specifically, the server 100 can identify one or more recognition sound frames by determining whether each of the prediction numerical information included in the output prediction value is equal to or greater than a predefined threshold value.
[0057] In this case, the server 100 may identify the first sound frame 411 as one or more recognition sound frames by identifying that the predicted numerical information corresponding to the sound of breaking glass in the first sound frame 411 is 82, which is equal to or greater than the predetermined critical value of 80 related to the sound of breaking glass. Also, the server 100 may identify that the predicted numerical information corresponding to the sound of breaking glass in the second sound frame 412 is 50, which is the predetermined critical value, which is less than the predetermined critical value of 80 related to the sound of breaking glass, and may not identify the second sound frame 412 as one or more recognition sound frames. In other words, the server 100 can identify, as one or more recognized audio frames, only audio frames having prediction numerical information equal to or greater than a predetermined threshold value among one or more audio frames constructed based on the audio data 400. That is, the server 100 can eliminate frames associated with less accurate recognition results among the audio frames and identify only frames having a certain degree of reliability or higher as one or more recognized audio frames.
[0058] According to an embodiment of the present invention, the server 100 may identify a converted audio frame through a time series analysis based on one or more recognized audio frames (S400). Here, the time series analysis may refer to an analysis of observing the time when audio data is acquired and determining whether there is an erroneously recognized audio. A detailed description of a method of identifying one or more converted audio frames through a time series analysis will be given below with reference to FIG. 7.
[0059] In one embodiment, the server 100 may identify predicted item information corresponding to each of one or more recognition sound frames (S410). That is, the server 100 may identify what kind of sound each of one or more recognition sound frames identified as a result of the threshold value analysis relates to. For example, the one or more recognition sound frames may include a first sound frame, a fourth sound frame, and a fifth sound frame. In this case, the server 100 may identify predicted item information of each sound frame. For example, the predicted item information of the first sound frame may include a "sound of breaking glass", the predicted item information of the fourth sound frame may include a "sound of a siren", and the predicted item information of the fifth sound frame may include a "sound of a siren". The above-mentioned specific descriptions of the one or more recognition sound frames and predicted item information are merely examples, and the present invention is not limited thereto. In an embodiment, the server 100 may determine whether the predicted item information is repeated a predetermined number of times or more during a predetermined reference time (S420). Specifically, the server 100 may have a reference time and a critical number of times preset for each predicted item information. For example, in the case of a puppy barking, the reference time may be preset as a time associated with two sound frames, and the critical number of times may be preset to two. In other words, when one or more recognition sound frames are associated with a puppy barking, the server 100 may identify a reference time and a critical number of times preset for the corresponding item information (i.e., a puppy barking) and determine whether the puppy barking has been repeated and recognized a predetermined number of times during the reference time. That is, the server 100 may determine whether a specific sound is continuously recognized by a predetermined reference value through one or more recognition sound frames.
[0060] The server 100 can identify a converted audio frame based on the determination result (S430). When the server 100 determines that a particular audio has not been continuously recognized for a set reference value through one or more recognition audio frames (i.e., that it has been repeated a predetermined critical number of times within a predetermined reference time), it can identify at least one of the one or more recognition audio frames as a converted audio frame. Here, the converted audio frame can refer to an audio frame that is to be converted to reduce the probability of misrecognition, i.e., to improve the recognition accuracy. According to an embodiment of the present invention, the server 100 may perform a transformation on a predicted value corresponding to the transformed audio frame (S500). In an embodiment, the transformation on the predicted value may include at least one of a noise transformation and an audio item transformation.
[0061] Noise transformation may refer to transforming the output of an acoustic recognition model based on a transformed acoustic frame into an unrecognized item, i.e., transforming the output (i.e., predictions) of an acoustic recognition model associated with a transformed frame into an unrecognized item (e.g., others). The acoustic item transformation may refer to transforming predicted item information associated with the transformed acoustic frame into calibrated predicted item information, where the calibrated predicted item information may be determined based on associations of the predicted item information. As a specific example, when prediction item information related to a converted audio frame includes information of "sound of washing hands", "sound of a toilet filling with water" that is associated with the corresponding sound of washing hands may be determined as proofread prediction item information. In this case, the server 100 may convert the prediction item information so that the converted audio frame related to the sound of washing hands is recognized as the sound of a toilet filling with water. The above-mentioned specific descriptions of the prediction item information and proofread prediction item information are merely examples, and the present invention is not limited thereto. As a result, when the server 100 determines that a particular sound has not been recognized consecutively for a set reference value through one or more recognized sound frames (i.e., determined that the sound has been repeated a predetermined number of times or more within a predetermined reference time), it can identify a conversion frame and perform conversion of the predicted value of the corresponding conversion frame. In this case, the conversion of the predicted value of the conversion frame can mean converting the conversion frame so that it is not recognized (i.e., converting it to an item not to be recognized) or converting it so that it is recognized as another sound that does not cause a recognition error when trying to recognize an event. Such conversion can be made possible by each of one or more sound frames partially overlapping with an adjacent sound frame. For example, a sound frame that is recognized alone in one sound frame because it is recognized to overlap partially through a second time unit can be identified as a conversion frame and converted. In other words, when trying to detect an event by targeting a specific sound, the recognition accuracy of the voice data can be improved by correcting (or converting) the sound frames related to misrecognition so that they do not cause recognition errors.
[0062] According to an embodiment of the present invention, the server 100 can identify a correlation between each of the one or more recognition audio frames based on prediction item information corresponding to each of the one or more recognition audio frames. For example, the one or more recognition audio frames can include a first audio frame and a second audio frame. The first audio frame can include prediction item information of "sound of flushing a toilet", and the second audio frame can include prediction item information of "sound of washing hands". The server 100 can identify a correlation between each of the audio frames. For example, the server 100 can identify a correlation in which acquisition of the second audio frame is predicted after acquisition of the first audio frame. In an embodiment, the server 100 may determine whether to adjust the threshold value and the threshold number of times corresponding to one or more sound frames based on the association relationship. The server 100 may adjust the threshold value and the threshold number of times corresponding to the sound frames based on the association relationship between the sound frames. That is, the threshold value and the threshold number of times preset for each sound item may be variably adjusted based on the association relationship between the sound frames. As a more specific example, the server 100 may be provided to detect an event related to the sound of water flushing a toilet. In this case, a first sound frame acquired based on sound data may include predicted item information of "the sound of water flushing a toilet," and a second sound frame may include predicted item information of "the sound of washing hands." For example, the sound related to the second sound frame may also relate to the sound of water flushing, and may be similar to the sound event (i.e., the sound of water flushing a toilet) detected or recognized by the server 100. Accordingly, the server 100 may identify a correlation between sound frames (i.e., a correlation that the acquisition of the sound of washing hands is predicted after the acquisition of the first sound frame) and adjust a threshold value and a threshold count corresponding to a predicted item related to the sound of washing hands.
[0063] For example, the server 100 may adjust a threshold value corresponding to a sound prediction item related to the sound of washing hands from the existing 80 to 95. Accordingly, the reference value for determining the sound of washing hands in the process of the threshold value analysis is improved, and the accuracy of the recognition may be improved. In this case, by setting a higher reference value than before, the probability that the sound frame is recognized as the sound of washing hands is reduced, and the accuracy of the event recognition related to the sound of flushing a toilet may be improved. As another example, the server 100 may adjust the threshold number of times for a sound prediction item related to the sound of washing hands after the sound of flushing a toilet from the current 2 times to 5 times. When the sound of washing hands is recognized alone, it is determined to have been recognized normally even if it is recognized only twice in a row, but it is determined to have been recognized only if it is acquired five times repeatedly after a related sound (i.e., the sound of flushing a toilet). That is, by variably adjusting the threshold value and the threshold number of times according to the correlation between sounds, the sound frame acquired at the next time point can be processed as an item not targeted for recognition (e.g., "others"). In other words, when a first sound frame is recognized, the threshold value and the threshold number of times associated with a second sound frame (e.g., the sound of washing hands) associated with the corresponding first sound frame (e.g., the sound of flushing a toilet) are adjusted to increase the reference value, so that when a second sound frame is acquired thereafter, it can be processed to be recognized as an item not targeted for recognition. As a result, the accuracy of recognition associated with the first sound frame to be detected can be maximized.
[0064] 8 shows an exemplary table for explaining an acoustic data correction process according to an embodiment of the present invention, and FIG 9 shows an exemplary diagram for explaining an acoustic data correction process according to an embodiment of the present invention. FIG. 8 may be a table related to predicted values output by the acoustic recognition model corresponding to a case where five acoustic frames are configured based on acoustic data. As shown in FIG. 8, the five acoustic frames may include a first acoustic frame corresponding to 0 to 1 seconds, a second acoustic frame corresponding to 0.5 to 1.5, a third acoustic frame corresponding to 1 to 2 seconds, a fourth acoustic frame corresponding to 1.5 to 2.5 seconds, and a fifth acoustic frame corresponding to 2 to 3 seconds. In this case, the first time unit 400a may be 1 second, and the second time unit 400b may be 0.5 seconds. The start time points of adjacent acoustic frames are configured to differ by the size of the second time unit 400b, which is smaller than the size of the first time unit 400a, so that each acoustic frame may at least partially overlap with each adjacent acoustic frame.
[0065] Also, the prediction item information corresponding to each sound frame and the prediction numerical information corresponding to each prediction item information may be as shown in FIG. 8. For example, the closer the prediction numerical information is to 1, the higher the prediction probability may be, and the closer it is to 0, the lower the prediction probability may be. For example, it can be confirmed that the output of siren corresponding to 0.5 to 1.5 seconds, i.e., the second sound frame, is 0.9, which is the highest. This may mean that the sound acquired during 0.5 to 1.5 seconds is highly likely to be siren. FIG. 9(a) is an example diagram showing a result of performing a threshold value analysis corresponding to the predicted value of FIG. 8. Referring to FIG. 9(a), in the case of siren, it can be seen that only frames (i.e., the second, third and fourth sound frames) that are equal to or greater than a threshold value (e.g., 0.6) are identified. In addition, in the case of scream, it can be seen that only frames (i.e., the first and fourth sound frames) that are equal to or greater than a threshold value (e.g., 0.3) are identified. In addition, in the case of glass break, it can be seen that only frames (i.e., the fifth sound frame) that are equal to or greater than a threshold value (e.g., 0.7) are identified. For example, sound frames equal to or greater than a threshold value corresponding to each predicted item can be identified as one or more recognized sound frames.
[0066] Figure 9(b) is an example diagram showing a result of performing a time series analysis corresponding to the predicted value of Figure 8. In this case, the predetermined reference time may be preset to a time associated with two acoustic frames, and the critical number may be preset to two. Referring to (a) and (b) of Figure 9, in the case of siren, it can be seen that when the recognition result of a recognition acoustic frame is observed two consecutive times, only the related frame remains as a recognition target.
[0067] Specifically, in FIG. 9(a), the second sound frame is identified as one or more recognized sound frames, but as a result of time series analysis, it can be seen that it has been converted as shown in FIG. 9(b). That is, in FIG. 9(a), the first sound frame and the second sound frame are not observed twice consecutively, so the server 100 can perform correction by identifying the second sound frame as a converted frame and converting it to "others". Accordingly, as shown in FIG. 9(b), an "x" may be displayed in the second sound frame area of "siren". This may indicate that the sound of "siren" has not been recognized in the corresponding section. With respect to a later time point, since the second sound frame and the third sound frame all have prediction numerical information equal to or greater than the critical value, it can be determined that "siren" has been recognized in the third sound frame. Similarly, since the third sound frame and the fourth sound frame all have prediction numerical information equal to or greater than the critical value, it can be determined that "siren" has been recognized in the fourth sound frame.
[0068] In addition, in the case of scream, the threshold value analysis result shows that a scream was detected to have occurred in relation to the first and fourth sound frames, as in Figure 9(a). However, since no scream was observed to occur twice consecutively in either the third or fourth sound frame during the time series analysis process, the server 100 may identify the fourth sound frame as a conversion frame and perform a correction to convert it to "others." Accordingly, as in Figure 9(b), an "x" may be displayed in the fourth sound frame area of the scream. This may indicate that the sound of the siren was not recognized in the corresponding section. In a further embodiment, the server 100 may provide information on the recognition result of the entire sound data to the user terminal 200. That is, the information on the recognition result of the entire sound data may include information on what sound was recognized at each time point (e.g., each sound frame) corresponding to the entire sound data acquired in time series. For example, the information on the recognition result of the entire sound data may be the same as that shown in FIG. 9(c).
[0069] Referring to FIG. 9(c), information that siren has been recognized in relation to the second sound frame may be displayed. In this case, the second sound frame related to siren may be converted (or corrected) since the sound of siren was not recognized in the time series analysis process as shown in FIG. 9(b). In the embodiment, when the server 100 provides information on the recognition result of the entire sound data, it may restore the result that exceeded the threshold value but was excluded in the time series analysis process. This may be to reflect the corresponding recognition result when providing the overall recognition information, although the sound recognition process uses it as a recognition target only if it is recognized consecutively two or more times.
[0070] The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in a software module executed by hardware, or in a combination thereof. The software module may reside in a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Flash Memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer readable recording medium commonly known in the art to which the present invention pertains. The components of the present invention may be embodied as a program (or application) and stored on a medium for execution in conjunction with a computer, which is hardware. The components of the present invention may be implemented as software programs or software elements, and similarly, embodiments may be implemented in programming or scripting languages such as C, C++, Java, assembler, etc., including various algorithms embodied as a combination of data structures, processes, routines, or other programming constructs. Functional aspects may be embodied as algorithms executed by one or more processors. Although the embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art to which the present invention pertains will understand that the present invention can be embodied in other specific forms without changing the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.
Claims
1. 1. A method performed by one or more processors of a computing device, comprising: constructing one or more acoustic frames based on the acoustic data; processing each of the one or more acoustic frames as an input to an acoustic recognition model and outputting a prediction corresponding to each acoustic frame; identifying one or more recognition sound frames through a threshold value analysis based on the predicted values corresponding to each sound frame; identifying a transformed acoustic frame through time series analysis based on the one or more recognition acoustic frames; and performing a transformation on a prediction value corresponding to the transformed acoustic frame.
2. The step of constructing one or more acoustic frames based on the acoustic data comprises:
2. The method for improving recognition accuracy of acoustic data according to claim 1, further comprising: dividing the acoustic data into predetermined first time unit sizes to form the one or more acoustic frames.
3. The start time of each of the one or more acoustic frames is 3. The method for improving recognition accuracy of acoustic data according to claim 2, wherein the difference between the start time points of adjacent acoustic frames and the magnitude of the second time unit is determined.
4. The predicted value is One or more pieces of forecast item information and forecast numerical information corresponding to each of the one or more pieces of forecast item information are included, The critical value analysis 2. The method for improving the recognition accuracy of acoustic data according to claim 1, wherein the analysis identifies the one or more recognition acoustic frames by determining whether each of the one or more pieces of predicted numerical information corresponding to each acoustic frame is equal to or greater than a predetermined threshold value corresponding to each piece of predicted item information.
5. The step of identifying the transformed acoustic frames through time series analysis comprises: identifying predicted item information corresponding to each of the one or more recognition acoustic frames; determining whether the identified predictive item information is repeated a predetermined number of times or more during a predetermined reference time period; and 5. The method for improving recognition accuracy of acoustic data according to claim 4, further comprising: identifying the transformed acoustic frame based on the discrimination result.
6. The method comprises: identifying a correlation between each of the one or more recognition acoustic frames based on predicted item information corresponding to each of the recognition acoustic frames; and The method for improving the recognition accuracy of acoustic data according to claim 5, further comprising: determining whether to adjust a critical value and a critical number of times corresponding to each of the one or more acoustic frames based on the correlation.
7. The transformation for the predicted value is 2. The method for improving the recognition accuracy of acoustic data according to claim 1, further comprising at least one of a noise transformation for converting an output of the acoustic recognition model based on the converted acoustic frame into an unrecognized item, and an acoustic item transformation for converting predicted item information related to the converted acoustic frame into calibrated predicted item information.
8. The calibration prediction item information is The method for improving the recognition accuracy of acoustic data according to claim 7, wherein the prediction item information is determined based on a correlation between the prediction items.
9. memory for storing one or more instructions; and a processor that executes the one or more instructions stored in the memory; 10. An apparatus, wherein the processor performs the method of claim 1 by executing the one or more instructions.
10. A computer program stored on a computer-readable recording medium so that the computer program can be combined with a computer that is hardware to perform the method according to claim 1.
Citation Information
Patent Citations
Scream detection device
JP2012048173A
Voice recognition accuracy deterioration factor estimation device, voice recognition accuracy deterioration factor estimation method and program
JP2019139010A
Non-audible murmur input alarm device, method, and program
WO2008007616A1
Apparatus for dectecting aucoustic event and method and operating method thereof
KR1020140143069A
