Artificial neural network model learning apparatus and inference system for speech and sound recognition

Abstract

The present invention relates to an artificial neural network model learning apparatus and inference system for speech and sound recognition, which comprise: a first learning agent that trains a first artificial neural network model for inference by using pieces of first audio data among a plurality of pieces of audio data collected from a first inference terminal; and a first evaluation agent that evaluates the first artificial neural network model trained by the first learning agent by using pieces of second audio data among the plurality of pieces of audio data collected from the first inference terminal and distributes, through a local network, the first artificial neural network model trained by the first inference terminal according to an evaluation result.

Description

An artificial neural network model training device and inference system for speech and acoustic recognition.

[0001] The present invention relates to an artificial neural network model learning device and an inference system for speech and sound recognition, and more particularly, to an artificial neural network model learning device and an inference system for speech and sound recognition capable of adapting an artificial neural network model for each individual environment by reflecting noise and sound in each individual environment in which the artificial neural network model is performed and performing inference directly adapted in each individual environment.

[0002] Vision and hearing are the primary senses through which humans perceive situations, and systems simulating these senses using artificial intelligence are being developed. Along with visual information, voice and sound (hereinafter collectively referred to as "audio," meaning signals with a bandwidth perceptible to the auditory sense) information are effective means of recognizing and interacting with situations.

[0003] Traditionally, physical contact means such as buttons and levers were primarily used for human interaction with machines, but with the development of artificial intelligence and voice recognition technology, voice user interface (VUI) is being used as the main interface technology for smartphones and Internet of Things (IoT) devices.

[0004] Advances in artificial intelligence and high-speed network technologies (e.g., 5G cellular) have made it possible for smartphones and IoT devices to provide voice recognition interfaces. Typically, voice information collected from the device's microphone is transmitted to a cloud server, where it is analyzed and the results are returned to the device. While the device operates based on the server's analysis, this process is delivered via high-speed network technology, allowing users to interact seamlessly with the device.

[0005] Voice recognition technology holds promise for user interfaces and intelligent services, but it is susceptible to interference from noise, depending on the environment. Furthermore, the patterns and intensities of noise and ambient sounds vary across different environments, leading to problems with devices accurately recognizing defined voices or sounds.

[0006] A representative technology used for voice recognition interfaces and services is Automatic Speech Recognition (ASR), which analyzes speech information and automatically converts it into sentences. Recent high-performance automatic speech recognition technologies are based on the Transformer model, which requires significant computational resources and memory space. Therefore, standalone operation on smartphones or IoT devices is difficult, and connection to a cloud server via a high-speed network is essential.

[0007] Transformer-based high-performance automatic speech recognition technology offers high accuracy and can decipher most speech, but it requires significant computational processing. Therefore, devices that struggle with high-performance neural network computation, such as microcontrollers (MCUs) or application processors (APs), are adopting keyword spotting (KWS) technology, which recognizes only predefined, specific words (keywords). This keyword spotting technology is typically used to recognize wake words or wake words in smartphones or Internet of Things (IoT) devices, but it can also be used for device control commands.

[0008] Although existing voice recognition technology can be applied in various fields, there are several problems that need to be solved.

[0009] First, an automatic speech inference system for voice recognition transmits collected voice information to a cloud server, where the server analyzes, processes, and transmits the inference results to a remote terminal in response. However, the voice information collected from the terminal may contain information unintended by the user, potentially exposing private or confidential information to the cloud server.

[0010] Furthermore, existing automatic speech inference systems require terminals to collect voice information and transmit it to a cloud server via the Internet. If the cloud server infers the voice information and the network is interrupted, voice service becomes unavailable. This poses a challenge for applications where reliability and stability are critical.

[0011] The automatic speech inference system performs inference using the same speech recognition model of the cloud server. However, when the cloud server performs speech recognition on speech information that includes noise or ambient sound other than speech, the performance of the artificial neural network model for speech recognition varies greatly depending on the intensity or pattern of the noise or ambient sound.

[0012] Moreover, the intensity, pattern, and shape of noise and ambient sound vary depending on the usage environment of the terminal transmitting voice information, making it difficult to learn and improve a voice recognition model that reflects noise or ambient sound.

[0013] In this way, an artificial neural network model learning device and inference system for voice and sound recognition that can improve various problems of existing known automatic speech recognition technologies are required.

[0014] The present invention has been devised to solve the above-described problems, and its purpose is to provide an artificial neural network model learning device and inference system that can eliminate or reduce the exposure of privacy or secret information accompanying audio recognition using an artificial neural network.

[0015] In addition, the present invention aims to provide an artificial neural network model learning device and inference system that can perform audio recognition using an artificial neural network and necessary actions therefor at low cost in an individual environment having a built-in system even in the event of network disconnection.

[0016] In addition, the present invention aims to provide an artificial neural network model learning device and inference system capable of audio recognition by adapting to noise in an implementation environment by collecting audio data having ambient noise or noise and training a speech recognition model that can be used in the field using the collected audio data.

[0017] In addition, the present invention aims to provide an artificial neural network model learning device and an inference system that train a voice recognition model using audio data collected for each individual environment, provide and perform the learned voice recognition model for each individual environment, and adaptively increase robust audio recognition efficiency in various environments with different noises and noises and perform necessary actions in the field.

[0018] The technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0019] An artificial neural network model learning device according to one aspect of the present invention learns an artificial neural network model distributed to an inference terminal using audio data collected from an inference terminal through a local network, and includes a first learning agent that learns a first artificial neural network model for inference using first audio data among a plurality of audio data collected from a first inference terminal, and a first evaluation agent that evaluates the first artificial neural network model learned by the first learning agent using second audio data among a plurality of audio data collected from the first inference terminal and distributes the first artificial neural network model learned to the first inference terminal through a local network based on the evaluation result.

[0020] In the above-described artificial neural network model learning device, the first learning agent learns a first artificial neural network model for inference in the second inference terminal using first audio data among a plurality of audio data collected through a local network from a second inference terminal different from the first inference terminal, and the first evaluation agent evaluates the first artificial neural network model learned for the second inference terminal in the first learning agent using second audio data among a plurality of audio data collected from the second inference terminal, and distributes the first artificial neural network model learned for the second inference terminal to the second inference terminal based on the evaluation result through the local network, wherein the local network is a private network.

[0021] In the above-described artificial neural network model learning device, a second learning agent that learns a second artificial neural network model for removing environmental noise occurring around a first inference terminal using third audio data among a plurality of audio data collected from a first inference terminal, and a second evaluation agent that evaluates the second artificial neural network model learned by the second learning agent using fourth audio data among a plurality of audio data collected from the first inference terminal and distributes the second artificial neural network model learned by the first inference terminal through a local network according to the evaluation result, wherein the learned first artificial neural network model and the second artificial neural network model are executed by the first inference terminal to remove environmental noise from audio data collected by the first inference terminal and recognize a target sound source from the removed audio data.

[0022] In the above-described artificial neural network model learning device, the artificial neural network model learning device learns a first artificial neural network model for a first inference terminal that is fine-tuned from an existing first artificial neural network model using a plurality of audio data collected during a learning period at a first inference terminal, and learns a first artificial neural model for a second inference terminal that is fine-tuned from an existing first artificial neural network model using a plurality of audio data collected during a learning period at a second inference terminal.

[0023] In the above-described artificial neural network model learning device, the first learning agent receives audio data synthesized with first audio data and pre-stored target sound source data as audio data for inference learning, the first evaluation agent receives audio data synthesized with second audio data and pre-stored target sound source data as audio data for inference evaluation, the second learning agent receives audio data synthesized with third audio data and pre-stored target sound source data as audio data for environmental sound removal learning, and the second evaluation agent receives audio data synthesized with fourth audio data and pre-stored target sound source data as audio data for environmental sound removal evaluation.

[0024] In addition, an artificial neural network inference terminal according to one aspect of the present invention includes a microphone for receiving an audio signal, a communication interface for transmitting and receiving a communication packet through a local network, a memory for storing audio data from the microphone and a program of a first artificial neural network model, an output unit for outputting a corresponding signal, and a microcontroller for executing a code of the program of the memory, wherein the microcontroller transmits a communication packet including audio data collected through the microphone to an artificial neural network model learning device connected to the local network through the communication interface, receives a first artificial neural network model for inference according to learning and evaluation of audio data collected from the artificial neural network model learning device through the local network, stores the first artificial neural network model in the memory, and outputs a corresponding signal to the output unit when a target sound source is recognized according to the execution of the first artificial neural network model.

[0025] In the artificial neural network inference terminal described above, the memory further stores a second artificial neural network model learned on audio data received from an artificial neural network model learning device and collected through a microphone, and a microcontroller executing the program of the second artificial neural network model and the program of the first artificial neural network model inputs audio data collected through the microphone into the second artificial neural network model to generate audio data from which environmental noise has been removed, inputs the audio data from which environmental noise has been removed into the first artificial neural network model to generate an inference result indicating whether a target sound source has been recognized, and generates a corresponding signal according to the inference result indicating recognition of the target sound source and outputs the signal to an output unit.

[0026] In the above artificial neural network inference terminal, the first artificial neural network model is a lightweight convolutional neural network model, the microcontroller is a controller based on a RISC (reduced instruction set computer) architecture that directly executes the programs of the first artificial neural network model and the second artificial neural network model, and the artificial neural network inference terminal is implemented as an embedded system.

[0027] In addition, an artificial neural network inference system according to one aspect of the present invention includes the artificial neural network model learning device, the first inference terminal for transmitting and receiving communication packets with the learning device through a local network, and the second inference terminal for transmitting and receiving communication packets with the learning device through a local network, wherein the learning device learns a first artificial neural network model performed in the first inference terminal using audio data collected from the first inference terminal, and learns the first artificial neural network model performed in the second inference terminal using audio data collected from the second inference terminal.

[0028] The artificial neural network model learning device and inference system according to the present invention as described above have the effect of eliminating or reducing the exposure of privacy or confidential information accompanying audio recognition using an artificial neural network.

[0029] In addition, the artificial neural network model learning device and inference system according to the present invention as described above has the effect of being able to perform audio recognition using an artificial neural network and necessary actions therefor at low cost in an individual environment having a built-in system even when the network is disconnected.

[0030] In addition, the artificial neural network model learning device and inference system according to the present invention as described above collects audio data having ambient noise or noise, and trains a voice recognition model that can be used in the field by utilizing the collected audio data, thereby adapting to the noise of the implementation environment and enabling audio recognition.

[0031] In addition, the artificial neural network model learning device and inference system according to the present invention as described above trains a voice recognition model using audio data collected for each individual environment, and provides and performs the learned voice recognition model for each individual environment, thereby increasing the efficiency of adaptively robust audio recognition in various environments with different noises and noises and enabling necessary actions to be taken in the field.

[0032] The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention belongs from the description below.

[0033] FIG. 1 is a diagram illustrating an example of an artificial neural network inference system according to the present invention.

[0034] Figure 2 is a block diagram illustrating an artificial neural network model learning device.

[0035] FIG. 3 is a drawing showing an exemplary functional block diagram of an artificial neural network model learning device according to the present invention.

[0036] Figure 4 is a drawing illustrating an exemplary hardware block diagram of an inference terminal.

[0037] Figure 5 is a drawing showing an exemplary functional block diagram of an inference terminal.

[0038] FIG. 6 is a diagram illustrating a schematic operation of an inference terminal and a learning device in an artificial neural network inference system according to the present invention and a control flow therebetween.

[0039] <Explanation of symbols>

[0040] 100: Learning Device

[0041] 101: Communication module 105: Storage module

[0042] 109: Control module

[0043] 151: Communication Agent 153: System Management Agent

[0044] 155: Learning data management agent 157: First learning agent

[0045] 159: First evaluation agent 161: Second learning agent

[0046] 163: Second Evaluation Agent

[0047] 500: Inference terminal

[0048] 501: Microphone 503: Communication Interface

[0049] 505: Output section 507: Memory

[0050] 509: Microcontroller 551: Audio receiving agent

[0051] 553: Audio data collection agent 555: Communication agent

[0052] 557: First artificial neural network model execution agent

[0053] 559: Agent executing the second artificial neural network model

[0054] 561: Output Management Agent

[0055] 900: Local network

[0056] The above-described objects, features, and advantages will become more apparent through the detailed description set forth below with reference to the accompanying drawings, thereby enabling those skilled in the art to easily implement the technical concepts of the present invention. Furthermore, in describing the present invention, detailed descriptions of known technologies related to the present invention will be omitted if they are deemed to unnecessarily obscure the gist of the present invention. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0057]

[0058] FIG. 1 is a diagram illustrating an example of an artificial neural network inference system according to the present invention.

[0059] As in the example of Fig. 1, the artificial neural network inference system is configured to include a learning device (100), one or more inference terminals (500), and a local network (900) connecting the learning device (100) and the inference terminals (500).

[0060] According to the present invention, an artificial neural network inference system comprises a learning device (100) and an inference terminal (500) connected to each other via a local network (900). The local network (900) is implemented as a private network separated from a public network such as the Internet (e.g., separated via a gateway or firewall). The local network (900) is a private network configured using communication protocols such as Ethernet, Zigbee, Wi-Fi, and Bluetooth.

[0061] The artificial neural network inference system of the present invention is configured such that the inference terminal (500) directly executes the artificial neural network model distributed from the learning device (100), thereby recognizing a target audio source from the collected audio data and taking necessary response actions based on the recognition of the target audio source. The inference terminal (500) is preferably implemented as an embedded system (built-in system).

[0062] The inference terminal (500) is distributed by the learning device (100) and executes a program of an artificial neural network model that is learned by reflecting the surrounding environmental sounds of the inference terminal (500). The inference terminal (500) can recognize a specific target sound source (data) and perform a specific response action (e.g., outputting a warning sound, stopping actuator operation, opening a door, etc.) set according to the recognition of the target sound source. The inference terminal (500) is implemented in the form of an embedded system and is installed in a specific place or location. Each of the plurality of inference terminals (500) is configured to transmit and receive communication packets through the learning device (100) and a local network (900).

[0063] The artificial neural network inference system of the present invention can be applied to various fields and locations. For example, the artificial neural network inference system can be installed in a subway station. The inference terminal (500) is installed on a subway screen door and, upon recognizing a predetermined target sound source (e.g., "Open"), operates a connected actuator to open the installed screen door.

[0064] For example, an artificial neural network inference system can be installed in an (automated) factory. The inference terminal (500) can be installed on various automated lines in the factory and, upon recognizing a predetermined target sound source (e.g., "stop") for an emergency stop of the automated line, drive an actuator for stopping the automated line or generate an emergency signal.

[0065] For example, an artificial neural network inference system can be installed for home intrusion detection and security services. An inference terminal (500) installed in a specific location within the home can generate an alarm sound based on the recognition of specific target sound sources, such as the sound of a window breaking, a latch shattering, or wood breaking, and output a signal indicating an external intrusion via a wired or wireless network. The artificial neural network inference system of the present invention can be utilized in various forms and environments in addition to the application examples described above.

[0066] Each of the inference terminals (500) of the artificial neural network inference system of the present invention collects audio data representing or including ambient environmental sounds used for learning and transmits the collected audio data to the learning device (100) via a local network (900). The inference terminal (500) receives an artificial neural network model trained with such audio data and is configured to adaptively recognize a specific target sound source in response to noise in the surrounding environment.

[0067] The inference terminal (500) in a noisy environment must adaptively operate in response to loud noises to recognize the target sound source. Since the ambient noise (noise) inevitably varies depending on the environment in which the inference terminal (500) is installed, the use of an artificial intelligence model that adaptively operates in response to ambient noise is essential for each inference terminal (500). The inference terminal (500) will be discussed in detail in FIG. 4 and below.

[0068] The learning device (100) trains an artificial neural network model executed in an inference terminal (500) and distributes the trained artificial neural network model to the inference terminal (500) via a local network (900). The learning device (100) fine-tunes (or fine-tunes) the artificial neural network model so that it can recognize a set target sound source by using ambient noise (environmental sound) collected in the inference terminal (500) as learning data. The learning device (100) can transmit the fine-tuned artificial neural network model to the inference terminal (500) via the local network (900). The transmitted artificial neural network model is preferably directly executed by the inference terminal (500). The learning device (100) is preferably configured to train the artificial neural network model executed in the individual inference terminal (500) by using audio data collected from the individual inference terminal (500).

[0069] The learning device (100) may be implemented as a personal computer or a workstation, etc. The learning device (100) uses audio data collected from individual or group inference terminals (500) as learning data to train an artificial neural network model executed in the individual or group inference terminals (500). The learning device (100) is configured to distribute the learning model to the individual or group inference terminals (500) based on the evaluation of the learning model. Depending on the design variation, the learning device (100) may also be built into the inference terminal (500).

[0070] The learning device (100) will be examined in more detail with reference to FIGS. 2, 3, and 6.

[0071]

[0072] Figure 2 is a drawing showing a block diagram of an artificial neural network model learning device (100).

[0073] The block diagram of the learning device (100) of FIG. 2 represents a conceptual HW block diagram, and the learning device (100) may further have other HW blocks not shown in FIG. 2. For example, the learning device (100) may further be configured to include input / output blocks.

[0074] The learning device (100) of FIG. 2 is connected to a local network (900) and is used to learn an artificial neural network model used in an inference terminal (500). The learning device (100) may be implemented as a personal computer or a workstation with medium / low specifications instead of a high-performance cloud server. The learning device (100) is configured to train an artificial neural network model distributed to the inference terminal (500) using audio data collected from the inference terminal (500) via the local network (900).

[0075] The learning device (100) of FIG. 2 is configured to include a communication module (101), a storage module (105), and a control module (109), and each module may be implemented on a board or implemented as a chipset and / or electronic circuit.

[0076] The communication module (101) is configured to include a chipset, circuits, and / or board for performing communication with a local network (900). For example, the communication module (101) may be configured to include an interface, chipset, circuits, antenna, etc. of a physical layer and / or a link layer for performing a set communication protocol of the local network (900). The communication module (101) is configured to be able to communicate with at least the inference terminal (500) and may further be configured to be able to communicate with a public network such as the Internet. At least, the communication module (101) is configured to be able to transmit and receive communication packets according to a communication protocol such as Ethernet, Zigbee, Wi-Fi, or Bluetooth.

[0077] The storage module (105) stores various data and programs, including volatile memory, non-volatile memory, and / or large-capacity storage media such as HDD. The storage module (105) stores various programs executed in the control module (109), various artificial neural network models (programs) transmitted to the inference terminal (500), and various audio data received from the inference terminal (500). The storage module (105) may include (store) a database for managing various data, artificial neural network models, and audio data collected from the inference terminal (500).

[0078] The control module (109) includes one or more CPUs and graphic cards. The control module (109) loads a program executed in the CPU or graphic card from the storage module (105) and controls and manages the learning device (100) according to the execution of the program. At least, the control module (109) is configured to train and evaluate one or more artificial neural network models for the inference terminal (500) and distribute the trained artificial neural network model to the inference terminal (500) through a local network (900) (communication module (101) connected thereto) according to the evaluation result. The control module (109) can execute the program(s) of the storage module (105) in the CPU or graphic card to train one or more artificial neural network models that are linked to each other and transmit the trained one or more artificial neural network models to the inference terminal (500) individually or in groups.

[0079]

[0080] FIG. 3 is a drawing showing an exemplary functional block diagram of an artificial neural network model learning device (100) according to the present invention.

[0081] Figure 3 is a functional block diagram preferably implemented by the control module (109) of the program(s) stored in the storage module (105). The functional blocks may be configured as individual programs, or some of the functional blocks may be configured as a single integrated program. The program may be implemented in various forms. For example, the program may be implemented in various forms such as a library, a linkable program module, an executable file, etc.

[0082] According to FIG. 3, the learning device (100) is configured to include a system management agent (153), one or more learning agents, one or more evaluation agents, a learning data management agent (155), a communication agent (151), etc. The learning device (100) preferably has at least two learning agents (157, 161) and at least two evaluation agents (159, 163).

[0083] Hereinafter, among the two learning agents, a specific learning agent is referred to as a "first learning agent" and the other specific learning agent is referred to as a "second learning agent." In addition, among the two evaluation agents, a specific evaluation agent that evaluates the artificial neural network model of the first learning agent (157) is referred to as a "first evaluation agent," and a specific other evaluation agent that evaluates the artificial neural network model of the second learning agent (161) is referred to as a "second evaluation agent."

[0084] The system management agent (153) controls and manages the learning device (100) and, furthermore, the artificial neural network inference system. The system management agent (153) distributes control policies for inference terminals (500) connected to the local network (900) through the communication agent (151). For example, the system management agent (153) can transmit collection-related control policies, such as the learning cycle, collection cycle, and transmission cycle of audio data, to each inference terminal (500).

[0085] In addition, the system management agent (153) is trained in advance with target sound source data reflecting the characteristics of various speakers and configures a base artificial neural network model (hereinafter referred to as the 'first artificial neural network model') for recognizing the target sound source and a base artificial neural network model (hereinafter referred to as the 'second artificial neural network model') for removing environmental noise (noise) around the inference terminal (500) from a series of audio data received from the inference terminal (500). The system management agent (153) provides the base first artificial neural network model to the first learning agent (157) and provides the base second artificial neural network model for noise removal to the second learning agent (161).

[0086] (Basic) The first artificial neural network model is preferably a Lightweight Convolutional Neural Network model, and the basic first artificial neural network model and the basic second artificial neural network model are prepared in advance by a system administrator, etc. The (basic) second artificial neural network model may be a Lightweight Convolutional Neural Network model or another type of artificial neural network model that is known or will become known.

[0087] The target audio source, which is the recognition target in the inference terminal (500), is predetermined according to the application environment of the inference system and may represent the voice of a specific word (speech keyword) or a specific sound. For example, the target audio source may represent words such as "open" or "stop," or may represent sounds such as the sound of a window breaking, a latch shattering, or wood breaking.

[0088] Target audio data is data representing the target audio source as audio data. The system management agent (153) may provide a plurality of target audio data representing the same target audio source, generated from various speakers and in various environments, to the learning agent (157, 161).

[0089] The communication agent (151) transmits and receives various data and control policies to and from the inference terminal (500) via the local network (900). For example, the communication agent (151) transmits a collection-related control policy of a specific inference terminal (500) received from the system management agent (153) to the specific inference terminal (500) via the local network (900). The communication agent (151) can receive collected audio data from the inference terminal (500) and transmit the received collected audio data to the learning data management agent (155).

[0090] The learning data management agent (155) manages data used for learning the first artificial neural network model and the second artificial neural network model. The learning data management agent (155) stores audio data collected from the inference terminal (500) in the database of the storage module (105) by inference terminal (500) (e.g., ID of the inference terminal (500)). According to a design variation example, audio data collected from the inference terminal (500) within each inference terminal group is stored by matching it to the corresponding inference terminal group (group ID). The inference terminals (500) may be grouped according to their installation locations. For example, inference terminals (500) within a set distance (e.g., 10 m) may be grouped into one group, and learning, evaluation, and distribution may be performed by inference terminal group.

[0091] As the learning cycle for a (group of) (specific) inference terminals (500) set by the system management agent (153) arrives, the learning data management agent (155) classifies audio data collected from the (group of) inference terminals (500) for which the learning cycle has arrived into learning audio data(s) and evaluation audio data(s). The learning cycle, which is linked to the classification ratio (e.g., 1:1, etc.) or the storage period of audio data in the database, may be set by the system management agent (153).

[0092] The learning data management agent (155) provides classified learning audio data(s) and target sound source data(s) to be recognized to the first learning agent (157) and the second learning agent (161), and provides classified evaluation audio data(s) and target sound source data(s) to be recognized to the first evaluation agent (159) and the second evaluation agent (163).

[0093] Classified learning audio data(s) and target sound source data(s) that are recognition targets and pre-stored in the storage module (105) are synthesized into audio by the learning data management agent (155) or by the first learning agent (157) and the second learning agent (161). For example, the first learning agent (157) and the second learning agent (161) can receive audio data synthesized by the learning data management agent (155) as audio data for inference learning and audio data for environmental noise removal learning from the learning data management agent (155).

[0094] The classified evaluation audio data(s) and the recognition target sound source data(s) are synthesized into audio by the learning data management agent (155) or by the first evaluation agent (159) and the second evaluation agent (163). For example, the first evaluation agent (159) and the second evaluation agent (163) can receive the audio data synthesized by the learning data management agent (155) as audio data for inference evaluation and audio data for environmental noise removal evaluation from the learning data management agent (155).

[0095] Depending on the various design examples, the classified learning audio data(s) provided to the first learning agent (157) and the second learning agent (161) may be the same or different. Additionally, the classified evaluation audio data(s) provided to the first evaluation agent (159) and the second evaluation agent (163) may be the same or different.

[0096] The first learning agent (157) trains the first artificial neural network model of the inference terminal (500) (group) using classified learning audio data (hereinafter also referred to as “first audio data”) among audio data collected for each inference terminal (500). The first learning agent (157) receives the first audio data and the target sound source data(s) of the target sound source to be recognized from the learning data management agent (155), inputs the synthesized first audio data and the target sound source data into the first artificial neural network model, and trains the first artificial neural network model of the inference terminal (500) (group) through (fine) tuning of the parameters (e.g., picture map) of the first artificial neural network model. The first learning agent (157) can train the first artificial neural network model using the audio data(s) collected as the learning cycle of the inference terminal (500) arrives.

[0097] In addition, the first learning agent (157) trains the first artificial neural network model of the other specific inference terminal (500) (group) using the classified first audio data among the audio data collected for each other specific inference terminal (500). The first learning agent (157) receives the first audio data and the target sound source data of the target sound source to be recognized from the learning data management agent (155), inputs the synthesized first audio data and the target sound source data into the first artificial neural network model of the other specific inference terminal (500) (group), and trains the first artificial neural network model of the other specific inference terminal (500) (group) through (fine) tuning of the parameters (e.g., picture map) of the first artificial neural network model. The first learning agent (157) can train the first artificial neural network model of the other specific inference terminal (500) using the audio data(s) collected as the learning cycle of the other specific inference terminal (500) arrives.

[0098] In this way, the first learning agent (157) trains the first artificial neural network model executed in the individual inference terminal (500) (group) using audio data collected for each individual inference terminal (500) (group). The individual inference terminal (500) (group) executing the trained first artificial neural network model dynamically adapts to ambient environmental sounds (noise, etc.) and can easily recognize target sound source data despite various types of different ambient environmental sounds generated in the individual inference terminal (500) (group).

[0099] The first evaluation agent (159) evaluates the first artificial neural network model learned by the inference terminal (500) (group) using audio data for evaluation (hereinafter also referred to as “second audio data”) classified from among audio data collected by each inference terminal (500) (group). The first evaluation agent (159) distributes the learned first artificial neural network model to the inference terminal (500) (group) through a local network (900) (via a communication agent (151)) according to the evaluation result.

[0100] The first evaluation agent (159) receives second audio data and target sound source data from the learning data management agent (155), inputs the synthesized second audio data and target sound source data into the trained first artificial neural network model, and receives a recognition result from the first artificial neural network model. The first evaluation agent (159) can input a plurality of synthesized audio data into the first artificial neural network model and receive a recognition result indicating a recognition probability. If the (multiple) recognition probabilities exceed a set threshold probability, the first evaluation agent (159) can determine that the trained first artificial neural network model has been improved and distribute the result to (a group of) the inference terminal (500) used for audio collection.

[0101] In addition, the first evaluation agent (159) evaluates the first artificial neural network model learned for the other specific inference terminal (500) (group) using second audio data among the audio data collected for each other specific inference terminal (500) (group). The first evaluation agent (159) distributes the learned first artificial neural network model to the other specific inference terminal (500) (group) through the local network (900) (via the communication agent (151)) according to the evaluation result.

[0102] The first evaluation agent (159) receives the second audio data collected from the other specific inference terminal (500) (group) and the target sound source data of the target sound source to be recognized from the learning data management agent (155), inputs the synthesized second audio data and the target sound source data into the trained first artificial neural network model, and receives the recognition result from the first artificial neural network model. The first evaluation agent (159) can input a plurality of synthesized audio data into the first artificial neural network model and receive the recognition result indicating the recognition probability. If the (multiple) recognition probabilities exceed a set threshold probability, the first evaluation agent (159) can determine that the trained first artificial neural network model has been improved and distribute it to the other specific inference terminal (500) (group) used for audio collection.

[0103] Here, the signal strength of the collected audio data (first audio data and second audio data) input to the synthesis during audio synthesis of the collected audio data can be changed. The collected audio data (first audio data and second audio data) with a signal strength lower (reduced, for example, 1 / 2, 1 / 3, etc.) than the signal strength of the original collected audio data can be synthesized with the target sound source data, and the first learning agent (157) and the first evaluation agent (159) can learn and evaluate with the synthesized audio data.

[0104] In this way, the first evaluation agent (159) can evaluate the first artificial neural network model for each inference terminal (500) using audio data collected from the inference terminal (500) and including ambient sound, thereby improving the recognition performance of the inference terminal (500). In addition, dynamic recognition performance can be improved even when changes in the ambient sound occur.

[0105] The second learning agent (161) trains a second artificial neural network model for removing environmental noise occurring around the inference terminal (500) (group) using classified learning audio data (hereinafter also referred to as “third audio data”) among audio data collected for each inference terminal (500) (group). The third audio data may be the same as or different from the first audio data.

[0106] The second learning agent (161) inputs audio data synthesized from the third audio data and the target sound source data into the second artificial neural network model of the inference terminal (500) (group) and trains the second artificial neural network model of the inference terminal (500) (group) through (fine) tuning of the parameters (e.g., picture map) of the second artificial neural network model. The second learning agent (161) can train the second artificial neural network model using the audio data(s) collected as the learning cycle of the inference terminal (500) arrives.

[0107] The second learning agent (161) can also train the second artificial neural network model of the other specific inference terminal (500) (group) using target sound source data and third audio data collected from the other specific inference terminal (500) (group) and synthesized audio data according to the arrival of the learning cycle of the other specific inference terminal (500) (group).

[0108] The second evaluation agent (163) evaluates the second artificial neural network model that has been trained by the inference terminal (500) (group) using the classified evaluation audio data (hereinafter also referred to as “fourth audio data”) among the audio data collected by each inference terminal (500) (group). The second evaluation agent (163) distributes the second artificial neural network model trained by the inference terminal (500) (group) through the local network (900) (via the communication agent (151)) according to the evaluation result. The fourth audio data may be the same as or different from the second audio data.

[0109] The second evaluation agent (163) receives the fourth audio data and target sound source data from the learning data management agent (155), inputs the synthesized fourth audio data and target sound source data into the second artificial neural network model trained by the second learning agent (161), and receives the recognition result from the second artificial neural network model. The second evaluation agent (163) can input a plurality of synthesized audio data into the second artificial neural network model and receive the recognition result indicating the recognition probability. If the (multiple) recognition probabilities exceed a set threshold probability, the second evaluation agent (163) can determine that the trained second artificial neural network model has been improved and distribute it to (a group of) the inference terminal (500) used for audio collection.

[0110] In addition, the second evaluation agent (163) uses the fourth audio data among the audio data collected for each other specific inference terminal (500) (group) to evaluate the second artificial neural network model that has been trained for each other specific inference terminal (500) (group) based on the recognition result and distributes it through the local network (900).

[0111] The first learning agent (157), the second learning agent (161), the first evaluation agent (159), and the second evaluation agent (163) and the first artificial neural network model and the second artificial neural network model are developed and implemented using a software platform capable of developing artificial neural networks. For example, the learning device (100) can implement the first learning agent (157), the second learning agent (161), the first evaluation agent (159), and the second evaluation agent (163) using the TensorFlow platform, the deep learning library PYTORCH, SPeechBrain, etc., and can implement and train the first artificial neural network model and the second artificial neural network model.

[0112] Through the functional blocks of Fig. 3, the first artificial neural network model and the second artificial neural network model that are learned are distributed to the corresponding inference terminal (500) and executed in the corresponding inference terminal (500). The inference terminal (500) that receives and executes the first artificial neural network model and the second artificial neural network model can remove ambient noise from audio data collected through the microphone (501) and recognize a target sound source from the removed audio data.

[0113] As shown in Fig. 3, the learning device (100) trains a first artificial neural network model for target sound source recognition and a second artificial neural network model for removing noise, etc., and distributes them to the inference terminal (500). By separating the first artificial neural network model and the second artificial neural network model, for example, when the target sound source changes, the second artificial neural network model, whose performance has already been improved, is applied while intensively training only the first artificial neural network model, thereby maintaining performance for target sound source recognition and achieving rapid improvement.

[0114] In this way, the learning device (100) learns a first artificial neural network model for one inference terminal (500) through fine tuning of the first artificial neural network model that is performed previously using audio data collected during a learning period in one inference terminal (500) and distributes it to the one inference terminal (500). In addition, the learning device (100) learns a first artificial neural network model for another inference terminal (500) through fine tuning of the first artificial neural network model that is performed previously using audio data collected during a learning period in another inference terminal (500) and distributes it to the other inference terminal (500). Accordingly, the learning device (100) is capable of voice recognition by removing or adapting to various types of noise that are caused or generated depending on the installation environment of the inference terminal (500).

[0115]

[0116] FIG. 4 is a drawing showing an exemplary hardware block diagram of an inference terminal (500).

[0117] As shown in the example of Fig. 4, the inference terminal (500) is configured to include a microphone (501), a communication interface (503), an output unit (505), a memory (507), and a microcontroller (509). The inference terminal (500) may further include other blocks not shown in Fig. 4. The inference terminal (500) is implemented as a (small) embedded system and may be installed at a specific location or place where the artificial neural network inference system operates.

[0118] A brief look at the hardware block of Fig. 4 shows that the microphone (501) receives an audio signal. The microphone (501) can convert a vibration signal into an electrical signal and output the converted electrical signal as an analog audio signal.

[0119] The communication interface (503) transmits and receives communication packets through the local network (900). The communication interface (503) includes a communication chipset (e.g., Pi chip) and circuits, antennas, connectors, etc. required for implementing communication protocols such as Ethernet, Zigbee, Wi-Fi, and / or Bluetooth, and can transmit and receive communication packets through the local network (900).

[0120] The memory (507) further includes non-volatile memory and volatile memory to store various data and programs. For example, the memory (507) may store audio data received through the microphone (501), a program of the first artificial neural network model and a program of the second artificial neural network model executed in the microcontroller (509), and may further store other management (control) programs.

[0121] The program of the first artificial neural network model and the program of the second artificial neural network model learned from audio data collected through the microphone (501) of the inference terminal (500) and transmitted to the learning device (100) are received from the learning device (100), stored in the memory (507), and executed by the microcontroller (509). The microcontroller (509) that executes the program of the first artificial neural network model and the program of the second artificial neural network model learned is configured to recognize a target sound source from audio data received from the microphone (501).

[0122] The output unit (505) is connected to the microcontroller (509) and outputs a corresponding signal received from the microcontroller (509). The output unit (505) has a circuit for operating a speaker, an actuator, and / or a circuit for opening a door, and receives a control signal for controlling the corresponding device from the microcontroller (509) and outputs a corresponding signal for controlling the corresponding device.

[0123] The output unit (505) may output, for example, a warning sound, a signal to stop the operation of an actuator such as a motor that is being driven, or a signal to open a closed door. Depending on the design example and various application examples, the output unit (505) may be implemented in various forms.

[0124] The microcontroller (509) controls the inference terminal (500). The microcontroller (509) is configured to execute the program code of the memory (507). The microcontroller (509) is implemented as a low-spec controller instead of a GPU (Graphics Processing Unit) or NPU (Neural Processing Unit) specialized for executing artificial neural networks or an AP (Application Processor) installed in a smartphone.

[0125] The microcontroller (509) is a controller based on a RISC (reduced instruction set computer) architecture and is implemented to execute program code in an ALU (Arithmetic Logic Unit) based on load / store instructions. The microcontroller (509) may have an ARM processor IP built-in (including) that directly executes (in an ALU) the programs of the first artificial neural network model and the second artificial neural network model of the memory (507) based on load / store instructions, or may have an Xtensa architecture IP built-in.

[0126] For example, the microcontroller (509) is configured to have a low-spec 32-bit ARM processor or Xtensa processor built in and to execute the programs of the first artificial neural network model and the second artificial neural network model in the ARM processor or Xtensa processor under an RTOS (Real-Time Operating System) environment to remove noise (ambient noise) from audio data from the microphone (501) and recognize the target sound source. According to the configuration of the inference terminal (500), the inference terminal (500) can be implemented inexpensively and can perform voice recognition efficiently and economically without using a cloud server or a remote server.

[0127] In FIGS. 5 and 6, we will examine in more detail the specific control related to target sound source recognition performed in the inference terminal (500) (microcontroller (509)).

[0128]

[0129] Figure 5 is a drawing showing an exemplary functional block diagram of an inference terminal (500).

[0130] According to FIG. 5, the inference terminal (500) is configured to include an audio receiving agent (551), an audio data collection agent (553), a communication agent (555), a first artificial neural network model execution agent (557), a second artificial neural network model execution agent (559), and an output management agent (561).

[0131] The audio receiving agent (551), the audio data collection agent (553), the communication agent (555), the first artificial neural network model execution agent (557), the second artificial neural network model execution agent (559), and the output management agent (561) may be configured as individual programs, or some of the functional blocks may be configured as a single integrated program. The program may be implemented in various forms, such as a library, a linkable program module, an executable file, etc.

[0132] An audio receiving agent (551), an audio data collection agent (553), a communication agent (555), a first artificial neural network model execution agent (557), a second artificial neural network model execution agent (559), and an output management agent (561), and further, a program of the first artificial neural network model and a program of the second artificial neural network model are executed by a microcontroller (509) of an inference terminal (500).

[0133] The audio receiving agent (551) receives audio data through a microphone (501). The audio receiving agent (551) provides a series of audio data according to a sampling period to the second artificial neural network model execution agent (559) and provides a series of audio data according to a sampling period to the audio data collection agent (553).

[0134] The communication agent (555) transmits and receives communication packets with the learning device (100) through a communication interface (503) that accesses a local network (900). The communication agent (555) (microcontroller (509)) receives collection-related control policies such as collection cycles and transmission cycles from the learning device (100), transmits the received collection-related control policies to the audio data collection agent (553), transmits the program of the learned first artificial neural network model to the first artificial neural network model execution agent (557), and transmits the program of the learned second artificial neural network model to the second artificial neural network model execution agent (559).

[0135] The program of the first artificial neural network model for inference based on learning and evaluation of audio data collected from the inference terminal (500) is stored in the memory (507), and the program of the second artificial neural network model for removing ambient noise, which is learned and evaluated using audio data collected for removing ambient noise, is also stored in the memory (507).

[0136] The audio data collection agent (553) stores audio data (audio data capable of recognizing ambient sounds) for a certain period of time from the audio receiving agent (551) in the memory (507) when the collection cycle arrives according to the collection-related control policy. When the transmission cycle (longer than the collection cycle) arrives, the audio data collection agent (553) can transmit a plurality of audio data collected according to the collection cycle to the learning device (100) through the communication agent (555). Depending on the design application, the audio data collection agent (553) can collect audio data during idle time when no corresponding signal is generated and transmit the collected audio data when the transmission cycle arrives.

[0137] In this way, the audio data collection agent (553) (microcontroller (509) that performs the same) transmits a communication packet containing audio data collected through a microphone (501) to an artificial neural network model learning device (100) connected to a local network (900) through a communication interface (503).

[0138] The second artificial neural network model execution agent (559) executes the program of the learned second artificial neural network model received from the communication agent (555) via the local network (900). The second artificial neural network model execution agent (559) (microcontroller (509) of the second artificial neural network model) inputs a series of audio data collected via the microphone (501) into the program of the learned second artificial neural network model to generate audio data from which ambient noise has been removed. The program of the second artificial neural network model is collected from the inference terminal (500) and trained with audio data having ambient noise, so that it can generate audio data from which ambient noise has been removed with high accuracy. In addition, it is possible to dynamically adapt to changes in ambient noise to remove ambient noise.

[0139] The first artificial neural network model execution agent (557) executes the program of the learned first artificial neural network model received from the communication agent (555) via the local network (900). The first artificial neural network model execution agent (557) (microcontroller (509)) receives audio data output from the program of the second artificial neural network model (audio data from which ambient noise is mainly removed by the second artificial neural network model) and recognizes a target sound source from the input audio data. The first artificial neural network model execution agent (557) can output an inference result indicating whether the target sound source is recognized.

[0140] The first artificial neural network model execution agent (557) inputs audio data from which ambient sounds have been (mainly) removed by the program of the second artificial neural network model into the first artificial neural network model, and can output an inference result indicating recognition of the target sound source when the recognition probability output from the first artificial neural network model is greater than or equal to the threshold probability.

[0141] In this way, the first artificial neural network model execution agent (557) can recognize the target sound source from audio data from which the ambient sound has been removed through dynamic learning of the ambient sound generated from the inference terminal (500). According to a design variation example, the execution and application of the second artificial neural network model execution agent (559) (and the second artificial neural network model program) may be omitted, so that audio data from the audio reception agent (551) may be directly supplied to the first artificial neural network model execution agent (557).

[0142] The output management agent (561) (microcontroller (509)) outputs a corresponding signal to the output unit (505) based on the inference result from the first artificial neural network model execution agent (557). The output management agent (561) can generate a corresponding signal and output it to the output unit (505) based on the reception of the inference result indicating recognition of the target sound source.

[0143] As described above, the inference terminal (500) receives an artificial neural network model learned from ambient environmental sounds generated around the inference terminal (500) from the learning device (100) according to a learning cycle and executes the received artificial neural network model on a RISC-based microcontroller (509). The inference terminal (500) can directly execute the artificial neural network model. Furthermore, the artificial neural network model can be individually tuned to the ambient environmental sounds and made lightweight enough to be executed on the microcontroller (509).

[0144]

[0145] FIG. 6 is a diagram illustrating a schematic operation of an inference terminal (500) and a learning device (100) in an artificial neural network inference system according to the present invention and a control flow therebetween.

[0146] The artificial neural network inference system may have multiple inference terminals (500), and the control flow of FIG. 6 may be performed for each individual inference terminal (500). Since the configuration and control of the inference terminal (500) and the learning device (100) have already been examined in detail through FIGS. 1 to 5, here we will briefly examine the control sequence.

[0147] First, the inference terminal (500) and the learning device (100) set up an environment for target sound source recognition (see ①). The learning device (100) transmits the program of the first (or basic) artificial neural network model and the program of the first (or basic) second artificial neural network model to the inference terminal (500) via a local network (900). In addition, the learning device (100) transmits collection-related control policies, such as collection cycles and transmission cycles, to the inference terminal (500) via the local network (900).

[0148] The inference terminal (500) collects and stores audio data through a microphone (501) according to a collection cycle (see ②). The inference terminal (500) transmits a plurality of collected audio data to a learning device (100) according to the arrival of a transmission cycle (see ③).

[0149] The learning device (100) matches the received audio data to the inference terminal (500) (group) and stores it in the storage module (105) (database), and as the learning cycle of the inference terminal (500) arrives, the collected audio data is classified into learning audio data(s) and evaluation audio data(s) (see ④). The learning device (100) uses the classified audio data to learn and evaluate (see ⑤) the artificial neural network model of the inference terminal (500).

[0150] The learning device (100) transmits the artificial neural network model to the inference terminal (500) through a local network (900) based on the evaluation result (see ⑥) and updates the current artificial neural network model (ID) being used in the inference terminal (500) in the database.

[0151] The inference terminal (500) receives and applies an artificial neural network model. Thereafter, the inference terminal (500) receives audio data through a microphone (501), inputs the received audio data into the applied artificial neural network model, attempts to recognize a target sound source, and outputs a corresponding signal when the target sound source is recognized (see ⑦).

[0152] The inference terminal (500) and the learning device (100) repeatedly perform processes ② to ⑦ according to the learning cycle (which is usually longer than the transmission cycle and the collection cycle). Here, processes ② to ⑦ of FIG. 6 are described as being performed sequentially, but processes ② to ⑦ may also be performed independently or in parallel.

[0153] As discussed above, the artificial neural network inference system according to the present invention can eliminate or reduce the exposure of privacy or confidential information associated with voice recognition. Furthermore, the artificial neural network inference system can recognize target sound sources (audio) adapted to the surrounding environment at low cost, even when the network is disconnected.

[0154] The present invention described above is not limited to the above-described embodiments and the attached drawings, as various substitutions, modifications, and changes can be made without departing from the technical spirit of the present invention by a person having ordinary skill in the art to which the present invention pertains.

Claims

1. A device that trains an artificial neural network model distributed to an inference terminal using audio data collected from an inference terminal through a local network, A first learning agent that trains a first artificial neural network model for inference using first audio data among a plurality of audio data collected from a first inference terminal; and A first evaluation agent that evaluates a first artificial neural network model learned by the first learning agent using second audio data among a plurality of audio data collected from the first inference terminal and distributes the learned first artificial neural network model to the first inference terminal through the local network according to the evaluation result; Artificial neural network model learning device.

2. In paragraph 1, The first learning agent trains a first artificial neural network model for inference in the second inference terminal by using first audio data among a plurality of audio data collected through the local network in a second inference terminal different from the first inference terminal, The first evaluation agent evaluates the first artificial neural network model learned for the second inference terminal from the first learning agent using second audio data among the plurality of audio data collected from the second inference terminal, and distributes the first artificial neural network model learned for the second inference terminal to the second inference terminal through the local network based on the evaluation result. The above local network is a private network, Artificial neural network model learning device.

3. In paragraph 1, A second learning agent that trains a second artificial neural network model for removing environmental noise occurring around the first inference terminal using third audio data among a plurality of audio data collected from the first inference terminal; and A second evaluation agent that evaluates the second artificial neural network model learned by the second learning agent using fourth audio data among a plurality of audio data collected from the first inference terminal and distributes the second artificial neural network model learned by the first inference terminal through the local network according to the evaluation result; The learned first artificial neural network model and the second artificial neural network model are performed in the first inference terminal to remove ambient sound from audio data collected in the first inference terminal and recognize a target sound source from the removed audio data. Artificial neural network model learning device.

4. In paragraph 2, The above artificial neural network model learning device learns a first artificial neural network model for the first inference terminal that is fine-tuned from an existing first artificial neural network model using a plurality of audio data collected during a learning period in the first inference terminal, and learns a first artificial neural model for the second inference terminal that is fine-tuned from an existing first artificial neural network model using a plurality of audio data collected during a learning period in the second inference terminal. Artificial neural network model learning device.

5. In paragraph 3, The above first learning agent receives audio data synthesized with the above first audio data and pre-stored target sound source data as audio data for inference learning, The above first evaluation agent receives audio data synthesized with the above second audio data and the above pre-stored target sound source data as audio data for inference evaluation. The second learning agent receives audio data synthesized with the third audio data and pre-stored target sound source data as audio data for environmental noise removal learning, The second evaluation agent receives audio data synthesized with the fourth audio data and the pre-stored target sound source data as audio data for environmental noise removal evaluation. Artificial neural network model learning device.

6. A microphone that receives audio signals; A communication interface that transmits and receives communication packets over a local network; A memory for storing audio data from the microphone and a program of the first artificial neural network model; An output section that outputs a corresponding signal; and A microcontroller that executes the code of the program of the memory; a microcontroller that transmits a communication packet including audio data collected through the microphone to an artificial neural network model learning device connected to a local network through the communication interface, receives a first artificial neural network model for inference according to learning and evaluation of the collected audio data from the artificial neural network model learning device through the local network, stores the first artificial neural network model in the memory, and outputs a corresponding signal to the output unit when a target sound source is recognized according to the execution of the first artificial neural network model; Artificial neural network inference terminal.

7. In paragraph 6, The above memory further stores a second artificial neural network model that is learned on audio data received from the artificial neural network model learning device and collected through the microphone, The microcontroller, which executes the program of the second artificial neural network model and the program of the first artificial neural network model, inputs audio data collected through a microphone into the second artificial neural network model to generate audio data with environmental noise removed, inputs the audio data with environmental noise removed into the first artificial neural network model to generate an inference result indicating whether a target sound source is recognized, and generates the corresponding signal according to the inference result indicating recognition of the target sound source and outputs it to the output unit. Artificial neural network inference terminal.

8. In paragraph 7, The above first artificial neural network model is a lightweight convolutional neural network model, The above microcontroller is a controller based on RISC (reduced instruction set computer) architecture that directly executes the programs of the first artificial neural network model and the second artificial neural network model. The above artificial neural network inference terminal is implemented as an embedded system. Artificial neural network inference terminal.

9. Artificial neural network model learning device of paragraph 1; The first inference terminal of claim 6, which transmits and receives communication packets through the learning device and a local network; and Including the second inference terminal of the sixth clause that transmits and receives communication packets through the learning device and the local network; The learning device trains a first artificial neural network model performed in the first inference terminal using audio data collected from the first inference terminal, and trains a first artificial neural network model performed in the second inference terminal using audio data collected from the second inference terminal. Artificial neural network inference system.

Images