Learning database construction method and system using voice personal information protection technology

Abstract

The present invention provides a method and system for constructing a learning database using voice personal information protection technology that can secure data including sound information whose personal information is protected as learning data for machine learning. [Solution] The present invention relates to a learning database construction method and system using voice personal information protection technology, the method includes the steps of receiving video data including audio data, separating the audio data from the video data, extracting background sound data from the audio data, and storing the video data and the background sound data from which the audio data has been removed as learning data. Thus, the present invention can secure data including audio information with protected personal information as learning data for machine learning.

Description

[Technical Field]

[0001] The present invention relates to a technology for generating training data for machine learning, and more particularly to a method and system for constructing a training database using a voice personal information protection technology that can secure data including sound information whose personal information is protected as training data for machine learning. [Background technology]

[0002] Machine learning methods are broadly classified into three types: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning is a method in which correct answer data (a pair of input data and its corresponding label) already exists, and the correct answer is communicated to the learning model, so that the error between the model's predicted value and the correct answer is minimized. Unsupervised learning is a learning method in which correct answer data does not exist (only input data exists), and classification is performed by analyzing the similarities between data and hidden characteristics. Reinforcement learning is a method of learning through the process of assigning rewards and punishments to the model's decisions in an environment in which correct answer data does not exist.

[0003] Supervised learning has the advantage of being easier to learn, more stable, and easier to evaluate than unsupervised learning or reinforcement learning because clear correct answer data already exists. However, the process of preparing the training data accounts for a large portion of supervised learning, requiring a lot of time and human resources. In addition, the quantity and quality of the training data have a significant impact on the recognition performance of the trained machine learning model, so the key to supervised learning is the generation of the training data.

[0004] On the other hand, although audio from video contains a lot of information that can be utilized, it is highly likely to infringe on personal privacy, so great care must be taken when generating training data based on video information. That is, even if audio modulation is applied, individuals can be identified by the intonation and tone of their voice, so in order to utilize sound information containing voice, the audio information must be processed in a way that makes it impossible to identify individuals.

[0005] In particular, a recognition sensor is essential for recognizing and judging the driving situation of a vehicle, and such a recognition sensor can generally be a camera, LIDAR, radar, etc. In machine learning, the data acquired by such a recognition sensor can be used to train a machine learning model. The more information contained in the data acquired by the sensor, the more advantageous it is for improving the performance of the machine learning model. Therefore, by adding sound information inside and outside the vehicle that is unrelated to cameras, LIDAR, and radar as machine learning data, the performance of the machine learning model can be expected to improve.

[0006] However, since the voice information contained in the sound is sensitive information that contains personal information that can identify an individual, it is not desirable to store and utilize this information without the user's consent. Methods such as voice modulation are used to protect individual privacy, but even with voice modulation, individuals can still be identified to some extent by the intonation and tone of their voice. Therefore, in order to utilize sound information containing voice, it is necessary to process the voice information so that it becomes impossible to identify an individual. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Korean Patent Registration No. 10-1581641 Summary of the Invention [Problem to be solved by the invention]

[0008] An embodiment of the present invention provides a method and system for constructing a learning database using voice personal information protection technology, which can secure data including sound information whose personal information is protected as learning data for machine learning.

[0009] One embodiment of the present invention provides a method and system for building a learning database using voice personal information protection technology, which can separate background sounds from voice data, encrypt only the voice using lossy encoding, convert the voice into text corresponding to the voice, and then remove personal information. [Means for solving the problem]

[0010] Among the embodiments, a method for constructing a learning database using audio personal information protection technology includes the steps of receiving video data including audio data, separating the audio data from the video data, extracting background audio data from the audio data, and storing the video data from which the audio data has been removed and the background audio data as learning data.

[0011] Separating the sound data may include applying at least one of a plurality of pre-processing methods to the sound data.

[0012] The step of extracting the background sound data may include the steps of: defining a machine learning-based network model including a deep neural network; constructing a first network model that receives the sound data as input and generates voice data as output; constructing a second network model that receives the sound data as input and generates the background sound data as output; and separating the voice data and the background sound data from the sound data based on the first and second network models.

[0013] The step of extracting the background sound data may include the steps of: constructing a third network model that receives the audio data as input and generates an audio feature vector as output; irreversibly encoding the audio data based on the third network model; and storing the audio feature vector generated by the irreversible encoding as the training data.

[0014] The step of extracting the background sound data may include the steps of constructing a fourth network model that receives the audio data as input and generates text data as output, and extracting the text data from the audio data based on the fourth network model.

[0015] The step of extracting the background sound data may include the steps of detecting personal information from the text data, converting the personal information from the text data into anonymous information, and storing the text data including the anonymous information as the training data.

[0016] The step of converting the personal information into anonymous information may include substituting the personal information with a higher class name based on a machine learning-based conversion model.

[0017] Among the embodiments, a learning database construction system using audio personal information protection technology includes a video receiving unit that receives video data including audio data, a sound extraction unit that separates the audio data from the video data, a background sound separation unit that extracts background sound data from the audio data, and a learning data storage unit that stores the video data from which the audio data has been removed and the background sound data as learning data. [Effects of the Invention]

[0018] The disclosed technology may have the following effects. However, this does not mean that a particular embodiment should include all of the following effects or only the following effects, and the scope of the disclosed technology should not be understood to be limited thereby.

[0019] A method and system for building a learning database using voice personal information protection technology according to an embodiment of the present invention can secure data containing voice information whose personal information is protected as learning data for machine learning.

[0020] A learning database construction method and system using voice personal information protection technology according to an embodiment of the present invention can separate background sounds from voice data, apply lossy encoding to only the voice, encrypt it, convert it into text corresponding to the voice, and then remove personal information. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram illustrating a learning database construction system according to the present invention. [Figure 2] FIG. 2 is a diagram illustrating the system configuration of the learning database construction device of FIG. [Figure 3] FIG. 2 is a diagram illustrating the functional configuration of the learning database construction device of FIG. [Figure 4] 1 is a flowchart illustrating a method for constructing a learning database using a voice personal information protection technology according to the present invention. [Figure 5] 1 is a diagram illustrating an embodiment of a background sound and speech separation method according to the present invention; [Figure 6] 1 is a diagram illustrating an embodiment of a feature vector calculation method according to the present invention and its non-reversible characteristics. [Figure 7] 1 is a diagram illustrating an embodiment of a feature vector calculation method according to the present invention and its non-reversible characteristics. [Figure 8] 1 is a diagram illustrating an embodiment of a text conversion method according to the present invention; [Figure 9] FIG. 1 is a diagram illustrating the overall concept of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] The description of the present invention is merely an embodiment for the purpose of structural or functional description, and therefore the scope of the present invention should not be interpreted as being limited by the embodiments described herein. In other words, since the embodiments can be modified in various ways and have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. Furthermore, the objectives or effects presented in the present invention do not mean that a particular embodiment should include all of these or only these effects, and therefore the scope of the present invention should not be understood as being limited thereby.

[0023] Meanwhile, the meanings of the terms used in this application should be understood as follows. Terms such as "first" and "second" are used to distinguish one component from another and should not be used to limit the scope of rights. For example, a first component may be called a second component, and similarly, a second component may be called a first component.

[0024] When a component is said to be "connected" to another component, it should be understood that it may be directly connected to the other component, but there may be other components in between. Conversely, when a component is said to be "directly connected" to another component, it should be understood that there are no other components in between. However, other expressions describing the relationship between components, such as "between" and "directly between," or "adjacent to" and "directly adjacent to," should be analyzed similarly.

[0025] Singular expressions shall be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as "comprise" or "have" shall be understood to specify the presence of embodied features, numerals, steps, operations, components, parts, or combinations thereof, and shall not be understood to preclude the presence or possible addition of one or more other features, numerals, steps, operations, components, parts, or combinations thereof.

[0026] The identifiers (e.g., a, b, c, etc.) used in each step are for convenience of explanation and do not dictate the order of each step, and each step may occur in a different order than specified unless the context clearly dictates a specific order. That is, each step may occur in the same order as specified, may occur substantially simultaneously, or may occur in the reverse order.

[0027] The present invention can be embodied as a computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Furthermore, the computer-readable recording medium can be distributed among computer systems connected via a network, so that the computer-readable code can be stored and executed in a distributed manner.

[0028] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by a person of ordinary skill in the art to which this invention pertains. Commonly used, predefined terms should be interpreted as being consistent with the meaning they have in the context of the relevant art, and should not be interpreted as having an idealized or overly formal meaning unless expressly defined in this application.

[0029] FIG. 1 is a diagram illustrating a learning database construction system according to the present invention. As shown in FIG. 1, a learning database construction system 100 may be implemented to include a user terminal 110, a learning database construction device 130, and a database 150.

[0030] The user terminal 110 may correspond to a terminal device operated by a user. In an embodiment of the present invention, a user may be understood as one or more users, and the multiple users may be divided into one or more user groups. Each of the one or more users may correspond to one or more user terminals 110. That is, a first user may correspond to a first user terminal, a second user may correspond to a second user terminal, ..., an nth user (where n is a natural number) may correspond to an nth user terminal.

[0031] The user terminal 110 is one of the devices constituting the learning database construction system 100 and may correspond to a computer device capable of performing user actions including generating, modifying, and deleting learning data. For example, the user terminal 110 may be embodied as a smartphone, a laptop, or a computer operable to be connected to the learning database construction device 130, but is not limited thereto and may be embodied as various devices including a tablet PC.

[0032] In addition, the user terminal 110 may install and execute a dedicated program or application for interfacing with the learning database construction device 130. For example, the user terminal 110 may transmit predetermined video data to the learning database construction device 130 to generate learning data, and may access the learning database constructed by the learning database construction device 130. This process may be performed through an interface provided by the dedicated program or application.

[0033] Meanwhile, the user terminal 110 can be connected to the learning database construction device 130 through a network, and a plurality of user terminals 110 can be connected to the learning database construction device 130 simultaneously.

[0034] The learning database construction device 130 may be implemented as a computer that performs the learning database construction method according to the present invention, or a server corresponding to a program. The learning database construction device 130 may be connected to the user terminal 110 via a wired network or a wireless network such as Bluetooth, Wi-Fi, or LTE, and may transmit and receive data to and from the user terminal 110 via the network.

[0035] In addition, the learning database construction device 130 may be embodied to operate in connection with an independent external system (not shown in FIG. 1) to collect or provide learning data. In one embodiment, the learning database construction device 130 may be embodied by a cloud server, and may satisfy various user needs regarding the construction and utilization of a learning database through cloud services.

[0036] The database 150 may correspond to a storage device that stores various information required in the operation process of the learning database construction device 130. For example, the database 150 may store video data collected from various sources or information related to a learning algorithm and a learning model for constructing a machine learning model, but is not limited thereto, and may store information collected or processed in various forms in the process of the learning database construction device 130 performing the learning database construction method using the voice personal information protection technology according to the present invention.

[0037] Also, in FIG. 1, the database 150 is shown as a device independent of the learning database construction device 130, but this is not necessarily limited to this, and it can of course be embodied as a logical storage device included in the learning database construction device 130.

[0038] FIG. 2 is a diagram illustrating the system configuration of the learning database construction device 130 of FIG. As shown in FIG. 2, the learning database construction device 130 may include a processor 210 , a memory 230 , a user input / output unit 250 , and a network input / output unit 270 .

[0039] The processor 210 can execute a learning database construction procedure according to an embodiment of the present invention, manage the memory 230 that is read or created during this process, and schedule synchronization times between the volatile memory and non-volatile memory in the memory 230. The processor 210 can control the overall operation of the learning database construction device 130, and is electrically connected to the memory 230, the user input / output unit 250, and the network input / output unit 270 to control the data flow between them. The processor 210 can be embodied as a CPU (Central Processing Unit) of the learning database construction device 130.

[0040] The memory 230 may include a secondary storage device embodied as a non-volatile memory such as a solid state disk (SSD) or a hard disk drive (HDD) and used to store all data required for the learning database construction device 130, or a primary storage device embodied as a volatile memory such as a random access memory (RAM). The memory 230 may also store a set of instructions that are executed by the processor 210 electrically connected thereto to perform the learning database construction method according to the present invention.

[0041] The user input / output unit 250 includes an environment for receiving user input and an environment for outputting specific information to a user, and may include, for example, an input device including an adapter such as a touchpad, touch screen, visual keyboard, or pointing device, and an output device including an adapter such as a monitor or touch screen. In one embodiment, the user input / output unit 250 may correspond to a computer device connected via a remote connection, and in that case, the learning database construction device 130 may be implemented as an independent server.

[0042] The network input / output unit 270 provides a communication environment for connecting with the user terminal 110 through a network, and may include adapters for communication with, for example, a LAN (Local Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), and a VAN (Value Added Network). The network input / output unit 270 may also include a Wi-Fi adapter for wireless transmission of learning data. It may be embodied to provide a short-range communication function such as WiFi or Bluetooth, or a wireless communication function of 4G or higher.

[0043] FIG. 3 is a diagram illustrating the functional configuration of the learning database construction device of FIG. As shown in FIG. 3, the learning database construction device 130 may include a video receiving unit 310 , a sound extracting unit 330 , a background sound separating unit 350 , a learning data storage unit 370 , and a control unit 390 .

[0044] The video receiving unit 310 may receive video data including audio data. For example, the video data may include black box video captured through a black box while the vehicle is running, video captured through a recognition sensor such as a camera, lidar, or radar, aerial video, medical video, etc. Audio data included in the video may include background sound, white noise, voice, etc. The video receiving unit 310 may receive video data via a network, receive video transmitted by the user terminal 110, or search for and receive video stored in the database 150.

[0045] The video receiving unit 310 can also receive audio data and video data independently, i.e., the video receiving unit 310 can sequentially receive video data without audio data and corresponding audio data, or can receive a pair of video data and audio data.

[0046] In one embodiment, the image receiving unit 310 may perform preprocessing operations on the received image data. For example, the image receiving unit 310 may perform preprocessing operations such as dividing the image data into sections of a predetermined length or converting the image data to a predetermined resolution. The image receiving unit 310 may also generate a single image by integrating the original image and the preprocessed image. The image receiving unit 310 may process the image data through various preprocessing operations into a form that can be used in subsequent steps, and the image received or processed by the image receiving unit 310 may be stored and managed in the database 150.

[0047] The audio extraction unit 330 can separate audio data from video data. The audio extraction unit 330 can extract audio from video using commercial software, and if necessary, can also utilize a method of recording audio and video through playback of video data and then removing the audio from the video. The audio extraction unit 330 can separate the video data and audio data using various methods, and the separated video and audio can be stored and managed in the database 150.

[0048] In one embodiment, the sound extraction unit 330 may apply at least one of a plurality of pre-processing methods to the sound data. That is, the sound data may undergo a predetermined pre-processing step to suit the data processing. In particular, various methods may be applied to the pre-processing operation, and may be performed by a single method or a combination of multiple methods. For example, the sound extraction unit 330 may convert one-dimensional sound data into a two-dimensional spectrogram, apply absolute values to the two-dimensional spectrogram, and perform a pre-processing operation to normalize the absolute values based on the maximum absolute value.

[0049] Here, a spectrogram can be a method of visualizing and expressing a sound spectrum in a graph. More specifically, a 2D spectrogram corresponding to 1D sound data can form a structure that combines a waveform that visually represents changes in the amplitude axis over time and a spectrum that visually represents changes in the amplitude axis corresponding to changes in frequency. For example, a 2D spectrogram can be a graph that represents differences in amplitude on both the time axis and the frequency axis as changes in hue and density.

[0050] The background sound separation unit 350 can extract background sound data from sound data. Here, the background sound data can correspond to the result of removing human voices from the sound data. The background sound separation unit 350 can also remove only predetermined targeted sound information from the sound data by utilizing a learning model.

[0051] In one embodiment, the background sound separation unit 350 defines a machine learning-based network model including a deep neural network, constructs a first network model that receives sound data as input and generates voice data as output, and constructs a second network model that receives sound data as input and generates background sound data as output, and can separate voice data and background sound data from the sound data based on the first and second network models. That is, the background sound separation unit 350 can independently extract voice and background sound from the sound data through the machine learning-based network model. To this end, the background sound separation unit 350 can independently construct network models depending on the sound to be extracted.

[0052] More specifically, the background sound separation unit 350 may construct a first network model that extracts voice data from sound data and a second network model that extracts background sound data from sound data based on a predefined network model. For example, the first and second network models may be implemented as a deep neural network composed of a plurality of neural network-based encoders and decoders. Once the network models are constructed, the background sound separation unit 350 may extract voice and background sound by sequentially applying the first and second network models to the sound data to be separated. The extracted voice and background sound may be temporarily stored in the memory 230, and the background sound separation unit 350 may store only the background sound in the database 150 and delete the voice without separate storage, thereby preventing the leakage of personal information.

[0053] In one embodiment, the background sound separation unit 350 may construct a third network model that receives audio data as input and generates an audio feature vector as output, irreversibly encode the audio data based on the third network model, and store the audio feature vector generated by the irreversible encoding as training data. The audio feature value may be defined as a fixed-length value having the same data format (e.g., 16-bit integer type or 32-bit floating point type) for each fixed time interval of audio information, and may be mathematically represented as a vector. That is, the background sound separation unit 350 may generate an audio feature vector as feature information corresponding to audio data, and may construct a dedicated network model for this purpose.

[0054] In particular, the third network model can generate a voice feature vector corresponding to the voice data, and the process of generating the voice feature vector via the third network model can correspond to a lossy encoding process in which the voice feature vector cannot be restored to voice data through decoding. Meanwhile, the voice feature vector cannot be used to identify an individual, but can be used to calculate the similarity between voices, and based on the similarity, it can be determined whether the voice data was generated by the same speaker.

[0055] In one embodiment, the background sound separator 350 may construct a fourth network model that receives audio data as input and generates text data as output, and extract text data from the audio data based on the fourth network model. That is, the audio data may be converted into text data through the fourth network model, which is a machine learning model, and the fourth network model may be constructed based on various speech recognition algorithms that convert speech into text. For example, the speech recognition algorithm may include Hidden Markov Models (HMM), Dynamic Time Warping (DTW), a neural network, etc.

[0056] In one embodiment, the background sound separator 350 may detect personal information from text data, convert the personal information in the text data into anonymous information, and store the text data including the anonymous information as training data. That is, in the process of converting the voice data into text data, personally identifiable feature information such as intonation and tone that only appears in voice may be removed, and information closely related to personal information contained in the text may be removed or replaced with anonymous information that does not allow for personal identification, thereby eliminating the possibility of identifying a person through the text. Meanwhile, the background sound separator 350 may utilize a personal information recognition machine learning model to detect personal information from text data.

[0057] In one embodiment, the background sound separator 350 may replace personal information with a higher class name based on a machine learning-based conversion model. The higher class name of personal information included in text may be used as anonymous information that does not identify individuals, but is not limited to this. Various names that provide anonymity may be used. In this case, a conversion model constructed through machine learning may be used. The conversion model may receive specific personal information as input and generate a higher class name of the corresponding personal information as output. For example, information related to a person, such as a person's name, height, age, or weight, may be replaced with a class name such as "person," and information related to a specific address, location, building, or area may be replaced with a class name such as "place."

[0058] In one embodiment, the background sound separator 350 may replace personal information included in text with anonymous information generated based on a voice feature vector. Here, the anonymous information may correspond to random information generated using the voice feature vector. For example, the background sound separator 350 may apply a predetermined hash function to the voice feature vector generated through lossy encoding and generate anonymous information based on the hash value. A hash table may be used to apply the hash function, and a conversion table independent of the hash table may be further used to generate anonymous information for the hash value.

[0059] As another example, the background sound separator 350 may generate a secret key for the encryption process based on the audio feature vector and perform an encryption operation to encrypt personal information using the secret key. In this case, irreversible encryption may be applied depending on the encryption algorithm used in the encryption operation, thereby indirectly achieving anonymity of personal information in text form.

[0060] The training data storage unit 370 may store the video data and background sound data from which sound data has been removed as training data. The sound information from which personal information has been removed can be used as training data, just like the video information, to improve the recognition performance of a machine learning model. Since the sound information does not contain personal information, it can also be used without restrictions in actual services. The training data storage unit 370 may store and manage the training data from which personal information has been removed in the database 150, and may store the training data independently according to data type. One piece of video data stored in the database 150 may be connected to background sound data, a sound feature vector, and text data from which personal information has been removed, and a search operation may be performed within the database 150 using the sound feature vector as a unique key value. That is, anonymous recording data may be searched and extracted based on the similarity between the sound feature vectors.

[0061] The control unit 390 controls the overall operation of the learning database construction device 130 and manages the control flow or data flow between the video receiving unit 310 , the sound extracting unit 330 , the background sound separating unit 350 and the learning data storage unit 370 .

[0062] FIG. 4 is a flowchart illustrating a method for constructing a learning database using the voice personal information protection technology according to the present invention. As shown in FIG. 4, the learning database construction device 130 may receive video data including audio data via the video receiving unit 310 (step S410). In one embodiment, the video receiving unit 310 may receive audio data and corresponding video data independently. That is, the video data may correspond to video without audio data. If video data including audio data is received, the video data may be transferred to the audio extracting unit 330, where a predetermined separation step may be performed.

[0063] The learning database construction device 130 can separate sound data from the video data through the sound extraction unit 330 (step S430). A video may include various sounds, and for example, in the case of a video captured through a black box installed in a vehicle, the sound data may include engine sounds generated while the vehicle is running, sounds of conversations between the driver and passengers inside the vehicle, and environmental sounds outside the vehicle.

[0064] In addition, the sound data extracted from the video data may undergo a predetermined pre-processing step. For example, the sound extractor 330 may perform a pre-processing operation to convert the sound data into a two-dimensional spectrogram, and the range of the spectrogram may be adjusted or a predetermined filter may be applied to the spectrogram for subsequent operation steps.

[0065] In addition, the learning database construction device 130 can extract background sound data from sound data through the background sound separation unit 350 (step S450). A pre-constructed learning network can be used in the sound data separation process, and the learning network model can be pre-constructed based on various machine learning-based network models. In one embodiment, the background sound separation unit 350 can extract independent background sound data by type from the sound data. For example, the background sound separation unit 350 can distinguish between vehicle internal sounds and vehicle external sounds from the sound data and extract the driver and passengers separately (or by user). That is, the background sound separation unit 350 can extract extractable sound data independently by type. In this case, type information related to the sound data can be defined in advance and used in the corresponding process.

[0066] In addition, the learning database construction device 130 may store the video data and background sound data from which the sound data has been removed as learning data via the learning data storage unit 370 (step S470). In one embodiment, the learning data storage unit 370 may group information extracted or generated in relation to one piece of video data into one piece of learning data and store the group in the database 150. For example, one piece of learning data may include background sound data extracted from the video data and sound data, an audio feature vector generated based on the audio data, and text data from which personal information has been removed. In another embodiment, the learning data storage unit 370 may generate an identification code for the corresponding learning data based on the audio feature vector of the specific learning data and store the identification code together.

[0067] FIG. 5 is a diagram illustrating an embodiment of the background sound and speech separation method according to the present invention. As shown in Fig. 5, the learning database construction device 130 may apply preprocessing and a machine learning model to sound data separated from video data or received independently of the video data to separate the sound data into audio data and background sound data. For example, the learning database construction device 130 may preprocess the sound data to generate a preprocessed spectrogram, from which an audio spectrogram and a background sound spectrogram may be extracted. Each extracted spectrogram may be separated into audio data and background sound data through a post-processing operation. In this case, the background sound data is stored as is in the database 150 as learning data, whereas an additional operation to remove personal information may be performed on the audio data.

[0068] 6 and 7 are diagrams illustrating an embodiment of a feature vector calculation method according to the present invention and its non-reversible characteristics. 6, the learning database construction device 130 may encode the voice data 610 through a machine learning model. In this case, the encoded voice feature vector 630 may be unable to be restored to the voice data 610. That is, the voice encoding process for generating the voice feature vector 630 corresponding to the voice data 610 may correspond to a lossy encoding process of the voice information.

[0069] As shown in Fig. 7, the voice feature vector 730 generated through lossy encoding of voice data 710 cannot be used to identify the speaker, but can be used to determine whether two voices are spoken by the same person based on the similarity between the voice feature vectors 730. In Fig. 7, even if the voices are recorded at different times and have different speech content, the voice feature vectors 730 of the same person may have similarity, with only small differences between them. In contrast, the voice feature vectors 730 of different people are less likely to have similarity, with relatively large differences between them.

[0070] That is, the learning database construction device 130 can effectively search for speech data 710 generated by the same person using the speech feature vector 730 among the training data constructed in the database 150. In addition, the learning database construction device 130 can also effectively determine whether the speakers of two speeches that cannot be identified based on the speech feature vector 730 are the same person.

[0071] FIG. 8 is a diagram illustrating an embodiment of the text conversion method according to the present invention. As shown in FIG. 8, the learning database construction device 130 can convert voice data 810 into text data 810 using a machine learning model. The learning database construction device 130 can effectively remove personal information from the text through a personal information recognition machine learning model that recognizes personal information and related information included in the text data 810. In addition, the learning database construction device 130 can remove personal information from the text and replace it with anonymous information that has anonymity. For example, the learning database construction device 130 can replace personal information in the text with a recognized upper class name 870 and use a machine learning model in the process of determining the upper class name 870 corresponding to the personal information. The text data 850 from which personal information has been removed can be stored and managed in the database 150 as learning data associated with video information.

[0072] FIG. 9 is a diagram illustrating the overall concept of the present invention. 9, the learning database construction device 130 can construct a machine learning database by separating sound information into voice information and background sound information, and storing voice feature vectors encoded from the voice information in an irreversible and undecodable manner and text information together with video. The learning database construction device 130 can also classify and extract anonymous recorded data from the constructed machine learning database based on the similarity between voice feature vectors.

[0073] In addition, assuming that black box footage collected during vehicle driving is managed, the learning database construction device 130 can extract driving records that are most similar to the voice feature vector calculated from the voice of a specific person according to a request submitted together with a warrant from a law enforcement agency. Meanwhile, the driving records can also record voice feature vectors of passengers in addition to the driver while the vehicle is driving.

[0074] The learning database construction device 130 according to the present invention can separate sounds contained in a video into background sounds and human voices by using a data processing technology utilizing many parameters, called deep learning or deep neural network, and convert the voices into feature vectors and text, and remove personal information that can identify individuals. In other words, the learning database construction device 130 can execute a method that can protect personal information by automatically removing personal information contained in the voices in the video, while effectively securing learning data about videos that are difficult to collect in machine learning.

[0075] Although the present invention has been described above with reference to preferred embodiments, it will be understood by those skilled in the art that various modifications and variations of the present invention may be made without departing from the spirit and scope of the present invention as set forth in the claims below. [Explanation of symbols]

[0076] 100 Learning database construction system 110 User terminal 130 Learning database construction device 150 databases 210 processors 230 memory 250 User input / output unit 270 Network Input / Output Unit 310 Video receiving unit 330 Sound extraction section 350 Background sound separation section 370 Learning data storage unit 390 Control Unit

Claims

1. A learning database construction system including a user terminal, a learning database construction device, and a database, The learning database construction device includes: receiving video data including audio data; separating the audio data from the video data; extracting voice data and background sound data from the sound data; and storing the video data from which the sound data has been removed, the audio data, and the background sound data as learning data, The step of extracting the background sound data includes: defining a machine learning-based network model including a deep neural network; constructing a first network model that receives the sound data and generates the voice data; constructing a second network model that receives the sound data and generates the background sound data; and separating the voice data and the background sound data from the sound data based on the first and second network models, respectively; The step of extracting the background sound data includes: constructing a fourth network model that receives the speech data and generates text data from which personally identifiable feature information, such as intonation and tone, that only appears in speech, has been removed; extracting the text data from the speech data based on the fourth network model; The step of extracting the background sound data includes: detecting personal information from the text data; converting the personal information from the text data into anonymous information; and storing the text data including the anonymous information as the training data.

2. The step of separating the sound data includes:

2. The method for constructing a learning database using voice personal information protection technology according to claim 1, further comprising the step of applying at least one of a plurality of preprocessing methods to the sound data.

3. The step of extracting the background sound data includes: constructing a third network model that receives the speech data and generates a speech feature vector; performing irreversible encoding of the audio data based on the third network model; The method for constructing a training database using the voice personal information protection technology of claim 1, further comprising a step of storing the voice feature vector generated by the lossy encoding as the training data.

4. The step of converting into anonymous information includes: The learning database construction method using voice personal information protection technology according to claim 1, further comprising a step of replacing the personal information with a higher class name based on a machine learning-based conversion model.

5. a video receiving unit that receives video data including audio data; and a sound extracting unit that separates the audio data from the video data; a background sound separation unit that extracts background sound data from the sound data; a learning data storage unit that stores the video data from which the sound data has been removed and the background sound data as learning data, the background sound separation unit defines a machine learning-based network model including a deep neural network, constructs a first network model that receives the sound data and generates voice data, constructs a second network model that receives the sound data and generates the background sound data, and separates the voice data and the background sound data from the sound data based on the first network model and the second network model; the background sound separation unit constructs a fourth network model that receives the audio data and generates text data, and extracts the text data from the audio data based on the fourth network model; The background sound separation unit detects personal information from the text data, converts the personal information into anonymous information in the text data, and stores the text data including the anonymous information as learning data. In the process of converting the voice data into the text data, personally identifiable characteristic information such as intonation and tone that only appears in voice is removed.

Images