Classification apparatus and method considering word order in document

Abstract

The present invention relates to a classification apparatus and method capable of classifying a document by considering all of meaning, context, and order information of each word present within the document. The classification apparatus comprises: a data collection module that collects document data; a context extraction module that generates a context channel on the basis of the document data, and generates a context feature map vector on the basis of the generated context channel and a CNN model; a sequence extraction module that generates a sequence channel on the basis of the document data, and generates a sequence feature map vector on the basis of the generated sequence channel and a transformer model; and a combination module that generates a classification result for the document data on the basis of the context feature map vector and the sequence feature map vector, wherein the context feature map vector includes having vectorized semantic information of each word included in the document data by reflecting context of the document data.

Description

Classification device and method considering word order in a document

[0001] The present invention relates to a classification device and method that considers the word order of a document.

[0002] Specifically, the present invention relates to a classification device and method capable of classifying a document by taking into account all of the meaning, context, and order information of each word present in the document.

[0003]

[0004] Recently, the development of the internet and the proliferation of smartphones have led to a surge in the use of social networking services (SNS), resulting in the rapid generation of massive amounts of text data. While this data is invaluable for understanding customer needs, its sheer volume makes manual analysis challenging. Furthermore, the advancement of generative artificial intelligence (AI) has also led to the proliferation of misinformation, such as fake news, highlighting the urgent need for technologies capable of automatically identifying and classifying such information.

[0005] Accordingly, active research is being conducted to improve automatic document classification performance by utilizing embeddings learned through artificial neural networks. In this process, word-level embedding vectors are generated and utilized. These vectors reflect the meaning of words, making them useful for document classification.

[0006]

[0007] The problem that the present invention seeks to solve is to provide a classification device and method that can classify a document by considering all of the meaning, context, and order information of each word existing in the document.

[0008] Specifically, the problem to be solved by the present invention is to provide a classification device and method capable of classifying a document by taking into account all of the meaning, context, and order information of each word existing in the document by generating an embedding vector containing the meaning, context, and order information of each word existing in the document using a neural network and classifying the document using the embedding vector.

[0009] The objectives of the present invention are not limited to those mentioned above. Other objectives and advantages of the present invention not mentioned above can be understood through the following description and will be more clearly understood through the embodiments of the present invention. Furthermore, it will be readily apparent that the objectives and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0010]

[0011] According to some embodiments of the present invention, a classification device includes a data collection module that collects document data, a context extraction module that generates a context channel based on the document data and generates a context feature map vector based on the generated context channel and a convolutional neural network (CNN) model, a sequence extraction module that generates a sequence channel based on the document data and generates a sequence feature map vector based on the generated sequence channel and a transformer model, and a combination module that generates a classification result for the document data based on the context feature map vector and the sequence feature map vector, wherein the context feature map vector may include a vectorized version of semantic information of each word included in the document data by reflecting a context of the document data, and the sequence feature map vector may include a vectorized version of semantic information of each word included in the document data by reflecting a sequence of each word in a sentence included in the document data.

[0012] In addition, the context extraction module may include a context channel generation unit that generates the context channel by extracting a plurality of features from the document data and converting each of the extracted features into an embedding vector of a predetermined dimension, and a context feature map vector generation unit that generates the context feature map vector by inputting the context channel into a context feature map vector generation model learned based on the CNN.

[0013] Additionally, the context channel generation unit can extract a predetermined number of features from the document data based on chi-square statistics.

[0014] In addition, the context feature map vector generation model may include a convolution layer that generates a feature map from the context channel by performing a convolution operation using a filter of a predetermined size when the context channel is input, a pooling layer that vectorizes the feature map, and a fully connected layer that performs dimensionality reduction on the vectorized feature map to generate the context feature map vector.

[0015] In addition, the sequence extraction module may include a sequence channel generation unit that generates the sequence channel by performing integer encoding on the document data, and a sequence feature map vector generation unit that generates the sequence feature map vector by inputting the sequence channel into a sequence feature map vector generation model learned based on an encoder of the transformer.

[0016] In addition, the sequence channel generation unit may generate a first integer list by performing a first integer encoding on the document data based on the frequency of words included in the document data, generate a second integer list by performing a second integer encoding on the document data based on the order of words included in the document data, and generate the sequence channel based on the generated first integer list and the second integer list.

[0017] In addition, the sequence channel generation unit can generate a first integer embedding and a second integer embedding by converting the first integer list and the second integer list into embedding vectors, and generate the sequence channel by combining the generated first integer embedding and the second integer embedding.

[0018] In addition, the sequence feature map vector generation model may include a transformer layer that reconstructs the sequence channel to generate a reconstructed channel when the sequence channel is input, a pooling layer that vectorizes the reconstructed channel, and a fully connected layer that performs dimensionality reduction on the vectorized reconstructed channel to generate the sequence feature map vector.

[0019] In addition, the combination module may generate the classification result using a pre-learned vector combination model, and the vector combination model may include a self-attention layer that performs major dimension amplification on the context feature map vector and the sequence feature map vector to generate an amplified context feature map vector and an amplified sequence feature map vector, and a fully connected layer that generates the classification result based on a combination of the context feature map vector, the amplified context feature map vector, the sequence feature map vector, and the amplified sequence feature map vector.

[0020] A classification method according to some embodiments of the present invention includes the steps of collecting document data, generating a context channel based on the document data, generating a context feature map vector based on the generated context channel and a CNN model, generating a sequence channel based on the document data, generating a sequence feature map vector based on the generated sequence channel and a transformer model, and generating a classification result for the document data based on the context feature map vector and the sequence feature map vector, wherein the context feature map vector may include a vectorized version of semantic information of each word included in the document data reflecting a context of the document data, and the sequence feature map vector may include a vectorized version of semantic information of each word included in the document data reflecting an order of each word in a sentence included in the document data.

[0021]

[0022] According to some embodiments of the present invention, a classification device and method generate an embedding vector containing semantic, contextual, and order information of each word within a document using a neural network, and classify the document using the embedding vector. This allows the document to be classified by considering all of the semantic, contextual, and order information of each word within the document, thereby improving the accuracy of document classification. At this time, by considering the order information of each word within the document, hidden information within the document can be amplified, thereby significantly improving the classification accuracy.

[0023] In addition, the classification device and method according to some embodiments of the present invention can be used for sentiment analysis of text data (e.g., posts, comments, etc.) existing in SNS, etc., through high classification accuracy, and new added value can be created when the analysis results are utilized for marketing.

[0024] In addition, the classification device and method according to some embodiments of the present invention have various applications, such as being applicable to the problem of classifying whether a given document was written by a human or generated by a generative AI model (e.g., ChatGPT).

[0025] In addition to the above-described contents, the specific effects of the present invention are described together with the specific matters for carrying out the invention below.

[0026]

[0027] FIG. 1 illustrates a classification system according to some embodiments of the present invention.

[0028] FIG. 2 is a block diagram of a classification device according to some embodiments of the present invention.

[0029] Figure 3 is a flowchart of a classification method according to some embodiments of the present invention.

[0030] FIG. 4 is a conceptual diagram illustrating a classification device and a classification method according to some embodiments of the present invention.

[0031] FIG. 5a is a detailed block diagram of a context extraction module according to some embodiments of the present invention.

[0032] FIG. 5b is a conceptual diagram illustrating the operation of a context channel generation unit according to some embodiments of the present invention.

[0033] FIG. 5c is a conceptual diagram illustrating the operation of a context feature map vector generation model used by a context feature map vector generation unit according to some embodiments of the present invention.

[0034] FIG. 6a is a detailed block diagram of a sequence extraction module according to some embodiments of the present invention.

[0035] FIG. 6b is a conceptual diagram illustrating the operation of a sequence channel generation unit according to some embodiments of the present invention.

[0036] FIG. 6c is a conceptual diagram illustrating the operation of a sequence feature map vector generation model used by a sequence feature map vector generation unit according to some embodiments of the present invention.

[0037] FIG. 7 is a conceptual diagram illustrating the operation of a coupling module according to some embodiments of the present invention.

[0038] FIG. 8 is a diagram illustrating a hardware implementation of a classification device that performs a classification method according to some embodiments of the present invention.

[0039]

[0040] The terms and words used in this specification and claims should not be interpreted based on their general or dictionary meanings. In accordance with the principle that inventors can define the concepts of terms and words to best describe their inventions, they should be interpreted in a way that is consistent with the technical concept of the present invention. Furthermore, the embodiments described in this specification and the configurations depicted in the drawings are merely examples of how the present invention can be realized and do not fully represent the technical concept of the present invention. Therefore, it should be understood that various equivalents, modifications, and applicable examples may exist as of the time of filing.

[0041] The terms first, second, A, B, etc. used in this specification and claims may be used to describe various components, but the components should not be limited by these terms. These terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term "and / or" includes any combination of a plurality of related listed items or any item among a plurality of related listed items.

[0042] The terminology used in this specification and claims is for the purpose of describing specific embodiments only and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "comprise" or "have" in this application do not preclude the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0043] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

[0044] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

[0045] In addition, each configuration, process, procedure or method included in each embodiment of the present invention may be shared within a scope that is not technically inconsistent with each other.

[0046] Hereinafter, with reference to FIGS. 1 to 8, a classification device and method according to some embodiments of the present invention and a system including the same will be described in detail.

[0047]

[0048] FIG. 1 illustrates a classification system according to some embodiments of the present invention.

[0049] Referring to FIG. 1, the classification system (1) may include a user terminal (100), a classification device (200), and a communication network (300).

[0050] The user terminal (100) is a device that transmits document data to the classification device (200). At this time, the document data may be target data for which a classification task is performed by the classification device (200). In other words, the user terminal (100) may transmit document data, which is data to be classified, to the classification device (200).

[0051] Document data may include text data. For example, document data may include multiple texts arranged in the form of paragraphs, paragraphs, sentences, words, etc. However, embodiments of the present invention are not limited thereto, and document data may also include image data, video data, etc.

[0052] However, unlike the above, the user terminal (100) may not transmit document data to the classification device (200), and in this case, the document data may be stored in advance in the classification device (200). However, for the convenience of the following description, it will be described assuming that the document data is provided to the classification device (200) by the user terminal (100).

[0053] Meanwhile, the user terminal (100) may be in the form of various types of electronic devices such as a smart phone, a computer, a laptop PC, a wearable device, a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, and a redundant array of inexpensive disks (RAID) system, but the embodiments of the present invention are not limited thereto.

[0054] The classification device (200) is a device that performs classification work on document data.

[0055]

[0056] In some examples, the classification device (200) may perform a classification task using text data included in the document data, thereby generating a classification result for the document data. At this time, the classification result may include a result of placing the document data into one of the pre-classified categories. In other words, the classification device (200) may generate a classification result regarding which category of the pre-classified categories the document data belongs to, using text data included in the document data.

[0057] At this time, the classification device (200) can perform a classification task based on a neural network. In other words, the classification device (200) can perform a classification task on document data using a pre-learned neural network model based on a neural network.

[0058] Specifically, the neural network model used by the classification device (200) can be trained based on machine learning. To explain in more detail, deep learning, a type of machine learning technology, is a technique that learns by going down to a deep level in multiple stages based on data. In other words, deep learning refers to a set of machine learning algorithms that extract core data from multiple data while increasing the level.

[0059] As examples, neural network models can utilize various well-known deep learning architectures. For example, neural network models can utilize structures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks (DBNs), graph neural networks (GNNs), generative adversarial networks (GANs), transformers, and autoencoders.

[0060] Specifically, a Convolutional Neural Network (CNN) is a model that mimics the human brain, based on the assumption that when recognizing an object, humans extract its basic features, then perform complex computations within the brain to recognize the object based on the results. CNNs can include, but are not limited to, well-known structures such as LeNet, AlexNet, VGGNet, GoogleNet, and ResNet.

[0061] RNN (Recurrent Neural Network) is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes over time. It can construct an artificial neural network structure by stacking layers at each moment.

[0062] A DBN (Deep Belief Network) is a deep learning structure constructed by stacking multiple layers of Restricted Boltzman Machines (RBMs), a deep learning technique. By repeatedly training RBMs (Restricted Boltzman Machines), a certain number of layers can be created, creating a DBN (Deep Belief Network) with that number of layers.

[0063] GNN (Graphic Neural Network, hereinafter referred to as GNN) represents an artificial neural network structure implemented in a way that derives similarity and feature points between modeling data by using modeling data modeled based on data mapped between specific parameters.

[0064] A Generative Adversarial Network (GAN) is an artificial neural network structure that uses a generative neural network and a discriminative neural network to generate new data in a similar form to the input data. GANs may include the well-known DCGAN (Deep Convolutional GAN), CGAN (Conditional GAN), WGAN (Wasserstein GAN), StyleGAN (Style-Based GAN), CycleGAN, etc., but embodiments of the present invention are not limited thereto.

[0065] Transformer is an artificial neural network with an attention-based encoder-decoder structure that can understand the overall meaning between input and output sequences. Transformer uses the attention mechanism to ensure that all elements of the input sequence influence the output sequence, allowing both the encoder and decoder to consider the entire sequence. Transformer can use natural language, time-series data, and even patched images as input.

[0066] An autoencoder is a deep learning architecture that extracts and reconstructs data features. Typically, an autoencoder comprises an encoder, which compresses input values, and a decoder, which restores the compressed data. The encoder transforms the input values ​​into a low-dimensional latent representation, and the decoder reconstructs the latent representation to the same dimensionality as the input values. Each encoder and decoder can be configured as a multilayer perceptron (MLP). When training an autoencoder, input data is input, and weights and biases are trained to minimize the difference between the output and the input values. This trained autoencoder can effectively extract input data features and reconstruct noisy input data. Autoencoders are primarily used in fields such as data compression, dimensionality reduction, noise removal, and data generation, and can also be utilized in areas such as image recognition, natural language processing, and speech recognition.

[0067] Meanwhile, artificial neural network learning in neural network models can be achieved by adjusting the weights of connections between nodes (and, if necessary, adjusting bias values) to produce the desired output for a given input. Furthermore, artificial neural networks can continuously update their weight values ​​through learning. Furthermore, methods such as backpropagation can be used for artificial neural network learning.

[0068] At this time, machine learning methods for artificial neural networks can include unsupervised learning, semi-supervised learning, and supervised learning. Furthermore, the neural network model can be controlled to automatically update its structure to output post-learning analysis data, depending on settings.

[0069] The specific process by which the classification device (200) performs classification work on document data based on this neural network model will be described later.

[0070] The communication network (300) refers to a communication means that performs data exchange between a user terminal (100) and a classification device (200).

[0071] At this time, the communication network (300) may include a network based on wired Internet technology, wireless Internet technology, and short-range communication technology. The wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN). The wireless Internet technology may include, for example, at least one of wireless LAN (WLAN), Digital Living Network Alliance (DMNA), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), and 5G NR (New Radio) technology. However, the present embodiment is not limited thereto. Short-range communication technologies may include, for example, at least one of Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio).However, this embodiment is not limited thereto.

[0072] Hereinafter, the operation of the classification device (200) according to some embodiments of the present invention will be described in more detail with reference to FIGS. 2 to 7.

[0073]

[0074] FIG. 2 is a block diagram of a classification device according to some embodiments of the present invention. FIG. 3 is a flowchart of a classification method according to some embodiments of the present invention. FIG. 4 is a conceptual diagram illustrating a classification device and a classification method according to some embodiments of the present invention. Each step (S100 to S400) of FIG. 3 can be performed by the classification device (200) of FIGS. 1 and 2.

[0075] Referring to FIGS. 1 to 4, a classification device (200) according to some embodiments of the present invention is a device that generates a classification result (hereinafter referred to as “CR”) for document data (hereinafter referred to as “DOC”), and may include a data collection module (210), a context extraction module (220), a sequence extraction module (230), and a combination module (240).

[0076] If we describe the operation of each component included in the classification device (200) in more detail, first, the data collection module (210) can collect document data (DOC) (S100).

[0077] Document data (DOC) may be target data for which a classification task is performed by a classification device (200). In some examples, document data (DOC) may include text data. For example, document data (DOC) may include multiple texts arranged in the form of paragraphs, paragraphs, sentences, words, etc. However, embodiments of the present invention are not limited thereto, and document data (DOC) may also include image data, video data, etc.

[0078] Meanwhile, the number of document data (DOC) may be plural (e.g., t). In other words, the document data (DOC) may include first document data (DOC_1) to t-th document data (DOC_t). In this case, the number (t) of document data (DOC) may be freely set and changed.

[0079] Next, the context extraction module (220) can generate a context feature map vector (hereinafter referred to as “CV”) based on document data (DOC) (S200), and the sequence extraction module (230) can generate a sequence feature map vector (hereinafter referred to as “SV”) based on document data (DOC) (S300).

[0080] For example, the context extraction module (220) may generate a context channel (hereinafter referred to as "CC") based on document data (DOC), and generate a context feature map vector (CV) based on the generated context channel (CC). At this time, when the context channel (CC) is generated, the context extraction module (220) may generate a context feature map vector (CV) based on the generated context channel (CC) and a convolutional neural network (CNN), which is a type of neural network described above.

[0081] At this time, the context feature map vector (CV) may include vectorized semantic information of each word included in the document data (DOC) by reflecting the context of the document data (DOC). In other words, the context extraction module (220) may vectorize semantic information of each word included in the document data (DOC) by considering the context in the document data (DOC), and generate the vectorized result as a context feature map vector (CV).

[0082] As another example, the sequence extraction module (230) may generate a sequence channel (hereinafter referred to as “SC”) based on document data (DOC), and generate a sequence feature map vector (SV) based on the generated sequence channel (SC). At this time, when the sequence channel (SC) is generated, the sequence extraction module (230) may generate a sequence feature map vector (SV) based on the generated sequence channel (SC) and a transformer, which is a type of neural network described above.

[0083] At this time, the sequence feature map vector (SV) may include the semantic information of each word included in the document data (DOC) vectorized by reflecting the order (Sequence) of each word in the document data (DOC). In other words, the sequence extraction module (230) may vectorize the semantic information of each word included in the document data (DOC) by considering the order of each word included in a sentence in the document data (DOC), and generate the vectorized result as a sequence feature map vector (SV).

[0084] Hereinafter, the process of generating a context feature map vector (CV) by the context extraction module (220) will be described in detail with reference to FIGS. 5a to 5c, and the process of generating a sequence feature map vector (SV) by the sequence extraction module (230) will be described in detail with reference to FIGS. 6a to 6c.

[0085]

[0086] FIG. 5A is a detailed block diagram of a context extraction module according to some embodiments of the present invention. FIG. 5B is a conceptual diagram illustrating the operation of a context channel generation unit according to some embodiments of the present invention. FIG. 5C is a conceptual diagram illustrating the operation of a context feature map vector generation model used by a context feature map vector generation unit according to some embodiments of the present invention.

[0087] Referring to FIGS. 4 to 5c, the context extraction module (220) is a module that generates a context feature map vector (CV) based on document data (DOC), and may include a context channel generation unit (221) and a context feature map vector generation unit (222).

[0088] The context channel creation unit (221) can create a context channel (CC) based on document data (DOC).

[0089] As some examples, the context channel generation unit (221) can generate a context channel (CC) by extracting a plurality of features (hereinafter referred to as "FTs") from document data (DOC) and converting each extracted feature into an embedding vector of a predetermined dimensionality. At this time, the context channel generation unit (221) can generate the context channel (CC) using a predefined context channel generation model (hereinafter referred to as "CCGM"). The context channel generation model (CCGM) may be a model based on an ELMo (Embeddings from Language Models) model, but embodiments of the present invention are not limited thereto.

[0090] As an example of FIG. 5b, first, the context channel generation model (CCGM) can extract a predetermined number (e.g., k) of features (FT) from document data (DOC) including the first document data (DOC_1) to the t-th document data (DOC_t). At this time, the features (FT) may be words included in the document data (DOC), but the embodiment of the present invention is not limited thereto. For example, the context channel generation model (CCGM) can use chi-square (chi-2) statistics when extracting features (FT) based on the document data (DOC). In other words, the context channel generation model (CCGM) can select a predetermined number (e.g., k) of features (FT) in order of increasing influence from the entire document data (DOC) including the first document data (DOC_1) to the t-th document data (DOC_t) based on the chi-square statistics.

[0091] Next, the context channel generation model (CCGM) can determine the number of features (FT) included in each of the first document data (DOC_1) to the t-th document data (DOC_t). In other words, the context channel generation model (CCGM) can determine how many features (FT) each of the first document data (DOC_1) to the t-th document data (DOC_t) includes.

[0092] Next, the context channel generation model (CCGM) can determine the number of features (FT) included in the document data with the maximum number of features (FT). In other words, the context channel generation model (CCGM) can search for document data (e.g., the first document data (DOC_1)) with the maximum number of features (FT) included, and determine the number of features (FT) included in the document data (e.g., the first document data (DOC_1)). Hereinafter, the number of features (FT) included in the document data with the maximum number of features (FT) included will be explained assuming that m is the case.

[0093] Next, the context channel generation model (CCGM) can generate a context channel (CC) in the form of matrix data having a predetermined size (m, n) based on the converted embedding vector (EV) by converting the features (FT) included in each of the first to t-th document data (DOC_1) to the t-th document data (DOC_t), respectively, into an embedding vector (EV) having a predetermined dimensionality (n, e.g., 256). In other words, the context channel generation model (CCGM) can generate the first to t-th context channels (CC_1) to the t-th document data (DOC_t), respectively, through a process of converting the embedding vector (EV) for the features (FT) included in each of the document data (DOC_1 to DOC_t). At this time, the size of the context channel (CC) is (m, n), where m is the number of features (FT) included in the document data with the maximum number of features (FT) included as described above, and n may be the dimensionality (e.g., 256) in the embedding conversion process for each feature (FT). However, the embodiments of the present invention are not limited thereto.

[0094] When a context channel (CC) is generated by the context channel generation unit (221) through this process, the context feature map vector generation unit (222) can generate a context feature map vector (CV) by inputting the context channel (CC) into a context feature map vector generation model (hereinafter referred to as “CVGM”).

[0095] At this time, the context feature map vector generation model (CVGM) may be a model learned based on CNN, which is a type of neural network. For example, the context feature map vector generation model (CVGM) may include a convolution layer (hereinafter referred to as "L_C") that generates a feature map (hereinafter referred to as "FM") from a context channel (CC) by performing a convolution operation using a filter of a predetermined size when a context channel (CC) is input, a pooling layer (hereinafter referred to as "L_P") that vectorizes the feature map (FM) to generate a vectorized feature map (hereinafter referred to as "V_FM"), and a fully-connected layer (hereinafter referred to as "L_FC") that generates a context feature map vector (CV) by performing dimensionality reduction on the vectorized feature map (V_FM).

[0096] Taking Fig. 5c as an example, first, the convolution layer (L_C) can receive a context channel (CC) in the form of matrix data having a size of (m, n). At this time, for convenience of explanation, Fig. 5c illustrates a case where m is 46 and n is 256. Next, the convolution layer (L_C) can generate a feature map (FM) from the context channel (CC) by performing a convolution operation using filters of a predetermined size (e.g., length 256, height 1) and a predetermined number (e.g., 1024). At this time, the number of generated feature maps (FM) may be the same as the number of filters (e.g., 1024). Next, the pooling layer (L_P) can vectorize the feature map (FM) to generate a vectorized feature map (V_FM). At this time, the pooling layer (L_P) can generate a vectorized feature map (V_FM) with a predetermined dimensionality (e.g., 1024) using global max pooling or global average pooling. Subsequently, the fully connected layer (L_FC) can perform dimensionality reduction (e.g., from 1024 to 256) on the vectorized feature map (V_FM) to generate a context feature map vector (CV).

[0097]

[0098] FIG. 6a is a detailed block diagram of a sequence extraction module according to some embodiments of the present invention. FIG. 6b is a conceptual diagram illustrating the operation of a sequence channel generation unit according to some embodiments of the present invention. FIG. 6c is a conceptual diagram illustrating the operation of a sequence feature map vector generation model used by a sequence feature map vector generation unit according to some embodiments of the present invention.

[0099] Referring to FIG. 4 and FIG. 6a to FIG. 6c, the sequence extraction module (230) is a module that generates a sequence feature map vector (SV) based on document data (DOC), and may include a sequence channel generation unit (231) and a sequence feature map vector generation unit (232).

[0100] The sequence channel generation unit (231) can generate a sequence channel (SC) based on document data (DOC). As some examples, the sequence channel generation unit (231) can generate a sequence channel (SC) by performing integer encoding on the document data (DOC).

[0101] For example, the sequence channel generation unit (231) may generate a first integer list (1st Integer List, hereinafter referred to as “IL_1”) by performing a first integer encoding based on the frequency of words included in the document data (DOC), generate a second integer list (2nd Integer List, hereinafter referred to as “IL_2”) by performing a second integer encoding based on the order of words included in the document data (DOC), and generate a sequence channel (SC) based on the generated first integer list (IL_1) and second integer list (IL_2).

[0102] Taking FIG. 6b as an example, first, the sequence channel generation unit (231) can determine the maximum length (m', e.g., 1000) of the document data (DOC). Next, the sequence channel generation unit (231) can perform a first integer encoding based on the frequency of words for each of the first document data (DOC_1) to the t-th document data (DOC_t) included in the document data (DOC) to generate a first integer list (IL_1), and perform a second integer encoding based on the order of the words to generate a second integer list (IL_2). At this time, padding can be performed according to the maximum length (e.g., 1000) of the document data (DOC) that is preset during the integer encoding process. Next, the sequence channel generation unit (231) may convert the first integer list (IL_1) and the second integer list (IL_2) generated for each of the first document data (DOC_1) to the t-th document data (DOC_t) into embedding vectors to generate a first integer embedding (hereinafter referred to as “IE_1”) and a second integer embedding (hereinafter referred to as “IE_2”) of a predetermined dimension (n’, e.g., 640). At this time, the sequence channel generation unit (231) may convert the first integer list (IL_1) and the second integer list (IL_2) into the first integer embedding (IE_1) and the second integer embedding (IE_2), respectively, using a predefined vector conversion algorithm. Next, the sequence channel generation unit (231) can generate a combined embedding (hereinafter referred to as "CE") by combining the first integer embedding (IE_1) and the second integer embedding (IE_2). For example, the sequence channel generation unit (231) can generate a combined embedding (CE) by combining the first integer embedding (IE_1) and the second integer embedding (IE_2).For example, the sequence channel generation unit (231) can add the first integer embedding (IE_1) and the second integer embedding (IE_2) and determine the added result as a combined embedding (CE). In the case of Fig. 6b, it is assumed that the maximum length (m') of the document data (DOC) is set to 1000, so the sequence channel generation unit (231) can create 1000 combined embeddings (CE) for each of the first document data (DOC_1) to the t-th document data (DOC_t).

[0103] Next, the sequence channel generation unit (231) can generate a sequence channel (SC) in the form of matrix data having a predetermined size (m', n') based on the combined embedding (CE) for each of the first document data (DOC_1) to the t-th document data (DOC_t) included in the document data (DOC). In other words, the sequence channel generation unit (231) can generate the first sequence channel (SC_1) to the t-th sequence channel (SC_t) in the form of matrix data having a predetermined size (m', n') through a combination of the combined embedding (CE) included in each of the document data (DOC_1 to DOC_t) for each of the first document data (DOC_1) to the t-th document data (DOC_t). At this time, the size of the sequence channel (SC) is (m', n'), where m' is a value set to the maximum length of the document data (DOC) as described above, and n' may be the dimensionality (e.g., 640) in the embedding conversion process for each integer list (IL_1, IL_2). However, the embodiments of the present invention are not limited thereto.

[0104] When a sequence channel (SC) is generated by the sequence channel generation unit (231) through this process, the sequence feature map vector generation unit (232) can generate a sequence feature map vector (SV) by inputting the sequence channel (SC) into a sequence feature map vector generation model (hereinafter referred to as “SVGM”).

[0105] At this time, the sequence feature map vector generation model (SVGM) may be a model learned based on an encoder of a transformer, which is a type of neural network. For example, the sequence feature map vector generation model (SVGM) may include a transformer layer (hereinafter referred to as "L_T") that reconstructs a sequence channel (SC) when a sequence channel (SC) is input to generate a reconstructed channel (hereinafter referred to as "RC"), a pooling layer (L_P) that vectorizes the reconstructed channel (RC) to generate a vectorized reconstructed channel (V_RC), and a fully connected layer (L_FC) that performs dimensionality reduction on the vectorized reconstructed channel (V_RC) to generate a sequence feature map vector (SV). At this time, the transformer layer (L_T) may include a self-attention layer (Self-Attention Layer in Transformer Layer, hereinafter referred to as “L_T_SA”), a fully-connected layer (Fully-Connected Layer in Transformer Layer, hereinafter referred to as “L_T_FC”), and a post-processing layer (Post-Processing Layer in Transformer Layer, hereinafter referred to as “L_T_PP”).

[0106] Taking Fig. 6c as an example, first, the transformer layer (L_T) can receive a sequence channel (SC) in the form of matrix data having a size of (m', n'). At this time, for convenience of explanation, Fig. 6c illustrates a case where m' is 1000 and n' is 640. Next, the sequence channel (SC) input to the transformer layer (L_T) can be reconstructed through a self-attention layer (L_T_SA), a fully connected layer (L_T_FC), and a post-processing layer (L_T_PP) and converted into a reconstructed channel (RC). In more detail, the self-attention layer (L_T_SA) outputs the highlighted and / or amplified main part in the sequence channel (SC), the fully connected layer (L_T_FC) reduces the dimension of the data that has passed through the self-attention layer (L_T_SA) (e.g., from 640 to 64) and then restores it (e.g., restores 64 to 640), and the post-processing layer (L_T_PP) can apply residual connection and layer normalization to the sequence channel (SC), the data that has passed through the self-attention layer (L_T_SA), and the data that has passed through the fully connected layer (L_T_FC). When the reconstructed channel (RC) is output from the transformer layer (L_T), the pooling layer (L_P) can then vectorize the reconstructed channel (RC) to generate a vectorized reconstructed channel (V_RC). At this time, the pooling layer (L_P) can generate a vectorized reconstructed channel (V_RC) with a predetermined dimensionality (e.g., 640) using global max pooling or global average pooling. Subsequently, the fully connected layer (L_FC) can perform dimensionality reduction (e.g., from 640 to 256) on the vectorized reconstructed channel (V_RC) to generate a sequence feature map vector (SV).

[0107]

[0108] Referring again to FIGS. 1 to 4, the combination module (240) can then generate a classification result (CR) based on the context feature map vector (CV) and the sequence feature map vector (SV) (S400).

[0109] At this time, the classification result (CR) may include the result of arranging the first document data (DOC_1) to the tth document data (DOC_t) included in the document data (DOC) into any one of the pre-defined categories. In other words, the classification result (CR) may include the result of determining whether each document data (DOC_1 to DOC_t) belongs to any of the pre-defined categories.

[0110] Hereinafter, the operation of the coupling module (240) will be described in more detail with reference to FIG. 7.

[0111]

[0112] FIG. 7 is a conceptual diagram illustrating the operation of a coupling module according to some embodiments of the present invention.

[0113] Referring to FIGS. 4 and 7, the combining module (240) can generate a classification result (CR) by combining a context feature map vector (CV) and a sequence feature map vector (SV) using a pre-learned vector combining model.

[0114] At this time, the vector combination model may include a self-attention layer (hereinafter referred to as "L_SA") that performs major dimension amplification on each of a context feature map vector (CV) and a sequence feature map vector (SV) to generate an amplified context feature map vector (hereinafter referred to as "CV_AMP") and an amplified sequence feature map vector (hereinafter referred to as "SV_AMP"), and a fully connected layer (L_FC) that generates a classification result (CR) based on the combination of the context feature map vector (CV), the amplified context feature map vector (CV_AMP), the sequence feature map vector (SV), and the amplified sequence feature map vector (SV_AMP).

[0115] As an example of Fig. 7, first, the self-attention layer (L_SA) can receive a context feature map vector (CV) and a sequence feature map vector (SV) having a predetermined dimensionality (e.g., 256). Next, the self-attention layer (L_SA) can perform principal dimension amplification on each of the context feature map vector (CV) and the sequence feature map vector (SV) to generate an amplified context feature map vector (CV_AMP) and an amplified sequence feature map vector (SV_AMP). At this time, the dimensionality of the amplified context feature map vector (CV_AMP) and the amplified sequence feature map vector (SV_AMP) can have the same value (e.g., 256) as that of the context feature map vector (CV) and the sequence feature map vector (SV). Next, the vector combination model can generate a combination vector (hereinafter referred to as "V_COM") by combining the context feature map vector (CV), the amplified context feature map vector (CV_AMP), the sequence feature map vector (SV), and the amplified sequence feature map vector (SV_AMP). Next, the fully connected layer (L_FC) can generate a classification result (CR) by applying dimensionality reduction and a softmax function to the combination vector (V_COM).

[0116]

[0117] FIG. 8 is a diagram illustrating a hardware implementation of a classification device that performs a classification method according to some embodiments of the present invention.

[0118] Referring to FIGS. 1, 2, and 8, a classification device (200) according to some embodiments of the present invention may be implemented as an electronic device (1000). The electronic device (1000) may include a controller (1010), an input / output device (1020), a memory device (1030), an interface (1040), and a bus (1050). The controller (1010), the input / output device (1020), the memory device (1030), and / or the interface (1040) may be coupled to each other via a bus (1050). In this case, the bus (1050) corresponds to a path through which data is transferred.

[0119] Specifically, the controller (1010) may include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a microprocessor, a digital signal processor, a microcontroller, an application processor (AP), and logic elements capable of performing functions similar thereto.

[0120] The input / output device (1020) may include at least one of a keypad, a keyboard, a touchscreen, and a display device.

[0121] The memory device (1030) can store data and / or programs, etc.

[0122] The interface (1040) may perform a function of transmitting data to or receiving data from a communication network. The interface (1040) may be wired or wireless. For example, the interface (1040) may include an antenna or a wired / wireless transceiver. Although not illustrated, the memory device (1030) may further include high-speed DRAM and / or SRAM as an operating memory for improving the operation of the controller (1010). The memory device (1030) may store programs or applications therein.

[0123] The classification device (200) according to embodiments of the present invention may be a system formed by connecting multiple electronic devices (1000) to each other via a network. In this case, each module or combination of modules may be implemented as an electronic device (1000). However, the present embodiment is not limited thereto.

[0124] Additionally, the classification device (200) may be implemented as at least one of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a redundant array of inexpensive disks (RAID) system, and an electronic document management (EDMS) system, but the present embodiment is not limited thereto.

[0125] Additionally, the classification device (200) can transmit data to the user terminal (100) via a network. The network may include a network based on wired Internet technology, wireless Internet technology, and short-range communication technology. The wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

[0126] The wireless Internet technology may include, for example, at least one of Wireless LAN (WLAN), Digital Living Network Alliance (DMNA), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), and 5G NR (New Radio) technologies. However, the present embodiment is not limited thereto.

[0127] Short-range communication technologies may include, for example, at least one of Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0128] A classification device (200) communicating through a network may comply with technical standards and standard communication methods for mobile communications. For example, the standard communication method may include at least one of GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0129] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art to which the present embodiment pertains may make various modifications and variations without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

Claims

1. Data collection module that collects document data; A context extraction module that generates a context channel based on the above document data and generates a context feature map vector based on the generated context channel and a CNN (Convolutional Neural Network) model; A sequence extraction module that generates a sequence channel based on the above document data and generates a sequence feature map vector based on the generated sequence channel and a transformer model; and Including a combination module that generates a classification result for the document data based on the context feature map vector and the sequence feature map vector, The above context feature map vector includes vectorized semantic information of each word included in the document data, reflecting the context of the document data, The above sequence feature map vector includes the semantic information of each word included in the document data, which is vectorized to reflect the order (Sequence) of each word in the sentence included in the document data. Classification device.

2. In paragraph 1, The above context extraction module, A context channel generation unit that extracts multiple features from the document data and generates the context channel by converting each extracted feature into an embedding vector of a predetermined dimension; A context feature map vector generation unit that generates the context feature map vector by inputting the context channel into a context feature map vector generation model learned based on the CNN. Classification device.

3. In paragraph 2, The above context channel creation unit, Extracting a predetermined number of features from the document data based on the chi-square statistic. Classification device.

4. In paragraph 2, The above context feature map vector generation model is, When the above context channel is input, a convolution layer that generates a feature map from the context channel by performing a convolution operation using a filter of a predetermined size, A pooling layer that vectorizes the above feature map, A fully connected layer that performs dimension reduction on the vectorized feature map to generate the context feature map vector. Classification device.

5. In paragraph 1, The above sequence extraction module, A sequence channel generation unit that generates the sequence channel by performing integer encoding on the above document data, and A sequence feature map vector generation unit that generates the sequence feature map vector by inputting the sequence channel into a sequence feature map vector generation model learned based on the encoder of the transformer. Classification device.

6. In paragraph 5, The above sequence channel generation unit, For the above document data, a first integer encoding is performed based on the frequency of words included in the document data to generate a first integer list, For the above document data, a second integer encoding is performed based on the order of words included in the document data to generate a second integer list, Generating the sequence channel based on the generated first integer list and the second integer list Classification device.

7. In paragraph 6, The above sequence channel generation unit, Converting the first integer list and the second integer list into embedding vectors to generate a first integer embedding and a second integer embedding, Generating the sequence channel by combining the generated first integer embedding and the generated second integer embedding. Classification device.

8. In paragraph 5, The above sequence feature map vector generation model is, When the above sequence channel is input, a transformer layer that reconstructs the sequence channel to generate a reconstructed channel, A pooling layer that vectorizes the above reconstruction channel, A fully connected layer comprising a dimensionality reduction layer for the vectorized reconstructed channel to generate the sequence feature map vector. Classification device.

9. In paragraph 1, The above combination module generates the classification result using a pre-learned vector combination model, The above vector combination model is, A self-attention layer that performs major dimension amplification on the context feature map vector and the sequence feature map vector to generate an amplified context feature map vector and an amplified sequence feature map vector, A fully connected layer comprising a combination of the context feature map vector, the amplified context feature map vector, the sequence feature map vector, and the amplified sequence feature map vector to generate the classification result. Classification device.

10. Step of collecting document data; A step of generating a context channel based on the above document data, and generating a context feature map vector based on the generated context channel and a CNN model; A step of generating a sequence channel based on the above document data, and generating a sequence feature map vector based on the generated sequence channel and a transformer model; and A step of generating a classification result for the document data based on the context feature map vector and the sequence feature map vector, The above context feature map vector includes vectorized semantic information of each word included in the document data, reflecting the context of the document data, The above sequence feature map vector includes the semantic information of each word included in the document data, which is vectorized to reflect the order of each word in the sentence included in the document data. Classification method.

Images