Feature extraction device and program

Abstract

To provide a feature extraction device capable of extracting features of a sentence by considering a label structure.SOLUTION: An input feature generation part outputs a token embedding corresponding to a token line on the basis of the token line corresponding to an input sentence described in natural language. A label structure generation part outputs a label structure comprising a feature amount of each label that belongs to a label set on the basis of the label set expressed in natural language. A substitution part receives the token embedding and the label structure, substitutes a vector originating from the label structure for a feature amount of each token in the token embedding with a key and a value in an attention mechanism as the label structure with a query in the attention mechanism as the token embedding, and outputs a label-substituted token embedding.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a feature extraction device and a program. [Background technology]

[0002] There is a task of classifying text by attaching labels (tags). In general, multiple labels are attached to a single text for classification. For example, in journalism, multiple labels are attached to text in a news article. A news article is assigned as many labels as there are aspects of the content it expresses. The number of labels attached to a single news article is variable (one or more). This allows readers to easily find articles in areas of interest when news articles are posted on websites, etc. A label represents a word or short phrase used for classification.

[0003] Traditionally, labeling has been done manually. For example, at news organizations that publish news daily, labelers read the text of news articles written by reporters, understand the content, and assign multiple labels appropriate to the text. The number of label types is around 100 or even several hundred. However, manual labeling is expensive and time-consuming, so automation has been desired.

[0004] For example, Patent Document 1, Non-Patent Document 1, Non-Patent Document 2, and Non-Patent Document 3 each describe a technique for assigning labels to sentences. These conventional techniques use a method for determining the features of words (tokens) that make up an input sentence, and learning the relationship between the features represented by a string of words and the label (correct answer) assigned to the sentence. In conventional multi-label classification models using neural networks, neural networks with a recurrent mechanism (RNN, Recurrent Neural Network) are often adopted. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-053730 [Non-patent literature]

[0006] [Non-Patent Document 1] Iwao Fujino, Satoru Ikeda, and Sora Yamamoto, "Multi-label classification of text documents using topic models and least mean squares estimation," Bulletin of the Faculty of Information and Communications Sciences, Tokai University, Vol. 10, No. 1, 2017, pp. 15-24, [online], [downloaded July 20, 2020], Internet <URL:https: / / www.u-tokai.ac.jp / academics / undergraduate / information_and_telecommu / kiyou / index / pdf / vol_10_001 / PP15-24.pdf> [Non-patent document 2] Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, Houfeng Wang, “Sequence Generation Model for Multi-label Classification”, Proceedings of the 27th International Conference on Computational Linguistics, pages 3915-3926, Santa Fe, New Mexico, USA, August 20-26, 2018, [online], [downloaded July 20, 2020], Internet<URL:https: / / www.aclweb.org / anthology / C18-1330.pdf> [Non-patent document 3] Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin, “Rethinking complex neural network architectures for document classification”, Proceedings of NAACL-HLT 2019, pages 4046-4051, Minneapolis, Minnesota, June 2 - June 7, 2019, Association for Computational Linguistics, [online], [downloaded July 20, 2020], Internet<URL:https: / / www.aclweb.org / anthology / N19-1408.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] In the conventional technology described above, the model determines the label to be assigned based only on information extracted from the input sentence. However, in multi-label classification problems that handle many labels, there is a problem that labels with similar meanings exist. In the conventional technology, the only information that the model handles is information about the input sentence and information about the correct label. In other words, in the conventional technology, the model does not handle the relationship between labels, i.e., the information contained in the set of labels (label structure).

[0008] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a feature extraction device and a program that can extract features of a sentence while taking into consideration the above-mentioned label structure. [Means for solving the problem]

[0009] [1] In order to solve the above problem, a feature extraction device according to one aspect of the present invention includes an input feature generation unit that outputs a token embedding corresponding to an input sentence written in a natural language, based on a token sequence corresponding to the token sequence; a label structure generation unit that outputs a label structure consisting of features of each label belonging to a label set, based on a label set expressed in a natural language; and a replacement unit that receives the token embedding and the label structure, and replaces the features of each token in the token embedding with a vector derived from the label structure, using a query in an attention mechanism as the token embedding and a key and value in the attention mechanism as the label structure, and outputs a label-replaced token embedding.

[0010] [2] In another aspect of the present invention, in the feature extraction device described above, the input feature generation unit further outputs features of CLS tokens added to the beginning of the token sequence based on the token sequence, and the device further includes a bidirectional gated regression processing unit that performs bidirectional gated regression processing based on the label-replaced token embedding to generate input sentence features that reflect the label structure, and a feature extraction unit that outputs a score for each label belonging to the label set based on the features of the CLS tokens and the input sentence features that reflect the label structure.

[0011] [3] In addition, according to one aspect of the present invention, the feature extraction device further comprises an output unit that outputs a set of labels whose scores are equal to or greater than a predetermined threshold.

[0012] [4] Furthermore, one aspect of the present invention is the feature extraction device described above, further comprising: a control unit that controls whether the feature extraction device operates in an estimation mode or a learning mode; and a loss calculation unit that calculates a loss between the score of each label output by the feature extraction unit and a correct score held by learning data that corresponds to the input sentence. When the feature extraction device operates in the estimation mode, the control unit controls the output unit to output a set of labels whose scores are equal to or greater than a predetermined threshold; and when the feature extraction device operates in the learning mode, the control unit performs error backpropagation based on the loss calculated by the loss calculation unit, and controls the feature extraction unit, bidirectional gated regression processing unit, replacement unit, and input feature generation unit to adjust internal parameters.

[0013] [5] Another aspect of the present invention is a program that causes a computer to function as a feature extraction device that includes: an input feature generation unit that, based on a token sequence corresponding to an input sentence written in a natural language, outputs a token embedding corresponding to the token sequence; a label structure generation unit that, based on a label set expressed in a natural language, outputs a label structure consisting of features of each label belonging to the label set; and a replacement unit that receives the token embedding and the label structure, and replaces the features of each token in the token embedding with a vector derived from the label structure, using a query in an attention mechanism as the token embedding and a key and value in the attention mechanism as the label structure, and outputs a label-replaced token embedding. [Effects of the Invention]

[0014] According to the present invention, it is possible to extract features of an input sentence taking into account the label structure, thereby improving the accuracy of determining the label to be assigned to the input sentence. [Brief explanation of the drawings]

[0015] [Figure 1]1 is a block diagram showing a schematic functional configuration of a feature extraction device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a schematic functional configuration of a feature extraction unit according to the embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of the internal configuration (configuration using a computer) of the feature extraction device according to the embodiment. [Figure 4] FIG. 10 is a schematic diagram showing the feature amount (CLS embedding) of a CLS token in 1 row and k columns and the feature amount (token embedding) of an m row and k columns calculated by the BERT unit according to the embodiment. [Figure 5] FIG. 5 is a schematic diagram showing only the feature amount (CLS embedding) of the CLS token in row 1, column k of the data shown in FIG. 4. [Figure 6] 10 is a schematic diagram showing data of a label structure with n rows and k columns calculated by a self-attention unit according to the embodiment. FIG. [Figure 7] This is a schematic diagram showing label-substituted token embedding data with m rows and k columns calculated by the source-target attention unit according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Next, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is a block diagram showing a schematic functional configuration of a feature extraction device according to this embodiment. As shown in the figure, the feature extraction device 1 includes a BERT unit 11, a self-attention unit 12, a source-target attention unit 13, a bidirectionally gated regression unit 14, a feature extraction unit 15, an output unit 16, an input sentence data storage unit 31, a label set data storage unit 32, a label structure data storage unit 33, a token embedding storage unit 34, a label-replaced token embedding storage unit 35, a CLS token feature storage unit 36, a loss calculation unit 21, and a control unit 9. The feature extraction device 1 having the above configuration can be realized, for example, using a computer and a program. Each of the above functional units may have a storage unit as needed. The storage unit may be, for example, a program variable or a memory allocated by program execution. Non-volatile storage units such as a magnetic hard disk drive or a solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0018] The feature extraction device 1 is a device that extracts features of an input sentence, which are necessary to accomplish the task of multi-label classification. The feature extraction device 1 is characterized in that it calculates a label structure based on the similarity between labels. The model realized by the feature extraction device 1 is called an SRN (Source Replacer Network).

[0019] The feature extraction device 1 of this embodiment uses attention mechanisms (self-attention and source-target attention). The attention mechanisms themselves are existing technologies. The attention mechanisms are also described in the following reference documents. References: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, “Attention Is All You Need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach,CA,USA,arXiv:1706.03762v5 [cs.CL], 6 Dec 2017,https: / / arxiv.org / abs / 1706.03762

[0020] The functions of the components constituting the feature extraction device 1 are outlined below.

[0021] The control unit 9 controls the operation of each unit of the feature extraction device 1 based on the operation mode. The feature extraction device 1 operates in either an estimation mode or a learning mode. In the estimation mode, the control unit 9 controls each unit in the feature extraction device 1 to estimate and output a label set corresponding to an input sentence using a trained model. In the learning mode, the control unit 9 controls the model in the feature extraction device 1 to be trained based on training data.

[0022] That is, the control unit 9 controls whether the feature extraction device 1 operates in estimation mode or learning mode. When the feature extraction device 1 operates in estimation mode, the control unit 9 controls the output unit 16 to output a set of labels whose scores are equal to or greater than a predetermined threshold. When the feature extraction device 1 operates in learning mode, the control unit 9 controls the feature extraction unit 15, the bidirectional gated regression unit 14, the source-target attention unit 13, and the BERT unit 11 to perform error backpropagation based on the loss calculated by the loss calculation unit 21 and adjust the internal parameters of each of them.

[0023] When operating the feature extraction device 1 in the learning mode, a group of labels (label space) of classification candidates is defined in advance. In addition, to train the model in the feature extraction device 1, a sufficient number of pairs of news articles (input sentences) and correct label sets for each article are prepared as learning data. For example, articles from NHK NEWS WEB (https: / / www3.nhk.or.jp / news / ) can be used as news articles.

[0024] The BERT unit 11 is also called an "input feature generation unit." The BERT unit 11 calculates features (token embedding) of an input sentence based on data of the input sentence (token sequence) stored in the input sentence data storage unit 31. That is, the BERT unit 11 outputs token embedding corresponding to a token sequence corresponding to an input sentence written in a natural language. The BERT unit 11 also outputs features of a CLS token added to the beginning of the token sequence based on the token sequence. BERT stands for Bidirectional Encoder Representations from Transformers. The BERT unit 11 performs processing using a BERT model.

[0025] The self-attention unit 12 is also called a "label structure generation unit." The self-attention unit 12 performs self-attention processing based on the label data stored in the label set data storage unit 32, and determines a label structure that takes into account the similarity between labels. In other words, the self-attention unit 12 outputs a label structure consisting of the feature quantities of each label belonging to the label set, based on a label set expressed in natural language.

[0026] The source-target attention unit 13 is also called a "replacement unit." The source-target attention unit 13 performs source-target attention processing and replaces the feature of each token output by the BERT unit 11 with a vector derived from the label structure. More specifically, the source-target attention unit 13 receives a token embedding and a label structure, and replaces the feature of each token in the token embedding with a vector derived from the label structure, using a query in the attention mechanism as the token embedding and a key and value in the attention mechanism as the label structure, and outputs a label-replaced token embedding.

[0027] The bidirectional gated regression unit 14 is also called a "bidirectional gated regression processor." The bidirectional gated regression unit 14 performs processing of a bidirectional gated regression unit (BiGRU). That is, the bidirectional gated regression unit 14 inputs sequence data C representing the features of a token sequence derived from a label structure and outputs a relational feature S. That is, the bidirectional gated regression unit 14 performs bidirectional gated regression processing based on the label-substituted token embedding to generate input sentence features (relational feature S) that reflect the label structure.

[0028] The feature extraction unit 15 calculates the features of the input sentence in the label space based on the features of the CLS tokens calculated by the BERT unit 11 and the relation feature S output by the bidirectional gated regression unit 14. That is, the feature extraction unit 15 outputs a score for each label belonging to the label set based on the features of the CLS tokens and the input sentence features reflecting the label structure.

[0029] When the feature extraction device 1 operates in the estimation mode, the output unit 16 outputs a set of labels corresponding to the input sentence based on the features extracted by the feature extraction unit 15. Specifically, the output unit 16 outputs a set of labels whose scores are equal to or greater than a predetermined threshold.

[0030] When the feature extraction device 1 operates in learning mode, the loss calculation unit 21 calculates the loss (difference) between the feature output by the feature extraction unit 15 and the amount based on the correct answer data. In other words, the loss calculation unit 21 calculates the loss between the score of each label output by the feature extraction unit 15 and the correct answer score corresponding to the input sentence held in the learning data. The vector corresponding to the correct answer data is, for example, the following vector. That is, when a certain label is included in the correct answer, the score corresponding to that label is set to 1.0, and when a certain label is not included in the correct answer, the score corresponding to that label is set to 0.0. Based on the loss calculated by the loss calculation unit 21, the feature extraction device 1 adjusts and updates the parameters within the model using the backpropagation method.

[0031] The input sentence data storage unit 31 stores data corresponding to an input sentence (a string of tokens).

[0032] The label set data storage unit 32 stores data corresponding to a set of all label candidates.

[0033] The label structure data storage unit 33 stores the data of the label structure obtained by the self-attention unit 12.

[0034] The token embedding storage unit 34 stores, at least temporarily, data of the token embedding (including the feature amount of the CLS token) obtained by the BERT unit 11.

[0035] The CLS token feature storage unit 36 ​​stores, at least temporarily, the feature of the CLS token calculated by the BERT unit 11. The CLS token is a token added to the beginning of the input sentence data. The feature of the CLS token represents the feature of the input sentence (token sequence).

[0036] The label-substituted token embedding storage unit 35 stores, at least temporarily, the vector embedding data derived from the label structure obtained by the source-target attention unit 13 .

[0037] 2 is a block diagram showing a schematic functional configuration of the feature extraction unit 15 of the feature extraction device 1. As shown in the figure, the feature extraction unit 15 includes an addition unit 150, an FFNN 151, and a sigmoid function calculation unit 152. The functions of each unit are as follows:

[0038] The addition unit 150 adds the feature of the CLS token and the relation feature S. The result of the addition by the addition unit 150 becomes an input to the next FFNN·151.

[0039] FFNN·151 is a one-layer feedforward neural network. FFNN·151 receives the output from adder 150 and outputs a vector with dimensions that match the number of candidate labels. Each element of the vector output by FFNN·151 corresponds to the score of each label for the given input sentence.

[0040] The sigmoid function calculation unit 152 normalizes the score of each of the above labels by applying a sigmoid function to the output from the FFNN 151. The sigmoid function is a known monotonically increasing continuous function with a domain of (-∞, +∞) and a range of (0, 1).

[0041] 2 shows an example of a vector output by the sigmoid function calculation unit 152 in the form of a graph. In this graph, labels l1, l2, . . . , l nThe elements of the vector corresponding to each of the above are real numbers greater than 0 and less than 1. Each element value may be considered as a score corresponding to a label. When a predetermined threshold is set, the output unit 16 can output labels whose score values ​​are greater than the threshold as inference results corresponding to the input sentence. As shown in the figure, scores corresponding to multiple labels may exceed the threshold. In the example shown in the figure, the scores corresponding to labels l1, l2, l4 exceed the threshold. In this example, the output unit 16 outputs labels l1, l2, l4, ...

[0042] The specific processing flow of the feature extraction device 1 will be described in detail below.

[0043] The input sentence data storage unit 31 stores data of an input sentence. Here, the input sentence data is data representing a sentence consisting of one sentence or multiple sentences. In other words, the input sentence data storage unit 31 stores information on the sentence (or sentence) to be labeled. Specifically, the data stored in the input sentence data storage unit 31 is, for example, data of a token sequence obtained by dividing the text of a sentence into tokens. The input sentence data can be represented, for example, as x below.

[0044]

number

[0045] Here, m is the number of tokens corresponding to the input sentence. i (1≦i≦m) is a token. Each token constituting a token sequence can be represented as a distributed representation vector. The correspondence between tokens and distributed representation vectors can be referenced, for example, from a correspondence table (not shown). In other words, a token sequence x is data equivalent to a distributed representation vector sequence.

[0046] The feature extraction device 1 may also include a token sequence generation unit (not shown). In this case, the token sequence generation unit divides the input sentence text into token units and generates a token sequence. The token sequence generation unit can write the generated token sequence (distributed representation vector sequence) to the input sentence data storage unit 31.

[0047] The label set data storage unit 32 stores data of a label set (label space) that can be assigned by the feature extraction device 1. The label space can be expressed as L below.

[0048]

number

[0049] Here, n is the number of types of labels in the label space. j (1≦j≦n) is a label. Each label also corresponds to a distributed representation vector. In other words, the label space L can be represented as a set of distributed representation vectors. The label set data storage unit 32 stores this set of distributed representation vectors.

[0050] The BERT unit 11 calculates the features of the CLS token (a token that represents the input sentence) and the features of other tokens (token embedding) based on the input sentence data. In other words, the BERT unit 11 calculates the features of the input sentence based on the input sentence data. The BERT unit 11 has an internal BERT model. BERT stands for "Bidirectional Encoder Representations from Transformers." BERT extracts features from the token sequence (input sentence). The BERT model itself is realized using existing technology. The BERT model is also described in the references below. References: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of NAACL-HLT 2019, pages 4171-4186, Minneapolis,Minnesota,June 2 - June 7,2019,Association for Computational Linguistics,https: / / www.aclweb.org / anthology / N19-1423.pdf

[0051] The features of the CLS tokens generated by the BERT unit 11 are k-dimensional vectors, in other words, a matrix with 1 row and k columns. The token embedding B generated by the BERT unit 11 is expressed as a matrix with m rows and k columns. Here, k is the number of dimensions of the token features. k may be, for example, several hundred or several thousand. However, the value of k is not limited to the example given here. As an example, k may be 768. In other words, the BERT unit 11 reads the data x of the input sentence and outputs the token embedding B expressed as follows:

[0052]

number

[0053] The BERT unit 11 writes the above-mentioned CLS token feature amount and token embedding B into the token embedding storage unit 34. The BERT unit 11 also writes the CLS token feature amount into the CLS token feature storage unit 36.

[0054] The self-attention unit 12 performs self-attention processing based on a label group L of classification candidates. First, the self-attention unit 12 inputs one-hot vectors of the labels of the classification candidates into a single-layer feedforward neural network (FFNN, Feedforward Neural Network, forward propagation type neural network). This extracts feature quantities E of all classification candidate labels. The feature quantities E of the classification candidate labels are expressed as follows. That is, E can be expressed as a matrix with n rows and k columns.

[0055]

number

[0056] The self-attention unit 12 inputs the above feature E into the self-attention mechanism to extract a label structure E(bar), which is a feature that takes into account the similarity between each label. Specifically, the self-attention unit 12 calculates the similarity between each label based on the feature E, and calculates the label structure E(bar) by multiplying the similarity by the feature of all classification candidate labels. In other words, the "label structure" in this embodiment is a label feature that takes into account the similarity between labels. The label structure E(bar) is expressed as follows. That is, E(bar) can be expressed as a matrix with n rows and k columns.

[0057]

number

[0058] This E(bar) is calculated using the formula below.

[0059]

number

[0060] Note that Softmax() is a softmax function. In other words, the elements of the matrix output by the softmax function can be treated as probability values. Also, W1, W2, b1, and b2 are learning parameters, as shown below. That is, W1, W 2 of Each is a matrix with k rows and k columns. Also, b 1 ,b 2 Each of these is a matrix with n rows and k columns.

[0061]

number

[0062] The self-attention unit 12 writes the obtained label structure E (bar) into the label structure data storage unit 33.

[0063] The token embedding B output by the BERT unit 11 and the label structure E (bar) output by the self-attention unit 12 are input to the source-target attention unit 13. That is, the source-target attention unit 13 reads the token embedding B from the token embedding storage unit 34 and reads the label structure E (bar) from the label structure data storage unit 33.

[0064] The source-target attention unit 13 converts the feature of each token corresponding to the input sentence into a vector derived from the label structure. This extracts information about the input sentence based on the label structure, based on the premise that each token has a relationship with each label.

[0065] The source-target attention unit 13 sets the Query in the attention mechanism as token embedding B. The source-target attention unit 13 also sets the Key and Value in the attention mechanism as label structure E (bar). As a result, the source-target attention unit 13 replaces the feature of each token with a vector derived from label structure E (bar). This replaced feature is called label-replaced token embedding (label-aware token embedding). The label-replaced token embedding, which is output from the processing of the source-target attention unit 13, can be considered as sequence data C below.

[0066]

number

[0067] In other words, the sequence data C is a matrix with m rows and k columns. This sequence data C is calculated according to the formula below.

[0068]

number

[0069] The Softmax() function is as explained above. W3, W4, b3, and b4 are learning parameters, as follows: W 3 ,W 4 of Each is a matrix with k rows and k columns. Also, b 3 is a matrix with m rows and k columns. 4 is a matrix with n rows and k columns.

[0070]

number

[0071] The source-target attention unit 13 writes the sequence data C into the label-substituted token embedding storage unit 35.

[0072] The sequence data C output by the source-target attention unit 13 is input to the bidirectional gated regression unit 14 (BiGRU). That is, the bidirectional gated regression unit 14 reads the sequence data C from the label-substituted token embedding storage unit 35.

[0073] The bidirectional gated regression unit 14 calculates the relation feature S between the token and the label structure E (bar) based on the sequence data C. That is, it is as shown in the following formula. This relation feature S is a k-dimensional vector. This relation feature S is text information that takes into account the label structure. In other words, the bidirectional gated regression unit 14 performs processing to combine the features of each token in the sequence data C into S, which is the feature of the text.

[0074]

number

[0075] As described above, the features of the CLS token and the relation feature S are both k-dimensional vectors.

[0076] The feature extraction unit 15 receives the feature of the CLS token and the above-mentioned relation feature S. That is, the feature extraction unit 15 reads out the feature of the CLS token from the CLS token feature storage unit 36. The feature extraction unit 15 also receives the relation feature S from the bidirectional gated regression unit 14. The feature extraction unit 15 calculates the following feature F based on the feature of the CLS token and the relation feature S. Specifically, the FFNN 151 in the feature extraction unit 15 is a one-layer FFNN, which receives the sum of the feature of the CLS token and the relation feature S and outputs the feature F.

[0077]

number

[0078] In other words, the FFNN·151 performs linear transformation processing based on the relational feature S, which is a k-dimensional vector, and outputs the feature F. In other words, F = SW + B. Here, W (a matrix with k rows and n columns) and B (an n-dimensional vector) are learnable parameters of the FFNN·151.

[0079] As mentioned above, n is the number of label types in the label space. In other words, the feature F output by the feature extraction unit 15 is a vector with dimensions (n ​​dimensions) that match the number of candidate labels. Each element of the vector F is a value that represents the degree to which the corresponding label candidate is suitable as a feature of the input sentence. The sigmoid function calculation unit 152 in the feature extraction unit 15 further applies a sigmoid function to the feature F to normalize it. Each element of the n-dimensional vector resulting from the normalization takes a value greater than 0 and less than 1. The value of each element represents the degree to which the corresponding label suits the input sentence.

[0080] The output unit 16 determines a set of labels to be output based on the features output by the feature extraction unit 15. Specifically, as shown in FIG. 2, the output unit 16 determines, as the label to be assigned to the input sentence, a label corresponding to an element of the above-mentioned normalized vector that has a value equal to or greater than a predetermined threshold. As an example, the threshold value may be 0.5. However, the threshold value may be set to another value. The output unit 16 may output multiple labels as labels corresponding to the input sentence. The label L (bar) output by the output unit 16 is an element of the set L, as follows:

[0081]

number

[0082] When the feature extraction device 1 operates in learning mode, the output unit 16 does not output labels, and instead performs backpropagation based on the loss calculated by the loss calculation unit 21. At this time, the loss calculation unit 21 calculates the cross entropy for binary classification between the n-dimensional vector calculated by the feature extraction unit and the vector corresponding to the ground truth data as the loss. The loss calculation unit 21 calculates the loss between vectors x and y, for example, using the following formula:

[0083]

number

[0084] FIG. 3 is a block diagram showing an example of the internal configuration of the feature extraction device 1. The feature extraction device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0085] At least some of the functions of the feature extraction device 1 can be implemented by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB flash drives, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs over networks like the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may be designed to implement some of the aforementioned functions, or may be capable of implementing the aforementioned functions in combination with a program already stored in the computer system.

[0086] As described above, according to this embodiment, the self-attention unit 12 extracts a label structure for multi-level classification (MLC). This process by the self-attention unit 12 takes advantage of the fact that the labels themselves are expressed in natural language. That is, the self-attention unit 12 converts each label candidate (classification destination candidate) expressed in natural language into a feature to extract the label structure. The source-target attention unit 13 then incorporates the label structure into the feature of the input sentence, i.e., the feature of the token sequence corresponding to the input sentence. That is, the source-target attention unit 13 outputs the token embedding after label replacement. The feature extraction device 1 having this configuration can extract features of the input sentence taking the label structure into consideration. Therefore, the feature extraction device 1 can determine the label to be assigned with higher accuracy than conventional techniques. In other words, classification accuracy is improved compared to existing multi-label classification techniques.

[0087] The above has described in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0088] [Example] Examples of data (sentences such as news articles) processed by the feature extraction device 1 of the above embodiment and corresponding labels are as follows: Suppose the input sentence is, for example, "Stock prices have fallen due to the coronavirus..." The input sentence data storage unit 31 stores a token sequence of [CLS], "coronavirus," "due to," "stock prices," "is," "fall," ... corresponding to the above input sentence. Note that [CLS] is a CLS token added to the beginning of a sentence. Also, suppose the labels present in the label space are, for example, "business," "sports," "medical care," "stock prices / foreign exchange," "infectious disease," ...

[0089] The BERT unit 11 inputs the above token sequence of [CLS], "coronavirus," "due to," "stock price," "is," "fall," ... and outputs a sequence of token features (token embeddings) such as [CLS], "coronavirus," "due to," "stock price," "is," "fall," ... (see FIG. 4). Of these, the feature of the CLS token (see FIG. 5) is passed to the feature extraction unit 15 via the CLS token feature storage unit 36.

[0090] On the other hand, the self-attention unit 12 inputs a set of candidate labels such as "business," "sports," "medical care," "stock prices / foreign exchange," "infectious disease," etc., and outputs a label structure. The label structure is data having k-dimensional vectors (features) corresponding to each label (see Figure 6).

[0091] The source-target attention unit 13 generates label-substituted token embeddings based on the above token embeddings and label structure (see FIG. 7). The bidirectional gated regression unit 14 generates relational features S (k-dimensional vectors) based on the label-substituted token embeddings.

[0092] The feature extraction unit 15 then adds the features (k-dimensional vector) of the CLS tokens and the relation feature S (k-dimensional vector), and based on the result, finds features (n-dimensional vector) corresponding to the label space. As a result, for example, labels corresponding to the input sentence, such as "business," "stock price / exchange," and "infectious disease," are determined. These labels are the multiple labels to be assigned to the input sentence.

[0093] 4, 5, 6, and 7 are schematic diagrams showing the configuration of data corresponding to the examples described herein. FIG. 4 is a schematic diagram showing the feature of a CLS token (CLS embedding) in 1 row and k columns and the feature of a token (token embedding) in m rows and k columns calculated by the BERT unit 11. FIG. 5 is a schematic diagram showing only the feature of a CLS token (CLS embedding) in 1 row and k columns of the data shown in FIG. 4. The CLS token feature storage unit 36 ​​stores the data of FIG. 5. FIG. 6 is a schematic diagram showing data with a label structure of n rows and k columns calculated by the self-attention unit 12. FIG. 7 is a schematic diagram showing label-substituted token embedding data in m rows and k columns calculated by the source-target attention unit 13.

[0094] [Evaluation experiment] An evaluation experiment was conducted to verify the effectiveness of the feature extraction device 1 according to the above embodiment. As in the previous study, micro-F1 was selected as the evaluation index. In addition, model training and evaluation were performed five times using different initial random seed values, and the evaluation index was the average value.

[0095] Reuters-21578 was selected as the dataset used in the evaluation experiments. This dataset was constructed based on news articles from Reuters, and matches the objectives that the feature extraction device 1 aims to achieve. Note that, for strict comparison with the comparison methods, the training data and test data were split in accordance with ModApte split.

[0096] CNN-RNN and wdLSTM were used as comparison methods for the evaluation experiments. This is because both models use Reuters-21578, which complies with ModApte split, for evaluation, allowing for fair evaluation. In addition, to verify the effectiveness of the mechanism adopted in this embodiment that takes into account the label structure of SRN, we also conducted experiments using the BERT Fine Tuning model, which uses only CLS tokens, which are the output of BERT, as features for classification, as a comparison target.

[0097] The results of the experiment using the above dataset are shown in Table 1 below. "BERT FT" in the table is the above BERT Fine Tuning model. This BERT Fine Tuning model is a model of processing only through the flow of BERT unit 11 ⇒ token embedding storage unit 34 ⇒ CLS token feature storage unit 36 ​​⇒ feature extraction unit 15 in FIG. 1 (i.e., a model that does not have processing through the flow of label set data storage unit 32 ⇒ self-attention unit 12 ⇒ label structure data storage unit 33 ⇒ source-target attention unit 13 ⇒ label-substituted token embedding storage unit 35 ⇒ bidirectionally gated regression unit 14). Note that "SRN" in the table is the model adopted by the feature extraction device 1 of this embodiment.

[0098] [Table 1]

[0099] The numbers in parentheses in Table 1 above represent standard deviations. The performance values ​​of the CNN-RNN and wdLSTM methods used for comparison are taken from the following references: CNN-RNN literature: G. Chen, D. Ye, Z. Xing, J. Chen and E. Cambria, "Ensemble application of convolutional and recurrent neural networks for multi-label text categorization," 2017 International Joint Conference on Neural Networks (IJCNN), Anchorage, AK, 2017, pp. 2377-2383, DOI: 10.1109 / IJCNN.2017.7966144. wdLSTM Literature: Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin, “Rethinking Complex Neural Network Architectures for Document Classification”, Proceedings of NAACL-HLT 2019, pages 4046-4051, Minneapolis, Minnesota, June 2 - June 7, 2019, Association for Computational Linguistics.

[0100] The results in Table 1 show that the micro-F1 value of the SRN of this embodiment is 1.5 points higher than that of a model using LSTM (wdLSTM), confirming the effectiveness of this embodiment. Furthermore, compared to the BERT Fine Tuning model, the SRN of this embodiment has a micro-F1 value that is 0.2 points higher, a Precision value that is 0.2 points higher, and a Recall value that is 0.3 points higher, demonstrating the effectiveness of the mechanism of this embodiment that takes into account the relationship between tokens and label structures. [Industrial Applicability]

[0101] The present invention can be used for tasks that extract features of sentences while taking into account the label structure. As an example, the present invention can be used for tasks that involve multi-label classification of sentences (text) such as news. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0102] 1. Feature extraction device 9 Control Unit 11 BERT section (input feature generation section) 12 Self-attention section (label structure generation section) 13 Source-target attention part (substitution part) 14. Bidirectional gated regression unit (bidirectional gated regression processing section) 15 Feature extraction unit 16 Output section 21 Loss calculation section 31 Input sentence data storage unit 32 Label set data storage unit 33 Label structure data storage unit 34 Token Embedding Memory 35 Label-substituted token embedding memory 36 CLS token feature memory unit 150 Addition section 151 FFNN 152 Sigmoid function calculation section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. an input feature generation unit that outputs a token embedding corresponding to an input sentence written in a natural language based on the token sequence; a label structure generation unit that calculates and outputs a label structure consisting of features of each label belonging to the label set using a self-attention mechanism based on the label set expressed in natural language; a replacement unit that receives the token embeddings and the label structure, calculates and outputs label-replaced token embeddings using a source-target attention mechanism; Equipped with The substitution portion is [Equation 1] The label-substituted token embedding C is calculated as follows: B is the token embedding represented as an m-by-k matrix, E is the label structure represented as a matrix with n rows and k columns, W ３ and W ４ Each of and is a k-row, k-column matrix and is a learning parameter, b 3 is a matrix with m rows and k columns and is a learning parameter; b 4 is a matrix with n rows and k columns and is a learning parameter; Softmax() is the softmax function), Feature extractor.

2. the input feature generation unit further outputs, based on the token sequence, a feature of a CLS token added to the beginning of the token sequence; moreover, a bidirectional gated regression processor that performs bidirectional gated regression processing based on the label-replaced token embedding to generate input sentence features that reflect the label structure; a feature extraction unit that outputs a score of each label belonging to the label set based on the feature of the CLS token and the input sentence feature that reflects the label structure; The feature extraction device according to claim 1 , comprising:

3. an output unit that outputs a set of labels whose scores are equal to or greater than a predetermined threshold; The feature extraction device according to claim 2 , further comprising:

4. a control unit that controls whether the feature extraction device operates in an estimation mode or a learning mode; a loss calculation unit that calculates a loss between the score of each label output by the feature extraction unit and a correct answer score corresponding to the input sentence in the learning data; Equipped with The control unit When the feature extraction device operates in an estimation mode, the output unit is controlled to output a set of labels whose scores are equal to or greater than a predetermined threshold; When the feature extraction device operates in a learning mode, error backpropagation is performed based on the loss calculated by the loss calculation unit, and control is performed to adjust internal parameters of a feature extraction unit, a bidirectional gated regression processing unit, a substitution unit, and an input feature generation unit. The feature extraction device according to claim 3 .

5. an input feature generation unit that outputs a token embedding corresponding to an input sentence written in a natural language based on the token sequence; a label structure generation unit that calculates and outputs a label structure consisting of features of each label belonging to the label set using a self-attention mechanism based on the label set expressed in natural language; a replacement unit that receives the token embeddings and the label structure, calculates and outputs label-replaced token embeddings using a source-target attention mechanism; Equipped with The substitution portion is [Equation 2] The label-substituted token embedding C is calculated as follows: B is the token embedding represented as an m-by-k matrix, E is the label structure represented as a matrix with n rows and k columns, W ３ and W ４ Each of and is a k-row, k-column matrix and is a learning parameter, b 3 is a matrix with m rows and k columns and is a learning parameter; b 4 is a matrix with n rows and k columns and is a learning parameter; Softmax() is the softmax function), A program that enables a computer to function as a feature extractor.

Images