Machine learning device, natural language processing device, and program

Abstract

To provide a machine learning device, a natural language processing device and a program that more accurately extracts the position of a subsequence to be analyzed that is included in a word string.SOLUTION: In an opinion analysis device (natural language processing device) 1, a word embedding unit 11 accepts input of a word string and outputs a word embedded expression string that corresponds to the word string. A series labeling unit 12 accepts the word embedded expression string outputted from the word embedding unit 11 as input and outputs a label series that corresponds to the word embedded expression string. A pause position prediction unit 22 accepts the word embedded expression string outputted from the word embedding unit 11 as input and outputs pause position information that corresponds to the word embedded expression string, and performs back propagation based on an error between the pause position information and correct-answer pause position information so as to adjust the parameters of an internal model. The word embedding unit 11 adjusts the parameters of the internal model by back propagation of an error from the series labeling unit 12 and back propagation of an error from the pause position prediction unit 22.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a machine learning device, a natural language processing device, and a program. [Background technology]

[0002] For example, it would be effective to automatically analyze the trends in opinions of people or society as a whole by automatically analyzing the large number of sentences posted on social networking services (SNS), etc. However, it is unrealistic to manually analyze tens of millions to hundreds of millions of sentences or more, and there is a strong demand for accurate automatic analysis.

[0003] Conventional techniques attempt to automatically label phrases or words contained in a sentence. For example, deep learning models have been used to extract opinion objects and opinion parts from a sentence.

[0004] Non-Patent Document 1 describes BERT (Bidirectional Encoder Representations from Transformers), which is a mechanism for analyzing text written in natural language using a deep learning model.

[0005] Non-Patent Document 2 describes a technique for "sequence labeling" that uses machine learning to label sequences of sentence components. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of NAACL-HLT 2019, pages 4171-4186, Association for Computational Linguistics, 2019. [Non-Patent Document 2] Takatomo Ishikawa, Machine Learning for Language Processing "5. Sequence Labeling", [online], Internet <URL:https: / / www.slideshare.net / Takatymo / ss-64274683, July 22, 2016>.<X [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] However, in the prior art, when labeling the components of a sentence using a deep learning analysis model, there is a problem that the segmentation of the sentence to be labeled cannot always be performed correctly. That is, there is a problem that an error can occur in the segmentation of the sentence, which is the preprocessing of the labeling process. This is a problem that an error can occur in the sentence break of the opinion target interval or the opinion interval in the task of extracting the opinion target interval or the opinion interval in the text. When such an error in interval extraction occurs, the labeling may also result in an error. That is, in the task of extracting the opinion target interval or the opinion interval, some of the expressions that should be included in those intervals may be missing, or incorrect expressions that should originally be outside the interval may be extracted and included in the interval. Also, when an error as described above occurs in the labeling, there may be a problem that the accuracy of the analysis in the subsequent processing deteriorates. For example, when statistically aggregating the trends of opinions on SNS, it also becomes a factor that deteriorates the accuracy.

[0008] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a machine learning device, a natural language processing device, and a program that enable more accurate extraction of the position of a subsequence to be analyzed contained in a word sequence (specific examples of this subsequence are the above-mentioned opinion target section and opinion section). [Means for solving the problem]

[0009] [1] In order to solve the above problem, a machine learning device according to one aspect of the present invention includes a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string, a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence, a phrase break position prediction unit that inputs the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence, and supplies a word string to be input to the word embedding unit and a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and a learning data supply unit that supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit, wherein the sequence labeling unit adjusts the parameters of an internal model by performing backpropagation based on the error between the label sequence and the correct label sequence, the phrase break position prediction unit adjusts the parameters of the internal model by performing backpropagation based on the error between the phrase break position information and the correct phrase break position information, and the word embedding unit adjusts the parameters of the internal model by backpropagating the error from the sequence labeling unit and the error from the phrase break position prediction unit.

[0010] [2] In another aspect of the present invention, in the machine learning device described above, the phrase break position prediction unit outputs phrase break position information corresponding to the input word embedding sequence by using a fully connected regression model based on all word embeddings included in the input word embedding sequence.

[0011] [3] In another aspect of the present invention, in the above-described machine learning device, the sequence labeling unit outputs at least a label indicating that a predetermined subsequence included in the original input word sequence is an opinion target, and a label indicating that the predetermined subsequence is an opinion.

[0012] [4] Furthermore, one aspect of the present invention is that in the above-mentioned machine learning device, the phrase break position prediction unit outputs, as the phrase break position information, a numerical value representing the start position and end position of the subsequence, and the correct phrase break position information supplied by the learning data supply unit is numerical information representing the correct start position and end position of the subsequence.

[0013] [5] Another aspect of the present invention is a natural language processing device comprising: a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string; and a sequence labeling unit that inputs the word embedding sequence output by the word embedding unit and outputs a label sequence corresponding to the word embedding sequence, wherein at least a model internal to the word embedding unit has been trained by the machine learning device according to any one of claims 1 to 4.

[0014] [6] Also, one aspect of the present invention is a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string; a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence; a phrase break position prediction unit that inputs the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence; and a word embedding unit that supplies a word string to be input to the word embedding unit and also supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies the phrase break position information output by the phrase break position prediction unit. the sequence labeling unit adjusts parameters of an internal model by performing backpropagation based on an error between the label sequence and the correct label sequence; the phrase break position prediction unit adjusts parameters of the internal model by performing backpropagation based on an error between the phrase break position information and the correct phrase break position information; and the word embedding unit adjusts parameters of the internal model by backpropagating the error from the sequence labeling unit and the error from the phrase break position prediction unit. [Effects of the Invention]

[0015] According to the present invention, it is possible to improve the accuracy when assigning a label sequence (tag sequence) to a word sequence. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an opinion analysis device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing in more detail the configuration of a model for predicting a phrase break position in the embodiment. [Figure 3] 2 is a schematic diagram showing an example of a text input to the opinion analysis device according to the embodiment and an example of labeling performed on the input text. FIG. [Figure 4]4 is a schematic diagram showing an example of a word sequence corresponding to an input text example (FIG. 3) and a correct tag sequence corresponding to the word sequence in the embodiment. FIG. [Figure 5] FIG. 4 is a schematic diagram showing another example of a word sequence corresponding to the input text example (FIG. 3) and a correct tag sequence corresponding to the word sequence in the embodiment. [Figure 6] 10 is a schematic diagram showing an example of correct answer data for training a phrase break position prediction model in the embodiment. FIG. [Figure 7] 10 is a flowchart showing a processing procedure when the opinion analysis device according to the embodiment learns a model. [Figure 8] 10 is a flowchart showing a procedure when the opinion analysis device according to the embodiment performs a sequence labeling process using a trained model. [Figure 9] FIG. 2 is a block diagram showing an example of the internal configuration of the opinion analysis device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Next, an embodiment of the present invention will be described with reference to the drawings. The device according to this embodiment is an opinion analysis device for analyzing opinions contained in input text. Specifically, the opinion analysis device of this embodiment assigns tags (labels) to elements (phrases) that make up the input text to distinguish them as opinion target sections, opinion sections, or sections that are neither.

[0018] A feature of the opinion analysis device of this embodiment is that it includes a regression model for predicting division points (phrase breaks) of the extracted section. This regression model specific to this embodiment may be referred to as the "phrase break position prediction model" below. The phrase break position prediction model, together with a model for performing sequence labeling (sequence labeling model), shares a distributed representation in a lower layer. The opinion analysis device of this embodiment performs machine learning on an internally stored model and is also called a "machine learning device." Furthermore, the opinion analysis device of this embodiment uses a trained model to provide a label sequence corresponding to a given unknown word sequence and is also called a "natural language processing device."

[0019] In this embodiment, in order to perform machine learning of the phrase break position prediction model, a loss function is used that gives the error between the phrase break position predicted by the phrase break position prediction model and the correct phrase break position in terms of cross entropy.

[0020] FIG. 1 is a block diagram showing a schematic functional configuration of an opinion analysis device according to this embodiment. As shown in the figure, the opinion analysis device 1 includes a word embedding unit 11, a sequence labeling unit 12, a sequence labeling loss function calculation unit 17, a phrase break position prediction unit 22, a phrase break position loss function calculation unit 27, and a training data supply unit 30. Each of these functional units can be implemented, for example, by a computer and a program. Each functional unit also includes a storage unit as needed. The storage unit may be, for example, a program variable or memory allocated by program execution. Nonvolatile storage units such as a magnetic hard disk drive or solid-state drive (SSD) may also be used as needed. At least some of the functions of each functional unit may be implemented as a dedicated electronic circuit rather than a program. The functions of each unit are described below.

[0021] The word embedding unit 11 receives a word string and outputs a word embedding representation string corresponding to the word string. The word embedding unit 11 includes an internal model capable of machine learning. The model included in the word embedding unit 11 is also called a "word embedding layer." The model included in the word embedding unit 11 is implemented, for example, using a neural network. In this embodiment, BERT is used as the deep learning model of the opinion target extraction device. BERT itself is an existing technology and is described in the aforementioned Non-Patent Document 1, etc. The BERT used by the word embedding unit 11 may be a general model pre-trained in Japanese. In this embodiment, the BERT included in the word embedding unit 11 can be further trained. The input sentence is divided into words using standard morphological analysis processing. In other words, the input sentence is equivalent to a word string. A word string corresponding to the input sentence is input to the word embedding unit 11 and converted into a word embedding representation string.

[0022] The sequence labeling unit 12 receives the word embedding sequence output from the word embedding unit 11 and outputs a label sequence corresponding to the word embedding sequence. In other words, the sequence labeling unit 12 assigns a label to each word. Specific examples of labels will be explained later. The sequence labeling unit 12 includes an internal model capable of machine learning. The model included in the sequence labeling unit 12 is called a "sequence labeling model."

[0023] In this embodiment, the sequence labeling unit 12 outputs at least a label indicating that a predetermined subsequence included in the input original word sequence is an opinion target, and a label indicating that the predetermined subsequence is an opinion. More specific examples of the labels assigned by the sequence labeling unit 12 will be described later.

[0024] The sequence labeling loss function calculation unit 17 calculates the error between the label sequence output by the sequence labeling unit 12 and the correct label sequence that is the correct answer. The error calculated by the sequence labeling loss function calculation unit 17 is the error that serves as the basis for error backpropagation performed by the sequence labeling unit 12 to adjust the parameters.

[0025] The phrase break position prediction unit 22 receives the word embedding sequence output from the word embedding unit 11 and outputs phrase break position information corresponding to the word embedding sequence. The phrase break position prediction unit 22 includes an internal model capable of machine learning. The model included in the phrase break position prediction unit 22 is called a "phrase break position prediction model."

[0026] The phrase break position prediction unit 22 may output phrase break position information corresponding to the input word embedding sequence by using a fully connected regression model based on all word embeddings included in the input word embedding sequence.

[0027] The phrase break position loss function calculation unit 27 calculates the error between the phrase break position information output by the phrase break position prediction unit 22 and the correct phrase break position information. The error calculated by the phrase break position loss function calculation unit 27 is the basis for the error backpropagation performed by the phrase break position prediction unit 22 to adjust the parameters.

[0028] When performing machine learning of the model, the word embedding unit 11 adjusts the parameters of the internal model based on both the backpropagation of errors from the sequence labeling unit 12 and the backpropagation of errors from the phrase break position prediction unit 22.

[0029] The learning data supplying unit 30 supplies learning data for training a model included in the opinion analysis device 1. Specifically, the learning data supplying unit 30 supplies a word sequence to be input to the word embedding unit 11. The learning data supplying unit 30 also supplies a correct label sequence that is the correct answer corresponding to the label sequence output by the sequence labeling unit 12, in correspondence with the word sequence. Furthermore, the learning data supplying unit 30 supplies correct phrase break position information that is the correct answer corresponding to the phrase break position information output by the phrase break position predicting unit 22, in correspondence with the word sequence.

[0030] The phrase break position prediction unit 22 outputs, as phrase break position information, numerical values representing the start and end positions of a subsequence of a word string corresponding to the input sentence. The number of subsequences may be one or more. The correct phrase break position information supplied by the training data supply unit 30 is numerical information representing the correct start and end positions of the subsequence. The start and end positions are numerical information representing the ordinal numbers of the words. The correct start and end positions of a subsequence are essentially integers. The start and end positions predicted and output by the phrase break position prediction unit 22 are not necessarily integers. Usually, the start and end positions predicted and output by the phrase break position prediction unit 22 are non-integer. These non-integer values can be considered to be predicted values that are approximations of the start and end positions, respectively.

[0031] FIG. 2 is a configuration diagram showing in more detail the configuration of the model for predicting phrase break positions. As shown in the figure, the model for predicting phrase break positions includes a fully connected recurrent layer 220. The fully connected recurrent layer 220 is a model included in the phrase break position prediction unit 22 shown in FIG. 1. The fully connected recurrent layer 220 includes two nodes (y1 and y2 in the figure) connected to all outputs from the word embedding layer 110 (BERT embedding layer). The word embedding layer 110 is a model included in the word embedding unit 11 shown in FIG. 1. The nodes included in the word embedding layer 110 (x1, x2, . . . , x D ) correspond to the embedded representations of words contained in the input text. y1 and y2 are numerical values that respectively represent the start position (start_position) and end position (end_position) of a segment break. These numerical values represent the ordinal positions of the words in the text. In other words, a model for predicting segment break positions predicts the start and end positions of one or more segments. In other words, the fully connected recurrent layer 220 is a two-dimensional regression model for predicting these two numerical values.

[0032] That is, the word embedding layer 110 is a D-dimensional continuous value matrix x1, x2, . . . , xD The fully connected recurrent layer 220 outputs the above x1, x2, . D The program outputs y1 and y2 calculated based on the above. y1 is the start position and y2 is the end position, and y1 and y2 are as shown in the following formula (1).

[0033]

number

[0034] In equation (1), f is the activation function. ij and b j are the internal parameters of the model of the fully connected recurrent layer 220. The values of these internal parameters are adjusted by learning. Note that j is either 1 or 2.

[0035] That is, the start position y1 and the end position y2 are the output values x1, x2, . . . , x D the weighted sum of (with a given parameter value b j It is calculated by applying an activation function to the

[0036] The outputs y1 and y2 from the fully connected recurrent layer 220 are compared with the intervals (correct answers) included in the training data. Specifically, the opinion analysis device 1 learns the internal parameters of the model using a criterion that minimizes the cross-entropy between the output from the fully connected recurrent layer 220 and the label intervals that appear in the training data. When calculating the criterion for the label intervals, all label intervals that appear in the training data are included in the criterion, without narrowing it down to one, except when pruning is performed for computational efficiency.

[0037] When training the model, the loss function L expressed by the following equation (2) is used. span (S) is used.

[0038]

number

[0039] In formula (2), SpanSet(S) is a set of annotation intervals that appear in sentence S, which is input text to the opinion analysis device 1. The start position of one interval l belonging to the set SpanSet(S) is Begin(l) in formula (2), and the end position is End(l). In other words, one interval (span) is defined by the value of the start position and the value of the end position. CrossEntropy(y,x) in formula (2) is the cross entropy between y (i.e., y1 or y2), which is one of the outputs from the fully connected recurrent layer 220, and the correct answer for the start position or end position given by the training data.

[0040] In other words, L span (S) is based on the sum of the cross entropy between y1 and the correct answer for the start position for each annotation section and the cross entropy between y2 and the correct answer for the end position for all annotation sections.

[0041] Cross entropy (y, x) is calculated by the following equation (3).

[0042]

number

[0043] In equation (3), k is an index representing a position within sentence S. Furthermore, max_seq_length is a value corresponding to the sequence length of sentence S. Furthermore, exp is an exponential function.

[0044] The opinion analysis device 1 calculates a loss function L for the above-mentioned phrase break position prediction model. span In addition to (S), the loss function L for the model held by the sequence labeling unit 12 (Fig. 1) seq (S) is used. Loss function L seq (S) is the loss function in the sequence labeling problem according to the prior art. The overall loss function L of the model when one sentence (S) is input to the opinion analysis device 1 istotal (S) is calculated as shown in equation (4) below.

[0045]

number

[0046] In other words, the overall loss function L total (S) is the loss function L for the phrase break position prediction model. span (S), and the loss function L according to the prior art for the model held by the sequence labeling unit 12. seq It is calculated as the sum of (S).

[0047] Of the model held by the sequence labeling unit 12 and the model held by the phrase break position prediction unit 22, the model held by the phrase break position prediction unit 22 is used only during learning, not during inference. In other words, as learning progresses so that the value of the overall loss function (the above equation (4)) decreases, the learning accuracy of the distributed representations in the lower layer (the word embedding layer 110 of the word embedding unit 11) improves. This improves the accuracy of labeling (the task of identifying opinion target sections and opinion sections) using the model held by the sequence labeling unit 12.

[0048] Fig. 3 is a schematic diagram showing an example of text input to the opinion analysis device 1 and an example of labeling performed on the input text. In other words, Fig. 3 is a schematic diagram showing an example of processing for extracting opinion target sections and opinion sections from input text.

[0049] The input text example shown in the figure is a single sentence posted to a social networking site. This text is "Violence is bad, it's bad #anime." Note that "#anime" in this text is an expression that follows the notation of tags (called hashtags because it uses "#"), which make it easier to search for specific topics. This input text is divided into eight words: "Violence / is / bad / , / bad / yo / # / anime" (slashes represent word separators). Here, punctuation marks and symbols such as "#" are treated as words for convenience. The opinion target sections extracted from this input text are "Violence" (start position 1, end position 1) and "#anime" (start position 7, end position 8). The opinion sections extracted are "Bad" (start position 3, end position 3) and "Bad" (start position 5, end position 5).

[0050] FIG. 4 is a schematic diagram showing an example of a word sequence corresponding to the example input text shown in FIG. 3 and a correct tag sequence (label sequence) corresponding to that sequence. The illustrated word sequence is "violence / is / not / , / is / not / yo / # / anime." In this diagram, numbers (position numbers) are assigned according to the order of the words. The correct tag sequence corresponding to such a word sequence is "TARGET / O / OPINION / O / OPINION / O / TARGET / TARGET." The tag "TARGET" indicates that the corresponding word belongs to the opinion target section. The tag "OPINION" indicates that the corresponding word belongs to the opinion section. The tag "O" indicates that the corresponding word does not belong to either the opinion target section or the opinion section.

[0051] In other words, this correct tag sequence indicates that the expression "violence" (the start position of the section is 1 and the end position is 1) and the expression "#anime" (the start position of the section is 7 and the end position is 8) in the word sequence are each the opinion target section. Also, the expressions "Ikenai" (the start position of the section is 3 and the end position is 3) and "Ikenai" (the start position of the section is 5 and the end position is 5) in the word sequence are each the opinion section.

[0052] FIG. 5 is a schematic diagram showing another example of a word sequence corresponding to the example input text shown in FIG. 3 and a corresponding correct tag sequence. Unlike the tags in FIG. 4, the example in FIG. 5 adds a prefix "B-" or "I-" to the tags "TARGET" and "OPINION," respectively. Such tags are called BIO-format tags. "B-" indicates the beginning of a section, while "I-" indicates the middle of a section. The tag assigned to the word at the end of a section also begins with "I-." In the example shown in FIG. 5, the correct tag sequence corresponding to the word sequence "violence / wa / bad / , / bad / yo / # / anime" is "B-TARGET / O / B-OPINION / O / B-OPINION / O / B-TARGET / I-TARGET."

[0053] In other words, this correct tag sequence indicates that the expression "violence" (the section starts at 1 and ends at 1) and the expression "#anime" (the section starts at 7 and ends at 8) in the word sequence are each the opinion section. Also, the expressions "Ikenai" (the section starts at 3 and ends at 3) and "Ikenai" (the section starts at 5 and ends at 5) in the word sequence are each opinion sections. In BIO format tags, adding "B-" or "I-" explicitly indicates that, for example, the seventh and eighth words in Figure 5 belong to the same section.

[0054] FIG. 6 is a schematic diagram showing an example of correct answer data for training a phrase break position prediction model. As shown in the figure, this correct answer data is given as a set of pairs of the start and end positions of sections. In the example shown, the correct answer data includes data on four sections. These sections are: start position 1 and end position 1, start position 3 and end position 3, start position 5 and end position 5, and start position 7 and end position 8. This correct answer data corresponds to the examples shown in FIGS. 4 and 5. The data shown in FIG. 6 represents the correct answers for the start and end positions also shown in FIG. 2.

[0055] 7 is a flowchart showing the procedure of processing when the opinion analysis device 1 learns a model. The procedure of learning a model will be described below with reference to this flowchart.

[0056] In step S11, the training data supply unit 30 supplies training data. The training data is a pair of an input text (word sequence) and correct answer data corresponding to the input data. The correct answer data includes the correct answer of the tag sequence and the position of the section (start position and end position). The word embedding unit 11 reads the input text (word sequence). Furthermore, the sequence labeling loss function calculation unit 17 reads the correct answer tag sequence. Then, the phrase break position loss function calculation unit 27 reads the correct answer data of the start position and end position of the section (for example, FIG. 6).

[0057] In step S12, the opinion analysis device 1 adjusts the internal parameters of each model based on the calculated loss function value. Specifically, the adjustment is performed as follows.

[0058] The word embedding unit 11 passes a sequence of embedded representations corresponding to the input word sequence to the sequence labeling unit 12 and the phrase break position prediction unit 22. The sequence labeling unit 12 calculates sequence labels using a sequence labeling model with internal parameter values at that time. Furthermore, the phrase break position prediction unit 22 determines the phrase break position using a phrase break position prediction model based on the internal parameter values at that time. The sequence labeling loss function calculation unit 17 calculates a loss function value based on the sequence label output by the sequence labeling unit 12 and the correct sequence label. The phrase break position loss function calculation unit 27 calculates a loss function value based on the phrase break position output by the phrase break position prediction unit 22 and the correct phrase break position (start position and end position). Based on these calculated loss function values, the internal parameters of each model are adjusted using the backpropagation method. Repeated adjustment of the internal parameters of such models works in the direction of decreasing the loss function value. In other words, repeated parameter adjustment enables each model to calculate an output value closer to the correct answer. Note that the method of adjusting model parameters using the backpropagation method itself belongs to existing technology.

[0059] In step S13, the opinion analysis device 1 saves the state of each model after adjusting the parameters. Specifically, the opinion analysis device 1 writes the adjusted parameter values into a non-volatile memory or the like.

[0060] Figure 8 is a flowchart showing the procedure when the opinion analysis device 1 performs sequence labeling processing using a trained model. Because the state of the model is saved in step S13 of the processing procedure shown in Figure 7, the opinion analysis device 1 can read out the state of each model. Below, the procedure for estimating sequence labels will be explained according to this flowchart.

[0061] In step S21, the word embedding unit 11 reads the input text (word sequence).

[0062] In step S22, the opinion analysis device 1 uses the model to infer a tag sequence corresponding to the input text. Specifically, the word embedding unit 11, which has a word embedding model, outputs a sequence of vectors calculated by the word embedding model. The sequence labeling unit 12, which has a sequence labeling model, calculates a tag sequence based on the sequence of vectors passed from the word embedding unit 11.

[0063] In step S23, the sequence labeling unit 12 outputs the tag sequence obtained in step S22. Examples of tag sequences corresponding to the input text (word string) are as shown in FIGS.

[0064] As described above, by using the opinion analysis device of this embodiment, the accuracy of tags assigned can be improved in the process of automatically assigning a tag sequence corresponding to an input word sequence. As a specific example, it becomes possible to accurately assign a tag indicating that the word is a word representing an opinion object or a tag indicating that the word is a word representing an opinion to each word in the input word sequence. In other words, it becomes possible to accurately extract parts of the input word sequence that have specific properties (e.g., opinion object, opinion, etc.).

[0065] The device of this embodiment can be used to extract portions of a specific nature (e.g., portions that are the subject of discussion, portions that are opinions, etc.) from text posted on an SNS. For example, when automatically analyzing reactions to viewing specific content (such as a broadcast program) based on the content posted on an SNS, even if the text of the posted content itself does not explicitly indicate which portions are the subject of discussion and which portions are opinions, these portions can be automatically and accurately extracted. Alternatively, even if the text of the posted content does not include hashtags or the like that indicate which content the reaction is directed to, it is possible to automatically extract portions that are the subject of discussion, portions that are opinions, etc. In other words, the device of this embodiment enables more powerful and accurate analysis than methods such as keyword search. In other words, this embodiment makes it possible to easily and accurately conduct large-scale market research, etc.

[0066] If it is possible to automatically extract opinion subjects and opinions from a large amount of text (for example, several thousand to several hundred thousand items) posted on social networking sites, etc., it will be possible to automatically tally up what kind of opinions were expressed and to what extent for each opinion subject. When tallying up, it is possible to tally up the same opinion subject or to perform clustering processing to group together similar opinion subjects. Conventional technology can be used as the tallying and clustering methods. For example, it will be possible to automatically tally up viewer opinions on a broadcast program.

[0067] FIG. 9 is a block diagram showing an example of the internal configuration of the opinion analysis device 1 of the above embodiment. The opinion analysis device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0068] At least some of the functions of the opinion analysis device 1 described above can be implemented by a computer. In this case, a program for implementing these functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. The term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB flash drives, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs over networks like the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within the computer systems that serve as the server or client. The program may be a program that implements some of the functions described above, or it may be a program that can be implemented in combination with a program already stored in the computer system.

[0069] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples.

[0070] [First Modification] In the above explanation, the text to be processed is written in Japanese, but this embodiment may also be applied to data other than Japanese. In the demonstration experiment described later, the text to be processed is in English as well as Japanese. Furthermore, text in other languages may also be processed.

[0071] [Second Modification] In the above embodiment, one opinion analysis device 1 both learns a model and performs a process of assigning (estimating) a label sequence using the trained model. As a variation, the opinion analysis device 1 may perform only one of model learning and label sequence estimation using the trained model. When the opinion analysis device 1 functions as a machine learning device, the parameters of the trained model can be transplanted to another device, and the destination device can perform a process of assigning (estimating) a label sequence. Alternatively, the opinion analysis device 1 may not perform machine learning itself, but may instead acquire parameters of a trained model and perform a process of assigning (estimating) a label sequence.

[0072] In the latter case, the opinion analysis device 1 functions as a natural language processing device including a word embedding unit that inputs a word string and outputs a word embedding sequence corresponding to the word string, and a sequence labeling unit that inputs the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence. In this case, at least the model held internally by the word embedding unit 11 is one that has been trained using the machine learning mechanism (machine learning device) described in the above embodiment.

[0073] The above has described in detail an embodiment of the present invention and its modified examples with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0074] [Implementation and Evaluation] The opinion analysis device 1 was implemented and evaluated through an experiment using actual posts on SNS (Twitter) and movie reviews. The evaluation results from the experiment will be described below.

[0075] The data used for the evaluation consisted of Japanese tweets about television programs and English reviews of movies. Specifically, the Japanese text used was tweets in Japanese from Twitter about the television program (NHK morning drama) "Natsuzora" over a nine-day period (24,610 tweets, 984,861 words). The English text used was reviews from the review website IMDb (986 reviews, 10,360 sentences, 321,807 words). Correct answers were manually annotated to the training data.

[0076] The evaluation measures used were precision, recall, and f1, which are defined by equations (5), (6), and (7), respectively. Note that here, TP is true positive, FP is false positive, and FN is false negative.

[0077]

number

[0078]

number

[0079]

number

[0080] Table 1 shows the experimental results when the labels were limited to the top level (first hierarchical level) for the task of analyzing Japanese text (tweets) about broadcast programs.

[0081] In this experiment, labels were organized into three layers, with two main types at the first layer. These two types are REFERENCE labels, which are assigned to parts that refer to the content of the broadcast program, and OPINION labels, which are assigned to parts that express the sender's subjective opinions. REFERENCE labels have the following labels at the second layer: TITLE, MUSIC, PERSON, PROGRAM, SCENE, STORY, and QUOTE. Meanwhile, OPINION labels have the following labels at the second layer: EVALUATION, ACTION, and INDEX. Of these, ECALUATION at the second layer has the following labels at the third layer: POSITIVE, NEGATIVE, NEUTRAL, and REQUEST.

[0082] In the experiment, the first-layer REFERENCE and OPINION labels were evaluated individually, and the entire system was also evaluated. The conventional technology used for comparison does not use the phrase break position prediction unit 22 (see Figure 1) even during training.

[0083] [Table 1]

[0084] It was confirmed that the results of this embodiment outperformed the results of the prior art in all other cases except for the REFERENCE label in the evaluation measure recall, in which the result value of this embodiment was slightly lower than the result of the prior art.

[0085] The effect of using this embodiment differs depending on the number of epochs used when training the model. At the initial stage of training (less than about 10 epochs), there is no significant difference in performance evaluated by the f1 value between the conventional technology and this embodiment. However, the effect of this embodiment begins to be seen after about 10 epochs, and the superiority of this embodiment (the difference in performance in the f1 value compared to the conventional technology) becomes significant after about 20 epochs.

[0086] The experimental results for the task of analyzing movie review text written in English, when the labels were limited to the top first layer, are shown in Table 2. As with Table 1, in Table 2 we evaluated the REFERENCE and OPINION labels in the first layer individually, as well as the overall evaluation.

[0087] [Table 2]

[0088] In this analysis of English text, it was confirmed that the present embodiment outperformed the results of the prior art in most cases. The evaluation result of the present embodiment was lower than the evaluation result of the prior art only for the REFERENCE label of the evaluation scale recall. In all other cases, the evaluation result of the present embodiment was higher than the evaluation of the prior art.

[0089] The significance level (p-value) indicating whether the improvement in this embodiment is significant compared to the conventional technology is shown in Table 3 below. The significance level shown here is the result of a McNemar test.

[0090] [Table 3]

[0091] The experimental results for the task of analyzing Japanese text (tweets) related to broadcast programs, without restricting the label hierarchy, are shown in Table 4 below. The values shown in Table 4 are the overall evaluation results (micro-average) of all labels.

[0092] [Table 4]

[0093] As shown in Tables 1, 2, and 4 above, it was confirmed by experiments that the accuracy of extracting expressions with specific positions in text (extracting opinion targets) according to this embodiment is better than that of the prior art. Furthermore, as shown in Table 3, it was confirmed by testing that the improvement in accuracy was significant. [Industrial Applicability]

[0094] The present invention can be used, for example, to extract elements of a specific position from a sentence written in a natural language. As an example, it can be used to extract elements of a specific position from text posted on a social networking site. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0095] 1. Opinion analysis device (machine learning device, natural language processing device) 11 Word Embedding 12 Series Labeling Section 17 Loss function calculation for sequence labeling 22 Phrase break position prediction section 27 Loss function calculation for phrase break position 30 Learning data supply unit 220 Fully connected recurrent layer 110 Word Embedding Layer (BERT Embedding Layer) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string; a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence; a phrase break position prediction unit that receives the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence; a learning data supply unit that supplies a word sequence to be input to the word embedding unit, supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit; Equipped with the sequence labeling unit adjusts parameters of an internal model by performing backpropagation based on an error between the label sequence and the correct label sequence; the phrase break position prediction unit adjusts parameters of an internal model by performing back propagation based on an error between the phrase break position information and the correct phrase break position information; the word embedding unit adjusts parameters of an internal model by backpropagating errors from the sequence labeling unit and backpropagating errors from the phrase break position prediction unit; Machine learning device.

2. the phrase break position prediction unit outputs phrase break position information corresponding to the word embedding sequence by using a fully connected regression model based on all word embeddings included in the input word embedding sequence. The machine learning device according to claim 1 .

3. the sequence labeling unit outputs at least a label indicating that a predetermined subsequence included in the original input word sequence is an opinion target, and a label indicating that the predetermined subsequence is an opinion; The machine learning device according to claim 1 or 2.

4. the phrase break position prediction unit outputs, as the phrase break position information, numerical values representing the start position and end position of the substring; the correct phrase-break position information provided by the learning data providing unit is numerical information indicating the correct start and end positions of the subsequence; The machine learning device according to claim 3 .

5. a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string; a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence; Equipped with At least the model internal to the word embedding unit has been trained by the machine learning device according to any one of claims 1 to 4. Natural language processing device.

6. a word embedding unit that receives a word string and outputs a word embedding string corresponding to the word string; a sequence labeling unit that receives the word embedding sequence output from the word embedding unit and outputs a label sequence corresponding to the word embedding sequence; a phrase break position prediction unit that receives the word embedding sequence output from the word embedding unit and outputs phrase break position information corresponding to the word embedding sequence; a learning data supply unit that supplies a word sequence to be input to the word embedding unit, supplies a correct label sequence corresponding to the label sequence output by the sequence labeling unit, and further supplies correct phrase break position information corresponding to the phrase break position information output by the phrase break position prediction unit; Equipped with the sequence labeling unit adjusts parameters of an internal model by performing backpropagation based on an error between the label sequence and the correct label sequence; the phrase break position prediction unit adjusts parameters of an internal model by performing back propagation based on an error between the phrase break position information and the correct phrase break position information; the word embedding unit adjusts parameters of an internal model by backpropagating errors from the sequence labeling unit and backpropagating errors from the phrase break position prediction unit; A program that enables a computer to function as a machine learning device.

Images