Validity determination device, validity determination method, and validity determination program

Abstract

A validity assessment device according to one embodiment of the present invention comprises: a data acquisition unit for acquiring a natural sentence; a meaning analyzer for implementing an analysis of meaning using the natural sentence as an input, thereby outputting a meaning expression; an alignment score calculation unit for calculating a first alignment score representing the relationship between the natural sentence and an input / output token of the meaning expression; a score adjustment unit for calculating a second alignment score in word units from the relationship between a word and a token included in the input / output token; a part-of-speech analyzer for implementing an analysis of parts of speech with respect to the natural sentence, thereby extracting a natural-sentence content word; a grammar analyzer for implementing an analysis of the grammar in the meaning expression, thereby extracting a meaning-expression content word; and a validity assessment unit for assessing whether or not the meaning expression is valid, the assessment being carried out on the basis of the second alignment score, the natural-sentence content word, and the meaning-expression content word.

Description

[Technical Field]

[0001] The present invention relates to a validity determination device, a validity determination method, and a validity determination program. [Background technology]

[0002] Conventionally, when determining the answerability of an input to a semantic analysis model, the validity determination device must prepare a dataset for determining answerability separate from the semantic analysis dataset and model, and must also design and train the model.

[0003] For example, Non-Patent Document 1 proposes a technology that distinguishes between four types of unanswerable questions in order to classify the intent of questions. Also, for example, Non-Patent Document 2 proposes a technology in which a semantic parser detects unanswerable questions and unclear parts and suggests rephrasing. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zhang, Yusen et al. “Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL.” ArXiv abs / 2010.12634 (2020) [Non-patent document 2] Zeng, Jichuan et al. “Photon: A Robust Cross-Domain Text-to-SQL System.” ACL (2020). Summary of the Invention [Problem to be solved by the invention]

[0005] The techniques proposed in Non-Patent Document 1 and Non-Patent Document 2 both have the problem that they require the creation of a dataset and the design and training of another model in addition to a semantic analyzer.

[0006] Furthermore, compared with the semantic analysis model, the answerability model has different accuracy and behavior, so there is a problem that the answerability is not consistent with the semantic analysis.

[0007] There is also a desire to control and analyze target systems, such as network / server resources, from natural language (hereinafter simply referred to as NL). That is, there is a desire to generate appropriate semantic representations (hereinafter simply referred to as MR) from NL and apply them to the target system. However, MRs generated from NL may not be suited to the target system, or executing an inappropriate MR may cause unintended behavior in the target system. Therefore, there is a risk in running the generated MR as is. Therefore, there is also a desire to determine whether a program can be applied to the target system, i.e., whether it is valid.

[0008] This invention was made in light of the above circumstances, and its purpose is to provide a technology that can determine the validity of output results from the correspondence between input and output of a trained semantic analysis model, without creating a special data set and designing and training a model. [Means for solving the problem]

[0009] In order to solve the above problem, one aspect of the present invention is a validity determination device that includes a data acquisition unit that acquires natural text, a semantic analyzer that performs semantic analysis using the natural text as input and outputs a semantic expression, an alignment score calculation unit that calculates a first alignment score that represents the relationship between input and output tokens of the natural text and the semantic expression, a score correction unit that calculates a second alignment score on a word-by-word basis from the relationship between tokens and words included in the input and output tokens, a part-of-speech analyzer that extracts natural text content words by performing a part-of-speech analysis of the natural text, a grammar analyzer that extracts semantic expression content words by performing a grammatical analysis of the semantic expression, and a validity determination unit that determines whether the semantic expression is valid based on the second alignment score, the natural text content words, and the semantic expression content words. [Effects of the Invention]

[0010] According to one aspect of the present invention, the validity determination device can use a trained model to determine the validity of an output result from the correspondence between input and output. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of a validity determination device according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of the validity determination device according to the embodiment in relation to the hardware configuration shown in FIG. [Figure 3] FIG. 3 is a flowchart showing an example of an operation performed by the validity determination device to determine MR validity. [Figure 4] FIG. 4 is a diagram showing an example of alignment scores of tokens and words. [Figure 5] FIG. 5 is a diagram showing an example of the alignment score to be extracted. [Figure 6] FIG. 6 is a diagram showing an example of SQL estimated from a query and correct SQL. [Figure 7] FIG. 7 is a diagram showing some of the alignment scores for each content word calculated by the MR validity determination unit. [Figure 8] FIG. 8 is a diagram showing an example of SQL estimated from a query and correct SQL. [Figure 9] FIG. 9 is a diagram showing some of the alignment scores for each content word calculated by the MR validity determination unit. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that, hereinafter, elements that are the same as or similar to elements that have already been described will be designated by the same or similar reference numerals, and duplicated descriptions will basically be omitted.

[0013] [Embodiment] (composition) FIG. 1 is a block diagram showing an example of a hardware configuration of a validity determination device 1 according to an embodiment. The validity determination device 1 may be, for example, a user terminal used by a user. Here, the user terminal may be any computer that can be generally used by a user, such as a PC (Personal Computer), a smartphone, a tablet terminal, or a wearable terminal. The validity determination device 1 may also be a server to which the user terminal is connected via a network. The server may also be any computer that can be used as a server.

[0014] The validity determination device 1 includes a control unit 10, a storage unit 20, and an input / output interface 30. The control unit 10, the storage unit 20, and the input / output interface 30 are communicatively connected to one another via a bus. The input / output interface 30 is communicatively connected to an input device 40 and an output device 50.

[0015] The control unit 10 controls the validity determination device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU).

[0016] The storage unit 20 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written to and read from at any time with a nonvolatile memory such as read-only memory (ROM), and stores middleware such as an operating system (OS) as well as application programs necessary for executing various control processes according to one embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The storage unit 20 may further include a storage medium that combines a nonvolatile memory such as an SSD that can be written to and read from at any time with a volatile memory such as random access memory (RAM).

[0017] The input / output interface 30 is connected to the input device 40, the output device 50, etc. The input / output interface 30 is an interface that enables transmission and reception of information between the input device 40 and the output device 50. The input / output interface 30 may also include a communication interface. For example, the validity determination device 1 and at least one of the input device 40 or the output device 50 may be wirelessly connected using short-range wireless technology or the like, and information may be transmitted and received using the short-range wireless technology. The communication interface may also include, for example, a communication module that provides a wired or wireless connection to a device or server used by another user via a network.

[0018] The input device 40 includes, for example, a keyboard, a pointing device, etc., which are used by the owner (e.g., a user, etc.) of the validity determination device 1 to input various information including data, etc., to the validity determination device 1. The input device 40 may also include a reader for reading data to be stored in the storage unit 20 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0019] The output device 50 includes a display that displays output data to be presented to the user from the validity determination device 1, a printer that prints the output data, and the like.

[0020] FIG. 2 is a block diagram showing the software configuration of the validity determination device 1 according to the embodiment in relation to the hardware configuration shown in FIG. The control unit 10 includes a data acquisition unit 101, a semantic analyzer 102, an alignment score calculation unit 103, a token-word score correction unit 104, a semantic representation (MR) grammar analyzer 105, a natural language (NL) part-of-speech analyzer 106, an MR validity determination unit 107, and an output control unit 108.

[0021] The data acquisition unit 101 acquires various data from the input device 40 through the input / output interface 30. For example, the data acquisition unit 101 acquires natural language, i.e., NL. For example, NL may be any natural language, such as an utterance, a query, or a question. The data acquisition unit 101 also outputs the acquired NL to the semantic analyzer 102 and the NL part-of-speech analyzer 106.

[0022] The semantic analyzer 102 may be a general, pre-trained one. Therefore, the semantic analyzer 102 has already trained a model, and no further model training is required. When NL is input, the semantic analyzer 102 can output a semantic representation (MR) using a trained model. Here, the MR may be any semantic representation such as Grammar, ASDL, SQL, etc. Here, MR means a semantic representation written in accordance with a structural grammar, in contrast to NL. For example, the MR output from the semantic analyzer 102 may be a program. Then, the semantic analyzer 102 outputs the model used to the alignment score calculation unit 103 and outputs the MR to the MR grammar analyzer 105. A general, pre-trained semantic analyzer 102 outputs text in token units, which are obtained by dividing words into smaller units.

[0023] The alignment score calculation unit 103 calculates an alignment score representing the relationship between the input and output tokens of the NL and MR. For example, the alignment score calculation unit 103 may calculate the alignment score of the input and output tokens of the model using an existing method. The existing method may be, for example, the method disclosed in Reference 1 (Lundberg, Scott M. and Su-In Lee. “A Unified Approach to Interpreting Model Predictions,” NIPS (2017)) or Reference 2 (Chen, Yun et al. “Accurate Word Alignment Induction from Neural Machine Translation,” EMNLP (2020)). For example, Reference 1 discloses that alignment can be calculated from a shapley value, and Reference 2 discloses that alignment can be calculated from the attention weight of a Transformer. The alignment score calculation unit 103 may calculate the alignment score according to, for example, the method described in Reference 1. Then, the alignment score calculation unit 103 outputs the calculated alignment score to the token-word score correction unit 104. The alignment score calculation unit 103 calculates an alignment score for the output of the semantic analyzer 102. Therefore, if the output of the semantic analyzer 102 is in units of tokens, the alignment score calculation unit 103 calculates an alignment score for each token. Since the validity determination in the MR validity determination unit 107, which will be described later, is performed in units of words, the alignment scores calculated in units of tokens need to be corrected to units of words by the token-word score correction unit 104.

[0024] The token-word score correction unit 104 calculates alignment scores between words based on the correspondence between input and output tokens and words. For example, when combining tokens into words, the token-word score correction unit 104 adds up the scores to calculate an alignment score for each word. Then, the token-word score correction unit 104 outputs the alignment score for each word to the MR validity determination unit 107.

[0025] The MR grammar analyzer 105 performs grammar analysis of the MR received from the semantic analyzer 102 according to a general grammar analysis method, and extracts MR content words. Then, the MR grammar analyzer 105 outputs the extracted MR content words to the MR validity determination unit 107.

[0026] The NL part-of-speech analyzer 106 performs part-of-speech analysis of the NL received from the data acquisition unit 101 according to a general NL part-of-speech analysis, and extracts NL content words. Then, the NL part-of-speech analyzer 106 outputs the extracted NL content words to the MR validity determination unit 107.

[0027] The MR validity determination unit 107 determines the validity of the MR based on the received alignment score, MR content words, and NL content words. The method for determining validity will be described in detail later. The MR validity determination unit 107 then outputs the determination result to the output control unit 108.

[0028] The output control unit 108 outputs the determination result to the output device 50 via the input / output interface 30.

[0029] (operation) FIG. 3 is a flowchart showing an example of the operation of the validity determination device 1 to determine MR validity. The control unit 10 of the validity determination device 1 reads out and executes the program stored in the storage unit 20, thereby realizing the operation of this flowchart.

[0030] The operation is initiated when natural language (NL) is received by the input device 40. The NL may be, for example, directly input by a user of the validity determination device 1 to the input device 40, may be an existing data set, or may be generated by an arbitrary program.

[0031] The data acquiring unit 101 acquires NL from the input device 40 through the input / output interface 30 (step ST101). The data acquiring unit 101 outputs the acquired NL to the semantic analyzer 102 and the NL part-of-speech analyzer 106.

[0032] The semantic analyzer 102 generates an MR from an NL (step ST102). The semantic analyzer 102 generates an MR from an NL using a trained model. Then, the semantic analyzer 102 outputs the generated MR to the MR grammar analyzer 105 and outputs the used model to the alignment score calculation unit 103.

[0033] The alignment score calculation unit 103 calculates a token-level alignment score (step ST103). The alignment score calculation unit 103 calculates alignment scores for input and output tokens of the model received from the semantic analyzer 102. For example, the alignment score calculation unit 103 calculates the alignment score from a Shapley value according to the method disclosed in the above-mentioned Reference 1 (hereinafter simply referred to as Shapley). Shapley is a method based on the Shapley value of game theory to explain the output of a model. Here, the Shapley value is, for example, a value that affects the output when a feature participates in or does not participate in the model, and indicates the amount of variation in the predicted value when a certain feature participates from the average predicted value. For example, when calculating using Shapley, the alignment score calculation unit 103 needs to refer to the change in the logit of the token when the input token is masked, in addition to the input and output of NL and MR. The alignment score calculation unit 103 outputs the calculated alignment score to the token-word score correction unit 104.

[0034] The token-word score correction unit 104 calculates an alignment score for each word (step ST104). The token-word score correction unit 104 calculates an alignment score for each word from the correspondence between tokens and words. Here, the correspondence between tokens and words is obtained by comparing each character string from the beginning. For example, when combining tokens into words, the alignment scores are added together to obtain the alignment score for each word. The token-word score correction unit 104 outputs the calculated alignment score for each word to the MR validity determination unit 107.

[0035] FIG. 4 is a diagram showing an example of alignment scores of tokens and words. As shown in Figure 4, if the score of the first token (_Se) for the word sepal is 0.103 and the score of the second token (pal) is 0.048, when combining the first and second tokens into the word (sepal), the alignment score for each word is 0.151, which is the sum of the scores of each token.

[0036] The MR grammar analyzer 105 analyzes the grammar of the MR (step ST105). The MR grammar analyzer 105 analyzes the grammar of the MR according to the grammar analysis method for each MR and extracts MR content words. Here, the method of analyzing the grammar of the MR and extracting MR content words may be a general method, and detailed description thereof will be omitted. Then, the MR grammar analyzer 105 outputs the extracted MR content words to the MR validity determination unit 107.

[0037] The NL part-of-speech analyzer 106 analyzes the parts of speech of the NL (step ST106). The NL part-of-speech analyzer 106 analyzes the parts of speech of the NL and extracts NL content words. Here, the method of analyzing the parts of speech of the NL and extracting NL content words may be a general method, and a detailed description thereof will be omitted. Then, the NL part-of-speech analyzer 106 outputs the extracted NL content words to the MR validity determination unit 107.

[0038] The MR validity determination unit 107 calculates the corrected alignment score s for each content word. w(step ST107). First, the MR validity determination unit 107 extracts content words for each NL and MR based on the received MR content words and NL content words, and extracts an alignment score for the other sequence.

[0039] FIG. 5 is a diagram showing an example of the alignment score to be extracted. In the table of Figure 5(a), row labels indicate some of the content words contained in the NL content words, and column labels indicate some of the content words contained in the MR content words. Each value is an alignment score calculated by the alignment score calculation unit 103. Figure 5(b) shows the alignment scores for each of the NL content words "name," "the," "number," "of," and "species" when the MR content word is "Species." Figure 5(c) shows the alignment scores for each of the MR content words "SELECT," "COUNT," "Species," "FROM," and "table" when the NL content words are "number" and "species."

[0040] Next, the MR validity determination unit 107 calculates a corrected alignment score s for each content word using softmax from the extracted alignment scores. w where w denotes each content word.

[0041] The MR validity determination unit 107 calculates the alignment score s corrected for each NL content word and each MR content word. w The maximum value of s max w (step ST108). For example, the maximum value s max w indicates the correspondence between content words.

[0042] The MR validity determination unit 107 determines the maximum value s max w The minimum value of s min (step ST109). Here, the minimum value s minis calculated by the following formula:

[0043]

number

[0044] Minimum score s min The smaller the value of s, the lower the correspondence between the content word and the most relevant content word. min If is small, it means that information about the corresponding content word is missing.

[0045] The MR validity determination unit 107 determines the minimum score s min The MR validity determination unit 107 determines whether the minimum value s min The validity of MR is judged based on the minimum score s min If it is determined that the score is less than the threshold value, the MR validity determination unit 107 determines that the MR is incorrect, and outputs the alignment score result to the output control unit 108. Then, the process proceeds to step ST111. On the other hand, if it is determined that the score is equal to or greater than the threshold value, the process proceeds to step ST112.

[0046] The output control unit 108 outputs the alignment score result to the output device 50 via the input / output interface 30 (step ST111). The output device 50 that has received the alignment score result may display the alignment score result. The output device 50 may also output the MR output by the semantic analyzer 102 to the output device 50.

[0047] The MR validity determination unit 107 determines that the MR is correct (step ST112). The MR validity determination unit 107 may output information indicating that the MR is correct to the output control unit 108. The output control unit 108 outputs information indicating that the MR is correct to the output device 50 via the input / output interface 30. In addition, the output control unit 108 may output the alignment score calculated by the MR validity determination unit 107 to the output device 50. The output device 50 that receives the information may display the information.

[0048] (Implementation example) In the following, we will explain an example implementation. First, we use WikiSQL as the input dataset. This dataset is a crowd-sourced Semantic Parsing dataset. We also assume the task of inferring the corresponding SQL from a natural language query (NL).

[0049] Next, the model used in the semantic analyzer 102 is t5-base-finetuned-wikiSQL. This model is a publicly available model that has been pre-trained. Furthermore, we will not use this model for new training, but will only perform inference.

[0050] The alignment score calculation unit 103 uses shapley. For example, the alignment score calculation unit 103 calculates the shapley value of the input token to the logit of the output token.

[0051] A sqlparser is used as the MR grammar analyzer 105. Here, the sqlparser used as the MR grammar analyzer 105 may be a general library implemented in, for example, Python, and therefore a detailed description thereof will be omitted here.

[0052] SpaCy is used as the NL part-of-speech analyzer 106. Here, SpaCy used as the NL part-of-speech analyzer 106 may be a general library implemented in, for example, Python, and therefore a detailed description thereof will be omitted here.

[0053] FIG. 6 is a diagram showing an example of SQL estimated from a query and correct SQL. 6(a) shows an example of a query and an example of the analysis result of the NL part-of-speech analyzer 106. As shown in FIG. 6(a), "name the number of species with sepal width of 3.4 and sepal length of 5.4" is input as a query. The NL part-of-speech analyzer 106 also analyzes each part of speech of the query as shown in the table.

[0054] Next, (b) of FIG. 6 shows an example of the estimated SQL, that is, the MR and the analysis result of the MR grammar analyzer 105. Furthermore, (c) of FIG. 6 shows the correct SQL. As shown in (b) of FIG. 6, the estimated SQL is the same as the correct SQL of (c) of FIG. 6. That is, the example shown in FIG. 6 shows that there is a one-to-one correspondence between the content words of NL and SQL. On the other hand, among the analysis results of the MR grammar analyzer 105, it shows that "table" and "length" are misidentified.

[0055] In the present application, the validity determination device 1 performs pipeline processing by combining multiple techniques. Therefore, an error in an earlier stage may affect a later stage. In the example shown in FIG. 6, the MR grammar analyzer 105 overlooks "length" and is unable to evaluate the correspondence from the content words of the SQL. However, the "length" in the analysis result of the NL part-of-speech analyzer 106 can be used to associate the "length" that the MR grammar analyzer 105 overlooked, so this is not a problem and the estimated SQL is the same as the correct SQL.

[0056] FIG. 7 is a diagram showing some of the alignment scores for each content word calculated by the MR validity determination unit 107. As shown in Figure 7, the conversion from number to Count clause has been successful. Furthermore, the example in Figure 7 shows that the correspondence between the input NL and the output SQL has been correctly grasped.

[0057] FIG. 8 is a diagram showing an example of SQL estimated from a query and correct SQL. 8(a) shows an example of a query and an example of the analysis result of the NL part-of-speech analyzer 106. As shown in FIG. 8(a), "What is the hometown of the pitcher whose school was Saint Joseph Regional High School?" is input as a query. The NL part-of-speech analyzer 106 also analyzes each part of speech of the query as shown in the table.

[0058] Next, Fig. 8(b) shows an example of the estimated SQL, that is, the output result and the analysis result of the MR grammar analyzer 105. Furthermore, Fig. 8(c) shows the correct SQL. As shown in Fig. 8(b), the estimated SQL does not match the correct SQL in Fig. 6(c), indicating that the information for "Pitcher" is missing.

[0059] FIG. 9 is a diagram showing some of the alignment scores for each content word calculated by the MR validity determination unit 107. As shown in Figure 9, the information for "Pitcher" is missing. As shown in Figure 9, the alignment score shows the correspondence between NL and SQL. Furthermore, it shows that there is a missing correspondence between the content words in NL and SQL, which leads to incorrect answers.

[0060] (Action and effect) According to the embodiment, the validity determination device 1 does not need to create a dataset, and does not need to design and train a model. Also, the validity determination device 1 can determine the possibility of an answer that is consistent with the behavior of the semantic analysis model. Furthermore, the validity determination device 1 can use a trained model to determine the validity of an output result from the correspondence between input and output.

[0061] [Other embodiments] In the above embodiment, the validity determination device 1 uses a published model as an example, but may use a trained model created by the user himself.

[0062] The techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be distributed by transmitting the program via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that implements this device loads the program stored on the storage medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by controlling the operation of the software means. The term "storage medium" as used herein is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0063] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in combination as appropriate as possible, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements. [Explanation of symbols]

[0064] 1...Validity determination device 10...Control unit 101...Data acquisition section 102...Semantic analyzer 103...Alignment score calculation unit 104...Token-word score correction unit 105...MR grammar analyzer 106...NL part-of-speech analyzer 107...MR validity judgment section 108...Output control unit 20...Storage section 30...Input / output interface 40...Input device 50...Output device

Claims

1. a data acquisition unit for acquiring natural language sentences; a semantic analyzer that performs semantic analysis using the natural language as input and outputs a semantic expression; an alignment score calculation unit that calculates a first alignment score that indicates a relationship between input and output tokens of the natural sentence and the semantic expression; a score correction unit that calculates a second alignment score for each word based on the relationship between the tokens and words included in the input / output tokens; a part-of-speech analyzer that extracts natural sentence content words by performing a part-of-speech analysis on the natural sentence; a grammar analyzer that performs grammar analysis on the semantic representation to extract semantic representation content words; a validity determination unit that determines whether the semantic expression is valid based on the second alignment score, the natural text content words, and the semantic expression content words; A validity determination device comprising:

2. 2. The validity determination device according to claim 1, wherein the validity determination unit extracts a third alignment score with the other series from the second alignment score for each of the natural sentence content words and the semantic content words, and calculates a corrected alignment score for each content word based on the third alignment score.

3. The validity determination device according to claim 2 , wherein the validity determination unit calculates a maximum value of the corrected alignment score for each of the natural sentence content words and the semantic content words.

4. The validity determination device according to claim 3 , wherein the validity determination unit calculates a minimum value of the maximum values ​​and determines whether the minimum value is less than a threshold value.

5. The validity determination device according to claim 4 , further comprising an output control unit that outputs the third alignment score when the validity determination unit determines that the minimum value is less than a threshold value.

6. The validity determination device according to claim 4 , wherein when it is determined that the minimum value is equal to or greater than a threshold value, the validity determination unit determines that the semantic expression is valid.

7. Acquiring natural language outputting a semantic expression by performing a semantic analysis using the natural language as an input; calculating a first alignment score representing a relationship between input and output tokens of the natural sentence and the semantic representation; calculating a second alignment score for each word from the relationships between the tokens and words included in the input / output tokens; extracting natural sentence content words by performing part-of-speech analysis on the natural sentence; extracting semantic representation content words by performing grammatical analysis on the semantic representation; determining whether the semantic expression is valid based on the second alignment score, the natural sentence content words, and the semantic expression content words; A validity determination method comprising:

8. A validity determination program that causes a processor to function as each unit of the validity determination device according to claim 1 .

Images