A cross-domain semantic analysis benchmark method, device, equipment and medium

By evaluating semantic analysis algorithms on cross-domain multilingual and multi-semantic representation data sets, the limitations of the evaluation methods in the existing technology are solved, and a comprehensive and fair evaluation of the performance of semantic analysis algorithms is achieved, which improves the effectiveness of the evaluation. .

CN113947089BActive Publication Date: 2024-11-01GUANGZHOU CIPHER CAMP INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111082692.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-15
Publication Date
2024-11-01
Estimated Expiration
2041-09-15

AI Technical Summary

Technical Problem

The existing semantic analysis algorithm evaluation methods lack comprehensive performance evaluation of cross-domain, multilingual and multi-semantic representations, making it difficult to make objective and impartial comparisons. The evaluation of new algorithms is often limited to specific data sets and semantic representations, which affects cross-domain Assessment of domain and cross-language competence.

Method used

A cross-domain semantic analysis benchmark method is proposed. By evaluating the preset baseline model and target model on a cross-domain multilingual and multisemantic representation dataset, combining pointers to generate network model, Glove word vector and fasttext multilingual model , configure the LSTM network and hyperparameters to compute the performance matrix to comprehensively evaluate the performance of the target model.

Benefits of technology

A comprehensive performance evaluation of the model prediction accuracy, cross-domain capabilities and cross-language capabilities of semantic analysis algorithms is realized, providing valuable references to cross-language and cross-domain models in the field of natural language semantic analysis, and improving the comprehensiveness of the evaluation and Validity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113947089B_ABST
    Figure CN113947089B_ABST
Patent Text Reader

Abstract

The present invention discloses a cross-domain semantic analysis benchmark method, device, equipment and medium, the method comprising: running a preset baseline model on a target data set to obtain a first performance of the semantic analysis task of the baseline model on the target data set; wherein the target data set adopts a cross-domain multi-language and multi-semantic representation; running a target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set; and determining a comprehensive evaluation result of the comprehensive performance of the target model on the target data set based on the first performance of the baseline model and the second performance of the target model. The present invention avoids the limitations brought by the semantic analysis task from a specific language to a specific semantic representation on a single domain data set, effectively improves the comprehensiveness and effectiveness of the evaluation of semantic analysis algorithms, and can be widely used in the field of semantic analysis technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of semantic analysis technology, and in particular to a cross-domain semantic analysis benchmark method, device, equipment and medium. Background Art

[0002] For a given semantic analysis algorithm, the semantic analysis benchmark method can evaluate the performance of the target algorithm, such as the accuracy of model prediction, cross-domain capability, cross-language capability, etc. In recent years, with the vigorous development of research in the field of natural language processing, more and more new semantic analysis algorithms have been born. These algorithms often claim to be able to achieve 90% or even higher accuracy in certain experiments. However, in fact, these results are only achieved on certain specific data sets or specific natural language and semantic representation combinations. Therefore, there is a lack of comparison between different algorithms, and it is difficult to have a comprehensive and objective evaluation of these algorithms. How to provide a set of evaluation methods that are applicable to most new algorithms and provide sufficient reference value for performance comparison has become an urgent problem to be solved in current research. The semantic analysis benchmark method can meet the above requirements.

[0003] Semantic analysis is an important research direction in the field of natural language processing. Semantic analysis focuses on how to convert natural language into machine-readable and executable logical statements, namely semantic representation (MR), such as SQL, Prolog, Python, etc. Semantic analysis algorithms are a general term for algorithms that can convert natural language into corresponding machine-readable logical expressions. The current performance evaluation method for semantic analysis algorithms is mainly to use new algorithms to train models on certain specific data sets, run the trained models on the test sets, and evaluate the performance of the model by calculating the prediction accuracy of the model. It is generally believed that the higher the accuracy, the better the performance of the target algorithm.

[0004] Although there are many datasets in different fields, languages ​​and with different semantic representations, there are still big problems in the evaluation of semantic analysis algorithms. First, although previous studies have provided versions of the above datasets in different languages ​​and with different semantic representations, these versions lack summary, resulting in that in actual tests, researchers often only use the mapping from a specific language to a specific semantic representation of a dataset as a carrier for performance evaluation. Obviously, this makes cross-domain, multi-language and multi-semantic representation data fail to be fully utilized. The evaluation of semantic analysis algorithms is actually no different from the previous tests of a single dataset, a single language and a single representation, and the cross-domain and cross-language performance of semantic analysis algorithms cannot be evaluated. Not only that, the above phenomenon may also lead to unfair evaluation of algorithms. The publishers of new algorithms often only publish the performance of the algorithm on a specific dataset. This may be because the algorithm meets certain specific conditions on the dataset and thus achieves particularly good results. Therefore, it is difficult to compare the performance of two algorithms using different testing methods. For example, some datasets use random partitioning in the number division of training sets and test sets, which makes the training sets and test sets conform to the same distribution, and the test results will be much higher than the performance on datasets that divide the training sets and test sets according to different databases. Furthermore, even if different semantic representation methods are used as output on the same dataset, the accuracy of the algorithm on the dataset will be significantly affected. For example, the semantic representation ability of Lambda statements is significantly higher than that of SQL, so using the Lambda version of the GeoQuery dataset to test the algorithm will result in an accuracy rate that is about 10% higher than that of the SQL version. All of these make it difficult to objectively evaluate new algorithms. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a cross-domain semantic analysis benchmark method, apparatus, device and medium.

[0006] One aspect of the present invention provides a cross-domain semantic analysis benchmark method, comprising:

[0007] Running a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set uses a cross-domain multi-language and multi-semantic representation;

[0008] Running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set;

[0009] According to the first performance of the baseline model and the second performance of the target model, a comprehensive evaluation result of the comprehensive performance of the target model on the target data set is determined.

[0010] Optionally, the baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism and a coverage mechanism on the basis of a seq2seq model; the target data set includes a first data set and a second data set.

[0011] Optionally, running a preset baseline model on a target data set to obtain a first performance of a semantic analysis task of the baseline model on the target data set includes:

[0012] The first word vector pre-training model using Glove 840B as English input;

[0013] Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English;

[0014] Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate;

[0015] Configure batch size and anti-overfitting parameters;

[0016] According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters;

[0017] Obtaining optimal hyperparameters according to the training results of the baseline model;

[0018] Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set;

[0019] A performance matrix of the best model is calculated, and a first performance of the baseline model is determined.

[0020] Optionally, running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set includes:

[0021] When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set;

[0022] When the natural language set is not empty, removing a candidate natural language from the natural language set;

[0023] When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set;

[0024] Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model;

[0025] Obtaining the test accuracy of the best model of the target model on the test set;

[0026] All test results are counted and the second performance of the target model is calculated.

[0027] Optionally, determining a comprehensive evaluation result of the comprehensive performance of the target model on the target data set according to the first performance of the baseline model and the second performance of the target model;

[0028] Obtain a first performance matrix of the first performance and a second performance matrix of the second performance;

[0029] If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model;

[0030] If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model;

[0031] If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model;

[0032] If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

[0033] The embodiment of the present invention also provides a cross-domain semantic analysis benchmark device, including:

[0034] The first module is used to run a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set adopts a cross-domain multi-language and multi-semantic representation;

[0035] A second module is used to run the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set;

[0036] The third module is used to determine a comprehensive evaluation result of the comprehensive performance of the target model on the target data set based on the first performance of the baseline model and the second performance of the target model.

[0037] Optionally, the baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism and a coverage mechanism on the basis of a seq2seq model; the target data set includes a first data set and a second data set.

[0038] Optionally, the first module is specifically used to:

[0039] The first word vector pre-training model using Glove 840B as English input;

[0040] Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English;

[0041] Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate;

[0042] Configure batch size and anti-overfitting parameters;

[0043] According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters;

[0044] Obtaining optimal hyperparameters according to the training results of the baseline model;

[0045] Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set;

[0046] Calculating a performance matrix of the best model to determine a first performance of the baseline model;

[0047] The second module is specifically used for:

[0048] When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set;

[0049] When the natural language set is not empty, removing a candidate natural language from the natural language set;

[0050] When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set;

[0051] Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model;

[0052] Obtaining the test accuracy of the best model of the target model on the test set;

[0053] Counting all test results and calculating the second performance of the target model;

[0054] The third module is specifically used for:

[0055] Obtain a first performance matrix of the first performance and a second performance matrix of the second performance;

[0056] If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model;

[0057] If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model;

[0058] If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model;

[0059] If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

[0060] An embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0061] The memory is used to store programs;

[0062] The processor executes the program to implement the method described above.

[0063] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0064] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0065] The embodiment of the present invention runs a preset baseline model on a target data set to obtain a first performance of the baseline model on the semantic analysis task of the target data set; wherein the target data set adopts a cross-domain multi-language and multi-semantic representation; runs the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set; and determines a comprehensive evaluation result of the comprehensive performance of the target model on the target data set based on the first performance of the baseline model and the second performance of the target model. The present invention can perform a performance evaluation on the target algorithm including indicators such as model prediction accuracy, cross-domain capability, cross-language capability, etc., and can provide a valuable reference for the research on cross-language and cross-domain models in the field of natural language semantic analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 is a flowchart of the overall steps of an embodiment of the present invention;

[0068] Figure 2 A schematic diagram of the structure of a baseline model provided by an embodiment of the present invention;

[0069] Figure 3 This is a diagram of the implementation process of steps S1 and S2 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] In view of the problems existing in the prior art, the present invention proposes a cross-domain semantic analysis benchmark method, including:

[0072] Running a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set uses a cross-domain multi-language and multi-semantic representation;

[0073] Running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set;

[0074] According to the first performance of the baseline model and the second performance of the target model, a comprehensive evaluation result of the comprehensive performance of the target model on the target data set is determined.

[0075] Optionally, the baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism and a coverage mechanism on the basis of a seq2seq model; the target data set includes a first data set and a second data set.

[0076] Optionally, running a preset baseline model on a target data set to obtain a first performance of a semantic analysis task of the baseline model on the target data set includes:

[0077] The first word vector pre-training model using Glove 840B as English input;

[0078] Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English;

[0079] Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate;

[0080] Configure batch size and anti-overfitting parameters;

[0081] According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters;

[0082] Obtaining optimal hyperparameters according to the training results of the baseline model;

[0083] Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set;

[0084] A performance matrix of the best model is calculated, and a first performance of the baseline model is determined.

[0085] Optionally, running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set includes:

[0086] When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set;

[0087] When the natural language set is not empty, removing a candidate natural language from the natural language set;

[0088] When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set;

[0089] Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model;

[0090] Obtaining the test accuracy of the best model of the target model on the test set;

[0091] All test results are counted and the second performance of the target model is calculated.

[0092] Optionally, determining a comprehensive evaluation result of the comprehensive performance of the target model on the target data set according to the first performance of the baseline model and the second performance of the target model;

[0093] Obtain a first performance matrix of the first performance and a second performance matrix of the second performance;

[0094] If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model;

[0095] If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model;

[0096] If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model;

[0097] If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

[0098] The embodiment of the present invention also provides a cross-domain semantic analysis benchmark device, including:

[0099] The first module is used to run a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set adopts a cross-domain multi-language and multi-semantic representation;

[0100] A second module is used to run the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set;

[0101] The third module is used to determine a comprehensive evaluation result of the comprehensive performance of the target model on the target data set based on the first performance of the baseline model and the second performance of the target model.

[0102] Optionally, the baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism and a coverage mechanism on the basis of a seq2seq model; the target data set includes a first data set and a second data set.

[0103] Optionally, the first module is specifically used to:

[0104] The first word vector pre-training model using Glove 840B as English input;

[0105] Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English;

[0106] Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate;

[0107] Configure batch size and anti-overfitting parameters;

[0108] According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters;

[0109] Obtaining optimal hyperparameters according to the training results of the baseline model;

[0110] Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set;

[0111] Calculating a performance matrix of the best model to determine a first performance of the baseline model;

[0112] The second module is specifically used for:

[0113] When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set;

[0114] When the natural language set is not empty, removing a candidate natural language from the natural language set;

[0115] When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set;

[0116] Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model;

[0117] Obtaining the test accuracy of the best model of the target model on the test set;

[0118] Counting all test results and calculating the second performance of the target model;

[0119] The third module is specifically used for:

[0120] Obtain a first performance matrix of the first performance and a second performance matrix of the second performance;

[0121] If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model;

[0122] If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model;

[0123] If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model;

[0124] If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

[0125] An embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0126] The memory is used to store programs;

[0127] The processor executes the program to implement the method described above.

[0128] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0129] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0130] The specific implementation principle of the present invention is described in detail below in conjunction with the accompanying drawings:

[0131] like Figure 1 As shown, the technical solution adopted by the present invention includes the following steps:

[0132] S1. Run the given baseline model on a specific cross-domain multi-language and multi-semantic representation dataset to obtain the performance of the baseline model in semantic analysis tasks on datasets in different domains, languages, and semantic representations.

[0133] S2. Run the target model on the same cross-domain multi-language and multi-semantic representation dataset to obtain the performance of the target model in semantic analysis tasks on datasets in different domains, languages, and semantic representations.

[0134] S3. Based on the experimental results of the baseline model and the target model, a comprehensive evaluation is conducted on the comprehensive performance of the target model’s cross-domain, cross-language, and cross-semantic representation.

[0135] First, give clear definitions of the specific terms mentioned in the above steps:

[0136] In step S1, the given baseline model is run on a specific cross-domain multi-language and multi-semantic representation dataset. The “given baseline model” refers to the pointer generation network model. This model adds attention mechanism, copy mechanism and coverage mechanism to the basic seq2seq model, so that the model can achieve the current first-class accuracy in the benchmark experiment. Its structure is as follows Figure 2 shown.

[0137] The basic seq2seq model mainly consists of two parts: encoder and decoder. Both encoder and decoder are essentially bidirectional long short-term memory neural networks. The encoder is responsible for processing the input sequence into an intermediate vector of a specified length. The vector is also called the intermediate layer state and can be regarded as the semantics of the input sequence. This process is called encoding. The decoder is responsible for generating a sequence of a specified representation based on the semantic vector. This process is called decoding. Through the above "encoding-decoding" method, the basic seq2seq model can complete basic semantic analysis tasks, that is, converting the input sequence of natural language into a logical representation with the same semantics under different semantic representations.

[0138] The attention mechanism is a mechanism proposed to imitate human attention. It enables the model to quickly filter out high-value information from a large amount of information. It is used to solve the problem that it is difficult to obtain a reasonable final vector representation when the input sequence of the LSTM / RNN model is long. The attention mechanism is implemented in the decoding process. It changes the shortcoming of the traditional decoder that each input is assigned the same vector. Instead, it assigns different weights to different input words. During the encoding process, what is generated is no longer an intermediate semantic of a fixed length, but a sequence of vectors of different lengths. The decoder will further process this sequence subset. For example, for a semantic analysis task, suppose the input sentence is: input_tok1, input_tok2...input_tok i ...input_tok n Assume the output statement is: output_tok1, output_tok2...output_tok j ...output_tok m If the attention mechanism is not added, then as the tokens in the input are input into the model one by one, when input to input_tok i When the model considers all the previously input tokens, that is, input_tok1, input_tok2...input_tok i , and each token will have an equal impact on the calculation of the current output, which will reduce the accuracy of the semantic analysis task, because when translating input_tok i , this token should contribute more to the calculation of the output than other tokens. The attention mechanism will jointly calculate all past inputs at each input step, and assign a probability to each token through the softmax function, so that the model's attention is more focused on the current input. The calculation process of the seq2seq model using the attention mechanism is as follows:

[0139]

[0140] a t =softmax(e t )

[0141]

[0142] Among them, h i is the i-th hidden state of the encoder, s t is the state of the decoder at time t, W h ,W s ,battn are all learning parameters. The model calculates the probability of generating a word from the vocabulary as output, and the calculation process is as follows:

[0143]

[0144] P(w)=P vocab

[0145] And according to the above probability, the loss function can be defined:

[0146]

[0147]

[0148] The copy mechanism is mainly used to solve the OOV (out of vocabulary) problem. When a word that is not in the encoder vocabulary appears in the input, the seq2seq model often finds it difficult to calculate what output needs to be generated under the current input. However, in fact, this type of OOV words are often proper nouns, such as place names, personal names, etc. These OOV words do not need to be translated and can be simply copied to the output content. The copy mechanism implements the above function, allowing the model to not only calculate and generate text through the input, but also directly reuse the original words in the input. gen It is a real number in the range [0,1], which is used to represent the probability of generating a word at the current moment or extracting an original word from the original text.

[0149]

[0150] Among them, x t is the decoder input at time t, s t is the hidden layer state of the decoder at time t

[0151] Combining the above probability formula with the attention mechanism, we can get the new output word probability P(w)

[0152]

[0153] The first part calculates the probability of generating an output from the encoder table, and the second part calculates the probability of extracting text from the original text.

[0154] The coverage mechanism mainly solves the problem that the alignment model cannot record historical states. Specifically, in the translation process under the "seq2seq+attention" framework, the "attention" of translating the current word is independent of the "attention" of translating the previous word. The current operation cannot obtain alignment-related information from the previous translation, which leads to the problems of "over-translation" and "missing translation". In terms of specific implementation, the coverage mechanism not only considers the hidden layer state at the current moment when calculating attention, but also considers the state generated previously, introducing contextvector:

[0155]

[0156] At the same time, the formula for calculating the weight matrix is ​​modified:

[0157]

[0158] Define coverage loss and add it to the loss function calculation:

[0159]

[0160] After the Coverage mechanism is introduced, the pointer generation network avoids repetition at the same location and thus repeatedly generates text.

[0161] In steps S1 and S2, the given baseline model is run on a specific cross-domain multilingual and multi-semantic representation dataset. The “specific cross-domain multilingual and multi-semantic representation dataset” consists of two datasets, including the MultilingualGeoQuery dataset and the Multilingual ATIS dataset.

[0162] GeoQuery is a dataset about querying the US geographic database. The embodiment of the present invention collects different language translations and different semantic representations of the dataset by scholars in the past twenty years, and summarizes them into a semantic analysis dataset Multilingual GeoQuery dataset with cross-language and cross-semantic representation. The Multilingual GeoQuery dataset consists of three parts: training set, development set and test set, with 548, 49 and 277 data respectively. Each data includes an input consisting of a natural language statement of a geographic query and an output with a logical representation of the same semantics. The dataset provides input in eight languages: English, Chinese, Thai, German, Greek, Indonesian, Swedish and Persian, and the output includes four semantic representations: SQL, FunQL, Prolog and Lambda. ATIS is a dataset about flight queries. The present invention collects different language versions of the dataset and summarizes them into a semantic analysis dataset Multilingual ATIS dataset with cross-language and cross-semantic representation. The Multilingual ATIS dataset consists of three parts: training set, development set and test set, with 2871, 317 and 314 data respectively. Each data includes input consisting of natural language sentences of flight query and logical expression output with the same semantics. The dataset provides input in nine languages ​​including English, Chinese, German, Spanish, Indonesian, Turkish and Japanese, and uses SQL, Lambda and intent recognition-slot filling statements as output. The data of Multilingual GeoQuery and Multiligual ATIS come from different fields, which can also effectively examine the cross-domain capabilities of the algorithm.

[0163] The various steps of the technical solution adopted by the present invention are described in detail below:

[0164] Each step of the technical solution adopted by the present invention is as follows Figure 1 Specifically, the processes of S1 and S2 are as follows: Figure 3 shown.

[0165] In step S1, the baseline model, i.e., the pointer generation network model, is run on the multilingual GeoQuery and multilingual ATIS datasets. First, the baseline model is trained on all tasks with different languages ​​and different semantic representation combinations in the training sets of the above datasets. For any dataset D in multilingual GeoQuery and multilingual ATIS, k ,remember For the dataset D k The set of all natural languages, is a set of semantic representations. For any language and Pairing <L i ,MR j >, the baseline model is based on L i Language sentences are input, MR j The training set is used for training on tasks with formal semantic representation as output, with a total of |L|×|MR| tasks. The specific training process is as follows: In terms of word embedding, Glove 840B is used as the word vector pre-training model for English input, and the fasttext multilingual model word vector pre-training model is used for other languages. In terms of hyperparameters, the hyperparameter list is set as follows: the number of layers of the codec LSTM network layer={1,2}, the number of neurons in the LSTM network rnn_size={32,64,128,256,512}, and the learning rate learning_rate={0.0005,0.001,0.005,0.01}. Batch processing is used for training, and the batch size batch_size={16,32,64,128}. At the same time, the model uses the dropout parameter to prevent overfitting, dropout={0.1,0.2,0.3,0.4,0.5}. The early stop mechanism is set during training, and the development set of the dataset is set to the dev parameter. After completing the above training process, the models of all parameter combinations obtained by training are run on <L i ,MR j >On the test set of the task, take the set of parameters with the highest accuracy on the task as the best hyperparameters for the task And get the best model of the baseline model on this task After training is complete, the performance matrix of the baseline model is calculated. The best model obtained exist <L i ,MR j >The accuracy achieved on the test set of the task is recorded as Then after traversing |L|×|MR| tasks, we get the performance matrix consisting of the accuracy of all tasks:

[0166]

[0167] In step S2, the target model is trained on the multilingual GeoQuery and multilingual ATIS datasets. Similarly, for any dataset D in multilingual GeoQuery and multilingual ATIS, k ,remember For the dataset D kThe set of all natural languages, is a set of semantic representations. For any language and Pairing <L i ,MR j >, the baseline model is based on L i Language sentences are input, MR j The training set is used for the tasks with formal semantic representation as output, and there are |L|×|MR| tasks in total. The models with all parameter combinations obtained by training are run on <L i ,MR j >On the test set of the task, take the set of parameters with the highest accuracy on the task as the best hyperparameters for the task And get the best model of the baseline model on this task After training is complete, the performance matrix of the baseline model is calculated. The best model obtained exist <L i ,MR j >The accuracy achieved on the test set of the task is recorded as Then after traversing |L|×|MR| tasks, we get the performance matrix consisting of the accuracy of all tasks:

[0168]

[0169] In step S3, according to the performance matrix of the baseline model and the target model and The comprehensive performance of the target model's cross-domain, cross-language, and cross-semantic representation is comprehensively evaluated.

[0170]

[0171] Evaluate the target algorithm:

[0172] If there is L i Make In D k The target algorithm is valid on the L i The language performance is insufficient.

[0173] If MR is present i Make In D k The target algorithm is established on the MR dataset. i Insufficient performance in semantic representation.

[0174] If there is D i and D j Make In D i and D jIf there are significant differences in the data sets, the algorithm does not have good cross-domain capabilities.

[0175] If there are no or only a few languages ​​that meet the above conditions, i , semantic representation MR j and dataset D k , the target algorithm has excellent performance and can achieve outstanding performance in semantic analysis tasks with cross-domain, multi-language and multi-semantic representations, and is currently a first-class semantic analysis algorithm.

[0176] Next, the embodiment of the present invention evaluates the basic seq2seq model through a semantic analysis benchmark method across multiple domains, languages, and multiple semantic representations.

[0177] According to step S1, the baseline model is run on the multilingual GeoQuery and multilingual ATIS datasets, respectively. First, the pointer generation network baseline model is trained on the multilingual GeoQuery dataset. The natural language input of multilingual GeoQuery includes eight languages, namely Chinese, English, Thai, German, Greek, Indonesian, Swedish and Persian, and the logical representation input of multilingual GeoQuery includes four forms, namely SQL, Prolog, FunQL and Lambda. The baseline model is trained on the above |L|×|MR|=32 tasks on the training set. In terms of word embedding, Glove 840B is used as the word vector pre-training model for English input, and the fasttext multilingual model word vector pre-training model is used for other languages. In terms of hyperparameters, the hyperparameter list is set as follows: the number of layers of the codec LSTM network layer = {1,2}, the number of neurons in the LSTM network rnn_size = {32,64,128,256,512}, and the learning rate learning_rate = {0.0005,0.001,0.005,0.01}. Batch processing is used for training, and the batch size batch_size = {16,32,64,128}. At the same time, the model uses the dropout parameter to prevent overfitting, dropout = {0.1,0.2,0.3,0.4,0.5}. The early stop mechanism is set during training, and the development set of the dataset is set to the dev parameter. After completing the above training process, all parameter combinations are tested on the test set, and the set of parameters with the highest accuracy on the task is taken as the best hyperparameters for the task. And get the best model of the baseline model on this task The accuracy of the model in the above |L|×|MR|=32 tasks is statistically analyzed, as shown in Table 1. Table 1 shows the experimental results of the baseline model on the Multilingual GeoQuery dataset:

[0178] Table 1

[0179]

[0180] Therefore, the performance matrix of the baseline model on the multilingual geoquery dataset is calculated as:

[0181]

[0182] Next, the pointer generation network baseline model is trained on the multilingual ATIS dataset. The natural language input of multilingual GeoQuery includes nine languages, namely Chinese, English, Japanese, German, Spanish, Indonesian, French, Hindi and Turkish. The logical representation input of multilingual GeoQuery includes three forms, namely SQL, Lambda and intent recognition-slot filling statements. The baseline model is trained on the above |L|×|MR|=27 tasks on the training set. In terms of word embedding, the hyperparameters are set to the same as the training process on the multilingual GeoQuery dataset. The best model for the corresponding task is obtained during training. Finally, the accuracy of the model in the above |L|×|MR|=27 tasks is counted, as shown in Table 2. Table 2 shows the experimental results of the baseline model on the Multilingual ATIS dataset:

[0183] Table 2

[0184]

[0185] Therefore, the performance matrix of the baseline model on the multilingual ATIS dataset is calculated as

[0186]

[0187] According to step S2, the target model is run on the multilingual GeoQuery and multilingual ATIS datasets respectively. First, the seq2seq target model is trained on the multilingual GeoQuery dataset. Since the seq2seq model also belongs to the generative model of the encoding-decoding structure, the training process exactly the same as in step S1 can be applied, which will not be repeated here. After training the baseline model for the above |L|×|MR|=27 tasks on the training set, the best model for the corresponding task is obtained. The accuracy of the models in all the above tasks is statistically analyzed, as shown in Table 3. Table 3 shows the experimental results of the target model on the Multilingual GeoQuery dataset:

[0188] Table 3

[0189]

[0190] Therefore, the performance matrix of the target model on the multilingual geoquery dataset is calculated as

[0191]

[0192] Next, we train the seq2seq target model on the multilingual ATIS dataset, applying exactly the same training process as in step S1, and train the best model for the corresponding task in the above |L|×|MR|=27 tasks. The accuracy of the statistical model is shown in Table 4, which shows the experimental results of the target model on the Multilingual ATIS dataset:

[0193] Table 4

[0194]

[0195] The calculated performance matrix of the target model on the multilingual ATIS dataset is

[0196]

[0197] According to step S3, the performance matrix of the baseline model and the target model obtained in steps S1 and S2 is used for comparison, so as to effectively evaluate the target algorithm. By comparing the performance matrix of the target algorithm and the baseline model, it can be found that the overall accuracy of seq2seq in the experiment is about 10%-20% lower than that of the baseline model. It can be concluded that the performance of the seq2seq basic model is insufficient and has a large difference from the first-class level.

[0198] In summary, the present invention proposes a semantic analysis benchmark method with cross-domain, multi-language and multi-semantic representation. For any given semantic analysis algorithm, the present semantic analysis benchmark method can perform performance evaluation on the target algorithm including indicators such as model prediction accuracy, cross-domain capability, cross-language capability and so on. At the same time, the benchmark method can provide valuable reference for the research on cross-language and cross-domain models in the field of natural language semantic analysis.

[0199] The method of the present invention is based on a pointer generation network model as a strong baseline algorithm, and obtains benchmark experimental results on semantic analysis datasets in multiple different fields, with multiple languages ​​as input and multiple semantic representations as output. By comprehensively comparing the above experimental results with the performance of the target semantic analysis algorithm on the same dataset, the various performances of the target algorithm are effectively evaluated. The present invention comprehensively considers datasets from different fields, multiple languages ​​and multiple semantic representation methods, avoids the limitations of semantic analysis tasks from specific languages ​​to specific semantic representations on a single field dataset, and effectively improves the comprehensiveness and effectiveness of the evaluation of semantic analysis algorithms.

[0200] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0201] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0202] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0203] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0204] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0205] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0206] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0207] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0208] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A cross-domain semantic analysis benchmark method, characterized in that: include: Running a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set uses a cross-domain multi-language and multi-semantic representation; Running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set; Determining a comprehensive performance evaluation result of the target model on the target data set according to the first performance of the baseline model and the second performance of the target model; The baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism, and a coverage mechanism on the basis of the seq2seq model; the target data set includes a first data set and a second data set; The step of running a preset baseline model on a target data set to obtain a first performance of the baseline model in a semantic analysis task on the target data set includes: The first word vector pre-training model using Glove 840B as English input; Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English; Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate; Configure batch size and anti-overfitting parameters; According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters; Obtaining optimal hyperparameters according to the training results of the baseline model; Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set; Calculating a performance matrix of the best model to determine a first performance of the baseline model; The step of running the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set includes: When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set; When the natural language set is not empty, removing a candidate natural language from the natural language set; When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set; Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model; Obtaining the test accuracy of the best model of the target model on the test set; All test results are counted and the second performance of the target model is calculated.

2. A cross-domain semantic analysis benchmark method according to claim 1, characterized in that: Determining a comprehensive performance evaluation result of the target model on the target data set based on the first performance of the baseline model and the second performance of the target model; Obtain a first performance matrix of the first performance and a second performance matrix of the second performance; If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model; If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model; If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model; If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

3. A cross-domain semantic analysis benchmark device, characterized in that: include: The first module is used to run a preset baseline model on a target data set to obtain a first performance of the baseline model on a semantic analysis task on the target data set; wherein the target data set adopts a cross-domain multi-language and multi-semantic representation; A second module is used to run the target model on the target data set to obtain a second performance of the semantic analysis task of the target model on the target data set; A third module is used to determine a comprehensive performance evaluation result of the target model on the target data set according to the first performance of the baseline model and the second performance of the target model; The baseline model is a pointer generation network model, and the pointer generation network model adds an attention mechanism, a copy mechanism, and a coverage mechanism on the basis of the seq2seq model; the target data set includes a first data set and a second data set; The first module is specifically used for: The first word vector pre-training model using Glove 840B as English input; Use the fasttext multilingual model as the second word vector pre-training model for input languages ​​other than English; Configure the number of codec LSTM network layers, the number of LSTM network neurons, and the learning rate; Configure batch size and anti-overfitting parameters; According to the first word vector pre-training model and the word vector pre-training model, the baseline model is trained by the number of codec LSTM network layers, the number of LSTM network neurons, the learning rate, the batch size and the anti-overfitting parameters; Obtaining optimal hyperparameters according to the training results of the baseline model; Determine, according to the optimal hyperparameters, the best model of the baseline model for the semantic analysis task on the target data set; Calculating a performance matrix of the best model to determine a first performance of the baseline model; The second module is specifically used for: When the target data set is not empty, selecting a candidate data set from the target data set, and removing a natural language set from the candidate data set; When the natural language set is not empty, removing a candidate natural language from the natural language set; When the semantic representation set in the target data set is not empty, removing a candidate semantic representation from the semantic representation set; Training the target model according to the candidate natural language and the candidate semantic representation to obtain an optimal model of the target model; Obtaining the test accuracy of the best model of the target model on the test set; All test results are counted and the second performance of the target model is calculated.

4. The cross-domain semantic analysis benchmark device according to claim 3, characterized in that: The third module is specifically used for: Obtain a first performance matrix of the first performance and a second performance matrix of the second performance; If there is a natural language that allows the first performance matrix and the second performance matrix to be established on the target data set, then generating a first performance determination result of the target model; If there is a semantic representation that enables the first performance matrix and the second performance matrix to be established on the target data, then generating a second performance determination result of the target model; If there is a first data set and a second data set such that the second performance matrix has a performance difference between the first data set and the second data set, generating a third performance determination result of the target model; If the target model does not have the first performance determination result, the second performance determination result or the third performance determination result, a fourth performance determination result of the target model is generated, and the fourth performance determination result indicates that the target model has excellent performance.

5. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to claim 1 or 2.

6. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to claim 1 or 2.