Semantic analysis model evaluation method

Abstract

The application discloses a semantic analysis model evaluation method, which comprises the following steps: testing the basic function of the semantic analysis model; testing the generalization ability of the semantic analysis model; testing the reverse recognition ability of the semantic analysis model; and obtaining a comprehensive score by comprehensively scoring the above steps. The semantic analysis model evaluation method provided by the application can evaluate the ability of the semantic analysis model from multiple angles, and the evaluation result is more accurate and comprehensive.

Description

Technical Field

[0001] This invention relates to a method for evaluating semantic analysis models. Background Technology

[0002] With the development of e-commerce, intelligent customer service is being used more and more. A good intelligent customer service can accurately identify the user's intent and thus provide targeted responses. The core of intelligent customer service lies in the performance of its semantic recognition model.

[0003] Current methods for evaluating semantic analysis models are relatively simplistic, primarily relying on a single metric from the test set, such as accuracy, precision, or F1 score. Therefore, even if a semantic analysis model performs well on existing test sets, it doesn't guarantee good performance in real-world online scenarios. Summary of the Invention

[0004] This invention provides a semantic analysis model evaluation method, which adopts the following technical solution:

[0005] A semantic analysis model evaluation method, comprising:

[0006] Testing the basic functionality of the semantic analysis model, specifically including:

[0007] Examine the model vocabulary of the semantic analysis model;

[0008] The semantic analysis model is tested using a simplified test set to determine its analysis results on the simplified test set.

[0009] The first score is obtained by scoring the above test results;

[0010] Testing the generalization ability of the semantic analysis model specifically includes:

[0011] The rewritten corpus is obtained by rewriting the corpus that can be accurately identified by several semantic analysis models without affecting the semantics of the sentences.

[0012] The rewritten corpus is then re-inputted into the semantic analysis model to determine whether it can be correctly identified.

[0013] The second score is obtained by scoring the above test results;

[0014] Testing the reverse recognition capability of the semantic analysis model specifically includes:

[0015] Reverse corpus is obtained by rewriting positive corpus that can be accurately identified by several semantic analysis models.

[0016] The rewritten reverse corpus is then re-input into the semantic analysis model to determine whether it can be correctly identified.

[0017] The test results above are scored to obtain a third score;

[0018] A comprehensive score is obtained by combining the scoring results from the above steps.

[0019] Furthermore, the specific method for examining the model vocabulary of the semantic analysis model is as follows:

[0020] The semantic analysis model's vocabulary is compared with the industry keyword database to determine whether the model's vocabulary is complete.

[0021] Furthermore, the industry keyword database was obtained through the following specific methods:

[0022] Obtain dialogue data from various industries;

[0023] The dialogue corpus is fed into an attention mechanism model to automatically identify keywords.

[0024] All extracted keywords were compiled into an industry keyword library.

[0025] Furthermore, the specific method for testing the semantic analysis model on a simplified test set and judging the analysis results of the semantic analysis model on the simplified test set is as follows:

[0026] Obtain several test statements;

[0027] Several test statements are input into multiple simplified semantic analysis models;

[0028] The test statements that scored high in each simplified semantic analysis model were selected to form a simplified test set;

[0029] Input a simplified test set into the semantic analysis model to obtain the analysis results.

[0030] Furthermore, the specific method for rewriting the corpus that several semantic analysis models can accurately identify without affecting the semantics of the sentences is as follows:

[0031] Obtain corpora that can be accurately identified by several semantic analysis models;

[0032] The rewritten corpus is obtained by applying at least one of the following processing methods to these corpora using the first rewriting model: adding typos, adding punctuation marks, replacing synonyms, and adding words that do not affect the semantics.

[0033] Furthermore, the specific method for rewriting the corpus that several semantic analysis models can accurately identify without affecting the semantics of the sentences is as follows:

[0034] Obtain corpora that can be accurately identified by several semantic analysis models;

[0035] These corpora are translated into English using a translation model and then translated back into Chinese to obtain rewritten corpora.

[0036] Furthermore, the specific method for rewriting the positive corpus that several semantic analysis models can accurately identify to obtain the negative corpus is as follows:

[0037] Obtain positive corpora that can be accurately identified by several semantic analysis models;

[0038] The reverse corpus is obtained by using a second rewriting model to process these positive corpora by adding negative words or replacing antonyms.

[0039] Furthermore, the specific method for rewriting the positive corpus that several semantic analysis models can accurately identify to obtain the negative corpus is as follows:

[0040] Obtain positive corpora that can be accurately identified by several semantic analysis models;

[0041] The reverse corpus is obtained by performing reverse semantic rewriting on these positive corpora using the third rewriting model.

[0042] Furthermore, the specific method for obtaining a comprehensive score by combining the scoring results of the above steps is to calculate the comprehensive score using the following formula:

[0043] score=α*s base +(1-α)[β*s invar +(1-β)*s dir ],

[0044] Where, score is the overall score, s base For the first score, s invar For the second-highest score, s dir The third score is given by α, which is the first factor, 0 ≤ α ≤ 1, and β, which is the second factor, 0 ≤ β ≤ 1.

[0045] Furthermore, the first factor α is 0.55, and the second factor β is 0.75.

[0046] The advantage of this invention lies in the semantic analysis model evaluation method provided, which evaluates the capabilities of the semantic analysis model from multiple perspectives, resulting in more accurate and comprehensive evaluation results. Attached Figure Description

[0047] Figure 1 This is a flowchart of the semantic analysis model evaluation method of the present invention. Detailed Implementation

[0048] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1The present invention provides a semantic analysis model evaluation method, specifically comprising: S1: testing the basic functions of the semantic analysis model; S2: testing the generalization ability of the semantic analysis model; S3: testing the reverse recognition ability of the semantic analysis model; S4: obtaining a comprehensive score by combining the scoring results of the above steps. Through these steps, the semantic analysis model is tested from multiple perspectives, resulting in a more accurate and comprehensive evaluation. The following details these steps.

[0050] For step S1: Test the basic functionality of the semantic analysis model.

[0051] Step S1 primarily tests the basic functionality of the semantic analysis model. The model needs to perform well in relatively simple cases. Generally, if the semantic analysis model performs poorly in these cases, it will significantly impact the user experience.

[0052] The basic functions of testing semantic analysis models specifically include:

[0053] Examine the semantic analysis model's vocabulary. Test the semantic analysis model using a simplified test set and evaluate its analysis results. Score the test results to obtain the highest score.

[0054] Specifically, the method for checking the semantic analysis model's vocabulary is as follows: compare the model's vocabulary with the industry keyword database to determine if the vocabulary is complete. Checking whether the vocabulary fully covers these keywords is crucial; if it's not comprehensive, the semantic analysis model may lack sensitivity to some industry keywords, thus reducing the overall performance of the model.

[0055] The industry keyword database was obtained through the following specific methods:

[0056] Acquire dialogue data from various industries. Obtain dialogue data from e-commerce platforms such as Taobao and JD.com. Input the dialogue data into an attention mechanism model to automatically identify keywords. Compile all extracted keywords into an industry keyword library.

[0057] The specific method for testing the semantic analysis model on a simplified test set and judging the analysis results of the semantic analysis model on the simplified test set is as follows:

[0058] Several test sentences are obtained. These test sentences are then input into multiple simplified semantic analysis models. In this invention, five simplified semantic analysis models are selected. Test sentences that score highly in each simplified semantic analysis model are selected to form a simplified test set. Specifically, in this invention, test sentences that score 0.98 or higher in each simplified semantic analysis model are selected to form the simplified test set. Here, simplified semantic analysis models refer to some simple semantic analysis models in the field. The simplified test set is input into the semantic analysis models to obtain analysis results. If the model fails to recognize these types of example sentences, the model performs poorly in the basic function test.

[0059] For step S2: Test the generalization ability of the semantic analysis model.

[0060] Testing the generalization ability of a semantic analysis model specifically includes:

[0061] Several corpora that can be accurately identified by semantic analysis models are rewritten without affecting the semantic meaning of the sentences to obtain rewritten corpora. These rewritten corpora are then re-inputted into the semantic analysis models to determine whether they can be correctly identified. The test results are scored to obtain a second score.

[0062] Specifically, the method for obtaining rewritten corpora by rewriting corpora that can be accurately identified by several semantic analysis models without affecting the semantic meaning of the sentences is as follows: Obtain several corpora that can be accurately identified by several semantic analysis models. Then, use a first rewriting model to process these corpora using at least one of the following methods: adding typos, adding punctuation marks, replacing synonyms, and adding words that do not affect the semantic meaning. For example, the intention of the example sentence "This piece of clothing is of really good quality" is "The buyer is satisfied." The example sentence can be modified to "Hello, this piece of clothing is of really good quality!" The added words do not change the main meaning of the original sentence. The modified corpora are then re-input into the semantic analysis model to see if it can still recognize the modified corpora.

[0063] As another rewriting method, the specific method for rewriting the corpus by rewriting several semantic analysis models that can be accurately identified without affecting the semantics of the sentences is as follows:

[0064] We acquire several corpora that can be accurately identified by semantic analysis models. These corpora are then translated into English using a translation model, and then back into Chinese to obtain rewritten corpora.

[0065] For example, the sentence "This clothing is of really good quality" means "The buyer is satisfied." A Chinese-to-English translation model translates this sentence as "This clothing has good quality," and then an English-to-Chinese translation model translates it as "This clothing has good quality." After two rounds of translation, the sentence still conveys the original meaning, but the expression is different. If the semantic analysis model can still identify "The buyer is satisfied," then the model performs well in the generalization performance test.

[0066] For step S3: Test the reverse recognition capability of the semantic analysis model.

[0067] Testing the reverse recognition capability of a semantic analysis model specifically includes:

[0068] Several positive corpora that the semantic analysis model can accurately identify are reverse-written to obtain negative corpora. These reverse corpora are then re-inputted into the semantic analysis model to determine if it can correctly identify the text. The test results are scored to obtain a third score.

[0069] The specific method for rewriting the positive corpus that several semantic analysis models can accurately identify to obtain the reverse corpus is as follows:

[0070] We obtain several positive corpora that can be accurately identified by semantic analysis models. We then use a second rewriting model to process these positive corpora by adding negative words or replacing antonyms, thus obtaining negative corpora.

[0071] Understandably, there's no semantic distinction between "positive" and "direction." As a pair of relative intentions, if one is considered positive, the other is considered negative. Specifically, the first step is to identify pairs of relative intention labels, such as "buyer expresses dissatisfaction" and "buyer expresses satisfaction." Then, sample some example sentences from the intentions that the model correctly identifies as belonging to one category, such as "buyer expresses satisfaction." In this case, "buyer expresses satisfaction" represents a positive intention. Next, add negative words or replace antonyms, for example, rewriting "This garment is of really good quality" as "This garment is of really poor quality" or "This garment is of really bad quality." Then, use the semantic model to test the rewritten example sentences. If the rewritten example sentence predicts "buyer expresses dissatisfaction," then the model performs well in the negative test.

[0072] As another alternative implementation method, the specific method for rewriting the positive corpus that several semantic analysis models can accurately identify to obtain the reverse corpus can also be as follows:

[0073] We obtain positive corpora that can be accurately identified by several semantic analysis models. We then use a third rewriting model to perform reverse semantic rewriting on these positive corpora to obtain reverse corpora.

[0074] For example, the model can rewrite "This garment is of really good quality" into "This garment is of poor quality". Then, the semantic model is used to test the rewritten example sentence. If the rewritten example sentence predicts "The buyer is dissatisfied", then the model performs well in the backtesting.

[0075] For step S4: A comprehensive score is obtained by combining the scoring results of the above steps.

[0076] The specific method for obtaining a comprehensive score by combining the scoring results of the above steps is to calculate the comprehensive score using the following formula:

[0077] score=α*s base +(1-α)[β*s invar +(1-β)*s dir ],

[0078] Where, score is the overall score, s base For the first score, s invar For the second-highest score, s dir The third score is given by α, which is the first factor, 0 ≤ α ≤ 1, and β, which is the second factor, 0 ≤ β ≤ 1.

[0079] Furthermore, the first factor α is 0.55, and the second factor β is 0.75.

[0080] The overall score obtained can accurately evaluate the functionality of the semantic analysis model.

[0081] A semantic analysis model that scores perfectly in basic functionality is generally considered acceptable, but a model lacking generalization ability is unacceptable. Therefore, only models that pass the basic functionality test and possess some generalization ability are considered qualified. Meanwhile, reverse engineering is a more challenging test, as it reveals model problems that are common and difficult to solve. Therefore, this third test carries a relatively low weight. The weighting of each type of test can be adjusted as needed.

[0082] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A semantic analysis model evaluation method, characterized in that, Include: Testing the basic functionality of the semantic analysis model specifically includes: Examine the model vocabulary of the semantic analysis model; The semantic analysis model is tested using a simplified test set to determine the analysis results of the semantic analysis model on the simplified test set: several test statements are obtained; the several test statements are respectively input into multiple simplified semantic analysis models; the test statements that each simplified semantic analysis model gives a high score are selected to form the simplified test set; The simplified test set is input into the semantic analysis model to obtain the analysis results; The test results are scored to obtain the first score; Testing the generalization ability of the semantic analysis model specifically includes: The rewritten corpus is obtained by rewriting the corpus that can be accurately identified by several semantic analysis models without affecting the semantics of the sentences. The rewritten corpus is then re-inputted into the semantic analysis model to determine whether it can be correctly identified. The test results are scored to obtain a second score; Testing the reverse recognition capability of the semantic analysis model specifically includes: The positive corpus that can be accurately identified by several of the aforementioned semantic analysis models is reverse-rewritten to obtain the reverse corpus; The rewritten reverse corpus is then re-inputted into the semantic analysis model to determine whether it can be correctly identified. The test results are scored to obtain a third score; A comprehensive score is obtained by combining the scoring results from the above steps; The method for obtaining a comprehensive score by combining the scoring results of the above steps is to calculate the comprehensive score using the following formula: score=α*S base +(1-α)[β*S invar +(1-β)*S dir ], where score is the overall score, S base For the first score, S invar For the second score, S dir The third score is defined as follows: α is the first factor, 0 ≤ α ≤ 1, and β is the second factor, 0 ≤ β ≤ 1.

2. The semantic analysis model evaluation method according to claim 1, characterized in that, The specific method for checking the model vocabulary of the semantic analysis model is as follows: The semantic analysis model's vocabulary is compared with the industry keyword database to determine whether the vocabulary is complete.

3. The semantic analysis model evaluation method according to claim 2, characterized in that, The industry keyword database was obtained through the following specific methods: Obtain dialogue data from various industries; The dialogue corpus is input into an attention mechanism model to automatically identify keywords; All extracted keywords are combined into the industry keyword library.

4. The semantic analysis model evaluation method according to claim 1, characterized in that, The specific method for obtaining rewritten corpus by rewriting the corpus that can be accurately identified by several semantic analysis models without affecting the semantics of the sentences is as follows: Obtain several corpora that can be accurately identified by the aforementioned semantic analysis models; The rewritten corpus is obtained by applying at least one of the following processing methods to these corpora using the first rewriting model: adding typos, adding punctuation marks, replacing synonyms, and adding words that do not affect the semantics.

5. The semantic analysis model evaluation method according to claim 1, characterized in that, The specific method for obtaining rewritten corpus by rewriting the corpus that can be accurately identified by several semantic analysis models without affecting the semantics of the sentences is as follows: Obtain several corpora that can be accurately identified by the aforementioned semantic analysis models; The rewritten corpus is obtained by translating these corpora into English using a translation model and then translating them back into Chinese.

6. The semantic analysis model evaluation method according to claim 1, characterized in that, The specific method for rewriting the positive corpus that can be accurately identified by several semantic analysis models to obtain the reverse corpus is as follows: Obtain several positive corpora that can be accurately identified by the semantic analysis model; The reverse corpus is obtained by using a second rewriting model to process these positive corpora by adding negative words or replacing antonyms.

7. The semantic analysis model evaluation method according to claim 1, characterized in that, The specific method for rewriting the positive corpus that can be accurately identified by several semantic analysis models to obtain the reverse corpus is as follows: Obtain several positive corpora that can be accurately identified by the semantic analysis model; The reverse corpus is obtained by performing reverse semantic rewriting on these positive corpora using a third rewriting model.

8. The semantic analysis model evaluation method according to claim 1, characterized in that, The first factor α is 0.55, and the second factor β is 0.75.

Images