Semantic parsing method and device based on contrast learning and semantic perception

Abstract

The application discloses a semantic parsing method and device based on contrast learning and semantic perception, comprising the following steps: collecting diverse positive and negative examples for each <sentence, semantic representation> pair in a training set; inputting the <sentence, semantic representation> pair and the positive and negative examples into a semantic parsing model after multi-level label representation of the <sentence, semantic representation> pair and the positive and negative examples; updating model parameters by the semantic parsing model through the overall similarity of the sentence and the semantic representation and the label; and inputting a sentence to be parsed into the trained semantic parsing model to obtain the semantic representation of the sentence to be parsed. The application makes the model focus on the overall semantics of the semantic representation, improves the discrimination ability of the model for fine granularity, and can more carefully and accurately divide examples for contrast learning, thereby reducing the noise caused by fuzzy examples to model building.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a semantic parsing method and apparatus based on contrastive learning and semantic perception. Background Technology

[0002] Semantic parsing is a core task in natural language processing and a key to achieving natural language understanding. In recent years, it has attracted considerable attention from researchers. For semantic parsing, each training instance is a <sentence, semantic representation> pair, and during the testing phase, given a sentence, the corresponding semantic representation is required to be output.

[0003] A key challenge in constructing neural semantic parsing models lies in the holistic and fine-grained nature of semantic representations. For semantic parsing tasks, the output semantic representation should be word-for-word accurate. If a word is changed or missing, the entire semantic representation changes significantly. For example, changing from "Property(λs(snum turnovers≥3))player" to "Property(λs(snum turnovers<3))player" with only one symbol alters the overall semantic meaning (representing players with 3 or more turnovers and players with fewer than 3 turnovers, respectively). Such fine-grained distinctions are common, and common semantic parsing methods optimize word-by-word, lacking modeling of the holistic and fine-grained characteristics of semantic representations. For instance, Chinese patent application CN105095186A discloses a semantic parsing method and apparatus. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a semantic parsing method and apparatus based on contrastive learning and semantic awareness. The method separates semantically different <sentence, semantic representation> pairs in the vector representation space, while bringing semantically identical <sentence, semantic representation> pairs closer together. This invention employs a multi-level partitioning approach to classify the collected examples. In the model optimization process, firstly, to improve overall sensitivity, this invention designs a holistic similarity function that compares semantic representations as a whole to accurately assess the similarity between sentences and semantic representations. Simultaneously, in conjunction with multi-level examples, this invention proposes a multi-level contrastive loss function, which groups semantically identical instances together at the representation layer and separates negative instances, thereby solving the problem of existing semantic parsing models being insensitive to fine-grained and holistic approaches.

[0005] The technical solution of the present invention includes:

[0006] A semantic parsing method based on contrastive learning and semantic awareness, comprising the following steps:

[0007] For each <sentence x, semantic representation y> pair in the training set, generate M positive sample sentences x′ of sentence x. m Obtain N semantic representations y′ of sentence x. n And pair <sentence x, semantic representation y> and <sentence x′>. m The semantic representation of y> is the relationship between sentence x and sentence x, and the semantic representation of y′ is the relationship between sentence x and sentence x. n >The sentences are respectively represented as <sentence x′ i , semantically represents y″ i Yes;

[0008] For each <sentence x″ i , semantically represents y″ i After labeling, the data is input into the semantic parsing model F. j-1 So that the semantic parsing model F j-1 Through sentence x″ i With semantic representation y″ i The overall similarity with the label is used to update the model parameters, resulting in the semantic parsing model F. j , where j is the training round;

[0009] When semantic parsing model F j When the set conditions are met, the semantic parsing model F will be... j F is the trained semantic parsing model;

[0010] Input the sentence to be parsed into the semantic parsing model F to obtain the semantic representation of the sentence.

[0011] Furthermore, generate sentence x′ m The methods include: restating sentence x.

[0012] Furthermore, obtain the semantic representation y′ n The methods include: inputting sentence x into semantic parsing model F j-1 .

[0013] Furthermore, the tags include: positive example tags, fuzzy example tags, or negative example tags, wherein the correct semantic representation of <sentence x″ i , semantically represents y″ i >For sentences labeled as positive examples, <sentence x′ m The semantic representation of y> is the same as that of <sentence x, and the semantic representation of y′ is the same as that of <sentence x. n For sentences labeled as fuzzy examples, the execution results differ, and the semantic representation is y′. n >The labels are for negative samples.

[0014] Furthermore, the overall similarity includes at least one of the following: average sequence representation similarity, attention sequence similarity, and conditional sequence similarity.

[0015] Furthermore, the average sequence representation similarity is calculated through the following steps:

[0016] 1) Calculate sentence x″ i The vector representation of h x ;

[0017] 2) Calculate the semantic representation y″ i The vector representation of g y ;

[0018] 3) Represent h as a vector x With vector representation g y After being transformed into the same space, similarity is calculated to obtain the average sequence representation of similarity.

[0019] Furthermore, the attention sequence similarity is calculated through the following steps:

[0020] 1) Calculate the semantic representation y″ i The t-th word y t The corresponding representation is g t ;

[0021] 2) Calculation word y t Attention-weighted representation at the sentence level

[0022] 3) Represent each of g t With each expression After being transformed into the same space, similarity calculation is performed to obtain the attention sequence similarity.

[0023] Further, conditional sequence similarity is calculated through the following steps:

[0024] 1) Calculate sentence x″ i The vector representation of h x ;

[0025] 2) Obtain the decoded semantic representation y″ i The t-th word y t When, the vector represents h x The weighted representation of c t ;

[0026] 3) Calculate the semantic representation y″ i The t-th word y t The corresponding representation is g t ;

[0027] 4) Represent each weighted value c t And representing g t After being transformed into the same space, similarity calculation is performed to obtain the conditional sequence similarity.

[0028] Furthermore, the semantic parsing model F is obtained through the following steps. j :

[0029] 1) Based on the overall similarity of <sentence x, semantic representation y> pairs, and the similarity of each <sentence x′ m The semantic representation of y> is used to determine the overall similarity and corresponding labels of pairs. Sentence-level contrastive learning is then performed to obtain the sentence-level contrastive loss.

[0030] 2) Based on the overall similarity of <sentence x, semantic representation y> pairs and the similarity of each <sentence x, semantic representation y′ n Based on the overall similarity and corresponding labels, semantic representation comparison learning is performed to obtain the semantic representation comparison loss.

[0031] 3) Calculate end-to-end decoding loss

[0032] 4) Loss based on sentence-level comparison Semantic representation contrast loss End-to-end decoding loss Obtain the overall loss of the model

[0033] 5) Use the overall model loss With the aforementioned tag, for <sentence x″ i , semantically represents y″ i >Train the semantic parsing model F to obtain the semantic parsing model. j .

[0034] A storage medium storing a computer program, wherein the computer program is configured to execute the method described above when run.

[0035] An electronic device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer to perform the methods described above.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1) The proposed holistic perception similarity function is used for contrastive learning, which can make the model pay attention to the overall semantics of the semantic representation and improve the model's ability to discriminate at a fine-grained level;

[0038] 2) The proposed multi-level sample partitioning method and the corresponding hierarchical comparison learning method can partition samples more meticulously and accurately for comparison learning, thereby reducing the noise brought by fuzzy samples to model modeling. Attached Figure Description

[0039] Figure 1 This is a flowchart of the method of the present invention.

[0040] Figure 2 This is an example diagram illustrating a multi-level instance partitioning method.

[0041] Figure 3 This is an example of the training process for a single training example. Detailed Implementation

[0042] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0043] This invention discloses a semantic parsing method based on contrastive learning and semantic awareness. First, it collects diverse positive and negative instances and uses a multi-level partitioning approach to divide the collected examples into three levels: positive examples, fuzzy examples, and negative examples. Then, it models the similarity between sentences and semantic representations at the overall level using a holistic perception similarity function. Finally, it trains the semantic parsing model using a multi-level contrastive loss function to obtain the final semantic parsing model. Its key technologies include:

[0044] 1) Utilizing a multi-level contrastive learning framework, during the contrastive learning process of multi-level examples, the similarity of positive pairs (sentences, semantic representations) is reduced at each level, thereby improving the model's perception of the overall and fine-grained distinction of semantic representations.

[0045] 2) Within the above framework, this invention utilizes a multi-level instance sampling and partitioning method to generate diverse and precisely partitioned examples, including examples with the same semantics but different expressions, as well as examples with different semantics but easily confused models. Simultaneously, parts that are difficult to distinguish as positive or negative are classified as fuzzy examples.

[0046] 3) Within the above framework, the hierarchical contrastive learning algorithm based on overall similarity models the similarity between sentences and semantic representations at the overall level, and hierarchically pulls and pushes instances at multiple levels in the vector representation space.

[0047] Specifically, the core technologies of this invention include:

[0048] 1. Multi-level instance sampling and partitioning methods

[0049] Contrastive learning algorithms learn better model representations by bringing positive examples closer and pushing negative examples further away. Positive and negative examples play a fundamental role in contrastive learning. However, in semantic analysis, many instances are ambiguous and very difficult to accurately classify into positive and negative categories. For example, in paraphrasing, a common method for constructing positive examples, the fine-grained semantics of the original sentence may be altered; two very different semantic representations may represent the same meaning and cannot be treated as negative samples. Therefore, this invention proposes a multi-level instance sampling and partitioning method.

[0050] Instance Sampling: 1) This invention randomly collects sentences and semantic representations in the training set; 2) Aug(x): This invention collects multiple paraphrases of sentences to enhance the diversity of text expression. This invention uses a publicly available paraphrasing model and generates 20 paraphrases for each sentence; 3) Parse(x): This invention collects the parsing results of the sentence under the current model. This invention uses a histogram search and retains the top 20 parsing results.

[0051] Multi-level instance sampling and partitioning method: This invention sorts the instances by Rank, with smaller Rank values ​​indicating a stronger likelihood of a positive instance.

[0052] 1) Rank=0: This level contains absolutely positive samples. This invention uses the annotations in the training corpus as the standard.

[0053] 2) Rank=1: This level contains ambiguous examples that are difficult to distinguish as positive or negative. For sentences, this invention classifies semantic representations with the same execution result and labeled semantic representation into this level. For semantic representations, restates of the sentence are classified into this category, as are sentences corresponding to semantic representations with the same execution result.

[0054] 3) Rank = 2: This level contains absolutely negative examples. For sentences, this invention classifies semantic representations whose execution results and labeled semantic representations differ into this level. For semantic representations, sentences corresponding to semantic representations with different execution results are also classified into this category. Because the execution results are different, this invention can determine that they cannot contain the same semantics.

[0055] A key characteristic of semantic parsing tasks is the requirement for accuracy; a single word error in the semantic representation can lead to an overall error. Therefore, rigorous judgment of correctness in comparisons is essential. Fuzzy examples are difficult to classify as either correct or incorrect. For instance, adding paraphrased sentences enhances the model's generalization ability, but paraphrasing subtly alters the semantics, potentially resulting in entirely different semantic representations. Semantic representations with identical execution results may simply be coincidental, or they may express the same meaning. Directly including these in positive or negative examples would introduce significant noise, hindering comparative learning. Therefore, these types need to be placed in fuzzy examples, at a distance greater than perfectly correct examples but certainly closer than negative examples.

[0056] 2. Hierarchical contrastive learning algorithm based on overall similarity

[0057] In contrastive learning, similarity calculation is crucial. The process of contrastive learning aims to achieve high similarity between positive examples while simultaneously lowering the similarity between negative examples. This invention measures similarity by treating semantic representation as a whole. Specifically, this invention designs three holistic similarity functions: average sequence representation similarity, attention sequence similarity, and conditional sequence similarity. They are:

[0058] φ sr (x, y) = mean(h) x ) T W s mean(g y )

[0059]

[0060]

[0061] Where, mean represents the average over the entire sentence. x and y represent the sentence and its semantic representation. h x and g y This represents the entire sentence and its semantic representation encoded into a vector. |y| is the length of the semantic representation, and g... t It is the t-th word y in the semantic representation t The corresponding representation. It is y t Attention-weighted representation at the sentence level. t It is used for decoding y t h x The weighted representation of . W is used to transform them into the same space for similarity calculation.

[0062] To complement the multi-level examples, this invention proposes a multi-level contrastive loss function. Given a sentence and a semantic representation (x, y), this invention performs contrastive learning on both the sentence and semantic representation levels:

[0063]

[0064]

[0065] Where φ θ (x, y) is the similarity function of this invention, θ is the model parameter, and r is the rank value. Through a multi-level contrastive loss function, this invention can hierarchically bring instances closer together and further apart in the vector representation space. The final loss function is the sum of the end-to-end decoding loss and the two multi-level contrastive loss functions of this invention:

[0066]

[0067] in, It is the training loss function of the semantic parsing decoder.

[0068] like Figure 3 As shown, the present invention is specifically illustrated by analyzing "Player with no less than 3 turnovers":

[0069] Scene:

[0070] Training corpus:

[0071] Example 1: Semantic representation of "Property(λs(s,num_turnovers,≥,3),player)" and the corresponding sentence "Who get 3 or more turnover each season".

[0072] Test corpus:

[0073] Test Example 1: Parse the sentence "Player with no less than 3 turnover".

[0074] Implementation:

[0075] Here, we take sentence-level comparative learning as an example:

[0076] (a) First, collect sentence-level samples, including randomly collecting sentences from the training set and generating new sentences through paraphrasing;

[0077] (ii) Then, according to the above multi-level division method, the sentence is divided into 3 levels according to the source of the sentence and the relationship between the marked semantic representation and the original semantic representation;

[0078] (III) Finally, the similarity between each sentence and semantic representation is calculated according to the overall perception similarity function described above, and a multi-level contrast loss function is used to bring the sample and semantic representation levels closer and further apart.

[0079] (iv) For test example 1, since similar representations are trained and reinforced in positive and fuzzy examples, they can be parsed more effectively.

[0080] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A semantic parsing method based on contrastive learning and semantic awareness, comprising the following steps: For each sentence in the training set Semantic representation Yes, generate a sentence. of A positive example sentence Get the sentence of semantic representation and will <sentence> Semantic representation >Yes, <sentence Semantic representation >Regarding <sentences Semantic representation >The sentences are represented as < Semantic representation Yes; For each sentence Semantic representation After labeling, the data is input into the semantic parsing model. To enable semantic parsing models Through sentences With semantic representation Based on the overall similarity with the label, update the model parameters to obtain the semantic parsing model. ,in For training rounds; When semantic parsing model When the set conditions are met, the semantic parsing model will be used. As a trained semantic parsing model ; Input the sentence to be parsed into the semantic parsing model This yields the semantic representation of the sentence to be parsed; The tags include: positive sample tags, fuzzy sample tags, or negative sample tags, wherein the correct semantic representation of the sentence is... Semantic representation >For sentences labeled as positive examples Semantic representation >Sentences with the same execution result Semantic representation >For sentences with different execution results for labels marked as fuzzy sample tags< Semantic representation >The labels are for negative samples.

2. The method as described in claim 1, characterized in that, Generate sentences The methods include: for sentences Perform restatement; obtain semantic representation The methods include: dividing the sentence Input semantic parsing model .

3. The method as described in claim 1, characterized in that, Overall similarity includes at least one of the following: average sequence representation similarity, attention sequence similarity, and conditional sequence similarity.

4. The method as described in claim 3, characterized in that, The average sequence representation similarity is calculated using the following steps: 1) Calculate sentences vector representation ; 2) Computational semantic representation vector representation ; 3) Representing vectors with vector representation After being transformed into the same space, similarity is calculated to obtain the average sequence representation of similarity.

5. The method as described in claim 3, characterized in that, Calculate attention sequence similarity using the following steps: 1) Computational semantic representation The Middle Word Corresponding representation ; 2) Calculation words Attention-weighted representation at the sentence level ; 3) Represent each With each expression After being transformed into the same space, similarity calculation is performed to obtain the attention sequence similarity.

6. The method as described in claim 3, characterized in that, Calculate conditional sequence similarity using the following steps: 1) Calculate sentences vector representation ; 2) Obtain the decoded semantic representation The Middle Word When, vector representation Weighted representation ; 3) Computing semantic representations The Middle Word Corresponding representation ; 4) Represent each weighted average and indicates After being transformed into the same space, similarity calculation is performed to obtain the conditional sequence similarity.

7. The method as described in claim 1, characterized in that, The semantic parsing model is obtained through the following steps. : 1) Based on <sentence> Semantic representation >Overall similarity of sentences, each <sentence Semantic representation Based on the overall similarity and corresponding labels, sentence-level contrastive learning is performed to obtain the sentence-level contrastive loss. ; 2) Based on <sentence> Semantic representation >Overall similarity of sentences, each <sentence Semantic representation Based on the overall similarity and corresponding labels, semantic representation comparison learning is performed to obtain the semantic representation comparison loss. ; 3) Calculate the end-to-end decoding loss ; 4) Comparison loss based on sentence end Semantic representation end contrast loss End-to-end decoding loss The overall loss of the model is obtained. ; 5) Use the overall model loss With the aforementioned tags, for <sentence> Semantic representation >Train the model to obtain the semantic parsing model. .

8. A storage medium storing a computer program, wherein, The computer program is configured to execute the method as described in any one of claims 1-7 at runtime.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method as claimed in any one of claims 1-7.

Images