Sentence coherence checking method

Abstract

The present invention provides a sentence coherence checking method, which extracts entities from sentences in an article coherence task data set to obtain a new data set; encodes the segmented and marked sentences through a Bert layer to obtain sentence representation; averages the vector groups of multiple entities inside the previous and next sentences to obtain an internal entity vector group, averages the vector groups of cohesive words between the internal entities of the previous and next sentences to obtain an internal cohesive word vector group, and averages the head and tail cohesive words between the previous and next sentences to obtain a sentence head and tail cohesive word vector group; cross-combines the entity vector group and the cohesive word vector group, and splices to obtain a sentence fusion vector that integrates cohesive words and entity information; then splices the above vectors to obtain a new sentence representation, connects the vector group in the new sentence representation to an attention mechanism attention layer, passes through a fully connected layer, and then performs softmax classification to judge the coherence of the new sentence representation, and if the output result is 1, the new sentence is judged to be coherent. The sentence coherence checking method of the present invention can quickly and accurately interpret the meaning of the text according to the semantic understanding of the sentence context.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a sentence coherence checking method. Background Art

[0002] In the paragraphs of a text document, the essence of text coherence is the existence of semantic relationships that integrate vocabulary and grammar and have logical meaning. On the one hand, it is reflected in the relevance of entity vocabulary in the sentence and the grammatical degree of the sentence. On the other hand, it is reflected in the logical expression between entities and the connecting words before and after the sentence. Coherence is often used to measure the importance of the expression process of the document sentence. Cohesion is mainly a surface form of the text structure. The logic of the sentences before and after is expressed through conjunctions and connecting words, ensuring that the semantics of the article are conveyed to the reader from the beginning to the end and from the shallow to the deep within the text framework, making it easier for them to perceive and explain the theme of the article.

[0003] For example, in the application of intelligent writing and reading comprehension, if the sentence structure of the recognized paragraph is not smooth, then the positioning of the answer will be biased due to the inability to grasp the semantics of the article, resulting in semantic misunderstanding. However, directly conducting coherence research at the sentence level will lose a lot of contextual information, and directly using the entire chapter for analysis and research will have many technical problems due to the large granularity. At the current technical and application level, semantic understanding is often not just about syntactic structure and appropriate word matching. The semantic fluency of a single sentence often cannot reflect the semantics of the entire paragraph. At present, only by grasping the coherence of the text and combining it with the contextual semantic understanding of the sentence can the meaning of the text be fully interpreted.

[0004] Therefore, it is necessary to provide a new technical solution. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention discloses a sentence coherence checking method, which can help understand the reading habits of an article and further determine the sentence coherence. The specific technical solution is as follows:

[0006] The present invention provides a sentence coherence checking method, comprising:

[0007] Entity extraction is performed on sentences in the article coherence task dataset to obtain a new dataset, which includes labeled sentences and entities, and the entities are segmented; one entity constitutes a vector group;

[0008] The segmented and labeled sentences are encoded through the Bert layer to obtain sentence representation;

[0009] In sentence representation, the vector groups of multiple entities in the previous and next sentences are averaged to obtain the internal entity vector group, the vector groups of the connecting words between the internal entities in the previous and next sentences are averaged to obtain the internal connecting word vector group, and the beginning and end connecting words between the previous and next sentences are averaged to obtain the sentence beginning and end connecting word vector group;

[0010] The vectors in the internal entity vector group, the internal cohesion word vector group and the sentence head-end cohesion word vector group are concatenated respectively, and an external distributed linear layer is connected to obtain the internal entity distribution vector, the internal cohesion word distribution vector and the sentence head-end cohesion word distribution vector;

[0011] Cross-concatenate the internal entity distribution vector and the internal cohesive word distribution vector to get the new sentence representation;

[0012] Concatenate the sentence head-tail connection distribution vector and the new sentence representation to obtain a new sentence representation;

[0013] The new sentence representation vector is connected to the attention mechanism attention layer, and then the new sentence representation vector passes through the fully connected layer. The new sentence representation vector is then classified through the softmax function and the judgment result is output.

[0014] Furthermore, the article coherence task dataset includes at least two sentences, each of which includes entities and linking words between entities;

[0015] In two adjacent sentences, the connection relationship between the former and the latter includes the beginning word of the former sentence, the ending word of the latter sentence, and the connecting words between the former and the latter sentence.

[0016] Furthermore, the new dataset includes at least two labeled sentences and at least two entities.

[0017] Furthermore, the segmented and labeled sentences are encoded through the Bert layer to obtain the sentence representation: H(i) = {H_A, (H_B, H_doler), H_D, (H_C, H_jin), H_E}, where,

[0018] H(i) represents the sentence representation;

[0019] H_A is the beginning word of the sentence representation, H_E is the ending word of the sentence representation, and H_D is the linking word between two sentences in the sentence representation;

[0020] (H_B, H_doler) is the first sentence of the two sentences, H_doler is the entity in the first sentence, and H_B is the linking word between entities in the second sentence;

[0021] (H_C_H_jin) is the latter sentence of the two sentences, H_jin is the entity inside the latter sentence, and H_C is the connecting word between the entities inside the latter sentence.

[0022] Furthermore, in sentence representation, the vector groups of multiple entities in the previous and next sentences are averaged through the entity representation layer to obtain an internal entity vector group.

[0023] Furthermore, in the sentence representation, the connection word vector groups between the internal entities of the previous and next sentences are averaged through the internal connection layer to obtain the internal connection word vector group.

[0024] Furthermore, in the sentence representation, the beginning and end words between the previous and next sentences are averaged through the beginning and end connection layer to obtain the sentence beginning and end connection word vector group.

[0025] Furthermore, the vectors in the internal entity vector group, the internal cohesion word vector group and the sentence head-end cohesion word vector group are concatenated respectively, and an external distributed linear layer is connected to obtain the internal entity distribution vector, the internal cohesion word distribution vector and the sentence head-end cohesion word distribution vector, which specifically includes:

[0026] Concatenate multiple vectors in the internal entity vector group and externally distribute the linear layer dense_entity to obtain the internal entity distribution vector H_doler, H_jin;

[0027] Concatenate multiple vectors in the internal cohesive word vector group and connect the external cohesive word distribution linear layer dense_BC to obtain the internal cohesive word distribution vectors H_B, H_C;

[0028] Concatenate multiple vectors in the sentence head-to-tail word vector group and connect the head-to-tail word distribution linear layer dense_ADE to obtain the sentence head-to-tail word distribution vector (H_A, H_D, H_E).

[0029] Furthermore, the internal entity distribution vector and the internal linking word distribution vector are cross-joined to obtain a new sentence represented as: He1(H_B,H_doler),He2(H_C,H_jin).

[0030] Furthermore, if the output result is 1, it is judged that the new sentence represents coherence, and if the output result is 0, it is judged that the new sentence represents incoherence.

[0031] The present invention has the following beneficial effects:

[0032] The sentence coherence checking method of the present invention can quickly and accurately interpret the meaning of a text based on the semantic understanding of the sentence context, simulate people's usual reading habits of using entity relationships and entity linking word connections (collocations) to help understand articles, and further determine sentence coherence.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 Model architecture diagram of the sentence coherence checking method of the present invention. DETAILED DESCRIPTION

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0037] The present invention provides a method for checking sentence coherence, comprising the following steps:

[0038] Entity extraction is performed on the sentence S in the article coherence task dataset to obtain a new dataset D, which includes a marked sentence Se and an entity e, and the entity e is segmented; one entity constitutes a word vector group. The article coherence task dataset includes at least two sentences, each sentence includes an entity and a linking word between entities; in two adjacent sentences, the linking relationship between the former sentence and the latter sentence includes the beginning word of the former sentence, the ending word of the latter sentence, and the linking word between the former sentence and the latter sentence. The new dataset D includes at least two marked sentences Se and at least two entities e. Specifically, taking two sentences in the article coherence task dataset D as an example, the article coherence task dataset D = {(S1, S2)}, where S1 and S2 represent two sentences, and the new data task set D = {Se1, Se2, e1, e2}, Se1 and Se2 represent the sentences after marking respectively, and e1 and e2 represent the corresponding entities extracted.

[0039] Construct the model and encode the segmented and labeled sentences through the Bert layer to obtain sentence representation H(i) = {H_A, (H_B, H_doler), H_D, (H_C, H_jin), H_E};

[0040] In sentence representation, the vector groups of multiple entities within the previous and next sentences are averaged to obtain the internal entity vector group, the vector groups of the connecting words between the internal entities of the previous and next sentences are averaged to obtain the internal connecting word vector group, and the beginning and end connecting words between the previous and next sentences are averaged to obtain the sentence beginning and end connecting word vector group.

[0041] The internal entity vector groups are horizontally spliced ​​and externally connected to the distributed linear layer dense_entity, the sentence head and tail connection word vector groups are spliced ​​and externally connected to the head and tail connection word distribution linear layer dense_ADE, and the internal connection word vector groups are spliced ​​and externally connected to the connection word distribution linear layer dense_BC to obtain a new sentence.

[0042] The internal connection word vector group and the internal entity vector group of the new sentence are concatenated to obtain the new sentence representation.

[0043] The vector group in the new sentence representation is connected to the attention layer, and then passes through the fully connected layer, and then softmax classification is performed to determine the coherence of the new sentence representation. If the output result is 1, the new sentence is considered coherent. If the output result is 0, the new sentence is considered incoherent.

[0044] Combine the following Figure 1 For detailed description, please refer to Figure 1 The sentence coherence checking method of the present invention comprises:

[0045] Step 1: Entity extraction preprocessing,

[0046] For a given article coherence task dataset D = {(S1, S2)}, where S1 and S2 represent two sentences, entities are extracted from S1 and S2 using tools, and a new data task set D = {Se1, Se2, e1, e2} is obtained, where Se1 and Se2 represent the marked sentences, and e1 and e2 represent the corresponding entities extracted.

[0047] For example: the sentence S1 entity consists of <e1> <f1>Segmentation is performed, and the sentence S2 entity is composed of <e2> <f2>The new data task set after segmentation is:

[0048] {"text_a":" <e1>today <f1>Outside <e1>rain <f1>","entity_a":["Today","It's raining"],"text_b":"So I can't play <e2>basketball <f2>","entity_b":["Basketball"],"label":1}

[0049] Step 2: Model Architecture

[0050] Figure 1 It is the entire architecture of the model. The model structure mainly includes the following four parts.

[0051] Below Figure 1 Taking as an example, the sentence coherence checking method of the present invention is further described. Figure 1 As shown,

[0052] (1) The sentence Se1 after extracting entity tags is concatenated with Se2 through the Bert layer to obtain the sentence representation:

[0053] H(i)={H_A, (H_B,H_doler),H_D,(H_C,H_jin),H_E}

[0054] (2) In sentence representation, the vector groups of multiple entities in the previous and next sentences are averaged to obtain the internal entity vector groups H_doler, H_jin, the vector groups of connecting words between the previous and next sentences are averaged to obtain the internal connecting word vector groups H_B,,H_C, and the vector groups of the beginning and end connecting words of Se1 and Se2 are averaged to obtain the beginning and end connecting word vector groups H_A,H_E,H_D,

[0055] For example: <e1>today <f1>yes <e1>sunny <f1>?yes <e2>cloudy day <f2>

[0056] Among them, [Today, sunny] is the entity vector group H_doler, and [cloudy] is the entity vector group H_jin.

[0057] And [is] is the internal connection word vector group H_B, and [is] is the internal connection word vector group H_C.

[0058] (3) Concatenate multiple vectors in the internal entity vector groups H_doler and H_jin, and externally connect the concatenated internal entity vector group to the distribution linear layer dense_entity to obtain the internal entity distribution vectors H_doler and H_jin; concatenate the vectors in the sentence internal connection word vector group and externally connect the connection word distribution linear layer dense_BC to obtain the internal connection word distribution vectors H_B and H_C; concatenate multiple vectors in the sentence start and end connection word vector groups (H_A, H_D, H_E), and externally connect the concatenated sentence start and end connection vector group to the start and end connection word distribution linear layer dense_ADE to obtain the start and end connection word distribution vectors (H_A, H_D, H_E).

[0059] Cross - combine and concatenate the internal entity distribution vectors and internal connection word distribution vectors obtained through the distribution linear layer to obtain a sentence fusion vector that fuses connection word and entity information, that is, two new sentence representations. The new sentence representation form is He1(H_B, H_doler), He2(H_C, H_jin), and the structure is (Entity1, Link1, Entity……).

[0060] The present invention aims to analyze the connection between the distributions of entities within a sentence by externally connecting the dense entity layer to obtain two entity distribution vector representations H_doler and H_jin; aims to analyze the connection between the distributions of connection words within a sentence by externally connecting the dense_BC layer to obtain two connection word distribution vector representations H_B and H_C; aims to analyze the connection between the distributions of the start and end connection words within a sentence by externally connecting the dense_ADE layer to obtain three connection word distribution vector representations (H_A, H_D, H_E).

[0061] (4) The new sentence representation is obtained by concatenating the sentence head-end linking word distribution vector (H_A, H_D, H_E) and the new sentence representation obtained in step (3). The new sentence representation vector group is connected to the attention mechanism attention layer. The attention mechanism captures the dependency between the head-end linking words and the sentence fusion vectors in the sentence, and outputs a new sentence representation vector with fusion attention. Finally, the output new sentence representation vector passes through the fully connected layer, and then the new sentence representation vector is classified by the softmax function and the result is output. If the output result is 1, the new sentence is judged to be coherent. If the output result is 0, the new sentence is judged to be incoherent.

[0062] The structure of the sentence representation can be understood as:

[0063] 1) H_A, H_E, and H_D represent the cohesion of the entire sentence;

[0064] H_A: the opening word of Se1;

[0065] H_E: the ending word of Se2;

[0066] H_D: linking word between Se1 and Se2;

[0067] 2)(HB, H_doler):Se1;

[0068] H_doler: entity inside Se1;

[0069] H_B: linking words between entities within Se1;

[0070] 3)(HC_H_jin):Se2;

[0071] H_jin: represents the entity inside Se2;

[0072] H_C: represents the linking word between entities within Se2.

[0073] The structure of the new sentence in the present invention remains unchanged. The linear layer is mainly used to focus on the distribution relationship of entities and the distribution relationship of linking words. The purpose of obtaining a new sentence is also to consider integrating this distribution relationship to obtain a new vector in this way. Specifically:

[0074] (1) Feature extraction

[0075] 1) Extract the vector after the head-to-tail connection layer and perform linear distribution processing to obtain its distribution vector;

[0076] 2) Extract the entities and linking words in the upper and lower sentences to obtain the corresponding vector group. Multiple vectors in the vector group are concatenated to obtain the vector group, which is passed through a linear layer to analyze the relationship between the sentence entities and the distribution of linking words, and obtain the entity distribution vector and the linking word distribution vector.

[0077] (2) Entity Fusion:

[0078] The entity distribution vector and linking word distribution vector obtained in step 2 of feature extraction are integrated, and the two distribution vectors are cross-joined to obtain the new representations He1 and He2 of the two sentences, with the structure of (Entity1, Link1, Entity…)

[0079] (3) Multi-head attention mechanism layer

[0080] The three vectors H_A, H_D, H_E obtained by concatenating feature extraction and the two new sentence representations obtained by the entity fusion layer.

[0081] Table 1 is an analysis table exploring semantic coherence from different angles.

[0082]

[0083] Please refer to Table 1. The first three models (BILSTM, BILSTM-Att, BILSTM-Entity) are word vector analysis obtained by using word2vec as pre-training corpus, while the latter (BERT, BERT-LSTM, BERT-Entity, BERT-EntityAtt-MH) are all model processing of word vectors of Bert pre-training model. Exploring the analysis of semantic coherence from different angles, it can be seen from Table 1 that in the baseline word representation model, the attention mechanism and entity distribution can greatly improve the coherence modeling effect.

[0084] Data performance: For the Bert-based model variant, the representation vector itself has the distribution of contextual semantics, and has good performance in various NLP tasks. Using its original vector sequence as input, the classification effect of softmax reached 73.53%. Secondly, fine-tuning in downstream tasks has a slight improvement.

[0085] The newly proposed model uses the prominent information of the position distribution of entity words and linking words to achieve a performance of 74.95% on the data, indicating that the combined distribution of entities and linking words is very important for the coherence task. By adding the Multi-Head attention mechanism on this basis, it is hoped that the input will be mapped to different subspaces through different linear changes, and the input sequence can be understood from different angles. The num_heads is set to 8, which effectively improves the model performance, and the acc accuracy rate is increased from 74.85% to 76.7%.

[0086] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0087] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify and vary the above embodiments within the scope of the present invention. < / e2> < / e1>

Claims

1. A method for checking sentence coherence, characterized in that: include, Entity extraction is performed on sentences in the article coherence task dataset to obtain a new dataset, which includes labeled sentences and entities, and the entities are segmented; one entity constitutes a vector group; The segmented and labeled sentences are encoded through the Bert layer to obtain sentence representation; In sentence representation, the vector groups of multiple entities in the previous and next sentences are averaged to obtain the internal entity vector group, the vector groups of the connecting words between the internal entities in the previous and next sentences are averaged to obtain the internal connecting word vector group, and the beginning and end connecting words between the previous and next sentences are averaged to obtain the sentence beginning and end connecting word vector group; Concatenate the vectors in the internal entity vector group, the internal cohesive word vector group, and the sentence head-end cohesive word vector group respectively, and connect the concatenation to an external distributed linear layer to obtain the internal entity distribution vector, the internal cohesive word distribution vector, and the sentence head-end cohesive word distribution vector respectively; Cross-concatenate the internal entity distribution vector and the internal cohesive word distribution vector to get the new sentence representation; Concatenate the sentence head-tail connection distribution vector and the new sentence representation to obtain a new sentence representation; The new sentence representation vector is connected to the attention mechanism attention layer, and then the new sentence representation vector passes through the fully connected layer. The new sentence representation vector is then classified through the softmax function and the judgment result is output.

2. The method according to claim 1, characterized in that The article coherence task dataset includes at least two sentences, each of which includes entities and linking words between entities; In two adjacent sentences, the connection relationship between the former and the latter includes the beginning word of the former sentence, the ending word of the latter sentence, and the connecting words between the former and the latter sentence.

3. The method according to claim 1, characterized in that The new dataset includes at least two labeled sentences and at least two entities.

4. The method according to claim 3, characterized in that The segmented and labeled sentences are encoded through the Bert layer to obtain the sentence representation: H(i) = {H_A, (H_B, H_doler), H_D, (H_C, H_jin), H_E}, where, H(i) represents the sentence representation; H_A is the beginning word of the sentence representation, H_E is the ending word of the sentence representation, and H_D is the linking word between two sentences in the sentence representation; (H_B, H_doler) is the first sentence of the two sentences, H_doler is the entity in the first sentence, and H_B is the linking word between entities in the second sentence; (H_C_H_jin) is the latter sentence of the two sentences, H_jin is the entity inside the latter sentence, and H_C is the connecting word between the entities inside the latter sentence.

5. The method according to claim 1, characterized in that In sentence representation, the entity representation layer averages the vector groups of multiple entities in the previous and next sentences to obtain the internal entity vector group.

6. The method according to claim 1, characterized in that In sentence representation, the internal connection layer averages the connection word vector groups between the internal entities of the previous and next sentences to obtain the internal connection word vector group.

7. The method according to claim 1, characterized in that In sentence representation, the head-to-tail connection words between the previous and next sentences are averaged through the head-to-tail connection layer to obtain the sentence head-to-tail connection word vector group.

8. The method according to claim 1, characterized in that The vectors in the internal entity vector group, the internal cohesion word vector group and the sentence head-end cohesion word vector group are concatenated respectively, and an external distributed linear layer is connected to obtain the internal entity distribution vector, the internal cohesion word distribution vector and the sentence head-end cohesion word distribution vector, which specifically includes: Concatenate multiple vectors in the internal entity vector group and externally distribute the linear layer dense_entity to obtain the internal entity distribution vector H_doler, H_jin; Concatenate multiple vectors in the internal cohesive word vector group and connect the external cohesive word distribution linear layer dense_BC to obtain the internal cohesive word distribution vectors H_B, H_C; Concatenate multiple vectors in the sentence head-to-tail word vector group and connect the head-to-tail word distribution linear layer dense_ADE to obtain the sentence head-to-tail word distribution vector (H_A, H_D, H_E).

9. The method according to claim 1, characterized in that: The internal entity distribution vector and the internal linking word distribution vector are cross-connected to obtain the new sentence representation: He1(H_B,H_doler), He2(H_C,H_jin).

10. The method according to claim 1, characterized in that If the output result is 1, the new sentence is judged to be coherent, and if the output result is 0, the new sentence is judged to be incoherent.

Images