An entity-enhanced text semantic representation method and system

Abstract

The application provides a text semantic representation method and system based on entity enhancement, comprising: obtaining a text encoding model and an entity enhancement encoding model, the entity enhancement encoding model being composed of a serially connected external entity vector representation module and a context-related entity information encoding module; extracting a first representation vector of a target text through the text encoding model; identifying a word sequence representing an entity in the target text through the external entity vector representation module, determining which entities in a knowledge base are associated with the word sequence, and obtaining an entity vector representation of the entity corresponding to the word sequence in the knowledge base; the entity information encoding module comprises a plurality of serially connected entity adapter layers, and the input of the entity information encoding module is an entity vector representation and a hidden state of an intermediate layer of the text encoding model; the features output by the last entity adapter layer are used as a second representation vector of the target text; and the first representation vector and the second representation vector are fused to obtain a semantic representation result of the target text.

Description

Technical Field

[0001] This invention relates to the field of information retrieval, and mainly to semantic representation methods of text in semantic-based retrieval. Background Technology

[0002] With the rapid development of the internet, massive amounts of content are widely circulated and generated online every day. As of June 2022, the number of websites hosted by the Communist Party of China had reached 3.98 million. Platforms such as Zhihu and Xiaohongshu have users creating a large amount of content every day. As of December 31, 2021, Zhihu had accumulated more than 490 million pieces of content, of which 420 million were questions and answers.

[0003] In the field of information retrieval, diverse search scenarios place higher demands on the characterization of semantic relevance. However, word-matching-based retrieval technologies have limitations, such as their inability to handle synonyms, polysemous words, and the overall semantics of sentences, making them ill-suited for diverse search scenarios. Therefore, semantic-based retrieval technologies have become a highly anticipated research direction.

[0004] With the introduction of pre-trained language model technology, deep semantic retrieval methods based on dense vector retrieval have achieved extremely high accuracy. Recently, they have successfully surpassed the traditional BM25 algorithm. For example, on the NaturalQuestions dataset, the semantic retrieval model based on dense vectors can outperform the BM25 algorithm with only 1,000 training data points.

[0005] However, current semantic-based retrieval models are lacking in capturing the semantics of entities within text, resulting in less than ideal performance when dealing with entity-specific queries. For example, for the query "Who plays Thoros of Myr in Game of Thrones?", the BM25 algorithm successfully matches the entity "Thoros of Myr" and finds the relevant text, while a dense vector retrieval model might retrieve an unrelated encyclopedia page for the actor, significantly negatively impacting retrieval performance.

[0006] Existing semantic retrieval models primarily use pre-trained language models, such as BERT, to encode queries or documents into fixed-length dense vectors, such as 128-dimensional or 768-dimensional vectors. Current research in this area mainly focuses on how to better obtain vector representations of text, such as by adjusting training strategies and mining hard-to-distinguish negative samples. Some studies also attempt to address the problem of poor semantic representation of text containing entities in current semantic retrieval methods.

[0007] For example, some researchers have leveraged the BM25 algorithm's superior performance in retrieving queries containing entities, hoping to transfer its advantages to dense vector retrieval models through knowledge distillation. However, experiments have shown that while this approach does enhance the retrieval performance of dense vector retrieval models for queries containing entities and learns the advantages of the BM25 algorithm, it also loses its own advantage in capturing semantic relevance of text.

[0008] Some researchers construct retrieval datasets based on triples representing relationships between entities, designing templates for each relationship. For example, for a triple like <China, capital, Beijing>, the query "Where is the capital of China?" would result in documents containing "The capital of China is Beijing". This ensures that all queries in the training set contain entities, forcing the text representation model in the semantic retrieval model to focus more on entities through data augmentation. While fine-tuning on the augmented dataset does enhance the focus on entities, testing on general retrieval datasets reveals a decline in retrieval performance.

[0009] Existing text semantic representation models based on pre-trained language models cannot effectively capture the semantics of entities. There are two main reasons for this: First, to reduce vocabulary size, current pre-trained language models typically use the WordPiece segmentation algorithm. For uncommon words, they are segmented into subwords. For example, the English word for Jinan, "Jinan," is not in the BERT model's vocabulary and would be segmented into "Jin" and "##an," where "##" indicates that "an" is a subword and not the beginning of the word. This segmentation makes it difficult to reconstruct the semantics of the original word "Jinan." Second, traditional semantic retrieval models are not explicitly trained to distinguish whether a term is an entity, part of an entity, or a regular term, especially for entities composed of multiple terms. For example, in the sentence "where is whitemarsh island," the model needs to recognize that "whitemarsh island" constitutes one entity, not two separate terms.

[0010] Furthermore, existing methods for improving entity semantic representation in enhanced semantic retrieval, such as BM25 distillation and data augmentation, lack generalization ability. While improving semantic retrieval performance on queries containing entities, they degrade performance on general retrieval datasets. Summary of the Invention

[0011] The purpose of this invention is to address the problems of poor entity semantic capture by pre-trained language models and poor generalization of existing improvement schemes in the prior art. This invention proposes a text representation method based on entity enhancement to improve the performance of semantic retrieval models in queries containing entities.

[0012] This invention proposes a text semantic representation method based on entity enhancement, which includes:

[0013] Step 1: Obtain the target text to be semantically represented, and obtain the text encoding model and entity augmentation encoding model. The entity augmentation encoding model consists of a concatenated external entity vector representation module and a context-dependent entity information encoding module; extract the first representation vector of the target text through the text encoding model.

[0014] Step 2: Use the external entity vector representation module to identify the word sequence representing entities in the target text, determine which entities in the knowledge base are associated with the word sequence, and obtain the entity vector representation of the entity corresponding to the word sequence in the knowledge base;

[0015] Step 3: The entity information encoding module includes multiple cascaded entity adapter layers. The input of the entity information encoding module is the vector representation of the entity in the target text and the hidden state of the intermediate layer of the text encoding model. The input of each entity adapter layer is the hidden state output by the corresponding intermediate layer of the text encoding model and the output of the previous entity adaptation layer. The feature output by the last entity adapter layer is used as the second representation vector of the target text.

[0016] Step 4: By using the entity information gating unit, the first representation vector and the second representation vector are fused to obtain the semantic representation result of the target text.

[0017] The aforementioned entity-enhanced text semantic representation method, wherein

[0018] Step 2 includes:

[0019] Named entity recognition identifies sequences of terms representing entities in the target text. Entity links determine the corresponding entities in the knowledge base. For each word or entity x, the vector representation E of the entity in the knowledge base is obtained using the word vectorization tool Wikipedia2vec. wikipedia (x);

[0020] Given vocabulary and entity table Mapping function E wikipedia : This maps words and entities to the same vector space;

[0021] According to Formula 1, a linear transformation matrix is ​​obtained. d PLM d represents the vector dimension in the text encoding model. Wikipedia The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity x in the dictionary is represented by a vector E obtained through the word vectorization tool Wikipedia2vec. wikipenta (x), after linear transformation, yields W·E wikipedia (x), the vector representation E obtained through the embedding layer of the text encoding model. PLM (x); the transformation matrix W such that the transformed vector W·E of all x is such that the vector W·E of all x is transformed. wikipedia (x) and E PLM The sum of the squares of the difference of (x) is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the text encoding model embedding layer is located;

[0022]

[0023] Based on Formula 2, the mapping function is obtained, resulting in the vector representation E after both words and entities are mapped to the input vector space of the text encoding model. Entity For those not in the entity table The words in the text are directly represented by vectors E obtained from the embedding layer of the text encoding model. PLM (x);

[0024]

[0025] Step 3 includes:

[0026] PLM-h j Let J be the hidden state output by the j-th intermediate layer of the text encoding model, with shape J. BatchSize is the number of samples in the current training batch, TextMaxLen is the maximum number of text tokens in the text encoder, and d hidden For the hidden layer dimension of the text encoder;

[0027] h i This represents the output of the i-th entity adapter, which is of shape [BatchSize, 1 + EntityLen, d]. hiddenThe tensor is ], where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is h0, the 0th position h0[:, 0,:] corresponds to a vector of all zeros, and the kth position h0[:, k,:] corresponds to the initial vector representation of the kth entity;

[0028] For the i-th entity adapter layer, the hidden state PLM-h output by the j-th intermediate layer [CLS] token of the text encoding model is calculated according to the following formula 3. j [:,0,:] is added to the 0th position h of the output of the entity adapter layer of layer i-1. i-1 [:,0,:];According to Formula 4, the obtained h i-1 As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from d... hidden Become d adapter According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimension, making the dimension increase from d. adapter Restore to d hidden and with h i-1 Add them together to form a residual connection;

[0029] h i-1 [:,0,:]=h i-1 [:,0,:]+PLM-h j [:,0,:] #(3)

[0030]

[0031] y i =Transformer-N(x) i #(5)

[0032]

[0033] As shown in Equation 7, the final output of the entity augmentation coding model is E. ent It is the output h of the last entity adaptation layer. K The encoding result corresponding to the 0th position in the code, where K is the number of entity adaptation layers.

[0034] E ent =h K [:,0,:]#(7)

[0035] The aforementioned entity-enhanced text semantic representation method, wherein step 4 includes:

[0036] The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector E encoded by the text encoding model based on the pre-trained language model. text and the second representation vector E ent , among which, [E text E ent ] indicates that E text and E ent The result of splicing, W g and b g denoted as the weights and biases of the gated network, respectively; σ represents the Sigmoid function; "·" represents the dot product operation; and ⊙ represents element-wise multiplication.

[0037]

[0038] E final =g⊙E text +(1-g)⊙E ent #(9)

[0039] g is a control weight with a value between 0 and 1.

[0040] As shown in Equation 9, the weight g is used to adjust E. text and E ent The final semantic representation result E of the target text is obtained by performing a weighted summation. final .

[0041] The text semantic representation method based on entity enhancement, wherein step 4 includes: performing text retrieval or text similarity judgment tasks with the extraction results of the semantic representation.

[0042] This invention also proposes a text semantic representation system based on entity enhancement, which includes:

[0043] Module 1: Obtain the target text to be semantically represented, and obtain the text encoding model and entity augmentation encoding model. The entity augmentation encoding model consists of a concatenated external entity vector representation module and a context-dependent entity information encoding module; extract the first representation vector of the target text through the text encoding model.

[0044] Module 2: The external entity vector representation module identifies the sequence of terms representing entities in the target text, determines which entities in the knowledge base are associated with the sequence of terms, and obtains the entity vector representation of the entity corresponding to the sequence of terms in the knowledge base;

[0045] Module 3: The entity information encoding module includes multiple cascaded entity adapter layers. The input of the entity information encoding module is the vector representation of the entity in the target text and the hidden state of the intermediate layer of the text encoding model. The input of each entity adapter layer is the hidden state output by the corresponding intermediate layer of the text encoding model and the output of the previous entity adaptation layer. The feature output by the last entity adapter layer is used as the second representation vector of the target text.

[0046] Module 4: By using the entity information gating unit, the first representation vector and the second representation vector are fused to obtain the semantic representation result of the target text.

[0047] The aforementioned entity-enhanced text semantic representation system, wherein

[0048] Module 2 includes:

[0049] Named entity recognition identifies sequences of terms representing entities in the target text. Entity links determine the corresponding entities in the knowledge base. For each word or entity x, the vector representation E of the entity in the knowledge base is obtained using the word vectorization tool Wikipedia2vec. wikipedia (x);

[0050] Given vocabulary and entity table Mapping function E wikipedia : This maps words and entities to the same vector space;

[0051] According to Formula 1, a linear transformation matrix is ​​obtained. d PLM d represents the vector dimension in the text encoding model. Wikipedia The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity x in the dictionary is represented by a vector E obtained through the word vectorization tool Wikipedia2vec. wikipedia (x), after linear transformation, yields W·E wikipedia (x), the vector representation E obtained through the embedding layer of the text encoding model. PLM (x); the transformation matrix W such that the transformed vector W·E of all x is such that the vector W·E of all x is transformed. wikipedia (x) and E PLM The sum of the squares of the difference of (x) is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the text encoding model embedding layer is located;

[0052]

[0053] Based on Formula 2, the mapping function is obtained, resulting in the vector representation E after both words and entities are mapped to the input vector space of the text encoding model. Entity For those not in the entity table The words in the text are directly represented by vectors E obtained from the embedding layer of the text encoding model. PLM (x);

[0054]

[0055] Module 3 includes:

[0056] PLM-h j Let [the hidden state output by the j-th intermediate layer of the text encoding model, with shape [BatchSize, TextMaxLen, d]]. hidden BatchSize is the number of samples in the current training batch, TextMaxLen is the maximum number of text tokens in the text encoder, and d hidden For the hidden layer dimension of the text encoder;

[0057] h i This represents the output of the i-th entity adapter, which is of shape [BatchSize, 1 + EntityLen, d]. hidden The tensor is ], where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is h0, the 0th position h0[:, 0,:] corresponds to a vector of all zeros, and the kth position h0[:, k,:] corresponds to the initial vector representation of the kth entity;

[0058] For the i-th entity adapter layer, the hidden state PLM-h output by the j-th intermediate layer [CLS] token of the text encoding model is calculated according to the following formula 3. j [:,0,:] is added to the 0th position h of the output of the entity adapter layer of layer i-1. i-1 [:,0,:];According to Formula 4, the obtained h i-1 As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from d... hidden Become d adapter According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimension, making the dimension increase from d. adapter Restore to d hidden and with h i-1 Add them together to form a residual connection;

[0059] h i-1 [:,0,:]=hi-1 [:,0,:]+PLM-h j [:,0,:] #(3)

[0060]

[0061] y i =Transformer-N(x) i #(5)

[0062]

[0063] As shown in Equation 7, the final output of the entity augmentation coding model is E. ent It is the output h of the last entity adaptation layer. K The encoding result corresponding to the 0th position in the code, where K is the number of entity adaptation layers.

[0064] E ent =h K [:,0,:]#(7)

[0065] The aforementioned entity-enhanced text semantic representation system, wherein module 4 includes:

[0066] The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector E encoded by the text encoding model based on the pre-trained language model. text and the second representation vector E ent , among which, [E text E ent ] indicates that E text and E ent The result of splicing, W g and b g denoted as the weights and biases of the gated network, respectively; σ represents the Sigmoid function; "·" represents the dot product operation; and ⊙ represents element-wise multiplication.

[0067]

[0068] E final =g⊙E text +(1-g)⊙E ent #(9)

[0069] g is a control weight with a value between 0 and 1.

[0070] As shown in Equation 9, the weight g is used to adjust E. text and E ent The final semantic representation result E of the target text is obtained by performing a weighted summation. final .

[0071] The entity-enhanced text semantic representation system includes module 4, which includes: performing text retrieval or text similarity judgment tasks based on the semantic representation extraction results.

[0072] The present invention also proposes a storage medium for storing a program that executes any of the entity-enhanced text semantic representation methods.

[0073] The present invention also proposes a client for any entity-enhanced text semantic representation system.

[0074] As can be seen from the above solutions, the advantages of the present invention are:

[0075] The advantages of this invention compared to existing technologies include: In terms of performance, it can better capture the semantics of entities contained in text and integrate the semantics of entities into the final text representation, thereby improving the text semantic representation model's ability to represent text containing entities. In terms of versatility, this invention, as a plugin, can be easily added to various existing text representation models to improve performance in downstream tasks such as semantic retrieval and text similarity calculation. Attached Figure Description

[0076] Figure 1 This is a structural diagram of the semantic representation model for entity enhancement in this invention;

[0077] Figure 2 This is a structural diagram of the entity adapter layer of the present invention. Detailed Implementation

[0078] The inventors discovered this issue while analyzing the text representation modules in existing semantic retrieval models. Existing technologies suffer from poor semantic representation of entities in text, primarily due to two problems. First, existing pre-trained language models (PLMs) are not adept at recognizing and representing entities in text when trained for retrieval tasks. This can be addressed by pre-extracting entities from the text and encoding them as a whole. Second, existing improvements to text representation models in semantic retrieval, which struggle to capture entity semantics, suffer from a lack of generalization. Specifically, while they improve performance on datasets where queries all contain entities, their performance deteriorates on general retrieval datasets. To solve this, a gating unit can be added during text encoding to control the influence of entity semantics on the original text representation. Furthermore, to enhance the generalization of the proposed method, the entity encoding module and gating module are designed as plug-ins, allowing them to be integrated into the text representation modules of current mainstream semantic retrieval models.

[0079] To achieve the above-mentioned technical effects, the present invention includes the following key technical points:

[0080] Key point 1: Introduction of external entity vector representation; Technical effect: By using external resources, well-encoded entity semantic representation vectors are obtained, avoiding the impact of pre-trained language models not being good at encoding entity information.

[0081] Key point 2: Context-dependent entity information encoding; Technical effect: Based on the semantic representation vector of the entity, the semantic information of the original text is added to obtain the context-dependent entity semantic representation, which is used for the final text semantic representation.

[0082] Key point 3, entity information gating unit; technical effect: by adjusting the contribution of entity information to text representation through adaptive weights, the generalization of entity augmentation methods can be enhanced.

[0083] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. Although the following embodiments are exemplified in English, this application is not limited to any particular language and can also be used for searching in other languages ​​such as Chinese.

[0084] Using textual features to represent entity semantics is still limited by the language model's text encoding capabilities, especially when the entity appears infrequently in the training corpus or is segmented into multiple words. To further address this issue, specialized entity encoding techniques can be used to directly obtain vector representations of entities, which are then incorporated as features into the text encoding process, ensuring that the resulting text encoding contains the entity's semantics. After obtaining the text vector representation, it can be used for tasks such as text retrieval and text similarity assessment. For example, in text retrieval tasks, it can achieve better retrieval results in scenarios requiring entity matching between queries and documents.

[0085] The overall model structure of this invention is as follows: Figure 1 As shown, Figure 1 The left side represents existing technologies, while the right side represents the technology proposed in this paper. The main approach involves adding pluggable components to existing text encoding models (such as BERT, RoBERta, and XLNET) to allow the semantics of entities within the text to be considered during the text encoding process. These components include an external entity vector representation module, an entity information encoding module, and entity gating units. This method can be integrated as a component into most existing text encoding models without requiring significant modifications, demonstrating excellent versatility.

[0086] The introduction of external entity vector representations is primarily achieved through an external entity vector representation module. The main goal is to identify the sequence of terms representing entities ("mentions") within a given query or document text, determine which entity in the Wikipedia knowledge base these mentions are associated with, and then obtain the vector representation of the corresponding entity in the knowledge base. For example, given the text "Jinan is the capital of Shandong province," named entity recognition first identifies the mentions "Jinan" and "shandong." Then, entity linking technology determines that these mentions correspond to the entities "Jinan" and "Shandong" in the knowledge base, respectively. This invention mainly uses word vectorization tools, such as Wikipedia2vec, to obtain the vector representation E of this entity in the knowledge base. wikipedia (Jinan), E wikipedia (shandong). Figure 1 The entities input into the EntityEmbedding Module are “Jin”, “##an”, and “Shandong”.

[0087] Wikipedia2vec is a toolkit that maps text or entities to a vector space with customizable dimensions. For each entity in the knowledge base, there is a pre-trained entity representation. Wikipedia2vec works similarly to Word2Vec, but it is trained on the Wikipedia corpus and incorporates words, entities, and anchor text during training. It uses a single-layer neural network to predict word neighbors to capture word similarity, predict relationships between entities based on neighboring entities, and predict word-to-entity mappings based on entity anchor text. Therefore, Wikipedia2vec can map words and entities to the same vector space. Given a vocabulary... and an entity table There is a mapping function E wikipedia : Map words and entities to the same vector space.

[0088] However, the entity vectors encoded by Wikipedia2vec and the pre-trained language model are not in the same space. A transformation rectangle needs to be learned to transform the entity vectors of Wikipedia2vec into the space of the text encoding model's embedding layer, as shown in Equation 1, resulting in a linear transformation matrix. d PLMd represents the vector dimension in the text encoding model. Wikipedia The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity x in the dictionary is represented by a vector E obtained through the word vectorization tool Wikipedia2vec. wikipedia (x), after linear transformation, yields W·E wikipedia (x), the vector representation E obtained through the embedding layer of the text encoding model. PLM (x); the transformation matrix W such that the transformed vector W·E of all x is such that the vector W·E of all x is transformed. wikipedia (x) and E PLM The sum of the squares of the difference of (x) is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the text encoding model embedding layer is located;

[0089]

[0090] The mapping function is obtained, resulting in the vector representation E after both words and entities are mapped to the input vector space of the text encoding model. Entity For those not in the entity table The words in the text are directly represented by vectors E obtained from the embedding layer of the text encoding model. PLM (x);

[0091]

[0092] Context-dependent entity information encoding. After obtaining the static vector representation of the entity, we want the entity adapter to adapt the entity vector representation according to different contexts to obtain a better entity semantic representation. Its input includes, in addition to, the external entity vector representation module E Entity In addition to the obtained entity vector representation, it also includes the hidden states output by the intermediate layers of the pre-trained language model used to encode the text. The structure of the entity adapter layer is as follows: Figure 2 As shown, each layer contains two projection layers and N transformer layers. There is a residual link between the two projection layers. The entire entity adapter has K entity adapter layers, which can take the states of the K intermediate layers of the text encoding model as input.

[0093] PLM-h j Let [the hidden state output by the j-th intermediate layer of the text encoding model, with shape [BatchSize, TextMaxLen, d]]. hidden BatchSize is the number of samples in the current training batch, TextMaxLen is the maximum number of text tokens in the text encoder, and dhidden For the hidden layer dimension of the text encoder;

[0094] h i This represents the output of the i-th entity adapter, which is of shape [BatchSize, 1 + EntityLen, d]. hidden The tensor is ], where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is h0, the 0th position h0[:, 0,:] corresponds to a vector of all zeros, and the kth position h0[:, k,:] corresponds to the initial vector representation of the kth entity;

[0095] For the i-th entity adapter layer, the hidden state PLM-h output by the j-th intermediate layer [CLS] token of the text encoding model is calculated according to the following formula 3. j [:,0,:] is added to the 0th position h of the output of the entity adapter layer of layer i-1. i-1 [:,0,:];According to Formula 4, the obtained h i-1 As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from d... hidden Become d adapter According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimension, making the dimension increase from d. adapter Restore to d hidden and with h i-1 Add them together to form a residual connection;

[0096] h i-1 [:,0,:]=h i-1 [:,0,:]+PLM-h j [:,0,:]#(3)

[0097]

[0098] y i =Transformer-N(x) i )#(5)

[0099]

[0100] As shown in Equation 7, the final output of the entity augmentation coding model is E. ent It is the output h of the last entity adaptation layer. K The encoding result corresponding to the 0th position in the code, where K is the number of entity adaptation layers.

[0101] E ent =h K [:,0,:]#(7)

[0102] The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector E encoded by the text encoding model based on the pre-trained language model. text and the second representation vector E ent , among which, [E text E ent ] indicates that E text and E ent The result of splicing, W g and b g denoted as the weights and biases of the gated network, respectively; σ represents the Sigmoid function; "·" represents the dot product operation; and ⊙ represents element-wise multiplication.

[0103]

[0104] E final =g⊙E text +(1-g)⊙E ent #(9)

[0105] g is a control weight with a value between 0 and 1.

[0106] As shown in Equation 9, the weight g is used to adjust E. text and E ent The final semantic representation result E of the target text is obtained by performing a weighted summation. final .

[0107] Training process:

[0108] Training data. The training set consists of several related text pairs, for example, one text pair might be: <Where is the capital of China? The capital of the People's Republic of China is Beijing.> Related document pairs can be obtained through manual annotation. For specific scenarios, such as search, the relevance between texts can also be determined based on user behavior.

[0109] During training, contrastive learning is employed, aiming to ensure that the vector representations of related texts are close in distance, while those of unrelated texts are far apart. For a given text, related texts are defined as positive samples, and unrelated texts as negative samples. Random sampling is typically used during training to obtain negative samples.

[0110] The loss function used is infoNCE loss:

[0111]

[0112] x i For a text, For is x i Positive samples, For x iA negative sample. f is a text encoding model, which takes text as input and outputs a vector representation of the text. Here, the text encoding model is the entity-enhanced text encoding model proposed in this invention. sim is a similarity calculation function, usually a dot product similarity or cosine similarity.

[0113] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0114] This invention also proposes a text semantic representation system based on entity enhancement, which includes:

[0115] Module 1: Obtain the target text to be semantically represented, and obtain the text encoding model and entity augmentation encoding model. The entity augmentation encoding model consists of a concatenated external entity vector representation module and a context-dependent entity information encoding module; extract the first representation vector of the target text through the text encoding model.

[0116] Module 2: The external entity vector representation module identifies the sequence of terms representing entities in the target text, determines which entities in the knowledge base are associated with the sequence of terms, and obtains the entity vector representation of the entity corresponding to the sequence of terms in the knowledge base;

[0117] Module 3: The entity information encoding module includes multiple cascaded entity adapter layers. The input of the entity information encoding module is the vector representation of the entity in the target text and the hidden state of the intermediate layer of the text encoding model. The input of each entity adapter layer is the hidden state output by the corresponding intermediate layer of the text encoding model and the output of the previous entity adaptation layer. The feature output by the last entity adapter layer is used as the second representation vector of the target text.

[0118] Module 4: By using the entity information gating unit, the first representation vector and the second representation vector are fused to obtain the semantic representation result of the target text.

[0119] The aforementioned entity-enhanced text semantic representation system, wherein

[0120] Module 2 includes:

[0121] Named entity recognition identifies sequences of terms representing entities in the target text. Entity links determine the corresponding entities in the knowledge base. For each word or entity x, the vector representation E of the entity in the knowledge base is obtained using the word vectorization tool Wikipedia2vec. wikipedia (x);

[0122] Given vocabulary and entity table Mapping function E wikipedia : This maps words and entities to the same vector space;

[0123] According to Formula 1, a linear transformation matrix is ​​obtained. dpLM is the vector dimension in the text encoding model, d Wikipedia The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity x in the dictionary is represented by a vector E obtained through the word vectorization tool Wikipedia2vec. wikipedia (x), after linear transformation, yields W·E wikipedia (x), the vector representation E obtained through the embedding layer of the text encoding model. PLM (x); the transformation matrix W such that the transformed vector W·E of all x is such that the vector W·E of all x is transformed. wikipedia (x) and E PLM The sum of the squares of the difference of (x) is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the text encoding model embedding layer is located;

[0124]

[0125] Based on Formula 2, the mapping function is obtained, resulting in the vector representation E after both words and entities are mapped to the input vector space of the text encoding model. Entity For those not in the entity table The words in the text are directly represented by vectors E obtained from the embedding layer of the text encoding model. PLM (x);

[0126]

[0127] Module 3 includes:

[0128] PLM-h j Let [the hidden state output by the j-th intermediate layer of the text encoding model, with shape [BatchSize, TextMaxLen, d]]. hidden BatchSize is the number of samples in the current training batch, TextMaxLen is the maximum number of text tokens in the text encoder, and d hidden For the hidden layer dimension of the text encoder;

[0129] h iThis represents the output of the i-th entity adapter, which is of shape [BatchSize, 1 + EntityLen, d]. hidden The tensor is ], where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is h0, the 0th position h0[:, 0,:] corresponds to a vector of all zeros, and the kth position h0[:, k,:] corresponds to the initial vector representation of the kth entity;

[0130] For the i-th entity adapter layer, the hidden state PLM-h output by the j-th intermediate layer [CLS] token of the text encoding model is calculated according to the following formula 3. j [:,0,:] is added to the 0th position h of the output of the entity adapter layer of layer i-1. i-1 [:,0,:];According to Formula 4, the obtained h i-1 As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from d... hidden Become d adapter According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimension, making the dimension increase from d. adapter Restore to d hidden and with h i-1 Add them together to form a residual connection;

[0131] h i-1 [:,0,:]=h i-1 [:,0,:]+PLM-h j [:,0,:] #(3)

[0132]

[0133] y i =Transformer-N(x) i #(5)

[0134]

[0135] As shown in Equation 7, the final output of the entity augmentation coding model is E. ent It is the output h of the last entity adaptation layer. K The encoding result corresponding to the 0th position in the code, where K is the number of entity adaptation layers.

[0136] E ent =h K [:,0,:]#(7)

[0137] The aforementioned entity-enhanced text semantic representation system, wherein module 4 includes:

[0138] The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector E encoded by the text encoding model based on the pre-trained language model. text and the second representation vector E ent , among which, [E text E ent ] indicates that E text and E ent The result of splicing, W g and b g denoted as the weights and biases of the gated network, respectively; σ represents the Sigmoid function; "·" represents the dot product operation; and ⊙ represents element-wise multiplication.

[0139]

[0140] E final =g⊙E text +(1-g)⊙E ent #(9)

[0141] g is a control weight with a value between 0 and 1.

[0142] As shown in Equation 9, the weight g is used to adjust E. text and E ent The final semantic representation result E of the target text is obtained by performing a weighted summation. final .

[0143] The entity-enhanced text semantic representation system includes module 4, which includes: performing text retrieval or text similarity judgment tasks based on the semantic representation extraction results.

[0144] The present invention also proposes a storage medium for storing a program that executes any of the entity-enhanced text semantic representation methods.

[0145] The present invention also proposes a client for any entity-enhanced text semantic representation system.

Claims

1. A text semantic representation method based on entity enhancement, characterized in that, include: Step 1: Obtain the target text to be semantically represented, and obtain the text encoding model and entity augmentation encoding model. The entity augmentation encoding model consists of a concatenated external entity vector representation module and a context-dependent entity information encoding module; extract the first representation vector of the target text through the text encoding model. Step 2: Use the external entity vector representation module to identify the word sequence representing entities in the target text, determine which entities in the knowledge base are associated with the word sequence, and obtain the entity vector representation of the entity corresponding to the word sequence in the knowledge base; Step 3: The entity information encoding module includes multiple cascaded entity adapter layers. The input of the entity information encoding module is the vector representation of the entity in the target text and the hidden state of the intermediate layer of the text encoding model. The input of each entity adapter layer is the hidden state output by the corresponding intermediate layer of the text encoding model and the output of the previous entity adaptation layer. The feature output by the last entity adapter layer is used as the second representation vector of the target text. Step 4: By using the entity information gating unit, fuse the first representation vector and the second representation vector to obtain the semantic representation result of the target text; Step 2 includes: Named entity recognition identifies sequences of terms representing entities in the target text. Entity links are used to determine the corresponding entities in the knowledge base for each sequence of terms. The vector representation of entities in the knowledge base is obtained through the word vectorization tool Wikipedia2vec. ; Given vocabulary and entity table Mapping function This is to map words and entities to the same vector space; According to Formula 1, a linear transformation matrix is ​​obtained. , For the vector dimension in the text encoding model, The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity in It uses vector representations obtained from the word vectorization tool Wikipedia2vec. After linear transformation, we obtain Vector representation obtained through the embedding layer of the text encoding model Transformation matrix The sum of squares of the L2 norm is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the embedding layer of the text encoding model is located; According to Formula 2, the mapping function is obtained, resulting in vector representations of both words and entities mapped to the input vector space of the text encoding model. For those not in the entity table The words in the text are directly represented by vectors obtained from the embedding layer of the text encoding model. ; Step 3 includes: Let J be the hidden state output by the j-th intermediate layer of the text encoding model, with shape J. BatchSize is the number of samples in the current training batch. This represents the maximum number of text tokens that can be encoded in a text encoder. For the hidden layer dimension of the text encoder; This represents the output of the i-th entity adapter, which is of shape... The tensor is where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is , Position 0 The corresponding vector of all zeros, the first Location The corresponding number is For the i-th entity adapter layer, the hidden state output by the j-th intermediate layer [CLS] token of the text encoding model is determined according to Formula 3 below. Add to the 0th position of the output of the entity adapter layer at layer i-1 According to Formula 4, the obtained As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from... become According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimensionality, making the dimensionality from... Restore to and with Add them together to form a residual connection; As shown in Equation 7, the final output of the entity augmentation coding model is: It is the output of the last entity adaptation layer. The encoding result corresponding to the 0th position in the middle, where The number of entity adaptation layers; .

2. The text semantic representation method based on entity enhancement as described in claim 1, characterized in that, Step 4 includes: The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector encoded by the text encoding model based on the pre-trained language model. and the second representation vector ,in, Indicates will and The result of splicing. and Let represent the weights and biases of the gated network, respectively, and let represent the sigmoid function. " indicates the dot product operation. ; 𝑔 represents the control weight, with values ​​between 0 and 1; As shown in Formula 9, using weights 𝑔 and The final semantic representation of the target text is obtained by performing a weighted summation. .

3. The text semantic representation method based on entity enhancement as described in claim 1, characterized in that, Step 4 includes: using the semantic representation to extract the results and performing text retrieval or text similarity judgment tasks.

4. A text semantic representation system based on entity enhancement, characterized in that, include: Module 1: Obtain the target text to be semantically represented, and obtain the text encoding model and entity augmentation encoding model. The entity augmentation encoding model consists of a concatenated external entity vector representation module and a context-dependent entity information encoding module; extract the first representation vector of the target text through the text encoding model. Module 2: The external entity vector representation module identifies the sequence of terms representing entities in the target text, determines which entities in the knowledge base are associated with the sequence of terms, and obtains the entity vector representation of the entity corresponding to the sequence of terms in the knowledge base; Module 3: The entity information encoding module includes multiple cascaded entity adapter layers. The input of the entity information encoding module is the vector representation of the entity in the target text and the hidden state of the intermediate layer of the text encoding model. The input of each entity adapter layer is the hidden state output by the corresponding intermediate layer of the text encoding model and the output of the previous entity adaptation layer. The feature output by the last entity adapter layer is used as the second representation vector of the target text. Module 4: By fusing the first representation vector and the second representation vector through the entity information gating unit, the semantic representation result of the target text is obtained; Module 2 includes: Named entity recognition identifies sequences of terms representing entities in the target text. Entity links are used to determine the corresponding entities in the knowledge base for each sequence of terms. The vector representation of entities in the knowledge base is obtained through the word vectorization tool Wikipedia2vec. ; Given vocabulary and entity table Mapping function This is to map words and entities to the same vector space; According to Formula 1, a linear transformation matrix is ​​obtained. , For the vector dimension in the text encoding model, The dimension of the entity vector encoded by Wikipedia2vec; for the defined vocabulary. and entity table Each word or entity in It uses vector representations obtained from the word vectorization tool Wikipedia2vec. After linear transformation, we obtain Vector representation obtained through the embedding layer of the text encoding model Transformation matrix The sum of squares of the L2 norm is minimized; through the linear transformation of Formula 1, the vector representations of words and entities obtained by Wikipedia2vec are mapped to the space where the embedding layer of the text encoding model is located; According to Formula 2, the mapping function is obtained, resulting in vector representations of both words and entities mapped to the input vector space of the text encoding model. For those not in the entity table The words in the text are directly represented by vectors obtained from the embedding layer of the text encoding model. ; Module 3 includes: Let J be the hidden state output by the j-th intermediate layer of the text encoding model, with shape J. BatchSize is the number of samples in the current training batch. This represents the maximum number of text tokens that can be encoded in a text encoder. For the hidden layer dimension of the text encoder; This represents the output of the i-th entity adapter, which is of shape... The tensor is where "1" corresponds to the position where the output of the text encoding model is stored; EntityLen is the number of entities; for the first entity adapter, its input is , Position 0 The corresponding vector of all zeros, the first Location The corresponding number is For the i-th entity adapter layer, the hidden state output by the j-th intermediate layer [CLS] token of the text encoding model is determined according to Formula 3 below. Add to the 0th position of the output of the entity adapter layer at layer i-1 According to Formula 4, the obtained As input, it first passes through a projection layer for dimensionality reduction, reducing the dimension from... become According to Formula 5, the data is fed into N transformer layers for encoding, and then, according to Formula 6, a projection layer is used to increase the dimensionality, making the dimensionality from... Restore to and with Add them together to form a residual connection; As shown in Equation 7, the final output of the entity augmentation coding model is: It is the output of the last entity adaptation layer. The encoding result corresponding to the 0th position in the middle, where The number of entity adaptation layers; 。 5. The text semantic representation system based on entity enhancement as described in claim 4, characterized in that, Module 4 includes: The control weights of the physical gating unit are calculated as shown in Equation 8. The input is the first representation vector encoded by the text encoding model based on the pre-trained language model. and the second representation vector ,in, Indicates will and The result of splicing. and Let represent the weights and biases of the gated network, respectively, and let represent the sigmoid function. " indicates the dot product operation. ; 𝑔 represents the control weight, with values ​​between 0 and 1; As shown in Formula 9, using weights 𝑔 and The final semantic representation of the target text is obtained by performing a weighted summation. .

6. The text semantic representation system based on entity enhancement as described in claim 4, characterized in that, Module 4 includes: performing text retrieval or text similarity judgment tasks based on the results extracted using the semantic representation.

7. A storage medium for storing a program that executes the entity-enhanced text semantic representation method according to any one of claims 1 to 3.

8. A client for the entity-enhanced text semantic representation system of any one of claims 4 to 6.

Images