A professional field term entity word vector self-correction method, system and device
By fine-tuning the pre-trained language model and constructing a domain terminology knowledge graph, the problem of polysemy in word vectors under different contexts was solved, generating entity word vectors containing semantic and structural information of the professional domain, thus improving the semantic learning ability of word vectors.
Patent Information
- Application Number
- CN202310147442.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-20
- Publication Date
- 2026-01-23
- Estimated Expiration
- 2043-02-20
AI Technical Summary
Existing technologies cannot effectively address the polysemy of the same word in different contexts when generating word vectors, and they lack information on the vocabulary and relational structures of professional domain terms.
By fine-tuning the pre-trained language model, encoding the entity explanation text of domain terms, constructing a domain terminology knowledge graph, and correcting the semantic word vectors of term entities through the entity relationship matrix, entity word vectors containing professional domain semantic and structural information are generated.
It effectively generates entity word vector sets containing semantic and structural information of professional fields, solves the problem of polysemy of word vectors in different contexts, and improves the semantic learning ability of word vectors.
Smart Images
Figure CN116383398B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and apparatus for self-correction of entity word vectors in professional terminology. Background Technology
[0002] Currently, in the field of natural language processing under artificial intelligence, natural language text corpora are segmented at different granularities, and each segmented sub-word can be converted into word vectors through machine learning or deep learning methods, realizing the conversion from text to mathematical language.
[0003] The conversion from text corpora to word vectors is a crucial step in enabling machines to understand and learn human information and knowledge. Traditional conversion methods mainly utilize bag-of-words models such as One-Hot, TF-IDF, and TextRank to achieve this conversion. Bag-of-words models segment the entire text into word units, ignoring word order and embedding each word into a high-dimensional vector, thus representing an entire article as a high-dimensional sparse matrix. This approach suffers from the curse of dimensionality and completely ignores the contextual information of each word within the specific text, as it only segments the text by word. The emergence of fixed-representation word vector models such as Word2vec, GloVe, and FastText has addressed this problem to some extent. Word2vec and FastText primarily use local text corpora and a sliding window mechanism to extract text features and generate word vectors; GloVe uses the entire text corpus and constructs a co-occurrence matrix to generate word vectors. These methods have significantly improved computational efficiency and semantic information learning in word vectors. However, the set of word vectors generated by these models is fixed and cannot address the issue of the same word having multiple meanings in different contexts.
[0004] With the emergence of large-scale pre-trained language models such as BERT and GPT, the ability to truly dynamically learn semantic information has been realized. These models can continuously adjust the vector distribution of each word in the semantic space during the learning process, allowing each word vector to learn different semantics based on different contexts. However, language models such as Word2vec, GloVe, and BERT generate word vectors on public, open-source datasets, and these models only learn textual semantic information from the training text corpus, without learning information such as the relationships between words. In 2018, Google used the structural information between concept nodes in the ConceptNet semantic network to correct the semantic word vector sets provided by Word2vec and GloVe. However, semantic networks like ConceptNet and WordNet are constructed manually on open-source datasets. If directly applied to certain specialized fields, they may fail to achieve the goal of word vector correction due to the lack of domain-specific terminology and the relationships between them. Summary of the Invention
[0005] The purpose of this invention is to provide a method, system, and apparatus for self-correction of entity vectors of professional terminology, so as to solve at least one of the above-mentioned technical problems existing in the prior art.
[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention provides a method comprising:
[0007] Step 1: Through the downstream relation extraction task, fine-tune the pre-trained language model and encode the domain term entity explanation text to obtain the domain term entity semantic word vectors;
[0008] Step 2: Construct a domain terminology knowledge graph using a relation extraction model, and then construct an entity relation matrix using the domain terminology knowledge graph;
[0009] Step 3: Correct the semantic word vectors of the domain terminology entities through the entity relationship matrix, and output the entity word vector set containing professional domain semantic and structural information.
[0010] In one feasible embodiment, step 1 includes:
[0011] Step 11: Explain the data features of the text using domain terminology and define the relationships between domain terms;
[0012] Step 12: Mark the original corpus of the domain terminology explanation text. In the original corpus, mark the start and end positions of the head entity with the first special symbol and the start and end positions of the tail entity with the second special symbol. Patch the relationship between the head entity and the tail entity in each corpus to the beginning of the sentence and add a third special symbol as the relationship label for that corpus.
[0013] The corpus is segmented at the character level using a pre-trained language model's word segmenter. Each explanatory text sequence is then concatenated into an input sequence of a preset character length. If the original text sequence exceeds the preset character length, it is truncated; if the original text sequence is shorter than the preset character length, it is padded with "0". A mask sequence is constructed based on the positions of the beginning and end entities, where the mask sequence is "1" at the corresponding entity position and "0" at the non-entity position, facilitating the subsequent extraction of term entity word vectors.
[0014] Step 13: Input the processed input sequence into the pre-trained language model. The model converts each input sequence into a vector matrix and encodes it through multiple layers of self-attention mechanism to obtain the vector representation of each word. Since the sentence vector contains the overall semantics of the text sequence, the sentence vector representation and term entity pair vector are extracted, and after being mapped to the same semantic space through several fully connected layers, they are concatenated and input into the fully connected layer, and then connected to the SoftMax layer to predict the relationship.
[0015] Step 14: After multiple rounds of training, once the loss function value for the model relation extraction task converges to a stable value, save the model. Encode domain terminology entities using the fine-tuned pre-trained language model, and output the vector representations of the first and last entities based on the entity mask sequence to obtain the terminology entity word vectors for that domain.
[0016] In one feasible embodiment, the method for defining domain terms in step 11 is as follows: based on an existing domain terminology knowledge graph, relationships between existing domain terms are searched in the existing domain terminology knowledge graph through remote supervision. Specifically, since this method may introduce a large amount of noisy data, it is necessary to manually determine whether the relationship conforms to the actual relationship between two entities in the domain terminology explanation text: if it does, the relationship is retained; if it does not, the relationship is discarded.
[0017] In one feasible embodiment, the method for defining domain terms in step 11 can also be: based on keywords of explicit relationships, such as "composed of," "includes," "...is also called," "...is subordinate to," "...is also known as," etc., to define five types of relationships: genus-species relationship, whole-part relationship, synonym relationship, antonymous relationship, and other relationships.
[0018] In one feasible embodiment, step 2 includes:
[0019] Step 21: Based on the original corpus, construct a unique index corresponding to each piece of original corpus. Input the index and the original corpus together into the relation extraction model, but do not participate in the training. The index is used to connect the relation labels output by the model with the term entity pairs in the original corpus mapped by this index to form a triple, such as (A, R, B), where A represents the head entity, R represents the relation label output by the model, and B represents the tail entity.
[0020] Step 22: Connect several triples into a domain terminology knowledge graph through relations, assign relation weights to obtain a weight matrix. For example, synonym relations are assigned 3, genus-species relations are assigned 2, whole-part relations are assigned 1, other relations are assigned 0, and antonyms are assigned -1, as shown in the table below:
[0021] Head Entity Tail entity relation Assignment forward shooter Synonyms 3.0 Team reserve forces Team strength Genus and species 2.0 Penalty kick Set piece Overall Part 1.0 Team Logistics Lawn maintenance other 0.0 attack Defense Antonyms -1.0
[0022] Step 23: Construct an entity relationship matrix based on the domain terminology knowledge graph. Specifically, an N*N adjacency matrix is constructed using N terminology nodes. The value of each row in the adjacency matrix represents the path information from a terminology node to other nodes in the graph: for terminology nodes directly connected to the current node, the matrix value is the sum of the weights of all relationships between these two nodes; for terminology nodes not directly connected to the current node, the matrix value is 0. The norm normalization method is used to ensure that the absolute sum of all weights in each row is 1, thus normalizing the weights from a node to the other nodes in the graph. The specific formula can be:
[0023]
[0024] x i-new =x i / ||x||1;
[0025] Where, x i This represents the i-th value in a row;
[0026] The PPMI positive point mutual information between term nodes is calculated based on each weight value. The specific formula is as follows:
[0027]
[0028] PPMI = max{0, PMI}
[0029] PMI(x, y) represents the point mutual information value, which is used to measure the correlation between two random variables. When two things (x, y) are completely unrelated, that is, when the probability p(x, y) of the two things occurring at the same time is 0, PMI(x, y) will tend to negative infinity. To avoid this situation, positive point mutual information PPMI is usually used for calculation.
[0030] After calculating the mutual information of the positive points, the weight matrix between term nodes is transformed into an entity relationship matrix. The normalized weight sum between each term node in the matrix is then transformed into the point mutual information value between the nodes, which can transform the simple weight sum into an intuitive relevance.
[0031] In one feasible embodiment, after obtaining the entity relationship matrix, a dimensionality reduction process is further included to remove invalid information. Specifically, the entity relationship matrix is decomposed into a left singular matrix U (m*m dimensional square matrix), a singular value diagonal matrix ∑ (m*n dimensional, with values only on the diagonal and all others being 0) and a right singular matrix V (n*n dimensional square matrix) by using SVD singular values. The first K singular values are selected, and the left singular matrix U is reduced from m*m dimensional to m*k dimensional, and the right singular matrix V is reduced from n*n dimensional to k*n dimensional.
[0032] This method allows for significant compression of the entity relationship matrix. The underlying principle is that in the singular value diagonal matrix ∑, the singular values are arranged from largest to smallest and decay rapidly. In most cases, the sum of the first 10%, or even 1%, of the singular values accounts for 99% of the total. Therefore, by retaining only the first portion of the singular values and their corresponding left and right singular matrices, and discarding the rest of the data, dimensionality reduction is achieved. This discards invalid information and compresses the matrix size. The specific formula is as follows:
[0033] A=U∑V T ;
[0034] U = AA T V = A T A ∑ i =AV i / U i i∈(0,k)
[0035] m, n→k 0<k<min(A.shape).
[0036] In one feasible embodiment, step 3 includes:
[0037] Step 31: Predict a new vector for each domain term entity semantic word vector. This new vector should be close to the original semantic vector in the semantic space, and also close to relevant neighbor nodes in the domain term knowledge graph based on the structural information of the terms in the entity relation matrix. The distance between the new vector and its neighbor nodes is defined as the Euclidean distance. The specific formula can be:
[0038]
[0039] in, Let the objective function of the correction process be defined, wherein the correction process is to continuously minimize the objective function. The entity semantic word vector representing term i; qi Represents a new vector; q j It is q i In the domain terminology knowledge graph, all neighboring nodes have a relationship (i, j) ∈ E; E represents a certain relationship between term i and term j. and β ij All are vector calibration constants;
[0040] Step 32, for Taking the first derivative and setting it to zero, we obtain each q. j For one iteration update, the formula can be:
[0041]
[0042] Step 33: Perform multiple iterations until the iteration termination condition is met, which forces the new vector to continuously move closer to the original semantic vector and neighboring nodes, and the Euclidean distance continuously decreases iteratively.
[0043] Through the above steps, each iteration calculates the sum of the dot product between the structural information of each domain term in the entity relation matrix and all semantic vectors, thereby injecting the structural information of each domain term node into the semantic vector of the domain term.
[0044] Secondly, based on the same inventive concept, this application also provides a self-correction system for entity vectors of professional domain terminology, including: a data acquisition module, a data processing module, and a result generation module.
[0045] The data acquisition module is used to collect raw corpora of domain terminology;
[0046] The data processing module includes a preprocessing unit, a model training unit, a relation extraction unit, a semantic vector encoding unit, a terminology graph construction unit, a structure matrix construction unit, and a correction unit.
[0047] The preprocessing unit labels the original corpus of domain terms and constructs a training set for the relation extraction model. The labeling format can be: relation label + third special symbol + domain term explanation text; the domain term explanation text includes a head entity with a first special symbol and a tail entity with a second special symbol.
[0048] The model training unit trains the relation extraction model based on the training set to obtain the trained relation extraction model.
[0049] The relation extraction unit, based on the original corpus of domain terms, extracts the relations corresponding to the term entity pairs through the trained relation extraction model to obtain triples.
[0050] The semantic vector encoding unit is used to obtain term entity word vectors from the encoding layer of the relation extraction model;
[0051] The terminology graph construction unit is used to connect the triples through relations to form a domain terminology knowledge graph;
[0052] The structural matrix construction unit is used to record and transform the structural information in the domain terminology knowledge graph into an entity relationship matrix.
[0053] The correction unit corrects the term entity word vectors using the entity relationship matrix.
[0054] The result generation module is used to organize the corrected term entity vectors into an entity vector set and send it out.
[0055] Thirdly, based on the same inventive concept, this application also provides a professional domain terminology entity vector self-correction device, including a processor, a memory, and a bus. The memory stores instructions and data that can be read by the processor. The processor is used to call the instructions and data in the memory to execute the professional domain terminology entity vector self-correction method as described above. The bus connects the functional components to transmit information.
[0056] By adopting the above technical solution, the present invention has the following beneficial effects:
[0057] This invention provides a method, system, and apparatus for self-correcting entity word vectors of professional domain terminology. It utilizes a trained relation extraction model to extract relations, mine the relationships between entities in professional domain terminology, construct a domain terminology knowledge graph, and correct the terminology entity word vectors through an entity relation matrix, thereby conveniently and effectively generating a set of entity word vectors with professional knowledge in that domain. Attached Figure Description
[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0059] Figure 1 A flowchart of a professional domain terminology entity vector self-correction method provided in this embodiment of the invention;
[0060] Figure 2 Example diagram of the marked original corpus provided in the embodiments of the present invention;
[0061] Figure 3This is a structural diagram of the Chinese BERT autoencoder pre-trained language model provided in an embodiment of the present invention;
[0062] Figure 4 An explanatory diagram illustrating the extraction of semantic word vectors of domain terminology entities using the relation extraction model provided in this embodiment of the invention;
[0063] Figure 5 An illustrative diagram illustrating the construction of an entity relationship matrix based on the domain terminology knowledge graph provided in this embodiment of the invention;
[0064] Figure 6 This is an illustrative diagram illustrating the dimensionality reduction of the entity relationship matrix using SVD, provided in an embodiment of the present invention.
[0065] Figure 7 An explanatory diagram illustrating the prediction of new vectors from semantic word vectors for each domain term entity, provided for embodiments of the present invention;
[0066] Figure 8 An explanatory diagram illustrating the correction of term entity word vectors using an entity relation matrix, provided for embodiments of the present invention;
[0067] Figure 9 A diagram of a professional domain terminology entity vector self-correction system provided for embodiments of the present invention. Detailed Implementation
[0068] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0069] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0070] The present invention will be further explained below with reference to specific embodiments.
[0071] like Figure 1 As shown, to solve the above-mentioned technical problems, the method provided by the embodiments of the present invention includes:
[0072] Step 1: Through the downstream relation extraction task, fine-tune the pre-trained language model - such as the Chinese BERT model, and encode the domain term entity explanation text to obtain the domain term entity semantic word vectors;
[0073] Step 2: Construct a domain terminology knowledge graph using a relation extraction model, and then construct an entity relation matrix using the domain terminology knowledge graph;
[0074] Step 3: Correct the semantic word vectors of domain terminology entities through the entity relationship matrix, and output a set of entity word vectors containing professional domain semantic and structural information.
[0075] Further, step 1 includes:
[0076] Step 11: Explain the data features of the text using domain terminology and define the relationships between domain terms;
[0077] Step 12: Mark the original corpus of the domain terminology explanation text. In the original corpus, use the first special symbol "&" to mark the start and end positions of the head entity, and the second special symbol "$" to mark the start and end positions of the tail entity. Concatenate the relationship between the head and tail entities in each corpus to the beginning of the sentence and add the third special symbol "\t" as the relationship label for that corpus. For example... Figure 2 As shown;
[0078] The corpus is segmented at the character level using the Chinese BERT model's word segmenter. Each explanatory text sequence is then concatenated into a 384-character input sequence. If the original text sequence exceeds 384 characters, it is truncated; if it is less than 384 characters, it is padded with zeros. A mask sequence is constructed based on the positions of the beginning and end entities. The mask sequence uses "1" at entity-corresponding positions and "0" at non-entity-corresponding positions to facilitate subsequent extraction of terminology entity word vectors, such as... Figure 3 As shown;
[0079] Step 13: Input the processed input sequence into the Chinese BERT model. The model converts each input sequence into a 384*768 vector matrix, encodes it through multiple layers of self-attention mechanism to obtain the vector representation of each sub-word; since the sentence vector contains the overall semantics of the text sequence, the sentence vector representation [CLS] and the term entity pair vectors [A] and [B] are extracted, and after being mapped to the same semantic space through three fully connected layers, they are concatenated to form a 1*2304 vector; after being input into a 2304*5 fully connected layer, it is then connected to the SoftMax layer to predict the relationship;
[0080] Step 14: After multiple rounds of training, when the loss function value of the model relation extraction task converges to a stable value F1, and the F1 value reaches its maximum value, save the model. Encode domain terminology entities using the fine-tuned model, and output the vector representations of the head and tail entities from the BERT encoding layer based on the entity mask sequence to obtain the terminology entity word vectors for that domain, such as... Figure 4 As shown.
[0081] Furthermore, the method for defining domain terms in step 11 is as follows: based on an existing domain terminology knowledge graph, relationships between existing domain terms are searched in the existing domain terminology knowledge graph through remote supervision. Specifically, since this method may introduce a large amount of noisy data, it is necessary to manually determine whether the relationship conforms to the actual relationship between two entities in the domain terminology explanation text: if it does, the relationship is retained; if it does not, the relationship is discarded.
[0082] Furthermore, the method for defining domain terms in step 11 can also be: based on keywords representing relationships, such as "composed of," "includes," "...is also called," "...is subordinate to," "...is also known as," etc., define five types of relationships: genus-species relationship, whole-part relationship, synonym relationship, antonymous relationship, and other relationships.
[0083] Further, step 2 includes:
[0084] Step 21: Based on the original corpus, construct a unique index corresponding to each piece of original corpus. Input the index and the original corpus together into the relation extraction model, but do not participate in the training. The index is used to connect the relation labels output by the model with the term entity pairs in the original corpus mapped by this index to form a triple, such as (A, R, B), where A represents the head entity, R represents the relation label output by the model, and B represents the tail entity.
[0085] Step 22: Combine N different triples (A i R j B i (i∈N, j∈{0,1,2,3,4}) are connected through relations to form a domain terminology knowledge graph. Weights are assigned to different relations to obtain a weight matrix: synonyms are assigned a value of 3, genus-species relations are assigned a value of 2, whole-part relations are assigned a value of 1, other relations are assigned a value of 0, and antonyms are assigned a value of -1, as shown in the table below:
[0086] Head Entity Tail entity relation Assignment forward shooter Synonyms 3.0 Team reserve forces Team strength Genus and species 2.0 Penalty kick Set piece Overall Part 1.0 Team Logistics Lawn maintenance other 0.0 attack Defense Antonyms -1.0
[0087] Step 23: Construct an entity relationship matrix based on the domain terminology knowledge graph. Specifically, an N*N adjacency matrix is constructed using N terminology nodes. The value of each row in the adjacency matrix represents the path information from a terminology node to other nodes in the graph: for terminology nodes directly connected to the current terminology node, the matrix value is the sum of the weights of all relationships between these two nodes; for terminology nodes not directly connected to the current terminology node, the matrix value is 0. Figure 5 As shown. Using the L1 norm normalization method, the sum of the absolute values of all weights in each row is made equal to 1. This normalizes the weights from a given node to the other nodes in the graph. The specific formula is as follows:
[0088]
[0089] x i-new =x i / ||x||1;
[0090] Where, x i This represents the i-th value in a row;
[0091] The PPMI positive point mutual information between term nodes is calculated based on each weight value. The specific formula is as follows:
[0092]
[0093] PPMI = max{0, PMI}
[0094] PMI(x, y) represents the point mutual information value, which is used to measure the correlation between two random variables. When two things (x, y) are completely unrelated, that is, when the probability p(x, y) of the two things occurring at the same time is 0, PMI(x, y) will tend to negative infinity. To avoid this situation, positive point mutual information PPMI is usually used for calculation, that is, "0" is used instead of "-∞".
[0095] After calculating the mutual information of the positive points, the weight matrix between term nodes is transformed into an entity relationship matrix. The normalized weight sum between each term node in the matrix is then transformed into the point mutual information value between the nodes, which can transform the simple weight sum into an intuitive relevance.
[0096] Furthermore, after calculating the mutual information at the positive points, dimensionality reduction processing is included to remove invalid information. Specifically, this process involves decomposing the entity relationship matrix into a left singular matrix U (m*m dimensional square matrix), a singular value diagonal matrix ∑ (m*n dimensional, with values only on the diagonal and all others being 0), and a right singular matrix V (n*n dimensional square matrix) using SVD singular value decomposition. Selecting the first K singular values reduces the left singular matrix U from m*m dimensional to m*k dimensional, and the right singular matrix V from n*n dimensional to k*n dimensional, as shown below. Figure 6 As shown.
[0097] This method allows for significant compression of the entity relationship matrix. The underlying principle is that in the singular value diagonal matrix ∑, the singular values are arranged from largest to smallest and decay rapidly. In most cases, the sum of the first 10%, or even 1%, of the singular values accounts for 99% of the total. Therefore, by retaining only the first portion of the singular values and their corresponding left and right singular matrices, and discarding the rest of the data, dimensionality reduction is achieved. This discards invalid information and compresses the matrix size. The specific formula is as follows:
[0098] A=U∑V T ;
[0099] U = AA T V = A T A ∑ i =AV i / U i i∈(0,k)
[0100] m, n→k 0<k<min(A.shape).
[0101] Further, step 3 includes:
[0102] Step 31: Predict a new vector for each domain term entity semantic word vector. This new vector should be close to the original semantic vector in the semantic space, and also close to relevant neighbor nodes in the domain term knowledge graph based on the structural information of the terms in the entity relation matrix. The distance between the new vector and its neighbor nodes is defined as the Euclidean distance, such as... Figure 7 As shown. The specific formula can be:
[0103]
[0104] in, Let the objective function of the correction process be defined, wherein the correction process is to continuously minimize the objective function. The entity semantic word vector representing term i; q i Represents a new vector; q j It is q i In the domain terminology knowledge graph, all neighboring nodes have a relationship (i, j) ∈ E; E represents a certain relationship between term i and term j. and β ij All are vector calibration constants;
[0105] Step 32, for Taking the first derivative and setting it to zero, we obtain each q. j The one-time iteration update can be represented by the formula:
[0106]
[0107] Step 33: After multiple updates and iterations, the iteration termination condition is met. This forces the new vector to continuously move closer to the original semantic vector and its neighboring nodes, thus the Euclidean distance continuously decreases iteratively.
[0108] Furthermore, the iteration termination condition is that the Euclidean distance reaches a preset length or the number of iterations reaches a preset number.
[0109] Through the above steps, each iteration calculates the sum of the dot product between the structural information of each domain term in the entity relation matrix and all semantic vectors, thereby injecting the structural information of each domain term node into the semantic vector of the domain term. Figure 8 As shown.
[0110] On the other hand, such as Figure 9 As shown, this embodiment of the invention also provides a self-correction system for entity vectors of professional domain terminology, including: a data acquisition module, a data processing module, and a result generation module.
[0111] The data acquisition module is used to collect raw corpora of domain terminology;
[0112] The data processing module includes a preprocessing unit, a model training unit, a relation extraction unit, a semantic vector encoding unit, a terminology graph construction unit, a structure matrix construction unit, and a correction unit.
[0113] The preprocessing unit labels the original corpus of domain terms and constructs a training set for the relation extraction model. The specific labeling format can be: relation label + third special symbol + domain term explanation text; the domain term explanation text includes a head entity with a first special symbol and a tail entity with a second special symbol, for example: relation label + \t + term explanation text (&head entity&, ..., $tail entity$);
[0114] The model training unit trains the relation extraction model based on the training set to obtain the trained relation extraction model.
[0115] The relation extraction unit, based on the original corpus of domain terms, extracts the relations corresponding to the term entity pairs through the trained relation extraction model to obtain triples.
[0116] The semantic vector encoding unit is used to obtain term entity word vectors from the encoding layer of the relation extraction model;
[0117] The terminology graph construction unit is used to connect the triples through relations to form a domain terminology knowledge graph;
[0118] The structural matrix construction unit is used to record and transform the structural information in the domain terminology knowledge graph into an entity relationship matrix - Entity2Entity matrix;
[0119] The correction unit corrects the term entity word vectors using the entity relationship matrix.
[0120] The result generation module is used to organize the corrected term entity vectors into an entity vector set and send it out.
[0121] In another aspect, embodiments of the present invention also provide a self-correction device for entity vectors of professional domain terminology, including a processor, a memory, and a bus. The memory stores instructions and data that can be read by the processor. The processor is used to call the instructions and data in the memory to execute the self-correction method for entity vectors of professional domain terminology as described above. The bus connects the functional components to transmit information.
[0122] In another implementation, this solution can be implemented using a device, which may include corresponding modules that perform one or more steps in the various embodiments described above. A module may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.
[0123] The processor executes the various methods and processes described above. For example, the method implementations in this scheme can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some implementations, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other implementations, the processor can be configured to execute one of the methods described above by any other suitable means (e.g., by means of firmware).
[0124] This device can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus connects various circuits, including one or more processors, memory, and / or hardware modules. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuitry, external antennas, etc.
[0125] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Component (EISA) buses, etc. Buses can be divided into address buses, data buses, control buses, etc.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for self-correcting entity vectors of professional domain terminology, characterized in that, include: Step 1: Through the downstream relation extraction task, fine-tune the pre-trained language model and encode the domain term entity explanation text to obtain the domain term entity semantic word vectors; Step 2: Construct a domain terminology knowledge graph using a relation extraction model, and then construct an entity relation matrix from the domain terminology knowledge graph; specifically including: Step 21: Based on the original corpus, construct an index corresponding to each piece of original corpus, input the index and the original corpus into the relation extraction model, output relation labels, and obtain a triple consisting of term entity pairs in the original corpus and the relation labels; Step 22: Connect several triples into a domain terminology knowledge graph through relations, and assign relation weights to obtain a weight matrix; Step 23: Construct an entity relationship matrix based on the domain terminology knowledge graph; construct an N*N adjacency matrix using N terminology nodes, where the value of each row in the adjacency matrix represents the path information from a terminology node to other nodes in the graph: for terminology nodes directly connected to the terminology node, the matrix value is the sum of the weights of all relationships between these two nodes; for terminology nodes not directly connected to the terminology node, the matrix value is 0; normalize the weights from a terminology node to other nodes in the graph by using the L1 norm normalization method, ensuring that the absolute sum of all weights in each row is 1. The specific formula is as follows: ； ； in, Indicates the first in a line One value; Indicates the total number of rows; Calculate the relationship between term nodes based on each weight value. The specific formula for point-to-point mutual information is as follows: ； ； in, This represents the point mutual information value, used to measure the correlation between two random variables; Indicates two things The probability of them occurring simultaneously; Indicates two things Edge counting; transforming the weight matrix into an entity relationship matrix through point mutual information calculation; Step 3: Correct the semantic word vectors of the domain terminology entities through the entity relationship matrix, and output the entity word vector set containing professional domain semantic and structural information.
2. The method according to claim 1, characterized in that, Step 1 includes: Step 11: Explain the data features of the text using domain terminology and define the relationships between domain terms; Step 12: Mark the original corpus of domain terminology explanation texts. In the original corpus, mark the start and end positions of the head entity using the first special symbol, and mark the start and end positions of the tail entity using the second special symbol. Concatenate the relationship between the head and tail entities in each corpus to the beginning of the sentence and add a third special symbol as the relationship label for that corpus. Use the word segmenter of the pre-trained language model to segment the corpus at the character level, and concatenate each explanation text sequence into an input sequence of a preset character length. Step 13: Input the processed input sequence into the pre-trained language model. The pre-trained language model converts each input sequence into a vector matrix, encodes it through multiple layers of self-attention mechanism, and obtains the vector representation of each sub-word. It extracts sentence vector representations and term entity pair vectors, maps them to the same semantic space through several fully connected layers, concatenates them, inputs them into the fully connected layer, and then connects them to the SoftMax layer to predict the relationship. Step 14: After multiple rounds of training, when the loss function value of the model relation extraction task converges to a stable value, save the pre-trained language model, encode domain term entities through the fine-tuned pre-trained language model, and output the vector representations of the head and tail entities according to the entity mask sequence to obtain the term entity word vectors of the domain.
3. The method according to claim 2, characterized in that, The method for defining the relationship between domain terms in step 11 includes: defining five types of relationships based on keywords that show the relationship: genus-species relationship, whole-part relationship, synonym relationship, antonymous relationship, and other relationships.
4. The method according to claim 1, characterized in that, After obtaining the entity relationship matrix, dimensionality reduction is performed, specifically including: decomposing the entity relationship matrix into an m*m dimensional left singular matrix, a singular value diagonal matrix, and an n*n dimensional right singular matrix using SVD singular values; selecting the first K singular values, the left singular matrix is reduced from m*m dimensional to m*k dimensional, and the right singular matrix is reduced from n*n dimensional to k*n dimensional.
5. The method according to claim 1, characterized in that, Step 3 specifically includes: Step 31: Predict a new vector for each domain term entity semantic word vector. The new vector should be close to the original semantic vector in the semantic space, and also close to the relevant neighbor nodes in the domain term knowledge graph based on the structural information of the terms in the entity relation matrix. The formula is as follows: ； in, The objective function representing the correction process; Representation of terms Entity semantic word vectors; Represent the new vector; yes In the domain terminology knowledge graph, there are relationships between all neighboring nodes. ; Representation of terms With terminology The relationship between them; and All are vector calibration constants; Step 32, for Taking the first derivative and setting it to zero, we get each The formula for one iteration update is: ； Step 33: Perform multiple iterations until the iteration termination condition is met.
6. The method according to claim 5, characterized in that, The iteration termination condition is reaching a preset number of iterations.
7. A self-correction system for entity vectors of professional domain terminology employing the method described in any one of claims 1-6, characterized in that, include: Data acquisition module, data processing module, and result generation module: The data acquisition module is used to collect raw corpora of domain terminology; The data processing module includes a preprocessing unit, a model training unit, a relation extraction unit, a semantic vector encoding unit, a terminology graph construction unit, a structure matrix construction unit, and a correction unit. The preprocessing unit labels the original corpus of the domain terms and constructs a training set for the relation extraction model; The model training unit trains the relation extraction model based on the training set. The relation extraction unit, based on the original corpus of domain terms, extracts the relations corresponding to the term entity pairs through the trained relation extraction model to obtain triples. The semantic vector encoding unit is used to obtain term entity word vectors from the encoding layer of the relation extraction model; The terminology graph construction unit is used to connect the triples through relations to form a domain terminology knowledge graph; The structural matrix construction unit is used to record and transform the structural information in the domain terminology knowledge graph into an entity relationship matrix. The correction unit corrects the term entity word vectors using the entity relationship matrix. The result generation module is used to organize the corrected term entity vectors into an entity vector set and send it out.
8. A self-correcting device for entity vectors of professional terminology, characterized in that, It includes a processor, a memory, and a bus, wherein the memory stores instructions and data that can be read by the processor; The processor is used to call instructions and data in the memory to execute the method as described in any one of claims 1 to 6; The bus connects the various functional components to transmit information.