A pre-training language model construction method, system and device

Abstract

The application provides a pre-training language model construction method, system and device, relates to the fields of artificial intelligence technology and semantic processing, and mainly comprises the following steps: based on a knowledge graph, a subgraph with a triple as a node unit is constructed; based on entity and relation description text, both are encoded through a conventional pre-training language model to obtain entity representation vectors and relation representation vectors; for the entity representation vectors and the relation representation vectors corresponding to the subgraph, modeling relation information is carried out based on a graph self-attention neural network, and the parameters of the graph neural network and the entity vectors are updated through a prediction link task; the parameters of the model are updated based on a contrast learning technology; and the parameters of the model are fine-tuned for a specific task. The scheme overcomes the problem that entity in the equipment field is sparse and a conventional pre-training language model cannot sufficiently learn entity semantics, and the updated pre-training language model has better understanding and reasoning capabilities in the application of the equipment field.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and semantic processing, and in particular to a method, system and apparatus for constructing a pre-trained language model. Background Technology

[0002] Currently, knowledge graphs are increasingly being used as a representation of symbolic knowledge. A knowledge graph is primarily a semantic network composed of entities, concepts, and various semantic relationships between entities and concepts. The basic unit of a knowledge graph is the head entity-relation-tail entity triple. Knowledge graphs organize and store symbolic knowledge in a simple and flexible way, and have been widely applied in various scenarios such as precise analysis, intelligent recommendation, and deep relational reasoning.

[0003] Pre-trained language models, as a product of the combination of big data, high computing power, and strong algorithms, condense the implicit knowledge inherent in big data and serve as a universal carrier for realizing various artificial intelligence applications. The working process of a pre-trained language model involves first training a deep neural network on a massive dataset, then transferring it to a target scenario, fine-tuning it using small datasets from the target scenario, and finally achieving the required performance. Pre-trained language models act as a bridge connecting the AI ​​technology ecosystem and domain applications. These models often require small-scale labeled data for secondary training to complete tasks in multiple application scenarios. Although pre-trained languages ​​possess knowledge transfer and generalization capabilities, their knowledge learning relies on traditional statistical learning paradigms and does not explicitly model the relationships between entities. This results in the model's inability to understand these relationships and poor reasoning ability. Particularly in the equipment domain, due to its high specialization and relatively sparse entities, there is a lack of entity sample resources, which hinders the direct modeling of relationships between entities in the equipment domain. Consequently, pre-trained language models cannot fully learn the semantics of equipment domain entities and the knowledge of relationships between them, weakening their understanding and reasoning abilities in equipment domain scenarios and thus hindering their application in the equipment domain.

[0004] Because conventional autoregressive pre-trained language models often rely on traditional statistical learning methods to learn semantics, they do not explicitly model the relationships between entities in the equipment domain. Furthermore, most existing techniques use translation-based methods to model entity relationships, which has two main drawbacks: firstly, this method can only model single triples and cannot model the topological structure of knowledge graphs; secondly, because it uses a different encoding scheme than the pre-trained language model for the input sequence, the generated entity representation vectors and the word vectors generated by the pre-trained language model reside in heterogeneous vector spaces, hindering subsequent processing. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, and apparatus for constructing a pre-trained language model, so as to solve at least one of the above-mentioned technical problems existing in the prior art.

[0006] To address the aforementioned technical problems, this invention provides a method for constructing a pre-trained language model, comprising: constructing a subgraph based on a knowledge graph, with triples as node units; encoding entities and relations using an initial pre-trained language model to obtain entity representation vectors and relation representation vectors; modeling relation information using a graph self-attention neural network for the entity representation vectors and relation representation vectors corresponding to the subgraph, and updating the parameters of the graph self-attention neural network and entity representation vectors by predicting linking tasks; updating the parameters of the pre-trained language model through contrastive learning to obtain a secondary pre-trained language model; and fine-tuning the parameters of the secondary pre-trained language model for specific tasks to obtain a domain-specific pre-trained language model.

[0007] The above method embeds entity association knowledge into entity representation vectors and injects entity semantics and its association knowledge into pre-trained language models through contrastive learning. It stores domain knowledge in the form of entity representation vectors and model parameters, thus overcoming the problem of poor robustness of existing pre-trained language models.

[0008] Preferably, the initial pre-trained language model is the "Pangu" pre-trained language model, but it can also be other pre-trained language models known in the art to achieve the same technical purpose.

[0009] Furthermore, the triple is composed of a head entity, a relation, and a tail entity, where the relation refers to the connection between the head entity and the tail entity.

[0010] By using triples as nodes and shared entities connecting triples as edges, multiple subgraphs of the knowledge graph can be constructed, which facilitates the simple and flexible organization and storage of symbolic knowledge and makes subsequent processing easier.

[0011] Furthermore, the pre-trained language model includes a vocabulary, word segmentation representation vector parameters, and model parameters:

[0012] The vocabulary is used to record basic semantic units, and the pre-trained language model uses the vocabulary to split the input text into several sub-words as the input sequence.

[0013] The word segmentation representation vector parameter is used to project the sub-words into an initial representation vector;

[0014] The model parameters are used to perform encoding layer operations on the initial representation vector to obtain the final representation vector.

[0015] Preferably, the final representation vector corresponding to the last semantic unit in the input sequence is used as the entity representation vector and the relation representation vector.

[0016] Furthermore, the graph self-attention neural network consists of two graph self-attention layers, each of which includes a linear variation layer and a bi-headed self-attention layer.

[0017] Furthermore, during the training of the graph self-attention neural network, the triples in the training data are scored. Correct triples are assigned high scores, and incorrect triples are assigned low scores. The scoring facilitates the subsequent calculation of the loss value.

[0018] Furthermore, the graph self-attention neural network also includes a cross-entropy loss function. The graph self-attention neural network obtains a loss value based on this cross-entropy loss function by predicting linking tasks: if the prediction is accurate, the loss value is small; if the prediction is inaccurate, the loss value is large. By minimizing the loss value and updating the parameters of the graph self-attention neural network using gradient descent, it is ensured that the encoded equipment domain entities possess correct entity association knowledge. Of course, other loss functions known in the art can also be used for evaluation to achieve the same technical objective.

[0019] Furthermore, the contrastive learning method includes: constructing training data including text references, positive examples, and negative examples; and updating model parameters through loss function calculation.

[0020] The text reference is a string with the same name as the entity, such as an entity in the equipment field;

[0021] The positive example is an entity that expresses the same semantics as the text reference item;

[0022] The negative example is an entity that expresses a different semantic meaning from the text reference item;

[0023] By using comparative learning, it is easy to inject equipment domain entities and their related knowledge into the pre-trained language model.

[0024] Preferably, the loss function is the triplet margin loss function, but other loss functions known in the art can also be used to achieve the same technical effect.

[0025] Furthermore, when fine-tuning the model parameters for specific tasks, a generative language model function is used as the loss function. These specific tasks include entity classification, relation classification, question answering, and summary generation.

[0026] On the other hand, the present invention also provides a pre-trained language model construction device for the above-mentioned pre-trained language model construction method. The device includes a processor, a memory, and a bus. The memory stores instructions and data that can be read by the processor. The processor is used to call the instructions and data in the memory. The bus connects the functional components to transmit information.

[0027] In another aspect, the present invention also provides a pre-trained language model construction system, including a data receiving module, a data processing module, and a model output module:

[0028] The data receiving module is used to receive entity data and relationship data;

[0029] The data processing module includes a subgraph construction unit, a pre-trained language model unit, a graph self-attention neural network unit, a contrastive learning unit, and a fine-tuning unit.

[0030] The subgraph construction unit constructs a subgraph with triples as node units based on entity data and relation data;

[0031] The pre-trained language model unit stores the pre-trained language model and encodes entity data and relation data respectively through the pre-trained language model to generate entity representation vectors and relation representation vectors;

[0032] The graph self-attention neural network unit encodes the relation representation vector for the entity representation vector and relation representation vector corresponding to the subgraph, and updates the parameters of the graph self-attention neural network and the entity representation vector through the prediction link task;

[0033] The contrastive learning unit updates the parameters of the pre-trained language model by calculating the loss function, based on the loss function, to obtain a second-level pre-trained language model.

[0034] The fine-tuning unit fine-tunes the parameters of the secondary pre-trained language model based on the specific task to obtain a domain pre-trained language model, and sends it to the model output module.

[0035] The model output module is used to output the domain pre-trained language model.

[0036] By adopting the above technical solution, the present invention has the following beneficial effects:

[0037] The solution provided by this invention overcomes the problem that the sparse entities in the equipment domain prevent conventional pre-trained language models from fully learning entity semantics. It models the relationships between entities in the equipment domain based on graph self-attention neural networks, avoiding the generation of entity representation vectors and relationship representation vectors that are heterogeneous with text vectors. Through comparative learning, it injects equipment domain knowledge into conventional pre-trained language models, enabling the updated pre-trained language models to have better understanding and reasoning abilities in equipment domain scenarios. Attached Figure Description

[0038] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a pre-trained language model construction method provided in an embodiment of the present invention;

[0040] Figure 2 A subgraph constructed based on a knowledge graph in the equipment field, provided in an embodiment of the present invention;

[0041] Figure 3 This is a structural diagram of a pre-trained language model provided in an embodiment of the present invention;

[0042] Figure 4 A semantic flowchart of entities and relationships in the field of equipment based on a "Pangu" pre-trained language model provided for embodiments of the present invention;

[0043] Figure 5 A flowchart for injecting relational knowledge into equipment domain entities based on a graph self-attention neural network, as provided in an embodiment of the present invention;

[0044] Figure 6 An explanatory diagram illustrating the construction of fine-tuned pre-trained language models for various downstream tasks in the equipment field, as provided in embodiments of the present invention;

[0045] Figure 7 This is a diagram of a pre-trained language model construction system provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The terms "first," "second," and "third," etc., in the specification, embodiments, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or modules. A method, system, product, or apparatus is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or apparatuses. "And / or" is used to indicate the selection of one or both of the two objects to which it is connected.

[0048] The present invention will be further explained below with reference to specific embodiments.

[0049] Before detailing the specific contents of the embodiments of the present invention, in order to facilitate the expression of the design concept of the embodiments of the present invention, the design starting point is summarized as follows:

[0050] To address the problem that existing autoregressive pre-trained language models cannot explicitly model the relationships between entities, this invention introduces graph neural networks for modeling, associates multidimensional entities, and abandons the traditional graph neural network approach of using entities as nodes, instead using triples as nodes, constructing edges for triples with the same entity, and using a feedforward neural network to encode the subgraph composed of several triples.

[0051] To address the issue of vector heterogeneity caused by translation-based encoding methods in existing technologies, this invention first encodes entities using an encoder based on an autoregressive pre-trained language model to generate entity representation vectors. Then, it further encodes entities and relationships using a graph self-attention neural network, thereby avoiding the generation of heterogeneous vectors.

[0052] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for constructing a pre-trained language model, comprising:

[0053] Step 1: Based on the equipment domain knowledge graph, construct a subgraph with triples as node units, such as... Figure 2 As shown, triples consisting of head entity-relationship-tail entity are selected as nodes, and shared entities connecting the triples are used as edges to construct multiple subgraphs of the knowledge graph.

[0054] Step 2: Based on the text corpus containing entities and relations, encode the entities and relations using a conventional pre-trained language model, such as a pre-trained language model for "Pangu". Figure 3As shown, entity representation vectors and relation representation vectors containing equipment domain semantics are obtained. The semantic dimension of the representation vectors can be 2560, or it can be set to other dimensions as needed. The vocabulary of the "Pangu" pre-trained language model has been segmented for entity descriptions and relations, as shown below. Figure 4 As shown, the word segmentation result is used as the input sequence of the "Pangu" pre-trained language model; after being encoded by the "Pangu" pre-trained language model, a representation vector of the same length as the input sequence is obtained, and the representation vector corresponding to the last semantic unit in the input sequence is extracted as the entity representation vector and the relation representation vector;

[0055] Step 3: For the entity representation vector and relation representation vector corresponding to the subgraph, after encoding the relations through a graph self-attention neural network, update the parameters of the graph self-attention neural network and the entity representation vector, such as... Figure 5 As shown, the graph self-attention neural network consists of two graph self-attention layers. Each graph self-attention layer includes a linear transformation layer and a bi-headed self-attention layer. In this embodiment, the calculation formula is preferably set as follows:

[0056]

[0057] Where k represents the number of self-attention heads, W represents the linearly varying layer with a matrix dimension of 2560*1280 to be trained, and e i Let e ​​represent the initial entity vector with dimension i = 2560. j Let e′ represent the initial entity vector with dimension j of 2560. i Let α represent the i-th encoded entity vector with dimension 2560. ij σ represents the attention level between the i-th entity and the j-th entity, σ is the ReLU nonlinear transformation, U represents the concatenation operation of vectors, and N represents the set of initial entity vectors;

[0058] The triples are used as the training set to train the graph-based attention neural network. The triples also include a score: correct triples are assigned a high score, and incorrect triples are assigned a low score. The score facilitates the subsequent calculation of the loss value.

[0059] Updating the parameters of the graph self-attention neural network by predicting linking tasks specifically includes: the graph self-attention neural network is based on the cross-entropy loss function, and obtains a loss value by predicting linking tasks: if the prediction is accurate, the loss value is small; if the prediction is inaccurate, the loss value is large; the parameters of the graph self-attention neural network are updated by minimizing the loss value and using gradient descent, which ensures that the encoded equipment domain entities have correct entity association knowledge; in this embodiment, the preferred cross-entropy loss function is as follows:

[0060]

[0061] Where (s,r,o) represents a triple, s represents the head entity, r represents the relation, and o represents the tail entity; T represents the set of triples; f represents the scoring function; W represents a diagonal matrix of dimension 2560*2560 to be trained; e s This represents the encoded head entity vector with a dimension of 2560; e o This represents the encoded tail entity vector with dimension 2560; y represents the label of the triple, assigned a value of 0 or 1: when assigned a value of 0, it represents an incorrect triple; when assigned a value of 1, it represents a correct triple.

[0062] Step 4: Update the parameters of the pre-trained language model through contrastive learning to obtain a secondary pre-trained language model, which is used to inject entity association knowledge into the entities. Specific steps include: constructing training data including text references, positive examples, and negative examples. The text references are strings with the same name as entities in the equipment domain; the positive examples are entities expressing the same semantics as the text references; and the negative examples are entities expressing different semantics from the text references. The parameters of the pre-trained language model are updated using a triplet marginal loss function. In this embodiment, the preferred triplet marginal loss function is as follows:

[0063]

[0064] Where e represents the representation vector of the text reference item, e pos The vector representing the positive example, e neg The vector representing the negative example;

[0065] Step 5: Fine-tune the parameters of the secondary pre-trained language model for specific tasks to obtain the equipment domain pre-trained language model, such as... Figure 6 As shown, test data is constructed for tasks such as entity classification, relation classification, question answering, and summary generation. A generative language model function is selected as the loss function for calculation, and the parameters of the secondary pre-trained language model are fine-tuned. In this embodiment, the preferred generative language model function is as follows:

[0066]

[0067] Where x represents a subword of the input sequence, and θ represents the parameters of the second-level pre-trained language model to be updated.

[0068] On the other hand, the present invention also provides a pre-trained language model construction system, such as Figure 7 As shown, it includes a data receiving module, a data processing module, and a model output module:

[0069] The data receiving module is used to receive entity data and relationship data in the field of equipment;

[0070] The data processing module includes a subgraph construction unit, a pre-trained language model unit, a graph self-attention neural network unit, a contrastive learning unit, and a fine-tuning unit.

[0071] The subgraph construction unit constructs a subgraph with triples as node units based on entity data and relation data;

[0072] The pre-trained language model unit stores a conventional pre-trained language model and encodes entity data and relation data respectively through the pre-trained language model to generate entity representation vectors and relation representation vectors;

[0073] The graph self-attention neural network unit encodes the relation representation vector for the entity representation vector and relation representation vector corresponding to the subgraph, and updates the parameters of the graph self-attention neural network and the entity representation vector through the prediction link task;

[0074] The contrastive learning unit updates the parameters of the pre-trained language model by calculating the loss function, based on the loss function, to obtain a second-level pre-trained language model.

[0075] The fine-tuning unit fine-tunes the parameters of the secondary pre-trained language model based on the specific task to obtain a pre-trained language model for the equipment domain, and sends it to the model output module.

[0076] The model output module is used to output the pre-trained language model for the equipment domain.

[0077] In another implementation, this solution can be implemented using a device, which may include corresponding modules that perform one or more steps in the various embodiments described above. A module may be one or more hardware modules specifically configured to perform the corresponding step, or implemented by a processor configured to perform the corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0078] The processor executes the methods and processes described above, with the following specific steps:

[0079] Step 1: Based on the knowledge graph of the equipment domain, construct a subgraph with triples as node units;

[0080] Step 2: Based on the text corpus of entities and relations, the entities and relations are encoded by a pre-trained language model to obtain entity representation vectors and relation representation vectors containing equipment neighborhood semantics;

[0081] Step 3: For the entity representation vector and relation representation vector corresponding to the subgraph, update the parameters of the graph neural network and entity representation vector after encoding the relation through the graph self-attention neural network;

[0082] Step 4: Update the parameters of the pre-trained language model through contrastive learning to inject the entity association knowledge into the entity. The specific steps include: constructing training data including text references, positive examples, and negative examples. The text references are strings with the same name as the equipment domain entity, the positive examples are entities that express the same semantics as the text references, and the negative examples are entities that express different semantics from the text references; update the parameters of the pre-trained language model through loss function calculation to obtain a second-level pre-trained language model.

[0083] Step 5: Construct test data for specific tasks such as entity classification, relation classification, question answering, and summary generation; select a generative language model function as the loss function for calculation; and fine-tune the parameters of the secondary pre-trained language model to obtain the equipment domain pre-trained language model.

[0084] For example, the method implementation in this scheme can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as memory. In some implementations, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by a processor, one or more steps of the method described above can be performed.

[0085] Alternatively, in other implementations, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0086] This device can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus connects various circuits, including one or more processors, memory, and / or hardware modules. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuitry, external antennas, etc.

[0087] This invention, based on a knowledge graph in the equipment domain, uses graph self-attention neural network technology to encode the knowledge graph topology structure on the basis of the initial entity representation vectors generated by the "Pangu" pre-trained language model. This embeds entity association knowledge into the entity representation vectors and injects entity semantics and their association knowledge into the pre-trained language model through contrastive learning. Equipment domain knowledge is stored in the form of entity representation vectors and model parameters, overcoming the problems of poor robustness and weak versatility of conventional pre-trained language models. In particular, it fills the gap in the application of pre-trained language models in the equipment domain.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a pre-trained language model, characterized in that, include: Based on knowledge graphs, a subgraph is constructed with triples as node units and shared entities connecting triples as edges; The triple is composed of a head entity, a relation, and a tail entity, where the relation refers to the connection between the head entity and the tail entity. Based on the entity and the relation, they are encoded using a pre-trained language model to obtain entity representation vectors and relation representation vectors. Relation information is modeled using a graph self-attention neural network, and the parameters of the graph self-attention neural network and the entity representation vectors are updated by predicting linking tasks. The parameters of the pre-trained language model are updated through contrastive learning to obtain a second-level pre-trained language model. The parameters of the second-level pre-trained language model are fine-tuned for specific tasks to obtain a domain-specific pre-trained language model. The contrastive learning method includes: constructing training data comprising text references, positive examples, and negative examples, wherein the text references are strings with the same names as entities in the equipment domain, the positive examples are entities expressing the same semantics as the text references, and the negative examples are entities expressing different semantics from the text references; and updating the parameters of the pre-trained language model through a triplet marginal loss function, the specific function formula being: ； in, The representation vector representing the textual reference. The representation vector of positive examples. The vector representing the negative example. Represents the loss function. Indicates textual references, Represents a set of textual references.

2. The pre-trained language model construction method according to claim 1, characterized in that, Pre-trained language models include a vocabulary, word segmentation representation vector parameters, and model parameters: The vocabulary is used to record basic semantic units, and the pre-trained language model uses the vocabulary to split the input text into several sub-words as the input sequence. The word segmentation representation vector parameter is used to project the sub-words into an initial representation vector; The model parameters are used to perform encoding layer operations on the initial representation vector to obtain the final representation vector.

3. The pre-trained language model construction method according to claim 2, characterized in that, The final representation vector corresponding to the last semantic unit in the input sequence is used as the entity representation vector and the relation representation vector.

4. The pre-trained language model construction method according to claim 1, characterized in that, The graph self-attention neural network consists of two graph self-attention layers, each of which includes a linear variation layer and a bi-headed self-attention layer.

5. The pre-trained language model construction method according to claim 1, characterized in that, include: When training the graph self-attention neural network, triples in the training data are scored, with correct triples assigned high scores and incorrect triples assigned low scores; the loss value is calculated by predicting linking tasks based on the cross-entropy loss function; and the parameters of the graph self-attention neural network are updated by minimizing the loss value and using gradient descent.

6. A pre-trained language model construction apparatus for the pre-trained language model construction method according to any one of claims 1 to 5, characterized in that, The device includes a processor, a memory, and a bus. The memory stores instructions and data that can be read by the processor; the processor is used to call the instructions and data in the memory; and the bus connects the various functional components to transmit information.

7. A pre-trained language model construction system employing any one of the pre-trained language model construction methods described in claims 1 to 5, characterized in that, It includes a data receiving module, a data processing module, and a model output module: The data receiving module is used to receive entity data and relationship data; The data processing module includes a subgraph construction unit, a pre-trained language model unit, a graph self-attention neural network unit, a contrastive learning unit, and a fine-tuning unit. The subgraph construction unit constructs a subgraph with triples as node units based on entity data and relation data; The pre-trained language model unit stores the pre-trained language model and encodes entity data and relation data respectively through the pre-trained language model to generate entity representation vectors and relation representation vectors; The graph self-attention neural network unit encodes the relation representation vector for the entity representation vector and relation representation vector corresponding to the subgraph, and updates the parameters of the graph self-attention neural network and the entity representation vector through the prediction link task; The contrastive learning unit updates the parameters of the pre-trained language model by calculating the loss function, based on the loss function, to obtain a second-level pre-trained language model. The fine-tuning unit fine-tunes the parameters of the secondary pre-trained language model based on the specific task to obtain a domain pre-trained language model, and sends it to the model output module. The model output module is used to output the domain pre-trained language model.

Images