A consensus library query method based on a large model

Abstract

The application relates to a large model-based consensus library query method, which improves the method for difficult problems of field knowledge query and applies the method in practice. In view of the confusion and errors easily caused by cross-field knowledge transfer, a field self-adaptive pre-training method is designed to enable the large model network to distinguish the knowledge of different fields; the large model network is improved; in view of the feature difference between knowledge injection and text input in the large model, an explicit alignment knowledge reinforcement representation learning method is designed to further improve the representation effect of the large model. Finally, the vector representation of the text and knowledge is obtained based on the improved hybrid large model network, and the consensus library is queried through vector matching, so that the actual application demand is better solved.

Description

Technical Field

[0001] This invention belongs to the field of knowledge reasoning and querying, and relates to a consensus database query method based on a large model. Background Technology

[0002] Early knowledge modeling methods in the field of artificial intelligence included expert systems. These systems involved manually constructing domain-specific knowledge and using rules to reason and query that knowledge in order to solve real-world problems. However, this technique suffers from incomplete coverage of domain knowledge, weak robustness, and low scalability.

[0003] The knowledge graph method proposed in the mid-term, originating from semantic networks, includes entities, attribute relationships, events, etc., and uses graph-based methods for representation and querying. Domain knowledge graphs focus on specific domains, mostly consisting of entities and concepts related to that domain, making it difficult to represent and query complex knowledge.

[0004] Recently proposed pre-trained large model methods can self-supervisedly learn general knowledge from massive datasets, implicitly storing this knowledge in the model parameters. Furthermore, by combining explicit knowledge from knowledge graphs, the large model is enhanced with a more robust representation. Based on this enhanced representation, retrieval matching algorithms can be used to easily query knowledge. However, transferring general-domain large models to the aerospace field for knowledge retrieval presents several technical challenges.

[0005] However, applying this method to the aerospace field faces the following difficulties and challenges:

[0006] (1) The general knowledge obtained from the pre-training of large models is domain-independent. Directly transferring it to the aerospace field will cause knowledge confusion and even introduce erroneous noise.

[0007] (2) The mainstream method of knowledge reinforcement representation learning injects knowledge graphs into the input of a large model and uses a single network (transformer) to represent the text input and the graph input. However, there is a gap between the feature representation spaces of the text input and the graph input, and a single network is difficult to effectively align and fuse them. Summary of the Invention

[0008] The technical problem solved by this invention is to overcome the shortcomings of the existing technology and propose a consensus library query method based on a large model.

[0009] The solution of the present invention is:

[0010] In a first aspect, the present invention provides a consensus library query method based on a large model, comprising:

[0011] The large model network is pre-trained for text representation and domain classification to obtain a large model network for text representation that can distinguish different domains.

[0012] An improved large-scale text representation network is proposed. Suppose the large-scale text representation network has N layers. The lower N / 2 layers are copied and initialized with the parameters obtained from the pre-training of the text representation network to obtain a hybrid large-scale network. N is an even number.

[0013] Pre-train the hybrid large model network with knowledge representation and update the parameters of the hybrid large model network;

[0014] In the hybrid large model network after parameter update, one of the two lower N / 2 layers represents the input text, and the other represents the input knowledge. The upper N / 2 layers are used to calculate the similarity between the text representation vector and the knowledge representation vector.

[0015] We use a hybrid large model network to obtain the representation vectors of all knowledge in the consensus base offline, and construct the vector library and vector index of the consensus base.

[0016] Based on the query statement entered by the user in real time, the representation vector of the query statement is obtained online using a hybrid large model network, and the vector library of the consensus database is searched to return the knowledge corresponding to the knowledge vector with the highest similarity.

[0017] Preferably, when pre-training the large model network for text representation and domain classification, the domain labels of the text data are used to design a domain classification task. Using general text corpora and domain text corpora, the parameters of the large model network are obtained through pre-training methods, thus obtaining the large model network for text representation.

[0018] Preferably, the large text representation model network can obtain representation vectors for any text and can also distinguish different domains.

[0019] The preferred learning objective for domain classification tasks is Loss. 领域分类 satisfy:

[0020] Loss 领域分类 = cross_entropy(Predict) 领域 Label 领域 )

[0021] `cross_entropy()` represents the cross-entropy loss function; `cross_entropy(Predict)` 领域 Label 领域 Label indicates the domain to which the text belongs. 领域 and prediction domain 领域 The cross-entropy loss function between them, i.e., the domain label of the text. 领域 and prediction domain 领域 The difference between them, Loss 领域分类The smaller the value, the more accurate the model's predictions will be.

[0022] The preferred learning objective for pre-training text representation and domain classification is Loss. union satisfy:

[0023] Loss union =Loss mlm +Loss nsp +Loss 领域分类

[0024] Among them, Loss mlm Loss represents the learning objective of a general pre-trained masked language model task. nsp Loss represents the learning objective of a general pre-trained sentence relation classification task. 领域分类 This represents the learning objective of the domain classification task in the domain pre-training.

[0025] Preferably, the hybrid large model network adopts an explicit alignment knowledge reinforcement representation learning method to solve the feature difference between knowledge injection and text input in the hybrid large model network, and improve the representation effect of the hybrid large model network on text or knowledge.

[0026] Preferably, the learning objective of the explicit alignment knowledge reinforcement representation learning method is Loss. align satisfy:

[0027] Loss align =max(0,δ-similar(pos_pair)+silimar(neg_pair))

[0028] Where `pos_pair` represents a matching text-knowledge pair, and `neg_pair` represents a non-matching text-knowledge pair; `similar()` calculates the similarity, corresponding to the fusion part of the large model network structure, and `δ` is the threshold; `Loss`... align The loss function value for aligning the target.

[0029] In a second aspect, the present invention provides a terminal device, comprising:

[0030] Memory, used to store at least one instruction executed by a processor;

[0031] A processor for executing instructions stored in memory to implement the method described in the first aspect above.

[0032] Thirdly, the present invention provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0033] The advantages of this invention compared to the prior art are:

[0034] Compared with existing technologies, this invention innovatively implements a domain knowledge query method based on a large model. It utilizes a domain-adaptive large model pre-training method, enabling the large model to distinguish knowledge from different domains; it designs and introduces an explicit alignment-based knowledge reinforcement representation learning method to address feature differences in injecting knowledge graphs into the large model input, achieving effective alignment and fusion, and improving the knowledge reinforcement representation effect of the large model; based on this large model representation, it constructs an aerospace consensus database query application, which can effectively solve practical application problems. Attached Figure Description

[0035] Figure 1 A schematic diagram of a domain-adaptive large model pre-training method;

[0036] Figure 2 A schematic diagram illustrating knowledge-enhanced representation learning that incorporates explicit alignment;

[0037] Figure 3 This is a schematic diagram illustrating the integration of knowledge query applications based on a hybrid large model network. Detailed Implementation

[0038] The invention will now be further described with reference to the accompanying drawings.

[0039] This invention addresses the shortcomings of existing technologies by designing a domain-adaptive large model pre-training method and introducing an explicit alignment knowledge reinforcement representation learning method, thereby better transferring large models to the aerospace field to solve knowledge retrieval problems.

[0040] The technical solution mainly consists of three parts: (1) domain-adaptive large model pre-training; (2) knowledge reinforcement representation learning with explicit alignment; and (3) knowledge query based on large model representation. The three parts are integrated in series to construct a consensus library query application method (or system) based on the large model.

[0041] The steps of this invention are as follows:

[0042] (1) Pre-train the large model network for text representation and domain classification to obtain a large model network for text representation that can distinguish different domains.

[0043] By utilizing general and domain-specific text corpora and employing pre-training methods such as self-supervised and weakly supervised training, the parameters of a large-scale text representation network are trained, enabling the large model to obtain the representation vector for any text. This large-scale text representation network can obtain the representation vector for any text and distinguish between different domains.

[0044] (2) Pre-training the knowledge representation of the large model network.

[0045] An improved large-scale text representation network is proposed. Assuming the large-scale text representation network has N layers, the lower N / 2 layers are copied. The copied lower N / 2 layers are initialized using parameters obtained from the pre-trained text representation network, resulting in a hybrid large-scale network; N is an even number, such as N = 12.

[0046] Pre-train the hybrid large model network with knowledge representation and update the parameters of the hybrid large model network;

[0047] In the hybrid large model network after parameter update, one of the two lower N / 2 layers represents the input text, and the other represents the input knowledge. The upper N / 2 layers are used to calculate the similarity between the text representation vector and the knowledge representation vector.

[0048] (3) Query consensus base based on hybrid large model network

[0049] We use a hybrid large model network to obtain the representation vectors of all knowledge in the consensus base offline, and construct the vector library and vector index of the consensus base.

[0050] Based on the query statement entered by the user in real time, the representation vector of the query statement is obtained online using a hybrid large model network, and the vector library of the consensus database is searched to return the knowledge corresponding to the knowledge vector with the highest similarity.

[0051] The key steps described above are implemented as follows:

[0052] (1) Domain-adaptive large model network pre-training

[0053] Pre-training tasks for general-domain large-scale model networks include Masked Language Modeling (MLM) and Sentence Relationship Prediction (NSP), which can learn general knowledge from massive amounts of data through self-supervised learning and are domain-independent. However, directly transferring this method to the aerospace domain can lead to knowledge confusion. To address this issue, a domain-adaptive large-scale model network pre-training method is designed. This method utilizes text data type labels (e.g., aerospace, aviation, etc.) to design domain classification pre-training tasks, which, together with general pre-training tasks, form a multi-task learning objective, enabling the large model to distinguish knowledge from different domains. By optimizing the multi-task learning objective, the large model network is pre-trained, allowing it to obtain text representation vectors and distinguish different domains. The design method is as follows:

[0054] Domain Adaptive Large Model Multi-Task Learning Objective Loss union as follows:

[0055] Loss union =Loss mlm +Loss nsp +Loss 领域分类

[0056] Among them, Loss mlm Loss represents the learning objective of a general pre-trained masked language model task. nsp Loss represents the learning objective of a general pre-trained sentence relation classification task. 领域分类 This represents the learning objective of the domain classification task in the domain pre-training. For example... Figure 1 As shown.

[0057] Loss of the domain classification pre-training task 领域分类 as follows:

[0058] Loss 领域分类 = cross_entropy(Predict) 领域 Label 领域 )

[0059] `cross_entropy()` represents the cross-entropy loss function; `cross_entropy(Predict)` 领域 Label 领域 Label indicates the domain to which the text belongs. 领域 and prediction domain 领域 The cross-entropy loss function between them, i.e., the domain label of the text. 领域 and prediction domain 领域 The difference between them, Loss 领域分类 The smaller the value, the more accurate the model's predictions will be.

[0060] (2) Introduce explicit alignment knowledge to enhance representation learning

[0061] Hybrid large-scale model networks employ an explicit aligned knowledge reinforcement representation learning method to address the feature differences between knowledge injection and text input in hybrid large-scale model networks, thereby improving the network's representation of text or knowledge. Utilizing knowledge reinforcement representation learning can further enhance the large model's ability to represent and retrieve knowledge. Traditional methods, when injecting knowledge graphs into the large model input, do not consider the differences in feature representation spaces between text input and graph input, directly using a single network for fusion, which affects the effectiveness of knowledge reinforcement representation. To address this issue, an explicit input feature alignment mechanism is designed, employing two independent networks to model the text input and graph input representations separately, and then fusing them, thus significantly improving the knowledge reinforcement representation effect of the large model. Figure 2 As shown.

[0062] The input feature alignment mechanism introduces the alignment target loss function as follows:

[0063] Loss align=max(0,δ-similar(pos_pair)+silimar(neg_pair))

[0064] Where pos_pair represents a matching text-knowledge pair, and neg_pair represents a non-matching text-knowledge pair; similar() calculates the similarity; δ is the threshold corresponding to the fusion part of the large model network structure; Loss align The loss function value for aligning targets represents the comparison between pos_pair similarity and neg_pair similarity. Optimization aims to maximize pos_pair similarity and minimize neg_pair similarity.

[0065] (3) Consensus library query based on hybrid large model network representation

[0066] Based on the above method, a large-scale knowledge reinforcement model for the aerospace field is obtained. This hybrid large-scale model network is used to calculate representation vectors for both the aerospace consensus base knowledge and the query statement, and similarity scores are also calculated. In practice, the representation vector library for the aerospace consensus base knowledge is calculated offline, while the query statement representation vector is calculated online. A vector retrieval tool is then used to retrieve the knowledge with the highest similarity from the library, which is returned as the result of the consensus base query. Figure 3 As shown.

[0067] The present invention also provides a terminal device, comprising: a memory for storing at least one instruction executed by a processor; and a processor for executing the instructions stored in the memory to implement the above method.

[0068] The present invention also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0069] This invention relates to a consensus database query method based on a large model, proposing an improved method and applying it in practice to address the challenges of domain knowledge querying. To address the confusion and errors that easily arise from cross-domain knowledge transfer, a domain-adaptive pre-training method is designed, enabling the large model network to distinguish knowledge from different domains. Furthermore, the large model network is improved by designing an explicit alignment-based knowledge reinforcement representation learning method to further enhance its representation performance, taking into account the feature differences between knowledge injection and text input. Finally, vector representations of text and knowledge are obtained based on the improved hybrid large model network, and consensus database queries are performed through vector matching, effectively meeting practical application needs.

[0070] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A large model-based consensus library query method, characterized in that, The method comprises the following steps: Text representation and domain classification pre-training are performed on a large model network to obtain a text representation large model network capable of distinguishing different domains; The text representation large model network is improved, and the text representation large model network has N layers, the lower N / 2 layers of the network are copied, the parameters obtained by text representation pre-training are used to initialize the copied lower N / 2 layers, and a hybrid large model network is obtained; N is an even number; Knowledge representation pre-training is performed on the hybrid large model network to update the parameters of the hybrid large model network; In the hybrid large model network after parameter updating, one of the two lower N / 2 layers represents the input text, and the other represents the input knowledge, and the upper N / 2 layers are used to calculate the similarity between the text representation vector and the knowledge representation vector; The representation vectors of all knowledge in the consensus library are obtained offline by using the hybrid large model network, and a vector library and a vector index of the consensus library are constructed. According to the query statement input by the user in real time, the representation vector of the query statement is obtained online by using the hybrid large model network, the vector library of the consensus library is searched, and the knowledge corresponding to the knowledge vector with the highest similarity is returned.

2. The consensus library query method based on a large model according to claim 1, characterized in that, When the large model network is pre-trained for text representation and domain classification, the domain labels of the text data are used to design a domain classification task, general text corpus and domain text corpus are used, and the pre-training method is used to obtain the parameters of the large model network, thereby obtaining the text representation large model network.

3. The method of claim 2, wherein, The text representation large model network can obtain the representation vector of any text and can distinguish different domains.

4. The large model-based consensus library query method according to claim 3, characterized in that, Learning objective Loss of domain classification task 领域分类 Satisfies: Loss 领域分类 = cross_entropy(Predict 领域 , Label 领域 ) cross_entropy() represents a cross-entropy loss function, cross_entropy(Predict 领域 , Label 领域 ) represents a cross-entropy loss function between the domain Label 领域 to which the text belongs and the predicted domain Predict 领域 , i.e. the difference between the domain Label 领域 to which the text belongs and the predicted domain Predict 领域 , Loss 领域分类 is smaller, the more accurate the domain predicted by the model is.

5. The consensus library query method based on a large model according to claim 2, characterized in that, Text representation and domain classification pre-training learning objective Loss union satisfies: Loss union = Loss mlm + Loss nsp + Loss 领域分类 wherein, Loss mlm represents the learning objective of the general pre-training mask language model task, Loss nsp represents the learning objective of the general pre-training sentence relation classification task, Loss 领域分类 represents the learning objective of the domain pre-training domain classification task.

6. The consensus library query method based on a large model according to claim 1, characterized in that, The hybrid large model network adopts an explicit alignment knowledge reinforced representation learning method to solve the feature difference between knowledge injection and text input in the hybrid large model network, and improves the representation effect of the hybrid large model network on text or knowledge.

7. The large model-based consensus library query method according to claim 6, characterized in that, The method comprises the following steps: Learning objective Loss of the explicit alignment knowledge reinforced representation learning method align satisfies: Loss align = max(0, δ - similar(pos_pair) + similar(neg_pair)) Wherein, pos_pair represents a matched text and knowledge pair, neg_pair represents a mismatched text and knowledge pair; similar() represents calculating similarity, corresponding to the fusion part of the large model network structure, and δ is a threshold; Loss align is the loss function value of the alignment target.

8. A terminal device, comprising: A memory is used to store instructions executed by at least one processor; A processor is used to execute the instructions stored in the memory to implement the method of any one of claims 1-7. The computer readable storage medium stores computer instructions, when the computer instructions run on the computer, the computer executes the method of any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, ​

Images