Image-to-text generation method and device based on large language model of few-sample learning

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for generating graphs from large language models based on few-sample learning. The method includes: constructing positive samples and negative samples, training the initial graph neural network to obtain a graph encoder, setting four different training tasks, obtaining respectively corresponding training samples, training the initial linear network to obtain a graph-to-text projector; selecting a few-sample learning sample, using the few-sample learning sample to fine-tune the pre-trained large language model, and obtaining a trained graph-to-text generation model based on the large language model according to the graph encoder, the graph-to-text projector and the fine-tuned large language model; obtaining a graph to be generated, inputting the graph to be generated into a trained graph-to-text generation model based on a large language model, and obtaining a generated text corresponding to the graph to be generated. The present invention can reduce the dependence on large-scale annotated data sets and improve the accuracy of the trained model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for generating graphs from text using a large language model based on few-sample learning. Background Art

[0002] In the field of Natural Language Processing (NLP), graph-to-text generation is an important research direction, which involves converting graphical data (such as graphs and knowledge graphs) into natural language descriptions. This task is crucial for downstream applications such as knowledge representation, information retrieval, and automatic summarization. With the development of deep learning technology, large language models (LLMs) have made significant progress in text generation, language understanding, and reasoning tasks. These models have acquired rich world knowledge and language capabilities through pre-training on large-scale text data.

[0003] The graph-to-text generation task aims to generate natural language descriptions from graph-structured data (such as semantic role annotations AMR, knowledge graphs KG, etc.). This task is at the intersection between graph structure and text, and is of great significance for downstream tasks such as knowledge-driven reasoning and text generation. Graph-to-text generation methods are mainly divided into two branches:

[0004] The first branch is graph-based methods: These methods focus on the graph structure itself and aim to better capture the structural information in the input graph. This includes using graph encoders to learn the representation of nodes and edges of the graph, or training neural networks through structure-aware learning objectives;

[0005] The second branch is methods based on pre-trained language models: These methods use pre-trained language models to generate fluent text. Usually, graph structure pre-training refers to pre-training graph encoders on large-scale unlabeled graph datasets using self-supervised learning strategies to capture structural information in the graph. A two-step structured generation method based on a pre-trained language model is introduced for KG-to-text generation. Although the use of graph pre-training methods can enable LLM to understand graphs, there are some shortcomings. In the process of trying to encode graphs into coherent text, some details and information contained in the graph may be lost, especially when the graph data is very rich. The effect of the graph encoder depends largely on the quality of the graph data used for pre-training; if the data is not rich or diverse enough, the generalization ability of the model may be limited.

[0006] In addition, in some current graph-to-text methods, representation learning of graph-structured data usually relies on supervised methods, which may require a large amount of labeled data and limit the generalization ability of the model when the data distribution changes. Summary of the invention

[0007] In order to solve the technical problems of the limited generalization ability of the model in the prior art and the need for a large amount of labeled data, the embodiment of the present invention provides a method and device for generating text from a large language model based on few-sample learning. The technical solution is as follows:

[0008] On the one hand, a method for generating text from a graph based on a large language model with few-sample learning is provided, the method being implemented by a device for generating text from a graph based on a large language model with few-sample learning, the method comprising:

[0009] S1. Construct positive and negative samples;

[0010] S2. Train the initial graph neural network through the constructed positive and negative samples to obtain a graph encoder.

[0011] S3. Set four different training tasks and obtain training samples corresponding to the four training tasks respectively;

[0012] S4, training the initial linear network according to the training samples corresponding to the four training tasks, and obtaining the image-to-text projector;

[0013] S5. Selecting a few-shot learning sample based on a few-shot image-to-text generation task;

[0014] S6. Fine-tune the pre-trained large language model using the few-shot learning sample, and obtain a trained large language model-based image-to-text generation model according to the image encoder, the image-to-text projector, and the fine-tuned large language model;

[0015] S7. Obtain a graph to be generated, and input the graph to be generated into a trained graph-to-text generation model based on a large language model to obtain a generated text corresponding to the graph to be generated.

[0016] On the other hand, a device for generating text from a graph based on a large language model with few sample learning is provided, and the device is applied to a method for generating text from a graph based on a large language model with few sample learning, and the device includes:

[0017] Construction module, used to construct positive samples and negative samples;

[0018] The training module is used to train the initial graph neural network through the constructed positive and negative samples to obtain the graph encoder;

[0019] A setting module is used to set four different training tasks and obtain training samples corresponding to the four training tasks respectively;

[0020] An optimization module is used to train the initial linear network according to the training samples corresponding to the four training tasks to obtain a graph-to-text projector;

[0021] A selection module is used to select few-shot learning samples based on a few-shot image-to-text generation task;

[0022] A fine-tuning module is used to fine-tune the pre-trained large language model using the few-shot learning samples, and obtain a trained image-to-text generation model based on the large language model according to the image encoder, the image-to-text projector, and the fine-tuned large language model;

[0023] The generation module is used to obtain the graph to be generated, input the graph to be generated into a trained graph-to-text generation model based on a large language model, and obtain the generated text corresponding to the graph to be generated.

[0024] On the other hand, a graph-to-text generation device of a large language model based on few-sample learning is provided, and the graph-to-text generation device of a large language model based on few-sample learning includes: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the graph-to-text generation methods of a large language model based on few-sample learning is implemented.

[0025] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned graph-to-text generation methods of a large language model based on few-sample learning.

[0026] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0027] The embodiment of the present invention enhances the ability of large language models to understand graph structures through the proposed structure-aware large language model framework and a specific pre-training stage for graph data. Through unsupervised graph comparison learning, the model learns the overall representation of the graph, improves the quality of the graph structure representation, and can distinguish the semantic differences between different graphs, so that the model can learn discriminative features without a large amount of labeled data. Secondly, four structure-aware instruction tasks are introduced to train the graph-to-text projector, so that the model can generate more accurate and coherent text based on the structural features of the graph, improve the semantic richness and structural consistency of the text, and bridge the field gap between graph representation and text representation. Finally, the system is fine-tuned through the few-sample learning technology, and the model is fine-tuned on specific downstream tasks, which reduces the dependence on large-scale labeled data sets and enhances the application potential of the model in data-scarce fields. Through the above process, the model not only learns the encoding ability of the graph structure, but also learns how to convert these structural information into text representation, and achieves better performance in the graph-to-text generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of a method for generating graphs from large language models based on few-sample learning provided by an embodiment of the present invention;

[0030] Figure 2 It is a schematic diagram of a structure for constructing positive examples and negative examples for graph contrast learning provided by an embodiment of the present invention;

[0031] Figure 3 It is a structural diagram of four different graph structure tasks provided by an embodiment of the present invention;

[0032] Figure 4 It is a block diagram of a device for generating text from a graph based on a large language model of few-sample learning provided by an embodiment of the present invention;

[0033] Figure 5 It is a structural schematic diagram of a graph-to-text generation device of a large language model based on few-sample learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0035] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0036] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0037] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] The embodiment of the present invention provides a method for generating text from a graph using a large language model based on few-sample learning. The method can be implemented by a device for generating text from a graph using a large language model based on few-sample learning. The device for generating text from a graph using a large language model based on few-sample learning can be a terminal or a server. Figure 1 The flowchart of the method for generating text from a graph using a large language model based on few-sample learning is shown. The processing flow of the method may include the following steps:

[0040] S1. Construct positive and negative samples.

[0041] Optionally, constructing positive samples and negative samples in S1 includes:

[0042] S11. Obtain a graph structure sample and use the graph structure sample as a root node.

[0043] S12. Rotate the two subtrees of the root node and use the graph structure obtained after the rotation as a positive sample.

[0044] S13. Delete some nodes in the root node to obtain the first part of negative example samples.

[0045] S14. Add some nodes to the root node to obtain the second part of negative example samples.

[0046] S15. Exchange some nodes at the root node to obtain the third part of negative example samples.

[0047] In a feasible implementation, Figure 2 As shown in the figure, the construction of the positive example is the graph obtained by rotating the two subtrees after the root node. The content of the graph does not change after the rotation. There are three types of counterexample construction. The first is to delete some nodes in the graph, the second is to add some nodes to the graph, and the third is to exchange the subtrees of some nodes in the graph. These operations all change the information of the graph.

[0048] S2. The initial graph neural network is trained by constructing positive and negative samples to obtain a graph encoder.

[0049] Optionally, the specific operation of S2 may be as follows:

[0050] The positive and negative samples are input into the initial graph neural network, and the initial graph neural network is trained according to the first loss function of the following formula (1):

[0051] (1)

[0052] in, represents the first loss function, represents the i-th graph in the obtained graph structure sample, i represents the order in the graph structure sample, Representation and Related positive examples, Representatives and Irrelevant negative samples, sim represents the similarity calculation function, represents a hyperparameter, and N represents the total number of graph structure samples obtained.

[0053] In a feasible implementation, the initial graph neural network (GNN) is a deep learning model specifically used to process graph structure data. Graph structure data is very common in many fields, such as social networks, molecular structures, transportation networks, etc. The core idea of ​​the graph neural network is to use the topological structure of the graph to capture the structural information of the graph by learning the features of nodes and edges. The embodiment of the present invention trains the graph neural network with initialized parameters, and the obtained graph encoder is used as part of the final generated graph-to-text generation model based on the large language model, which enhances the ability of the final generated graph-to-text generation model based on the large language model to understand the graph structure.

[0054] The goal of the above formula (1) is to minimize the similarity between each image and its positive samples, while maximizing the dissimilarity with negative samples.

[0055] S3. Set four different training tasks and obtain training samples corresponding to the four training tasks respectively.

[0056] Optionally, four different training tasks of S3 include: node degree prediction task, triple completion task, subgraph filling task and graph depth prediction task, where:

[0057] The node degree prediction task is used to predict the input and output degrees of a specific node. This task facilitates the model to capture the local structure of the graph.

[0058] The triple completion task is used to complete a given triple. This task can guide the model to learn the relationship between nodes based on the input graph.

[0059] The subgraph filling task is used to fill the mask subgraph according to the neighbor graph. This task can help the model learn subgraph level structural information.

[0060] The graph depth prediction task is used to predict the depth of the input graph. The depth of a graph refers to the length of the longest path from the root node to a specific node. This task helps the model achieve a deeper understanding of the graph structure.

[0061] In one feasible implementation, these four tasks are designed in the format of Graph Question-Answering, with the purpose of training a graph-to-text projector and promoting knowledge transfer from graph to text.

[0062] For example, if Figure 3 As shown in Figure 2, the node degree prediction task can be: What is the input node of deep learning? The answer is set to: 1.

[0063] The triple completion task can be: Complete this triple (?, contains, knowledge graph)? The answer is set to: artificial intelligence.

[0064] The subgraph filling task can be: Complete the missing subgraph? The answer is set to: The missing subgraph is xxx.

[0065] The graph depth prediction task can be: What is the height of this input graph? The answer is set to: 3.

[0066] Through these structure-aware instruction fine-tuning tasks, the model not only learns the ability to encode graph structures, but also learns how to transform this structural information into text representation, thereby achieving better performance in graph-to-text generation tasks.

[0067] S4. According to the training samples corresponding to the four training tasks, the initial linear network is trained to obtain a graph-to-text projector.

[0068] Optionally, the specific operation of S4 may include the following steps S41-S44:

[0069] S41. Construct an instruction prompt template according to the training samples corresponding to the four training tasks respectively. The instruction prompt template includes a given diagram, task instructions and answers.

[0070] In a feasible implementation, for the construction of the instruction templates for the four tasks proposed in step S3, the instruction prompt template includes three parts: system message, task description, and answer. In addition, two special tags are added ( <graph> , < / graph> ) to distinguish whether the answer is a textual representation or a graphical representation.

[0071] S42. According to the following formula (2), the probability of answer A appearing under the given graph g and task instruction I is calculated:

[0072] (2)

[0073] in, represents the set of all elements before the jth element in the generated answer sequence A, Represents the total number of elements in the answer sequence A. The probability of answer A appearing is calculated by multiplying the joint probabilities of all possible answer sequences.

[0074] S43. According to the above formula (2), the loss function of the structure-aware guided adjustment task is constructed as follows (3):

[0075] (3)

[0076] in, It is the guidance adjustment dataset, that is, the set of training samples corresponding to the four training tasks, including the triples of given graph, task instruction and answer. In the given graph and instructions In the case of , generate the log-likelihood of answer A.

[0077] This loss function is the sum of the negative log-likelihood over all training examples and is used to optimize the model to correctly predict the answer. This is a summation operation over all training examples.

[0078] S44. Adjust the loss function of the task according to the structure-aware guidance, train the initial linear network, and obtain the image-to-text projector.

[0079] In a feasible implementation, the initial linear network can be a network structure commonly used in the prior art, such as a multilayer perceptron (MLP), a linear convolutional neural network, etc., whose network structure and principle are all prior art, and the embodiment of the present invention will not be described in detail. The initial linear network is used for training, and the obtained graph-to-text projector is used as part of the final generated graph-to-text generation model based on a large language model, so that the model can generate more accurate and coherent text according to the structural features of the graph, improve the semantic richness and structural consistency of the text, and bridge the field gap between graph representation and text representation.

[0080] S5. According to the few-sample image-to-text generation task, select the few-sample learning samples.

[0081] In a feasible implementation, an AMR-to-text task and a KG-to-text task are selected. For the AMR-to-text task, MR2.0 and AMR3.0 corpora are selected for task-aware fine-tuning and evaluation; for KG-to-text, WebNLG extracted from DBpedia is used.

[0082] S6. Use the few-shot learning samples to fine-tune the pre-trained large language model, and obtain a trained image-to-text generation model based on the large language model according to the image encoder, the image-to-text projector, and the fine-tuned large language model.

[0083] In a feasible implementation, before fine-tuning, an instruction template is constructed for each task:

[0084] Prompt: Let's step by step, complete this task!

[0085] Optionally, the specific operation of S6 may include the following steps:

[0086] According to the few-shot learning sample and the target loss function of the following formula (4), the pre-trained large language model is fine-tuned to obtain a fine-tuned large language model.

[0087] (4)

[0088] in, represents the target loss function. It is a few-shot learning sample dataset, which includes input graph H, task instructions and triples of the target text Y. Given an input graph H and a task instruction The log-likelihood of the probability of the target text Y appearing under the condition .

[0089] In one possible implementation, conditional probability is used to measure the model's ability to generate target text by maximizing the model to learn how to generate accurate text output based on the input graph and task instructions.

[0090] After obtaining the fine-tuned large language model, the graph encoder, the image-to-text projector, and the fine-tuned large language model are assembled together to obtain a trained image-to-text generation model based on the large language model.

[0091] S7. Obtain a graph to be generated, and input the graph to be generated into a trained graph-to-text generation model based on a large language model to obtain a generated text corresponding to the graph to be generated.

[0092] In a feasible implementation method, the graph to be generated is input into a graph encoder of a trained graph-to-text generation model based on a large language model to obtain a vector corresponding to the graph to be generated, the vector corresponding to the graph to be generated is input into a graph-to-text projector to obtain a text sequence corresponding to the graph to be generated, and the text sequence is input into a fine-tuned large language model to obtain a generated text corresponding to the graph to be generated.

[0093] The embodiment of the present invention enhances the ability of large language models to understand graph structures through the proposed structure-aware large language model framework and a specific pre-training stage for graph data. Through unsupervised graph comparison learning, the model learns the overall representation of the graph, improves the quality of the graph structure representation, and can distinguish the semantic differences between different graphs, so that the model can learn discriminative features without a large amount of labeled data. Secondly, four structure-aware instruction tasks are introduced to train the graph-to-text projector, so that the model can generate more accurate and coherent text based on the structural features of the graph, improve the semantic richness and structural consistency of the text, and bridge the field gap between graph representation and text representation. Finally, the system is fine-tuned through the few-sample learning technology, and the model is fine-tuned on specific downstream tasks, which reduces the dependence on large-scale labeled data sets and enhances the application potential of the model in data-scarce fields. Through the above process, the model not only learns the encoding ability of the graph structure, but also learns how to convert these structural information into text representation, and achieves better performance in the graph-to-text generation task.

[0094] Figure 4 The block diagram of a device for generating text from a graph based on a large language model of few-sample learning is shown according to an exemplary embodiment. The device is used for a method for generating text from a graph based on a large language model of few-sample learning. Figure 4The device includes a construction module 410, a training module 420, a setting module 430, an optimization module 440, a selection module 450, a fine-tuning module 460 and a generation module 470. Among them:

[0095] A construction module 410 is used to construct positive samples and negative samples;

[0096] A training module 420 is used to train the initial graph neural network through the constructed positive samples and negative samples to obtain a graph encoder;

[0097] A setting module 430 is used to set four different training tasks and obtain training samples corresponding to the four training tasks respectively;

[0098] An optimization module 440 is used to train the initial linear network according to the training samples corresponding to the four training tasks to obtain a graph-to-text projector;

[0099] A selection module 450 is used to select a few-sample learning sample according to a few-sample image-to-text generation task;

[0100] A fine-tuning module 460 is used to fine-tune the pre-trained large language model using the few-shot learning sample, and obtain a trained large language model-based image-to-text generation model according to the image encoder, the image-to-text projector, and the fine-tuned large language model;

[0101] The generation module 470 is used to obtain the graph to be generated, input the graph to be generated into the trained graph-to-text generation model based on the large language model, and obtain the generated text corresponding to the graph to be generated.

[0102] Optionally, the construction module 410 is used to:

[0103] S11, obtaining a graph structure sample, and taking the graph structure sample as a root node;

[0104] S12, rotate the two subtrees of the root node, and use the graph structure obtained after the rotation as a positive example;

[0105] S13, deleting some nodes in the root node to obtain the first part of negative example samples;

[0106] S14, adding some nodes to the root node to obtain the second part of negative example samples;

[0107] S15. Exchange some nodes at the root node to obtain the third part of negative example samples.

[0108] Optionally, the training module 420 is used to:

[0109] The positive and negative samples are input into the initial graph neural network, and the initial graph neural network is trained according to the first loss function of the following formula (1):

[0110] (1)

[0111] in, represents the first loss function, represents the i-th graph in the obtained graph structure sample, i represents the order in the graph structure sample, Representation and Related positive examples, Representatives and Irrelevant negative samples, sim represents the similarity calculation function, represents a hyperparameter, and N represents the total number of graph structure samples obtained.

[0112] Optionally, the four different training tasks include: a node degree prediction task, a triple completion task, a subgraph filling task and a graph depth prediction task, wherein:

[0113] The node degree prediction task is used to predict the input and output degrees of a specific node;

[0114] The triple completion task is used to complete a given triple;

[0115] The sub-graph filling task is used to fill the mask sub-graph according to the neighbor graph;

[0116] The graph depth prediction task is used to predict the depth of an input graph, where the depth of a graph refers to the length of the longest path from a root node to a specific node.

[0117] Optionally, the optimization module 440 is used to:

[0118] S41, constructing an instruction prompt template according to the training samples corresponding to the four training tasks respectively, wherein the instruction prompt template includes a given diagram, a task instruction and an answer;

[0119] S42. According to the following formula (2), the probability of answer A appearing under the given graph g and task instruction I is calculated:

[0120] (2)

[0121] in, represents the set of all elements before the jth element in the generated answer sequence A, Represents the total number of elements in the answer sequence A;

[0122] S43. According to the above formula (2), the loss function of the structure-aware guided adjustment task is constructed as follows (3):

[0123] (3)

[0124] in, It is the guidance adjustment dataset, that is, the set of training samples corresponding to the four training tasks, including the triples of given graph, task instruction and answer. In the given graph and instructions In the case of , generate the log-likelihood of answer A;

[0125] S44. Adjust the loss function of the task according to the structure-aware guidance, train the initial linear network, and obtain the image-to-text projector.

[0126] Optionally, the fine-tuning module 460 is used to:

[0127] According to the few-shot learning sample and the target loss function of the following formula (4), the pre-trained large language model is fine-tuned to obtain a fine-tuned large language model;

[0128] (4)

[0129] in, represents the target loss function; It is a few-shot learning sample dataset, which includes input graph H, task instructions and the triple of the target text Y; Given an input graph H and a task instruction The log-likelihood of the probability of the target text Y appearing under the condition .

[0130] The embodiment of the present invention enhances the ability of large language models to understand graph structures through the proposed structure-aware large language model framework and a specific pre-training stage for graph data. Through unsupervised graph comparison learning, the model learns the overall representation of the graph, improves the quality of the graph structure representation, and can distinguish the semantic differences between different graphs, so that the model can learn discriminative features without a large amount of labeled data. Secondly, four structure-aware instruction tasks are introduced to train the graph-to-text projector, so that the model can generate more accurate and coherent text based on the structural features of the graph, improve the semantic richness and structural consistency of the text, and bridge the field gap between graph representation and text representation. Finally, the system is fine-tuned through the few-sample learning technology, and the model is fine-tuned on specific downstream tasks, which reduces the dependence on large-scale labeled data sets and enhances the application potential of the model in data-scarce fields. Through the above process, the model not only learns the encoding ability of the graph structure, but also learns how to convert these structural information into text representation, and achieves better performance in the graph-to-text generation task.

[0131] Figure 5 is a structural schematic diagram of a device for generating graphs from large language models based on few-sample learning provided by an embodiment of the present invention, such as Figure 5 As shown, the graph-to-text generation device based on a large language model of few-sample learning may include the above Figure 4 The illustrated method of fine-tuning the pre-trained large language model based on a few samples to obtain the fine-tuned large language model is an apparatus for generating graphs from large language models learned. Optionally, the apparatus 510 for generating graphs from large language models learned based on a few samples may include a first processor 2001.

[0132] Optionally, the graph-to-text generation device 510 based on a large language model of few-sample learning may further include a memory 2002 and a transceiver 2003 .

[0133] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0134] Combine the following Figure 5 The components of the graph-to-text generation device 510 for a large language model based on few-sample learning are specifically introduced as follows:

[0135] The first processor 2001 is the control center of the device 510 for generating graphs from texts based on a large language model of few-sample learning, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0136] Optionally, the first processor 2001 can perform various functions of the graph-to-text generation device 510 of a large language model based on few-sample learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0137] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0138] In a specific implementation, as an embodiment, the graph-to-text generation device 510 based on a large language model of few-sample learning may also include multiple processors, such as Figure 5 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0139] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0140] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0141] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0142] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0143] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0144] It should be noted that Figure 5 The structure of the graph-to-text generation device 510 of the large language model based on few-sample learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0145] In addition, the technical effects of the graph-to-text generation device 510 based on a large language model of few-sample learning can refer to the technical effects of the graph-to-text generation method based on a large language model of few-sample learning described in the above method embodiment, and will not be repeated here.

[0146] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0147] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0148] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0149] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0150] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0151] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0152] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0158] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A graph-to-text generation method based on a large language model with few-shot learning, characterized in that: The method comprises: S1. Construct positive and negative samples; S2. Train the initial graph neural network through the constructed positive and negative samples to obtain a graph encoder. S3. Set four different training tasks and obtain training samples corresponding to the four training tasks respectively; S4, training the initial linear network according to the training samples corresponding to the four training tasks, and obtaining the image-to-text projector; S5. Selecting a few-shot learning sample based on a few-shot image-to-text generation task; S6. Fine-tune the pre-trained large language model using the few-shot learning sample, and obtain a trained large language model-based image-to-text generation model according to the image encoder, the image-to-text projector, and the fine-tuned large language model; S7, obtaining a graph to be generated, inputting the graph to be generated into a trained graph-to-text generation model based on a large language model, and obtaining a generated text corresponding to the graph to be generated; The constructing of positive samples and negative samples in S1 includes: S11, obtaining a graph structure sample, and taking the graph structure sample as a root node; S12, rotate the two subtrees of the root node, and use the graph structure obtained after the rotation as a positive example; S13, deleting some nodes in the root node to obtain the first part of negative example samples; S14, adding some nodes to the root node to obtain the second part of negative example samples; S15, exchanging some nodes at the root node to obtain the third part of negative example samples; The S2 trains the initial graph neural network by constructing positive samples and negative samples, including: The positive and negative samples are input into the initial graph neural network, and the initial graph neural network is trained according to the first loss function of the following formula (1): （1） in, represents the first loss function, h i represents the i-th graph in the obtained graph structure sample, i represents the order in the graph structure sample, h i + Indicates that i The relevant positive examples, h i - Represents and h i Irrelevant negative samples, sim represents the similarity calculation function, represents a hyperparameter, and N represents the total number of graph structure samples obtained; The step S4 trains the initial linear network according to the training samples corresponding to the four training tasks to obtain the image-to-text projector, including: S41, constructing an instruction prompt template according to the training samples corresponding to the four training tasks respectively, wherein the instruction prompt template includes a given diagram, a task instruction and an answer; S42, according to the following formula (2), calculate the g and mission instructions I Under the condition, the probability of answer A is: （2） in, Represents the generated answer A Middle j The set of all elements before the element, Indicates the total number of elements in answer A; （3） in, It is the guidance adjustment dataset, that is, the set of training samples corresponding to the four training tasks, including the triples of given graph, task instruction and answer. In the given graph and instructions In the case of A The log-likelihood of ; S44. Adjust the loss function of the task according to the structure-aware guidance, train the initial linear network, and obtain the image-to-text projector.

2. The method for generating text from a large language model using few-sample learning according to claim 1, characterized in that: The four different training tasks of S3 include: node degree prediction task, triple completion task, subgraph filling task and graph depth prediction task, where: The node degree prediction task is used to predict the input and output degrees of a specific node; The triple completion task is used to complete a given triple; The sub-graph filling task is used to fill the mask sub-graph according to the neighbor graph; The graph depth prediction task is used to predict the depth of an input graph, where the depth of a graph refers to the length of the longest path from a root node to a specific node.

3. The method for generating text from a large language model using few-sample learning according to claim 1, characterized in that: The S6 uses a few-shot learning sample to fine-tune the pre-trained large language model, including: According to the few-shot learning sample and the target loss function of the following formula (4), the pre-trained large language model is fine-tuned to obtain a fine-tuned large language model; （4） in, represents the target loss function; is a few-shot learning sample dataset, which contains the input graph , Mission Instructions and the triple of the target text Y; Given an input graph and mission instructions The log-likelihood of the probability of the target text Y appearing under the condition .

4. A device for generating text from a graph based on a large language model with few-sample learning, wherein the device for generating text from a graph based on a large language model with few-sample learning is used to implement the method for generating text from a graph based on a large language model with few-sample learning as claimed in any one of claims 1 to 3, characterized in that: The device comprises: Construction module, used to construct positive samples and negative samples; The training module is used to train the initial graph neural network through the constructed positive and negative samples to obtain the graph encoder; A setting module is used to set four different training tasks and obtain training samples corresponding to the four training tasks respectively; An optimization module is used to train the initial linear network according to the training samples corresponding to the four training tasks to obtain a graph-to-text projector; A selection module is used to select few-shot learning samples based on a few-shot image-to-text generation task; A fine-tuning module is used to fine-tune the pre-trained large language model using the few-shot learning samples, and obtain a trained image-to-text generation model based on the large language model according to the image encoder, the image-to-text projector, and the fine-tuned large language model; A generation module, used for obtaining a graph to be generated, inputting the graph to be generated into a trained graph-to-text generation model based on a large language model, and obtaining a generated text corresponding to the graph to be generated; Wherein, the building block is used for: S11, obtaining a graph structure sample, and taking the graph structure sample as a root node; S12, rotate the two subtrees of the root node, and use the graph structure obtained after the rotation as a positive example; S13, deleting some nodes in the root node to obtain the first part of negative example samples; S14, adding some nodes to the root node to obtain the second part of negative example samples; S15, exchanging some nodes at the root node to obtain the third part of negative example samples; Optionally, the training module is used to: The positive and negative samples are input into the initial graph neural network, and the initial graph neural network is trained according to the first loss function of the following formula (1): （1） in, represents the first loss function, h i represents the i-th graph in the obtained graph structure sample, i represents the order in the graph structure sample, h i + Indicates that i The relevant positive examples, h i - Represents and h i Irrelevant negative sample, sim represents the similarity calculation function, represents the hyperparameter, and N represents the total number of graph structure samples obtained; Wherein, the optimization module is used to: S41, constructing an instruction prompt template according to the training samples corresponding to the four training tasks respectively, wherein the instruction prompt template includes a given diagram, a task instruction and an answer; S42, according to the following formula (2), calculate the g and mission instructions I Under the condition, the probability of answer A is: （2） in, Represents the generated answer A Middle j The set of all elements before the element, Indicates the total number of elements in answer A; S43. According to the above formula (2), the loss function of the structure-aware guided adjustment task is constructed as follows (3): （3） in, It is the guidance adjustment dataset, that is, the set of training samples corresponding to the four training tasks, including the triples of given graph, task instruction and answer. In the given graph and instructions In the case of A The log-likelihood of ; S44. Adjust the loss function of the task according to the structure-aware guidance, train the initial linear network, and obtain the image-to-text projector.

5. A graph-to-text generation device based on a large language model of few-sample learning, characterized in that: The device for generating text from a graph using a large language model based on few-sample learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images