Method and device for evaluating structured knowledge prompt framework, equipment and medium

Abstract

The embodiment of the invention provides a method and device for evaluating a structured knowledge prompt framework, equipment and a medium. According to the scheme, the method comprises the steps that first cue word information is converted into vectorized first sample data, first structured background knowledge is converted into vectorized first structured knowledge cue information through a structured knowledge cue framework, and the first structured background knowledge is background knowledge related to the first cue word information; training the first large language model based on vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information to obtain a first trained large language model; testing the first trained large language model by using the first test data, and determining a first performance index of the first trained large language model; and evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model evaluation, and in particular to a method and device for evaluating a structured knowledge prompting framework, equipment and a medium. BACKGROUND

[0002] When using a large language model to process various tasks, structured information generated by providing a structured knowledge prompting framework is usually used to enhance the reasoning ability of the large language model, thereby improving the effect of task processing.

[0003] However, the specific role and effectiveness of the structured knowledge prompting framework in improving the reasoning performance of the large language model have not been deeply explored in the current technology. In actual application, the enhancement effect of the structured knowledge prompting framework on the large language model may not be as significant as expected, resulting in that the large language model fails to achieve the expected performance when performing tasks.

[0004] Therefore, how to evaluate the structured knowledge prompting framework has become a technical problem to be solved. SUMMARY

[0005] The embodiments of the present specification provide a method, device, equipment and medium for evaluating a structured knowledge prompting framework, so as to realize the evaluation of the structured knowledge prompting framework.

[0006] To solve the above technical problems, the embodiments of the present specification are implemented as follows.

[0007] In a first aspect, the embodiments of the present specification provide a method for evaluating a structured knowledge prompting framework, the structured knowledge prompting framework being used to convert structured knowledge into vector information for inputting into a large language model, and the method comprising: vectorizing the first prompt word information to generate vectorized first sample data; converting first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompting information by using the structured knowledge prompting framework; training the first large language model based on vectorized first training data generated by splicing the first sample data and the first structured knowledge prompting information, to obtain a first trained large language model; testing the first trained large language model by using first test data, to determine a first performance indicator of the first trained large language model; and evaluating the generalization ability of the structured knowledge prompting framework in a target direction according to the first performance indicator.

[0008] In a second aspect, the embodiments of the present specification provide a device for evaluating a structured knowledge prompting framework, the structured knowledge prompting framework being used to convert structured knowledge into vector information for inputting into a large language model, and the device comprising: The generating module is configured to perform vector conversion on the first prompt word information to generate vectorized first sample data. The converting module is configured to convert first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompt information by using the structured knowledge prompt framework. The training module is configured to train the first large language model based on vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information, to obtain a first trained large language model. The determining module is configured to test the first trained large language model by using first test data, and determine a first performance index of the first trained large language model. The evaluating module is configured to evaluate the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index.

[0009] In a third aspect, an apparatus for computing is provided, and the apparatus includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for evaluating a structured knowledge prompt framework.

[0010] In a fourth aspect, a computer-readable medium is provided, and the computer-readable medium stores computer-readable instructions executable by a processor to implement the method for evaluating a structured knowledge prompt framework.

[0011] At least one embodiment of the present specification can achieve the following beneficial effects: The present solution combines sample data with structured prompt information generated by a structured knowledge prompt framework to construct training data for training a large language model. After training the large language model using these training data, the performance of the trained large language model is evaluated using test data, and then a performance index is obtained. By analyzing these performance indexes, the generalization ability of the structured knowledge prompt framework in enhancing the performance of the large language model can be evaluated to achieve the evaluation of the structured knowledge prompt framework. Further, according to the evaluation result of the structured knowledge prompt framework, the corresponding parameters of the structured knowledge prompt framework can be reasonably adjusted to enable the structured knowledge prompt framework to effectively enhance the ability of the large language model, and thus improve the performance of the large language model. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments described in the present specification, and all other drawings obtained by those of ordinary skill in the art based on these drawings without creative labor should also fall within the scope of protection of one or more embodiments of the present specification.

[0013] Figure 1 is an application scenario diagram of a method for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification. Figure 2 is a flow diagram of a method for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification. Figure 3 is a structural diagram of a device for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification. Figure 2 Figure 4 is a structural diagram of an apparatus for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of one or more embodiments of the present specification more clear, the technical solutions of one or more embodiments of the present specification will be described clearly and completely below in combination with specific embodiments of the present specification and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present specification, not all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of one or more embodiments of the present specification.

[0015] In order to facilitate the understanding of the present specification, at least some of the terms of the present specification are explained as follows: Large language model: Large language model (LLM) refers to a deep learning model trained using a large amount of text data, so that the model can generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on various topics by training on a large dataset. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, to a certain extent, to simulate the language cognition and generation process of human beings.

[0016] ​Structured knowledge prompting: structured knowledge prompting (SKP) is used to encode structured knowledge through a structure encoder to obtain a structured representation vector, and use these vectors as prompts to enhance large language models.

[0017] Prompt information: generally refers to specific guiding or leading content given when interacting with a large language model, which is used to help the large language model understand the user's needs, questions or tasks. Prompt information can help the large language model better parse the user's intent to ensure that appropriate and accurate responses can be provided.

[0018] Current large language models in the field of natural language processing have shown excellent performance in text generation, but there is still room for further improvement in the accuracy of the generated text. The structured knowledge prompting framework, as a method of integrating external knowledge into large language models, has achieved certain results in many knowledge-intensive tasks by combining structured representations. However, existing research mostly focuses on the ability of the structured knowledge prompting framework to handle specific problems, and lacks in-depth exploration of its generalization ability in different scenarios and performance boundaries. Therefore, it is urgent to conduct more comprehensive research and analysis on the generalization ability of the structured knowledge prompting framework to fully exploit its potential in large language models.

[0019] The technical solutions provided by the embodiments of the present specification will be described in detail below with reference to the accompanying drawings. Figure 1 is a schematic diagram of an application scenario for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification.

[0020] As shown in Figure 1 , the application scenario schematic diagram includes a large language model 101 and a server 102.

[0021] In the embodiments of the present specification, the large language model 101 can be an artificial intelligence model trained through a large amount of text data, wherein the large language model 101 can include but is not limited to at least one of the GPT series model, the BERT system model, the T5 series model, and the DeepSeek series model.

[0022] The server 102 can include but is not limited to at least one of any device, equipment, platform, device cluster or cloud computing service center with computing and processing capabilities.

[0023] A communication connection is established between the large language model 101 and the server 102. The communication connection method may include, but is not limited to, a local area network connection, a wide area network connection, an Internet connection, a short-range communication connection, or other types of data network connections. Short-range communication connections include, but are not limited to, Near Field Communication (NFC), local area network, Bluetooth, infrared, and other connection methods.

[0024] A structured knowledge hinting framework can be deployed on the large language model 101 to enhance its ability to handle tasks. Based on this framework, the large language model 101's task-handling capabilities can be improved. The server 102 can train the large language model 101, test the trained model, and evaluate the generalization ability of the structured knowledge hinting framework in enhancing the large language model 101's task-handling capabilities based on performance metrics determined after testing.

[0025] Next, a method for evaluating a structured knowledge prompting framework, provided in the embodiments of the specification, will be described in conjunction with the accompanying drawings.

[0026] Figure 2 This is a flowchart illustrating a method for evaluating a structured knowledge prompting framework provided in an embodiment of this specification. From a programming perspective, the entity executing the process can be a program mounted on a server used for evaluating the structured knowledge prompting framework. From a hardware perspective, the entity executing the process can be a device used for evaluating the structured knowledge prompting framework.

[0027] like Figure 2 As shown, the process may include the following steps: Step 202: Perform vector transformation on the first prompt word information to generate vectorized first sample data.

[0028] In the embodiments of this specification, cue word information can be input information used in a large language model to guide the model to generate specific outputs. It tells the model how to generate responses or perform reasoning. By precisely designing cue word information, the model can be better guided to complete a task. Cue word information can include simple text, as well as information about the task's structure, context, or specific requirements. The task's structure can be the organization of the task itself and the relationships between its components when performing a specific task. It can describe the task's steps, the required information, and the framework for processing that information.

[0029] In the embodiments of the present specification, the manner of vector conversion for the first prompt word information can be to convert the first prompt word information into a digital form that can be understood by the large language model. Specifically, the vector conversion can be performed on the first prompt word information through word embedding technology, which can map each word to a fixed-length real-valued vector.

[0030] Step 204: converting first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompt information using the structured knowledge prompt framework.

[0031] In the embodiments of the present specification, the structured knowledge prompt framework (SKP framework) can be deployed as a functional module on the large language model, or it can also be deployed as an independent device and interact with the large language model through communication connection.

[0032] In the embodiments of the present specification, the SKP framework can be a framework for systematically organizing and prompting knowledge, aiming to help the large language model better understand and apply knowledge, especially to enhance the ability for specific tasks, which can include at least one of text generation tasks, question and answer tasks, classification tasks, dialogue tasks, information extraction tasks, and reasoning tasks. The SKP framework can convert structured knowledge into vectorized data. Structured knowledge can be represented in a systematic and organized manner, so that each part of the knowledge has a clear hierarchy, category and relationship. This way, knowledge can be more easily understood, processed and applied.

[0033] In practical applications, the first structured background knowledge can be knowledge associated with the first prompt word information, such as: if the first prompt word information is "which drugs should diabetic patients take to control blood sugar", then the corresponding first structured background knowledge can be structured knowledge related to diabetes, specifically knowledge related to controlling blood sugar levels, such as a knowledge graph or related medical information structured content.

[0034] In the embodiments of the present specification, the SKP framework can be used to convert the first structured background knowledge into vectorized first structured knowledge prompt information. Specifically, the knowledge graph embedding technology can be used to convert the first structured background knowledge into vectorized first structured knowledge prompt information, where the knowledge graph embedding technology can map the nodes (entities) and edges (relationships) in the knowledge graph into a continuous vector space, where each node and edge is represented by a low-dimensional vector. Through this representation, the semantic and structural information between entities and relationships is preserved in the vector space, thereby supporting the large language model for further learning and reasoning.

[0035] Step 206: training the first large language model based on the vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information, to obtain a first trained large language model.

[0036] In the embodiments of the present specification, the vectorized first sample data and the vectorized first structured knowledge prompt information can both be vectorized data recognizable by the large language model. The first sample data and the first structured knowledge prompt information are arranged in an element order together to form a new and longer vector, to obtain the first training data. Specifically, the two vectors can be combined into one vector through vector splicing technology or vector merging technology. In actual application, if the first sample data is [a1, a2, a3] and the first structured knowledge prompt information is [b1, b2], the generated first training data can be [a1, a2, a3, b1, b2].

[0037] In the embodiments of the present specification, the first training data can include a plurality of training data. The first large language model is trained based on the first training data to obtain a trained large language model. Each training data can include question information and answer information. In actual application, in the process of training the first large language model using the first training data, the question information in the training data is first input into the model, and output information is generated by the model. Then, the difference between the output information and the answer information is compared, and the corresponding loss value is calculated. Based on the loss value, the parameters of the model are adjusted to obtain an updated large language model. Then, the second batch of training data is used to train the adjusted model again. This process is iterated until a preset number of training times is reached, or the accuracy of the model meets a predetermined standard. Finally, a trained large language model is obtained.

[0038] Step 208: testing the first trained large language model using first test data to determine a first performance indicator of the first trained large language model.

[0039] In the embodiments of the present specification, the first test data can be vectorized data generated by splicing vectorized target prompt word information and vectorized target structured knowledge prompt information. The target structured knowledge prompt information can be information generated by vector conversion of target structured background knowledge related to the target prompt word information using the SKP framework.

[0040] In the embodiments of the present specification, the first training data can be data for training the first large language model in the ability of processing the first task. The first test data can be data for testing the ability of the first trained large language model in processing the first task, or the first test data can also be data for testing the ability of the first trained large language model in processing the second task, wherein the first task and the second task are different tasks.

[0041] In the embodiments of the present specification, the first test data can include question information and answer information. In the process of testing the first trained large language model by using the first test data, the question information in the test data is first input into the model, the output information is generated by the model, then the difference information between the output information and the answer information is compared, and the performance index of the first trained large language model in processing the target task is determined based on the difference information. The target task can be the task corresponding to the first test data.

[0042] In actual application, the first test data can include several test data, and a difference information can be determined based on each test data. The performance index of the first trained large language model in processing the target task based on the difference information can include: obtaining each target difference information in each difference information meeting a preset requirement, calculating a first quantity value corresponding to each target difference information, calculating a total quantity value corresponding to each difference information, determining the performance index of the first trained large language model based on the ratio between the first quantity value and the total quantity value, wherein the performance index can be the accuracy.

[0043] Step 210: evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index.

[0044] In the embodiments of the present specification, the generalization ability in the target direction can include at least one of the generalization ability for reflecting whether the SKP framework has the ability to enhance the large language model to process the task on the corresponding data granularity type of structured knowledge prompt information, the generalization ability for reflecting whether the SKP framework has the transferability of the enhancement effect when processing different tasks of the large language model, the generalization ability for reflecting whether the SKP framework has the scalability of the enhancement effect for different computing capabilities of the large language model of the same type, and the generalization ability for reflecting whether the SKP framework has the universality of the enhancement effect for different types of large language models.

[0045] In actual applications, if the first performance index is greater than or equal to the preset threshold, it indicates that the generalization capability of the SKP framework in the target direction meets the expected effect; if the first performance index is less than the preset threshold, it indicates that the generalization capability of the SKP framework in the target direction does not meet the expected effect, and then the corresponding parameters of the SKP framework can be adjusted based on the evaluation result of the SKP framework to improve the generalization capability of the SKP framework in the target direction.

[0046] It should be understood that the order of some of the steps in the method described in one or more embodiments of the present specification can be exchanged according to actual needs, or some of the steps can be omitted or deleted.

[0047] Figure 2 The method in the method can evaluate the generalization capability of the SKP framework in enhancing the performance of the large language model by analyzing the performance index tested for the trained large language model, so as to realize the evaluation of the SKP framework.

[0048] Further, based on the current parameter settings (such as hyperparameters, architecture information, etc.) of the SKP framework, structured knowledge prompt information is generated, and the large language model is trained based on the structured knowledge prompt information. After the training of the large language model is completed, the performance of the large language model is tested, and the test index can include accuracy, recall rate, loss function value, etc., depending on the nature of the task processed by the large language model. According to the test result, feedback information is generated, the performance of the large language model is analyzed, and problems (for example, overfitting, underfitting, inaccurate prediction of the model for a specific class, etc.) are found out. According to the test result and the feedback information, the parameters of the SKP framework can be adjusted, such as increasing or decreasing the hyperparameter value of the SKP framework, increasing or decreasing the number of layers of the architecture of the SKP framework, changing the activation function or node number of the neural network in the SKP framework, etc. Further, after adjusting the parameters of the SKP framework based on the test result and the feedback information, the ability of the SKP framework to enhance the large language model can be improved, thereby improving the performance of the large language model.

[0049] Based on the method, Figure 2 The embodiments of the present specification also provide some specific implementation schemes of the method, which are described below.

[0050] In the embodiments of the present specification, a specific implementation is proposed for the evaluation method for reflecting whether the SKP framework has the ability to enhance the large language model to process tasks on the structured knowledge prompt information of the corresponding data granularity type.

[0051] Optionally, the data granularity type of the first prompt word information includes any one of entity granularity, triple granularity and subgraph granularity; and the data granularity type of the first structured background knowledge is the same as the data granularity type of the first prompt word information.

[0052] Correspondingly, the evaluating, according to the first performance indicator, the generalization capability of the structured knowledge prompt framework in the target direction can specifically include: if the first performance indicator is greater than or equal to a first threshold, determining that the structured knowledge prompt framework has a generalization capability of enhancing the reasoning capability of the first large language model in a first granularity dimension; the first granularity is a data granularity type of the first prompt word information.

[0053] In the embodiments of the present specification, the data granularity type corresponding to the first prompt word information can include any one of entity granularity (EG), triple granularity (TG), and subgraph granularity (SG). The entity granularity can represent processing with a single entity in a knowledge graph as the smallest unit of information, and each entity can be a specific object, concept, thing, etc. For example, in the medical field, the entity granularity can involve entities such as "diabetes" or "insulin", which are represented and processed individually. The triple granularity can represent a triple (entity-relation-entity) in a knowledge graph as the smallest unit of information, and the triple is the core structure in the knowledge graph, used to represent the relationship between entities, and each triple is composed of a head entity, a relationship, and a tail entity, usually represented as (entity1, relationship, entity2). At this granularity, the information in the knowledge graph can be represented as different triple combinations, and the relationship of each triple is considered as the semantic connection between entities. For example, in the medical field, the triple is used to represent the relationship between entities for treating diabetes as "diabetes" - "treatment" - "insulin". The subgraph granularity can represent a part of the structure (i.e., subgraph) in the knowledge graph as a processing unit, and the subgraph can be composed of multiple entities and their relationships, representing a certain specific part or field in the knowledge graph. The subgraph granularity can contain one or more triples, forming a local structure about a specific topic or concept. This granularity can provide richer contextual information, thereby better supporting reasoning and complex tasks. For example, when processing a subgraph of a disease, the subgraph can contain all entities and relationships related to "diabetes", such as the relationship between diabetes and insulin, treatment methods, complications, etc.

[0054] In the embodiments of the present specification, since the first structured background knowledge is knowledge related to the first prompt word information, the data granularity type of the first structured background knowledge is the same as the data granularity type of the first prompt word information.

[0055] In the embodiments of the present specification, the first training data can be data for training the first large language model in the ability of processing the first task. The first test data can also be data for testing the ability of the first trained large language model in processing the first task. The data granularity type of the prompt word information used to generate the first training data is the same as the data granularity type of the prompt word information used to generate the first test data, and the data granularity type of the prompt word information used to generate the first test data is the same as the data granularity type of the structured background knowledge used to generate the first test data.

[0056] In the embodiments of the present specification, the first threshold value can be a value determined according to actual needs, such as 90%, 95%, or any other value. It should be noted that the larger the first threshold value, the greater the enhancement effect of the structured knowledge prompt framework on the reasoning ability of the first large language model in the first granularity dimension.

[0057] In the embodiments of the present specification, when the first performance indicator is greater than or equal to the first threshold value, if the data granularity type of the first prompt word information is entity granularity, it is determined that the SKP framework has an enhancing effect on the reasoning ability of the first large language model in the entity granularity dimension; if the data granularity type of the first prompt word information is triple granularity, it is determined that the SKP framework has an enhancing effect on the reasoning ability of the first large language model in the triple granularity dimension; if the data granularity type of the first prompt word information is subgraph granularity, it is determined that the SKP framework has an enhancing effect on the reasoning ability of the first large language model in the subgraph granularity dimension.

[0058] In the embodiments of the present specification, when the first performance indicator is less than the first threshold value, if the data granularity type of the first prompt word information is entity granularity, it is determined that the SKP framework does not have an enhancing effect on the reasoning ability of the first large language model in the entity granularity dimension; if the data granularity type of the first prompt word information is triple granularity, it is determined that the SKP framework does not have an enhancing effect on the reasoning ability of the first large language model in the triple granularity dimension; if the data granularity type of the first prompt word information is subgraph granularity, it is determined that the SKP framework does not have an enhancing effect on the reasoning ability of the first large language model in the subgraph granularity dimension.

[0059] In the embodiments of the present specification, in the evaluation process, for each evaluation, in addition to being able to evaluate the generalization ability of the SKP framework in enhancing the reasoning ability of the first large language model in one granularity dimension, the generalization ability of the SKP framework in enhancing the reasoning ability of the first large language model in multiple granularity dimensions can also be evaluated.

[0060] Optionally, the method can further include: training the first large language model based on the vectorized second training data generated by splicing the second sample data and the second structured knowledge prompt information, to obtain a second trained large language model, wherein the second sample data is vectorized data generated by performing vector conversion on the second prompt word information, and the second structured knowledge prompt information is vectorized information generated by performing vector conversion on the second structured background knowledge related to the second prompt word information using the structured knowledge prompt framework; the data granularity type of the second prompt word information is different from the data granularity type of the first prompt word information, and the data granularity type of the second structured background knowledge is the same as the data granularity type of the second prompt word information; and testing the second trained large language model using second test data to determine a second performance indicator of the second trained large language model.

[0061] Correspondingly, the evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance indicator can specifically include: if the first performance indicator is greater than or equal to the first threshold value, and the second performance indicator is greater than or equal to the second threshold value, it is determined that the structured knowledge prompt framework has a generalization ability of enhancing the reasoning ability of the first large language model in the first granularity dimension and the second granularity dimension, and the second granularity is the data granularity type of the second prompt word information.

[0062] In the embodiments of the present specification, the explanations of the second prompt word information, the second structured knowledge prompt information, the second sample data, and the second training data can correspond to the explanations of the first prompt word information, the first structured knowledge prompt information, the first sample data, and the first training data respectively, which will not be repeated here. The data granularity type of the second prompt word information is different from the data granularity type of the first prompt word information, that is, if the data granularity type of the first prompt word information is entity granularity, the data granularity type of the second prompt word information can be triple granularity or subgraph granularity. The data granularity type of the second structured background knowledge is the same as the data granularity type of the second prompt word information, that is, the data granularity type of the second structured background knowledge and the data granularity type of the second prompt word information can be any one of entity granularity, triple granularity, and subgraph granularity.

[0063] In the embodiments of the present specification, the manner of training the first large language model using the second training data can refer to the manner of training the first large language model using the first training data, which will not be repeated here.

[0064] In the embodiments of the present specification, the explanation for the second test data can refer to the explanation for the first test data, which is not repeated here. The second training data can be data for training the first large language model in the ability of processing the second task. The second test data can also be data for testing the ability of the first trained large language model in processing the second task. The data granularity type of the prompt word information used to generate the second training data is the same as the data granularity type of the prompt word information used to generate the second test data, and the data granularity type of the prompt word information used to generate the second test data is the same as the data granularity type of the structured background knowledge used to generate the second test data.

[0065] In the embodiments of the present specification, the way of testing the second trained large language model with the second test data can refer to the way of testing the first trained large language model with the first test data, and the explanation for the second performance indicator can refer to the explanation for the first performance indicator, which is not repeated here.

[0066] In the embodiments of the present specification, the determination manner of the second threshold value can refer to the determination manner of the first threshold value, the second threshold value can be the same as the first threshold value, or the second threshold value can be different from the first threshold value, which is not limited.

[0067] In actual applications, the first granularity can be the data granularity type of the first prompt word information, and the second granularity can be the data granularity type of the second prompt word information. If the first performance indicator is greater than or equal to the first threshold value, and the second performance indicator is greater than or equal to the second threshold value, it can be determined that the SKP framework has a generalization ability of enhancing the inference ability of the first large language model in the first granularity dimension and the second granularity dimension. If the first performance indicator is greater than or equal to the first threshold value, and the second performance indicator is less than the second threshold value, it can be determined that the SKP framework has a generalization ability of enhancing the inference ability of the first large language model in the first granularity dimension, but not in the second granularity dimension. If the first performance indicator is less than the first threshold value, and the second performance indicator is greater than or equal to the second threshold value, it can be determined that the SKP framework has a generalization ability of not enhancing the inference ability of the first large language model in the first granularity dimension, but enhancing the inference ability of the first large language model in the second granularity dimension. If the first performance indicator is less than the first threshold value, and the second performance indicator is less than the second threshold value, it can be determined that the SKP framework does not have a generalization ability of enhancing the inference ability of the first large language model in the first granularity dimension and the second granularity dimension.

[0068] In the embodiments of the present specification, in order to improve the evaluation diversity of the method for enhancing the task processing capability of the large language model by the SKP framework, the performance of the framework in the corresponding data granularity type can be evaluated, so as to more comprehensively evaluate the applicability and performance of the framework in different scenarios, and then based on the evaluation result, the user can reasonably adjust the parameters of the SKP framework, so that the SKP framework can more effectively enhance the capability of the large language model, thereby improving the performance of the large language model.

[0069] In the embodiments of the present specification, a specific embodiment is proposed for the evaluation method for reflecting whether the generalization capability of the SKP framework has the transferability of the enhancement effect when the large language model processes different tasks.

[0070] Optionally, the first large language model is trained based on the vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information, and specifically can include training the first large language model by using the first training data for instructing the large language model to perform the first task.

[0071] Optionally, the first trained large language model is tested by using the first test data, and specifically can include testing the first trained large language model by using the first test data for instructing the large language model to perform the second task; the first test data includes third sample data and third structured knowledge prompt information; the data granularity type of the third prompt information corresponding to the third sample data is different from the data granularity type of the first prompt information, and / or the task type corresponding to the first task is different from the task type corresponding to the second task; the data granularity type includes any one of entity granularity, triple granularity and subgraph granularity, and the task type includes any one of binary classification task, multiple choice task and description generation task.

[0072] Optionally, the generalization capability of the structured knowledge prompt framework in the target direction is evaluated according to the first performance index, and specifically can include: if the first performance index is greater than or equal to a first threshold, it is determined that the structured knowledge prompt framework has the generalization capability of transferability; the transferability is used to reflect that the structured knowledge prompt framework has the characteristic of processing new tasks.

[0073] In the embodiments of the present specification, the first training data can be data for training the first large language model in processing the first task, and the first test data can be data for testing the first trained large language model in processing the second task, and the first task and the second task are different tasks.

[0074] In the embodiments of the present specification, the first test data can be vectorized data generated by splicing the vectorized third sample data and the vectorized third structured knowledge prompt information. The third sample data can be data generated by vector conversion on the third prompt word information, and the third structured knowledge prompt information can be data generated by vector conversion on the third structured background knowledge related to the third prompt word information using the SKP framework. The explanation of the third sample data, the third structured knowledge prompt information, the third prompt word information, and the third structured background knowledge can correspond to the explanation of the first sample data, the first structured knowledge prompt information, the first prompt word information, and the first structured background knowledge, respectively, which will not be repeated here.

[0075] In the embodiments of the present specification, the first training data is different from the first test data, and the difference can include that the data granularity type of the first prompt word information is different from the data granularity type of the third prompt word information, and the task type corresponding to the first task is also different from the task type corresponding to the second task; or the data granularity type of the first prompt word information is the same as the data granularity type of the third prompt word information, and the task type corresponding to the first task is different from the task type corresponding to the second task; or the data granularity type of the first prompt word information is different from the data granularity type of the third prompt word information, and the task type corresponding to the first task is the same as the task type corresponding to the second task. The data granularity type can include any one of entity granularity, triple granularity, and subgraph granularity. The task type can include any one of binary classification (CLS), multiple choice (MC), and description generation (DESC).

[0076] In actual applications, the binary classification task can be a task of dividing input information into two categories. The multiple choice task can be a task of selecting a correct answer from multiple predefined options. The description generation task can be a task of generating a natural language text description according to input information.

[0077] In the embodiments of the present specification, after testing the first trained large language model based on the first test data, the first performance indicator of the first trained large language model in processing the second task can be determined. If the first performance indicator is greater than or equal to the first threshold, it is determined that the SKP framework has the generalization ability of transferability. If the first performance indicator is less than the first threshold, it is determined that the SKP framework does not have the generalization ability of transferability. The transferability is used to reflect the characteristic of the structured knowledge prompt framework in processing new tasks.

[0078] In the embodiments of the present specification, the task can include nine categories, respectively, a binary classification task based on entity granularity type data construction (Entity CLS), a multiple choice task based on entity granularity type data construction (Entity MC), a description generation task based on entity granularity type data construction (Entity DESC), a binary classification task based on triple granularity type data construction (Triple CLS), a multiple choice task based on triple granularity type data construction (Triple MC), a description generation task based on triple granularity type data construction (Triple DESC), a binary classification task based on subgraph granularity type data construction (Subgraph CLS), a multiple choice task based on subgraph granularity type data construction (Subgraph MC), and a description generation task based on subgraph granularity type data construction (Subgraph DESC).

[0079] In order to facilitate those skilled in the art to understand the present scheme, the present specification illustrates whether the SKP framework has the characteristics of processing new tasks by a specific example. It is assumed that a large language model is trained using the training data corresponding to the binary classification task based on entity granularity type data construction to obtain a trained large language model, and the trained large language model is tested using test data corresponding to other tasks. If the performance index of the trained large language model is greater than a first threshold value, it indicates that the SKP framework has the characteristics of processing new tasks.

[0080] In the embodiments of the present specification, in order to improve the evaluation diversity of the method for enhancing the task processing capability of the large language model by the SKP framework, it can be evaluated whether the framework has transferability, so as to more comprehensively evaluate its applicability and performance in different scenarios, and then based on the evaluation result, the user can reasonably adjust the parameters of the SKP framework, so that the SKP framework can more effectively enhance the capability of the large language model, thereby improving the performance of the large language model.

[0081] In the embodiments of the present specification, a specific embodiment is proposed for the evaluation method for reflecting whether the SKP framework has the scalability of the enhancement effect of the large language model with different computing capabilities for the same type.

[0082] Optionally, the method can further include training a second large language model using the first training data to obtain a third trained large language model, the first large language model and the second large language model being the same type of model, and the computing capability of the second large language model being greater than that of the first large language model; and testing the third trained large language model using the first test data to determine a third performance index of the third trained large language model.

[0083] Optionally, the evaluating the generalization capability of the structured knowledge prompt framework in the target direction according to the first performance indicator can specifically include: if the third performance indicator is greater than the first performance indicator, determining that the structured knowledge prompt framework has the generalization capability of scalability; the scalability is used to reflect that the enhancement effect of the inference capability of the structured knowledge prompt framework on the large language model increases with the increase of the computing capability of the large language model.

[0084] In the embodiments of the present specification, the first large language model and the second large language model are of the same type, for example, can be any one of the GPT series model, the BERT system model, the T5 series model, and the DeepSeek series model. The computing capability of the second large language model is greater than the computing capability of the first large language model. The computing capability of the large language model can refer to the computing resource requirement of the large language model in the training and inference process, such as hardware, storage requirement, delay, throughput, and energy efficiency.

[0085] In the embodiments of the present specification, the first training data used to train the second large language model and the first training data used to train the first large language model can be the same data. The first test data used to test the third trained large language model and the first test data used to test the first trained large language model can be the same data.

[0086] In the embodiments of the present specification, the training of the second large language model using the first training data can refer to the training of the first large language model using the first training data. The testing of the third trained large language model using the first test data can refer to the testing of the first trained large language model using the first test data. Both of which will not be repeated here.

[0087] In the embodiments of the present specification, the explanation of the third performance indicator can refer to the explanation of the first performance indicator, which will not be repeated here. If the third performance indicator is greater than the first performance indicator, it is determined that the SKP framework has the generalization capability of scalability; if the third performance indicator is less than the first performance indicator, it is determined that the SKP framework does not have the generalization capability of scalability. The scalability is used to reflect that the enhancement effect of the inference capability of the SKP framework on the large language model increases with the increase of the computing capability of the large language model, that is, for the same SKP framework, the stronger the computing capability of the large language model, the stronger the enhancement capability of the SKP framework on the large language model.

[0088] In the embodiments of the present specification, another specific embodiment is proposed for the evaluation method for reflecting whether the enhancement effect of the SKP framework on different computing capability large language models of the same type has the generalization capability of scalability.

[0089] Optionally, the first structured knowledge prompt information is generated based on a first structured knowledge prompt framework; the method can further include training the first large language model by using third training data containing the first sample data and fourth structured knowledge prompt information to obtain a fourth trained large language model; the fourth structured knowledge prompt information is generated based on a second structured knowledge prompt framework, and the second structured knowledge prompt framework has a greater computing capability than the first structured knowledge prompt framework; and the fourth trained large language model is tested by using the first test data to determine a fourth performance index of the fourth trained large language model.

[0090] Optionally, the generalization capability of the structured knowledge prompt framework in the target direction is evaluated according to the first performance index, and specifically can include: if the fourth performance index is greater than the first performance index, it is determined that the structured knowledge prompt framework has a generalization capability of scalability; the scalability is used to reflect that the enhancement effect of the reasoning capability of the structured knowledge prompt framework on the large language model increases with the increase of the computing capability of the structured knowledge prompt framework.

[0091] In the embodiments of the present specification, the first SKP framework and the second SKP framework can belong to the same type of framework system, and the computing capability of the second SKP framework is better than that of the first SKP framework. The computing capability of the SKP framework refers to the computing resources required in the process of generating structured knowledge prompt information, including hardware, storage requirements, delay, throughput and energy efficiency and other aspects of resource consumption.

[0092] In the embodiments of the present specification, the fourth structured knowledge prompt information can be information generated by vector conversion of the first structured background knowledge related to the first prompt word information by using the second SKP framework.

[0093] In the embodiments of the present specification, the first sample data in the third training data and the first sample data in the first training data can be the same data, and the first structured background knowledge used to generate the fourth structured knowledge prompt information and the first structured background knowledge used to generate the first structured knowledge prompt information can be the same data.

[0094] In the embodiments of the present specification, the first test data can be vectorized data generated by splicing vectorized sample data and vectorized structured knowledge prompt information. The sample data in the first test data for testing the fourth trained large language model can be the same data as the sample data in the first test data for testing the first trained large language model. The structured knowledge prompt information in the first test data for testing the fourth trained large language model can be data generated by vector conversion of the structured background knowledge A using the second SKP framework, and the structured knowledge prompt information in the first test data for testing the first trained large language model can be data generated by vector conversion of the structured background knowledge A using the first SKP framework.

[0095] In the embodiments of the present specification, the manner of training the first large language model using the third training data can refer to the manner of training the first large language model using the first training data. The manner of testing the fourth trained large language model using the first test data can refer to the manner of testing the first trained large language model using the first test data. Both of which will not be repeated here.

[0096] In the embodiments of the present specification, the explanation for the fourth performance indicator can refer to the explanation for the first performance indicator, which will not be repeated here. If the fourth performance indicator is greater than the first performance indicator, it is determined that the SKP framework has the generalization ability of scalability; if the fourth performance indicator is less than the first performance indicator, it is determined that the SKP framework does not have the generalization ability of scalability. Wherein, the scalability is used to reflect that the enhancement effect of the reasoning ability of the SKP framework on the large language model increases with the increase of the computing ability of the SKP framework, that is, for the same large language model, the stronger the computing ability of the SKP framework, the stronger the enhancement ability of the SKP framework on the large language model.

[0097] In the embodiments of the present specification, in order to improve the evaluation diversity of the method for enhancing the task processing ability of the large language model by the SKP framework, it can be evaluated whether the framework has scalability, so as to more comprehensively evaluate its applicability and performance in different scenarios, and then based on the evaluation result, the user can reasonably adjust the parameters of the SKP framework, so that the SKP framework can more effectively enhance the ability of the large language model, thereby improving the performance of the large language model.

[0098] In the embodiments of the present specification, a specific embodiment is proposed for the evaluation manner for reflecting whether the generalization ability of the SKP framework for enhancing different types of large language models has universality.

[0099] Optionally, the method can further include: training a third large language model by using the first training data to obtain a fifth trained large language model, the third large language model being different from the first large language model in type; and testing the fifth trained large language model by using the first test data to determine a fifth performance index of the fifth trained large language model.

[0100] Optionally, the evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index can specifically include: if the first performance index is greater than or equal to a first threshold value and the fifth performance index is greater than or equal to a third threshold value, it is determined that the structured knowledge prompt framework has a generalization ability of universality; the universality is used to reflect that the structured knowledge prompt framework has an effect of enhancing the reasoning ability of the first large language model and the reasoning ability of the third large language model.

[0101] In the embodiments of the present specification, the first large language model and the third large language model are different in type, for example, the first large language model is a GPT series model, and the third large language model can be any one of a BERT system model, a T5 series model, and a DeepSeek series model other than the GPT series model.

[0102] In the embodiments of the present specification, the first training data used for training the third large language model can be the same data as the first training data used for training the first large language model, and the first test data used for testing the fifth trained large language model can be the same data as the first test data used for testing the first trained large language model.

[0103] In the embodiments of the present specification, the manner of training the third large language model by using the first training data can refer to the manner of training the first large language model by using the first training data, and the manner of testing the fifth trained large language model by using the first test data can refer to the manner of testing the first trained large language model by using the first test data. Details are not repeated here.

[0104] In the embodiments of the present specification, the explanation of the fifth performance index can refer to the explanation of the first performance index, and details are not repeated here. The manner of determining the third threshold value can refer to the manner of determining the first threshold value, and the third threshold value can be the same as the first threshold value, or the third threshold value can be different from the first threshold value, which is not limited.

[0105] In the embodiments of the present specification, if the first performance index is greater than or equal to the first threshold value, and the fifth performance index is greater than or equal to the third threshold value, it is determined that the SKP framework has the generalization ability of universality; if the first performance index is less than the first threshold value, and / or the fifth performance index is less than the third threshold value, it is determined that the SKP framework does not have the generalization ability of universality. Wherein, the universality is used to reflect that the SKP framework has the effect of enhancing the reasoning ability of the first large language model, and also has the effect of enhancing the reasoning ability of the third large language model.

[0106] In the embodiments of the present specification, in order to improve the evaluation diversity of the method for enhancing the task processing ability of the large language model by the SKP framework, it can be evaluated whether the framework has universality, so as to more comprehensively evaluate its applicability and performance in different scenarios, and then based on the evaluation result, the user can reasonably adjust the parameters of the SKP framework, so that the SKP framework can more effectively enhance the ability of the large language model, thereby improving the performance of the large language model.

[0107] In order to improve the efficiency of obtaining task instances, the task instances can be generated based on the instruction prompt word template.

[0108] Optionally, the vector conversion is performed on the first prompt word information to generate vectorized first sample data, which can specifically include: determining an instruction prompt word template corresponding to a target task processed by the first large language model; inputting the first prompt word information into the instruction prompt word template to obtain a task instance corresponding to the target task; and performing vector conversion on the task instance to generate vectorized first sample data.

[0109] In the embodiments of the present specification, the types of target tasks can refer to the 9 types of tasks in the above embodiments, and for each type of target task, an instruction prompt word template can be set in advance. By inputting the prompt word information into the instruction prompt word template corresponding to the target task, a task instance corresponding to the target task can be obtained.

[0110] In actual application, for a binary classification task (CLS) scenario, the constructed task instance can include a certain number of sample positive examples and a certain number of sample negative examples. The number of sample positive examples and sample negative examples can be equal or unequal, which is not limited. If the prompt word information is entity granularity (EG) data, the sample positive example can be a task instance with consistent entity ID and entity name, and the sample negative example can be a task instance with inconsistent entity ID and entity name. If the prompt word information is triple granularity (TG) or subgraph granularity (SG) data, the sample positive example can be data split from a known knowledge graph, and the sample negative example can be data obtained by randomly perturbing the sample positive example.

[0111] For the multiple choice (MC) scenario, if the prompt information is entity granularity (EG) data, the constructed task instance can be to give an entity ID, and then select the entity corresponding name from multiple name options; if the prompt information is triple granularity (TG) data, the constructed task instance can be to give a triple data missing some entity, and select the missing entity in the triple data from multiple entity options; if the prompt information is subgraph granularity (SG) data, the constructed task instance can be to give a subgraph data missing some entity, and select the missing entity in the subgraph data from multiple entity options.

[0112] For the description generation task (DESC) scenario, if the prompt information is entity granularity (EG) data, the constructed task instance can be to generate a description for the information in the entity; if the prompt information is triple granularity (TG) data, the constructed task instance can be to generate a description for the information in the triple; if the prompt information is subgraph granularity (SG) data, the constructed task instance can be to generate a description for the information in the subgraph.

[0113] In the embodiments of the present specification, after determining the task instance corresponding to the target task, the task instance can be converted into vectorized data recognizable by the large language model, to obtain first sample data for training the large language model.

[0114] In order to improve the objectivity of evaluating the structured knowledge prompting framework, the conditional information in the task instance for assisting the large language model to generate reply information can be deleted, and only the conditional information is retained in the structured knowledge prompt information.

[0115] Optionally, the vector conversion for the task instance generates vectorized first sample data, which can specifically include: obtaining user question information in the task instance; deleting conditional information in the user question information for limiting model reply information to obtain an updated task instance; performing vector conversion on the updated task instance to generate vectorized first sample data.

[0116] Optionally, the first large language model is trained based on the vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information, which can specifically include: splicing the first sample data and the first structured knowledge prompt information to obtain the first training data, the first structured knowledge prompt information containing structured knowledge for the conditional information; training the first large language model based on the first training data.

[0117] In an embodiment of the present specification, the question information and the corresponding answer information can be included in each task instance, and the question information can include condition information used to assist the large language model in replying. For example, if the question information is "What medicine should a diabetic person take to control blood sugar?", the condition information can be "diabetic person".

[0118] In an embodiment of the present specification, the condition information in the question information included in the task instance is deleted to obtain an updated task instance. The updated task instance is converted into vectorized data recognizable by the large language model to obtain first sample data used to train the large language model.

[0119] In an embodiment of the present specification, the first sample data and the first structured knowledge prompt information are spliced into first training data used to train the large language model, wherein the first sample data does not include condition information used to limit the model reply information, and only the first structured knowledge prompt information includes the condition information used to limit the model reply information, so as to verify whether the large language model can accurately complete the reply information to the question information based on the condition information included in the first structured knowledge prompt information, so as to improve the accuracy of the evaluation of the SKP framework.

[0120] In order to improve the accuracy of the evaluation of the SKP framework, the data required for evaluating the SKP framework can be constructed based on known knowledge graph data.

[0121] Optionally, before the vector conversion of the first prompt word information to generate the vectorized first sample data, the method can further include: constructing the first prompt word information based on the known knowledge graph data; and the first prompt word information includes at least one of entity granularity prompt word data, triple granularity prompt word data, and subgraph granularity prompt word data.

[0122] In an embodiment of the present specification, the known knowledge graph data can be knowledge graph data whose correctness of data is greater than a preset threshold, or the known knowledge graph data can also be knowledge graph data whose usage frequency satisfies a preset frequency.

[0123] In an embodiment of the present specification, the first preset number of entity granularity prompt word data, the second preset number of triple granularity prompt word data, and the third preset number of subgraph granularity prompt word data can be constructed from the known knowledge graph data. Since the first prompt word information is constructed based on the known knowledge graph data, the accuracy of the first prompt word information can be improved, and the accuracy of the evaluation of the structured knowledge prompt framework can be improved.

[0124] In an embodiment of the present specification, a specific embodiment is also proposed for the specific manner of constructing the first prompt word information.

[0125] Optionally, if the first prompt word information includes entity granularity prompt word data, the constructing of the first prompt word information based on the known knowledge graph data can specifically include: randomly extracting a first preset number of nodes from the known knowledge graph data; determining the entities corresponding to the nodes as the entity granularity prompt word data.

[0126] In the embodiments of the present specification, the first preset number can be set according to actual needs, for example, the first preset number can be 20,000, which is not limited.

[0127] In actual application, the entity granularity prompt word data can include description information of the corresponding entity, such as ID information of the entity, name information of the entity, etc.

[0128] Optionally, if the first prompt word information includes triple granularity prompt word data, the constructing of the first prompt word information based on the known knowledge graph data can specifically include: splitting a second preset number of triple data from the known knowledge graph data to obtain triple granularity prompt word data.

[0129] In the embodiments of the present specification, the way of splitting triple data from the known knowledge graph data can be split by a random splitting manner.

[0130] In the embodiments of the present specification, the second preset number can be set according to actual needs, for example, the second preset number can be 10,000, which is not limited.

[0131] In actual application, the triple granularity prompt word data can include description information of the corresponding triple, such as information of entities in the triple, relationship information between entities in the triple, etc.

[0132] Optionally, if the first prompt word information includes subgraph granularity prompt word data, the constructing of the first prompt word information based on the known knowledge graph data can specifically include: selecting a third preset number of target nodes from the first preset number of nodes; for each target node, sampling graph data in a preset domain containing the target node from the known knowledge graph data to construct a subgraph, and obtaining subgraph granularity prompt word data.

[0133] In the embodiments of the present specification, the third preset number of target nodes are randomly selected from the first preset number of nodes selected above, wherein the third preset number can be set according to actual needs, for example, the third preset number can be 5,000.

[0134] In the embodiments of the present specification, based on the determined target node, the graph data in the preset domain is sampled with the target node as the reference to construct subgraph data. Specifically, the graph data in the N-hop neighborhood of the target node can be sampled to construct subgraph data.

[0135] In practical applications, the prompt word data of the subgraph granularity can include description information of the corresponding subgraph, such as information of entities in the subgraph, relationship information between entities in the subgraph, and information of the triple data contained in the subgraph, etc.

[0136] In order to facilitate those skilled in the art to understand the present scheme, the present specification further proposes a specific example 1 for the way of evaluating the SKP framework.

[0137] Example 1, assuming that the generalization ability evaluated for the SKP framework includes: generalization ability 1, generalization ability 2, generalization ability 3, and generalization ability 4.

[0138] Generalization ability 1: data granularity (Granularity), which structured knowledge of which granularity in the knowledge graph can the SKP framework integrate into the large language model.

[0139] Generalization ability 2: transferability (Transferability), can the SKP framework be transferred between different tasks? Can the SKP handle new elements that have never been seen before? Generalization ability 3: scalability (Scalability), does the SKP framework show an expansion rule? Generalization ability 4: universality (Universality), can the SKP framework be applied to different large language models? In order to evaluate whether the SKP framework has the above 4 types of generalization ability, 9 types of task instances are constructed in the present specification, and each type of task instance is divided into training set instances, validation set instances and test set instances, as shown in Table 1 below.

[0140] Table 1

[0141] For each type of task in Table 1, the performance indicators of the large language model on the task can be determined based on the training set, the validation set, and the test set. In actual applications, the steps of determining the performance indicators of the large language model on the target task can include: first, training the large language model using the training data, and in the training process, a plurality of trained large language models can be determined based on different training times. Then, the validation data is used to validate each trained large language model, and a trained large language model that meets the preset requirements is selected from the trained large language models to obtain a plurality of validated large language models. Each validated large language model is then tested using the test data to test the performance indicators of each validated large language model, and finally, the performance indicators of the large language model on the target task are determined based on the performance indicators. Alternatively, a target validated large language model is selected from the validated large language models, and the performance indicators of the target validated large language model are tested using the test data to test the performance indicators of the target validated large language model, and the performance indicators of the target validated large language model are determined as the performance indicators of the large language model on the target task. The target validated large language model can be the validated large language model corresponding to the first validated result in the order from best to worst. It should be noted that the target ratio between the first number of training data, the second number of validation data, and the third number of test data can be set according to actual needs, such as 8:1:1, which is not limited in the present specification.

[0142] Based on the performance indicators of the large language model on various tasks, the generalization ability of the SKP framework can be evaluated.

[0143] For example: based on Entity CLS, Entity MC, and Entity DESC, it can be evaluated whether the SKP framework can integrate the structured knowledge of entity granularity in the knowledge graph into the large language model; based on Triple CLS, Triple MC, and Triple DESC, it can be evaluated whether the SKP framework can integrate the structured knowledge of triple granularity in the knowledge graph into the large language model; based on Subgraph CLS, Subgraph MC, and Subgraph DESC, it can be evaluated whether the SKP framework can integrate the structured knowledge of subgraph granularity in the knowledge graph into the large language model.

[0144] For example: based on any two types of tasks in Table 1, such as Entity CLS and Triple CLS, the large language model is trained using the training set in Entity CLS, and the performance indicators of the trained large language model are tested using the test set in Triple CLS, which can evaluate whether the SKP framework can be transferred between different tasks.

[0145] For example, based on any type of task in Table 1, such as Entity MC, the performance metrics of the first large language model can be determined using Entity MC, and then the performance metrics of the second large language model can be determined using Entity MC. The first and second large language models are of the same type, and the computational power of the first large language model is stronger than that of the second large language model. This can be used to evaluate whether the SKP model has scalability.

[0146] For example, based on any type of task in Table 1, such as Triple CLS, we can use Triple CLS to determine the performance metrics of the first language model, and then use Triple CLS to determine the performance metrics of the second language model. The first and second language models are different types of models, which can be used to evaluate whether the SKP framework has universality.

[0147] Based on the same idea, embodiments of this specification also provide apparatus corresponding to the above methods. Figure 3 The embodiments provided in this specification correspond to Figure 2 A schematic diagram of a device for evaluating a structured knowledge prompting framework. (Example) Figure 3 As shown, the device may include: The generation module 302 is used to perform vector transformation on the first prompt word information to generate vectorized first sample data; The conversion module 304 is used to convert the first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompt information using the structured knowledge prompt framework; Training module 306 is used to train the first large language model based on the vectorized first training data generated by concatenating the first sample data and the first structured knowledge prompt information, so as to obtain the first trained large language model. The determination module 308 is used to test the first trained large language model using the first test data and determine the first performance index of the first trained large language model. Evaluation module 310 is used to evaluate the generalization ability of the structured knowledge hinting framework in the target direction based on the first performance index.

[0148] based on Figure 3 The present specification provides some specific implementation methods for the apparatus, which are described below.

[0149] Optionally, the data granularity type of the first prompt word information includes any one of entity granularity, triplet granularity, and subgraph granularity; the data granularity type of the first structured background knowledge is the same as the data granularity type of the first prompt word information.

[0150] The evaluation module 310 can specifically include: The first determination unit is configured to determine that the structured knowledge prompt framework has generalization capability of enhancing inference capability of the first large language model in the first granularity dimension, if the first performance index is greater than or equal to a first threshold.

[0151] Optionally, the apparatus can further include a second training module configured to train the first large language model based on vectorized second training data generated by splicing second sample data and second structured knowledge prompt information, to obtain a second trained large language model, wherein the second sample data is vectorized data generated by vector conversion on second prompt information, the second structured knowledge prompt information is vectorized information generated by vector conversion on second structured background knowledge related to the second prompt information by using the structured knowledge prompt framework, the data granularity type of the second prompt information is different from the data granularity type of the first prompt information, and the data granularity type of the second structured background knowledge is the same as the data granularity type of the second prompt information.

[0152] Optionally, the evaluation module 310 can specifically include: The second determination unit is configured to determine that the structured knowledge prompt framework has generalization capability of enhancing inference capability of the first large language model in the first granularity dimension and the second granularity dimension, if the first performance index is greater than or equal to the first threshold, and the second performance index is greater than or equal to a second threshold, wherein the second granularity is the data granularity type of the second prompt information.

[0153] Optionally, the training module 306 can specifically include a first training unit configured to train the first large language model by using first training data for instructing the large language model to perform a first task.

[0154] Optionally, the determination module 308 can specifically include: The test unit is configured to test the first trained large language model by using first test data for instructing the large language model to perform a second task; the first test data comprises third sample data and third structured knowledge prompt information; the third sample data corresponds to third prompt word information of a data granularity type different from a data granularity type of the first prompt word information, and / or a task type corresponding to the first task is different from a task type corresponding to the second task; the data granularity type comprises any one of entity granularity, triple granularity, and subgraph granularity, and the task type comprises any one of a binary classification task, a multiple-choice task, and a description generation task.

[0155] Optionally, the evaluation module 310 can specifically include a third determination unit configured to determine that the structured knowledge prompt framework has a generalization capability of migratability if the first performance indicator is greater than or equal to a first threshold; the migratability is used to reflect that the structured knowledge prompt framework has a characteristic of processing a new task.

[0156] Optionally, the apparatus can further include a third training module configured to train a second large language model by using the first training data to obtain a third trained large language model; the first large language model and the second large language model are models of the same type, and a computing capability of the second large language model is greater than a computing capability of the first large language model. The third determination module is configured to test the third trained large language model by using the first test data to determine a third performance indicator of the third trained large language model.

[0157] Optionally, the evaluation module 310 can specifically include a fourth determination unit configured to determine that the structured knowledge prompt framework has a generalization capability of scalability if the third performance indicator is greater than the first performance indicator; the scalability is used to reflect that an enhancement effect of the structured knowledge prompt framework on reasoning capability of a large language model increases as a computing capability of the large language model increases.

[0158] Optionally, the first structured knowledge prompt information is generated based on a first structured knowledge prompt framework; the apparatus can further include a fourth training module configured to train the first large language model by using third training data containing the first sample data and fourth structured knowledge prompt information to obtain a fourth trained large language model; the fourth structured knowledge prompt information is generated based on a second structured knowledge prompt framework, and a computing capability of the second structured knowledge prompt framework is greater than a computing capability of the first structured knowledge prompt framework. The fourth determination module is configured to test the fourth trained large language model by using the first test data to determine a fourth performance indicator of the fourth trained large language model.

[0159] Optionally, the evaluation module 310 can specifically include a fifth determination unit configured to determine that the structured knowledge prompt framework has a generalization capability of scalability if the fourth performance index is greater than the first performance index, wherein the scalability is used to reflect that an enhancement effect of inference capability of the structured knowledge prompt framework on the large language model increases with an increase of computing capability of the structured knowledge prompt framework.

[0160] Optionally, the device can further include a fifth training module configured to train a third large language model using the first training data to obtain a fifth trained large language model, wherein the third large language model is different from the first large language model in type. A fifth determination module is configured to test the fifth trained large language model using the first test data to determine a fifth performance index of the fifth trained large language model.

[0161] Optionally, the evaluation module 310 can specifically include a sixth determination unit configured to determine that the structured knowledge prompt framework has a generalization capability of universality if the first performance index is greater than or equal to a first threshold value and the fifth performance index is greater than or equal to a third threshold value, wherein the universality is used to reflect that the structured knowledge prompt framework has an effect of enhancing inference capability of the first large language model and inference capability of the third large language model.

[0162] Optionally, the generation module 302 can specifically include a seventh determination unit configured to determine an instruction prompt word template corresponding to a target task processed by the first large language model. An input unit is configured to input the first prompt word information into the instruction prompt word template to obtain a task instance corresponding to the target task. A generation unit is configured to perform vector conversion on the task instance to generate vectorized first sample data.

[0163] Optionally, the generation unit can specifically include an acquisition subunit configured to acquire user question information in the task instance. A deletion subunit is configured to delete condition information for limiting model reply information in the user question information to obtain an updated task instance. A generation subunit is configured to perform vector conversion on the updated task instance to generate vectorized first sample data.

[0164] Optionally, the training module 306 can specifically include a splicing unit configured to splice the first sample data and the first structured knowledge prompt information to obtain the first training data, wherein the first structured knowledge prompt information contains structured knowledge for the condition information. A second training unit is configured to train the first large language model based on the first training data.

[0165] Optionally, the apparatus can further include a constructing module configured to construct first prompt word information based on known knowledge graph data, wherein the first prompt word information includes at least one of entity-granularity prompt word data, triple-granularity prompt word data, and subgraph-granularity prompt word data.

[0166] Optionally, if the first prompt word information includes entity-granularity prompt word data, the constructing module can specifically include an extracting unit configured to randomly extract a first preset number of nodes from the known knowledge graph data, and an eighth determining unit configured to determine an entity corresponding to the nodes as the entity-granularity prompt word data. Optionally, if the first prompt word information includes triple-granularity prompt word data, the constructing module can specifically include a splitting unit configured to split a second preset number of triple data from the known knowledge graph data to obtain the triple-granularity prompt word data.

[0167] Optionally, if the first prompt word information includes subgraph-granularity prompt word data, the constructing module can specifically include a selecting unit configured to select a third preset number of target nodes from the first preset number of nodes, and a sampling unit configured to sample graph data within a preset domain containing the target nodes from the known knowledge graph data to construct a subgraph for each target node, and obtain the subgraph-granularity prompt word data.

[0168] Based on the same idea, the present specification also provides a device corresponding to the above method.

[0169] Figure 4 is a structural schematic diagram of an apparatus for evaluating a structured knowledge prompting framework provided by an embodiment of the present specification. As shown in Figure 4 The apparatus 400 can include at least one processor 410 and a memory 430 in communication with the at least one processor. Wherein the at least one processor is configured to execute the method of the present specification. Figure 2As shown in the method, the memory 430 stores instructions 420 executable by the at least one processor 410, and the instructions are executed by the at least one processor 410 to enable the at least one processor 410 to: perform vector conversion on first prompt word information to generate vectorized first sample data; convert first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompt information by using a structured knowledge prompt framework; train the first large language model based on vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information to obtain a first trained large language model; test the first trained large language model by using first test data to determine a first performance indicator of the first trained large language model; and evaluate the generalization ability of the structured knowledge prompt framework in a target direction according to the first performance indicator.

[0170] Based on the same idea, the present specification also provides a computer readable medium corresponding to the above method. The computer readable medium stores computer readable instructions executable by a processor to implement the above method of evaluating a structured knowledge prompt framework.

[0171] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device shown, Figure 4 As the device shown is basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0172] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards in relevant regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0173] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to a software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is only necessary to logically program a method flow using the above-mentioned hardware description languages and program it into an integrated circuit to easily obtain a hardware circuit that implements the logical method flow.

[0174] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is also possible to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to perform the same functions by logically programming the method steps. Such a controller can therefore be considered as a hardware component, and the means included therein for performing various functions can also be considered as structures within the hardware component. Alternatively, the means for performing various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0175] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0176] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware in the implementation of the present application.

[0177] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0179] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0180] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0181] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0182] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. A

[0183] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0184] It should also be noted that the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0185] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system or computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The present application can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including storage devices.

[0187] The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A method for evaluating a structured knowledge prompting framework for converting structured knowledge into vector information for inputting into a large language model, the method comprising: vector converting first prompt information to generate vectorized first sample data; converting first structured background knowledge related to the first prompt information into vectorized first structured knowledge prompting information using the structured knowledge prompting framework; training the first large language model based on vectorized first training data generated by concatenating the first sample data and the first structured knowledge prompting information to obtain a first trained large language model; testing the first trained large language model using first test data to determine a first performance indicator of the first trained large language model; evaluating the generalization capability of the structured knowledge prompting framework in a target direction according to the first performance indicator.

2. The method of claim 1, wherein the data granularity type of the first prompt information comprises any one of entity granularity, triple granularity, and subgraph granularity; and the data granularity type of the first structured background knowledge is the same as the data granularity type of the first prompt information. The evaluation of the generalization capability of the structured knowledge prompting framework in the target direction according to the first performance indicator specifically comprises: if the first performance indicator is greater than or equal to a first threshold value, determining that the structured knowledge prompting framework has a generalization capability of enhancing the inference capability of the first large language model in a first granularity dimension; the first granularity is the data granularity type of the first prompt information.

3. The method of claim 2, further comprising: training the first large language model based on vectorized second training data generated by concatenating second sample data and second structured knowledge prompting information to obtain a second trained large language model, wherein the second sample data is vectorized data generated by vector converting second prompt information, and the second structured knowledge prompting information is vectorized information generated by vector converting second structured background knowledge related to the second prompt information using the structured knowledge prompting framework; the data granularity type of the second prompt information is different from the data granularity type of the first prompt information, and the data granularity type of the second structured background knowledge is the same as the data granularity type of the second prompt information; testing the second trained large language model using second test data to determine a second performance indicator of the second trained large language model; The evaluation of the generalization capability of the structured knowledge prompting framework in the target direction according to the first performance indicator specifically comprises: if the first performance indicator is greater than or equal to the first threshold value, and the second performance indicator is greater than or equal to a second threshold value, determining that the structured knowledge prompting framework has a generalization capability of enhancing the inference capability of the first large language model in both the first granularity dimension and a second granularity dimension; the second granularity is the data granularity type of the second prompt information.

4. The method of claim 1, wherein the training the first large language model based on the vectorized first training data generated by concatenating the first sample data and the first structured knowledge prompt information comprises: training the first large language model using first training data for instructing the large language model to perform a first task; and wherein the testing the first trained large language model using the first test data comprises: testing the first trained large language model using first test data for instructing the large language model to perform a second task, wherein the first test data comprises third sample data and third structured knowledge prompt information, wherein a data granularity type of third prompt information corresponding to the third sample data is different from a data granularity type of the first prompt information, and / or wherein a task type corresponding to the first task is different from a task type corresponding to the second task, wherein the data granularity type comprises any one of an entity granularity, a triple granularity, and a subgraph granularity, and wherein the task type comprises any one of a binary classification task, a multiple-choice task, and a description generation task; and wherein the evaluating the generalization capability of the structured knowledge prompt framework in the target direction according to the first performance indicator comprises: determining that the structured knowledge prompt framework has a generalization capability of migratability if the first performance indicator is greater than or equal to a first threshold value, wherein the migratability reflects that the structured knowledge prompt framework has a characteristic of processing a new task.

5. The method of claim 1, further comprising: training a second large language model using the first training data to obtain a third trained large language model, wherein the first large language model and the second large language model are the same type of model, and wherein a computing capability of the second large language model is greater than a computing capability of the first large language model; testing the third trained large language model using the first test data to determine a third performance indicator of the third trained large language model; and wherein the evaluating the generalization capability of the structured knowledge prompt framework in the target direction according to the first performance indicator comprises: determining that the structured knowledge prompt framework has a generalization capability of scalability if the third performance indicator is greater than the first performance indicator, wherein the scalability reflects that an enhancement effect of the structured knowledge prompt framework on inference capability of the large language model increases as the computing capability of the large language model increases.

6. The method of claim 1, wherein the first structured knowledge prompt information is generated based on a first structured knowledge prompt framework, and wherein the method further comprises: training the first large language model using third training data comprising the first sample data and fourth structured knowledge prompt information to obtain a fourth trained large language model; and wherein the fourth structured knowledge prompt information is generated based on a second structured knowledge prompt framework, and wherein a computing capability of the second structured knowledge prompt framework is greater than a computing capability of the first structured knowledge prompt framework. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ test the fourth trained large language model by using the first test data to determine a fourth performance index of the fourth trained large language model; the evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index specifically comprises: if the fourth performance index is greater than the first performance index, it is determined that the structured knowledge prompt framework has the generalization ability of scalability; the scalability is used to reflect that the enhancement effect of the reasoning ability of the structured knowledge prompt framework on the large language model increases with the increase of the computing capacity of the structured knowledge prompt framework.

7. The method of claim 1, further comprising: training a third large language model by using the first training data to obtain a fifth trained large language model, the third large language model being different from the first large language model in type; testing the fifth trained large language model by using the first test data to determine a fifth performance index of the fifth trained large language model; the evaluating the generalization ability of the structured knowledge prompt framework in the target direction according to the first performance index specifically comprises: if the first performance index is greater than or equal to a first threshold value and the fifth performance index is greater than or equal to a third threshold value, it is determined that the structured knowledge prompt framework has the generalization ability of universality; the universality is used to reflect that the structured knowledge prompt framework has the effect of enhancing the reasoning ability of the first large language model and the reasoning ability of the third large language model.

8. The method of claim 1, wherein the vector conversion on the first prompt word information to generate the vectorized first sample data specifically comprises: determining an instruction prompt word template corresponding to a target task processed by the first large language model; inputting the first prompt word information into the instruction prompt word template to obtain a task instance corresponding to the target task; performing vector conversion on the task instance to generate the vectorized first sample data.

9. The method of claim 8, wherein the vector conversion on the task instance to generate the vectorized first sample data specifically comprises: obtaining user question information in the task instance; deleting condition information for limiting model reply information in the user question information to obtain an updated task instance; performing vector conversion on the updated task instance to generate the vectorized first sample data; the training the first large language model based on the vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information specifically comprises: splicing the first sample data and the first structured knowledge prompt information to obtain the first training data, the first structured knowledge prompt information containing structured knowledge for the condition information; training the first large language model based on the first training data.

10. The method of claim 1, further comprising, before the vector conversion on the first prompt word information to generate the vectorized first sample data: construct first prompt word information based on known knowledge graph data; The first prompt word information includes at least one of entity granularity prompt word data, triple granularity prompt word data, and subgraph granularity prompt word data.

11. The method of claim 10, if the first prompt word information includes entity granularity prompt word data, the constructing first prompt word information based on known knowledge graph data specifically comprises: randomly extracting a first preset number of nodes from the known knowledge graph data; determining the entities corresponding to the nodes as the entity granularity prompt word data; or, if the first prompt word information includes triple granularity prompt word data, the constructing first prompt word information based on known knowledge graph data specifically comprises: splitting a second preset number of triple data from the known knowledge graph data to obtain triple granularity prompt word data; or, if the first prompt word information includes subgraph granularity prompt word data, the constructing first prompt word information based on known knowledge graph data specifically comprises: selecting a third preset number of target nodes from the first preset number of nodes; for each target node, sampling graph data within a preset domain containing the target node from the known knowledge graph data to construct a subgraph, and obtaining subgraph granularity prompt word data.

12. An apparatus for evaluating a structured knowledge prompt framework for converting structured knowledge into vector information for inputting into a large language model, the apparatus comprising: a generation module configured to perform vector conversion on first prompt word information to generate vectorized first sample data; a conversion module configured to convert first structured background knowledge related to the first prompt word information into vectorized first structured knowledge prompt information using the structured knowledge prompt framework; a training module configured to train the first large language model based on vectorized first training data generated by splicing the first sample data and the first structured knowledge prompt information, to obtain a first trained large language model; a determination module configured to test the first trained large language model using first test data and determine a first performance indicator of the first trained large language model; an evaluation module configured to evaluate the generalization ability of the structured knowledge prompt framework in a target direction according to the first performance indicator.

13. A computing device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for evaluating a structured knowledge prompt framework according to any one of claims 1 to 11.

14. A computer readable medium having stored thereon computer readable instructions executable by a processor to implement the method for evaluating a structured knowledge prompt framework according to any one of claims 1 to 11.

Images