Semantic relationship prediction method based on multi-section prompt optimization of large language model

Abstract

The invention relates to the field of computer application, in particular to a semantic relation prediction method based on large language model multi-section prompt optimization. The method comprises the steps of obtaining a data set, defining an initial prompt as a current prompt of a first round of iteration, starting iteration, and inputting the current prompt and a training data set into a large language model for prediction to obtain an error case; obtaining a section set according to the error case; obtaining a candidate prompt set by using a second error analysis method; taking the candidate prompt set as the current prompt to participate in the next round of iteration; respectively inputting prompts in each candidate prompt set and statements in the test data set into a large language model for prediction, obtaining prediction results, evaluating the prediction results obtained by each prompt, and selecting k prompts; and the selected prompt is combined with the to-be-predicted statement to be input into the large language model for prediction. According to the method, prompts can be automatically constructed from multiple sections to utilize the large language model, so that the labor cost is reduced, and meanwhile, the prediction effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer applications, and in particular to a semantic relationship prediction method based on multi-faceted prompt optimization of a large language model. Background Art

[0002] In recent years, large language models have emerged. Large language models are artificial intelligence models based on deep learning technology, especially the Transformer architecture. These models are trained with large amounts of data and are able to understand and generate natural language text. Large language models have a large number of parameters and a complex structure, which enables them to perform well in tasks such as language understanding, generation, and translation. Due to the excellent capabilities and ease of use of large language models, they have brought great changes to the computer field, especially the field of natural language processing.

[0003] Determining whether a sentence contains a causal relationship is an important task in natural language processing (NLP). The goal of this task is to automatically identify and classify causal relationships in text, such as determining whether the sentence "because X, so Y" expresses a causal relationship. This is widely used in text analysis, information extraction, and automatic question-answering systems. Traditional methods usually use neural network-based methods to solve this problem, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and Transformers to directly learn features and patterns from data.

[0004] With the rise of large language model technology, it has become possible to apply it to a variety of tasks, which has given rise to strategies for using such models to perform specific tasks. Among them, a simple and efficient method focuses on prompt engineering, that is, guiding large language models to generate expected outputs through carefully designed "prompt words" or "prompt sentences." This method significantly enhances the versatility and flexibility of the model, enabling it to perform a wide range of tasks without in-depth training on specific tasks.

[0005] However, the implementation of prompt engineering is also accompanied by challenges. The core is the need to manually conceive and construct accurate and inspiring "prompt words" or "prompt sentences". This process not only requires the creator to have a deep language foundation, but also a deep understanding of the inherent logic of the target task and the working mechanism of the model, which leads to relatively high manpower investment and cost. Therefore, how to efficiently and creatively construct prompts to maximize the potential of large language models has become an important topic in current research and practice. Summary of the invention

[0006] The purpose of the present invention is to solve the problems of the prior art and to propose a semantic relationship prediction method based on multi-faceted prompt optimization of a large language model.

[0007] In order to achieve the above object, the technical solution provided by the present invention is: A semantic relationship prediction method based on multi-aspect prompt optimization of a large language model, comprising: Acquire a data set, the data set including sentences having a semantic relationship and sentences not having the semantic relationship, whether or not having the semantic relationship is used as a true label of the sentence, and divide the data set into a training data set and a test data set; Define an initial prompt as the current prompt of the first round of iteration to start the iteration, input the current prompt and training data set into the large language model for prediction, and get the error case based on the true label; Input the current prompt and the error case into the large language model based on the first error analysis method to obtain the first error cause, and input the current prompt and the first error cause as the second prompt into the large language model to obtain a section set; Input the current prompt, error case and aspect set as the third prompt into the large language model, obtain the second error cause by using the second error analysis method, input the current prompt and the second error cause as the fourth prompt into the large language model to obtain a candidate prompt set; Input the candidate prompt set, error cases and aspect set into the large language model, output a new candidate prompt set as the current prompt to participate in the next round of iteration, iterate a preset number of times, and obtain a candidate prompt set each time; Each prompt in all candidate prompt sets is combined with each sentence in the test data set and input into the large language model for prediction, and the prediction results are obtained. The evaluation index of the prediction results obtained from each prompt is calculated, and the k prompts with the highest evaluation index are selected; The selected k prompts are combined to obtain a final prompt set, and the final prompt set is combined with the sentence to be predicted and input into the large language model for prediction to obtain a prediction result of whether the sentence to be predicted has the semantic relationship.

[0008] Furthermore, the current prompt and the training data set are input into the large language model for prediction, and error cases are obtained according to the true labels, including: The training data set D is sampled by cluster sampling or random sampling. train Sampling is performed to obtain the sampled data set D F sample ; The sampled data set D F sample And the current prompt are input into the large language model for prediction to get the predicted label; Compare the predicted labels with the true labels, and sample the sentences with inconsistent comparison results to get error cases.

[0009] Furthermore, the current prompt and the training data set are input into the large language model for prediction, and error cases are obtained according to the true labels, including: The training data set D train And the current prompt are input into the large language model for prediction to get the predicted label; Compare the predicted labels with the true labels, and sample the sentences with inconsistent comparison results to form a data set, denoted as D S wrong ; The dataset D is sampled by clustering. S wrong Sampling is performed to obtain sampled error cases.

[0010] Furthermore, the cluster sampling includes: The dataset to be sampled is converted into an embedded representation and divided into predefined K clusters according to the distance of the embedded representation in the semantic space; Determine whether random sampling is used. If it is random sampling, randomly sample from each cluster. If it is not random sampling, use the k-centerGreedy algorithm to sample each cluster.

[0011] Furthermore, the first error analysis method includes a naïve_reasons analysis method, a main_mistake analysis method and a topic_system2 analysis method. The naïve_reasons analysis method is to input the current prompt and error cases into the large language model; The main_mistake analysis method is to input the current prompt, error cases, and instructions for identifying the main error cases and performing error cause analysis based on the main error cases into the large language model; The topic_system2 analysis method is to input the current prompt, error cases, and instructions for identifying the topic of the current prompt and analyzing the cause of the error based on the topic into the large language model.

[0012] Furthermore, the second error analysis method includes a naïve_aspects analysis method, an orderwise analysis method and an asp-system2 analysis method. The naïve_aspects analysis method is to input the current prompt, error case and aspect set into the large language model; The orderwise analysis method is to input the current prompt, error case, aspect set and the instruction that all aspects should be considered comprehensively in each analysis into the large language model; The asp-system2 analysis method is to input the current prompt, error case, aspect set and the instruction of first screening the aspect set and then analyzing the cause of the error into the large language model.

[0013] Furthermore, the evaluation indicators include area under the curve, accuracy, precision, recall and F1 score.

[0014] Furthermore, the step of inputting each prompt in all candidate prompt sets into the large language model for prediction in combination with each sentence in the test data set includes: The test data set is sampled using the Kmeans clustering method or the random selection method; Each prompt in all candidate prompt sets combined with each sentence in the sampled test dataset are input into the large language model for prediction.

[0015] Compared with the prior art, the present invention has the following significant advantages: it can automatically construct prompts from multiple aspects to utilize a large language model, reducing labor costs and human errors, while improving the prediction effect of identifying whether a sentence contains semantic relationships. It can be used to help determine whether a sentence contains semantic relationships such as causal relationships, and to assist in text analysis, information extraction, and the construction of automatic question-answering systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an overall flow chart of a semantic relationship prediction method based on multi-faceted prompt optimization of a large language model according to the present invention; Figure 2 It is a flow chart of the aspect extraction based on the large language model of the present invention; Figure 3 is a flow chart of a method for sampling a data set using a kmeans clustering algorithm in the present invention; Figure 4 is a flow chart of constructing a candidate prompt set according to the present invention; Figure 5 It is a flow chart of the present invention for evaluating and selecting the predicted results of the prompt. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] like Figure 1 As shown, the present invention provides a semantic relationship prediction method based on multi-faceted prompt optimization of a large language model, taking the task of predicting causal relationships as an example, comprising the following steps: 1) Dataset acquisition: This task is a causal detection task based on medical literature. There are two datasets used: BioCausal-Small dataset and SMCausal dataset. The BioCausal-Small dataset is a dataset compiled by Kyriakakis et al. The dataset contains 2000 sentences from the PUBMED database, of which 1113 sentences contain causal relationships. Given the limited size of the BioCausal-Small dataset, this embodiment performs annotation work to create an SMCausal dataset specifically for biomedical causal detection. The annotations are performed by annotators familiar with biomedical terminology. The entire dataset is public and consists of sentences extracted from article titles and abstracts in the SemMedDB dataset, a total of 79762 sentences, of which 7631 sentences contain causal relationships. Both PUBMED and SemMedDB are professional databases in the biomedical field.

[0019] The BioCausal-Small dataset and the SMCausal dataset are split into training dataset and test dataset in a ratio of 7:3.

[0020] 2) Define the initial prompt as the current prompt for the first iteration: In the automation process of the causal detection task, it is crucial to define an appropriate initial prompt. This preliminary step needs to be completed through manual input to lay the foundation for subsequent automated processing. The main function of the initial prompt is to guide the large language model to perform preliminary predictive analysis, which will serve as the starting point of the analysis process, and the prediction result will be further used in the subsequent process to guide the model to identify and analyze possible causes of errors. The design of the initial prompt needs to be carefully adjusted for specific tasks to ensure that it can provide clear direction and sufficient information for the model. In the causal detection task, the purpose of the initial prompt is to stimulate the model's ability to identify whether there is a causal relationship in the text. Therefore, in the task of causal prediction, a typical initial prompt is: "Does the sentence have a causal relationship?" By asking such a question, the large language model is guided to deeply analyze the text content and find and confirm any signs that express or imply a causal relationship. Through such an initial prompt, the large language model can be guided to perform more sophisticated text analysis, providing a solid starting point for identifying and understanding causal relationships. Subsequently, based on the preliminary prediction results provided by the model, the model can be used to analyze which factors may lead to inaccurate predictions, thereby continuously iterating and optimizing the prompt set, and ultimately forming an efficient and accurate causal detection process.

[0021] 3) Aspect extraction: The core idea of ​​aspect-oriented programming is to separate cross-cutting concerns to improve the maintainability and scalability of the code. This embodiment uses a new method, aspect extraction, to apply the essence of aspect-oriented programming to the automated causal detection task. Here, you only need to manually enter an initial prompt p 0 : "Does the sentence have a causal relationship?". Subsequent tasks are performed automatically, and the large language model will generate a set of facets based on this initial prompt. There are two ways to extract facets: facet extraction based on a large language model and manually constructed facet extraction. Manually constructed facet extraction can be added to facet extraction based on a large language model, and the facets generated by these two methods can be merged. Facet extraction based on a large language model uses the power of a large language model to automatically generate relevant facets from text data. Figure 2 As shown in the figure, the specific process of facet extraction based on the large language model is as follows:

[0022] 3.1) Given an initial prompt p 0 The initial prompt is a prompt constructed by humans based on the current task, which is then input into the large language model. For example, for the current task causal detection task, the initial prompt p 0 It could be “Does the sentence have a causal relationship?”.

[0023] 3.2) Determine the training data set D first train Do sampling and then prediction or use the entire training data set for prediction and then sampling.

[0024] If sampling is performed before prediction, the sampling method is divided into two methods: kmeans clustering sampling and random sampling. The final data set D obtained after sampling is F sample Then the sampled data set D F sample and the current prompt (the current prompt in the first iteration is the initial prompt p 0 ) is input into the large language model for prediction. Each sentence will get a predicted label (0 or 1), 0 means that there is no causal relationship in the sentence, and 1 means that the current sentence contains a causal relationship. After the large language model completes the prediction, the error cases of prediction errors are sampled from the prediction results, that is, the samples where the true label and the predicted label are inconsistent, recorded as D F wrong .

[0025] If prediction is performed first and then sampling, the training data set D is directlytrain and the current prompt (the current prompt in the first iteration is the initial prompt p 0 ) is input into the large language model for prediction. Similarly, each sentence in the training data set will get a label. As in the above sampling first part, the sample of the prediction error is sampled from the prediction result, denoted as D S wrong Then use the kmeans clustering algorithm to cluster the data set D S wrong Sampling is performed to obtain the sampled data set D S sample .

[0026] Finally, the dataset D obtained by sampling and then predicting is used F wrong Or first predict and then sample the data set D S sample As an error case, it is input into the large language model, and the reasons for the prediction errors are summarized to obtain the first error reason. The input format is the current prompt plus the error case and the instruction based on the first error analysis method. The first error analysis method here includes "naïve_reasons", "main_mistake" and "topic_system2". The "naïve_reasons" method is the simplest method, which seeks to directly cause the large language model to generate errors. The "main_mistake" method allows the large language model to focus on the error cause analysis. Specifically, this analysis method allows the large language model to consider the main error cases in the sample and give the reasons for the error. The "topic_system2" method guides the large language model to analyze the error cause. Specifically, it allows the large language model to analyze the error cause in two steps: the first step is to identify the topic in the current prompt; the second step is to analyze the error cause based on the topic identified in the previous step.

[0027] If the "naïve_reasons" analysis method is selected, the large language model is required to directly analyze the cause of the error based on the given error case, for example: ## My current prompt is: Does the sentence have a causalrelationship? ##But this prompt gets the following examples wrong: ###Example1 Text: "In animal models, EPIT induces sustained unresponsiveness and prevents further sensitization mediated by Tregs." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model for genetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation,ie a single-step method for genomicevaluation." Label: No Prediction: Yes ##give 3 reasons why the prompt could have gotten these examples wrong.”.

[0028] If the "main_mistake" analysis method is selected, the large language model is required to consider the main error cases among all error cases, and then analyze the causes of the errors based on the identified main error cases, for example: ## My current prompt is: Does the sentence have a causalrelationship? ##But this prompt gets the following examples wrong: ###Example1 Text: "In animal models, EPIT induces sustained unresponsiveness and prevents further sensitization mediated by Tregs." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model for genetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation,ie a single-step method for genomicevaluation." Label: No Prediction: Yes ## Consider the main mistake,give 3 reasons why the prompt could havegotten these examples wrong.".

[0029] If the "topic_system2" analysis method is selected, the large language model is guided to analyze the cause of the error in steps. The main steps are divided into two steps: the first step is to identify the topic in the current prompt; the second step is to analyze the cause of the error based on the topic identified in the previous step. For example:

[0030] ## My current prompt is: Does the sentence have a causalrelationship? ##But this prompt gets the following examples wrong: ###Example1 Text: "In animal models, EPIT induces sustained unresponsiveness and prevents further sensitization mediated by Tregs." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model for genetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation,ie a single-step method for genomicevaluation." Label: No Prediction: Yes ## ###please use flowing steps to analyze the error reasons: 1.identify the topics of the current prompt,give at most 2 topics. 2.give the topics above,give 3 reasons why the prompt could havegotten these examples wrong.".

[0031] Then, the first error reason is input into the large language model again to classify the error reason, that is, what aspects need to be paid attention to during prediction. The prompt format for inputting the large language model here is the current prompt and the first error reason, for example:

[0032] ## My current prompt is:Does the sentence have a causalrelationship? Based on errors examples the problems with this prompt are: ###reason 0:The prompt may have gotten the examples wrong because itis not considering the complexity and nuances of causal relationships insentences. It may be oversimplifying the task by expecting a binary Yes or Noanswer,leading to misclassification of sentences that have subtle or indirectcausal relationships. ###reason 1:The prompt may have gotten the examples wrong because itis too focused on identifying causal relationships without considering the context and content of the text. It may not be capturing the nuances of the language used in the sentences, leading to incorrect classifications. The 5 new aspects are:".

[0033] At this point, we have obtained 5 new aspects. The aspects can be understood as the aspects that need to be considered when generating a new set of prompts: "aspect 0: Causality is when one event causes another even without explicit markers, aspect 1: Consider implicit causal phrases such as suggest and implying, aspect 2: Consider nuanced language, aspect 3: Capture complex relationships, aspect 4: Include contextual information". These 5 aspects are used for subsequent use.

[0034] like Figure 3 As shown in the figure, the Kmeans clustering sampling algorithm here converts the data set into an embedded representation, and divides it into a predefined number of K clusters according to the distance of the embedded representation in the semantic space. Then sampling is performed according to different pre-selected sampling methods. The sampling methods are divided into random sampling and k-centerGreedy algorithm sampling. Random sampling is random sampling from each cluster. The k-centerGreedy algorithm is a greedy algorithm used to select data subsets or solve facility location problems. It can help select a set of representative sample points, thereby effectively reducing computational costs and improving the overall performance of the task. For each cluster, the k-centerGreedy algorithm is used to sample to obtain a representative sample.

[0035] In the case of manually constructed aspect extraction, domain experts provide aspects based on specific tasks, that is, you can manually input aspects that need to be considered, that is, add new content to the aspects generated above. When only aspects generated based on a large language model are used, all aspects generated based on a large language model constitute an aspect set. When aspects generated based on a large language model and manually constructed aspects are used, aspects generated based on a large language model and manually constructed aspects together constitute an aspect set.

[0036] 4) Construct a prompt collection: Figure 4 As shown, this embodiment adopts a systematic approach to build and optimize the prompt set. This method includes combining the facet set obtained in the previous step, the current prompt and the error case with the advanced analysis capabilities of the large language model. In this way, the quality of the prompt can be further refined and improved. The following is a detailed description of this process:

[0037] 4.1) First, the extracted aspect set, the current prompt, and some randomly extracted error cases from the error cases generated by the current prompt are combined to form the third prompt and sent to the large language model as input. At this time, the large language model uses the second error analysis method to explore the potential cause of the error. The second error analysis method can choose one of the three second error analysis methods: "naïve_aspects", "orderwise" and "asp-system2".

[0038] 4.2) In the "naïve_aspects" analysis method, a simple and direct error analysis is sought by directly comparing aspects. This approach is suitable for analyzing errors in a specific single aspect, focusing on the clear error point. The "orderwise" method focuses on the interaction and combined effect between aspects, not only looking at each aspect separately, but also considering them as a whole to analyze the cause of the error. The "orderwise" method fully considers the possible mutual influence between aspects, so it can provide a more holistic perspective. Unlike the previous two, the "asp-system2" method starts from a global perspective, selects the key aspects that are most relevant to the error, and then generates an explanation of the cause of the error based on these aspects. This method focuses on identifying the core aspects that have a significant impact on the error, and combines these core aspects to build a more accurate error analysis.

[0039] If the "naïve_aspects" analysis method is selected, it seeks to directly compare aspects for simple and direct error analysis, that is, given all aspect sets, generate error causes. Only one aspect is considered when generating a cause. The prompt format for inputting the large language model here is the current prompt plus the error case and the aspect set generated in step 3). The final prompt is "

[0040] ## My current prompt is: Does the sentence have a causalrelationship? ## But this prompt gets the following examples wrong: ###Example1 Text: "In animal models, EPIT induces sustained unresponsiveness and prevents further sensitization mediated by Tregs." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model for genetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation, ie a single-step method for genomicevaluation." Label: No Prediction: Yes Please analyze the errors from following aspects: aspect 0: Causality is when one event causes another even without explicit markers, aspect 1: Consider implicit causal phrases such as suggest and implying, aspect 2: Consider nuanced language, aspect 3: Capture complex relationships, aspect 4: Include contextual information ## Give 3 reasons why the prompt could have gotten these examples wrong. ”.

[0041] If the "orderwise" analysis method is selected, the focus is on the interaction and comprehensive effect between aspects. Instead of looking at each aspect separately, they are considered as a whole to analyze the cause of the error. That is, all aspects are considered comprehensively when generating an error cause. The prompt format for inputting the large language model here is the current prompt plus the error case and the aspect set generated in step 3) and the instruction to consider all aspects comprehensively when analyzing a cause. The final prompt is "

[0042] ## My current prompt is: Does the sentence have a causalrelationship? ## But this prompt gets the following examples wrong: ###Example1 Text: "In animal models,EPIT induces sustained unresponsiveness andprevents further sensitization mediated by Tregs ." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model forgenetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation , ie a single-step method for genomicevaluation ." Label: No Prediction: Yes Please analysis the errors synthesize the following aspects: aspect 0:Causality is when one event causes another even withoutexplicit markers, aspect 1: Consider implicit causal phrases such as suggest andimplying, aspect 2: Consider nuanced language, aspect 3: Capture complex relationships, aspect 4: Include contextual information Each analysis should take into account all aspects previouslymentioned. ## Give 3 reasons why the prompt could have gotten these examples wrong. ”.

[0043] If the "asp-system2" analysis method is selected, it starts from a global perspective, selects the key aspects most related to the error, and then generates an explanation of the cause of the error based on these aspects. That is, first filter the aspect set generated in the third step, and then analyze the cause of the error based on the filtered aspect. When generating an error cause, only one aspect in the filtered aspect set will be considered. The prompt format for inputting the large language model here is the current prompt plus the error case and aspect set, and the instruction to first filter the aspect set and then analyze the cause of the error. The final prompt is "

[0044] ## My current prompt is: Does the sentence have a causalrelationship? ## But this prompt gets the following examples wrong: ###Example1 Text: "In animal models, EPIT induces sustained unresponsiveness and prevents further sensitization mediated by Tregs." Label: Yes Prediction: No ###Example 2 Text: "Using combined relationship matrices results in a model forgenetic evaluation where both pedigree and marker genotypes are usedsimultaneously for genetic evaluation , ie a single-step method for genomicevaluation ." Label: No Prediction: Yes Given the following aspects: aspect 0:Causality is when one event causes another even withoutexplicit markers, aspect 1: Consider implicit causal phrases such as suggest andimplying, aspect 2: Consider nuanced language, aspect 3: Capture complex relationships, aspect 4: Include contextual information Analysis the errors using following steps: 1.select the most related at most 3 aspects. 2.using the selected aspects to analysis the errors. ## Give 3 reasons why the prompt could have gotten these exampleswrong. ”。

[0045] 4.3) Identify the second error cause through the second error analysis method, feed back the current prompt and the second error cause to the large language model in the form of a prompt (the fourth prompt), and generate an improved new prompt (a set of candidate prompts). The format of the fourth prompt input to the large language model here is the current prompt, the second error cause generated in step 4.2), and the instruction to generate a new prompt. So the fourth prompt here is "

[0046] ## My current prompt is: Does the sentence have a causalrelationship? Based on errors examples the problems with this prompt are: ###reason 0:The prompt may have gotten the examples wrong because itis not considering the complexity and nuances of causal relationships insentences. It may be oversimplifying the task by expecting a binary Yes or Noanswer,leading to misclassification of sentences that have subtle or indirectcausal relationships. ###reason 1:The prompt may have gotten the examples wrong because itis too focused on identifying causal relationships without considering the context and content of the text.It may not be capturing the nuances of the language used in the sentences, leading to incorrect classifications. Based on the above information, generate a new prompt. ”.

[0047] The newly generated prompts need to be closely related to the original prompts in terms of semantics, but with appropriate adjustments and changes in wording and expression to ensure more effective causal detection tasks.

[0048] Repeat steps 3) and 4) using the previously obtained candidate prompt set as the current prompt, and obtain a new candidate prompt set, iterating a preset number of times. Through continuous rounds of feedback and optimization, the relevance and accuracy of the prompts can be continuously improved.

[0049] After the above process, we finally get a series of corrected and repeatedly optimized candidate prompt sets. These sets are designed to provide more professional and refined guidance for problems that may arise in causal detection tasks, helping researchers to more effectively determine whether there is a causal relationship in Chinese literature.

[0050] 5) Prompt evaluation and selection: Figure 5As shown, according to the prompts in the candidate prompt set obtained in step 4), each prompt and each sentence in the test data set are combined and input into the large language model for prediction. For each prompt in the candidate prompt set, the large language model is called for prediction and a result is obtained. In order to evaluate the effectiveness of these prompts, this embodiment adopts a set of widely recognized statistical indicators, including AUC (area under the curve), accuracy, precision, recall and F1 score. The five indicators are comprehensively sorted, with the highest priority indicator being F1 score, followed by recall, precision, accuracy, and AUC. Sort by F1 score first, and if the F1 scores are the same, sort based on recall. If the recalls are the same, sort based on precision, and so on. These indicators work together to provide a comprehensive and in-depth understanding of the performance of the prompts. Finally, sort according to the comprehensive indicators of different prompts, and finally select the top k prompts. The test data set is divided according to the division ratio mentioned above, accounting for approximately 30% of the total data volume. However, in actual operation, if the test data set is too large, it may lead to high computational costs, especially considering the cost of using high-level large language models (such as GPT-4), measures may need to be taken to reduce this financial burden. Therefore, this embodiment adopts clustering and random selection methods to select a part of the data set for centralized testing. This not only improves the efficiency of the test, but also helps us control costs. The selection methods include clustering and random selection methods. Specifically, the Kmeans clustering method is used to select representative test data. This method can ensure the diversity and wide coverage of the test data, thereby avoiding the selection of biased data and ensuring the accuracy and reliability of the results. Of course, in addition to clustering, random selection is also an effective method, especially when it is necessary to quickly reduce the scale of the test.

[0051] 6) Based on the resulting prompts, a large language model is used to determine whether sentences in medical literature contain causal relationships.

[0052] The above-mentioned embodiments only express one or several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A semantic relationship prediction method based on multi-aspect prompt optimization of a large language model, characterized in that: The semantic relationship prediction method based on multi-aspect prompt optimization of a large language model includes: Acquire a data set, the data set including sentences having a semantic relationship and sentences not having the semantic relationship, whether or not having the semantic relationship is used as a true label of the sentence, and divide the data set into a training data set and a test data set; Define an initial prompt as the current prompt of the first round of iteration to start the iteration, input the current prompt and training data set into the large language model for prediction, and get the error case based on the true label; Input the current prompt and the error case into the large language model based on the first error analysis method to obtain the first error cause, and input the current prompt and the first error cause as the second prompt into the large language model to obtain a section set; Input the current prompt, error case and aspect set as the third prompt into the large language model, obtain the second error cause by using the second error analysis method, input the current prompt and the second error cause as the fourth prompt into the large language model to obtain a candidate prompt set; Input the candidate prompt set, error cases and aspect set into the large language model, output a new candidate prompt set as the current prompt to participate in the next round of iteration, iterate a preset number of times, and obtain a candidate prompt set each time; Each prompt in all candidate prompt sets is combined with each sentence in the test data set and input into the large language model for prediction, and the prediction results are obtained. The evaluation index of the prediction results obtained from each prompt is calculated, and the k prompts with the highest evaluation index are selected; The selected k prompts are combined to obtain a final prompt set, and the final prompt set is combined with the sentence to be predicted and input into the large language model for prediction to obtain a prediction result of whether the sentence to be predicted has the semantic relationship.

2. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1 is characterized in that: The current prompt and training data set are input into the large language model for prediction, and error cases are obtained according to the true labels, including: The training data set D is sampled by cluster sampling or random sampling. train Sampling is performed to obtain the sampled data set D F sample ; The sampled data set D F sample And the current prompt are input into the large language model for prediction to get the predicted label; Compare the predicted labels with the true labels, and sample the sentences with inconsistent comparison results to get error cases.

3. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1, characterized in that: The current prompt and training data set are input into the large language model for prediction, and error cases are obtained according to the true labels, including: The training data set D train And the current prompt are input into the large language model for prediction to get the predicted label; Compare the predicted labels with the true labels, and sample the sentences with inconsistent comparison results to form a data set, denoted as D S wrong ; The dataset D is sampled by clustering. S wrong Sampling is performed to obtain sampled error cases.

4. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 2 or 3, characterized in that: The cluster sampling includes: The dataset to be sampled is converted into an embedded representation and divided into predefined K clusters according to the distance of the embedded representation in the semantic space; Determine whether random sampling is used. If it is random sampling, randomly sample from each cluster. If it is not random sampling, use the k-centerGreedy algorithm to sample each cluster.

5. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1, characterized in that: The first error analysis method includes a naïve_reasons analysis method, a main_mistake analysis method and a topic_system2 analysis method. The naïve_reasons analysis method is to input the current prompt and error cases into the large language model; The main_mistake analysis method is to input the current prompt, error cases, and instructions for identifying the main error cases and performing error cause analysis based on the main error cases into the large language model; The topic_system2 analysis method is to input the current prompt, error cases, and instructions for identifying the topic of the current prompt and analyzing the cause of the error based on the topic into the large language model.

6. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1, characterized in that: The second error analysis method includes a naïve_aspects analysis method, an orderwise analysis method and an asp-system2 analysis method. The naïve_aspects analysis method is to input the current prompt, error case and aspect set into the large language model; The orderwise analysis method is to input the current prompt, error case, aspect set and the instruction that all aspects should be considered comprehensively in each analysis into the large language model; The asp-system2 analysis method is to input the current prompt, error case, aspect set and the instruction of first screening the aspect set and then analyzing the cause of the error into the large language model.

7. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1, characterized in that: The evaluation indicators include area under the curve, accuracy, precision, recall and F1 score.

8. The semantic relationship prediction method based on large language model multi-faceted prompt optimization according to claim 1, characterized in that: The step of inputting each prompt in all candidate prompt sets into the large language model for prediction in combination with each sentence in the test data set includes: The test data set is sampled using the Kmeans clustering method or the random selection method; Each prompt in all candidate prompt sets combined with each sentence in the sampled test dataset are input into the large language model for prediction.

Images