Sample data determination method and device based on large model and storage medium

Abstract

The invention provides a sample data determination method and device based on a large model and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of deep learning, natural language processing, large models and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a sample question under a time reasoning task, a plurality of reference texts of the sample question and an answer generation rule; determining a first answer of the sample question according to a large model, the answer generation rule, the plurality of reference texts and respective first publishing time of the plurality of reference texts; and generating a sample answer pair set for training a time inference model according to the sample question and the first answer. Therefore, in combination with the large model, the sample question, the answer generation rule and the multiple reference texts, the answer of the sample question can be accurately obtained, so that the answer of the sample question does not need to be determined manually, and the model training cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as deep learning, natural language processing, and large models, and in particular to a method, device, and storage medium for determining sample data based on a large model. Background Art

[0002] When training a temporal reasoning model for answering questions in temporal reasoning tasks, how to obtain question-answer pairs for training the temporal reasoning model in a convenient way is very important for reducing the training cost of the model. Summary of the invention

[0003] The present disclosure provides a method, device and storage medium for determining sample data based on a large model.

[0004] According to one aspect of the present disclosure, a method for determining sample data based on a big model is provided, the method comprising: obtaining a sample question under a temporal reasoning task, a plurality of reference texts for the sample question, and an answer generation rule; determining a first answer to the sample question based on the big model, the answer generation rule, the plurality of reference texts, and a first publication time of each of the plurality of reference texts; and generating a set of sample answer pairs for training a temporal reasoning model based on the sample question and the first answer.

[0005] According to another aspect of the present disclosure, a sample data determination device based on a large model is provided, the device comprising: an acquisition module for acquiring sample questions under a temporal reasoning task, multiple reference texts for the sample questions, and answer generation rules; a first determination module for determining a first answer to the sample question based on the large model, the answer generation rules, the multiple reference texts, and the first publication time of each of the multiple reference texts; and a generation module for generating a set of sample answer pairs for training a temporal reasoning model based on the sample question and the first answer.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the large model-based sample data determination method proposed above in the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the large model-based sample data determination method proposed above in the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the large model-based sample data determination method proposed above in the present disclosure.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0012] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0015] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0017] Figure 7 It is a block diagram of an electronic device used to implement the method for determining sample data based on a large model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0019] In the related art, the answers to sample questions under the temporal reasoning task are usually determined manually, so as to train the temporal reasoning model based on the sample questions and answers, and use the trained temporal reasoning model to answer the pending questions under the temporal reasoning task. However, this manual method of obtaining question-answer pairs leads to a high cost for training the temporal reasoning model. Therefore, how to obtain the answer pairs for training the temporal reasoning model in a convenient way is a technical problem that needs to be solved urgently.

[0020] In response to the above problems, the present disclosure proposes a method, device and storage medium for determining sample data based on a large model. The scheme proposes the following technical concepts: obtaining sample questions, multiple reference texts and answer generation rules for sample questions under a temporal reasoning task; determining the first answer to the sample question based on the large model, the answer generation rule, the multiple reference texts and the first publication time of each of the multiple reference texts; generating a set of sample answer pairs for training the temporal reasoning model based on the sample question and the first answer. Thus, by combining the large model, the sample question, the answer generation rule and the multiple reference texts, the answer to the sample question can be accurately obtained, thereby eliminating the need to manually determine the answer to the sample question, reducing the cost of obtaining sample data for training the temporal reasoning model, and thereby improving the efficiency of the temporal reasoning model trained based on the sample data.

[0021] Figure 1 It is a schematic diagram according to the first embodiment of the present disclosure. It should be noted that the sample data determination method based on a large model in the embodiment of the present disclosure can be applied to a sample data determination device based on a large model. The device can be an electronic device, or can be configured in an electronic device so that the electronic device can perform the sample data determination function based on the large model.

[0022] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, and other hardware devices with various operating systems, touch screens and / or display screens.

[0023] It should be noted that, in the following embodiments, the sample data determination device based on the large model is taken as an electronic device as an example for description.

[0024] like Figure 1 As shown, the method for determining sample data based on a large model may include the following steps:

[0025] Step 101, obtaining sample questions under the temporal reasoning task, multiple reference texts of the sample questions and answer generation rules.

[0026] Among them, temporal reasoning tasks refer to tasks that require combining temporal information to infer and judge the relationship between events, such as the order of occurrence, duration, time interval, causal relationship, etc. of events.

[0027] Among them, the sample problems under the temporal reasoning task refer to the specific instance problems used in performing the temporal reasoning task. These problems are used to train the model's reasoning ability about temporal relationships.

[0028] The reference text refers to the text material needed to answer the sample questions, which can usually be obtained from various sources on the Internet.

[0029] The answer generation rule refers to the rule based on which the answer to the sample question is generated based on multiple reference texts.

[0030] In some embodiments, a possible implementation method for obtaining sample problems under the temporal reasoning task is: obtaining an application scenario of the temporal reasoning task, and obtaining sample problems in the temporal reasoning task under the application scenario.

[0031] Among them, the application scenarios in this embodiment may include one or more.

[0032] Among them, obtaining application scenarios of temporal reasoning tasks refers to obtaining scenarios in which temporal reasoning tasks will be used. For example, application scenarios may include but are not limited to news analysis, medical diagnosis, legal cases, etc., among which news analysis can be divided into: science and technology news analysis, economic news analysis, entertainment news analysis, health news analysis, environmental news analysis, etc.

[0033] Step 102, determining a first answer to the sample question based on the large model, answer generation rules, multiple reference texts, and the first publication time of each of the multiple reference texts.

[0034] In some embodiments, answer generation rules, multiple reference texts, the first publishing time of each of the multiple reference texts, and a sample question may be input into a large model to generate a first answer to the sample question through the large model.

[0035] Among them, the large model in this embodiment can be a large model with question answering capabilities, for example, the Generative Pre-trained Transformer-4 (GPT-4) model, the GPT-4o model, etc. This embodiment does not specifically limit the large model.

[0036] In other embodiments, a first prompt word is generated based on a sample question, an answer generation rule, multiple reference texts, and the first publication time corresponding to each of the multiple reference texts, wherein the first prompt word is used to indicate that an answer to the sample question is generated based on the answer generation rule using multiple reference texts and the first publication time corresponding to each of the multiple reference texts; the first prompt word is input into the macro model to obtain the first answer to the sample question. Thus, the macro model is given clear instructions through the first prompt word, so that the model can better understand the generation requirements, which helps to improve the accuracy of the first answer generated by the macro model.

[0037] Step 103: Generate a set of sample answer pairs for training the temporal reasoning model based on the sample question and the first answer.

[0038] In this embodiment, a sample answer pair may be formed based on the sample question and the first answer, and a sample answer pair set for training the temporal reasoning model may be generated based on the sample answer pair.

[0039] It should be noted that the temporal reasoning model refers to a model that can answer questions in temporal reasoning tasks.

[0040] The sample data determination method based on the big model of the disclosed embodiment obtains sample questions under the temporal reasoning task, multiple reference texts of the sample questions, and answer generation rules; determines the first answer to the sample question based on the big model, the answer generation rules, the multiple reference texts, and the first publication time of each of the multiple reference texts; and generates a set of sample answer pairs for training the temporal reasoning model based on the sample question and the first answer. Thus, by combining the big model, the sample question, the answer generation rules, and the multiple reference texts, the answer to the sample question can be accurately obtained, thereby eliminating the need to manually determine the answer to the sample question, reducing the cost of obtaining sample data for training the temporal reasoning model, and further improving the efficiency of the temporal reasoning model trained based on the sample data.

[0041] In order to obtain more diverse answer pairs, the first publication time of multiple reference texts can also be modified, and the second answer of the sample question can be determined by combining the large model, multiple reference texts, the second publication time of the multiple reference texts, and the answer generation rules, and the sample question, the first answer, and the second answer can be combined to generate a set of answer pairs for training the temporal reasoning model. Figure 2 This process is described exemplarily.

[0042] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure.

[0043] like Figure 2 As shown, the method may also include the following steps:

[0044] Step 201, obtaining sample questions under the temporal reasoning task, multiple reference texts of the sample questions and answer generation rules.

[0045] It should be noted that, for the specific description of step 201, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0046] Step 202 , modifying the first publishing time of each of the plurality of reference texts to obtain a second publishing time corresponding to each reference text, wherein the first publishing time and the second publishing time corresponding to each reference text are different.

[0047] It is understandable that in different application scenarios, the first publication time of each of the multiple reference texts is modified to obtain the second publication time corresponding to each reference text in different ways, which are exemplarily described as follows:

[0048] As an example, the first publication time of multiple reference texts is randomly shuffled to obtain the second publication time corresponding to each reference text. Thus, by randomly shuffling the publication time of multiple reference texts, more reference texts with different publication times are constructed, data enhancement of the reference texts is achieved, and then more sampled sample answer pairs can be obtained.

[0049] For example, the number of reference texts is 3, namely reference text 1, reference text 2 and reference text 3, among which the first publication time of reference text 1 is January 1, 2022, the first publication time of reference text 2 is November 7, 2024, and the first publication time of reference text 3 is April 30, 2007. After randomly shuffling the first publication time of these three reference texts, the second publication time of these three reference texts can be respectively: the second publication time of reference text 1 is April 30, 2007, the second publication time of reference text 2 can be January 1, 2022, and the second publication time of reference text 3 can be November 7, 2024.

[0050] In some embodiments, a second prompt word may be generated according to the first publication time of each of the multiple reference texts, wherein the second prompt word is used to indicate that the first publication time is randomly shuffled. Correspondingly, the second prompt word is input into the large model to obtain the second publication time corresponding to each reference text through the large model. Thus, the publication time of the reference text can be modified quickly through the large model, thereby improving the efficiency of obtaining reference texts with different publication times.

[0051] For a reference text, the first publishing time and the second publishing time of the reference text are different.

[0052] In another embodiment, the first publishing time of each of the plurality of reference texts may be input into a preset time random scrambling program, so as to obtain the second publishing time of each of the reference texts through the time random scrambling program.

[0053] As another example, the earliest publication time and the latest publication time are determined from the first publication time of each of the multiple reference texts; for each reference text, a time is randomly selected between the earliest publication time and the latest publication time as the second publication time corresponding to the reference text. Thus, the publication time of the reference text is accurately modified in combination with the earliest publication time and the latest publication time of the reference text, thereby constructing more reference texts with different publication times, achieving data enhancement of the reference text, and then obtaining more sampled sample answer pairs.

[0054] For example, the number of reference texts is 3, and the 3 reference texts are reference text 1, reference text 2, and reference text 3, wherein the first publication time of reference text 1 is January 01, 2022, the first publication time of reference text 2 is November 07, 2024, and the first publication time of reference text 3 is April 30, 2007. By analyzing the publication time of these three reference texts, it can be determined that the earliest publication time is April 30, 2007, and the latest publication time is November 07, 2024. For each reference text, a time can be randomly selected between the earliest publication time and the latest publication time as the second publication time of the reference text. Among them, the second publication time of the three reference texts can be respectively: the second publication time of reference text 1 can be October 08, 2023, the second publication time of reference text 2 can be September 9, 2008, and the second publication time of reference text 3 can be November 12, 2011.

[0055] Step 203, determining a first answer to the sample question based on the large model, answer generation rules, multiple reference texts, and the first publication time of each of the multiple reference texts.

[0056] The answer generation rules in this embodiment may include at least one of the following:

[0057] In the case of a first reference text with factual conflict among multiple reference texts, answering sample questions based on a second reference text among the multiple reference texts and the publication time corresponding to each of the second reference texts, wherein the second reference text is a reference text among the multiple reference texts other than the first reference text;

[0058] In the case where there is no factually conflicting reference text among the multiple reference texts, answer the sample questions based on the multiple reference texts and the publication time of each of the multiple reference texts;

[0059] Answer sample questions based on pre-set answers when there are factual conflicts in each reference text;

[0060] In the case that at least two third reference texts among the multiple reference texts conflict with respect to the same fact, a fourth reference text with the latest publication time is obtained from the third reference texts, and a sample question is answered based on the fourth reference text, the publication time of the fourth reference text, the fifth reference text, and the publication time of the fifth reference text, wherein the fifth reference text is a reference text other than the third reference text among the multiple reference texts;

[0061] In the case that there is a sixth reference text among multiple reference texts that does not include the occurrence time of an event, answer the sample question based on the publication time of the sixth reference text, the event in the sixth reference text, the event in the seventh reference text and the corresponding occurrence time, wherein the seventh reference text is a reference text among the multiple reference texts other than the sixth reference text.

[0062] Among them, the event and the corresponding occurrence time in the seventh reference text refer to the event in the seventh reference text and the occurrence time of the event, that is, when the event and the occurrence time of the event are recorded in the seventh reference text, the occurrence time of the event recorded in the seventh reference text shall prevail.

[0063] It can be understood that, since the sixth reference text does not include the time of occurrence of the event, at this time, the publication time of the sixth reference text is used as the time of occurrence of the event.

[0064] The preset answer may be preset according to actual needs. For example, the preset answer may be: I have not learned the relevant knowledge, please check XXX.

[0065] In this embodiment, by combining various situations included in the answer generation rules, the large model can combine the answer generation rules to accurately generate answers to sample questions in various situations.

[0066] In some embodiments, the answer generation rule may further include: when it is determined that the sample question includes a preset tense word, it may be determined that the time in the sample question is the past year, that is, from one year ago to now.

[0067] Among them, preset tense words may include but are not limited to recently, recent, now, nowadays, currently, presently, today, etc.

[0068] Among them, in some other embodiments, the above-mentioned answer generation rules may also include: adding a preset disclaimer prompt to the answer while ensuring that the generated answer is fluent.

[0069] Among them, the preset disclaimer is pre-set according to actual needs. For example, the preset disclaimer can be: The above knowledge is as of x month, x year, 20xx, and you can go to the official website to check the latest relevant published results.

[0070] Step 204 , determining a second answer to the sample question based on the large model, the multiple reference texts, the second publishing time of the multiple reference texts, and the answer generation rules.

[0071] In some embodiments, multiple reference texts, the second publishing time of each of the multiple reference texts, a sample question, and an answer generation rule may be input into a large model to generate a second answer to the sample question through the large model.

[0072] In some embodiments, a corresponding third prompt word may be generated according to the sample question, multiple reference texts, the second publishing time of each of the multiple reference texts, and the answer generation rule, wherein the third prompt word is used to indicate that the sample question is answered based on the answer generation rule according to the multiple reference texts and the second publishing time. Correspondingly, the third prompt word is input into the large model to pass the second answer to the sample question.

[0073] It should be noted that the sequence numbers of the steps in the embodiments of the present disclosure are not used to limit the execution order of the steps. On the premise of meeting the purpose of the invention, those skilled in the art can adjust the execution order of the steps, for example, swapping the order of steps 203 and 204, or executing steps 203 and 204 in parallel.

[0074] Step 205: Generate a set of sample answer pairs for training a temporal reasoning model based on the sample question, the first answer, and the second answer.

[0075] In some embodiments, a first sample answer pair may be formed based on the sample question and the first answer, a second sample answer pair may be formed based on the sample question and the second answer, and a sample answer pair set for training a temporal reasoning model may be formed based on the first sample answer pair and the second sample answer pair.

[0076] In this embodiment, the publication time of the reference text is modified to construct more reference texts with different publication times, and the first answer to the sample question is determined by combining the large model, the answer generation rule, multiple reference texts, and the first publication time of each of the multiple reference texts, and the second answer to the sample question is determined by combining the large model, multiple reference texts, the second publication time of each of the multiple reference texts, and the answer generation rule, and a set of sample answer pairs for training the temporal reasoning model is generated based on the sample question, the first answer, and the second answer. Thus, by modifying the publication time of the reference text, data enhancement of the reference text is achieved, so that more diverse sample data can be generated.

[0077] In some embodiments, in order to obtain more diverse sample problems, the definition information of multiple types of temporal reasoning problems under the temporal reasoning task, the application scenarios of the temporal reasoning task and the large model can be combined to generate sample problems of different categories. In order to make the process clear, the following is combined with Figure 3 This process is described exemplarily.

[0078] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure.

[0079] like Figure 3 As shown, the method may include the following steps:

[0080] Step 301, obtaining definition information of multiple types of temporal reasoning problems under the temporal reasoning task.

[0081] Among them, the multiple categories of time reasoning problems include at least two of the following: absolute time problems, time error problems, time missing problems, time relative problems, time calculation problems, and time fusion problems. As a result, the large model can generate first questions of different categories, and then the sample answer pairs can include sample questions of various types of time reasoning problems, providing diversified sample questions for training the time reasoning model. Among them, the diversified sample questions can enable the trained time reasoning model to handle various types of problems under the time reasoning task, thereby improving the processing effect of the trained output time reasoning model.

[0082] Among them, absolute time questions are questions that ask about the specific time of an event or about an event that occurred at a certain time.

[0083] Among them, time error issues refer to time-fact inquiries that do not conform to the actual facts, or inquiries about facts related to a person who has stopped or passed away. For example, asking about a person's "recent activities" when the person has already retired or passed away. Another example is asking about the details of an event that "happened on April 2, 2017", but the actual time of the event should be April 2, 2018.

[0084] Among them, the time missing problem refers to the problem that certain time information (such as time points, time intervals, etc.) is missing or not provided, and it is necessary to combine the time information of the reference text to infer the missing time information.

[0085] Among them, time-relative questions refer to the relative time given in the question (for example, asking about this year, now, last year, last time, the last time, the latest time, etc.). It is necessary to combine the current time as a reference time point to infer the specific inquiry time or event information.

[0086] Among them, time calculation problems refer to a type of problem that requires time calculation, such as asking how old someone is this year, or how many times something has happened.

[0087] Among them, time calculation problems can be further divided into association problems, intervention problems and counterfactual problems.

[0088] Among them, the related problem is the problem of real time calculation. For example, if it is known that a person died in 1977, the related question can be "how old was a person in 1976".

[0089] Among them, intervention problems refer to questions that require time calculation under real hypothetical situations, for example, if someone died in 1977, how old was he in 1978.

[0090] Among them, counterfactual questions refer to questions that require time calculation under assumptions that violate the actual situation. For example, the actual situation is that a person died in 1977, and the question "If someone died in 1979, how old was he in 1978" is asked. In this question, the assumption is inconsistent with the actual situation. Therefore, the subsequent questions must also violate the actual facts. Therefore, this type of question is called a counterfactual question.

[0091] Among them, time fusion problems refer to problems that require combining reasoning to summarize events with a time span. For example, asking about what happened from a certain year to a certain year, asking about the history of a certain position, asking about the life resume of a certain person, etc.

[0092] Step 302: Obtain an application scenario for the temporal reasoning task.

[0093] Step 303: Generate multiple first questions in the application scenario based on the large model, the application scenario and the definition information, wherein the category of each first question corresponds to one category of the multiple categories of temporal reasoning problems.

[0094] In some embodiments, definition information of an application scenario and multiple types of temporal reasoning problems may be input into a large model to generate multiple first problems in the application scenario through the large model.

[0095] In some embodiments, a fourth prompt word may be generated according to the definition information of the application scenario and multiple types of time reasoning problems, wherein the fourth prompt word is used to indicate that multiple problems of the application scenario are generated according to the definition information, and the categories of the generated problems correspond to one category of the multiple types of time reasoning problems. Correspondingly, the fourth prompt word is input into the large model to obtain multiple first problems in the application scenario. Thus, prompting the large model with the fourth prompt word helps to improve the accuracy of the first problem generated by the large model.

[0096] It can be understood that there may be multiple application scenarios in this embodiment.

[0097] In some embodiments, when there are multiple application scenarios, the definition information of multiple application scenarios and multiple types of time reasoning problems can be directly input into the big model, so that the big model can generate multiple first problems corresponding to multiple application scenarios at one time.

[0098] In other embodiments, when there are multiple application scenarios, for each application scenario, the definition information of the application scenario and multiple types of time reasoning problems can be input into the big model, so that the big model only generates multiple first problems under the corresponding application scenario each time.

[0099] In some embodiments, in order to enable the large model to accurately generate multiple first questions in the application scenario, correspondingly, example questions of various types of time reasoning problems in the application scenario can also be obtained, and correspondingly, multiple first questions in the application scenario are generated according to the large model, example questions, application scenario and definition information. Thus, the large model can accurately generate multiple first questions in the application scenario based on the definition information and example questions of various types of time reasoning problems.

[0100] In this embodiment, for each type of time reasoning problem, multiple real problems of this type of time reasoning problem in the application scenario can be obtained, and example problems can be determined from the multiple real problems.

[0101] In some embodiments, application scenarios, definition information of various types of temporal reasoning problems, and example problems may be input into a large model to generate multiple first problems in the application scenario through the large model.

[0102] In some embodiments, a fifth prompt word may be generated according to the application scenario, the definition information of multiple types of temporal reasoning problems, and the example problems, wherein the fifth prompt word is used to indicate that multiple problems of the application scenario are generated according to the definition information and the example problems, and the categories of the generated problems correspond to one category of the multiple types of temporal reasoning problems, and correspondingly, the fifth prompt word is input into the large model to obtain multiple first problems in the application scenario. Thus, prompting the large model with the fifth prompt word helps to improve the accuracy of the first questions generated by the large model.

[0103] Step 304: determine sample questions based on the first question.

[0104] In some embodiments, the plurality of first questions may be directly used as sample questions respectively.

[0105] In other embodiments, for each first question, it can be determined whether the first question includes the first time keyword, and if it is determined that the first question includes the first time keyword, the first time keyword in the first question is rewritten to obtain a second question, wherein the second time keyword in the second question is different from the first time keyword. Correspondingly, the first question and the second question are respectively used as sample questions. Thus, by rewriting the time keyword in the first question, data enhancement of the first question can be achieved, so that more sample questions can be obtained, and then more sample answer pairs can be obtained.

[0106] For example, the first question is: "By 2022, how many meetings has the XX organization held?", where the first time keyword in the first question is 2022. Suppose the second question obtained by rewriting the first question is: "By this year, how many meetings has the XX organization held?", where the second time keyword in the second question is "this year".

[0107] For another example, the first question is: "By 2022, how many meetings has the XX organization held?", where the first time keyword in the first question is 2022. Suppose the second question obtained by rewriting the first question is: "By 2023, how many meetings has the XX organization held?", where the second time keyword in the second question is "2023".

[0108] For another example, the first question is: "By 2022, how many meetings has the XX organization held?", where the first time keyword in the first question is 2022. Suppose the second question obtained by rewriting the first question is: "By last year, how many meetings has the XX organization held?", where the second time keyword in the second question is "last year".

[0109] In some embodiments, the first question can be rewritten by the large model to obtain the second question. Thus, the second question is determined conveniently and quickly. As an example, a corresponding target prompt word can be generated according to the first question, wherein the target prompt word is used to indicate that the first time keyword in the first question is to be rewritten. Then, the target prompt word is input into the large model to obtain the second question through the large model.

[0110] Step 305, obtaining multiple reference texts and answer generation rules for sample questions.

[0111] Step 306, determining a first answer to the sample question based on the large model, answer generation rules, multiple reference texts, and the first publication time of each of the multiple reference texts.

[0112] Step 307: Generate a set of sample answer pairs for training the temporal reasoning model based on the sample question and the first answer.

[0113] It should be noted that, for the specific description of step 306 and step 307, reference may be made to the relevant description in other embodiments, which will not be repeated here.

[0114] In this embodiment, the definition information of multiple types of temporal reasoning problems under the temporal reasoning task, as well as the application scenarios and large models of the temporal reasoning task are combined to generate different types of sample problems under different application scenarios, thereby providing a variety of sample problems, and then obtaining more sample answer pairs, which helps to improve the accuracy of the trained output temporal reasoning model.

[0115] In order to clearly understand the present disclosure, Figure 4 The method of this embodiment is described exemplarily.

[0116] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure.

[0117] like Figure 4 As shown, the method may include the following steps:

[0118] Step 401, obtaining definition information of multiple types of temporal reasoning problems under the temporal reasoning task.

[0119] Step 402: Obtain an application scenario for the temporal reasoning task.

[0120] Step 403, generating multiple first questions in the application scenario according to the large model, the application scenario and the definition information, wherein the category of the type question of each first question corresponds to one category of the multiple categories of temporal reasoning problems.

[0121] Step 404: determine sample questions based on the first question.

[0122] It should be noted that for the specific description of steps 401 to 404, reference may be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0123] Step 405, obtaining multiple reference texts and answer generation rules for sample questions.

[0124] Step 406, modifying the first publishing time of each of the plurality of reference texts to obtain a second publishing time corresponding to each reference text, wherein the first publishing time and the second publishing time corresponding to each reference text are different.

[0125] Step 407, determining a first answer to the sample question based on the large model, answer generation rules, multiple reference texts, and the first publication time of each of the multiple reference texts.

[0126] Step 408, determining a second answer to the sample question based on the large model, the multiple reference texts, the second publishing time of the multiple reference texts, and the answer generation rules.

[0127] Step 409: Generate a set of sample answer pairs for training the temporal reasoning model based on the sample question, the first answer, and the second answer.

[0128] It can be understood that, for the specific description of steps 406 to 409, reference can be made to the relevant descriptions in other embodiments, which will not be repeated here.

[0129] In this embodiment, a large model is used to generate the first questions of various types of time reasoning problems in different application scenarios, and the generated first questions are used as sample questions. The publication time of the reference text corresponding to the sample questions is modified, thereby achieving data enhancement of the reference text, thereby generating more diverse sample data. As a result, while eliminating the need for manual answers to sample questions, diverse sample data can be automatically constructed, which helps to improve the accuracy of the trained time reasoning model.

[0130] Based on any of the above embodiments, after generating a set of sample answer pairs, a temporal reasoning model can be trained according to the set of sample answer pairs to obtain a trained temporal reasoning model, wherein the trained temporal reasoning model is used to determine the answers to pending questions in the temporal reasoning task.

[0131] In some embodiments, the initial temporal reasoning model may be trained in combination with supervised fine-tuning (SFT) and direct preference optimization (DPO) to obtain a trained temporal reasoning model.

[0132] The following example is used to illustrate this. Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5The method may include the following steps: Step 501, generating the first question of various time reasoning problems in multiple different application scenarios through a large model. Step 502, in the case where the first question has a first time keyword, rewrite the first question to obtain a second question, and use the first question and the second question as sample questions, respectively, wherein the second time keyword in the second question is different from the first time keyword in the first question. Step 503, obtaining multiple reference texts and corresponding first publication times corresponding to the sample question, rewriting the first publication times of the multiple reference texts, and obtaining the second publication time corresponding to each reference text. Step 504, generating a first target prompt word according to the answer generation rule, the sample question, the multiple reference texts, and the first publication time of each of the multiple reference texts, and inputting the first target prompt word into the large model to obtain the first answer to the sample question. Step 505, generating a second target prompt word according to the answer generation rule, the sample question, the multiple reference texts, and the second publication time of each of the multiple reference texts, and inputting the second target prompt word into the large model to obtain the second answer to the sample question. Step 506, forming a first answer pair according to the sample question and the first answer, and forming a second answer pair according to the sample question and the second answer, and generating a set of sample answer pairs for training the temporal reasoning model according to the first answer pair and the second answer pair.

[0133] The answer generation rules may include but are not limited to the following rules:

[0134] Rule 1: When it is determined that the sample question includes a preset tense word, it can be determined that the time in the sample question is the past year, that is, from one year ago to now.

[0135] Rule 2: When at least two first target reference texts among multiple reference texts conflict with each other regarding the same event, obtain the second target reference text with the latest publication time from the first target reference texts, and take the event in the second target reference text as accurate, that is, when answering the sample question, answer the sample question based on the event in the second target reference text.

[0136] Rule 3: In the event of a factual conflict in a reference text, the reference text may be determined to be unavailable.

[0137] Rule 4: If the reference text includes the time of occurrence of an event, answer the sample question based on the event and the time of occurrence of the event. If the reference text does not include the time of occurrence of the event, use the publication time of the reference text as the time of occurrence of the event. That is, answer the sample question in combination with the publication time of the reference text and the event included in the reference text.

[0138] Rule 5: If there is at least partially available third target reference text among multiple reference texts, the answer to the sample question can be inferred by combining the third target reference text and the corresponding publication time.

[0139] Rule 6: If the above-mentioned multiple reference texts are all unavailable, the sample question is answered according to the preset answer, that is, the large model outputs the preset answer. For example, the preset answer may be "I have not learned the relevant knowledge, please refer to XXX".

[0140] Rule 7: Require that facts in the generated answers are found from multiple reference texts.

[0141] Rule 8: While ensuring the generated answer is coherent, add a preset disclaimer to the answer.

[0142] Among them, the preset disclaimer is pre-set according to actual needs. For example, the preset disclaimer can be: The above knowledge is as of x month, x year, 20xx, and you can go to the official website to check the latest relevant published results.

[0143] Rule 9: Provide as many details as possible while ensuring that the facts are correct. For example, if there is an event, if you can infer the time of the event, write the time in your answer.

[0144] Rule 10: When determining the answers to sample questions, if the content in the reference text can be combined, then combine it. If the content in the reference text cannot be combined, then answer the sample questions based on the content in the reference text with the latest publication time.

[0145] Rule 11: Output the answer in the format of total score.

[0146] In this embodiment, a method for automatically generating answers to sample questions is provided, thereby avoiding manual determination of answers to sample questions, reducing the cost of obtaining sample answer pairs for training temporal reasoning models, and further reducing the training cost of temporal reasoning models.

[0147] In order to implement the above embodiments, the present disclosure also provides a sample data determination device based on a large model.

[0148] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure.

[0149] like Figure 6 As shown, the large model-based sample data determination device 60 may include: an acquisition module 601 , a first determination module 602 and a generation module 603 .

[0150] The acquisition module 601 is used to acquire sample questions, multiple reference texts of the sample questions and answer generation rules under the temporal reasoning task.

[0151] The first determination module 602 is used to determine the first answer to the sample question according to the large model, the answer generation rule, the multiple reference texts and the first publication time of each of the multiple reference texts.

[0152] The generation module 603 is used to generate a set of sample answer pairs for training the temporal reasoning model according to the sample question and the first answer.

[0153] As a possible implementation of the embodiment of the present disclosure, the device may further include:

[0154] A modification module, used to modify the first publication time of each of the plurality of reference texts to obtain a second publication time corresponding to each reference text, wherein the first publication time and the second publication time corresponding to each reference text are different;

[0155] A second determination module is used to determine a second answer to the sample question based on the large model, the multiple reference texts, and the second publication time and answer generation rule of the multiple reference texts;

[0156] The generation module 603 is specifically used to generate a set of sample answer pairs for training the temporal reasoning model according to the sample question, the first answer and the second answer.

[0157] As a possible implementation manner of the embodiment of the present disclosure, the modification module is specifically used to: randomly shuffle the first publication time of each of the multiple reference texts to obtain the second publication time corresponding to each reference text.

[0158] As a possible implementation of the embodiment of the present disclosure, the modification module is specifically used to:

[0159] The earliest publication time and the latest publication time are determined from the first publication time of each of the multiple reference texts; for each reference text, a time is randomly selected between the earliest publication time and the latest publication time as the second publication time corresponding to the reference text.

[0160] As a possible implementation of the embodiment of the present disclosure, the acquisition module 601 may include:

[0161] A first acquisition unit, used to acquire definition information of multiple types of temporal reasoning problems under a temporal reasoning task;

[0162] A second acquisition unit, used to acquire an application scenario of the temporal reasoning task;

[0163] A generating unit, configured to generate a plurality of first problems in the application scenario according to the large model, the application scenario and the definition information, wherein a category of each first problem corresponds to a category of the plurality of temporal reasoning problems;

[0164] The determining unit is used to determine a sample question according to the first question.

[0165] As a possible implementation manner of the embodiment of the present disclosure, the determining unit is specifically configured to:

[0166] In the case where it is determined that the first question includes the first time keyword, rewriting the first time keyword in the first question to obtain a second question, wherein the second time keyword in the second question is different from the first time keyword;

[0167] The first question and the second question are taken as sample questions respectively.

[0168] As a possible implementation of the embodiment of the present disclosure, the acquisition module 601 may further include: a third acquisition unit, configured to acquire example problems of various types of time reasoning problems in application scenarios.

[0169] The generation unit is specifically used to generate multiple first questions in the application scenario according to the large model, example questions, application scenarios and definition information.

[0170] As a possible implementation method of an embodiment of the present disclosure, the multiple types of time reasoning problems include at least two of the following: absolute time problem, time error problem, time missing problem, relative time problem, time calculation problem, and time fusion problem.

[0171] As a possible implementation method of an embodiment of the present disclosure, the first determination module is specifically used to: generate a first prompt word according to a sample question, an answer generation rule, multiple reference texts, and a first publication time corresponding to each of the multiple reference texts, wherein the first prompt word is used to indicate that an answer to the sample question is generated based on the answer generation rule using multiple reference texts and a first publication time corresponding to each of the multiple reference texts; and input the first prompt word into a large model to obtain a first answer to the sample question.

[0172] As a possible implementation of the embodiment of the present disclosure, the answer generation rule includes at least one of the following:

[0173] In the case of a first reference text with factual conflict among multiple reference texts, answering sample questions based on a second reference text among the multiple reference texts and the publication time corresponding to each of the second reference texts, wherein the second reference text is a reference text among the multiple reference texts other than the first reference text;

[0174] In the case where there is no factually conflicting reference text among the multiple reference texts, answer the sample questions based on the multiple reference texts and the publication time of each of the multiple reference texts;

[0175] Answer sample questions based on pre-set answers when there are factual conflicts in each reference text;

[0176] In the case that at least two third reference texts among the multiple reference texts conflict with respect to the same fact, a fourth reference text with the latest publication time is obtained from the third reference texts, and a sample question is answered based on the fourth reference text, the publication time of the fourth reference text, the fifth reference text, and the publication time of the fifth reference text, wherein the fifth reference text is a reference text other than the third reference text among the multiple reference texts;

[0177] In the case that there is a sixth reference text among multiple reference texts that does not include the occurrence time of an event, answer the sample question based on the publication time of the sixth reference text, the event in the sixth reference text, the event in the seventh reference text and the corresponding occurrence time, wherein the seventh reference text is a reference text among the multiple reference texts other than the third reference text.

[0178] It should be noted that the aforementioned explanation of the embodiment of the method for determining sample data based on a large model is also applicable to the device for determining sample data based on a large model of this embodiment, and will not be repeated here.

[0179] The sample data determination device based on the big model of the disclosed embodiment obtains sample questions, multiple reference texts of the sample questions and answer generation rules under the temporal reasoning task; determines the first answer to the sample question based on the big model, the answer generation rules, the multiple reference texts and the first publication time of each of the multiple reference texts; and generates a set of sample answer pairs for training the temporal reasoning model based on the sample question and the first answer. Thus, by combining the big model, the sample question, the answer generation rules and the multiple reference texts, the answer to the sample question can be accurately obtained, so that the answer to the sample question does not need to be determined manually, which reduces the cost of model training.

[0180] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with the relevant laws and regulations, and do not violate public order and good morals.

[0181] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0182] Figure 7It is a block diagram of an electronic device for implementing the sample data determination method based on a large model of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0183] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0184] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0185] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a sample data determination method based on a large model. For example, in some embodiments, the sample data determination method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the sample data determination method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the large model-based sample data determination method in any other appropriate manner (eg, by means of firmware).

[0186] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0188] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0190] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0191] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0192] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0193] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining sample data based on a large model, comprising: Obtaining a sample question under a temporal reasoning task, a plurality of reference texts for the sample question, and an answer generation rule; Determine a first answer to the sample question according to the macro model, the answer generation rule, the plurality of reference texts, and the first publishing time of each of the plurality of reference texts; Based on the sample question and the first answer, a set of sample answer pairs for training a temporal reasoning model is generated.

2. The method according to claim 1, wherein: The method further comprises: Modifying the first publishing time of each of the plurality of reference texts to obtain a second publishing time corresponding to each of the reference texts, wherein the first publishing time and the second publishing time corresponding to each of the reference texts are different; Determining a second answer to the sample question according to the large model, the multiple reference texts, the second publishing time of each of the multiple reference texts, and the answer generation rule; The step of generating a set of sample answer pairs for training a temporal reasoning model according to the sample question and the first answer includes: Based on the sample question, the first answer and the second answer, a set of sample answer pairs for training a temporal reasoning model is generated.

3. The method according to claim 2, wherein: The step of modifying the first publishing time of each of the plurality of reference texts to obtain a second publishing time corresponding to each of the reference texts includes: The first publishing time of each of the multiple reference texts is randomly shuffled to obtain a second publishing time corresponding to each of the reference texts.

4. The method according to claim 2, wherein: The step of modifying the first publishing time of each of the plurality of reference texts to obtain a second publishing time corresponding to each of the reference texts includes: Determining the earliest publication time and the latest publication time from the first publication time of each of the plurality of reference texts; For each reference text, a time is randomly selected between the earliest publication time and the latest publication time as the second publication time corresponding to the reference text.

5. The method according to claim 1, wherein: Get sample problems under the temporal reasoning task, including: Obtaining definition information of multiple types of temporal reasoning problems under the temporal reasoning task; Obtaining an application scenario of the temporal reasoning task; Generate a plurality of first problems in the application scenario according to the large model, the application scenario and the definition information, wherein a category of each of the first problems corresponds to a category in the plurality of categories of temporal reasoning problems; According to the first question, the sample question is determined.

6. The method according to claim 5, wherein: Determining the sample question according to the first question includes: In the case where it is determined that the first question includes a first time keyword, rewriting the first time keyword in the first question to obtain a second question, wherein the second time keyword in the second question is different from the first time keyword; The first question and the second question are respectively used as the sample questions.

7. The method according to claim 5, wherein: The method further comprises: Obtain example problems of various types of temporal reasoning problems in the application scenarios; The step of generating a plurality of first questions in the application scenario according to the large model, the application scenario and the definition information includes: A plurality of first questions in the application scenario are generated according to the large model, the example questions, the application scenario and the definition information.

8. The method according to claim 5, wherein: The multiple types of time reasoning problems include at least two of the following: absolute time problem, time error problem, time missing problem, relative time problem, time calculation problem, and time fusion problem.

9. The method according to claim 1, wherein: Determining the first answer to the sample question according to the macro model, the answer generation rule, the plurality of reference texts, and the first publishing time of each of the plurality of reference texts includes: Generate a first prompt word according to the sample question, the answer generation rule, the multiple reference texts, and the first publication time corresponding to each of the multiple reference texts, wherein the first prompt word is used to indicate that an answer to the sample question is generated based on the answer generation rule by using the multiple reference texts and the first publication time corresponding to each of the multiple reference texts; The first prompt word is input into the large model to obtain a first answer to the sample question.

10. The method according to claim 1, wherein: The answer generation rule includes at least one of the following: In the case where there is a first reference text with factual conflict among the multiple reference texts, answer the sample question according to a second reference text among the multiple reference texts and the publication time corresponding to each of the second reference texts, wherein the second reference text is a reference text among the multiple reference texts except the first reference text; In the case that there is no reference text with factual conflict among the multiple reference texts, answering the sample question according to the multiple reference texts and the publication time of each of the multiple reference texts; In the event of factual conflicts in each of the reference texts, answer the sample questions according to the preset answers; In the case that at least two third reference texts among the plurality of reference texts conflict with respect to the same fact, obtaining a fourth reference text with the latest publication time from the third reference texts, and answering the sample question according to the fourth reference text, the publication time of the fourth reference text, a fifth reference text and the publication time of the fifth reference text, wherein the fifth reference text is a reference text among the plurality of reference texts except the third reference text; In the case that there is a sixth reference text among the multiple reference texts that does not include the occurrence time of an event, the sample question is answered based on the publication time of the sixth reference text, the event in the sixth reference text, the event in the seventh reference text and the corresponding occurrence time, wherein the seventh reference text is a reference text among the multiple reference texts other than the third reference text.

11. A sample data determination device based on a large model, comprising: An acquisition module, used to acquire sample questions under the temporal reasoning task, multiple reference texts of the sample questions, and answer generation rules; A first determination module, configured to determine a first answer to the sample question according to the macro model, the answer generation rule, the plurality of reference texts, and the first publication time of each of the plurality of reference texts; A generation module is used to generate a set of sample answer pairs for training a temporal reasoning model based on the sample question and the first answer.

12. The device according to claim 11, wherein The device also includes: A modification module, configured to modify the first publication time of each of the plurality of reference texts to obtain a second publication time corresponding to each of the reference texts, wherein the first publication time and the second publication time corresponding to each of the reference texts are different; A second determination module, configured to determine a second answer to the sample question according to the large model, the multiple reference texts, the second publishing time of each of the multiple reference texts, and the answer generation rule; Wherein, the generation module is specifically used for: Based on the sample question, the first answer and the second answer, a set of sample answer pairs for training a temporal reasoning model is generated.

13. The device according to claim 12, wherein: The modification module is specifically used for: The first publishing time of each of the multiple reference texts is randomly shuffled to obtain a second publishing time corresponding to each of the reference texts.

14. The device according to claim 12, wherein: The modification module is specifically used for: Determining the earliest publication time and the latest publication time from the first publication time of each of the plurality of reference texts; For each reference text, a time is randomly selected between the earliest publication time and the latest publication time as the second publication time corresponding to the reference text.

15. The device according to claim 11, wherein The acquisition module comprises: A first acquisition unit, configured to acquire definition information of multiple types of temporal reasoning problems under the temporal reasoning task; A second acquisition unit, used to acquire an application scenario of the temporal reasoning task; A generating unit, configured to generate a plurality of first problems in the application scenario according to the large model, the application scenario and the definition information, wherein a category of each of the first problems corresponds to a category of the plurality of temporal reasoning problems; A determining unit is used to determine the sample question according to the first question.

16. The device according to claim 15, wherein: The determining unit is specifically configured to: In the case where it is determined that the first question includes a first time keyword, rewriting the first time keyword in the first question to obtain a second question, wherein the second time keyword in the second question is different from the first time keyword; The first question and the second question are respectively used as the sample questions.

17. The device according to claim 15, wherein: The acquisition module further includes: A third acquisition unit is used to acquire example problems of various types of time reasoning problems in the application scenario; Wherein, the generating unit is specifically used for: A plurality of first questions in the application scenario are generated according to the large model, the example questions, the application scenario and the definition information.

18. The device according to claim 15, wherein: The multiple types of time reasoning problems include at least two of the following: absolute time problem, time error problem, time missing problem, relative time problem, time calculation problem, and time fusion problem.

19. The device according to claim 11, wherein: The first determining module is specifically configured to: Generate a first prompt word according to the sample question, the answer generation rule, the multiple reference texts, and the first publication time corresponding to each of the multiple reference texts, wherein the first prompt word is used to indicate that an answer to the sample question is generated based on the answer generation rule by using the multiple reference texts and the first publication time corresponding to each of the multiple reference texts; The first prompt word is input into the large model to obtain a first answer to the sample question.

20. The device according to claim 11, wherein The answer generation rule includes at least one of the following: In the case where there is a first reference text with factual conflict among the multiple reference texts, answer the sample question according to a second reference text among the multiple reference texts and the publication time corresponding to each of the second reference texts, wherein the second reference text is a reference text among the multiple reference texts except the first reference text; In the case that there is no reference text with factual conflict among the multiple reference texts, answering the sample question according to the multiple reference texts and the publication time of each of the multiple reference texts; In the event of factual conflicts in each of the reference texts, answer the sample questions according to the preset answers; In the case that at least two third reference texts among the plurality of reference texts conflict with respect to the same fact, obtaining a fourth reference text with the latest publication time from the third reference texts, and answering the sample question according to the fourth reference text, the publication time of the fourth reference text, a fifth reference text and the publication time of the fifth reference text, wherein the fifth reference text is a reference text among the plurality of reference texts except the third reference text; In the case that there is a sixth reference text among the multiple reference texts that does not include the occurrence time of an event, the sample question is answered based on the publication time of the sixth reference text, the event in the sixth reference text, the event in the seventh reference text and the corresponding occurrence time, wherein the seventh reference text is a reference text among the multiple reference texts other than the third reference text.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images