Long sequence data construction method and device

Abstract

A long sequence data construction method comprises: extracting K topics from N documents, N>=1, K>=1, wherein a topic is used to indicate a field related to at least one document; generating a first question related to a first topic in the K topics based on a first context and the first topic, wherein the first context includes a document related to the first topic in the N documents; generating a first answer related to the first question and a first derivation process of the first answer based on the first question and the first context, wherein the first derivation process explicitly paraphrases first content related to the first question in the first context; and concatenating the first question, the first context, the first derivation process and the first answer in sequence to obtain first long sequence data, which is used for supervised training of a neural network model. The long sequence data constructed by the method can improve the model training efficiency and accuracy, and can enhance the model processing of complex problems.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for constructing long sequence data. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, Large Language Models (LLMs) have played an increasingly important role in the field of natural language processing. Through pre-training on massive amounts of text data, LLMs possess powerful text generation and understanding capabilities, achieving remarkable results in tasks such as machine translation, text summarization, and dialogue systems. However, with the continuous improvement of application demands, expectations for LLMs are no longer limited to short text processing; instead, they are expected to handle long texts and perform complex logical reasoning. This capability is crucial for achieving deep text understanding, human-computer interaction, and knowledge discovery. For example, in scenarios such as legal document analysis, academic paper reviews, and novel comprehension, LLMs need to extract relevant information from large amounts of text and perform multi-step logical reasoning to complete specific tasks.

[0003] Current LLM models primarily demonstrate information retrieval and basic summarization capabilities when processing long texts. While LLMs can extract key information from long texts to some extent, they often struggle to effectively integrate information and perform in-depth analysis in tasks involving complex logical reasoning, multi-step reasoning, and cross-paragraph connections. For example, in extremely long texts, LLMs need to quickly and accurately locate task-relevant information and analyze, synthesize, and reason to arrive at the correct conclusions. However, limited by deficiencies in model memory and attention mechanisms, weak logical reasoning abilities, and a disconnect between information retrieval and reasoning, LLMs have significant shortcomings in integrating information retrieval and logical reasoning, making it difficult to simultaneously meet the requirements of efficient retrieval and deep reasoning. This limits their performance in long text reasoning tasks. Summary of the Invention

[0004] This application provides a method, apparatus, computing device cluster, computer-readable storage medium, and computer program product containing instructions for constructing long sequence data, which can construct long sequence data that can improve the ability of neural network models to integrate information retrieval and logical reasoning.

[0005] In a first aspect, this application provides a method for constructing long sequence data, comprising: extracting K topics from N documents, where N≥1 and K≥1, wherein a topic is used to indicate a domain involved in at least one document; generating a first question related to the first topic based on a first context and the first topic among the K topics, wherein the first context includes documents related to the first topic among the N documents; generating a first answer and a first derivation process of the first answer related to the first question based on the first question and the first context, wherein the first derivation process explicitly restates the first content related to the first question in the first context; and concatenating the first question, the first context, the first derivation process, and the first answer in sequence to obtain a first long sequence data, wherein the first long sequence data is used for supervised training of a neural network model.

[0006] Thus, in the first long sequence of data, since the first derivation process explicitly restates the content related to the first question in the first context, and this content is located at the end of the input of the model to be trained subsequently, training the model using this first long sequence of data aligns with the model's preference for short-distance dependencies. This allows the model to focus more on these dependencies, shortening the backtracking distance required for the model to answer, thereby improving training efficiency and accuracy. Furthermore, since the first long sequence of data contains the derivation process of the answer, the model can refer to this process during training, enabling it to perform multi-step logical reasoning based on this information. This cultivates the model's reasoning chain ability and enhances its ability to handle complex problems.

[0007] In one possible implementation, before extracting K topics from N documents, the process includes: filtering the knowledge base based on a preset character count range to obtain N documents, where the character count in each of the N documents falls within the preset character count range. This ensures that the length of the generated long sequence data is suitable for long text training tasks.

[0008] In one possible implementation, before generating the first question related to the first topic based on the first context and the first topic among K topics, the method further includes: randomly sampling from an irrelevant dataset to obtain irrelevant documents unrelated to the first topic; concatenating the irrelevant documents with the documents related to the first topic among N documents to obtain the first context. In this way, the documents related to the first topic in the first context provide the core information needed for the first question, while the irrelevant documents increase the difficulty and diversity of the question, requiring the model to perform more complex reasoning and judgment when answering the question. This allows the model trained using the constructed long sequence data to enhance its ability to retrieve question-related information.

[0009] In one possible implementation, before generating the first answer and the first derivation process of the first answer based on the first question and the first context, the method further includes: performing a search based on the first question and the first context to obtain first content; generating the first answer and the first derivation process of the first answer based on the first question and the first context, including: generating a first prompt word based on the first question, the first context, and the first content, wherein the first prompt word is used to guide the large language model to generate the first answer and explicitly restate the first content in the answer; and inputting the first prompt word into the large language model to obtain the first answer and the first derivation process. This first prompt word helps the large language model better understand the background and context of the question, thereby enabling more effective reasoning and judgment, and improving the accuracy and relevance of the answer.

[0010] In one possible implementation, before sequentially concatenating the first question, first context, first derivation process, and first answer, the method further includes ensuring that the quality of the first derivation process and the first answer meets the requirements. This can improve the accuracy of the generated long sequence data and enhance the precision of subsequent model training.

[0011] In one possible implementation, the first problem relates to multiple documents out of N documents.

[0012] In one possible implementation, the method further includes training the neural network model using a first long sequence of data.

[0013] Secondly, this application provides a long sequence data construction apparatus, comprising: a topic sampling module, configured to extract K topics from N documents, where N≥1 and K≥1, wherein a topic is used to indicate a domain involved in at least one document; a question generation module, configured to generate a first question related to the first topic based on a first context and a first topic among the K topics, wherein the first context includes documents related to the first topic among the N documents; an answer generation module, configured to generate a first answer and a first derivation process of the first answer related to the first question based on the first question and the first context, wherein the first derivation process explicitly restates the first content related to the first question in the first context; and a long sequence data generation module, configured to concatenate the first question, the first context, the first derivation process, and the first answer in sequence to obtain a first long sequence data, wherein the first long sequence data is used for supervised training of a neural network model.

[0014] In one possible implementation, before extracting K topics from N documents, the topic sampling module is also used to: filter the knowledge base based on a preset character count range to obtain N documents, wherein the number of characters in each of the N documents is within the preset character count range.

[0015] In one possible implementation, before generating a first question related to the first topic based on the first context and the first topic among K topics, the topic sampling module is also used to: perform random sampling from the irrelevant dataset to obtain irrelevant documents unrelated to the first topic; and concatenate the irrelevant documents with the documents related to the first topic among N documents to obtain the first context.

[0016] In one possible implementation, before the answer generation module generates the first answer and the first derivation process of the first answer based on the first question and the first context, the question generation module is further configured to: perform a search based on the first question and the first context to obtain the first content. At this point, when the answer generation module generates the first answer and the first derivation process of the first answer based on the first question and the first context, it specifically performs the following: generates a first prompt word based on the first question, the first context, and the first content, wherein the first prompt word is used to guide the large language model to generate the first answer and explicitly restate the first content in the answer; and inputs the first prompt word into the large language model to obtain the first answer and the first derivation process.

[0017] In one possible implementation, before the long sequence data generation module concatenates the first question, the first context, the first derivation process, and the first answer in sequence, the answer generation module is also used to: determine whether the quality of the first derivation process and the first answer meets the requirements.

[0018] In one possible implementation, the first problem relates to multiple documents out of N documents.

[0019] One possible implementation also includes a training module for training the neural network model using the first long sequence of data.

[0020] Thirdly, embodiments of this application provide a long sequence data construction apparatus, including at least one processor and an interface; the at least one processor obtains program instructions through the interface; the at least one processor is used to execute program line instructions to implement the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, this long sequence data construction apparatus may be, but is not limited to, a chip.

[0021] Fourthly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0022] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the computing device cluster may include one or more computing devices.

[0023] Sixthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in the first aspect or any possible implementation thereof. Exemplarily, the cluster of computing devices may include one or more computing devices.

[0024] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating a technical concept for constructing long sequence data according to an embodiment of this application;

[0026] Figure 2 This is a schematic diagram illustrating a process for constructing long sequence data according to an embodiment of this application;

[0027] Figure 3 This is a flowchart illustrating a method for constructing long sequence data according to an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a long sequence data construction device provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of another long sequence data construction device provided in the embodiments of this application. Detailed Implementation

[0033] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0034] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0035] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0036] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0037] First, some of the technical terms involved in the embodiments of this application will be introduced.

[0038] (1) Prompt

[0039] A prompt is a text instruction used to guide a neural network model in generating specific content. Its core function is to provide the model with context and task objectives. To provide context, a prompt helps the model understand the semantic environment of the input data by including relevant information or background. To specify the task objective, a prompt tells the model the task to be performed through explicit instructions, such as "check for errors in the data and correct them" or "analyze trends in the data." This instruction guides the model to think and generate content in a specific direction.

[0040] (2) Large Language Model

[0041] Large language models are artificial intelligence models based on deep learning techniques. They are trained on large-scale text datasets to learn the syntax, semantics, and pragmatics of a language. These models have numerous parameters and can generate high-quality language output from input text, such as answering questions, translating text, and creating stories, among other natural language processing tasks.

[0042] Next, the solutions provided in the embodiments of this application will be introduced.

[0043] For example, to enhance the long-text reasoning capabilities of LLM, a specialized long-sequence dataset can be constructed during the LLM training phase, and this data can be used to train the LLM. Through this training method based on long-sequence data, LLM can better understand the complex structures and logical relationships within long texts, thereby significantly improving its performance in long-text reasoning tasks. For instance, when handling complex tasks such as legal document analysis and academic paper reviews, it can more accurately extract key information and perform multi-step reasoning.

[0044] When constructing long-sequence data, similarity-based concatenation and data synthesis methods can be used. The similarity-based concatenation method mainly involves two steps: first, by calculating the similarity between documents, a relationship graph is constructed based on the similarity values; then, by traversing the relationship graph, more "similar" documents are concatenated into a long-sequence data of the target length (e.g., 32k). This method essentially reorganizes the data, hoping to make LLM (Limited Language Management) more likely to learn the "common" semantics of the data by concatenating similar text into the same context. However, it has two limitations: first, the "similarity" calculation is highly dependent on the document embedding representation, and the accuracy of this calculation is low in massive and diverse data scenarios; second, this method cannot enhance the reasoning ability of the data itself, and if the reasoning knowledge of the original data is weak, the potential for improving the reasoning ability of LLM is limited.

[0045] The data synthesis-based approach primarily comprises two steps: designing task prompts and generating long-text questions. In the task prompt design step, users can design different types of task prompts (such as "explain," "compare," and "predict"). These prompts act as "instructions" to the LLM, guiding it to generate diverse long-text content. For example, if the task is "explain," the LLM might generate a detailed explanation of a complex concept; if it's "compare," it might generate text comparing different things. In the long-text question generation step, based on these task prompts, the LLM generates a series of questions covering multiple parts of the long text. For instance, for a long text about climate change, the LLM might generate questions like: "What are the main causes of climate change?", "What are the impacts of climate change on ecosystems?", and "How can we address climate change?" These questions not only cover different parts of the text but also guide the LLM to generate more comprehensive and coherent long-text content. The core of this approach is leveraging the powerful language generation capabilities of the LLM to generate instruction data related to long texts through diverse task prompts. It's like assigning the model different "homework," allowing it to learn to handle more complex long-text content while completing these tasks. This approach helps models better understand the structure and logic of long texts, thereby improving their performance in long text reasoning tasks. However, this method has limitations: first, it generates data based on only a single long text, limiting the input length of long sequence data; second, because it only involves a single document, the generated data cannot enhance the model's ability to reason across documents in long texts.

[0046] In view of this, embodiments of this application provide a long-sequence data construction scheme, aiming to improve the performance of LLM in handling long text reasoning tasks, especially in the retrieval of relevant information (i.e., information fragments or data that directly affect the reasoning task) and complex multi-step logical reasoning. Since the long-text reasoning ability of LLM is closely related to the position of relevant information in the document, especially when the relevant information is at the end of the LLM input, the reasoning performance of LLM is usually better. This is because when generating output, LLM tends to pay more attention to the most recent contextual information, and information at the end is more easily included in the current reasoning scope. For example, in a long text reasoning task, if key clues or facts are located at the end of the text, the model can more efficiently utilize this information for reasoning. Furthermore, in multi-step reasoning tasks, the reasoning ability of LLM is also affected by the location of relevant information. If LLM can accurately locate this relevant information and integrate it into the reasoning process, its reasoning efficiency and accuracy will be significantly improved. For example, through conscious guidance, LLM can more effectively plan reasoning steps, thereby better handling complex multi-step reasoning tasks. Therefore, by enhancing the retrieval capabilities of LLMs, allowing them to explicitly restate relevant information from the context, and thus moving this information to the end of the context, the multi-step reasoning ability of LLMs can be improved. The long sequence data construction scheme provided in this application is based on this concept.

[0047] For example, Figure 1 This illustration shows a schematic diagram of a technical concept for constructing long sequence data according to an embodiment of this application. Figure 1 As shown, the overall architecture for constructing long-sequence data includes: a topic sampling part, a question generation part, an answer generation part, and a long-sequence data generation part. The topic sampling part is primarily responsible for filtering documents from the existing knowledge base and extracting a large subset of topics from these documents to obtain the topics themselves, as well as documents related to and unrelated to those topics. For example, as... Figure 2 As shown, in this section, in S21, documents are first filtered from the knowledge base, ensuring that the document length is suitable for long text reasoning tasks, thus selecting documents of appropriate length. These filtered documents will serve as reading content for reading comprehension. In S22, a large subset of topics is randomly selected from the filtered documents. These filtered documents can then be used as contextual input to the LLM for processing.

[0048] The question generation section is primarily responsible for receiving sampled documents, generating a context for input to the LLM based on these documents, and constructing a question prompt to guide the LLM in generating topic-related questions. Additionally, this section is also responsible for finding question-related content from the context. For example, such as... Figure 2As shown, in this section, in S23, for each topic in the topic subset, documents related to that topic, as well as irrelevant documents randomly sampled from irrelevant datasets, can be concatenated as context input to the LLM. Simultaneously, a question prompt can be constructed and input into the LLM. This question prompt guides the LLM to generate questions related to the topic. In S24, a prompt is designed to extract question-related content, guiding the LLM to find relevant content from the context and organizing the extracted information according to a predetermined structure.

[0049] The answer generation section is primarily responsible for guiding the LLM (Local Management Module) to explicitly restate question-related content in the response based on the context and the generated question, in order to generate the final answer. For example, such as... Figure 2 As shown, in this section, S25, a prompt for generating the answer can be designed first. By designing the prompt, the LLM explicitly restates the content related to the question in the answer, so as to move important information to the end of the LLM input, which is in line with the model's preference for short-distance dependencies. For example, the LLM can provide a detailed analysis and reasoning process in the answer, and explicitly restate the content related to the question in the given reasoning process. S26, the generated answer is checked to ensure that its logic, accuracy, and format meet the requirements. Here, the answer generated in this part contains both the answer to the question and the reasoning process of the answer.

[0050] The long sequence data generation section is primarily responsible for concatenating the aforementioned context, specific question, answer to the specific question, and reasoning process of the answer according to a specific format to obtain the final long sequence data. For example, such as... Figure 2 As shown, in this section, at S27, the questions obtained above, the documents related to the questions, the randomly sampled irrelevant documents, and the answers generated in the answer generation section can be organized into long sequence data. This long sequence data can be used as long text inference data for supervised fine-tuning of neural network models.

[0051] As can be seen from the above concept of constructing long-sequence data, the final generated long-sequence data includes documents unrelated to the question. Therefore, when using this long-sequence data to train a neural network model, the complexity and diversity of the questions can be increased, requiring the neural network model to perform more complex reasoning and judgment when answering the questions, thereby enhancing the retrieval ability of the neural network model. In addition, the final generated long-sequence data also contains the reasoning process for the answer, and the reasoning process explicitly restates the content related to the question. This part of the content is located at the end of the neural network model's input, and this part of the content is highly related to the answer to the question. This allows the neural network model to pay more attention to this content when generating the output, which is in line with the model's preference for short-distance dependencies and reduces the difficulty of model training.

[0052] The following describes the specific implementation process of the above technical concept.

[0053] For example, Figure 3 This illustration shows a flowchart of a long-sequence data construction method provided in an embodiment of this application. It is understood that this method can be executed by any device, apparatus, platform, or cluster of devices with computing and processing capabilities. For example, the method can be executed by a long-sequence data construction apparatus, which can be implemented by software and / or hardware, and can be, but is not limited to, configured in an electronic device or server. Figure 3 As shown, this long sequence data construction method may include the following steps:

[0054] S301. Extract K topics from N documents, where N≥1 and K≥1, where a topic is used to indicate an area covered by at least one document.

[0055] In this embodiment, the N documents can be, but are not limited to, randomly selected from the knowledge base. During the selection process, to ensure that the text length is suitable for long text reasoning tasks, a range of character counts in the required documents can be preset (e.g., 8000 to 16000 characters). Then, based on the preset character count range, the knowledge base is filtered to obtain N documents. In this way, the character counts in the selected documents all meet the requirements, i.e., they are all within the preset character count range.

[0056] Furthermore, K topics can be randomly selected from the N selected documents. In this embodiment, a topic can be used to indicate a field covered by a document. For example, the documents can be processed using text mining and natural language processing techniques to extract topics from them. In some embodiments, when the document is a paper, each index term in the paper can be considered as a topic.

[0057] S302. Based on the first context and the first topic among K topics, generate a first question related to the first topic, wherein the first context includes documents related to the first topic among N documents.

[0058] In this embodiment, for each topic, a question related to that topic can be generated. For ease of description, the following example illustrates generating a first question related to the first topic among K topics. Documents related to the first topic can be used as the first context, and LLM is used to process this first context and the first question to obtain the first question. For example, a prompt can be constructed to guide LLM in generating questions, ensuring that the first question is closely related to the first topic. For instance, the prompt can instruct LLM to first analyze the first context to find content related to the first question; then, summarize this content; and finally, generate the question from the summarized content. In some embodiments, to increase the complexity and diversity of the first question, irrelevant documents unrelated to the first topic are introduced into the first context. Thus, documents related to the first topic in the first context provide the core information required for the first question, while irrelevant documents increase the difficulty and diversity of the question, requiring LLM to perform more complex reasoning and judgment when answering it. Irrelevant documents can be randomly sampled from an irrelevant dataset to obtain documents unrelated to the first topic. Furthermore, irrelevant documents and documents related to the first topic among N documents can be concatenated to obtain the first context. In some embodiments, to mimic human thinking, the first question can be related to multiple documents among the N documents, i.e., the first question involves multiple documents.

[0059] S303. Based on the first question and the first context, generate the first answer related to the first question and the first derivation process of the first answer, wherein the first derivation process explicitly restates the first content related to the first question in the first context.

[0060] In this embodiment, a first question and a first context can be input into an LLM (Limited Language Management) system to generate a first answer and a first derivation process related to the first question. For example, the first question and the first context can be embedded in a prompt template to obtain a prompt. Then, this prompt can be input into the LLM system to obtain a first answer and a first reasoning process. The prompt generated in this step can guide the LLM system to explicitly restate the first content related to the first question in the first context and generate the first answer to the first question in its response. For example, the prompt can guide the LLM system to first analyze the first context to find content related to the first question; then summarize this content; and finally, generate a reasoning process and a corresponding answer based on the summarized content in a predetermined format. In some embodiments, to ensure that the LLM system can accurately reference key content in a document when answering a question, the first content can be extracted from the first context by the LLM system, and a prompt can be generated using the first question, the first context, and the first content. In this way, the prompt helps the LLM model better understand the background and context of the problem, enabling more effective reasoning and judgment, thereby improving the accuracy and relevance of the answer. For example, to ensure that the LLM-generated answer meets requirements in terms of logic, accuracy, and format, the quality of the first answer and the first derivation process can be checked. For instance, the prompt mechanism can be continued, and LLM can be used to perform quality checks on the first answer and the first derivation process.

[0061] S304. The first question, the first context, the first derivation process, and the first answer are concatenated in sequence to obtain the first long sequence data, which is used for supervised training of the neural network model.

[0062] In this embodiment, the first question, the first context, the first derivation process, and the first answer can be concatenated sequentially to obtain the first long sequence data. Furthermore, this first long sequence data can be used to perform supervised training on a neural network model. For example, the format of the first long sequence data can conform to the input and output of the neural network model during supervised training, in order to facilitate model training.

[0063] Thus, in the first long sequence of data, since the first derivation process explicitly restates the content related to the first question in the first context, and this content is located at the end of the input of the model to be trained subsequently, training the model using this first long sequence of data aligns with the model's preference for short-distance dependencies. This allows the model to focus more on these dependencies, shortening the backtracking distance required for the model to answer, thereby improving training efficiency and accuracy. Furthermore, since the first long sequence of data contains the derivation process of the answer, the model can refer to this process during training, enabling it to perform multi-step logical reasoning based on this information. This cultivates the model's reasoning chain ability and enhances its ability to handle complex problems.

[0064] To facilitate Li Jiao'er, the following section uses the generation of long sequence data from the Arxiv paper dataset as an example, combined with web page data as an irrelevant reference dataset, to describe a specific implementation process of this technical solution.

[0065] Step 1: Select documents of appropriate length from Arxiv. To ensure the text length is suitable for long text inference tasks, this length is set to 8,000 to 16,000 characters in this embodiment. These documents will serve as the reading content for the reading comprehension exercise.

[0066] Step 2: Randomly select a large number of topic subsets from the documents filtered in the previous step. In this embodiment, each keyword in the paper is considered a topic subset, meaning a paper can belong to multiple topic subsets. Each topic must contain one or more related documents.

[0067] Step 3: Generate a question for each topic. The relevant documents for that topic, along with other randomly sampled and concatenated additional irrelevant data, are used as the text content for the LLM input. In this step, the question prompt can be generated from a pre-set source or by the LLM itself, aiming to ensure the question is relevant to the topic. This approach ensures the generated questions are closely related to the topic while introducing irrelevant information to increase the complexity and diversity of the questions. Specifically, relevant documents provide the core information needed for the question, while irrelevant information increases the difficulty and diversity of the question, requiring the model to perform more complex reasoning and judgment when answering it.

[0068] For example, the generated Prompt for the question could be:

[0069] "{content}\n Summarize the above paper.Your answer should start with\"The paper titled{title}is mainly about...\"."

[0070] Here, "content" represents the contextual text pieced together for each topic. "title" refers to the title of the currently referred to paper.

[0071] Input the above Prompt into an existing LLM and let it generate questions involving multiple articles.

[0072] Step 4: Based on the question, design a prompt to extract relevant information, guiding the LLM to find question-related content from the document. By extracting information directly related to the question, the LLM can accurately reference key content from the document when answering the question, thereby improving the accuracy and relevance of the answer. Furthermore, information extraction can help the model better understand the background and context of the question, enabling more effective reasoning and judgment.

[0073] For example, the prompt could be:

[0074] "This is the{paper}that is to be analyzed.This is the{sentence}thatcites that paper.Find out all relevant parts in that paper."

[0075] This prompt can effectively guide the model to retrieve relevant content related to the question.

[0076] The next step is to organize the extracted information and structure it according to a predetermined format.

[0077] Step 5: Based on the previously generated questions and extracted relevant information, design an answer generation prompt. The prompt should not only include the questions and information, but also guide the LLM to restate relevant information in the answer, so as to move important information to the end of the input. When using this generated data to train the model, it conforms to the characteristic that the model is more friendly to short-distance dependencies, and achieves better training results.

[0078] For example, the prompt could be:

[0079] "You are analyzing why a certain sentence in a paper cited another paper.\nThis is the sentence:{original_sentence}\nThis is the name of the cited paper:{original_paper_title}.\nThis is the summary of the cited paper:{cited_paper_summary}\n\nFormat your analysis in the format of\"In the original paper,we can see the sentence{{sentence in original paper}},and thepaper{{cited paper title}}mainly talks about{{summarize the cited paperbriefly}},therefore{{why the original sentence would cite the citedpaper}}.\"\n\nDO NOT answer anything other than specified by the format."

[0080] The basic structure consists of restating the problem and related information, and finally guiding the model to restate the information and output in a formatted manner.

[0081] Input the prompt into an existing LLM and let it generate an answer.

[0082] Step 6: Check the generated answers to ensure that they meet the requirements in terms of logic, accuracy, and format.

[0083] Step 7: Organize the obtained questions, irrelevant information, relevant information, and answers into long-sequence inference data for supervised fine-tuning of the neural network model. In this embodiment, it consists of topic-related papers, topic-irrelevant papers, questions generated by the large model, and the content of the answers.

[0084] For example:{

[0085] "text":{given papers and irrelevant web text},

[0086] "input":"There are{paper_num}papers in the above{text},titled{paper_titles}respectively.Which of the papers in{paper_choice_titles}may the paper{original_paper_title}have cited?",

[0087] "output":"To assess whether the papers in{paper_choice_titles}mayhave been cited by{original_paper_title},we should summarize the respective papers first.\n\nThen according to the summaries,we can assess whether the papers were cited based on their relevance to paper{original_paper_title}.\n{summaries_and_reasons}\nThe rest of the papers provided do not show strong correlation to the paper in question,and therefore may have not been cited.\n\nIn conclusion, the paper{original_paper_title}may have cited papers{cited_paper_titles}."

[0088] As can be seen from the above description, the long sequence data construction scheme provided in this application can construct long sequence data across multiple related or unrelated documents in a specific format. Furthermore, this scheme can explicitly strengthen the inference part through steps such as summarizing or restating. In addition, the long sequence data constructed in this way can enhance the model training effect compared to general formats.

[0089] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments and features described in the above embodiments can be combined according to actual conditions, and the combined solutions are still within the protection scope of this application.

[0090] Next, based on the methods in the above embodiments, a long sequence data construction apparatus provided in this application will be introduced.

[0091] For example, Figure 4 A schematic diagram of a long sequence data construction apparatus provided in an embodiment of this application is shown. Figure 4 As shown, the long sequence data construction device 400 includes: a topic sampling module 401, a question generation module 402, an answer generation module 403, and a long sequence data generation module 404. The topic sampling module 401 is used to extract K topics from N documents, where N≥1 and K≥1, and each topic indicates a domain covered by at least one document. The question generation module 402 is used to generate a first question related to the first topic based on a first context and the first topic among the K topics, where the first context includes documents related to the first topic from the N documents. The answer generation module 403 is used to generate a first answer and a first derivation process for the first answer based on the first question and the first context, where the first derivation process explicitly restates the first content related to the first question in the first context. The long sequence data generation module 404 is used to sequentially concatenate the first question, the first context, the first derivation process, and the first answer to obtain first long sequence data, which is used for supervised training of a neural network model.

[0092] In some embodiments, before extracting K topics from N documents, the topic sampling module 401 is further configured to: filter the knowledge base based on a preset character count range to obtain N documents, wherein the number of characters in each of the N documents is within the preset character count range.

[0093] In some embodiments, before generating a first question related to the first topic based on the first context and the first topic among K topics, the topic sampling module 401 is further configured to: perform random sampling from the irrelevant dataset to obtain irrelevant documents unrelated to the first topic; and concatenate the irrelevant documents with the documents related to the first topic among N documents to obtain the first context.

[0094] In some embodiments, before the answer generation module 403 generates a first answer and a first derivation process related to the first question based on the first question and the first context, the question generation module 402 is further configured to: perform a search in the first context based on the first question to obtain the first content. At this time, when the answer generation module 403 generates the first answer and the first derivation process related to the first question based on the first question and the first context, it is specifically configured to: generate a first prompt word based on the first question, the first context, and the first content, wherein the first prompt word is used to guide the large language model to generate the first answer and explicitly restate the first content in the answer; input the first prompt word into the large language model to obtain the first answer and the first derivation process.

[0095] In some embodiments, before the long sequence data generation module 404 concatenates the first question, the first context, the first derivation process, and the first answer in sequence, the answer generation module 403 is further configured to: determine that the quality of the first derivation process and the first answer meets the requirements.

[0096] In some embodiments, the first problem relates to multiple documents out of N documents.

[0097] In some embodiments, the system further includes a training module (not shown) for training a neural network model using a first long sequence of data.

[0098] In some embodiments, Figure 4 The topic sampling module 401, question generation module 402, answer generation module 403, and long sequence data generation module 404 shown can all be implemented in software or hardware. For example, the implementation of the topic sampling module 401 will be described below. Similarly, the implementation of the question generation module 402, answer generation module 403, and long sequence data generation module 404 can refer to the implementation of the topic sampling module 401.

[0099] As an example of a software functional unit, the topic sampling module 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the topic sampling module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0100] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0101] As an example of a hardware functional unit, the topic sampling module 401 may include at least one computing device, such as a server. Alternatively, the topic sampling module 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0102] The multiple computing devices included in the topic sampling module 401 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the topic sampling module 401 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the topic sampling module 401 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0103] It should be noted that, in other embodiments, the topic sampling module 401 can be used to execute any step in the long sequence data construction method described in the above embodiments, and any one or more of the question generation module 402, answer generation module 403, and long sequence data generation module 404 can also be used to execute any step in the long sequence data construction method described in the above embodiments. Furthermore, the steps implemented by the topic sampling module 401, question generation module 402, answer generation module 403, and long sequence data generation module 404 can also be specified as needed, and different steps in the long sequence data construction method described in the above embodiments can be implemented by the topic sampling module 401, question generation module 402, answer generation module 403, and long sequence data generation module 404 respectively. Figure 4 The long sequence data construction device 400 shown has all the functions.

[0104] This application also provides a computing device 500. For example... Figure 5 As shown, the computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. The processor 504, the memory 506, and the communication interface 508 communicate with each other via the bus 502. The computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 500.

[0105] Bus 502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus 504 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 504 may include a path for transmitting information between various components of the computing device 500 (e.g., memory 506, processor 504, communication interface 508).

[0106] Processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0107] Memory 506 may include volatile memory, such as random access memory (RAM). Processor 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0108] The memory 506 stores executable program code, and the processor 504 executes the executable program code to implement the aforementioned functions respectively. Figure 4 The topic sampling module 401, question generation module 402, answer generation module 403, and long sequence data generation module 404 shown herein perform their functions to implement the long sequence data construction method described in the above embodiments. That is, the memory 506 stores instructions for executing the long sequence data construction method described in the above embodiments.

[0109] Alternatively, the memory 506 stores executable code, and the processor 504 executes the executable code to implement the aforementioned functions respectively. Figure 4 The long sequence data construction apparatus 400 shown in the diagram functions to implement the long sequence data construction method described in the above embodiments. That is, the memory 506 stores instructions for executing the long sequence data construction method described in the above embodiments.

[0110] The communication interface 508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 500 and other devices or communication networks.

[0111] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0112] like Figure 6As shown, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the long sequence data construction method described in the above embodiments.

[0113] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the long sequence data construction method described in the above embodiments. In other words, a combination of one or more computing devices 500 can jointly execute instructions for executing the long sequence data construction method described in the above embodiments.

[0114] It should be noted that the memory 506 in different computing devices 500 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 4 The long sequence data construction device 400 shown contains some of the functions. That is, the instructions stored in the memory 506 in different computing devices 500 can implement the functions of one or more of the modules: topic sampling module 401, question generation module 402, answer generation module 403, and long sequence data generation module 404.

[0115] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 7 One possible implementation is shown. For example... Figure 7 As shown, the two computing devices 500A and 500B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 506 in computing device 500A stores instructions for executing the functions of the topic sampling module 401 and the question generation module 402. Simultaneously, the memory 506 in computing device 500B stores instructions for executing the functions of the answer generation module 403 and the long sequence data generation module 404.

[0116] It should be understood that Figure 7 The functions of computing device 500A shown can also be performed by multiple computing devices 500. Similarly, the functions of computing device 500B can also be performed by multiple computing devices 500.

[0117] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 6 and Figure 7The connection method of the computing device cluster is different in that the memory 506 of one or more computing devices 500 in the computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0118] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the aforementioned long-sequence data construction method. In other words, a combination of one or more computing devices 500 can jointly execute the instructions for executing the aforementioned long-sequence data construction method.

[0119] It should be understood that each step of the above method embodiments can be accomplished by hardware logic circuits or software instructions in a processor.

[0120] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that the computing device can store, or a data storage device such as a data center containing one or more available media. Available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0121] Based on the methods in the above embodiments, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices containing at least one computing device, cause the cluster of computing devices to perform the methods in the above embodiments.

[0122] Based on the methods in the above embodiments, this application also provides a long sequence data construction apparatus. Please refer to... Figure 8 , Figure 8 This is a schematic diagram of another long sequence data construction apparatus provided in an embodiment of this application. Figure 8 As shown, the long-sequence data construction apparatus 800 includes one or more processors 801 and interface circuitry 802. Optionally, the long-sequence data construction apparatus 800 may also include a bus 803. Wherein:

[0123] The processor 801 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the processor 801 or through software instructions. The processor 801 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0124] The interface circuit 802 can be used to send or receive data, instructions or information. The processor 801 can use the data, instructions or other information received by the interface circuit 802 to process the data, instructions or other information, and can send the processed information out through the interface circuit 802.

[0125] Optionally, the long sequence data construction apparatus 800 also includes a memory, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0126] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0127] Optionally, the interface circuit 802 can be used to output the execution results of the processor 801.

[0128] It should be noted that the functions of the processor 801 and the interface circuit 802 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0129] It should be understood that each step of the above method embodiments can be completed by hardware logic circuits or software instructions in a processor.

[0130] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0131] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0132] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0133] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing long sequence data, characterized in that, The method includes: Extract K topics from N documents, where N≥1 and K≥1, where each topic is used to indicate an area covered by at least one of the documents; Based on a first context and a first topic among the K topics, a first question related to the first topic is generated, wherein the first context includes documents related to the first topic among the N documents; Based on the first question and the first context, a first answer related to the first question and a first derivation process of the first answer are generated, wherein the first content related to the first question in the first context is explicitly restated in the first derivation process; The first question, the first context, the first derivation process, and the first answer are concatenated in sequence to obtain a first long sequence of data, wherein the first long sequence of data is used for supervised training of the neural network model.

2. The method according to claim 1, characterized in that, Before extracting K topics from N documents, the process also includes: Based on a preset character count range, the knowledge base is filtered to obtain the N documents, wherein the number of characters in each of the N documents is within the preset character count range.

3. The method according to claim 1 or 2, characterized in that, Before generating a first question related to the first topic based on the first context and the first topic among the K topics, the method further includes: Random sampling is performed from the irrelevant dataset to obtain irrelevant documents that are unrelated to the first topic; The irrelevant documents and the documents related to the first topic among the N documents are concatenated to obtain the first context.

4. The method according to any one of claims 1-3, characterized in that, Before the first derivation process of generating a first answer related to the first question and the first answer based on the first question and the first context, the method further includes: Based on the first question, a search is performed in the first context to obtain the first content; The process of generating a first answer related to the first question and a first derivation process of the first answer based on the first question and the first context includes: Based on the first question, the first context, and the first content, a first prompt word is generated, wherein the first prompt word is used to guide the large language model to generate the first answer and to explicitly restate the first content in the answer; The first prompt word is input into the large language model to obtain the first answer and the first derivation process.

5. The method according to any one of claims 1-4, characterized in that, Before sequentially concatenating the first question, the first context, the first derivation process, and the first answer, the method further includes: The quality of the first derivation process and the first answer is confirmed to meet the requirements.

6. The method according to any one of claims 1-5, characterized in that, The first problem relates to multiple documents among the N documents.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The neural network model is trained using the first long sequence of data.

8. A long sequence data construction apparatus, characterized in that, include: A topic sampling module is used to extract K topics from N documents, where N≥1 and K≥1, wherein one of the topics is used to indicate a field involved in at least one of the documents; The question generation module is used to generate a first question related to the first topic based on a first context and a first topic among the K topics, wherein the first context includes documents related to the first topic among the N documents; The answer generation module is used to generate a first answer related to the first question and a first derivation process of the first answer based on the first question and the first context, wherein the first derivation process explicitly restates the first content related to the first question in the first context; The long sequence data generation module is used to concatenate the first question, the first context, the first derivation process, and the first answer in sequence to obtain the first long sequence data, wherein the first long sequence data is used for supervised training of the neural network model.

9. The apparatus according to claim 8, characterized in that, Before extracting K topics from N documents, the topic sampling module is also used for: Based on a preset character count range, the knowledge base is filtered to obtain the N documents, wherein the number of characters in each of the N documents is within the preset character count range.

10. The apparatus according to claim 8 or 9, characterized in that, Before generating a first question related to the first topic based on the first context and the first topic among the K topics, the topic sampling module is further configured to: Random sampling is performed from the irrelevant dataset to obtain irrelevant documents that are unrelated to the first topic; The irrelevant documents and the documents related to the first topic among the N documents are concatenated to obtain the first context.

11. The apparatus according to any one of claims 8-10, characterized in that, Before the answer generation module generates a first answer related to the first question and a first derivation process of the first answer based on the first question and the first context, the question generation module is further configured to: Based on the first question, a search is performed in the first context to obtain the first content; When the answer generation module generates a first answer related to the first question and a first derivation process of the first answer based on the first question and the first context, it is specifically used for: Based on the first question, the first context, and the first content, a first prompt word is generated, wherein the first prompt word is used to guide the large language model to generate the first answer and to explicitly restate the first content in the answer; The first prompt word is input into the large language model to obtain the first answer and the first derivation process.

12. The apparatus according to any one of claims 8-11, characterized in that, Before the long sequence data generation module concatenates the first question, the first context, the first derivation process, and the first answer in sequence, the answer generation module is further configured to: The quality of the first derivation process and the first answer is confirmed to meet the requirements.

13. The apparatus according to any one of claims 8-12, characterized in that, The first problem relates to multiple documents among the N documents.

14. The apparatus according to any one of claims 8-13, characterized in that, Also includes: The training module is used to train the neural network model using the first long sequence data.

15. A long sequence data construction apparatus, characterized in that, Includes at least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is configured to execute the program line instructions to implement the method as described in any one of claims 1-7.

16. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

17. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-7, wherein the cluster of computing devices includes at least one computing device.

18. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-7, wherein the computing device cluster includes at least one computing device.

Images