Document question answering data generation method, training method, and related products

Abstract

The present disclosure provides a document question and answer data generation method, a training method, a generation device, an electronic device, a computer readable storage medium and a computer program product, which are related to the technical fields of artificial intelligence such as large language models and deep learning. The method comprises: performing page content extraction on a page image in a document to obtain description information corresponding to a seed page in the document; generating a thought reasoning chain corresponding to the seed page by using a preset question and answer data generation model according to the description information, a question definition of a preset question type and a question and answer instance of the preset question type; and in response to the thought reasoning chain constituting a question thought chain corresponding to the preset question type, generating question and answer data corresponding to the preset question type by using the question and answer data generation model based on the question thought chain. The method can improve the generation quality and question complexity of the document question and answer data.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to the fields of artificial intelligence technology such as large language models and deep learning, and particularly to a document question-and-answer data generation method, training method, generation device, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Currently, document-based automated data synthesis processes often employ simple template generation or question reuse methods. This approach results in low-quality, undiverse synthesized data, leading to often simplistic document-based question-and-answer datasets that fail to cover complex question types. Summary of the Invention

[0003] This disclosure provides a document question-and-answer data generation method, training method, generation device, electronic device, computer-readable storage medium, and computer program product.

[0004] In a first aspect, embodiments of this disclosure propose a document question-and-answer data generation method, comprising: extracting page content from page images in a document to obtain descriptive information corresponding to a seed page in the document; generating a thought reasoning chain corresponding to the seed page using a preset question-and-answer data generation model based on the descriptive information, a question definition of a preset question type, and question-and-answer instances of the preset question type; wherein the preset question type includes multi-hop questions and / or set questions; in response to the thought reasoning chain forming a question thought chain corresponding to the preset question type, generating question-and-answer data corresponding to the preset question type based on the question thought chain using the question-and-answer data generation model.

[0005] Secondly, embodiments of this disclosure propose a training method for a question-answering model, comprising: acquiring document question-answering data samples, wherein the document question-answering data samples are constructed using the document question-answering data generation method described in the first aspect above; inputting the document question-answering data samples into a question-answering model, and training the question-answering model.

[0006] Thirdly, this disclosure provides a document question-and-answer data generation device, comprising: a page extraction module for extracting page content from page images in a document to obtain descriptive information corresponding to a seed page in the document; a reasoning module for generating a thought reasoning chain corresponding to the seed page based on the descriptive information, a question definition of a preset question type, and question-and-answer instances of the preset question type using a preset question-and-answer data generation model; wherein the preset question type includes multi-hop questions and / or set questions; and a data generation module for generating question-and-answer data corresponding to the preset question type based on the question-and-answer data generation model in response to the thought reasoning chain forming a question thought chain corresponding to the preset question type.

[0007] Fourthly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the document question-and-answer data generation method as described in the first aspect.

[0008] Fifthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the document question-and-answer data generation method as described in the first aspect.

[0009] In a sixth aspect, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the document question-and-answer data generation method as described in the first aspect.

[0010] The document question-and-answer data generation method provided in this disclosure first extracts page content from page images in a document to obtain descriptive information of seed pages. Then, based on the descriptive information, the definition of a preset question, and instances of the preset question, it uses a preset question-and-answer data generation model to perform reasoning to determine whether question-and-answer data corresponding to the preset question can be generated based on the descriptive information. If question-and-answer data corresponding to the preset question can be generated based on the descriptive information, the corresponding question-and-answer data is generated. The preset question types include multi-hop questions and / or set questions. Thus, this disclosure can generate complex question-and-answer data containing multi-hop questions and / or set questions based on the descriptive information, improving the quality, diversity, and difficulty of the question-and-answer data.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart illustrating a document question-and-answer data generation method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating another document question-and-answer data generation method provided in this disclosure embodiment; Figure 4A flowchart illustrating a document question-and-answer data generation method in an application scenario provided by an embodiment of this disclosure; Figure 5 A structural block diagram of a document question-and-answer data generation device provided in this embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device suitable for performing a document question-and-answer data generation method, provided as an embodiment of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0015] In the field of data processing, especially in document processing and retrieval enhancement, document question answering tasks can comprehensively utilize the data information in documents to provide users with more comprehensive and accurate content understanding and retrieval services for various questions. To achieve a deep understanding of document content and accurately answer user questions, current document question answering systems generally rely on a large amount of high-quality question-and-answer training data. Related technologies typically employ manual annotation methods or automated synthesis methods to acquire document question answering data.

[0016] Manual annotation methods are typically performed by professional annotators or crowdsourcing teams. The specific process for generating document question-and-answer data involves: first, collecting documents containing various types of information; then, annotators reading each document, combining the different information contained within, designing targeted questions and answers, annotating the answers to each question, and conducting quality checks to form a structured question-and-answer dataset. However, this method has limitations in data scale, making large-scale expansion difficult; the annotation process is time-consuming, labor-intensive, and costly; moreover, this method relies on human experience, making it prone to subjective bias and difficult to guarantee data consistency and comprehensiveness.

[0017] Automated question-answering (QA) generation methods are generally based on pre-set templates or rule engines. The specific process for generating document QA data includes: first, structured parsing of the document content to extract text paragraphs, image descriptions, table data, and other information; then, automatically generating questions related to the document content based on pre-set QA templates or generation rules, and extracting or calculating answers from the document; finally, organizing the generated questions and answers to form an automated QA dataset. However, this method, due to its reliance on template generation or simple rules, results in a limited range of question types, making it difficult to cover complex reasoning, multi-hop question answering, and other high-order tasks. Therefore, questions generated using this method are mostly superficial fact extractions, lacking cross-modal and deep-understanding question design, resulting in insufficient data diversity and quality, which affects the model's training performance and practical application capabilities.

[0018] Based on the problems existing in the above-mentioned related technologies, the present disclosure provides a document question-answer data generation scheme that can automatically generate high-quality question-answer pairs, significantly improving the diversity and complexity of the data.

[0019] Figure 1 An exemplary system architecture 100 is shown, to which embodiments of the document question-and-answer data generation method, apparatus, electronic device, and computer-readable storage medium of this disclosure can be applied.

[0020] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0021] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include data transmission applications and instant messaging applications.

[0022] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0023] Server 105 can provide various services through its built-in applications. Taking a document question-and-answer data generation application as an example, when running this application, server 105 can achieve the following: First, extract page content from the page images in the document to obtain description information corresponding to the seed page in the document; based on the description information, the definition of preset question types, and instances of preset question types, generate a thought process chain corresponding to the seed page using a preset question-and-answer data generation model; wherein, the preset question types include multi-hop questions and / or set questions; in response to the thought process chain forming a question thought chain corresponding to the preset question type, generate question and answer data corresponding to the preset question type based on the question thought chain using the question-and-answer data generation model.

[0024] It should be noted that the page images, preset question type definitions, and instances of the preset question types in the document can be obtained from the terminal devices 101, 102, and 103 via the network 104, or they can be pre-stored locally on the server 105 in various ways. Therefore, when the server 105 detects that this data is already stored locally (e.g., when it starts processing the page images, preset question type definitions, and instances of the preset question types in a previously stored document), it can choose to directly obtain this data from locally. In this case, the exemplary system architecture 100 may not include the terminal devices 101, 102, and 103 and the network 104.

[0025] Since generating document question-and-answer data requires significant computing resources and power, the document question-and-answer data generation methods provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the document question-and-answer data generation device is also generally located within the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also complete the aforementioned calculations performed by the server 105 through their installed document question-and-answer data generation applications, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the document question-and-answer data generation application determines that the terminal device has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the document question-and-answer data generation device can also be located within terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0026] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0027] Please refer to Figure 2 , Figure 2 A flowchart of a document question-and-answer data generation method provided in this disclosure embodiment, which can be applied to, for example... Figure 1 The system architecture 100 shown (e.g.) Figure 1 The server 105 shown can also be applied to processors or electronic devices, etc., and this disclosure does not limit it in this regard. Process 200 includes the following steps: Step 201: Extract page content from the page images in the document to obtain description information corresponding to the seed page in the document.

[0028] Specifically, the document can be a multimodal document, including text, images, tables, and other modal information. This multimodal document... For a series of page images The set consists of N, where N is the total number of pages in the multimodal document and is a positive integer greater than 0. In other words, a page image refers to the page corresponding to each page of the document in the multimodal document, and the page image can include multimodal information such as documents, images, and tables.

[0029] Extracting page content from page images in a document specifically involves extracting structured content such as text blocks, tables, and image descriptions from each page image in the document, generating descriptive information for each page image, and then using this descriptive information to obtain descriptive information related to a seed page in the document. The seed page refers to one page image selected from all page images in the document. The seed page typically serves as the starting point in the document question-and-answer data generation process. After obtaining the document content described above, a question-and-answer data structure can be constructed starting from the seed page. The problem starts with the documentation. One page image is randomly selected from all page images. , as a seed page.

[0030] Optionally, the description information corresponding to the seed page can be obtained solely from the page content in the seed page, or it can be obtained from the page content in other page images besides the seed page.

[0031] Specifically, the descriptive information corresponding to the seed page in the document serves as the basis for generating document question-and-answer data. If the information contained in the current seed page is sufficient, the corresponding descriptive information can be obtained solely based on the content of the seed page. If the content contained in the seed page is limited, the content of the seed page can be supplemented based on the content of other page images in the document, in addition to the seed page, to obtain the descriptive information corresponding to the seed page.

[0032] Optionally, a large language model can be used to perform joint visual and textual understanding of text, images, and tables in the page images of the document, thereby generating descriptive information for each page image. Optionally, a neural network model can also be used to perform semantic recognition and image and table recognition of text, images, and tables in the page images of the document through feature extraction, thereby generating descriptive information for each page image; this disclosure does not limit this approach.

[0033] This disclosure embodiment extracts page content from multimodal documents, enabling multimodal document question-answering tasks to comprehensively utilize various modal information such as text, images, and tables in the document, providing users with more comprehensive and accurate content understanding and retrieval services for various questions.

[0034] Step 202: Based on the description information, the problem definition of the preset question type, and the question and answer examples of the preset question type, generate a thought reasoning chain corresponding to the seed page using the preset question and answer data generation model.

[0035] The preset problem types include multi-hop problems and / or set problems.

[0036] Specifically, the pre-defined question-and-answer data generation model can be a large language model. It first identifies core topics, key entities (such as names, locations, and technical terms), important dates, data, causal relationships, and operational steps based on the descriptive information. Then, combining the question definition and question-and-answer examples of the pre-defined question type, it logically infers the key entities, important dates, causal relationships, and operational steps in the descriptive information according to the question definition and the question-and-answer examples of the pre-defined question type, generating a reasoning chain. Based on this reasoning chain, it determines whether the descriptive information can generate question-and-answer data corresponding to the pre-defined question type. Here, the pre-defined question type examples refer to question-and-answer examples of the pre-defined question type, enabling the question-and-answer data generation model to more accurately infer whether the descriptive information can generate question-and-answer data corresponding to the pre-defined question type, based on the pre-defined question definition and specific examples of pre-defined questions.

[0037] Multi-hop problems refer to those where the answer cannot be found directly from a single source or passage. Answering multi-hop problems requires multiple "reasoning jumps," that is, connecting multiple pieces of information and performing logical reasoning or comprehensive judgment. The core characteristic of multi-hop problems is that they involve a single entity, requiring reasoning and multiple steps based on that entity.

[0038] An example of a multi-hop question-and-answer problem could be: "What is the birthday of A's wife?" This multi-hop question involves only one entity, A, and then reasoning is performed based on entity A to arrive at the final answer. First, the first hop determines A's wife, B, based on A, and then the second hop determines B's birthday.

[0039] Set problems are those where the answer is not a single entity, but a set (i.e., a list or group of entities). The goal of the problem is to enumerate all items that meet specific conditions. The core characteristics of set problems are the involvement of multiple entities and the complexity of the answer.

[0040] An example of a set problem could be: "Who are some Chinese people who are both directors and actors?" This set problem involves two entities: directors and actors. First, we need to determine the set of all Chinese directors, then we need to determine the set of all Chinese actors, and finally, we find the intersection of the two sets to obtain the final answer.

[0041] Optionally, to further generate high-quality, complex-structured questions, the question-answer instances of the preset question type can also include instances that include both multi-hop questions and set questions. Thus, when the descriptive information is sufficient, based on the question definition of the preset question type and the question-answer instances of the preset question type, complex question-answer data containing multi-hop and set questions can also be generated.

[0042] For example, an instance that includes both multi-hop problems and set problems could be: "Who are the wives of Chinese people who are both directors and actors?" This example adds a multi-hop problem to the set problem example above. First, based on the set problem, we determine the Chinese person C who is both a director and an actor. Then, based on C, we perform multi-hop problems to determine C's wives, thus obtaining the final answer.

[0043] Of course, instances that include both multi-hop problems and set problems can also include set problems interspersed between multi-hop problems. This disclosure does not limit the question-and-answer instances of the preset question types.

[0044] There can be one or more preset question types. When there is only one preset question type, the specified question and answer data can be generated; when there are multiple preset question types, the most matching question and answer data among the multiple preset question types can be generated based on the description information.

[0045] Specifically, in practical application scenarios, for documents Typically, the type and number of questions to be generated are specified. For example, 100 multi-hop questions might be generated first, followed by 200 set questions. In this case, multi-hop and set questions can be generated sequentially based on requirements. Therefore, when generating multi-hop questions, only the description information, the question definition, and question-answer instances of the multi-hop questions need to be input into the preset question-answer data generation model; similarly, when generating set questions, only the description information, the question definition, and question-answer instances of the set questions need to be input into the preset question-answer data generation model. That is, if the question type for which question-answer data needs to be generated is specified in the actual application scenario, then when generating the corresponding thought process chain using the preset question-answer data generation model, only the question definition and question-answer instances of the specified question type need to be input.

[0046] In other application scenarios, it may be possible to omit the question type from which the question-and-answer data is generated. In this case, all the definitions of the preset question types and all the question-and-answer instances of the preset question types need to be used as input to the preset question-and-answer data generation model. In this way, the preset question-and-answer data generation model can generate the best document question-and-answer data based on the descriptive information, thereby improving the quality and efficiency of the generated question-and-answer data.

[0047] Step 203: In response to the formation of a thought reasoning chain and a question thinking chain corresponding to the preset question type, based on the question thinking chain, use the question-and-answer data generation model to generate question-and-answer data corresponding to the preset question type.

[0048] Specifically, in this disclosure, the correspondence between a thought process chain and a preset problem type means that the thought process chain corresponds to at least a portion of the preset problem type. In other words, if the preset problem type is a multi-hop problem, then the correspondence between the thought process chain and the preset problem type means that the thought process chain corresponds to a multi-hop problem; similarly, if the preset problem type is a set problem, then the correspondence between the thought process chain and the preset problem type means that the thought process chain corresponds to a set problem; if the preset problem type is both a multi-hop problem and a set problem, then the correspondence between the thought process chain and the preset problem type means that the thought process chain corresponds to a multi-hop problem and / or a multi-hop problem.

[0049] If the reasoning chain output by the question-and-answer data generation model includes a chain-like reasoning chain corresponding to a multi-hop problem, it means that a reasoning chain corresponding to a multi-hop problem can be generated based on the descriptive information, thus indicating that a multi-hop problem can be constructed based on the descriptive information. Therefore, the question-and-answer data generation model further generates corresponding multi-hop question-and-answer data based on the reasoning chain and the problem definition of the multi-hop problem. Similarly, if the reasoning chain output by the question-and-answer data generation model includes a chain-like reasoning chain corresponding to a set problem, it means that a reasoning chain corresponding to a set problem can be generated based on the descriptive information, thus indicating that a set problem can be constructed based on the descriptive information. Therefore, the question-and-answer data generation model further generates corresponding set question-and-answer data based on the reasoning chain and the problem definition of the set problem.

[0050] Furthermore, when the reasoning chain includes both chain-based reasoning chains corresponding to multi-hop problems and set operation reasoning chains corresponding to set problems, it indicates that the descriptive information can generate both the reasoning chains corresponding to multi-hop problems and the reasoning chains corresponding to set problems. This demonstrates that the descriptive information can be used to construct both multi-hop and set problems. Therefore, the question-answering data generation model further generates comprehensive question-answering data containing both multi-hop and set elements based on the reasoning chain, the problem definition of set problems, and the problem definition of multi-hop problems.

[0051] The document question-and-answer data generation method provided in this disclosure first extracts page content from page images in a document to obtain descriptive information of seed pages. Then, based on the descriptive information, the definition of a preset question, and instances of the preset question, it uses a preset question-and-answer data generation model to perform reasoning to determine whether question-and-answer data corresponding to the preset question can be generated based on the descriptive information. If question-and-answer data corresponding to the preset question can be generated based on the descriptive information, the corresponding question-and-answer data is generated. The preset question types include multi-hop questions and / or set questions. Thus, this disclosure can generate complex question-and-answer data containing multi-hop questions and / or set questions based on the descriptive information, improving the quality, diversity, and difficulty of the question-and-answer data.

[0052] In some embodiments, when the preset question types include multi-hop questions and set problems, the corresponding question thought chain is a chain-like reasoning chain corresponding to multi-hop questions and / or a set operation reasoning chain corresponding to set problems; the above Figure 2 Step 203: In response to the formation of a thought reasoning chain corresponding to a preset question type, based on the question reasoning chain, using a question-and-answer data generation model, generate question-and-answer data corresponding to the preset question type, including: in response to the formation of a thought reasoning chain corresponding to a preset question type, based on the question reasoning chain, using a question-and-answer data generation model, generate multi-hop question-and-answer data corresponding to the chained reasoning chain and / or set question-and-answer data corresponding to the set operation reasoning chain.

[0053] Specifically, when the preset question types are multi-hop questions and set problems, the corresponding preset question instances include instances of multi-hop questions and instances of set problems. Therefore, the question-answering data model can generate two types of question thought chains based on the sufficiency of the descriptive information: chain-based reasoning chains corresponding to multi-hop questions and set operation reasoning chains corresponding to set problems. Specifically, when the question-answering data model, based on the descriptive information and the definitions of multi-hop questions and set problems, determines that the descriptive information only includes multi-hop logic corresponding to multi-hop questions, it can generate a chain-based reasoning chain corresponding to multi-hop questions. When it determines that the descriptive information only includes set logic corresponding to data sets, it can generate a set operation reasoning chain corresponding to set problems. And when it determines that the descriptive information includes both multi-hop logic corresponding to multi-hop questions and set logic corresponding to sets, it can generate both chain-based reasoning chains corresponding to multi-hop questions and set operation reasoning chains corresponding to set problems.

[0054] In response to the above Figure 2 Step 201 involves extracting page content from the page images in the document to obtain description information corresponding to the seed page in the document. A specific implementation method is given below.

[0055] refer to Figure 3 , Figure 3 This is a flowchart illustrating yet another document question-and-answer data generation method provided in this embodiment of the disclosure. For example... Figure 3 As shown, the method specifically includes the following steps: Step 301: Extract the page content from the page images in the document to obtain the text information of the page images.

[0056] Specifically, the content of each page is first fully analyzed to extract text, images, tables, etc., from the page images. The text content is then converted and output as text, yielding the text information of the page images. The formula is expressed as: ,in Represents page image The result after text content conversion.

[0057] Step 302: Using a preset multimodal model, process the page image and corresponding text information to obtain the description information corresponding to the seed page in the document.

[0058] Specifically, based on step 301, a multimodal large model can be further utilized. Taking a page image and its corresponding text as input, the program outputs a corresponding descriptive message. The formula is expressed as: Each of them This can be viewed as a statement of an objective fact, such as "Model X achieves result C for metric B on dataset A." It is worth noting that the declarative description can come from any content on the page, such as images, text, tables, etc. Optionally, step 301 above can also extract page content from page images in the document based on the multimodal model to obtain text information from the page images; this disclosure does not limit this approach.

[0059] This embodiment of the disclosure processes the page image using a preset multimodal model, employing both the page image and the corresponding text information in two modes. Compared to directly obtaining the corresponding descriptive information from the text information extracted from the page image, the descriptive information generated by this embodiment is more accurate and comprehensive.

[0060] Step 303: Based on the description information, the question definition of the preset question type, and the question-and-answer examples of the preset question type, generate a thought reasoning chain corresponding to the seed page using the preset question-and-answer data generation model; wherein, the preset question type includes multi-hop questions and / or set questions.

[0061] Step 304: In response to the formation of a thought reasoning chain that corresponds to the preset question type, generate question and answer data corresponding to the preset question type using a question and answer data generation model based on the question and answer chain.

[0062] Steps 301 to 302 above are as follows Figure 2 One specific implementation of step 201 shown, steps 303-304 are as follows: Figure 2 Steps 202-203 shown are the same. For the same parts, please refer to the corresponding parts of the previous embodiment. They will not be repeated here.

[0063] To ensure that the obtained seed page description information is more comprehensive and richer, the above... Figure 2Step 201: Extract page content from the page images in the document to obtain descriptive information corresponding to the seed page in the document. This may include descriptive information of the seed page and supplementary descriptive information from other page images. A specific implementation method is given below.

[0064] In some embodiments, extracting page content from page images in a document to obtain descriptive information corresponding to a seed page in the document includes: extracting page content from the page images of the seed page using a preset multimodal model to generate text information corresponding to the seed page, and generating declarative descriptive information of the seed page based on the page images and corresponding text information; ranking the page images in the document by relevance based on the declarative descriptive information of the seed page to obtain associated pages of the seed page; extracting page content from the page images of the associated pages using a multimodal model to generate text information corresponding to the associated pages, and processing the page images and corresponding text information of the associated pages to generate supplementary descriptive information of the seed page; and obtaining the descriptive information of the seed page based on the declarative descriptive information and the supplementary descriptive information.

[0065] Specifically, this disclosure first uses a pre-defined multimodal model to obtain a descriptive description of the seed page; then, after obtaining the descriptive description information of the seed page, it can retrieve the most similar related pages from the document. For example, by calculating the similarity between the images of each page in the document and the seed page, a sorted list of pages in the document can be obtained, and then related pages can be obtained based on the sorted list. Optionally, the similarity between the images of each page in the document and the seed page can be calculated using a cosine similarity calculation function; alternatively, one or more keywords can be determined based on the descriptive description information of the seed page, and then the similarity between the images of each page in the document and the seed page can be determined based on the frequency of the keywords appearing on each page; other methods can also be used to determine the similarity between the images of each page in the document and the seed page, and this disclosure does not limit this method.

[0066] It should be noted that after obtaining related pages based on page ranking, seed pages need to be excluded; finally, a multimodal model is used to obtain supplementary information related to the seed pages from the related pages. Specifically, one or more page images with the highest relevance can be selected as related pages based on the richness of descriptive information of the seed pages and the page ranking.

[0067] The process of obtaining descriptive information from seed pages and supplementary descriptive information from associated pages using a multimodal model is similar to the above. Figure 3 The corresponding implementation methods are similar and will not be repeated here.

[0068] The question-and-answer data generation method provided in this disclosure, for the descriptive description information of the seed page, searches for supplementary descriptions related to the descriptive description in the images of other pages in the document besides the seed page, which can make the obtained description information more complete, thereby generating higher quality question-and-answer data based on the description information.

[0069] When searching for supplementary descriptions related to the declarative description information based on the seed page, "related" can be defined as duplicates of the declarative description information, the existence of the same entity as the declarative description information, or the sharing of the same relationship as the declarative description information. A specific implementation method is given below.

[0070] In some embodiments, based on the descriptive information of the seed page, page images in the document are ranked by relevance to obtain associated pages of the seed page, including: determining the entities and / or relationships contained in the seed page according to the descriptive information of the seed page; retrieving page images containing entities and / or relationships from the page images in the document to obtain candidate pages; and ranking the candidate pages by relevance to obtain associated pages of the seed page.

[0071] Specifically, this disclosure first identifies entities and relationships in the descriptive information. Then, pages that share entities and / or relationships with the seed page are selected as candidate pages. Finally, all candidate pages are sorted according to relevance, and one or more candidate pages with the highest similarity are selected from the candidate pages based on the richness of the descriptive information or a preset threshold to identify related pages.

[0072] In this context, an entity is an objectively existing thing or object that can be distinguished from another, such as model X, a person, or a location. A relationship describes the way entities are associated, and can be a conclusive description, an attribute description, etc.

[0073] Optionally, since the multi-hop problem involves deeper logical reasoning about a single entity, page images that share entities with the seed page can be considered as candidate pages for the multi-hop problem. Optionally, since the set problem involves set operations on multiple entities, page images that share relationships with the seed page can be considered as candidate pages for the set problem. Furthermore, to enrich the descriptive information of the seed page, page images that share both entities and relationships with the seed page can be considered as candidate pages; this disclosure does not limit this approach.

[0074] This disclosure embodiment selects page images with the same entities (e.g., both describing model X) and / or the same relationships (both conclusive descriptions) as candidate pages, and then determines associated pages based on the candidate pages. Thus, the supplementary descriptive information obtained based on the associated pages can improve the generation efficiency and generation quality of question-and-answer data corresponding to multi-hop questions and / or set questions.

[0075] In determining the associated pages, it is necessary to retrieve page images related to the seed page from the page images in the document and calculate the corresponding similarity. This disclosure can also improve retrieval efficiency by encoding the descriptive information of the seed page and each page image. A specific implementation method is given below.

[0076] In some embodiments, based on the descriptive information of the seed page, the page images in the document are ranked according to their relevance to obtain the associated pages of the seed page. This includes: performing vector encoding on each page image in the document to obtain an encoding vector corresponding to each page image; performing vector encoding on the descriptive information of the seed page to obtain an encoded description corresponding to the descriptive information; calculating the similarity score between each encoded vector and the encoded description to obtain a similarity ranking; and determining the associated pages of the seed page based on the similarity ranking.

[0077] Specifically, a pre-trained visual language model, denoted as VLM, can be used to convert each page image into a vector form, thus obtaining the encoding vector corresponding to each page image. This is used for subsequent retrieval. A VLM can encode both text and images, mapping both to a fixed-length vector. This process can be represented as: ,in, For a length of A matrix composed of vectors, The vector representation of the image on each page of the document.

[0078] Obtain the descriptive information corresponding to the seed page. Subsequently, the pre-trained Visual Language Model (VLM) can be used to encode the declarative information, obtaining an encoded description corresponding to the declarative information. Then, similarity calculations can be performed between the encoded vector and the encoded description to rank the images on each page of the document based on their similarity.

[0079] Optionally, candidate pages similar to the seed page can be retrieved from each encoding vector based on the encoding description, and then similarity can be calculated based on the encoding vector corresponding to the candidate page and the encoding description to obtain the similarity ranking. This disclosure can reduce the computational amount of similarity score calculation and improve computational efficiency.

[0080] This disclosure improves the efficiency of identifying associated pages by converting both page images and descriptive information of seed pages into vector representations, and then performing similar page image retrieval and similarity calculation based on the vector representation.

[0081] If, based on the declarative and supplementary descriptive information of the seed page, the question-answering data generation model determines that the descriptive information still cannot constitute the preset question type (multi-hop question and / or set question), it means that the current descriptive information is insufficient. In this case, the next page image can be selected based on the relevance ranking of the page images in the previously determined document to supplement the descriptive information. A specific implementation method is given below.

[0082] In some embodiments, the method includes not only steps 201-203, but also: in response to the reasoning chain not containing a reasoning chain corresponding to a preset question type, expanding the associated pages based on relevance ranking to obtain expanded pages; using a multimodal model to process the page image and corresponding text information of the expanded pages to generate expanded description information of the seed page; and regenerating the reasoning chain corresponding to the seed page based on the expanded description information.

[0083] The process of processing the page image and corresponding text information of the expanded page to generate the expanded description information of the seed page is similar to the process of obtaining the descriptive description information and the supplementary description information, and will not be repeated here.

[0084] This disclosure expands the associated pages based on relevance ranking, thereby generating a thought process chain that better matches the preset question type based on expanded descriptive information, declarative descriptive information, and supplementary descriptive information.

[0085] Optionally, before obtaining the expanded pages, the images of each page in the document can be re-sorted based on descriptive information (including descriptive and supplementary descriptive information), the relevance ranking can be updated based on the ranking result, and the related pages can be expanded based on the updated relevance ranking.

[0086] Specifically, because descriptive information contains limited details, relevance ranking based on it may contain errors. This disclosure improves the accuracy of relevance ranking by ranking based on both descriptive and supplementary descriptive information, allowing for more comprehensive information-based judgment. This results in expanded pages being more relevant to seed and related pages, leading to more accurate expanded descriptive information. Consequently, the generated reasoning chain based on expanded, descriptive, and supplementary descriptive information better matches the preset question types (multi-hop questions and / or set questions), improving the efficiency of document question-and-answer data generation.

[0087] In response to the above Figure 2 In step 202, during the process of generating the thought reasoning chain corresponding to the seed page using the preset question-and-answer data generation model, different types of question definitions and question instances can be selected to generate the thought reasoning chain based on the different numbers of entities in the description information. Two specific implementation methods are given below.

[0088] In some embodiments, based on the description information, the question definition of a preset question type, and question-answer instances of a preset question type, a thought reasoning chain corresponding to the seed page is generated using a preset question-answer data generation model, including: in response to using the preset question-answer data generation model, determining that the description information includes only one entity; using the preset question-answer data generation model, determining the jump relationship of the entity in the description information based on the question definition of a multi-hop question and question-answer instances of a multi-hop question; and generating a thought reasoning chain corresponding to the seed page based on the jump relationship.

[0089] In some embodiments, based on the description information, the question definition of a preset question type, and question-answer instances of the preset question type, a thought reasoning chain corresponding to the seed page is generated using a preset question-answer data generation model, including: in response to using the preset question-answer data generation model, determining that the description information contains multiple entities; using the preset question-answer data generation model, determining the set relationship of the multiple entities in the description information based on the question definition of a set question and question-answer instances of a set question; and generating a thought reasoning chain corresponding to the seed page based on the set relationship.

[0090] Specifically, as mentioned above, multi-hop problems generally involve deeper logical reasoning on the same entity, while set problems generally involve combining multiple entities. Therefore, when the description information contains only one entity, it is highly likely that a multi-hop problem can be formed based on that description information. Thus, the pre-defined question-answering data generation model can directly perform relevant reasoning on the description information based on the problem definition and question-answer instances of multi-hop problems. Similarly, when the description information contains multiple entities, it is highly likely that a set problem can be formed based on that description information. Therefore, the pre-defined question-answering data generation model can directly perform relevant reasoning on the description information based on the problem definition and question-answer instances of set problems. This improves the efficiency and accuracy of the question-answering data generation model in generating reasoning chains.

[0091] In some embodiments, set problems include set intersection problems, set union problems, and set difference problems.

[0092] Specifically, set intersection problems require finding common elements or information among multiple objective facts, with the answer being the intersection of these facts. For example, finding the highest score among students who are simultaneously on the basketball team and in the math club requires first identifying the students who are in both clubs, and then determining their highest score.

[0093] The set union problem requires finding a logical OR relationship between multiple objective facts, and the answer is the union of these objective facts. For example, to find the highest salary among employees working in the finance department or human resources department, we need to first find the employees in the finance department or human resources department, and then solve for the maximum salary.

[0094] The set difference problem requires finding multiple objective facts that are either A or B, meaning two conditions cannot be satisfied simultaneously. The answer is the difference set of these objective facts. For example, to find the highest salary among employees who work in the sales department but not in the marketing department, we need to first find the employees in the sales department, check if they work in the marketing department, find the employees who do not work in the marketing department, and then solve for their highest salary.

[0095] By using different problem definitions and examples for different set types in set problems, we can generate reasoning chains for different types of set problems based on the descriptive information, thereby improving the richness and diversity of the final generated document question-and-answer data.

[0096] The seed page in this disclosure can be a page image randomly selected from all page images in the document, or it can be selected based on specific needs. A specific implementation method is given below.

[0097] In some embodiments, the method includes not only steps 201-203 described above, but also: selecting a page image from the document as a seed page based on the business domain and business objectives corresponding to the document.

[0098] Specifically, seed pages are typically selected from the pages in a document that best represent the core business, kickstart the project, or provide an overview. These might be the cover, table of contents, executive summary, key content pages, etc., used as a reference, starting point, or promotional material. However, the location of the corresponding core business element in the document may differ depending on the specific business area and objectives.

[0099] For example, if the document is a business plan, its corresponding business area is startup / technology, and its business goal is to attract investors or partners. For business plans, the executive summary typically provides a condensed version of the entire plan, including the business concept, market opportunity, financial projections, and investment needs. Therefore, using the summary as a seed page results in more targeted and higher-quality Q&A data.

[0100] For example, this document is a product brochure, the business area is consumer goods, and the business objective is to attract potential customers and stimulate their desire to purchase. For product brochures, the introduction of the main product is generally the core of the brochure. Therefore, the page image corresponding to the main product introduction can be used as a seed page. Thus, using the page image corresponding to the main product introduction as a seed page results in Q&A data that better reflects the features of the main product, thereby attracting more user attention, achieving the product promotion goal, and improving the quality of the generated Q&A data.

[0101] This disclosure embodiment selects a seed page based on the business domain and business objectives corresponding to the document, and then generates question and answer data starting from the seed page, making the generated question and answer data more specific and targeted, and improving the quality of the question and answer data.

[0102] To enhance understanding, this disclosure also provides a specific implementation scheme based on a particular application scenario. Please refer to the example below. Figure 4 The process 400 shown specifically includes the following steps: Step 401: Multimodal document content transformation and indexing.

[0103] Specifically, a multimodal document Viewed as a series of page images The set is composed of [images]. First, a multi-modal large language model (MLLM) is used to comprehensively analyze the content of each page image. MLLM takes images and text as input and outputs text. Specifically, for each page image, MLLM is used for joint visual and text understanding to extract structured content such as text blocks, tables, and image captions, which are then output as text. The formula is as follows: in Page The result after text content conversion. Following this, a pre-trained visual language model, denoted as VLM, is used to convert each page image into a vector for subsequent retrieval. VLM can encode both text and images, mapping them to a fixed-length vector. This process can be represented as: Where E is a matrix composed of vectors of length d, corresponding to the vector representation of each page in the document.

[0104] Step 402: Select seed page.

[0105] Specifically, after obtaining the aforementioned document content, starting from the seed page, a framework targeting the document is constructed. The problem is this: First, randomly select one page from all the pages. This serves as a seed page. Based on the text content of this page... Using a multimodal large language model, a declarative description is obtained. The process is as follows:

[0106] Each This can be viewed as a statement of an objective fact, such as "Model X achieves result C for metric B on dataset A." It's worth noting that declarative descriptions can come from any content on this page, such as images, text, tables, etc.

[0107] Step 403: Rank the page images by similarity.

[0108] Specifically, in obtaining a declarative description Then, a pre-trained visual language model (VLM) is used to analyze this declarative description. Encode and retrieve information related to the declarative description. The most similar candidate pages are selected, and a list of pages is obtained by calculating similarity. The pages in this list contain elements that are similar to the declarative description. Relevant information. Additionally, it should be noted that after obtaining the page ranking, seed pages need to be excluded. The overall process is as follows:

[0109] in, Indicates a declarative description Vector encoding, It is the cosine similarity calculation function. yes For each page in the document, a similarity score is calculated, and based on this score, a ranking of each page can be obtained. .

[0110] Then, using Multimodal Large Model (MLLM), based on the page ranking described above... Retrieve the pages with the highest similarity in order, and perform the following steps 404~406 sequentially: Step 404: Expand the declarative description.

[0111] For declarative descriptions, search the current page for related supplementary descriptions. Related descriptions can be defined as those that overlap with the declarative description, such as the same entity (e.g., both describing model X) or the same relationship (e.g., both conclusive descriptions). Extract the page content using a multimodal large model. ,in This indicates additional descriptions on this page.

[0112] Step 405: Sufficiency determination.

[0113] Combining the existing supplementary description and the original declarative description, and considering the definition of the pre-defined problem (multi-hop problem or set problem), determine whether it constitutes the pre-defined problem. Optionally, a Large Language Model (LLM) can be used as the inference engine, outputting a binary value of yes or no, indicating whether the current supplementary and declarative descriptions are sufficient. ,in, It is a binary value indicating whether a predefined problem can be formed. It is a thought process, representing the reasons for making a judgment. This indicates the specific definition of the presupposed problem. This represents a context instance of the preset question. If the preset question can be constituted based on the above LLM judgment, then proceed to step 406; if the preset question cannot be constituted based on the above LLM judgment, then return to step 404, continue to select the next page based on similarity sorting, and continue to expand and supplement the description until the preset question can be constituted based on the above LLM judgment, then proceed to step 406.

[0114] Step 406: Combining the question and answer.

[0115] Specifically, if the current declarative and supplementary descriptions satisfy the definition of a random problem, i.e., they constitute a predefined problem, then a Large Language Model (LLM) is used to synthesize the problem. Based on the sufficiency criteria obtained in step 405, and combining the declarative and supplementary descriptions, the problem and answer are synthesized:

[0116] in, This indicates the definition of the pre-defined problem. Explain the reasons for the sufficiency determination. , These represent the question and answer to the problem, respectively.

[0117] The preset problems can be multi-hop problems, intersection problems, union problems, and difference problems. The main difference for different types of problems is that the definitions of different types of problems in step 405 and the context examples of the corresponding problems are different. Other processes are the same as the synthesis process of multi-hop problems.

[0118] The document question-and-answer data generation method disclosed herein first extracts page content from page images in a document to obtain descriptive information of seed pages. Then, based on the descriptive information, the definition of a preset question, and instances of the preset question, it uses a preset question-and-answer data generation model to perform reasoning to determine whether question-and-answer data corresponding to the preset question can be generated based on the descriptive information. If question-and-answer data corresponding to the preset question can be generated based on the descriptive information, the corresponding question-and-answer data is generated. The preset question types include multi-hop questions and / or set questions. Thus, this disclosure can generate complex question-and-answer data containing multi-hop questions and / or set questions based on the descriptive information, improving the quality, diversity, and difficulty of the question-and-answer data.

[0119] Furthermore, document question-answering data can be applied to document processing and retrieval enhancement. In document processing and retrieval enhancement, multimodal question-answering models are typically used to gain a deep understanding of document content and accurately answer user questions.

[0120] This disclosure also provides a training method for a question-answering model, which specifically includes: obtaining document question-answering data samples; inputting the document question-answering data samples into the question-answering model to train the question-answering model.

[0121] The document question-and-answer data samples are constructed using the method described in any of the above embodiments of the document question-and-answer data generation method, and will not be repeated here.

[0122] This disclosure provides embodiments for constructing training data for a question-answering model based on document question-answering data obtained from any of the above-described document data generation embodiments. Since the document question-answering data obtained according to any of the above embodiments covers complex question types such as multiple inference and set theory questions, the embodiments of the above document data generation method automatically construct complex, multi-hop, multimodal document questions and their corresponding answers through operations such as multi-hop questions, intersections, unions, and differences of sets. This effectively improves the scale and quality of the training data, providing a solid data foundation for model training and meeting the actual training data needs of products such as intelligent document processing and intelligent retrieval. Therefore, the question-answering model trained based on this document question-answering data has stronger generalization ability and better performance in practical applications.

[0123] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a document question-and-answer data generation apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0124] like Figure 5 As shown, the document question-and-answer data generation device 500 of this embodiment may include: a page extraction module 501, a reasoning module 502, and a data generation module. The page extraction module 501 is used to extract page content from page images in a document to obtain descriptive information corresponding to a seed page in the document. The reasoning module 502 is used to generate a thought reasoning chain corresponding to the seed page based on the descriptive information, a question definition of a preset question type, and question-and-answer instances of the preset question type, using a preset question-and-answer data generation model. The preset question types include multi-hop questions and / or set questions. The data generation module 503 is used to generate question-and-answer data corresponding to the preset question type based on the question reasoning chain forming a question thought chain corresponding to the preset question type, using the question-and-answer data generation model.

[0125] In this embodiment, the specific processing of the page extraction module 501, the reasoning module 502, and the data generation module in the document question-and-answer data generation device 500, and the resulting technical effects, can be found in reference to [reference needed]. Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0126] In some embodiments, the preset question types include multi-hop questions and set questions. Correspondingly, the question reasoning chain is a chain-like reasoning chain corresponding to multi-hop questions and / or a set operation reasoning chain corresponding to set questions. The data generation module 503 is specifically used to, in response to the reasoning chain containing a chain-like reasoning chain corresponding to multi-hop questions and / or a set operation reasoning chain corresponding to set questions, generate multi-hop question-and-answer data corresponding to the chain-like reasoning chain and / or set question-and-answer data corresponding to the set operation reasoning chain using a question-and-answer data generation model. In some embodiments, the page extraction module 501 is specifically used to extract page content from page images in the document to obtain text information of the page images; and to process the page images and corresponding text information using a preset multimodal model to obtain descriptive information corresponding to seed pages in the document.

[0127] In some embodiments, the description information includes declarative description information and supplementary description information; the page extraction module 501 is specifically used to extract page content from the page image of the seed page using a preset multimodal model, generate text information corresponding to the seed page, and generate declarative description information of the seed page based on the page image of the seed page and the corresponding text information; based on the declarative description information of the seed page, sort the page images in the document by relevance to obtain the associated pages of the seed page; use the multimodal model to extract page content from the page images of the associated pages, generate text information corresponding to the associated pages, and process the page images of the associated pages and the corresponding text information to generate supplementary description information of the seed page; and obtain the description information of the seed page based on the declarative description information and the supplementary description information.

[0128] In some embodiments, the page extraction module 501 is specifically configured to determine the entities and / or relationships contained in the seed page based on the descriptive information of the seed page; retrieve page images containing the entities and / or relationships from the page images in the document to obtain candidate pages; and sort the candidate pages by relevance to obtain the associated pages of the seed page.

[0129] In some embodiments, the page extraction module 501 is specifically used to perform vector encoding on each page image in the document to obtain an encoding vector corresponding to each page image; to perform vector encoding on the descriptive information of the seed page to obtain an encoded description corresponding to the descriptive information; to calculate the similarity score between each encoded vector and the encoded description to obtain a similarity ranking; and to determine the associated pages of the seed page based on the similarity ranking.

[0130] In some embodiments, the document question-and-answer data generation device 500 includes not only a page extraction module 501, a reasoning module 502, and a data generation module, but also an expansion module. This module is used to expand the associated pages based on relevance ranking in response to the reasoning chain not containing a reasoning chain corresponding to a preset question type, thereby obtaining expanded pages; to process the page image and corresponding text information of the expanded pages using a multimodal model, generating expanded description information for the seed page; and to regenerate the reasoning chain corresponding to the seed page based on the expanded description information.

[0131] In some embodiments, the reasoning module 502 is specifically configured to: in response to using a preset question-and-answer data generation model, determine that the description information includes only one entity; using the preset question-and-answer data generation model, determine the jump relationship of the entity in the description information based on the question definition of a multi-hop question and the question-and-answer instance of a multi-hop question; and generate a thought reasoning chain corresponding to the seed page based on the jump relationship.

[0132] In some embodiments, the reasoning module 502 is specifically used to: in response to using a preset question-and-answer data generation model to determine that the description information contains multiple entities; using the preset question-and-answer data generation model, based on the question definition of the set question and the question-and-answer instances of the set question, to determine the set relationship of the multiple entities in the description information; and based on the set relationship, to generate a thought reasoning chain corresponding to the seed page.

[0133] In some embodiments, the set problems include the set intersection problem, the set union problem, and the set difference problem.

[0134] In some embodiments, in addition to the page extraction module 501, the inference module 502, and the data generation module, a selection module is also included. The selection module is specifically used to select a page image from the document as a seed page based on the business domain and business objectives corresponding to the document.

[0135] This embodiment is a device embodiment corresponding to the method embodiment described above. The document question-and-answer data generation device provided in this embodiment first extracts page content from page images in a document to obtain description information of seed pages. Then, based on the description information, the definition of a preset question, and instances of the preset question, it uses a preset question-and-answer data generation model to perform reasoning to determine whether question-and-answer data corresponding to the preset question can be generated based on the description information. If question-and-answer data corresponding to the preset question can be generated based on the description information, the corresponding question-and-answer data is generated. The preset question types include multi-hop questions and / or set questions. Thus, this disclosure can generate complex question-and-answer data containing multi-hop questions and / or set questions based on the description information, improving the quality, diversity, and difficulty of the question-and-answer data.

[0136] According to embodiments of this disclosure, this disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the document question-and-answer data generation method described in any of the above embodiments.

[0137] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the document question-and-answer data generation method described in any of the above embodiments when executed.

[0138] According to embodiments of this disclosure, this disclosure also provides a computer program product that, when executed by a processor, can implement the document question-and-answer data generation method described in any of the above embodiments.

[0139] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0141] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the document question-and-answer data generation method. For example, in some embodiments, the document question-and-answer data generation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the document question-and-answer data generation method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the document question-and-answer data generation method by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0148] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0149] According to the technical solution of this disclosure, firstly, page content is extracted from page images in a document to obtain description information of the seed page. Then, based on the description information, the definition of a preset question, and instances of the preset question, a preset question-and-answer data generation model is used for reasoning to determine whether question-and-answer data corresponding to the preset question can be generated based on the description information. If question-and-answer data corresponding to the preset question can be generated based on the description information, the corresponding question-and-answer data is generated. The preset question types include multi-hop questions and / or set questions. Thus, this disclosure can generate complex question-and-answer data containing multi-hop questions and / or set questions based on the description information, improving the quality, diversity, and difficulty of the question-and-answer data.

[0150] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0151] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating document question-and-answer data, comprising: Extract page content from page images in the document to obtain descriptive information corresponding to the seed page in the document; Based on the description information, the question definition of the preset question type, and the question-and-answer instance of the preset question type, a thought reasoning chain corresponding to the seed page is generated using a preset question-and-answer data generation model. The thought reasoning chain is used to determine whether question-and-answer data corresponding to the preset question type can be generated based on the description information. The preset question type includes multi-hop questions and / or set questions. In response to the thought reasoning chain forming a question thought chain corresponding to the preset question type, based on the question thought chain, the question-and-answer data generation model is used to generate question-and-answer data for the document that corresponds to the preset question type.

2. The method according to claim 1, wherein, The preset problem types include multi-hop problems and set problems. Correspondingly, the problem thinking chain is a chain reasoning chain corresponding to the multi-hop problem and / or a set operation reasoning chain corresponding to the set problem. The step involves responding to the thought reasoning chain forming a question thought chain corresponding to the preset question type, and based on the question thought chain, using the question-and-answer data generation model to generate question-and-answer data for the document and corresponding to the preset question type, including: In response to the thought reasoning chain forming a question thought chain corresponding to the preset question type, based on the question thought chain, the question-and-answer data generation model is used to generate multi-hop question-and-answer data for the document and corresponding to the chained reasoning chain and / or set question-and-answer data corresponding to the set operation reasoning chain.

3. The method according to claim 1, wherein, The step of extracting page content from page images in the document to obtain description information corresponding to the seed page in the document includes: Extract page content from page images in a document to obtain text information of the page images; Using a preset multimodal model, the page image and corresponding text information are processed to obtain description information corresponding to the seed page in the document.

4. The method according to claim 3, wherein, The descriptive information includes declarative descriptive information and supplementary descriptive information; The step of extracting page content from page images in the document to obtain description information corresponding to the seed page in the document includes: Using a preset multimodal model, the page content of the seed page is extracted from the page image, text information corresponding to the seed page is generated, and descriptive information of the seed page is generated based on the page image of the seed page and the corresponding text information. Based on the descriptive information of the seed page, the page images in the document are sorted by relevance to obtain the associated pages of the seed page; Using the multimodal model, page content is extracted from the page image of the associated page, generating text information corresponding to the associated page. Based on the page image of the associated page and the corresponding text information, supplementary description information of the seed page is generated. The description information of the seed page is obtained based on the declarative description information and the supplementary description information.

5. The method according to claim 4, wherein, The step of ranking the page images in the document based on the descriptive information of the seed page to obtain the associated pages of the seed page includes: Based on the descriptive information of the seed page, determine the entities and / or relationships contained in the seed page; Retrieve page images containing the entities and / or relationships from the page images in the document to obtain candidate pages; The candidate pages are sorted by relevance to obtain the associated pages of the seed page.

6. The method according to claim 4, wherein, The step of ranking the page images in the document based on the descriptive information of the seed page to obtain the associated pages of the seed page includes: The images on each page of the document are vector-encoded to obtain the encoded vectors corresponding to each page image. The declarative description information of the seed page is vector-encoded to obtain an encoded description corresponding to the declarative description information; Calculate the similarity score between each encoded vector and the encoded description to obtain a similarity ranking; The associated pages of the seed page are determined based on the similarity ranking.

7. The method according to claim 4, wherein, Also includes: In response to the fact that the thought reasoning chain does not constitute a thought chain corresponding to the preset question type, the associated page is expanded based on the relevance ranking to obtain an expanded page; Using the multimodal model, the page image and corresponding text information of the expanded page are processed to generate the expanded description information of the seed page; Based on the expanded description information, a thought process chain corresponding to the seed page is regenerated.

8. The method according to claim 1, wherein, The step of generating a thought process chain corresponding to the seed page based on the description information, the question definition of the preset question type, and the question-answer instance of the preset question type, using a preset question-answer data generation model, includes: In response to using a preset question-and-answer data generation model, it is determined that the description information includes only one entity; Using a pre-defined question-and-answer data generation model, the jump relationship of the entity in the description information is determined based on the question definition and question-and-answer instances of the multi-hop question. Based on the jump relationship, a thought process chain corresponding to the seed page is generated.

9. The method according to claim 1, wherein, The step of generating a thought process chain corresponding to the seed page based on the description information, the question definition of the preset question type, and the question-answer instance of the preset question type, using a preset question-answer data generation model, includes: In response to using a preset question-and-answer data generation model, it is determined that the description information contains multiple entities; Using a pre-defined question-and-answer data generation model, the set relationship of multiple entities in the description information is determined based on the question definition of the set question and the question-and-answer instances of the set question; Based on the set relationship, a thought reasoning chain corresponding to the seed page is generated.

10. The method according to any one of claims 1-9, wherein, The set problems mentioned include the set intersection problem, the set union problem, and the set difference problem.

11. The method according to claim 1, further comprising: Based on the business domain and business objectives corresponding to the document, a page image is selected from the document as the seed page.

12. A method for training a question-answering model, comprising: Obtain document question-and-answer data samples, wherein the document question-and-answer data samples are constructed using the method described in any one of claims 1-11; The document question-and-answer data samples are input into the question-and-answer model to train the model.

13. A document question-and-answer data generation apparatus, comprising: The page extraction module is used to extract page content from page images in a document and obtain description information corresponding to the seed page in the document. The reasoning module is used to generate a thought reasoning chain corresponding to the seed page based on the description information, the question definition of the preset question type, and the question-answer instance of the preset question type, using a preset question-answer data generation model, so as to determine whether question-answer data corresponding to the preset question type can be generated based on the description information based on the thought reasoning chain; wherein, the preset question type includes multi-hop questions and / or set questions; The data generation module is used to respond to the thought reasoning chain forming a question thought chain corresponding to the preset question type, and based on the question thought chain, use the question and answer data generation model to generate question and answer data for the document and corresponding to the preset question type.

14. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the document question-and-answer data generation method according to any one of claims 1-11.

15. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the document question-and-answer data generation method according to any one of claims 1-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the document question-and-answer data generation method according to any one of claims 1-11.

Images