A question and answer pair augmentation method, apparatus, medium, and device

Abstract

The application discloses a question and answer pair augmentation method and device, medium and equipment, and relates to the technical field of computers. The application first performs semantic segmentation on a domain corpus, constructs a local and global semantic hypergraph of text, and constructs a cross-modal logical topology hypergraph in combination with visual analysis. On this basis, anchor points are determined and diversified question candidate groups are generated through multi-structure sampling. Finally, a search path is planned according to the complexity of the question, and corresponding predicted answers are generated, realizing a question and answer generation process that fuses multi-modal semantic structures. In the question and answer pair augmentation, the application constructs a three-layer complementary semantic structure including a macro semantic hypergraph, a local semantic hypergraph and a cross-modal logical topology hypergraph, realizes the collaborative modeling of global macro semantic correlation, local entity high-order relationship and cross-modal logical topology, effectively breaks down the "semantic island" and supports deep reasoning and multi-modal question and answer generation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a question-and-answer pair augmentation method, apparatus, medium, and device. Background Technology

[0002] Currently, question-answering augmentation techniques for specific domains (such as healthcare and law) mainly rely on simple text block segmentation followed by the generation of questions and answers using a Large Language Model (LLM). However, this method generates questions and answers based on only a single text block, resulting in overly superficial augmented content that fails to cover deep reasoning scenarios.

[0003] In existing technologies, Retrieval Augmented Generation (RAG) technology improves the response quality and domain adaptability of large language models by incorporating external knowledge, reducing illusion phenomena. To address the problem that traditional flattened text block-based retrieval cannot capture semantic relationships, related technologies use graph modeling to model the relationships between multiple entities in the text. Meanwhile, in question-answering tasks, existing technologies often employ question-answer pair augmentation to expand training samples and knowledge coverage. The basic process typically includes: First, extracting factual fragments, entity relationships, event descriptions, or topic summaries from the original corpus, document snippets, or existing labeled data; second, constructing multiple semantically similar but differently expressed questions around the same knowledge content based on template generation, rule rewriting, synonym rewriting, question reorganization, or automatic generation by a large language model, and matching corresponding answers to each question to form a question-answer pair set; then, further performing deduplication, filtering, consistency verification, and quality assessment on the generated question-answer pairs, eliminating low-quality samples with ambiguity, inconsistent answers, or incomplete reasoning chains; finally, using the augmented question-answer pairs for retrieval library construction, supervised fine-tuning, or retrieval machine training to improve the model's adaptability to different expression methods, different query granularities, and complex reasoning scenarios. While these methods can enhance knowledge recall and question-answering robustness to some extent, most still rely on textual semantic similarity as the core organizational method. They lack support for visual element associations, structured logical relationships, and multi-hop reasoning processes in cross-modal scenarios. Therefore, when facing complex text-image question-answering tasks, they still suffer from problems such as scattered retrieval evidence, incomplete reasoning chains, and insufficient stability in answer generation.

[0004] However, existing question-answering augmentation methods are limited to entity-level representations between text blocks, neglecting the global macro-semantic organization and alignment across text blocks. They lack a unified organization of macro-semantic knowledge and a semantic-driven reasoning mechanism, making it difficult to support hierarchical information integration from macro-level understanding to micro-level details. This results in problems such as "semantic silos," logical redundancy, and insufficient deep reasoning ability in the augmented question-answering data. Summary of the Invention

[0005] Therefore, it is necessary to provide a question-and-answer pair augmentation method, apparatus, medium, and device to address the aforementioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a question-answer pair augmentation method, comprising: Based on the semantic segmentation results of the original corpus in the target domain, a set of text blocks is generated; based on the semantic relationships of entities within the text blocks, a local semantic hypergraph is constructed; based on the macro-semantic associations between text blocks, a global semantic hypergraph is constructed. Visual elements in the original corpus are parsed to obtain visual semantic descriptions and visual feature fingerprints of the visual elements; a cross-modal logical topology hypergraph is constructed based on the visual semantic descriptions and visual feature fingerprints. Sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question anchor point, and generate question candidate groups by performing diversified sampling based on the question anchor point; Based on the complexity of each question in the question candidate group, the retrieval path based on the local semantic hypergraph and the global semantic hypergraph is determined and retrieved, and the predicted answer for each question in the question candidate group is generated.

[0007] This invention provides a question-and-answer pair augmentation device, comprising: The corpus preprocessing module is used to generate a set of text blocks based on the semantic segmentation results of the original corpus in the target domain; construct a local semantic hypergraph based on the semantic relationships of entities within the text blocks; and construct a global semantic hypergraph based on the macro-semantic relationships between text blocks. The multimodal hypergraph construction module is used to parse visual elements in the original corpus to obtain visual semantic descriptions and visual feature fingerprints of the visual elements; and to construct a cross-modal logical topology hypergraph based on the visual semantic descriptions and visual feature fingerprints. The question generation module is used to sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question generation anchor point, and generate question candidate groups based on the question generation anchor point through diversified sampling. The problem-solving module is used to determine and retrieve the search paths based on the complexity of each problem in the problem candidate group, using both local and global semantic hypergraphs, and to generate answers for each problem in the problem candidate group.

[0008] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described question-and-answer pair augmentation method.

[0009] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described question-and-answer pair augmentation method.

[0010] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: This invention first performs semantic segmentation on the domain corpus, constructing local and global semantic hypergraphs, and then combines visual parsing to construct a cross-modal logical topological hypergraph. Based on this, anchor points are determined through multi-structure sampling, and diverse question candidate groups are generated. Finally, retrieval paths are planned according to question complexity to generate corresponding predicted answers, realizing a question-answering generation process that integrates multi-modal semantic structures. In question-answer pair augmentation, this invention constructs a three-layer complementary semantic structure including a macro-level semantic hypergraph, a local semantic hypergraph, and a cross-modal logical topological hypergraph. This achieves collaborative modeling of global macro-level semantic associations, high-order local entity relations, and cross-modal logical topology, effectively breaking down "semantic silos" and supporting deep reasoning and multi-modal question-answering generation. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0012] Figure 1 This invention provides a flowchart illustrating a question-answer pair augmentation method. Figure 2 A schematic diagram of the overall framework of a question-answering pair augmentation method provided by the present invention; Figure 3 A schematic diagram of the framework of a question-generating agent reinforcement learning fine-tuning module provided by the present invention; Figure 4 A schematic diagram of the framework of a problem-solving agent reinforcement learning fine-tuning module provided by the present invention; Figure 5 This is a schematic diagram of a computer device for implementing a question-answering pair augmentation method provided by the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0014] Currently, question-answer pair augmentation for specific domains (such as healthcare and law) mainly relies on simple text block segmentation followed by LLM (Limited Language Modeling) to generate questions and answers. However, this approach has several problems. First, there is the issue of semantic silos. Existing methods generate questions and answers based on only a single text block, ignoring the global macro-semantic logic across text blocks, resulting in question-answer pairs lacking a macro perspective. Second, existing methods cannot capture high-order complex relationships between multiple entities, leading to augmented question-answer content that is too superficial and fails to cover deep reasoning scenarios. Finally, the generated question-answer pairs often contain logical redundancy or semantic deviations, lacking unified macro-semantic constraints.

[0015] Retrieval Enhancement Generation (RAG) technology improves the response quality and domain adaptability of large language models (LLMs) by incorporating external knowledge, reducing illusion phenomena. To address the problem that traditional flattened, text-block-based retrieval cannot capture semantic relationships, related technologies are mainly divided into two categories:

[0016] Graph-enhanced RAG methods (such as GraphRAG and LightRAG) construct knowledge graphs by extracting entities and binary relations between entities from text. They leverage graph structures to enhance entity-level indexing and retrieval, explicitly capturing semantic associations to optimize information organization, but they only focus on low-order pairwise relations between entities. Hypergraph-enhanced RAG methods (such as Hyper-RAG) model complex high-order relations between multiple entities using hyperedges, overcoming the limitation of traditional graphs that only support binary relations. However, these methods are still limited to entity-level representations between text blocks, neglecting global macro-level semantic organization and alignment across text blocks. They lack a unified organization of macro-level semantics and a semantic-driven reasoning mechanism, making it difficult to support hierarchical information integration from macro-level understanding to micro-level details.

[0017] The aforementioned existing technologies, when performing question-answer pair augmentation, have failed to achieve collaborative modeling of the global macro semantic structure and the high-order relationships of local entities, and the generation logic lacks top-down cognitive guidance, resulting in problems such as "semantic silos," logical redundancy, and insufficient deep reasoning ability in the augmented data.

[0018] Specifically, existing question-answer pairs generated by current methods exhibit a "semantic silo" phenomenon, lacking global relevance across text blocks. Existing question-answer pair augmentation methods primarily rely on simple text segmentation followed by LLM (Limited Graph Model) generation. Because existing techniques (including some hypergraph augmentation methods) neglect global macro-semantic logic and alignment across text blocks, the generated question-answer pairs are typically based only on local information from a single text block. This "semantic silo" problem results in augmented data lacking a macro-level perspective, failing to generate comprehensive questions requiring information from multiple paragraphs and chapters to answer, leading to deficiencies in the structure and coherence of the training data.

[0019] Secondly, the question-and-answer content is too superficial and fails to cover deep reasoning scenarios. Existing graph-augmented RAG methods mainly focus on low-order binary relationships between entities (such as GraphRAG), or although they introduce hypergraphs, they lack effective macro-level semantic organization (such as Hyper-RAG). This limitation means that when automatically generating question-and-answer pairs, the system cannot capture high-order complex relationships between multiple entities (such as causal chains, event co-occurrence, etc.). Therefore, the generated question-and-answer questions often remain at the level of simple fact queries, failing to cover complex scenarios requiring multi-hop reasoning or deep logical analysis, thus limiting the performance of downstream models in deep reasoning tasks.

[0020] Finally, the generated answers suffer from logical redundancy and semantic shifts, lacking unified macro-level semantic constraints. Existing technologies for question-and-answer generation often employ single-stage retrieval or direct generation, lacking top-down cognitive guidance and the constraints of a global macro-level semantic framework. This leads to logical redundancy in the generated answers, or deviations from the core macro-level semantics during the generation process (semantic shift). Particularly in highly structured professional fields such as medicine, the lack of precise alignment between global macro-level semantics and local details often results in question-and-answer pairs that fail to meet the requirements for building high-quality knowledge bases in terms of comprehensiveness, consistency, and logical coherence.

[0021] Furthermore, existing methods such as GraphRAG and Hyper-RAG perform retrieval enhancement and answer generation through fixed workflows because: 1. User questions in the dataset all involve queries to RAG databases, which are not inherent in the knowledge of the large model; 2. The retrieved RAG databases are relatively simple, containing only a single RAG database. However, these methods can lead to over-calling issues for simple questions or multiple / multi-layered RAG databases. For example, for a question that a large model can answer correctly, the operation of retrieving RAG databases can greatly affect the model's response time, increase the cost for users, and negatively impact the user experience.

[0022] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Figure 1 This is a schematic diagram of a question-answering pair augmentation method according to the present invention, which specifically includes the following steps: S101: Generate a set of text blocks based on the semantic segmentation results of the original corpus in the target domain; construct a local semantic hypergraph based on the semantic relationships of entities within the text blocks; and construct a global semantic hypergraph based on the macro-semantic associations between text blocks.

[0024] S102: Parse the visual elements in the original corpus to obtain the visual semantic description and visual feature fingerprint of the visual elements; construct a cross-modal logical topology hypergraph based on the visual semantic description and visual feature fingerprint.

[0025] S103: Sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question anchor point, and generate question candidate groups by performing diversified sampling based on the question anchor point.

[0026] S104: Based on the complexity of each question in the question candidate group, determine and retrieve the retrieval path based on the local semantic hypergraph and the global semantic hypergraph, and generate the predicted answer for each question in the question candidate group.

[0027] For ease of explanation, the following description focuses solely on the server as the executing entity. The server mentioned in this invention can be a server set up on a business platform, or a device such as a desktop computer or laptop computer capable of executing the solution of this invention.

[0028] This invention proposes an automated question-and-answer pair augmentation method and system based on hypergraph semantic alignment. To ensure that the augmented question-and-answer pairs go beyond surface-level text, this scheme constructs a three-layer complementary semantic structure: **Global Semantic Question Answering (Global QA):** Captures the narrative thread across text blocks (e.g., summarizing the core logic of the entire text) through a "global semantic hypergraph," supporting the generation of macro-summary and panoramic question-and-answer pairs. **Local Reasoning Question Answering (Local Reasoning QA):** Models high-order complex relationships between entities (e.g., causal chains, event co-occurrence) using a "local semantic hypergraph," automatically generating deep question-and-answer pairs requiring multi-hop reasoning. **Multimodal Logic Question Answering (Multimodal QA):** Transforms flowcharts and diagrams into structured units and incorporates them into the hypergraph using OCR and VLM, thereby supporting the generation of richly illustrated question-and-answer pairs such as "operational logic based on flowcharts" or "data comparison based on diagrams."

[0029] In the question-answer pair augmentation stage of retrieval and generation, this scheme introduces a reinforcement learning fine-tuning framework based on the multi-agent-GRPO (Group Relative Policy Optimization) algorithm. This framework uses a multi-agent approach to jointly optimize both the agent that generates the questions and the agent that solves them, ensuring the accuracy of the augmented question-answer pairs while simultaneously optimizing their diversity and difficulty.

[0030] The question-and-answer pair generation agent, within the multi-agent collaborative framework, aims to generate question-and-answer pairs that deeply cover hypergraph knowledge, possess high-difficulty reasoning capabilities, and exhibit diverse distributions. Its optimization process is primarily based on the GRPO algorithm, achieving self-evolution in a discriminator-free network through an intra-group relative scoring mechanism. Multi-granularity anchor extraction: The question-and-answer agent first randomly samples key entities, higher-order hyperedges, or visual logic nodes as "question anchors" from the "macro-semantic hypergraph," "local semantic hypergraph," and "cross-modal logical topology hypergraph" based on the preprocessed text blocks. Intra-group diversified sampling: An adaptive temperature adjustment mechanism is used to generate a group of candidate questions for the same knowledge anchor. The optimization of the question-and-answer agent is driven by a specially designed reward function to ensure the quality of the generated question-and-answer pairs: Semantic alignment reward: Calculates the matching degree between the generated question and higher-order relationships (such as causal chains and event co-occurrence) in the hypergraph. If the question successfully triggers a multi-hop logical path in the hypergraph, a high reward is given to prevent the question from becoming merely a superficial fact. Difficulty Reward: This reward is determined by the feedback from the "problem-solving agent." This scheme selects problems where the problem-solving agent's accuracy is between (0, 0.9) for reinforcement training. Diversity Gain Reward: By calculating the semantic vector similarity of problems within a group, it encourages the generation of problems that differ significantly from the known problem library, avoiding logical redundancy in augmented data.

[0031] The question-and-answer pair solving agent, through the establishment of a multi-dimensional reward system including format rewards, mixed answer quality rewards, call frequency rewards, and retrieval relevance rewards, induces the model to evolve into a "top-down" cognitive decision-making ability similar to that of a human expert: Adaptive strategy selection: The model can autonomously determine the calling path of the two-layer hypergraph library based on the complexity of the question (such as isolated fact type, macro summary type, or deep reasoning type), effectively avoiding excessive calls to the RAG library and reducing system response time; Explainable reasoning generation: When generating question-and-answer pairs, the system first establishes a macro semantic framework, then fills in micro factual details, and forces the output of a structured response containing thought chains (CoT) (such as... <think>The tag content ensures that the generated question-and-answer pairs have rigorous logic, multi-hop reasoning ability, and global factual consistency.

[0032] 1. System Overall Architecture This system consists of several core modules: a corpus preprocessing module, a multimodal semantic extraction module, a multimodal hypergraph construction module, and a retrieval enhancement module based on reinforcement learning fine-tuning. These modules work collaboratively through a data transmission interface. The overall architecture is as follows: Figure 1 As shown.

[0033] Corpus preprocessing module: responsible for cleaning, segmenting and format standardization of raw corpus, providing basic text units for subsequent hypergraph construction; Multimodal semantic extraction module: Utilizes OCR and visual language model (VLM) to extract deep information of visual elements in documents and transforms them into structured text descriptions to compensate for the lack of recognition of complex logic diagrams by pure text processing.

[0034] The multimodal hypergraph construction module includes a macro-semantic hypergraph construction submodule and a local semantic hypergraph construction submodule, which respectively complete the modeling of macro-semantic associations across text blocks and the modeling of high-order local semantic relationships within text blocks. When constructing the hypergraph, hyperedges are allowed to connect text vertices with image / chart vertices. For example, a "macro-semantic hyperedge" can simultaneously associate a surgical description and its corresponding surgical diagram, thereby supporting the generation of richly illustrated question-and-answer pairs.

[0035] A question-answering pair enhancement module fine-tuned using multi-agent reinforcement learning. Utilizing the GRPO (Group Relative Policy Optimization) algorithm, without relying on a complex critic network, it automatically optimizes the probability distribution of macro / local keyword extraction and the selection strategy of retrieval paths through a relative scoring mechanism within groups.

[0036] The overall system architecture's data flow is designed as follows: The corpus preprocessing module first cleans and semantically segments the raw data, and the output text blocks are transmitted to the hypergraph construction module as basic units; the hypergraph data generated by the construction module is persistently stored in the graph database; after the global semantic hypergraph and local semantic hypergraph are constructed, this method uses the GRPO algorithm to automatically optimize the LLM to select whether to query the global semantic hypergraph or the local semantic hypergraph, and then selects different calling strategies for different questions: for deep reasoning and detailed verification questions, the global semantic hypergraph is queried first, and then the local semantic hypergraph is retrieved; for macro-summary and panoramic scanning questions, only the global semantic hypergraph needs to be retrieved; for isolated fact query questions, only the local semantic hypergraph needs to be retrieved. This reduces the overall system response time while ensuring accuracy. Table 1 shows the key features of each module mentioned above.

[0037] Table 1 Key features of each module 2. Detailed Design of Core Modules (1) Corpus preprocessing module Existing technologies often employ fixed-length, non-overlapping segmentation or segmentation based solely on character count, which can easily lead to semantic fragmentation. This method uses a hybrid segmentation strategy combining semantic awareness and a fixed window, ensuring both segmentation efficiency and maximizing the preservation of text block semantic integrity. The specific steps are as follows:

[0038] Corpus cleaning: The original corpus (such as books, reports, medical manuals, etc.) is deduplicated and noise-reduced, invalid information (such as whitespace characters, repeated paragraphs, format marks) is removed, and the text encoding format (UTF-8) is unified. Semantic breakpoint detection: Using natural language processing tools (such as spaCy's syntactic analyzer) to identify sentence boundaries, paragraph boundaries, and semantic transition keywords (such as "however," "therefore," "in conclusion") in the text, and marking potential semantic breakpoints; Hybrid segmentation: A fixed-length sliding window with partial overlap is used for segmentation. The window length is set to 512 characters (adaptively adjustable according to the corpus type), and the overlap rate is 20%. During segmentation, if a semantic breakpoint is detected to coincide with the window boundary, the window length is automatically adjusted by ±128 characters to ensure that semantically complete units are not split.

[0039] Text block standardization: Add a unique identifier (Chunk-ID), source document identifier (Doc-ID), and position information (start / end character index) to each text block, and store the original content and preprocessed content of the text block to form a standardized text block set.

[0040] This module combines semantic breakpoint detection with fixed window segmentation, solving the semantic fragmentation problem caused by segmentation in existing technologies and providing a semantically coherent basic unit for subsequent macroscopic semantic hypergraph construction.

[0041] (2) Multimodal analysis module To address the limitations of existing technologies that rely solely on text descriptions, leading to the loss of visual details and their inability to handle images with similar visual styles but significantly different text descriptions or without text annotations, this method designs a dual-track multimodal parsing module. This module not only utilizes Visual Modeling (VLM) for textual logic extraction but also directly models the visual features of the image for similarity through a visual encoder. The specific steps are as follows:

[0042] The visual element recognition and multidimensional feature extraction system automatically locates visual elements (such as charts, flowcharts, and photographs) in documents during the corpus preprocessing stage. This module first uses a visual language model (such as Qwen-2.5-VL or GPT-4o) to classify and semantically parse the elements. Simultaneously, a pre-trained visual encoder (such as CLIP-VIT-L-14) is introduced to extract high-dimensional feature vectors (visual embeddings) from the images, capturing native visual features such as color, shape, texture, and spatial topology.

[0043] Semantic logical structured path. Visual Modeling (VLM) is used to reason about the deep logic within images. For flowcharts, logical transitions between nodes (e.g., "Node A -> Decision B") are parsed and a structured logical sequence is output. For statistical charts, axis data, trend changes (e.g., "steady growth"), and key extreme points are extracted and converted into descriptive text blocks. Finally, alignment verification is performed using text tags extracted from OCR to ensure the accuracy of professional terminology.

[0044] A direct modeling path for visual similarity. This module constructs a visual similarity matrix by calculating the cosine similarity or Euclidean distance between feature vectors of different visual elements. Cross-graph visual clustering: Images with high visual similarity (such as photos of the same medical device from different angles, or financial statements from different years but with the same format) are identified as "visually equivalence classes" or "visually related clusters". Visual association hyperedge generation: When the distance between two images in the visual feature space is less than a set threshold, the system automatically generates a "visually similar hyperedge" and includes it in the hypergraph index, even if their contextual text blocks are not directly related. This enables the system to answer questions based on visually intuitive comparison, such as "finding cases with similar operational logic to this surgical diagram".

[0045] The cross-modal semantic-feature fusion output module binds the parsed structured visual description (semantic dimension) with the corresponding feature vector (feature dimension), assigning a unique Chunk-ID. The output not only includes the image's "textual description" but also a "visual feature fingerprint" and a "list of associated images" that are visually similar. This information is then output to the "multimodal hypergraph construction module" to build cross-modal logical topological relationships.

[0046] The technical effects and innovations of this module: This solution overcomes the limitations of traditional RAG solutions that rely solely on "converting everything to text" by introducing "direct visual similarity modeling." It not only preserves the symbolic logic (semantic path) of images but also their signal features (feature path), solving the retrieval distortion problem caused by ambiguous text descriptions. In the question-answer generation stage, this allows the model to perform dual retrieval comparisons based on both "what it looks like" and "what the text says," significantly improving the accuracy and diversity of question-answer pair augmentation in complex visual scenarios such as medical image analysis and industrial drawing comparison.

[0047] (3) Multimodal hypergraph semantic index structure Unlike existing technologies that only model single entity relationships or local high-order relationships, this method constructs hypergraphs with three complementary semantic granularities to achieve collaborative modeling of global macro-semantics, local entity relationships, and cross-modal logical topology: Global Semantic Hypergraph Module: Used to capture macro-level semantic relationships across text blocks (such as storylines, narrative frames, summaries, etc.), providing global macro-level semantic organization.

[0048] Construction Method / Steps: The corpus (such as books, reports, and manuals) is segmented into a set of text blocks using a fixed-length sliding window with partial overlap; Large Language Models (LLMs) are used in conjunction with preset extraction prompts to perform semantic parsing on each text block, automatically extracting potential macro-semantics and related key entities; a macro-semantic hypergraph is constructed with key entities as vertices and the narrative macro-semantics of the text blocks as hyperedges. ,in For the set of key entities, It is a set of macro-semantic hyperedges.

[0049] Local Semantic Hypergraph Module: Used to model fine-grained high-order semantic relationships between entities within text blocks, supporting accurate capture of local semantic associations.

[0050] Construction process: For each text block, use LLMs combined with entity extraction prompts to extract entities (such as people, concepts, events, organizations, etc.) and their descriptions as a vertex set. Based on the semantic relationships between entities, two types of hyperedges are constructed: low-order hyperedges. (Capturing fundamental pairwise relations) and higher-order hyperedges (Model complex semantic relationships between multiple entities, such as event co-occurrence, causal links, etc.); organize entities, low-order hyperedges, and high-order hyperedges into local semantic hypergraphs and store them in a hypergraph database.

[0051] Cross-modal logic topology hypergraph: Specifically stores non-linear logic jump relationships extracted from flowcharts, schematic diagrams, and structure diagrams.

[0052] Construction process: Based on the structured logical semantic sequence in the visual semantic description of visual elements, determine the visual logical vertices and the logical path hyperedges between them; when the visual element is a real-world image, use the target object, component object, spatial location object, and attribute object as visual logical vertices, and determine the visual logical vertices based on composition relationships, adjacency relationships, attribute constraints, or the feature space of visual feature fingerprints. Nearest neighbor relationships are used to construct object relationship hyperedges; for statistical charts, association relationships are constructed based on the similarity between the text descriptors of the statistical chart names and the text descriptors in the text blocks; the text blocks corresponding to the visual logical vertices are determined as text semantic vertices, and visual similarity hyperedges are established based on the similarity between visual feature fingerprints, and cross-modal semantic mapping hyperedges are established based on the correspondence between visual logical vertices and text semantic vertices; a cross-modal logical topology hypergraph is constructed based on visual logical vertices, text semantic vertices, logical path hyperedges, statistical relationship hyperedges, object relationship hyperedges, visual similarity hyperedges, and cross-modal semantic mapping hyperedges, and stored in the hypergraph database.

[0053] (5) Reinforcement learning fine-tuning of the retrieval enhancement module of the question-generating agent The core objective of the question-generating agent is to generate question-answer pairs with deep coverage, high-difficulty reasoning capabilities, and diverse distribution. This module abandons the traditional fixed-template generation method and adopts a fine-tuning framework based on the Group Relative Policy Optimization (GRPO) algorithm to achieve self-play and policy iteration without a discriminator network. Through an intra-group relative scoring mechanism, the question-generating agent can spontaneously learn how to extract logical anchors from complex hypergraph topologies and transform them into high-quality question-answer data.

[0054] Multi-granularity anchor point dynamic extraction: The question-generating agent receives preprocessed text blocks and a constructed three-layer hypergraph as input. It extracts narrative context from the "macro-semantic hypergraph," higher-order entity relationships (such as causal chains and event co-occurrence) from the "local semantic hypergraph," and visual logic nodes from the "cross-modal logical topology hypergraph" as question-generating anchor points. Diverse sampling strategy within groups: For the same knowledge anchor point, the system uses an adaptive temperature adjustment mechanism for sampling. By dynamically adjusting the sampling temperature within the model's controllable exploration range, the agent generates multiple question candidate groups with distinct logical paths in a single iteration, ensuring the model can explore a broad potential reasoning space and avoid falling into a single question-generating pattern.

[0055] To induce the question-generating agent to develop expert-level thinking logic, this module constructs a four-dimensional joint reward function: Semantic alignment reward: Used to evaluate the fit between the generated question and the higher-order relationships in the hypergraph. If the question successfully reaches a multi-hop retrieval path or cross-module logic in the hypergraph, a high-weight reward is given to prevent the generated question from being merely a superficial fact.

[0056] Diversity Gain Reward: By calculating the semantic vector similarity between multiple questions generated within the same group, rewarding question statements with high differences, thereby effectively suppressing semantic redundancy in augmented data.

[0057] Cross-modal fusion rewards: Specifically designed for corpora containing charts and flowcharts, rewards are given for questions that require a combination of "visual logic nodes" and "textual explanation blocks" to answer accurately, ensuring that the augmented question-answer pairs have true multimodal reasoning capabilities.

[0058] (5) Reinforcement learning fine-tuning of the retrieval enhancement module of the problem-solving agent This module aims to optimize the retrieval and generation process in a multimodal hypergraph environment through reinforcement learning (RL). By introducing a group relative policy optimization algorithm (GRPO), the system can adaptively select the optimal retrieval path between the macro-semantic hypergraph and the local semantic hypergraph based on the complexity of different questions, and generate high-quality question-answer pairs with interpretable reasoning chains (CoT).

[0059] The specific process of data complexity partitioning is as follows: The first step is to directly input the question into the large model. If the large model can answer it correctly, the question has the first complexity; otherwise, proceed to the second step.

[0060] The second step is to input the problem and all global hypergraph content into the large model. If the large model can answer correctly, the problem has a second complexity; otherwise, proceed to the third step.

[0061] The third step is to input the problem and all local hypergraph content into the large model. If the large model can answer correctly, the problem has a third complexity; otherwise, proceed to the fourth step.

[0062] The fourth step is to input the problem, all global hypergraphs, and all local hypergraphs into the large model. If the large model can answer the question correctly, the problem has a fourth complexity; otherwise, the problem is discarded.

[0063] In each step, the temperature is set to 1, and the large model is asked to answer N times. If it answers correctly even once, it is considered to be able to answer correctly; otherwise, it is considered to be unable to answer.

[0064] Data type selection for reinforcement learning. In the reinforcement learning process, the first step is to process the data used. We first divide all questions into four categories using a workflow: 1. Questions that the large model can answer correctly without any RAG library retrieval; 2. Questions that the large model can answer correctly by retrieving the global semantic hypergraph; 3. Questions that the large model can answer correctly by retrieving the local semantic hypergraph; 4. Questions that the large model needs to retrieve both the global and local semantic hypergraphs to answer correctly. During reinforcement learning training, the proportion of each category of questions is controlled at 1:1:1:1.

[0065] Data difficulty screening for reinforcement learning. In order to select the most valuable problems for optimization in reinforcement learning, this method sets up a corresponding RAG workflow for each type of problem, selects problems with an accuracy in the range of (0-0.9) for reinforcement learning training, and excludes data with an accuracy greater than 0.9 and an accuracy of 0.

[0066] Reinforcement learning uses the GRPO algorithm for optimization, and the optimized reward function is: Formatting Reward: Checks whether the model's output conforms to a standard discriminant format. That is, the model's thought process... <think>< / think> In the tag, the model's rag call process is located in <tool>< / tool> In the tags,

[0067] Hybrid answer reward: The objective is to determine whether the model's output matches the standard answer. The fluency and factual accuracy of the generated answer are evaluated using Bartscore or a similar metric. Simultaneously, this method employs an LLM-as-judge approach, using the qwen3-30B-A3B model to judge the correctness of the model's output based on the standard answer.

[0068] Call Reward. The goal of this method is to achieve adaptive RAG retrieval library calls through reinforcement learning. Therefore, while ensuring the correctness of the task, it is expected that the LLM will make the fewest calls while answering the user's questions, hence the introduction of a call reward.

[0069] Relevance Reward. To measure the effectiveness of large models in retrieving the RAG library during inference, this method trains a reward model to evaluate the relevance between the content retrieved by the large model through keywords and the correct answer, encouraging the model to generate the most suitable keywords.

[0070] Adaptive Temperature Regulation. Diversity is a key factor in GRPO training. The core of GRPO lies in comparing positive and negative trajectories to increase the probability of positive trajectories appearing. The diversity of sampled trajectories is crucial, mainly in two aspects: high diversity prompts the model to produce both positive and negative trajectories simultaneously during a single trajectory generation, thereby strengthening the contrast effect between trajectories; high diversity allows the model to explore a wider range of potential inference paths, helping to avoid the model becoming overconfident in a narrow set of patterns too quickly. Therefore, during the GRPO algorithm training process, this scheme adaptively adjusts the temperature coefficient to the sampling temperature based on the model's controllable exploration range, thereby improving diversity while ensuring consistent performance.

[0071] (7) Question-generating Agent-Solving Agent Self-evolutionary Framework To achieve a self-evolving framework, the first step is to optimize the parameters of the problem-solving agent using the GRPO algorithm while keeping the weights of the problem-generating agent fixed. Then, while keeping the parameters of the problem-solving agent fixed, the parameters of the problem-generating agent are optimized to achieve the self-evolution of the entire system.

[0072] System process execution steps The complete execution flow of this system includes the following steps, forming a closed loop from corpus preprocessing to answer generation.

[0073] Step 1: Corpus Input and Preprocessing. The raw corpus is input into the corpus preprocessing module, where it undergoes cleaning, semantic breakpoint detection, hybrid segmentation, and standardization to generate a set of text blocks.

[0074] Step 2: Multimodal Analysis. The multimodal analysis module is used to locate visual elements in the document. Logical transitions and data trends in charts and flowcharts are extracted using OCR and a Visual Language Model (VLM). At the same time, image feature vectors are extracted using a visual encoder to generate structured visual semantic descriptions and visual feature fingerprints.

[0075] Step 3: Hypergraph Construction. Input the text block set into the dual hypergraph construction module. The macro-semantic hypergraph construction submodule generates a macro-semantic hypergraph through macro-semantic extraction, key entity extraction, and hyperedge construction. The local semantic hypergraph construction submodule generates a local semantic hypergraph through entity extraction, low-order / high-order hyperedge construction, and cross-text block integration. And store the dual hypergraphs in the hypergraph database;

[0076] Step 4: Question-Answer Pair Sampling and Generation by the Question Generator. The question generator randomly samples key entities or logical nodes from the three-layer hypergraph as "question anchors," and uses an adaptive temperature adjustment mechanism to perform diversified sampling within groups, generating candidate question groups covering macro-summary, deep reasoning, and multimodal logic types. It then fine-tunes and optimizes these groups based on GRPO reinforcement learning.

[0077] Step 5: Adaptive Retrieval and Problem Solving by the Problem-Solving Agent. The problem-solving agent autonomously decides the retrieval path based on the complexity of the problem and fine-tunes it using reinforcement learning based on GRPO. Utilizing the Group Relative Policy Optimization (GRPO) algorithm, the generated trajectory is scored according to a multi-dimensional reward system (format reward, mixed answer reward, call count reward, retrieval relevance reward, semantic alignment reward, etc.). The system filters high-value data with accuracy between (0, 0.9) and automatically optimizes model parameters through a group relative scoring mechanism.

[0078] Step 6: Agent Collaborative Self-Evolution and Augmented Output. The system employs an alternating optimization strategy, training the question-generating agent and the question-solving agent alternately while keeping one side's parameters fixed, thus achieving the framework's self-evolution. The final output is a high-quality set of automated augmented question-answer pairs with high comprehensiveness, strong logic, and low redundancy.

[0079] Specifically, step 2, extracting visual feature fingerprints based on the type of visual element, includes: First, the visual elements undergo image preprocessing, including resolution unification, noise filtering, and edge enhancement. Second, pre-trained visual encoding models (CLIP, ViT, Qwen3VL-emb, etc.) are used to encode the visual elements, obtaining fixed-dimensional visual feature vectors. Then, the visual feature vectors are normalized to generate visual feature fingerprints. Finally, the visual feature fingerprints are bound to the identifiers of the corresponding visual elements and stored in a multimodal index database, where the visual feature fingerprint is a normalized vector of length d. If the visual image is a flowchart or statistical chart, the names of the flowcharts or charts are encoded using pre-trained text encoding models (Bert, Sentence Bert, Qwen3VL-emb, etc.).

[0080] Specifically, step 4 involves generating specific candidate problem groups, including: Based on the text block set, event units, topic units, entity units, and visual units are extracted; narrative paths are extracted from the global semantic hypergraph according to the temporal order, causal relationship, or chapter subordination relationship between event units; entity relationship subgraphs are extracted from the higher-order hyperedges in the local semantic hypergraph; visual logic paths or visual similarity clusters are extracted from the cross-modal logical topology hypergraph; the narrative paths, entity relationship subgraphs, or visual logic paths are encoded into a unified anchor point representation to obtain question anchor points, wherein the question anchor points include at least an anchor point identifier, an anchor point type, a set of associated vertices, a set of associated hyperedges, and a source text block identifier; for the same question anchor point, diverse sampling within the group is performed based on preset template constraints, sampling temperature, and historical question similarity thresholds to generate multiple semantically different questions with verifiable answers, forming a question candidate group. Specifically, firstly, the agent that generates questions randomly samples a subgraph on the constructed hypergraph. The subgraph includes macroscopic nodes and local semantic nodes. The corresponding macroscopic nodes, macroscopic hyperedges (which constitute the overall narrative context), internal microscopic entity nodes, and internal entity hyperedges (which constitute the participating entities and the relationships between entities) are used as anchor points to be input into the agent to generate question pairs.

[0081] Specifically, the solution process in step 5 includes: Under the premise of freezing the problem-solving model, for each problem in the generated problem candidate group, the semantic alignment reward is determined based on the alignment degree between the generated problem and the input topic, entity, and path. Specifically: 1. The topic alignment reward is calculated by calculating the similarity between the semantic vectors of the generated problem and the candidate topic; 2. In addition, relevant entities are determined by calculating the similarity between the problem and multiple entity texts that is greater than a threshold. The more relevant entities involved and the more hyperedges involved, the more complex the proof problem is, and the number of relevant entities and the number of hyperedges involved are used as the entity alignment reward; 3. Based on 2, the maximum depth of the hypergraph formed by the relevant entities and hyperedges is calculated as the reasoning path length of the problem, and the reasoning length is used as the reward. Problems with larger reasoning lengths, i.e., problems that require more reasoning steps, are rewarded.

[0082] Based on the semantic vector similarity between the question and other questions in the question candidate group, the overall difference between the question and other questions in the question candidate group is determined to determine the diversity gain reward of the question; the overall difference is positively correlated with the diversity gain reward of the question; The cross-modal fusion reward for a problem is determined based on whether it simultaneously references visual logical vertices and corresponding textual semantic vertices, and whether it requires combining visual similarity hyperedges or cross-modal semantic mapping hyperedges to obtain the standard answer. Specifically, the same problem-solving model is used to answer the same problem under three different conditions: Condition 1: only text input; Condition 2: only image input; Condition 3: both image and text input. If a problem is cross-modal, then viewing an image or text alone should be less effective than viewing both simultaneously. The specific approach is to answer the same problem under three different input conditions. n The accuracy rate of text-only responses is recorded as follows: The accuracy rate of responses based solely on image input is denoted as... The accuracy of mixed inputs is denoted as: The final reward is recorded as follows: - ( + ) / 2.

[0083] The question reward is determined based on the semantic alignment reward, diversity gain reward, and cross-modal fusion reward of the question. The generation strategy of the question generation model is updated by optimizing the group relative strategy with the optimization objective of maximizing the relative average reward within the same question candidate group. The generation strategy includes at least the anchor point selection probability distribution, the question template selection probability distribution, and the sampling temperature control parameter.

[0084] The problem-solving model is further refined, specifically including: Under the premise of freezing the question generation model, for the predicted answer of each question in the generated question candidate group, the format of the predicted answer is determined according to the preset problem-solving format template and the model output, and the format reward is determined. The mixed answer reward for the generated answer is determined based on the deviation between the predicted answer and the preset standard answer for the question. The reward for generating the predicted answer is determined based on the number of retrieval calls made by the problem-solving model; the number of calls and the reward for generating the answer are negatively correlated. The retrieval relevance reward for the predicted answer is determined based on the relevance between the keywords used in the retrieval process of generating the predicted answer by the problem-solving model and the preset standard answer of the question; relevance and relevance reward are positively correlated. The answer reward for the predicted answer is determined based on the format reward, mixed answer reward, call count reward, and retrieval relevance reward. The problem-solving strategy of the problem-solving model is updated by optimizing the group relative strategy based on the answer rewards of each question in the generated question candidate group.

[0085] based on Figure 1 The question-answering pair augmentation method presented in this invention first performs semantic segmentation on the domain corpus, constructs local and global semantic hypergraphs, and combines visual parsing to construct a cross-modal logical topological hypergraph. Based on this, anchor points are determined through multi-structure sampling, and diverse question candidate groups are generated. Finally, retrieval paths are planned according to question complexity to generate corresponding predicted answers, realizing a question-answering generation process that integrates multi-modal semantic structures. In question-answering pair augmentation, this invention constructs a three-layer complementary semantic structure including a macro-level semantic hypergraph, a local semantic hypergraph, and a cross-modal logical topological hypergraph. This achieves collaborative modeling of global macro-level semantic associations, high-order local entity relations, and cross-modal logical topology, effectively breaking down "semantic silos" and supporting deep reasoning and multi-modal question-answering generation.

[0086] The present invention provides an automated question-answering pair augmentation method and system based on reinforcement learning for hypergraph semantic alignment. This method and system have significant technical effects in improving the quality of question-answering pairs, enhancing deep reasoning capabilities, and optimizing system operating efficiency.

[0087] First, this solution effectively addresses the "semantic silo" problem in existing technologies by constructing a three-layer complementary hypergraph structure that includes macro-level semantics, local semantics, and cross-modal logic. This collaborative modeling approach can capture the global narrative context and high-order complex relationships (such as causal chains and event co-occurrence) across text blocks, enabling the augmented question-answer pairs to go beyond the surface facts of a single text block and possess a macro-level perspective and multi-hop reasoning depth. Furthermore, the system mandates the output of structured responses containing thought chains (CoT), ensuring that the generated question-answer pairs have rigorous logic and consistency with global facts.

[0088] At the algorithmic level, this scheme introduces a reinforcement learning fine-tuning framework based on the multi-agent-GRPO (Group Relative Policy Optimization) algorithm, realizing a self-evolutionary game between the question-generating agent and the question-solving agent. Without relying on a complex commentator network, this framework significantly improves the distribution diversity and inference difficulty of the augmented data while ensuring correctness through an intra-group relative scoring mechanism, adaptive temperature adjustment, and a four-dimensional joint reward function (semantic alignment, difficulty evolution, diversity gain, and cross-modal fusion), effectively suppressing logical redundancy and semantic shift.

[0089] In terms of system efficiency and adaptability, the problem-solving agent demonstrates top-down cognitive decision-making capabilities reminiscent of human experts. Through adaptive strategy selection, the system can autonomously determine the calling path to the two-layer hypergraph library based on the complexity of the problem (e.g., isolated facts, macro-level summaries, etc.). This mechanism effectively avoids excessive calls to the RAG library, significantly reducing the overall system response time and user costs while ensuring high accuracy.

[0090] Finally, regarding the processing of multimodal information, this scheme introduces "direct visual similarity modeling" through a dual-track multimodal parsing module, overcoming the retrieval distortion limitations caused by the traditional "everything is converted to text" approach. By preserving the feature paths and semantic paths of images, the system can perform dual retrieval by combining visual intuitive comparison with textual descriptions, significantly improving the augmented accuracy of question-answer pairs in complex visual scenarios such as medical imaging and industrial drawing. Experimental data show that this method outperforms existing technologies such as NaiveRAG and GraphRAG in multiple dimensions, including comprehensiveness (87.0), logicality (87.5), and overall score (85.39).

[0091] The key technical point of this invention is to construct a three-layer complementary semantic structure that includes a macro-semantic hypergraph, a local semantic hypergraph, and a cross-modal logical topology hypergraph. This enables collaborative modeling of global macro-semantic associations, local entity high-order relations, and cross-modal logical topology, effectively breaking down "semantic silos" and supporting deep reasoning and multimodal question answering generation.

[0092] This innovative approach employs a hybrid segmentation strategy combining semantic awareness and fixed windows. This maximizes the preservation of semantic integrity of text blocks while ensuring segmentation efficiency, providing semantically coherent foundational units for hypergraph construction. A dual-track multimodal parsing module is designed, using Visual Modeling (VLM) to extract the semantic logic structure of visual elements. Simultaneously, a visual encoder extracts high-dimensional feature vectors from images for direct visual similarity modeling, overcoming the limitations of traditional "everything is converted to text" methods and improving the accuracy of question-answering augmentation in complex visual scenarios. A reinforcement learning fine-tuning framework based on the multi-agent-GRPO algorithm is introduced to construct a self-evolving game system between a question-generating agent and a question-solving agent. The question-generating agent generates question-answer pairs with deep coverage, high-difficulty reasoning, and diverse distribution through dynamic extraction of multi-granularity anchor points and diversified sampling strategies within groups, combined with a four-dimensional joint reward function of semantic alignment, difficulty evolution, diversity gain, and cross-modal fusion. The question-solving agent autonomously optimizes the hypergraph call path through a multi-dimensional reward system and an adaptive strategy selection mechanism, avoiding excessive calls and generating structured responses containing thought chains. By alternately optimizing the parameters of the question-generating agent and the question-solving agent, the entire system achieves self-evolution. At the same time, the data type and difficulty are accurately screened during the reinforcement learning process to further ensure the quality of augmented question-answer pairs and the efficiency of system optimization. Ultimately, the comprehensive improvement of question-answer pair augmentation in terms of comprehensiveness, logic, reasoning depth, and system operating efficiency is achieved.

[0093] When applying the question-and-answer pair augmentation method provided by this invention, it is not necessary to... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this invention does not impose any restrictions on it.

[0094] The above describes a question-answer pair augmentation method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding question-answer pair augmentation device, including: The corpus preprocessing module is used to generate a set of text blocks based on the semantic segmentation results of the original corpus in the target domain; construct a local semantic hypergraph based on the semantic relationships of entities within the text blocks; and construct a global semantic hypergraph based on the macro-semantic relationships between text blocks. The multimodal hypergraph construction module is used to parse visual elements in the original corpus to obtain visual semantic descriptions and visual feature fingerprints of the visual elements; and to construct a cross-modal logical topology hypergraph based on the visual semantic descriptions and visual feature fingerprints. The question generation module is used to sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question generation anchor point, and generate question candidate groups based on the question generation anchor point through diversified sampling. The problem-solving module is used to determine and retrieve the search paths based on the complexity of each problem in the problem candidate group, using both local and global semantic hypergraphs, and to generate answers for each problem in the problem candidate group.

[0095] For specific limitations regarding the question-and-answer pair augmentation device, please refer to the limitations of the question-and-answer pair augmentation method above, which will not be repeated here. Each module in the aforementioned question-and-answer pair augmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0096] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided Q&A approach enhances the method.

[0097] The present invention also provides Figure 5 The schematic diagram of the computer device shown is as follows: Figure 5 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 The provided Q&A pairs enhance the approach.

[0098] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.< / think>

Claims

1. A question-and-answer pair augmentation method, characterized in that, include: Based on the semantic segmentation results of the original corpus in the target domain, a set of text blocks is generated; Based on the semantic relationships of entities within text blocks, construct a local semantic hypergraph; based on the macro-semantic relationships between text blocks, construct a global semantic hypergraph. The visual elements in the original corpus are analyzed to obtain the visual semantic description and visual feature fingerprint of the visual elements; Construct a cross-modal logical topology hypergraph based on visual semantic description and visual feature fingerprint; Sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question anchor point, and generate question candidate groups by performing diversified sampling based on the question anchor point; Based on the complexity of each question in the question candidate group, the retrieval path based on the local semantic hypergraph and the global semantic hypergraph is determined and retrieved, and the predicted answer for each question in the question candidate group is generated.

2. The question-and-answer pair augmentation method as described in claim 1, characterized in that, The step of generating a set of text blocks based on the semantic segmentation results of the original corpus in the target domain specifically includes: The original corpus of the target domain is obtained, and the original corpus is deduplicated and noise-reduced. The text encoding format is then unified to obtain the cleaned corpus text. Identify sentence boundaries, paragraph boundaries, and semantic turning points in the corpus text, and mark semantic breakpoints in the corpus text; The text corpus is segmented by partially overlapping sliding windows. When the semantic breakpoint coincides with the window boundary, the window length is adjusted so that the window boundary avoids the semantic breakpoint, resulting in a set of text blocks formed by multiple text blocks.

3. The question-and-answer pair augmentation method as described in claim 1, characterized in that, The visual elements in the original corpus include at least one of flowcharts, statistical charts, and physical images; The process of parsing visual elements in the original corpus to obtain visual semantic descriptions and visual feature fingerprints of the visual elements specifically includes: By using a visual language model to perform semantic analysis on visual elements in the original corpus, the logical transitions between flowchart nodes can be determined, resulting in a structured logical semantic sequence; alternatively, coordinate axis data, trend changes, and extreme points of statistical charts can be extracted and transformed into descriptive text blocks; or, the semantic categories of physical images can be determined as visual semantic descriptions of visual elements. Visual feature fingerprints are extracted based on the type of visual elements, specifically including: Visual elements are preprocessed, including resolution unification, noise filtering, and edge enhancement. A pre-trained visual encoding model is used to encode visual elements to obtain visual feature vectors of fixed dimensions; The visual feature vectors are normalized to generate visual feature fingerprints; The visual feature fingerprint is bound to the identifier of the corresponding visual element and stored in a multimodal index database, wherein the visual feature fingerprint has a length of [missing information]. d The normalized vector.

4. The question-and-answer pair augmentation method as described in claim 3, characterized in that, The construction of a cross-modal logical topology hypergraph based on visual semantic description and visual feature fingerprint specifically includes: Based on the structured logical semantic sequence in the visual semantic description of visual elements, determine the visual logical vertices and the logical path hyperedges between visual logical vertices; When the visual element is a real-world image, the target object, component object, spatial location object, and attribute object are used as visual logic vertices, and the logic is determined based on compositional relationships, adjacency relationships, attribute constraints, or the feature space of visual feature fingerprints. Nearest neighbor relationships are used to construct object relationship hyperedges; for statistical charts, association relationships are constructed based on the similarity between the text descriptor of the statistical chart name and the text descriptor in the text block; The text block corresponding to the visual logic vertex is determined as the text semantic vertex, and a visual similarity hyperedge is established based on the similarity between visual feature fingerprints. A cross-modal semantic mapping hyperedge is established based on the correspondence between the visual logic vertex and the text semantic vertex. Construct a cross-modal logical topology hypergraph based on visual logical vertices, textual semantic vertices, logical path hyperedges, statistical relation hyperedges, object relation hyperedges, visual similarity hyperedges, and cross-modal semantic mapping hyperedges.

5. The question-and-answer pair augmentation method as described in claim 1, characterized in that, The process of sampling text block sets, local semantic hypergraphs, global semantic hypergraphs, and cross-modal logical topology hypergraphs to determine question anchor points, and generating candidate question groups through diversified sampling based on these anchor points, specifically includes: Extract event units, topic units, entity units, and visual units based on text block sets; Based on the temporal sequence, causal relationship, or chapter hierarchy among event units, narrative paths are extracted from the global semantic hypergraph; entity relationship subgraphs are extracted from the higher-order hyperedges in the local semantic hypergraph; visual logic paths or visual similarity clusters are extracted from the cross-modal logical topology hypergraph; narrative paths, entity relationship subgraphs, or visual logic paths are encoded into unified anchor point representations to obtain question anchor points; wherein, the question anchor point includes at least an anchor point identifier, an anchor point type, a set of associated vertices, a set of associated hyperedges, and an identifier of the source text block; For the same question anchor point, diverse sampling within the group is carried out based on preset template constraints, sampling temperature and historical question similarity threshold to generate multiple questions with different semantics and verifiable answers, forming a question candidate group.

6. The question-and-answer pair augmentation method as described in claim 1, characterized in that, The step of determining and retrieving retrieval paths based on local and global semantic hypergraphs according to the complexity of each question in the question candidate group, and generating predicted answers for each question in the question candidate group, specifically includes: The complexity of each question in the candidate question group is determined based on the semantic features, reasoning step features, entity coverage features, and visual cue features of the question. The semantic features include at least the number of topic keywords and the level of abstraction. The reasoning step features include at least the shortest reasoning path length of the question in the hypergraph. The entity coverage features include at least the number of associated entities. The visual cue features include at least whether visual logic nodes or visual feature fingerprint matching are involved. Based on the complexity of each question in the candidate question group, the preset complexity threshold, and the correspondence between complexity intervals and retrieval paths, the retrieval path for each question in the candidate question group is determined. No retrieval is performed when a question falls into the first complexity interval; when a question falls into the second complexity interval, only the global semantic hypergraph is retrieved; when a question falls into the third complexity interval, only the local semantic hypergraph is retrieved; when a question falls into the fourth complexity interval, both the global and local semantic hypergraphs are retrieved simultaneously; when a question contains visual cues, a cross-modal logical topology hypergraph is further retrieved based on the corresponding retrieval path; and the retrieval results from different hypergraphs are structured and organized according to source type, relevance score, and time order to form a contextual evidence package. The contextual evidence package is input into the problem-solving model. First, a macro-semantic framework is generated based on global semantic evidence. Then, entity facts, relational facts, and visual facts are supplemented based on local semantic evidence and cross-modal evidence to generate predicted answers for each question in the question candidate group. The determination of the complexity of each problem in the candidate problem group specifically includes: The first step is to directly input the question into the large model. If the large model can answer it correctly, the question has the first complexity; otherwise, proceed to the second step. The second step is to input the problem and all global hypergraph content into the large model. If the large model can answer correctly, the problem has a second complexity; otherwise, proceed to the third step. The third step is to input the problem and all local hypergraph content into the large model. If the large model can answer correctly, the problem has a third complexity; otherwise, proceed to the fourth step. The fourth step is to input the problem, all global hypergraphs, and all local hypergraphs into the large model. If the large model can answer the question correctly, the problem has a fourth complexity; otherwise, the problem is discarded. In each step, the temperature is set to 1, and the large model is asked to answer N times. If it answers correctly even once, it is considered to be able to answer correctly; otherwise, it is considered to be unable to answer.

7. The question-and-answer pair augmentation method as described in claim 1, characterized in that, A question candidate group is generated using a question generation model; a predicted answer for each question in the question candidate group is generated using a question solving model. Training the question-generating model specifically includes: Under the premise of freezing the problem-solving model, for each problem in the generated problem candidate group, the semantic alignment reward of the problem is determined according to the degree of alignment between the generated problem and the input topic, entity, and path. The semantic alignment reward includes topic alignment reward, entity alignment reward, and inference length reward. Specifically, the topic alignment reward is calculated by calculating the similarity between the semantic vectors of the generated problem and the candidate topic; relevant entities are determined by calculating the similarity between the problem and multiple entity texts that is greater than a threshold, and the number of relevant entities and the number of hyperedges involved are used as entity alignment reward; the maximum depth of the hypergraph formed by the relevant entities and hyperedges is calculated as the inference path length of the problem, and the inference length is used as the reward. Based on the semantic vector similarity between the question and other questions in the question candidate group, the overall difference between the question and other questions in the question candidate group is determined to determine the diversity gain reward of the question; the overall difference is positively correlated with the diversity gain reward of the question; The cross-modal fusion reward for a problem is determined based on whether it simultaneously references visual logical vertices and corresponding textual semantic vertices, and whether it requires combining visual similarity hyperedges or cross-modal semantic mapping hyperedges to obtain the standard answer. Specifically, this includes: fixing the same problem-solving model and answering the same problem under three different conditions: Condition 1: only text input, Condition 2: only image input, and Condition 3: both image and text input; answering the same problem under three different input conditions. n The accuracy rate of text-only responses is recorded as follows: The accuracy rate of responses based solely on image input is denoted as... The accuracy of mixed inputs is denoted as: The final reward is recorded as follows: -( + ) / 2, as a cross-modal fusion reward; The question reward is determined based on the semantic alignment reward, diversity gain reward, and cross-modal fusion reward of the question. The generation strategy of the question generation model is updated by optimizing the group relative strategy with the optimization objective of maximizing the relative average reward within the same question candidate group. The generation strategy includes at least the anchor point selection probability distribution, the question template selection probability distribution, and the sampling temperature control parameter.

8. The question-answer pair augmentation method as described in claim 7, characterized in that, Training the problem-solving model specifically includes: Under the premise of freezing the question generation model, for the predicted answer of each question in the generated question candidate group, the format of the predicted answer is determined according to the preset question format template and the model output, and the format reward is determined. The mixed answer reward for the generated answer is determined based on the deviation between the predicted answer and the preset standard answer for the question. The reward for generating the predicted answer is determined based on the number of retrieval calls made by the problem-solving model; the number of calls and the reward for generating the answer are negatively correlated. The retrieval relevance reward for the predicted answer is determined based on the relevance between the keywords used in the retrieval process of generating the predicted answer by the problem-solving model and the preset standard answer of the question; relevance and relevance reward are positively correlated. The answer reward for the predicted answer is determined based on the format reward, mixed answer reward, call count reward, and retrieval relevance reward. The problem-solving strategy of the problem-solving model is updated by optimizing the group relative strategy based on the answer rewards of each question in the generated question candidate group.

9. A question-and-answer pair augmentation device, characterized in that, include: The corpus preprocessing module is used to generate a set of text blocks based on the semantic segmentation results of the original corpus in the target domain; Based on the semantic relationships of entities within text blocks, construct a local semantic hypergraph; based on the macro-semantic relationships between text blocks, construct a global semantic hypergraph. The multimodal hypergraph construction module is used to parse visual elements in the original corpus to obtain visual semantic descriptions and visual feature fingerprints of the visual elements; Construct a cross-modal logical topology hypergraph based on visual semantic description and visual feature fingerprint; The question generation module is used to sample the text block set, local semantic hypergraph, global semantic hypergraph and cross-modal logical topology hypergraph to determine the question generation anchor point, and generate question candidate groups based on the question generation anchor point through diversified sampling. The problem-solving module is used to determine and retrieve the search paths based on the complexity of each problem in the problem candidate group, using both local and global semantic hypergraphs, and to generate answers for each problem in the problem candidate group.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 8.

Images