Question and answer pair augmentation method and device in professional field, equipment and medium

Abstract

The invention provides a question and answer pair augmentation method and device, equipment and a medium. Relates to the technical field of natural language processing. The method comprises the steps that a question data set and a vectorization database in the professional field are obtained, an intelligent agent is constructed and trained, in each round of training, the intelligent agent gradually executes a vision-text search operation, a vision area labeling operation and an answer generation operation in the vectorization database according to responses, and the question data set and the vectorization database are obtained; taking a result returned by the answer generation operation as a target answer; evaluating the target answer and the generated response, and adjusting a generation strategy of the next state response according to an evaluation result; and augmenting question and answer pairs in the professional field based on the trained intelligent agent. According to the method, a traceable thinking chain is constructed for the evaluation of the generated response through the hierarchical visual processing flow of the intelligent agent, the logic of answer generation can be verified, the response of semantic deviation can be traced, and the professional accuracy and reliability of the generated question and answer pair are improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more particularly to a question-answer pair augmentation method, apparatus, device, and medium in a specialized field. Background Technology

[0002] In the process of implementing artificial intelligence applications in specialized vertical industries, constructing high-quality, scenario-based professional question-answering datasets is the primary challenge. Vertical domains are characterized by knowledge-intensive, highly specialized, and deeply coupled across systems; however, the scarcity of high-quality question-answering samples, uneven data distribution, and difficulties in knowledge extraction severely restrict the practical application of AI technology. As intelligent question-answering systems are a core means of improving business efficiency and reducing labor costs, their reliance on high-quality question-answering datasets is increasingly prominent. Therefore, there is an urgent need to develop efficient and controllable professional question-answering dataset augmentation technologies to automatically expand domain-specific question-answering datasets and improve the accuracy and coverage of intelligent question-answering systems.

[0003] Retrieval-Augmented Generation (RAG) technology provides technical support for augmenting specialized question-answer pairs. RAG combines external knowledge retrieval with the generative capabilities of a large language model to construct a standard "index-retrieval-enhancement-generation" process: Domain-specific documents are segmented into structured text blocks using semantic segmentation and vectorized using an embedding model to build a vector database; in the retrieval stage, user online queries are vectorized using the same embedding model, and relevant text fragments are quickly located in the vector database through similarity calculation; in the enhancement stage, the retrieval results are combined with the original query according to the suggested template, transforming it into a context-based closed-domain question; finally, in the generation stage, the large language model understands and integrates the provided context to generate an accurate answer.

[0004] However, existing RAG technology relies solely on textual semantic surface similarity matching when handling question-answer pair augmentation tasks. It lacks logical verification of the generated answers and cannot effectively parse multimodal information such as data charts and device diagrams in professional documents. This makes it difficult to construct and align cross-modal question-answer pairs between text and images, ultimately leading to semantic bias in the augmented question-answer pairs and reducing the professional accuracy and reliability of the generated question-answer pairs. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, device, and medium for augmenting question-answer pairs in a professional field to address the aforementioned technical problems, thereby resolving the semantic bias and other modal information loss that may occur in the augmented question-answer pairs generated in the prior art, and achieving a significant improvement in the professional accuracy and reliability of the generated question-answer pairs.

[0006] The following technical solution is adopted in this specification: This specification provides a professional-domain question-and-answer pair augmentation method, including: Obtain a problem dataset and a vectorized database for the professional field. The problem dataset includes professional problems in the professional field; the vectorized database includes text element vectors corresponding to text information in the professional field and visual element vectors corresponding to non-text information in the professional field. Construct an agent to augment question-answer pairs in a specialized domain, and train the agent through reinforcement learning. During the reinforcement learning training process: The professional questions in the problem dataset are taken as the current state of the agent; a response containing multiple operation instructions is generated based on the current state. The operation instructions have a sequential order, and each operation instruction corresponds to a specific operation of the agent in the vectorized database. The agent parses the operation instructions in the response and executes each operation instruction in sequence, including: When the operation command is visual-text search, the system retrieves text element vectors and visual element vectors related to the professional question from the vectorized database to obtain a multimodal answer. When the operation instruction is visual region annotation, the image corresponding to the visual element vector in the initial answer is located and annotated to generate a visually annotated answer. When the operation instruction is to generate an answer, the multimodal answers and visually labeled answers accumulated from the previously executed visual-text search and visual region annotation are integrated to generate the target answer for the current state; The target answer and the generated response are evaluated, and the generation strategy for the next state response is adjusted based on the evaluation results; The trained agent is used to augment the question-and-answer pairs in the relevant professional domain.

[0007] Furthermore, the intelligent agent parses the operation instructions in the response and executes each operation instruction sequentially, specifically including: I. Initialize the initial sequence trajectory of the response and operation counter ; II. Determine the operation counter If the number of operations is not greater than the preset maximum number of operations T, proceed to steps III to X; otherwise, proceed to step XI. III. Based on Professional Problem Vectors and policy network Generate response sequence ; IV. Response Sequence spliced ​​to the initial sequence trajectory of the response To obtain the updated sequence trajectory ; V. In the updated sequence trajectory Detect visual-text search tag action commands. If a visual-text search tag action command is detected, proceed to step VI; otherwise, proceed to step VII. VI. In the updated sequence trajectory Parse the search query And based on the search query Searching for text-visual elements related to professional issues in vectorized databases. ; VII. In the updated sequence trajectory The system detects visual region labeling action commands. If a visual region labeling action command is detected, step VIII is executed; otherwise, step IX is executed. VIII. From the updated sequence trajectory Visual perception tokens were parsed from the data. loc And based on visual perception tokens loc Text-visual elements The visual elements in the image are subjected to pixel localization, pixel cropping and pixel encoding operations to obtain the visual region vector; IX. In the updated sequence trajectory If the action command to generate an answer label is detected, proceed to step X; otherwise, update the operation counter. Return to step II; X. Concatenate the visual region vectors to the updated sequence trajectory. In order to Update again; XI. Output the updated response sequence trajectory .

[0008] Furthermore, the visually perceived token loc Text-visual elements The visual elements in the image undergo pixel localization, pixel cropping, and pixel encoding operations, specifically including: By using visual perception tokens loc As input to the preset visual space mapping function, text-visual elements are provided to the preset visual space mapping function. Pixel localization of visual elements in text is achieved by using a preset visual space mapping function to map text-visual elements. Crop the visual elements in the image: in, This represents the key pixel region obtained from cropping that is relevant to the specific problem. This indicates the relationship between text and visual elements. Cropping and enlarging visual elements in the image; Representing text-visual elements Visual elements in; Represents visual perception tokens loc The pixel boundaries of the location, The minimum x-coordinate boundary, The boundary of the maximum x-coordinate. The minimum ordinate boundary, Maximum ordinate boundary; and Images representing visual elements respectively Width and height dimensions; and These represent the maximum processing width and maximum processing height of the preset visual encoder, respectively; The key pixel regions related to the professional problem are obtained through cropping using a preset visual encoder. Encode it.

[0009] Furthermore, the evaluation of the target answer and the generated response specifically includes: A retrieval efficiency reward is awarded, which is used to quantitatively evaluate the relevance of the target answer to the visual region vector and the reasonableness of the target answer's ranking. The formula for calculating the retrieval efficiency reward is as follows: in, Incentives are awarded for improved search efficiency; It is the set of all visual region vectors in the response sequence generated by the agent for a specific problem; This is the collection of images most relevant to the specific professional issue. It is the normalized loss cumulative gain. ; It is a correlation indicator function. ; For the first Images related to the retrieved professional questions; It is the ideal cumulative gain after loss. ; A reward is awarded for answer accuracy. This reward is used to quantitatively assess the similarity between the target answer and the reference answer generated by the pre-set expert model. The formula for calculating the answer accuracy reward is as follows: in, For professional issues; Reference answers to professional questions generated by a pre-defined expert model; The answer obtained by the intelligent agent; For random variables The probability distribution; Rewards are given for accurate answers, and compliance is based on the expertise of the question. Reference Answer and the answers obtained by the agent The probability distribution, reference answer and the answers obtained by the agent The higher the similarity, the more accurate the answer is rewarded. The larger; A pattern normalization reward is awarded, which is used to quantitatively evaluate the degree of format normalization of the responses generated by the policy network. The formula for calculating the pattern normalization reward is as follows: in, Rewards for standardized practices; It is the generated response sequence; Indicates the updated sequence trajectory The process of detecting visual region annotation label action commands and obtaining visual region vectors based on visual region annotation label action commands; The evaluation reward value is obtained by weighting and integrating the rewards for retrieval efficiency, answer accuracy, and pattern standardization.

[0010] Furthermore, the steps for obtaining the problem dataset in the aforementioned professional field specifically include: Construct a structured problem template for a professional field, wherein the structured problem template includes at least one variable slot; The generative artificial intelligence service interface is invoked to perform multi-dimensional augmentation operations on the structured problem template to obtain a problem dataset in a specific domain; the multi-dimensional augmentation operations include: Select semantically related entities and fill or replace the variable slots; Rewrite the problem after variable slot filling or replacement through generative artificial intelligence service interface; By combining virtual scenarios, document fragments, or dialogue history related to the question as contextual information with the question, a scenario-based question can be generated.

[0011] Furthermore, the steps for obtaining the vectorized database specifically include: Batch acquisition of original PDF documents in professional fields; conversion of text content and non-text visual elements in the original PDF documents into unified image file object representations; and semantic segmentation processing of the image file object representations to obtain a set of structured knowledge units. Perform adaptive resolution optimization on the visual elements in the structured knowledge unit set, and convert all visual elements in the structured knowledge unit set to the same color mode to obtain a standardized knowledge unit set; Perform vector encoding operations on each standardized knowledge unit set to obtain a knowledge unit vector set, each knowledge unit vector set including text element vectors and visual element vectors; An index is built on the knowledge unit vector set to construct a vectorized database that includes the index, text feature vectors, visual feature vectors, and the original PDF document content.

[0012] Furthermore, the augmentation of the question-and-answer pairs based on the trained agent in the professional domain includes: A multi-layered dynamic prompt template is constructed, comprising role definition prompts, process control prompts, and action specification prompts. Role definition prompts define the identity of a domain expert and the task objective for generating question-answer pairs. Process control prompts guide the agent to execute a standardized process of problem understanding, multimodal retrieval, information integration, and answer generation through structured instruction sequences. Action specification prompts define the triggering conditions and output format requirements for visual-text search operations, visual region annotation operations, and answer generation operations. Obtain a dataset of professional questions in the professional fields to be expanded. Based on the complexity and type of questions in the professional question dataset, dynamically adjust the weights of role definition prompts, process control prompts, and action specification prompts in the multi-level dynamic prompt template to generate the optimal prompt strategy that matches the professional question dataset. The agent sequentially performs visual-text search, visual region annotation, and answer generation operations based on the optimal prompting strategy to obtain the original output of the question-answer pair. The original output of the question-answer pairs is subjected to structured parsing, rule-based post-processing, and output standardization to obtain an augmented dataset of question-answer pairs for the desired professional domain.

[0013] This specification provides a question-and-answer pair augmentation device for a specific field, comprising: The prior dataset acquisition module obtains a problem dataset and a vectorized database for the professional field. The problem dataset includes professional questions in the professional field; the vectorized database includes text element vectors corresponding to text information in the professional field and visual element vectors corresponding to non-text information in the professional field. The agent training module is used to build agents for augmenting question-answer pairs in a specific domain. The agents are trained through reinforcement learning. During the reinforcement learning training process: The professional questions in the problem dataset are taken as the current state of the agent; a response containing multiple operation instructions is generated based on the current state. The operation instructions have a sequential order, and each operation instruction corresponds to a specific operation of the agent in the vectorized database. The agent parses the operation instructions in the response and executes each operation instruction in sequence, including: When the operation command is visual-text search, the system retrieves text element vectors and visual element vectors related to the professional question from the vectorized database to obtain a multimodal answer. When the operation instruction is visual region annotation, the image corresponding to the visual element vector in the initial answer is located and annotated to generate a visually annotated answer. When the operation instruction is to generate an answer, the multimodal answers and visually labeled answers accumulated from the previously executed visual-text search and visual region annotation are integrated to generate the target answer for the current state; The target answer and the generated response are evaluated, and the generation strategy for the next state response is adjusted based on the evaluation results; The question-answer pair augmentation module is used to augment question-answer pairs based on the trained agent in a specific domain.

[0014] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the question-answer pair augmentation method described above in the field of expertise.

[0015] This specification provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the question-answer pair augmentation method described above in the field of expertise.

[0016] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: This invention utilizes an intelligent agent to progressively execute a hierarchical visual processing flow—"visual-text search → visual region annotation → answer generation"—within a vectorized database. This completes the progressive analysis of visual elements, from coarse-grained search and location to fine-grained annotation and understanding, achieving a deep understanding of complex visual elements and ensuring semantic alignment between text and images. Furthermore, each step of the hierarchical visual processing flow constructs a traceable thought chain for evaluating the generated response. This traceable thought chain not only verifies the logic of answer generation but also traces back to semantically biased responses during the generation of the target answer. Based on the evaluation reward values ​​of the responses and the target answer, iterative updates to the weight parameters of the policy network through reinforcement learning can reduce the probability of generating semantically biased responses, thereby improving the professional accuracy and reliability of the generated question-and-answer pairs. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is one of the flowcharts illustrating a question-and-answer pair augmentation method for a specific field, as provided in this specification. Figure 2 This is a flowchart illustrating the process by which an intelligent agent performs visual-text search, visual region annotation, and answer generation operations step by step in a vectorized database based on responses, as provided in this specification. Figure 3 This is the second flowchart illustrating a question-and-answer pair augmentation method for a specific professional field, as provided in this specification. Figure 4 This specification provides a structural diagram of a question-and-answer pair augmentation device for a specific field. Figure 5 This is a schematic diagram of a computer device provided for this specification. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0020] In the development of RAG technology, several representative optimization schemes have emerged. DeepRAG models the retrieval process as a Markov decision process, achieving adaptive retrieval and query decomposition. Through reinforcement learning, it dynamically determines retrieval timing and strategies, significantly improving the efficiency of handling complex problems. Causal-Counterfactual RAG introduces explicit causal graphs and counterfactual reasoning into the retrieval process, enhancing the robustness of the answers. By constructing causal graphs and reasoning about counterfactual scenarios, it effectively identifies and eliminates biases caused by confounding factors. RankCoT combines re-ranking and chain reasoning, optimizing the purity of knowledge extraction through a self-reflection mechanism. It utilizes chain reasoning to perform multi-step verification and filtering of retrieval results, effectively reducing the interference of noisy documents on the generation process.

[0021] However, existing RAG technologies still face significant challenges when processing domain-specific datasets. First, regarding illusion suppression, most existing methods rely on superficial semantic similarity of text, lacking deep logical verification and consistency assurance mechanisms for the retrieved data. This results in generated answers that may still be poorly relevant to the query or contain factual biases. Second, existing technologies are entirely limited to text modality processing, failing to effectively understand multimodal data that may exist in domain-specific datasets, such as data charts, equipment diagrams, and financial document images, making it difficult to achieve cross-modal alignment and joint reasoning between text and images. These problems severely limit the accuracy and practicality of intelligent question-answering systems in professional domains and real-world business scenarios.

[0022] Therefore, this invention provides a domain-specific question-answer pair augmentation method, comprising: 1) calling an Application Programming Interface (API) to augment a small number of question templates into a large number of questions, obtaining a domain-specific question dataset; 2) processing domain-specific documents to establish a domain-specific vectorized database; 3) training an agent using the GRPO (Group Relative Policy Optimization) reinforcement learning algorithm, performing multimodal, multi-step interaction and reasoning in each training round; 4) constructing an artificial intelligence agent for the trained multimodal interaction model to obtain the question-answer pair augmentation dataset. This invention significantly improves the agent's multimodal understanding ability. By introducing a dynamic visual perception mechanism, it overcomes the limitations of traditional text retrieval augmentation generation systems in processing multimodal documents.

[0023] The following is combined with Figures 1-3 This invention describes a question-and-answer pair augmentation method within the field of expertise.

[0024] Figure 1 This document provides one of the flowcharts for a question-and-answer pair augmentation method in a specific field, such as... Figure 1 As shown, the method includes: S100. Obtain a problem dataset and a vectorized database for the professional field. The problem dataset includes professional problems in the professional field. The vectorized database includes text element vectors corresponding to text information in the professional field and visual element vectors corresponding to non-text information in the professional field.

[0025] For example, to construct a comprehensive and robust domain-specific problem dataset, a small number of high-quality problem templates are first designed. These templates are then augmented into a large number of problems, defining the semantic structure and logical framework of core problems within the domain. Subsequently, problem augmentation is performed by calling an API with natural language generation capabilities (such as a large language model API). The core task of this API is to perform large-scale and diverse augmentations based on these templates, which can include: 1) entity replacement, automatically extracting relevant entities from the domain knowledge base (such as equipment names in the mechanical field, drug names in the pharmaceutical field, and legal clauses in the legal field) to fill placeholders in the templates; 2) semantic paraphrasing, rewriting the generated problems with synonyms and changing sentence structures to ensure diverse expressions while retaining the original meaning; and 3) context injection, adding specific constraints or scenario backgrounds to the problems to increase their complexity and realism. Through this process, dozens of basic templates can be efficiently expanded into multiple domain-specific problems with broad coverage and diverse expressions, ultimately forming a large-scale and high-quality professional problem dataset. After obtaining the professional question dataset, the professional questions in the professional field are transformed into a uniformly formatted input representation using a preset structured prompt template. The uniformly formatted input representation includes text instructions, visual markers, and tool calling specifications. Among them, text instructions are used to define the steps and logic for generating answers to professional questions, visual markers are used to locate visual elements related to professional questions during the answer generation process, and tool calling specifications are used to define the types of tools that can be called during the answering process. The uniformly formatted input representation is encoded into professional question vectors using a preset text encoder, and a professional question dataset is constructed based on the professional question vectors.

[0026] For example, the construction of a vectorized database may include: The steps of processing domain-specific documents to establish a domain-specific vectorized database may include: S111: Batch acquire original PDF documents in the professional field, convert the text content and non-text visual elements in the original PDF documents into a unified image file object representation; and perform semantic block processing on the image file object representation to obtain a set of structured knowledge units.

[0027] For example, a standardized pipelined processing architecture based on LlamaIndex is adopted to achieve deep structural processing of professional domain documents. First, the SimpleDirectoryReader component reads the original PDF documents in batches, and a high-performance parsing engine extracts the text content and visual elements (including charts, flowcharts, and professional diagrams) from the documents, converting them into unified image file object representations. Each image file object is then semantically segmented by SimpleFileNodeParser to ensure the integrity and independence of knowledge units.

[0028] S112. Perform adaptive resolution optimization on the visual elements in the structured knowledge unit set, and convert all visual elements in the structured knowledge unit set to the same color mode to obtain a standardized knowledge unit set.

[0029] For example, the core processing is implemented using a dedicated multimodal embedding model, vl_Embedding, which is based on the ColQwen2-VL architecture and has cross-modal semantic alignment capabilities. During processing, the system performs adaptive resolution optimization on the visual content: when the total number of image pixels exceeds the model's processing limit, the image is scaled proportionally to within the 2048×2048 pixel range, while ensuring that the processed image does not fall below the minimum resolution requirement of 512×512 pixels, thus balancing computational efficiency and visual detail preservation. All images are uniformly converted to RGB mode to ensure consistency in input format.

[0030] S113. Perform vector encoding operation on each standardized knowledge unit set to obtain a knowledge unit vector set, each knowledge unit vector set including text element vectors and visual element vectors; For example, multimodal embedding models pre-trained and optimized with domain data (such as improved BERT or CLIP models based on the Transformer architecture) perform vector encoding operations: for textual elements in knowledge units (including technical terms, technical descriptions, logical relationship expressions, etc.), semantic features are extracted through the model's text encoding layer and transformed into structured numerical vectors; for visual elements (including diagrams, charts, key area annotation maps, etc.), spatial features, texture features, and visual semantic associations are parsed through the visual encoding branch to generate corresponding feature vectors. Ultimately, each knowledge unit is mapped to a unified 768-dimensional floating-point vector representation. This vector not only fully preserves the core information of both textual and visual elements but also achieves cross-modal feature alignment and fusion, ensuring that subsequent vector retrieval, similarity calculation, and other operations can accurately capture the multi-dimensional semantic associations of knowledge units, providing high-dimensional feature support for cross-modal knowledge matching and reasoning.

[0031] S114. Build an index for the knowledge unit vector set to construct a vectorized database that includes the index, text element vectors, visual element vectors, and the original PDF document content.

[0032] For example, the vector corresponding to each knowledge unit is organized using a FAISS index structure to construct an efficient multimodal retrieval database that supports approximate nearest neighbor search. This database not only stores the original content but also retains complete metadata information (including document source, page location, and content type), providing contextual support for subsequent accurate retrieval. A multi-threaded parallel execution mode is used to process documents, and through batch processing, a multimodal vector knowledge base covering domain knowledge is constructed.

[0033] S200. Construct an agent for augmenting professional domain question-answer pairs. Train the agent using reinforcement learning. During reinforcement learning training: use the professional questions in the question dataset as the agent's current state; generate a response containing multiple operation instructions based on the current state, wherein the operation instructions have a sequential order and each operation instruction corresponds to a specific operation of the agent in the vectorized database; parse the operation instructions in the response and execute each operation instruction in sequence, including: when the operation instruction is visual-text search, retrieve text element vectors and visual element vectors related to the professional question from the vectorized database to obtain a multimodal answer; when the operation instruction is visual region annotation, perform region localization and annotation on the image corresponding to the visual element vector in the initial answer to generate a visually annotated answer; when the operation instruction is answer generation, integrate the multimodal answers and visually annotated answers accumulated from the executed visual-text search and visual region annotation to generate the target answer for the current state; evaluate the target answer and the generated response, and adjust the generation strategy for the next state response based on the evaluation results.

[0034] For example, a large model including a policy network, a value network, and an agent is constructed. This large model is trained on a problem dataset and a vectorized database to obtain a multimodal interaction model. In each training round: a specialized question is extracted from the problem dataset and input into the policy network (which can be a visual language model). The policy network generates a response, which triggers the agent to perform visual-text search, visual region labeling, and answer generation operations in the vectorized database. The agent then progressively performs these operations based on the response, using the result of the answer generation operation as the target answer. The value network evaluates the response generated by the policy network and the target answer, outputting an evaluation reward value. The weight parameters of the policy network are iteratively updated based on the evaluation reward value. For example, the large model is trained using the GRPO reinforcement learning algorithm. The steps for multimodal, multi-step interaction and inference in each training round may include:

[0035] S210, Initialization Phase: Load a domain-specific question set. Each question is transformed into an initial state for a multi-turn dialogue using a structured prompt template. This template integrates text instructions, visual markers, and tool invocation specifications. The pre-set structured prompt template transforms professional questions in the domain into a uniformly formatted input representation. This uniformly formatted input representation includes text instructions, visual markers, and tool invocation specifications. Text instructions define the steps and logic for generating answers to professional questions; visual markers locate visual elements related to the professional questions during the answer generation process; and tool invocation specifications define the types of tools that can be invoked during the answering process. A pre-set text encoder encodes the uniformly formatted input representation into a professional question vector. Simultaneously, the policy network (Actor), reference policy network (RefPolicy), and value network (Critic) are initialized. These networks share basic parameters but perform different functions. The visual encoder and text encoder complete pre-trained weight loading, preparing for subsequent multimodal feature extraction.

[0036] S220. Enter the reinforcement learning training loop and conduct multiple rounds of interaction. At each time step, the policy network generates a response containing thought chains and action instructions based on the current state. The response parsing module extracts three structured operations: search query ( <search>), region labeling ( <region>) or answer generation ( <answer>After performing the corresponding operations on the obtained domain-specific vectorized database, new observations are returned, and the state management module updates the dialogue history to ensure context integrity.

[0037] S230. If the system issues a search command, the vector similarity-based retrieval module is invoked. This module encodes the query text into a vector and performs an approximate nearest neighbor search in a pre-built multimodal knowledge base, returning the visual-text information fragment with the highest relevance. The retrieval process employs an improved NDCG evaluation mechanism to ensure the accuracy and reasonable ranking of the returned results.

[0038] S240. If the system issues a visual region annotation instruction, a precise coordinate mapping and image processing flow is executed: First, the normalized coordinates output by the model are mapped back to the original image space, and then a cropping operation is performed on the original high-resolution image. After adaptive resolution adjustment, the cropped region is re-encoded into a high-density token sequence by the visual encoder, significantly improving the ability to perceive visual details. The entire process simulates the human "coarse-to-fine" observation strategy, gradually focusing on information-dense areas.

[0039] S250. If the system issues an answer generation instruction, the loop will exit and complete trajectory information and answer will be generated.

[0040] S260. Evaluate the current results using a three-dimensional reward function: retrieval efficiency reward. Evaluate the relevance and ranking quality of search results; reward answer accuracy. Calculate the similarity between the generated answer and the standard answer; pattern normative reward. Verify the response format conformity. The reward calculation module integrates these signals to generate a scalar reward value.

[0041] S270. Policy optimization is performed using the GRPO algorithm. The advantage estimation module calculates the relative value of each action, and the policy gradient module updates the network parameters. The value function is updated through temporal difference learning, and the Critic network provides accurate predictions of state values. The KL divergence penalty term ensures that the policy update magnitude is within a safe range, preventing policy abrupt changes.

[0042] S280 monitors training progress through multiple metrics, including average reward, answer accuracy, and retrieval efficiency. Sliding window statistics ensure the stability of performance evaluation. Training automatically terminates when validation set performance no longer improves over several consecutive epochs. This results in a complete multimodal interactive large-scale model.

[0043] S300: Based on the trained agent, the question-and-answer pairs in the professional field are augmented.

[0044] For example, the steps for constructing an AI agent to acquire augmented datasets for question-answer pairs, based on a trained multimodal interaction model, may include: S310. Construct an AI agent architecture based on prompting engineering. The system designs a multi-layered prompting template system, where the role definition layer clarifies the agent's domain expert identity and question-answer pair generation task objectives; the process control layer guides the agent to execute a standardized "problem understanding - multimodal retrieval - information integration - answer generation" process through structured instruction sequences; and the action specification layer... <search> 、 <region> 、 <answer>The three core action categories have detailed trigger conditions and output format requirements. The prompt templates employ a dynamic context-aware mechanism, which can automatically adjust the weight of prompt words based on the current interaction state, ensuring that the agent maintains consistent responses during batch processing.

[0045] S320. Implement a question-answer pair generation process for batch processing. The system loads the generated domain-specific question dataset and uses a dynamic prompt scheduler to match the optimal prompt strategy based on question complexity and type characteristics. The dynamic prompt scheduler, which integrates question semantic parsing and feature classification modules, first deeply decomposes the input domain-specific questions: it identifies question complexity (e.g., simple concept query, complex scene analysis, multi-condition reasoning, etc.) based on natural language processing technology, and extracts question type features (e.g., text description, visual association, cross-modal fusion, etc.). Then, it combines a pre-set prompt strategy library (including optimized prompt templates for different complexities and types), and adaptively selects the most suitable prompt strategy for the current question through similarity matching and effect priority ranking, ensuring accurate adaptation of prompt instructions. The execution engine adopts an asynchronous pipeline architecture, supporting concurrent processing of multiple question instances, with each instance independently maintaining its prompt session state. During processing, the agent sequentially executes multimodal retrieval, visual region annotation, and answer generation operations based on the pre-set prompt templates. The system records the complete interaction trajectory and intermediate results for each question in real time.

[0046] S330. Post-processing and quality verification of question-and-answer pair data. The system performs structured parsing on the raw output generated by the agent, extracts standardized question-and-answer pair formats, and performs rule-based post-processing, including answer deduplication, format unification, and obvious error filtering. Simultaneously, the system re-injects low-quality samples and adjusts the prompting strategy for regeneration.

[0047] S340, the agent output module generates a standardized question-answer pair dataset, realizing the acquisition of the question-answer pair augmented dataset.

[0048] This invention offers two improvements over traditional retrieval enhancement generation methods. The main improvement is to consider the multimodal features of domain-specific documents, enhancing the multimodal recognition capability of the retrieval enhancement generation method to improve the accuracy of question-answering pairs on the dataset. The beneficial effects of this invention include:

[0049] 1. This invention significantly enhances the multimodal understanding capabilities of intelligent agents. By introducing a dynamic visual perception mechanism, it completely overcomes the limitations of traditional text-based RAG systems in processing multimodal documents. The hierarchical visual processing paradigm adopted by the system enables the agent to progressively extract visual information from coarse to fine granular levels, achieving a deep understanding of complex visual elements such as charts, diagrams, and professional graphics.

[0050] 2. This invention achieves multi-dimensional control over the quality of generated answers through a three-dimensional reward function design and the GRPO reinforcement learning algorithm. The retrieval efficiency reward ensures the relevance and reasonable ranking of retrieval results; the answer accuracy reward provides fine-grained quality feedback through expert model evaluation; and the pattern standardization reward guarantees the standardization of response formats. This comprehensive reward mechanism enables the system to significantly reduce the illusion rate while maintaining a high recall rate.

[0051] The architecture of this invention exhibits excellent domain adaptability, enabling rapid deployment across various professional fields. The system supports the processing of multiple types of visual documents through a unified interface specification, including technical manuals, academic papers, financial reports, and medical images. Simultaneously, it ensures fast domain database retrieval and accurate answer generation. This comprehensive improvement in accuracy allows the generated question-and-answer dataset to be directly used to train high-performance domain expert models, significantly reducing the cost and time investment of manual annotation.

[0052] Based on the above Figure 1 The illustrated embodiment, for example, demonstrates a breakthrough innovation in visual information processing through step S300. By constructing a hierarchical dynamic visual perception mechanism, it changes the traditional retrieval enhancement generation processing mode. Its core lies in establishing a progressive visual information acquisition system from coarse-grained to fine-grained. Region labeling is added to the action instructions in reinforcement learning. <region>The main purpose of this action is to select, crop, and scale the most relevant region to the query from the retrieved image, improving the image clarity of that region to retrieve and collect richer image information. This technology is based on a precise coordinate mapping system. When the agent performs visual region annotation operations in the vectorized database based on the response, it parses the visual perception token from the response. loc Visual perception token loc As input to the preset visual space mapping function, text-visual elements are provided to the preset visual space mapping function. Pixel localization of visual elements in text is achieved by using a preset visual space mapping function to map text-visual elements. The visual elements in the image are cropped. The visual space mapping function is defined as:

[0053] ; in, Represents the original image, i.e. Representing text-visual elements Visual elements in; Image boundaries representing region-labeled action annotations, i.e., visual perception tokens. loc The pixel boundaries of the location, The minimum x-coordinate boundary, The boundary of the maximum x-coordinate. The minimum ordinate boundary, The maximum ordinate boundary is also the area of ​​the image after cropping and magnification; and Images representing visual elements respectively Width and height dimensions; and These represent the preset maximum processing width and maximum processing height dimensions of the visual encoder, respectively, which are determined by the maximum processing capacity of the visual encoder. Decide; This represents the cropping and enlarging operation of the image. This mapping function achieves a precise conversion from the normalized coordinate space to the original image pixel space. This mathematical mapping ensures that regardless of the resolution characteristics of the input image, the system can maintain the accuracy and consistency of the geometric transformation within the encoder's processing range, laying a solid foundation for subsequent visual feature extraction.

[0054] This technical solution exhibits three significant advantages: First, through region-selective processing and re-encoding strategies, the information density of visual tokens is increased by approximately 3-5 times, greatly enhancing the model's ability to perceive fine-grained visual features. Second, the system perfectly simulates the "coarse-to-fine" observation strategy of the human visual system, gradually focusing on information-dense areas through a multi-level attention mechanism, achieving more efficient visual information processing. Finally, this technology fundamentally resolves the contradiction between high-resolution requirements and model computational constraints in visually rich document processing, providing a feasible technical path for practical applications. Based on the above Figure 1 In the illustrated embodiment, for example, in step S200 above, when the value network evaluates the response and target answer generated by the policy network, a significant theoretical breakthrough has been achieved in the design of the reward function. A multi-dimensional reward evaluation system has been constructed, completely surpassing the limitations of traditional outcome-oriented rewards. The system's comprehensive reward function adopts a weighted fusion architecture:

[0055] ; in, , and The coefficients representing the three types of rewards meet the conditions. + + =1; Represents a reward for search efficiency; Rewards are given for accurate answers; Rewards are given for standardizing the model.

[0056] For retrieval efficiency rewards Its evaluation mechanism is based on an improved Normalized Discounted Cumulative Gain (NDCG) algorithm. The reward calculation function is defined as:

[0057] ; in This represents the set of all retrieved images or visual region vectors in the trajectory / response sequence generated by the agent in response to the problem. This represents the set of images most relevant to the professional issue. The formula for calculating the Discounted Cumulative Gain (DCG) is as follows: ; Among them, the correlation indicator function Using binary evaluation method: ; in, For the first Images related to the retrieved professional questions.

[0058] The ideal loss cumulative gain is constructed based on the perfect ordering assumption, and its calculation formula is as follows: .

[0059] Rewards for answer quality This invention introduces a more advanced expert model for in-depth evaluation of answers. This model is trained on large-scale multimodal data and possesses powerful answer quality evaluation capabilities. The corresponding expression is: in, This refers to the reference answers to the professional questions generated by the pre-defined expert model. The answer to be evaluated obtained by the agent. Representing current professional issues, For random variables The probability distribution; This represents the process of evaluating the correctness of the final answer. For random variables The probability distribution; Obedience is based on professional issues Reference Answer and the answers obtained by the agent The probability distribution, reference answer and the answers obtained by the agent The higher the similarity, the more accurate the answer is rewarded. The larger.

[0060] The evaluation process comprehensively considers multiple dimensions such as semantic consistency, factual accuracy, and logical coherence. Compared with traditional exact matching methods, this evaluation method can better reflect the actual quality of the answer, allows for reward allocation for partially correct cases, and provides more granular training signals.

[0061] Model-based rewards By parsing the structured features of action sequences, behavioral normativity is ensured. A pattern matching algorithm based on regular expressions is employed to ensure that the action format conforms to predefined normative requirements. This normative constraint not only improves the reliability of the system but also significantly enhances the interpretability and predictability of the generated results. The corresponding expression is:

[0062] in, It is the generated response sequence; This represents the process of detecting visual-text search tag action commands and extracting visual elements.

[0063] Based on the above Figure 1 In the illustrated embodiment, for example, in step S200 above, the agent performs the corresponding operation in the vectorized database based on the response. Combining the two core technologies mentioned above, a deep integration of visual processing technology and reinforcement learning mechanism is achieved, and this integration produces a significant synergistic effect. The hierarchical characteristics of visual processing and the multi-step decision-making process of reinforcement learning form a natural technical fit. Each visual pruning operation corresponds to a clear decision point, together constituting a complete state-action-reward loop system. Figure 2 This is a flowchart illustrating how an agent, based on a response, progressively performs visual-text search, visual region annotation, and answer generation operations in a vectorized database, as provided in this specification. Figure 3 As shown, the algorithm can be summarized in the following steps, where the input is a professional question vector (query). Policy Network Vectorized database Maximum number of iterations T; output is the final response sequence trajectory. The specific steps are as follows:

[0064] I. Initialize the sequence trajectory Operation counter ; II. Determine the operation counter If the number of operations is not greater than the preset maximum number of operations T, proceed to steps III to X; otherwise, proceed to step XI. III. Based on Professional Problem Vectors and policy network Generate response sequence ; IV. Response Sequence spliced ​​to the initial sequence trajectory of the response To obtain the updated sequence trajectory ; V. In the updated sequence trajectory Detection of visual-text search tag action commands <search>If a visual-text search tag action instruction is detected, proceed to step VI; otherwise, proceed to step VII. VI. In the updated sequence trajectory Parse the search query And based on the search query Searching for text-visual elements related to professional issues in vectorized databases. ; VII. In the updated sequence trajectory Instructions for labeling visual regions in the detection system <region>If a visual region labeling action command is detected, proceed to step VIII; otherwise, proceed to step IX. VIII. From the updated sequence trajectory Visual perception tokens were parsed from the data. And based on visual perception tokens loc Text-visual elements The visual elements in the image are subjected to pixel localization, pixel cropping, and pixel encoding operations to obtain visual region vectors. ; IX. In the updated sequence trajectory Instructions for generating tags for answer detection <answer>If an answer-generating-label action instruction is detected, execute step X; otherwise, update the operation counter. Return to step II; X. Concatenate the visual region vectors to the updated sequence trajectory according to the user's role. In order to Update again; XI. Output the updated sequence trajectory .

[0065] Table 1 is the pseudocode for the iterative reasoning and interaction process between the Visual Language Model (VLM) and the external environment provided by this invention.

[0066] Table 1 Figure 3 This document provides a second flowchart illustrating a professional-domain question-and-answer pair augmentation method. Figure 3 As shown, the method includes: S1 calling an API to augment a small number of question templates into a large number of questions, obtaining a domain-specific question dataset; S2 processing domain-specific documents to establish a domain-specific vectorized database; S3 training an agent using the GRPO reinforcement learning algorithm, performing multimodal, multi-step interaction and reasoning in each training round; S4 constructing an artificial intelligence agent based on the trained multimodal interaction model to achieve the acquisition of the augmented dataset for question answering. This invention significantly improves the agent's multimodal understanding ability. By introducing a dynamic visual perception mechanism, it completely solves the limitations of traditional text retrieval augmentation generation systems in processing multimodal documents.

[0067] The question-and-answer pair augmentation device provided by the present invention is described below. The question-and-answer pair augmentation device described below can be referred to in correspondence with the question-and-answer pair augmentation method described above.

[0068] Figure 4 For an example of a question-and-answer pair augmentation device in a specific field provided in this specification, please refer to the following diagram. Figure 4 As shown, the question-and-answer pair augmentation device in this professional field may include: The prior dataset acquisition module obtains a problem dataset and a vectorized database for the professional field. The problem dataset includes professional questions in the professional field; the vectorized database includes text element vectors corresponding to text information in the professional field and visual element vectors corresponding to non-text information in the professional field.

[0069] The agent training module is used to build agents for augmenting question-answer pairs in a specific domain. The agents are trained through reinforcement learning. During the reinforcement learning training process: The professional questions in the problem dataset are taken as the current state of the agent; a response containing multiple operation instructions is generated based on the current state. The operation instructions have a sequential order, and each operation instruction corresponds to a specific operation of the agent in the vectorized database. The agent parses the operation instructions in the response and executes each operation instruction in sequence, including: When the operation command is visual-text search, the system retrieves text element vectors and visual element vectors related to the professional question from the vectorized database to obtain a multimodal answer. When the operation instruction is visual region annotation, the image corresponding to the visual element vector in the initial answer is located and annotated to generate a visually annotated answer. When the operation instruction is to generate an answer, the multimodal answers and visually labeled answers accumulated from the previously executed visual-text search and visual region annotation are integrated to generate the target answer for the current state; The target answer and the generated response are evaluated, and the generation strategy for the next state response is adjusted based on the evaluation results.

[0070] The question-answer pair augmentation module is used to augment question-answer pairs based on the trained agent in a specific domain.

[0071] For specific limitations on the domain-specific question-and-answer augmentation device, please refer to the limitations on domain-specific question-and-answer augmentation methods mentioned above, which will not be repeated here. Each module in the aforementioned domain-specific question-and-answer augmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0072] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The Q&A provided in the professional field enhances the approach.

[0073] Figure 5 This specification provides a schematic diagram of a computer device, such as... Figure 5 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 The Q&A provided in the professional field enhances the approach.

[0074] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0075] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.< / answer> < / region> < / search> < / region> < / answer> < / region> < / search> < / answer> < / region> < / search>

Claims

1. A question-and-answer pair augmentation method in a professional field, characterized in that, The method comprises the following steps: obtaining a professional field question dataset and a vectorization database, wherein the professional field question dataset comprises professional questions in the professional field, and the vectorization database comprises text element vectors corresponding to professional field text information and visual element vectors corresponding to professional field non-text information; constructing an agent for augmenting professional field question and answer pairs, training the agent through reinforcement learning, and in the reinforcement learning training process: taking the professional questions in the question dataset as the current state of the agent; generating a response containing multiple operation instructions according to the current state, wherein the operation instructions have a sequence, and each operation instruction corresponds to a specific operation of the agent in the vectorization database; the agent analyzes the operation instructions in the response and executes each operation instruction in sequence, including: when the operation instruction is visual-text search, retrieving the text element vectors and visual element vectors related to the professional question in the vectorization database to obtain a multi-modal answer; when the operation instruction is visual region labeling, performing region positioning and labeling on the image corresponding to the visual element vector in the initial answer to generate a visual labeling answer; when the operation instruction is answer generation, integrating the multi-modal answer and the visual labeling answer accumulated by the executed visual-text search and visual region labeling to generate a target answer for the current state; evaluating the target answer and the generated response, and adjusting the generation strategy of the next state response according to the evaluation result; augmenting the professional field question and answer pairs based on the trained agent.

2. The expert domain question and answer augmentation method of claim 1, wherein, The agent analyzes the operation instructions in the response and executes each operation instruction in sequence, specifically including: I. Initialize response initial sequence track and operation counter ; II. determining operation counter whether not greater than a preset maximum operation number T, if yes, executing step III~step X; otherwise, executing step XI; III. Professional problem vector and policy network Generating a response sequence ; IV. concatenating the response sequence to the response initial sequence track to obtain an updated sequence track ; V. In the updated sequence track detecting a visual-text search tag action instruction, and if the visual-text search tag action instruction is detected, performing step VI; otherwise, performing step VII; VI. In the updated sequence trajectory Parse the search query And based on the search query Searching for text-visual elements related to professional issues in vectorized databases. ; VII. In the updated sequence track detecting a visual region annotation tag action instruction, and if the visual region annotation tag action instruction is detected, performing step VIII; otherwise, performing step IX; VIII. Parsing visual perception tokens from the updated sequence track loc loc performing pixel positioning, pixel cropping and pixel encoding operations on the visual elements in the text-visual elements​​​ IX. In the updated sequence track detecting an answer generation tag action instruction, if the answer generation tag action instruction is detected, performing step X; otherwise updating the operation counter returning to performing step II; X. Concatenate the visual region vectors to the updated sequence trajectory. In order to Update again; XI. outputting the updated response sequence trajectory .

3. The expert domain question and answer augmentation method of claim 2, wherein, The visual perception token loc Pixel positioning, pixel cropping and pixel encoding operations are performed on the visual elements in the text-visual element ​ by providing a visual perception token loc as input to the preset visual space mapping function, providing the preset visual space mapping function with a text-visual element in the visual element, cropping the visual element in the text-visual element by the preset visual space mapping function: wherein, represents a cropped key pixel region related to the professional problem obtained; represents a crop and zoom operation for the visual element in the text-visual element ; represents the visual element in the text-visual element ; represents a visual perception token loc positioned at a pixel boundary, is a minimum x-coordinate boundary, is a maximum x-coordinate boundary, is a minimum y-coordinate boundary, is a maximum y-coordinate boundary; and respectively represent a width dimension parameter and a height dimension parameter of an image corresponding to the visual element; and respectively represent a maximum processing width dimension and a maximum processing height dimension of a preset visual encoder; obtaining a key pixel region related to the professional problem by a preset visual encoder encoding.

4. The expert domain question and answer pair augmentation method of claim 2, wherein, The evaluation of the target answer and the generated response specifically includes: obtaining a retrieval efficiency reward, which is used to quantitatively evaluate the relevance of the target answer to the visual region vector and the rationality of the sorting of the target answer, and the calculation formula of the retrieval efficiency reward is: wherein, is a retrieval efficiency reward; is a set of all visual region vectors in a response sequence generated by the agent for the professional question; is a set of images most relevant to the professional question; is a normalized discount cumulative gain, ; is a relevance indicator function, ; is a th retrieved professional question relevant image; is an ideal discount cumulative gain, ; obtaining an answer accuracy reward, which is used to quantitatively evaluate the similarity between the target answer and the reference answer generated by the preset expert large model, and the calculation formula of the answer accuracy reward is: wherein, is a professional question; is a reference answer corresponding to the professional question generated by the preset expert large model; is an answer obtained by the agent; is a probability distribution of a random variable ; is an answer accuracy reward, subject to a probability distribution based on the professional question , the reference answer and the answer obtained by the agent , the higher the similarity between the reference answer and the answer obtained by the agent , the greater the answer accuracy reward ; obtaining a mode specification reward, which is used to quantitatively evaluate the format standardization degree of the response generated by the strategy network, and the calculation formula of the mode specification reward is: wherein, is a pattern normative reward; is a generated response sequence; represents a process of detecting a visual region annotation label action instruction in the updated sequence trajectory and obtaining a visual region vector according to the visual region annotation label action instruction; weighting and fusing the retrieval efficiency reward, the answer accuracy reward and the mode specification reward to obtain an evaluation reward value.

5. The expert domain question and answer augmentation method of claim 1, wherein, The acquisition step of the professional field question dataset specifically includes: constructing a structured question template of the professional field, wherein the structured question template comprises at least one variable slot; calling a generative artificial intelligence service interface to perform a multi-dimensional augmentation operation on the structured question template to obtain the professional field question dataset; the multi-dimensional augmentation operation includes: selecting semantically related entities to fill or replace the variable slot; rewriting the question after filling or replacing the variable slot through the generative artificial intelligence service interface; combining the question with context information such as a virtual scene, a document segment or a dialogue history to generate a scene question.

6. The expert domain question and answer augmentation method of claim 1, wherein, The acquisition step of the vectorization database specifically includes: Batch acquisition of original PDF documents in a professional field, convert the text content and non-text visual elements in the original PDF document into a unified image file object representation; and perform semantic block processing on the image file object representation to obtain a structured knowledge unit set; Perform adaptive resolution optimization on the visual elements in the structured knowledge unit set, and convert all visual elements in the structured knowledge unit set to the same color mode to obtain a standardized knowledge unit set; Perform vector encoding on each standardized knowledge unit set to obtain a knowledge unit vector set, each of which includes a text element vector and a visual element vector; Index the knowledge unit vector set to build a vectorization database including the index, text element vector, visual element vector, and original PDF document content.

7. The expert domain question and answer augmentation method of claim 1, wherein, The trained agent is used to augment the question and answer pairs in the professional field, including: Construct a multi-level dynamic prompt template including role definition prompts, process control prompts, and action specification prompts; wherein the role definition prompts are used to define the professional field expert identity and the question and answer pair generation task target; the process control prompts are used to guide the agent to perform a standardized question understanding-multimodal retrieval-information integration-answer generation process through a structured instruction sequence; the action specification prompts are used to define the trigger conditions and output format requirements of the visual-text search operation, visual region labeling operation, and answer generation operation; Obtain a professional problem dataset in the professional field to be augmented, and dynamically adjust the weights of the role definition prompts, process control prompts, and action specification prompts in the multi-level dynamic prompt template according to the question complexity and question type in the professional problem dataset to generate an optimal prompt strategy matching the professional problem dataset; The agent performs the visual-text search operation, visual region labeling operation, and answer generation operation in sequence based on the optimal prompt strategy to obtain the question and answer pair original output; Perform structured parsing, rule-based post-processing, and standardized output operations on the question and answer pair original output to obtain the question and answer pair augmented dataset in the professional field to be augmented.

8. A professional domain question and answer augmentation apparatus, characterized by, It includes: A priori dataset acquisition module, obtain a question dataset and a vectorization database in a professional field, wherein the question dataset includes professional questions in the professional field; the vectorization database includes text element vectors corresponding to professional field text information and visual element vectors corresponding to professional field non-text information; An agent training module for constructing an agent for augmenting professional field question and answer pairs, training the agent through reinforcement learning, and during the reinforcement learning training process: The professional questions in the question dataset are used as the current state of the agent; a response containing multiple operation instructions is generated based on the current state, the operation instructions have a sequence, and each operation instruction corresponds to a specific operation of the agent in the vectorization database; The agent parses the operation instructions in the response and executes each operation instruction in sequence, including: When the operation instruction is a visual-text search, a text element vector and a visual element vector related to the professional problem are retrieved from the vectorized database to obtain a multi-modal answer; When the operation instruction is a visual region labeling, a region positioning and labeling are performed on an image corresponding to the visual element vector in the initial answer to generate a visual labeling answer; When the operation instruction is an answer generation, multi-modal answers and visual labeling answers accumulated by the performed visual-text search and visual region labeling are integrated to generate a target answer of a current state; The target answer and the generated response are evaluated, and a generation strategy of a next state response is adjusted according to an evaluation result; The question and answer pair augmentation module is configured to augment the question and answer pair of the professional field based on the trained agent.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the professional field question and answer pair augmentation method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the professional field question and answer pair augmentation method of any one of claims 1 to 7.

Images