A structured question-answer generation and tracing method and system based on document classification

Abstract

The application belongs to the technical field of document intelligent processing, and discloses a structured question and answer generation and tracing method and system based on document classification, which comprises the following steps: receiving an input text, generating a stable document identifier according to the input text and an original file name; performing document-level three-level classification on the processed text to form a category label path; selecting a corresponding candidate question generation template and an answer generation template according to the category label path, and determining question and answer generation constraint information and tracing index information; performing graph-text organization and structured cutting on the processed text to obtain multiple text blocks and block-level information corresponding to each text block; obtaining a question and answer item according to the text block, the candidate question generation template and the answer generation template, and performing verification on the question and answer item; establishing a tracing relationship among the category label path, the stable text block identifier and the question and answer item according to the block-level information; and writing the text block, the question and answer item, the tracing relationship and the evidence item into a storage set.

Description

Technical Field

[0001] This invention belongs to the field of document intelligent processing technology, specifically relating to a structured question-and-answer generation and tracing method and system based on document classification. Background Technology

[0002] In intelligent question-answering scenarios involving standard documents, policy documents, notification documents, research materials, and mixed text and image documents, existing technologies typically employ two approaches: one uses a unified question-answer template to generate questions and answers for different document types, while the other uses structured segmentation followed by segmentation to generate questions and answers and establish a segment-level source mapping. The former approach fails to differentiate between the purpose, information structure, and evidence organization methods of different document categories, making it difficult to simultaneously address technical indicators in standard documents, responsibilities and obligations in policy documents, and time limits in notification documents, resulting in unstable question-answer generation results. The latter approach, while better preserving the correspondence between text blocks and question-answer items, typically uses segmentation results only as text mapping units, still struggling to create differentiated question-answering objectives based on document categories and failing to reliably transfer classification results to subsequent question-answer generation processes.

[0003] Furthermore, for mixed text and image documents containing charts, flowcharts, tables, or diagrams, existing image optical character recognition, image summarization, or image description technologies typically only obtain isolated image content. It's difficult to establish a stable correspondence between image content and specific text blocks, and it's also difficult to determine whether the image content semantically matches the current text block, making it difficult for image evidence to effectively enter the question-and-answer generation process. Existing question-and-answer database entry schemes typically only store a single-layer association between question-and-answer content and original text fragments or text blocks, making it difficult to further associate image identifiers, evidence fragments, category label paths, and corresponding confidence information. This results in low efficiency for subsequent retrieval, verification, auditing, and result backtracking. At the same time, existing schemes typically lack a closed-loop mechanism that uses category label paths to connect template selection, evidence organization, and database indexing, making it difficult to support automated question-and-answer data construction for different types of documents.

[0004] Therefore, it is necessary to provide a new technical solution that, while retaining the advantages of structured segmentation and stable traceability, introduces a document classification template selection mechanism, a mixed text and image content organization mechanism, and a traceability relationship construction mechanism to improve the question-and-answer generation adaptation capability and the ability to utilize text and image evidence in different types of documents. Summary of the Invention

[0005] This invention aims to address the shortcomings of existing technologies and provides the following solutions: A structured question-answering generation and source tracing method based on document classification includes the following steps: Receive input text, preprocess the input text to obtain processed text, and generate a stable document identifier based on the processed text and the original file name; The processed text is then subjected to document-level three-level classification to generate category label paths; Based on the category label path, select the corresponding candidate question generation template and answer generation template, and determine the question and answer generation constraint information and source index information corresponding to the category label path; The processed text is organized into text and structured blocks to obtain multiple text blocks for question and answer generation, as well as block-level information corresponding to each text block. The block-level information includes: block number, stable text block identifier, title path, parent path, and hierarchy depth. Question and answer items are obtained based on the text block, the candidate question generation template, and the answer generation template, and the question and answer items are validated. Based on the correspondence between the block sequence number and the stable text block identifier in the block-level information, the generation stage source number in the question and answer item is rewritten as the stable text block identifier, and the tracing relationship between the category tag path, the stable text block identifier and the question and answer item is established; Write the text block, the question and answer items, the source relationship, and the evidence items into the storage set.

[0006] Preferably, the method for obtaining the stable document identifier includes: When the input file is a text file, the text content is read directly. When the input file is a non-text file, the optical character recognition service is invoked to obtain the textualized result. When the textification result contains both lightweight markup language content and plain text content, the lightweight markup language content is used as the processed text. A content summary value is generated based on the content of the processed text; The stable document identifier is obtained by concatenating the original filename with the content digest value and then performing a hash calculation.

[0007] Preferably, the method for performing the three-level classification includes: For the processed text, document-level structural features, text features, and rule features are extracted; The primary, secondary, and tertiary categories are obtained based on the structural features, text features, and rule features. The first-level, second-level, and third-level categories are combined in hierarchical order to form the category label path, and the corresponding confidence level and classification basis are obtained.

[0008] Preferably, the method for obtaining the question-and-answer generation constraint information and the source index information includes: Parse the target category hierarchy information in the category label path; Select the candidate question generation template and the answer generation template that correspond to the target category hierarchy information from the preset template set; The candidate question generation template is used to determine the question type, information extraction target, and evidence organization rules. The answer generation template is used to constrain the answer expression method and verification rules. The question-and-answer generation constraint information and the source index information are determined according to the category label path.

[0009] Preferably, the method for organizing text and images includes: Replace the image content in the processed text with the corresponding image markers, and record the position of the image markers in the text; Based on the text block where the image tag is located, extract one or more pieces of information from the preceding text of the image, the following text of the image, the text block summary, or the title path to construct the image context; Perform image analysis on the image content to obtain the image analysis results.

[0010] Preferably, the method for structured segmentation includes: The processed text is divided into blocks according to the title structure, text separation rules, or preset segmentation rules; Assign hierarchical numbers to each text block according to the order in which the heading or text block appears under the same parent path; The block number, the stable text block identifier, the title path, the parent path, and the level depth are generated based on the level number.

[0011] Preferably, the method for performing validation on the question-and-answer items includes: The generated question and answer items are subjected to field normalization processing, question type normalization verification, source fact missing verification, fuzzy reference verification, source fact granularity verification, and original text support verification based on the corresponding text block content. If the number of questions and answers generated initially does not reach the preset target, does not meet the question type plan, fails the original text support verification, or fails the image and text evidence verification, a corrective regeneration will be performed.

[0012] Preferably, the method for validating the question-and-answer items further includes validating the textual and image evidence: The image-text fit score is calculated based on the semantic similarity between the image analysis results and the content of the text block, the overlap between the image keywords and the text block keywords, the proximity between the image marker position and the text block boundary, the matching degree between the image type and the template evidence requirements, and the matching degree between the image title or caption and the title path. When the image-text matching score is greater than or equal to a preset threshold, the corresponding image evidence is retained. When the image-text matching score is lower than a preset threshold, the corresponding image evidence is removed or prohibited from participating in answer generation.

[0013] Preferably, the method for establishing the source relationship includes: Each question and answer item is associated with one or more of the stable document identifier, the stable text block identifier, and the evidence fragment identifier; When performing verification on image and text evidence, further associate one or more of the image identifier and category label paths; Index information based on the stable text block identifier, the image identifier, the question-and-answer identifier, and the category tag path is established in the storage set respectively; Upon receiving a query request, the system uses the stable text block identifier, the image identifier, the question and answer identifier, or the category tag path as filtering conditions to query the associated question and answer items, evidence items, and corresponding category tag paths from the storage set.

[0014] This invention also provides a structured question-and-answer generation and tracing system based on document classification. The system applies the above-mentioned method and includes: a preprocessing module, a classification module, a template selection module, a structured segmentation module, a question-and-answer generation and verification module, a tracing module, and a database query module. The preprocessing module is used to receive input text, preprocess the input text to obtain processed text, and generate a stable document identifier based on the processed text and the original file name; The classification module is used to perform document-level three-level classification on the processed text to form category label paths; The template selection module selects the corresponding candidate question generation template and answer generation template according to the category tag path, and determines the question and answer generation constraint information and source index information corresponding to the category tag path; The structured segmentation module is used to perform graphic organization and structured segmentation on the processed text to obtain multiple text blocks for question and answer generation and block-level information corresponding to each text block. The block-level information includes: block number, stable text block identifier, title path, parent path, and hierarchy depth. The question-and-answer generation and verification module obtains question-and-answer items based on the text block, the candidate question generation template, and the answer generation template, and performs verification on the question-and-answer items; The tracing module rewrites the generation stage source number in the question and answer item into a stable text block identifier based on the correspondence between the block sequence number and the stable text block identifier in the block-level information, and establishes the tracing relationship between the category tag path, the stable text block identifier and the question and answer item. The database query module is used to write the text block, the question and answer item, the source relationship, and the evidence item into the storage set.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention adds a document-level three-level classification step before question and answer generation, and combines the first-level category, the second-level category and the third-level category to form a category label path. The category label path then drives the selection of question and answer generation template, determination of information extraction target, evidence organization and database index configuration. This enables standard documents, legal and regulatory documents, document notices and other types of documents to correspond to different information extraction targets and evidence organization methods, thereby improving the stability of question and answer generation in different document types and enhancing the subsequent traceability and closed-loop capability.

[0016] (2) This invention solves the problem that image evidence is difficult to participate in question and answer generation in mixed text documents by inserting image markers into the input text and establishing the correspondence between image identifiers, image markers and stable text block identifiers, and generating image analysis results and image context at the same time. This allows image content to enter the question and answer generation chain together with text content.

[0017] (3) This invention constructs image context information such as image background, text block summary and title path, and calculates image-text fit by combining one or more of semantic similarity, keyword overlap, position proximity and image type matching degree, so that only image evidence that semantically matches the current text block participates in answer generation, thereby reducing the risk of misuse or cross-block drift of image analysis results.

[0018] (4) This invention rewrites the generation stage source number in the question and answer item into a stable text block identifier and establishes a multi-level tracing relationship between the question and answer item and the text block, image content, evidence fragments and category tag path. At the same time, the category tag path is included in the database index and related queries. It not only supports reverse lookup of question and answer items based on the stable text block identifier, but also supports related retrieval based on image identifier, question and answer identifier or category tag path, thereby improving the efficiency of result verification and subsequent reuse.

[0019] (5) This invention generates stable document identifiers based on input text and original file name, so that the same document can maintain consistent association when repeated parsing, regenerating questions and answers and cross-processing review, thereby forming a unified identifier chain of document-level identifiers, text block-level identifiers and question-and-answer item-level traceability relationships.

[0020] (6) By implementing differentiated template selection, evidence organization and question-and-answer verification based on category label path, this invention can automatically construct high-quality and traceable question-and-answer data, thereby reducing the cost of manually constructing question-and-answer datasets. Attached Figure Description

[0021] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the three-level classification and template selection process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structured segmentation and image tagging and positioning process for mixed text and image documents according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the question-and-answer generation, image and text evidence verification, and repair-based regeneration process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the multi-level tracing relationship and storage structure in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Example 1: In this embodiment, as Figure 1 As shown, a structured question-answering generation and tracing method based on document classification includes the following steps: S1. Receive input text, preprocess the input text to obtain processed text, and generate a stable document identifier based on the processed text and the original file name.

[0026] The methods for obtaining stable document identifiers include: directly reading the text content when the input file is a text file; calling the optical character recognition service to obtain the textification result when the input file is a non-text file; when the textification result contains both lightweight markup language content and ordinary text content, the lightweight markup language content is used as the processed text first; generating a content digest value based on the content of the processed text; and performing hash calculation after concatenating the original file name and the content digest value to obtain the stable document identifier.

[0027] In this embodiment, one or more uploaded files are first received, and it is determined whether each input file needs to undergo optical character recognition (OCR) textification processing. For files with a text extension, text content, or those identified as text content, their text content is directly read. For portable document format files, image files, word processing document files, or other files identified as non-text, they are added to the OCR candidate queue.

[0028] The candidate file is processed using an optical character recognition service. If the returned result contains both lightweight markup language (JCL) content and plain text content, the JCL content is prioritized as the input for subsequent classification, image and text organization, and structured segmentation to preserve heading levels, paragraph boundaries, and list structures as much as possible. If the returned result does not contain JCL content, plain text content is used as the input for subsequent processing. If the returned result also contains image slice results, the image slice results are used as the input for subsequent image tagging and localization, image analysis result generation, and image context construction.

[0029] In one specific implementation, the input processing stage can also assign a document-level task identifier to the input file and record one or more of the following: original file name, file type, page number information, image page position information, and text processing status. These serve as auxiliary metadata for subsequent category tag path recording, image location, structured segmentation, and question-and-answer tracing.

[0030] Furthermore, let the input text be... T _ doc The original file name is N _ doc For input text T _ doc The digest value is obtained after performing the content digest calculation. h _ doc Then stable document identifier D _ doc It can be represented as: , , in, HThis represents the hash calculation function. Using the above method, the stable document identifier remains consistent when the original filename and input text remain unchanged; however, when the input text changes, the digest value changes accordingly, thus changing the stable document identifier.

[0031] S2. Perform document-level three-level classification on the processed text to generate category label paths.

[0032] The method for three-level classification includes: extracting document-level structural features, text features, and rule features from the processed text; obtaining primary, secondary, and tertiary categories based on the structural, text, and rule features; combining the primary, secondary, and tertiary categories in hierarchical order to form category label paths, and obtaining the corresponding confidence scores and classification criteria.

[0033] In this embodiment, as Figure 2 As shown, document-level structural features, text features, and rule features are extracted from the input text; based on the structural features, text features, and rule features, first-level categories, second-level categories, and third-level categories are output; and the first-level categories, second-level categories, and third-level categories are combined to form category label paths.

[0034] Specifically, the system extracts the title, first paragraph, local structural regions, keywords, numbering patterns, date patterns, chapter patterns, and other structural signals from the input text. It then outputs first-level, second-level, and third-level categories using rule-based classification, model-based classification, or a combination of rule and model classification. Subsequently, the first-level, second-level, and third-level categories are connected using separators in a preset order to form a category label path, preferably a path string of "first-level category / second-level category / third-level category". When a category level is empty, a preset catch-all category can be used to fill in the gap. Simultaneously, the system records the classification confidence and the basis for the hit. Classification confidence may include one or more of the following: route confidence, second-level category confidence, third-level category confidence, and final label confidence. The basis for the hit may include one or more of the following: the hit rule identifier, the candidate category with the highest score, or the evidence summary output by the model.

[0035] S3. Select the corresponding candidate question generation template and answer generation template according to the category label path, and determine the question and answer generation constraint information and traceability index information corresponding to the category label path.

[0036] The methods for obtaining question-and-answer generation constraint information and source index information include: parsing the target category level information in the category label path; selecting candidate question generation templates and answer generation templates corresponding to the target category level information from a preset template set; using the candidate question generation templates to determine the question type, information extraction target and evidence organization rules, using the answer generation templates to constrain the answer expression method and verification rules, and determining the question-and-answer generation constraint information and source index information based on the category label path.

[0037] In this embodiment, the main template type is selected based on the category tag path, such as... Figure 2 As shown, the process involves supplementing sub-template constraints based on second-level and third-level categories; and outputting candidate question generation templates and answer generation templates. Specifically, based on the preset hierarchical category information in the category label path, corresponding candidate question generation templates and answer generation templates are selected from the template set. Preferably, the first-level category information in the category label path is parsed first, and then the first-level category information is mapped to template type identifiers; for the same main template type, sub-template constraints or preset field constraints can also be supplemented based on second-level and third-level categories to adjust the question type ratio, key extraction fields, evidence planning granularity, or answer expression method respectively. The candidate question generation template is used to determine the question type, information extraction target, evidence planning fields, and disabled question modes, while the answer generation template is used to constrain the answer expression method, evidence organization method, source fact field output, original text support method, and empty result judgment conditions. In addition, the question and answer item output fields, evidence type selection strategy, and database index field configuration can be determined based on the category label path, so that the category label path runs through the question and answer generation, evidence organization, and result tracing process.

[0038] Furthermore, it can output the classification confidence score corresponding to the category label path. The classification confidence score can be determined by combining the classification scores of the first-level category, second-level category, and third-level category. Whether to directly use the current category label path or switch to the fallback template can be determined based on whether the classification confidence score is greater than a preset threshold.

[0039] S4. Perform graphic organization and structured segmentation on the processed text to obtain multiple text blocks for question-and-answer generation, along with block-level information corresponding to each text block. This block-level information includes: block number, stable text block identifier, title path, parent path, and hierarchy depth. For example... Figure 3 As shown.

[0040] The method for organizing images and text includes: replacing the image content in the processed text with the corresponding image tags and recording the position of the image tags in the text; extracting one or more pieces of information from the preceding text, following text, text block summary, or title path based on the text block where the image tags are located to construct the image context; and performing image analysis on the image content to obtain the image analysis results.

[0041] In this embodiment, image markers are inserted at the corresponding image positions in the input text, and the offset position, page, or region information of the image markers in the text is recorded. Preferably, each image marker adopts a unified marker format, which can be a marker string containing a preset prefix and an image sequence number. After inserting the image markers, the correspondence between the image markers and image identifiers is recorded, and the block number or text block identifier of the text block where the image marker is located is also recorded, thereby establishing the positioning relationship between the image identifier, the image marker, and the text block. For the image content, image analysis results are further generated. The image analysis results can be generated by one or more of the following: image optical character recognition, layout recognition, table recognition, flowchart recognition, image description model, and image classification model, and can include at least one or more of the following: image text, image type, image summary, key entities in the image, relationships in the image, or structured fields in the image. Furthermore, one or more information from the preceding text, following text, text block summary, and title path can be extracted from the text block where the image marker is located as the image context.

[0042] The structured segmentation method includes: segmenting the processed text into blocks according to the heading structure, text separation rules, or preset segmentation rules; assigning hierarchical numbers to each text block according to the order in which the heading or text block appears under the same parent path; and generating block numbers, stable text block identifiers, heading paths, parent paths, and hierarchical depths based on the hierarchical numbers.

[0043] In this embodiment, during the structured segmentation stage, instead of performing a simple fixed-length truncation of the entire text, the document's heading structure, paragraph boundaries, list boundaries, and preset segmentation lengths are prioritized for segmentation, and corresponding block-level information is generated for each text block. Specifically, the heading path corresponding to the text block is first extracted; then, under the same parent path, each level is assigned a sequence number according to the order in which the heading or text block appears, and the sequence numbers of each level are combined layer by layer to form a hierarchical path; subsequently, the parent path, root path, and level depth are generated based on the hierarchical path to clarify the position of the text block in the document tree. For text blocks containing image markers, the correspondence between the image markers and the text block is preserved, so that the text block contains both text structure information and image-text association information.

[0044] In a specific implementation, block-level information may include one or more of the following: stable text block identifier, block number, hierarchical path, title path, parent path, root path, hierarchical depth, block text, image tag set, and input text used to represent the semantics of the text block. The structured block segmentation step simultaneously outputs text block results for question-answering generation and block-level metadata results for source tracing management. The former is used as input to the question-answering generation model, and the latter is used for subsequent source number rewriting and multi-level source tracing relationship establishment.

[0045] In a specific implementation, let the stable document identifier be...D _ doc The hierarchical path is L _ idx The title path is L _ title The text block content is T _ blk Then the stable text block identifier C _ blk It can be represented as: .

[0046] Using the above method can ensure that text blocks maintain stable identification during repeated data entry, Q&A retrieval, and cross-process review.

[0047] The lowest-level text block is written as an input object to the text block collection. The written content includes at least the text block body, path information representing the text block's position in the document tree, image tag information, and a vector representation generated from the representation text. In a specific implementation, a stable text block identifier can be generated jointly by the task identifier, document-level task identifier, hierarchical path, title path, and block text, thus giving text blocks in different documents or at different locations a stable and distinguishable identifier. For any text block, the parent title prefix in the current text block's title path can also be extracted, and concatenated with the current block text to obtain the representation text used to represent the hierarchical semantics of the text block. By using the title path prefix and the block body together for subsequent vectorization representation, the hierarchical semantics of the current text block can be preserved during the representation stage, thus providing a unified text block storage foundation for block-level retrieval, question-answering tracing, and association lookup.

[0048] S5. Obtain question and answer items based on text blocks, candidate question generation templates, and answer generation templates, and perform validation on the question and answer items.

[0049] The methods for verifying question and answer items include: performing field normalization processing, question type normalization verification, source fact missing verification, fuzzy reference verification, source fact granularity verification, and original text support verification based on the corresponding text block content on the generated question and answer items; when the number of question and answer items generated for the first time does not reach the preset target, does not meet the question type plan, fails the original text support verification, or fails the graphic evidence verification, a repair-style regeneration is performed.

[0050] The method for verifying question-and-answer items also includes verifying image and text evidence: performing image analysis on the image content to obtain the image analysis results; calculating the image-text fit score based on the semantic similarity between the image analysis results and the content of the text block, the overlap between image keywords and text block keywords, the proximity of the image marker position to the text block boundary, the matching degree between the image type and the template evidence requirements, and the matching degree between the image title or caption and the title path; when the image-text fit score is greater than or equal to a preset threshold, the corresponding image evidence is retained; when the image-text fit score is lower than the preset threshold, the corresponding image evidence is removed or prohibited from participating in answer generation.

[0051] In this embodiment, as Figure 4 As shown, the question-and-answer generation model is called concurrently according to text blocks; candidate question-and-answer items are output according to question type patterns and template constraints; field normalization, question type constraints, source fact verification, and original text support verification are performed on the candidate question-and-answer items; image and text evidence verification is performed when image analysis results exist; and repair regeneration is performed when the number of valid question-and-answer items is insufficient, question types are missing, or evidence verification fails.

[0052] Specifically, each text block is treated as an independent question-and-answer generation unit, and parallel question-and-answer generation processing is performed on multiple text blocks according to a preset scheduling strategy to improve the efficiency of question-and-answer generation in multi-text-block scenarios. For any text block, corresponding question-and-answer items are generated according to the preset target number of questions and answers and question type pattern for each block. Each question-and-answer item includes at least the question content, answer content, source factual text, and source field; in a specific implementation, the question-and-answer item may also include one or more of the following: answer description, question type field, source text block title path field, supporting evidence stable text block identifier set field, knowledge category field, knowledge category confidence field, and knowledge category basis field.

[0053] During the question-and-answer generation phase, each text block is first assigned a generation phase source number, which is then appended to the candidate question-and-answer items. Subsequently, field normalization, question type constraints, source fact verification, fuzzy referencing verification, original text support verification, and image-text evidence verification are performed on the question-and-answer items. For mixed image-text scenarios, an image-text fit score is calculated based on the matching relationship between the image analysis results and the current text block content. Preferably, the image-text fit score is obtained by weighting one or more of the following: semantic similarity between the image analysis text and the text block content; overlap between image keywords and text block keywords; proximity between the image marker position and the text block boundary; matching degree between image type and template evidence requirements; and matching degree between image title or caption and title path. When the image-text fit score is greater than or equal to a preset threshold, the corresponding image evidence is retained and allowed to participate in answer generation; when the image-text fit score is lower than the preset threshold, the corresponding image evidence is removed or prohibited from participating in answer generation.

[0054] In a specific implementation, let the semantic similarity score between the image analysis text and the text block content be . s _ sem The overlap score between image keywords and text block keywords is: s _ kw The proximity score between the image marker location and the text block boundary is s _ pos The matching score between the image type and the template evidence requirement is: s _ type The matching score between the image title or caption and the title path is... s _ title The corresponding weighting coefficients are respectively , , , , Then the image and text matching score S _ img _ txt It can be represented as: , .

[0055] Furthermore, when the image-text relevance score is... S _ img _ txt Greater than or equal to the preset threshold T _ img When the image and text match score is high, retain the current image evidence; S _ img _ txt Less than the preset threshold T _ img At that time, the current image evidence will be removed or it will be prohibited from participating in answer generation.

[0056] If the initial generated results do not reach the preset target number, do not meet the preset question type plan, fail the original text support verification, or fail the image and text evidence verification, a corrective regeneration is performed. The corrective regeneration can reconstruct the constraint input based on the missing question type, the field that failed the verification, or the missing evidence type to supplement the valid question and answer items.

[0057] S6. Based on the correspondence between block sequence number and stable text block identifier in block-level information, rewrite the generation stage source number in the question and answer item as stable text block identifier, and establish the tracing relationship between category label path, stable text block identifier and question and answer item.

[0058] The methods for establishing traceability relationships include: associating each question-and-answer item with one or more of stable document identifiers, stable text block identifiers, and evidence fragment identifiers; when performing verification on text and image evidence, further associating it with one or more of image identifiers and category tag paths; establishing index information in the storage set based on stable text block identifiers, image identifiers, question-and-answer identifiers, and category tag paths respectively; and upon receiving a query request, querying the associated question-and-answer items, evidence items, and corresponding category tag paths from the storage set based on stable text block identifiers, image identifiers, question-and-answer identifiers, or category tag paths as filtering conditions.

[0059] In this embodiment, as Figure 5 As shown, the source field of the question and answer item is rewritten according to the correspondence between the source number of the generation stage and the stable text block identifier; one or more of the stable text block identifier, image identifier, evidence fragment identifier and category label path are associated with the question and answer item.

[0060] Specifically, after the question-and-answer item verification is passed, based on the correspondence between the generation stage source number and the stable text block identifier in the block-level metadata, the generation stage source number in the candidate question-and-answer item is rewritten as the stable text block identifier to form a source field that can be directly used for database entry and retrieval. For question-and-answer items containing image evidence, image identifier, image type field, image fit score field, or image description field can be further associated; for all question-and-answer items, category tag path, knowledge category confidence, and knowledge category basis can be further associated.

[0061] S7. Write the text block, question and answer items, source relationships, and evidence items into the storage set.

[0062] In this embodiment, as Figure 5 As shown, text block records, question and answer records, and evidence records are written to their respective storage sets; related queries are provided based on stable text block identifiers, image identifiers, question and answer identifiers, or category tag paths.

[0063] Specifically, the rewritten question and answer items are written into a storage set. The storage set preferably has an index structure based on stable text block identifiers, image identifiers, question and answer identifiers, and category tag paths. Upon receiving a query request, the storage set can be queried for related question and answer items, evidence items, image information, and corresponding category tag paths based on stable text block identifiers, image identifiers, question and answer identifiers, or category tag paths as filtering conditions, enabling bidirectional or multidirectional tracing.

[0064] Two-way or multi-way indexes can be built based on question-and-answer identifiers, stable text block identifiers, image identifiers, evidence fragment identifiers, and category label paths to support reverse lookup of text blocks from question-and-answer items, reverse lookup of question-and-answer items from text blocks, and related retrieval based on image identifiers or category label paths.

[0065] Example 2: In this embodiment, a structured question-and-answer generation and tracing system based on document classification includes: a preprocessing module, a classification module, a template selection module, a structured segmentation module, a question-and-answer generation and verification module, a tracing module, and a database query module.

[0066] The preprocessing module receives the input text, preprocesses it to obtain the processed text, and generates stable document identifiers based on the processed text and the original filename. The classification module performs document-level three-level classification on the processed text, forming category label paths. The template selection module selects corresponding candidate question generation templates and answer generation templates based on the category label paths, and determines the question-and-answer generation constraints and source index information corresponding to the category label paths. The structured segmentation module performs graphic organization and structured segmentation on the processed text, obtaining multiple text blocks for question-and-answer generation, as well as the corresponding text blocks. The module generates and verifies the corresponding block-level information, including: block number, stable text block identifier, title path, parent path, and hierarchy depth. The question-and-answer generation and verification module generates question-and-answer items based on the text block, candidate question generation template, and answer generation template, and performs verification on the question-and-answer items. The tracing module rewrites the generation stage source number in the question-and-answer items with the stable text block identifier based on the correspondence between the block number and the stable text block identifier in the block-level information, and establishes the tracing relationship between the category label path, the stable text block identifier, and the question-and-answer items. The database query module is used to write the text block, question-and-answer items, tracing relationship, and evidence items into the storage set.

[0067] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A structured question-answering generation and source tracing method based on document classification, characterized in that, Includes the following steps: Receive input text, preprocess the input text to obtain processed text, and generate a stable document identifier based on the processed text and the original file name; The processed text is then subjected to document-level three-level classification to generate category label paths; Based on the category label path, select the corresponding candidate question generation template and answer generation template, and determine the question and answer generation constraint information and source index information corresponding to the category label path; The processed text is organized into text and structured blocks to obtain multiple text blocks for question and answer generation, as well as block-level information corresponding to each text block. The block-level information includes: block number, stable text block identifier, title path, parent path, and hierarchy depth. Question and answer items are obtained based on the text block, the candidate question generation template, and the answer generation template, and the question and answer items are validated. Based on the correspondence between the block sequence number and the stable text block identifier in the block-level information, the generation stage source number in the question and answer item is rewritten as the stable text block identifier, and the tracing relationship between the category tag path, the stable text block identifier and the question and answer item is established; Write the text block, the question and answer items, the source relationship, and the evidence items into the storage set.

2. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The methods for obtaining the stable document identifier include: When the input file is a text file, the text content is read directly. When the input file is a non-text file, the optical character recognition service is invoked to obtain the textualized result. When the textification result contains both lightweight markup language content and plain text content, the lightweight markup language content is used as the processed text. A content summary value is generated based on the content of the processed text; The stable document identifier is obtained by concatenating the original filename with the content digest value and then performing a hash calculation.

3. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The method for performing the three-level classification includes: For the processed text, document-level structural features, text features, and rule features are extracted; The primary, secondary, and tertiary categories are obtained based on the structural features, text features, and rule features. The first-level, second-level, and third-level categories are combined in hierarchical order to form the category label path, and the corresponding confidence level and classification basis are obtained.

4. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The methods for obtaining the question-and-answer generation constraint information and the source index information include: Parse the target category hierarchy information in the category label path; Select the candidate question generation template and the answer generation template that correspond to the target category hierarchy information from the preset template set; The candidate question generation template is used to determine the question type, information extraction target, and evidence organization rules. The answer generation template is used to constrain the answer expression method and verification rules. The question-and-answer generation constraint information and the source index information are determined according to the category label path.

5. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The method for organizing text and images includes: Replace the image content in the processed text with the corresponding image markers, and record the position of the image markers in the text; Based on the text block where the image tag is located, extract one or more pieces of information from the preceding text of the image, the following text of the image, the text block summary, or the title path to construct the image context; Perform image analysis on the image content to obtain the image analysis results.

6. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The method for structured segmentation includes: The processed text is divided into blocks according to the title structure, text separation rules, or preset segmentation rules; Assign hierarchical numbers to each text block according to the order in which the heading or text block appears under the same parent path; The block number, the stable text block identifier, the title path, the parent path, and the level depth are generated based on the level number.

7. The structured question-answering generation and tracing method based on document classification according to claim 1, characterized in that, The method for performing validation on the question-and-answer items includes: The generated question and answer items are subjected to field normalization processing, question type normalization verification, source fact missing verification, fuzzy reference verification, source fact granularity verification, and original text support verification based on the corresponding text block content. If the number of questions and answers generated initially does not reach the preset target, does not meet the question type plan, fails the original text support verification, or fails the image and text evidence verification, a corrective regeneration will be performed.

8. The structured question-answering generation and tracing method based on document classification according to claim 7, characterized in that, The method for validating the question-and-answer items also includes validating the textual and image evidence: The image-text fit score is calculated based on the semantic similarity between the image analysis results and the content of the text block, the overlap between the image keywords and the text block keywords, the proximity between the image marker position and the text block boundary, the matching degree between the image type and the template evidence requirements, and the matching degree between the image title or caption and the title path. When the image-text matching score is greater than or equal to a preset threshold, the corresponding image evidence is retained. When the image-text matching score is lower than a preset threshold, the corresponding image evidence is removed or prohibited from participating in answer generation.

9. The structured question-answering generation and tracing method based on document classification according to claim 8, characterized in that, The methods for establishing the aforementioned source relationship include: Each question and answer item is associated with one or more of the stable document identifier, the stable text block identifier, and the evidence fragment identifier; When performing verification on image and text evidence, further associate one or more of the image identifier and category label paths; Index information based on the stable text block identifier, the image identifier, the question-and-answer identifier, and the category tag path is established in the storage set respectively; Upon receiving a query request, the system uses the stable text block identifier, the image identifier, the question and answer identifier, or the category tag path as filtering conditions to query the associated question and answer items, evidence items, and corresponding category tag paths from the storage set.

10. A structured question-answering generation and tracing system based on document classification, wherein the system applies the method described in any one of claims 1-9, characterized in that, include: The module includes a preprocessing module, a classification module, a template selection module, a structured segmentation module, a question and answer generation and validation module, a traceability module, and an input query module. The preprocessing module is used to receive input text, preprocess the input text to obtain processed text, and generate a stable document identifier based on the processed text and the original file name; The classification module is used to perform document-level three-level classification on the processed text to form category label paths; The template selection module selects the corresponding candidate question generation template and answer generation template according to the category tag path, and determines the question and answer generation constraint information and source index information corresponding to the category tag path; The structured segmentation module is used to perform graphic organization and structured segmentation on the processed text to obtain multiple text blocks for question and answer generation and block-level information corresponding to each text block. The block-level information includes: block number, stable text block identifier, title path, parent path, and hierarchy depth. The question-and-answer generation and verification module obtains question-and-answer items based on the text block, the candidate question generation template, and the answer generation template, and performs verification on the question-and-answer items; The tracing module rewrites the generation stage source number in the question and answer item into a stable text block identifier based on the correspondence between the block sequence number and the stable text block identifier in the block-level information, and establishes the tracing relationship between the category tag path, the stable text block identifier and the question and answer item. The database query module is used to write the text block, the question and answer item, the source relationship, and the evidence item into the storage set.

Images