Anchors graph based multi-modal evidence question answering method

Abstract

The application discloses a multi-modal evidence question answering method based on an anchor graph, and the method comprises the following steps: establishing a vector index comprising a plurality of stable evidence segments; obtaining a query request, generating a query vector based on the query request, and obtaining a preliminary screening set comprising a plurality of preliminary screening evidence segments from the vector index matched with the query vector; determining a minimum evidence set satisfying a constraint condition based on the preliminary screening set; generating an answer content based on the minimum evidence set, and cutting the answer content into a plurality of answer units; determining a support alignment score between each answer unit and each stable evidence segment in the minimum evidence set, establishing an answer unit support graph based on each support alignment score, and outputting a proof package of each answer unit based on the answer unit support graph.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a multimodal evidence question-answering method based on anchor graphs. Background Technology

[0002] Existing large-scale question answering and retrieval enhancement generation systems typically only output document links, paragraph numbers, or source tags at the end of the answer. While such solutions can provide some indication of the answer's source, they usually only achieve document-level or fragment-level citations, making it difficult to trace further to fine-grained locations such as logical text positions, local image regions, audio time segments, video frames, and table cell blocks.

[0003] For multimodal objects such as text, images, audio, video, and tables, existing technologies typically employ separate location representations, making it difficult to establish a unified, computable location representation across different modalities. Simultaneously, existing solutions often use a method of directly concatenating recalled fragments into the prompt context after sorting by similarity during the retrieval phase, lacking a process for calculating the minimum sufficient evidence set required to satisfy the answer, easily introducing a large amount of redundant noise. Furthermore, even when existing solutions output citations, they usually only indicate that a piece of evidence is relevant to the answer, without further quantifying whether a piece of evidence is truly necessary for a particular answer. Summary of the Invention

[0004] To address the existing technical problems, this application provides a multimodal evidence question-answering method based on anchor graphs, which can quantify the evidence corresponding to the answers and remove redundant noise, thereby improving the verifiability of the questions and answers.

[0005] In a first aspect, a multimodal evidence question-answering method based on anchor graphs is provided, comprising: establishing a vector index including multiple stable evidence fragments; obtaining a query request, generating a query vector based on the query request, and selecting a preliminary set of multiple preliminary evidence fragments from the vector index that match the query vector; determining a minimum evidence set that satisfies constraints based on the preliminary set; generating answer content based on the minimum evidence set and dividing the answer content into multiple answer units; determining the support alignment score between each answer unit and each stable evidence fragment in the minimum evidence set, and establishing an answer unit support graph based on each support alignment score; and outputting a proof package for each answer unit based on the answer unit support graph.

[0006] This application addresses the problems of coarse retrieval granularity and untraceable evidence sources in traditional schemes by establishing a vector index containing multiple stable evidence fragments, thus refining multimodal raw data into searchable and traceable evidence units. Based on this, it achieves cross-modal semantic-driven evidence retrieval by acquiring query requests and generating query vectors, and semantically recalling the initial screening set from the vector index, overcoming the poor generalization ability of keyword matching. Furthermore, by solving for the minimum evidence set satisfying sufficiency and minimumity constraints based on the initial screening set, redundant noise is eliminated while ensuring the integrity of the response information, solving the problems of invalid content occupying the context window and broken evidence chains leading to model comprehension bias. Based on this, a minimum evidence set is then used to further refine the retrieval process. The system generates and segments response content into multiple response units, providing a fine-grained structural foundation for sentence-by-sentence attribution. This overcomes the shortcomings of traditional solutions, which only offer document-level overall references and lack sufficient granularity for attribution. Furthermore, by determining the support alignment scores between each response unit and each stable evidence fragment and establishing a support graph for each response unit, the system outputs a proof package for each response unit based on the support graph. This achieves structured auditable proof that includes necessary evidence attribution, necessity score, version context, tamper-proof signature, and credibility assessment. The entire technical solution forms a complete closed loop from multimodal evidence digitization, semantic retrieval and recall, evidence minimization and compression, response generation and fine-grained segmentation, sentence-by-sentence attribution graph construction to the output of structured auditable proof. Attached Figure Description

[0007] Figure 1 This is an application environment diagram of a multimodal evidence question-answering method based on anchor point graphs in one embodiment; Figure 2 This is a flowchart of a multimodal evidence question-answering method based on anchor graphs in one embodiment; Figure 3 This is a flowchart illustrating the process of establishing a vector index comprising multiple stable evidence fragments in one embodiment. Figure 4 This is a schematic diagram of a multimodal evidence question-and-answer device based on an anchor point graph in one embodiment; Figure 5 This is a schematic diagram of a computing device in one embodiment. Detailed Implementation

[0008] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0009] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0010] In the following description, the expression “some embodiments” refers to a subset of all possible embodiments. However, it should be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0011] See Figure 1 In one embodiment, the application environment diagram of the multimodal evidence question-answering method based on anchor graphs includes multiple terminal devices 100 and a computing device 200. The terminal devices 100 can communicate with the computing device 200. The multimodal evidence question-answering method based on anchor graphs provided in this application embodiment can be applied to the computing device 200, which communicates with a database. The computing device 200 obtains query requests from the terminal devices 100 and outputs proof packets for each answer unit corresponding to the query request.

[0012] Existing solutions, even when outputting citations, typically only indicate a correlation between evidence and an answer, but cannot further quantify whether a particular piece of evidence is truly necessary for a particular answer. Especially when changes occur in object content, preprocessing flow, embedding model, index parameters, rearrangement rules, or prompt templates, existing solutions can usually only observe changes in the answer, but struggle to attribute the source of those changes in fine detail. At the engineering deployment level, for scenarios with long contexts, complex multimodal tasks, or multiple answer units, performing complete edge-replay of all evidence individually would incur high computational overhead and latency costs, making stable deployment in multimodal AI infrastructures difficult.

[0013] Please see Figure 2 This is a flowchart of a multimodal evidence question-answering method based on anchor graphs provided in an embodiment of this application. The anchor graph-based multimodal evidence question-answering method is applied in a computing device and includes the following steps: S10. Establish a vector index that includes multiple stable evidence fragments.

[0014] In this embodiment, a stable evidence fragment is an evidence block obtained by concatenating one or more atomic units, i.e., the smallest multimodal slices, such as a sentence, a table cell, or an image region, according to semantic integrity, structural coherence, and logical consistency. A stable evidence fragment is the smallest usable unit of evidence that the system provides services to the outside world; every sentence in the response must ultimately be attributed to a stable evidence fragment. A vector index is a storage structure that converts the semantic information of stable evidence fragments into high-dimensional real-valued vectors and organizes these vectors using specific data structures, such as HNSW graphs or inverted indexes, to support efficient approximate nearest neighbor retrieval.

[0015] S11. Obtain the query request, generate a query vector based on the query request, and select a preliminary set of multiple preliminary evidence fragments from the vector index that match the query vector.

[0016] A query request is a natural language question submitted by a user to the system, serving as the business input for the entire process. It expresses the information the user wants to retrieve from the multimodal database. A query vector is the mathematical representation of the query request, mapping the natural language text to a fixed-dimensional, high-dimensional floating-point vector, used for similarity retrieval within a vector index.

[0017] For the input query request Generate query vectors ,in, For the query encoder, the same or compatible semantic encoding model is used as the stable evidence fragment embedding. The initial screening set is retrieved from the vector index. , This sets the maximum number of responses to be retrieved. Optionally, using the query vector as the retrieval key, an approximate nearest neighbor search algorithm can be employed to quickly filter the vector index for responses with the highest semantic similarity. A set of stable evidence fragments as the initial screening set .

[0018] S12. Based on the initial screening set, determine the minimum set of evidence that satisfies the constraints.

[0019] Constraints refer to a series of pre-defined criteria and optimization conditions that stable evidence fragments in the initial screening set must meet before being adopted as final evidence. The minimum evidence set refers to the set of stable evidence fragments that meet the constraints selected from the initial screening set.

[0020] S13. Based on the minimum set of evidence, generate the answer content and divide the answer content into multiple answer units.

[0021] In this embodiment, based on the minimum set of evidence Construct a context for prompting and drive the generative model to obtain the answer. , To generate a calling function for a large language model, in response to a query request. With minimal evidence set As input, the generator model outputs the answer. The generative model can be obtained through pre-training.

[0022] Regarding the answer content The structured segmentation yields a set of response units. , The set of answer units is segmented by the answer segmentation module based on syntactic structure, named entity distribution, and statement relations. Cut into This is achieved by dividing a response into two independent and verifiable semantic units, such as "penalty triggering condition unit" and "penalty cap unit". For the first One answer unit; To answer the number of units.

[0023] S14. Determine the support alignment score between each response unit and each stable evidence fragment in the minimum evidence set, and establish a support graph for each response unit based on each support alignment score.

[0024] The support alignment score is used to quantify the degree of matching between the response unit and the stable evidence fragment in four dimensions: semantic content, entity information, numerical information, and temporal information. A higher support alignment score indicates that the stable evidence fragment is more likely to be a genuine source of support for the response unit. The response unit support graph is a weighted bipartite graph with response units and stable evidence fragments as vertices. An edge is established between two vertices if their support alignment score is not lower than a preset support edge threshold, and the weight of the support edge is the support alignment score. This response unit support graph explicitly expresses the support relationship between the response unit and the stable evidence fragment, and provides clear operational boundaries and a topological computational basis for subsequent necessity verification.

[0025] S15. Based on the answer unit support graph, output the proof package for each answer unit.

[0026] In this embodiment, the proof package of the response unit is a machine-readable, auditable, and attributable structured evidence report. Each response unit generates a separate proof package, which fully records the response content of the response unit, the core evidence it relies on, the necessity score of each piece of evidence, the system version status at the time of generation, the tamper-proof fingerprint of the evidence, and the comprehensive credibility assessment. It is the final output of the entire evidence closed-loop verification process.

[0027] In the above embodiments, by establishing a vector index containing multiple stable evidence fragments, the multimodal raw data is finely granularized into searchable and traceable evidence units, solving the problems of coarse retrieval granularity and untraceable evidence sources in traditional schemes. Based on this, by acquiring query requests and generating query vectors, a preliminary screening set is semantically recalled from the vector index, achieving cross-modal semantic-driven evidence retrieval and overcoming the poor generalization ability of keyword matching. Furthermore, by solving for the minimum evidence set satisfying sufficiency and minimumity constraints based on the preliminary screening set, redundant noise is eliminated while ensuring the integrity of the response information, solving the problems of invalid content occupying the context window and broken evidence chains leading to model comprehension bias. Based on this, based on the minimum... The small evidence set generates answer content and segments it into multiple answer units, providing a fine-grained structural foundation for sentence-by-sentence attribution. This overcomes the deficiency of traditional solutions, which can only achieve document-level overall citation and lack sufficient attribution granularity. Furthermore, by determining the support alignment score between each answer unit and each stable evidence fragment and establishing an answer unit support graph, the solution outputs the proof package for each answer unit based on the answer unit support graph. This achieves structured auditable proof that includes necessary evidence attribution, necessity score, version context, tamper-proof signature, and credibility assessment. The entire technical solution forms a complete closed loop from multimodal evidence digitization, semantic retrieval and recall, evidence minimization and compression, answer generation and fine-grained segmentation, sentence-by-sentence attribution graph construction to structured auditable proof output.

[0028] In some embodiments, such as Figure 3 As shown, Figure 3 A flowchart for establishing a vector index comprising multiple stable evidence fragments in one embodiment includes: S31. Obtain the original object of the multimodal model and extract the set of atomic units corresponding to the original object.

[0029] Suppose a multimodal primitive object set , This represents the collection of raw objects read by the multimodal object access module from user-submitted files, databases, message queues, or object storage. Indicates the first An object can be a text file, image file, audio file, video file, or structured table file. This represents the total number of original objects, a positive integer, calculated by the access module. Each original object has a modal tag. , Represents the primitive object The modal tag is automatically determined by the multimodal object access module based on the file extension, Multipurpose Internet Mail Extensions (MIME) type, or modal field in the metadata, and can be one of text, image, audio, video, or table.

[0030] Extract the set of atomic units for each original object. , This indicates that the atomic unit extraction module extracts data from the original object. The extracted set of atomic units, the specific way to call it depends on... The corresponding modality type. For text, it calls sentence segmentation; for images, it calls layout detection and region proposal; for audio, it calls speech activity detection and segmentation; for video, it calls shot segmentation and keyframe extraction; and for tables, it calls row and column parsing. express The first in Each atomic unit is the smallest positioning processing unit in this invention; The number of atomic units is obtained from the number of segments output by the analytical algorithm described above.

[0031] S32. For each atomic unit in the set of atomic units, construct the standard anchor vector for each atomic unit.

[0032] Optionally, the standard anchor vectors for constructing each atomic unit include: For any atomic unit, obtain the atomic attributes of the atomic unit, where the atomic attributes include the modality type code corresponding to the atomic unit, the positioning position range corresponding to the atomic unit, the positioning quality score corresponding to the positioning position range, and the preprocessing confidence level corresponding to the atomic unit. Based on the atomic properties of atomic units, determine the anchor tuple of atomic units; Using the tuple expansion function, the fields in the anchor tuple are concatenated in a preset order to obtain the standard anchor vector.

[0033] In this embodiment, to unify the local positions of different modes, for each atomic unit Constructing a unified anchor tuple , This represents the modal type encoding of an atomic unit. For integers, and A one-to-one correspondence is used, for example, text is encoded as 1, image as 2, audio as 3, video as 4, and table as 5. Modal encoding is directly written by the atomic unit extraction module when generating atomic units, and is used for subsequent selection of the modal mapping matrix and zero-value filling determination for fields not belonging to the current modality. The location range includes at least one of the following: normalized bounding box of an image region, audio or video time interval, video frame interval, and table row and column range. Represents the normalized bounding box of the image region. and These are the coordinates of the top left and bottom right corners, respectively, and their values ​​are taken from... The output is directly from the image layout detection model. Indicates an audio or video time range. These are the start and end times, in seconds, output by the audio segmenter or video lens splitter. Indicates the video frame interval. These are the start and end frame numbers, respectively, output by the video keyframe extractor. This indicates the range of rows and columns in the table, obtained from the cell boundaries output by the table parser. Indicates a range of logical positions in the text. The start and end character offsets are output by the text segmentation tool. This represents the location quality score. It is directly obtained from the confidence score output by the corresponding parsing algorithm. For text, it is the confidence score for sentence segmentation; for images, it is the confidence score for page layout detection bounding boxes; for audio, it is the confidence score for speech activity detection; for video, it is the confidence score for shot boundary detection; and for tables, it is the confidence score for table structure parsing. This indicates the preprocessing confidence level. The confidence level of each preprocessing step performed by this atomic unit, such as OCR, ASR, denoising, layout parsing, table parsing, and keyframe extraction, is calculated by averaging or weighted averaging the output confidence levels. If a preprocessing step does not output a confidence level, the default confidence level for that step is used. Fields not belonging to the current atomic unit's modality are uniformly filled with zero values ​​to ensure a fixed tuple dimension.

[0034] A standard anchor vector is the result of mapping anchor tuples of multimodal atomic units to a unified-dimensional real-number vector space. It transforms the positional information of atomic units from different modalities such as text, images, audio, video, and tables into a unified, computable, and comparable mathematical representation. This involves mapping anchor tuples to standard anchor vectors. , This is a tuple expansion function used to concatenate the fields in the anchor tuple into a fixed-dimensional real number column vector in a preset order. Fields that do not belong to the current modality are filled with 0. Its function is to represent the position information of heterogeneous modalities as an intermediate vector with a uniform form and fixed dimensions. For each modality, there is an affine mapping matrix and a parameter matrix maintained by each modality. The number of rows is equal to the dimension of the unified anchor vector, and the number of columns is equal to the output dimension of the tuple expansion function. Its function is to linearly map the intermediate vector to the shared standard anchor space, so that the atomic units of different modalities are comparable in this space. This is the bias vector for the corresponding mode. and It can be predefined by domain experts or learned from training data with cross-modal alignment annotations.

[0035] S33. Construct an anchor point diagram based on the standard anchor point vector of each atomic unit.

[0036] In this embodiment, the anchor graph is a weighted undirected graph with atomic units as vertices and multi-dimensional proximity relationships between atomic units as edges.

[0037] Optionally, based on the standard anchor vectors of each atomic unit, the anchor graph can be constructed as follows: For any two atomic units, based on the standard anchor vectors corresponding to each atomic unit, calculate the semantic similarity, positional continuity, temporal continuity, and structural correlation between the two atomic units. Weighting is applied to semantic similarity, positional continuity, temporal continuity, and structural relevance to determine the edge weights corresponding to two atomic units; When the edge weight is greater than the edge weight threshold, an edge is established between two atomic units in the anchor graph.

[0038] Construct an anchor point diagram based on standard anchor point vectors. For any two atomic units and If at least one of semantic proximity, structural proximity, temporal proximity, or positional proximity is satisfied, then an edge is established between the corresponding vertices. and Boundary weights , for and The semantic similarity between them is determined by Corresponding semantic embedding vector and Corresponding semantic embedding vector The cosine similarity is obtained. for and The positional continuity between characters in the text is determined by the range of character positions. The overlap or adjacency of the image is obtained from the bounding box. The IoU or distance is obtained by dividing the table by row and column intervals. The degree of overlap is obtained. for and The temporal continuity between time intervals is determined by the time interval. or frame interval The interval is according to The conversion yields, where This represents the time interval between two atomic units. This represents the time normalization constant. for and The structural correlation between the two is obtained by matching the page level, table blocks, or video shot groups of the two according to preset rules. The weights are non-negative and satisfy the following conditions: It can be set a priori by domain experts or obtained through supervised learning on cross-modal aligned data.

[0039] S34. Based on the anchor point graph, perform fragment stabilization on each atomic unit to obtain multiple stable evidence fragments, generate chain evidence fingerprints for each stable evidence fragment, and write them into the vector index.

[0040] Optionally, based on the anchor graph, fragment stabilization is performed on each atomic unit to obtain multiple stable evidence fragments, including: For any two adjacent atomic units that are naturally adjacent, determine the semantic distance, structural jump distance, temporal continuity distance, and anchor vector distance between the two adjacent atomic units; Weighted distances are applied to semantic distance, structural jump distance, temporal continuity distance, and anchor vector distance to determine the boundary score between two adjacent atomic units; When the boundary score is greater than the boundary threshold, a candidate boundary is established between two adjacent atomic units. Based on the candidate boundary, the continuous atomic units are grouped to obtain candidate evidence fragments. For any candidate evidence fragment, determine the semantic cohesion within the candidate evidence fragment, the coverage of the candidate evidence fragment to entities and constraints in the database, the topological continuity of the candidate evidence fragment in the anchor graph, and the evidence chain breakage penalty value corresponding to the candidate evidence fragment. The stability of candidate evidence fragments is determined by weighting semantic cohesion, coverage, topological continuity, and evidence chain breakage penalty values. Based on the stability of each candidate evidence fragment, fragment stabilization is performed to obtain multiple stable evidence fragments.

[0041] In this embodiment, for two atomic units that are adjacent in natural order and For example, the system segments the contract text into multiple atomic units according to the PDF reading order. The first sentence of the text on page 2 of the PDF is... The second sentence of the main text on page 2 of the PDF reads, "The breaching party shall pay the penalty within 15 days of receiving the notice." This means that "the maximum penalty for breach of contract is 20% of the total contract price." and These are two atomic units that are adjacent in natural order. and The corresponding boundary score: .

[0042] in The weights are non-negative real numbers, satisfying Semantic distance ,in, They are respectively The corresponding semantic embedding vectors are obtained by the semantic encoder. This is the inner product of the two. Structural jump distance. , for and The change in heading level between different headings is obtained by subtracting the heading level field from the document parsing results. for and The amount of layout variation, such as from body text to list, or from paragraph to table, is obtained by subtracting the style tags output by the layout parsing model. for and The amount of change in structural type, such as from the chapter header to the chapter body, is obtained by subtracting the structural type labels. The weights are non-negative real numbers. Time continuity distance. ,in, for The end time of the time interval, for The start time of the time interval; This is the time normalization constant, in seconds. Anchor point vector distance. ,in, They are respectively The standard anchor vector.

[0043] when At that time, in the atomic unit and Establish candidate boundaries between them; where The boundary threshold is set to a value of 1. .

[0044] After calculating boundary scores for each pair of atomic unit sequences arranged in natural order, the boundary scores of each atomic unit pair are compared with preset boundary thresholds. If the boundary score between two adjacent atomic unit pairs is greater than the boundary threshold, a candidate boundary is established between the two adjacent atomic units. This candidate boundary indicates that the system determines there is a significant information gap, making it unsuitable to include the content on both sides in the same evidence block. After traversing all adjacent atomic unit pairs, all established candidate boundaries divide the entire atomic unit sequence into several intervals. Each interval consists of all continuous atomic units between any two adjacent candidate boundaries. The starting point of the interval is the beginning position of the sequence or the first atomic unit after the previous candidate boundary, and the ending point of the interval is the last atomic unit before the next candidate boundary or the end position of the sequence. In this way, multiple non-overlapping candidate evidence fragments are obtained. These candidate evidence fragments will be used as input for subsequent fragment stabilization processing.

[0045] Candidate evidence fragments , Stability: , The semantic cohesion within the candidate evidence fragment is obtained by the mean of the pairwise semantic similarity of all atomic units within the candidate evidence fragment. The coverage of entities and constraints by candidate evidence fragments is obtained by normalizing the number of entities and constraints identified by the named entity recognition and constraint extraction module in the candidate evidence fragments according to preset rules. Entities refer to named entities with specific semantics, such as person names, organization names, place names, contract numbers, product models, monetary values, dates, etc. Constraints refer to logical, conditional, or rule-based statements, such as restrictive phrases like "must not", "must", "exceed", "under certain conditions", "if...then...", etc. For candidate evidence fragments in the anchor point diagram The topological continuity is obtained by the ratio of the size of the connected subgraph formed by the vertices of the candidate evidence fragment on the anchor graph to the size of the candidate evidence fragment. This is a penalty for a broken chain of evidence, calculated according to preset rules based on the number of times a candidate piece of evidence spans a title, chapter, or scene change. The weights are non-negative real numbers.

[0046] Optionally, fragment stabilization includes fragment merging, boundary shifting, and shortest bridging; Based on the stability of each candidate evidence fragment, fragment stabilization is performed, resulting in multiple stable evidence fragments, including: When the stability of two adjacent candidate evidence segments is less than the first stability threshold, the two adjacent candidate evidence segments are merged to obtain a stable evidence segment. When the stability of the adjacent evidence fragments of a candidate evidence fragment is lower than the second stability threshold, the boundary of the candidate evidence fragment is shifted towards the direction of the adjacent evidence fragments until the stability of the updated adjacent evidence fragments is higher than or equal to the second stability threshold. When the shortest bridging cost between two adjacent candidate evidence fragments is less than the bridging cost threshold, the minimum bridging cost between the candidate evidence fragment and the adjacent evidence fragment on the anchor graph is calculated. When the minimum bridging cost is less than the bridging cost threshold, the shortest bridging is performed, and the atomic units traversed by the bridging path corresponding to the minimum bridging cost are incorporated into the candidate evidence fragment.

[0047] In this embodiment, adjacent evidence segments are candidate evidence segments that are geographically adjacent to candidate evidence segments. Adjacent candidate evidence segments are two candidate evidence segments that are geographically adjacent. ,in The first stability threshold is used for fragment merging. Fragment merging refers to merging two adjacent candidate evidence fragments into a longer fragment when both have low stability. For example, combining two incorrectly separated sentences into a single paragraph. When the stability of adjacent evidence fragments of a candidate evidence fragment is below the second stability threshold, boundary shifting is performed. Boundary shifting refers to shifting the boundary of a candidate evidence fragment by several atomic units towards the adjacent evidence fragment when the stability of an adjacent evidence fragment on one side of the candidate evidence fragment is low. For example, pulling the first sentence that has mistakenly fallen into the next paragraph back into the current paragraph. Shortest bridging refers to merging the bridge between two adjacent candidate evidence fragments on the anchor point diagram at the minimum bridging cost to maintain the continuity of the evidence chain. For example, bridging a small footnote next to a signature image with the main text of a contract clause to avoid a break in the evidence chain at abrupt changes in layout. Minimum bridging cost. , and Each of the adjacent candidate evidence fragments is selected as one atomic unit; For the corresponding vertices in the anchor point graph The edge weights on the graph have been calculated during the anchor graph construction phase. To prevent small positive numbers with a denominator of zero; The minimum bridging cost among all atomic unit pairs between the two is selected; the smaller the cost, the stronger the connection between the two can be. Bridge is executed at that time. This is the bridging cost threshold. From this, we obtain the set of stable evidence fragments. ,in To stabilize the number of evidence fragments, the calculation method for the shortest bridging cost between two adjacent candidate evidence fragments is similar to that for the minimum bridging cost, and will not be repeated here.

[0048] Optionally, fragment vectors of stable evidence fragments are also generated. This data, along with the chain evidence fingerprint and index slots, is written into the vector index for retrieval purposes. For each stable evidence fragment... Construct a chain of evidence fingerprints. Preprocess signatures. , To act on stable evidence fragments A set of preprocessing procedures, such as This is obtained from the preprocessing scheduler. The parameter set corresponding to the preprocessing process, such as OCR model version, font recognition threshold, and language prior, is collected uniformly by the preprocessing scheduler. This is a summary of the preprocessing results, which is a short hash summary of the preprocessed output of stable evidence fragments after serialization. This is a serialization function that follows preset rules.

[0049] For cryptographic hash functions, such as SHA-256. Content signature definition. .

[0050] This is a text summary of the candidate evidence fragments. If it is a text or table fragment, it is obtained directly from the fragment text. If it is an image fragment, it is obtained from the OCR output. If it is an audio or video fragment, it is obtained from the ASR output. This is a summary of the layout and table structure, obtained by serializing the structured objects output by the layout parsing model or the table parsing model. For audio and video keyframe or time slice summaries, null values ​​are used for image segments, sampled feature hashes are used for audio, and perceptual hashes of keyframe images are used for video; null values ​​are used to fill in fields where no corresponding modality field exists. Chain evidence fingerprints: , To stabilize evidence fragments Belonging to the original object The object identifier is assigned by the multimodal object access module when it is added to the database. An anchor segment identifier is a unique logical position code assigned to a stable evidence segment. This is the embedded model version, generated by the embedding service. Write at that time. This is the indexed version, written by the vector indexing system. To stabilize the index slot of the evidence fragment in the vector index, it is written back by the vector index. For the chain fingerprint of the previous stable evidence fragment in a preset order, The initial fingerprint (e.g., a string of all zeros) is retrieved at the specified time. This is a byte string concatenation operation.

[0051] This application addresses the issue of broken evidence chains caused by fragmented atomic units, such as cutting a complete clause in half, in traditional schemes by performing fragment stabilization on each atomic unit based on anchor graphs. It leverages the semantic proximity, structural correlation, temporal continuity, and positional continuity between atomic units in the anchor graph, and employs a dual-scale boundary detection and consensus repair mechanism to synthesize scattered atomic units into stable evidence fragments with precise boundaries and logical integrity. This provides high-quality semantic units for subsequent retrieval and attribution. Furthermore, it generates a chain-like evidence fingerprint for each stable evidence fragment, including object identifier, anchor segment identifier, preprocessing signature, content signature, embedded model version, index version, and index. By using chained hashing to lock the slot and the fingerprint of the previous segment, the problem of existing systems being unable to trace changes in the source of evidence when changes occur in the preprocessing process, embedding model, or index parameters is fundamentally solved. Changes in any link will lead to fingerprint mismatch, making version drift accurately detectable and located. Finally, the segment vector and the chained evidence fingerprint are written into the vector index together, so that the retrieval stage can not only recall relevant evidence based on semantic similarity, but also simultaneously obtain its complete verifiability status. This provides a data foundation with both semantic comparability and source credibility for subsequent sufficiency assessment, necessity verification, and proof package generation, ultimately realizing a complete closed loop from fragmented atomic units to structured credible evidence units.

[0052] In some embodiments, determining the minimum set of evidence that satisfies the constraints based on the initial screening set includes: Starting with an empty set as the initial evidence set, calculate the comprehensive score corresponding to each preliminary screening evidence fragment in the preliminary screening set. Based on the comprehensive score corresponding to each preliminary screening evidence fragment, sort the preliminary screening set to obtain an initialized sorted fragment queue. According to the order of the sorted fragment queue, select the initial screening evidence fragment that maximizes the increase in the sufficiency function value and add it to the current evidence set; Calculate the necessary pre-screening score for each currently screened evidence fragment in the current evidence set. The necessary pre-screening score reflects the degree of decrease in the coverage of each evidence evaluation indicator after deleting the currently screened evidence fragment from the current evidence set. Currently screened evidence fragments with necessary pre-screening scores lower than a preset necessary pre-screening threshold are marked as suspected redundant fragments. For a suspected redundant fragment, if the sufficiency function value of the current evidence set is greater than or equal to the sufficiency threshold after deleting the suspected redundant fragment, the suspected redundant fragment is permanently deleted; if the sufficiency function value of the current evidence set is less than the sufficiency threshold after deleting the suspected redundant fragment, the suspected redundant fragment is retained, and the current evidence set is updated. Within the updated current evidence set, select high-cost preliminary screening evidence fragments with costs higher than the first cost threshold. Obtain low-cost preliminary screening evidence fragments from the sorted fragment queue that were not selected into the updated current evidence set and whose costs are lower than the second cost threshold. Exchange the high-cost preliminary screening evidence fragments with the low-cost preliminary screening evidence fragments. If the sufficiency function value of the current evidence set after the exchange is not lower than the sufficiency threshold and the optimization objective function value is lower than before the exchange, then perform the exchange. Based on the current evidence set and the remaining preliminary screening evidence fragments in the preliminary screening set, update the sorted fragment queue, and continue to select the preliminary screening evidence fragments that maximize the increase in the sufficiency function value according to the order of the sorted fragment queue and add them to the current evidence set until the termination iteration condition is met. The current evidence set after the termination iteration condition is met is taken as the minimum evidence set.

[0053] In this embodiment, the initial screening evidence fragments are stable evidence fragments from the initial screening set. For each initial screening evidence fragment... ,calculate The overall score between the query request and the query result is as follows: .

[0054] in Cosine similarity; For initial screening of evidence fragments The coverage of query elements is determined by the matching ratio of entities, constraints, numerical values, and time elements within the fragment to the query elements. for The verifiability of a chain of evidence fingerprints is obtained by combining the fingerprint hash verification pass rate and the preprocessing confidence level, and its value is taken from... . for The connectivity of the current set of evidence to the anchor point graph. As a redundancy penalty, take and The maximum similarity of stable evidence fragments in the data; for In anchor point diagram Topological importance; to The weights are non-negative. This represents the current set of evidence and is an intermediate state variable that is continuously updated during the iteration process.

[0055] During the initialization phase, the current evidence set is initialized to an empty set. Subsequently, the current evidence set is updated once for each initially screened evidence fragment selected, and is dynamically updated with each iteration. After convergence, the value is taken as the minimum evidence set. . for The segment vector. Define the sufficiency function: , For key entity coverage, by Entities and query requests appearing in The ratio of the named entity recognition set is obtained. To determine the constraint coverage, by The ratio of the number of covered constraints to the number of constraints required by the query is used to obtain the result. This represents the coverage rate of numerical features. This refers to the coverage rate of time elements. To ensure the verifiable coverage of chain evidence fingerprints, by The ratio of the number of segments that pass fingerprint verification to the total number of segments is used to obtain the result. to The weights are non-negative.

[0056] Construct the objective function to minimize topological constraints: , For fragments The cost of entering the context can be obtained by combining the fragment length, the number of tokens, the parsing complexity, or the call cost. for The sum of the pairwise redundancy of internal segments reflects the degree of repetition in the set; for In anchor point diagram The severed edges and weights reflect the completeness of the chain of evidence. The weights are non-negative. The topological constraints are: , , The sufficiency threshold is set to a value of . It is used to determine whether the evidence set meets the overall coverage required for an answer. express Satisfying topological constraints, i.e., from Deleting any segment will lower the sufficiency threshold below the sufficiency threshold, thus ensuring... There are no redundant segments. The minimum evidence set is obtained when the objective function is minimized. Indicates from delete .

[0057] The iterative process is as follows: initialization In each round, select the current preliminary screening evidence fragment according to the following formula: in, is the topological reward coefficient, and is a non-negative real number; For initial screening of evidence fragments The cost, It is a small positive number. Until This stage ends when the time is right. For example, in a contract review scenario, this stage might first select the clause that directly states the maximum amount of liquidated damages.

[0058] Calculate the current evidence set The necessary pre-screening score for the current initial screening evidence fragments: .

[0059] in, From respectively Delete the current initial screening evidence fragment The decrease in the coverage of key entities, constraints, numerical elements, time elements, and chain evidence fingerprints before and after; to Non-negative weights; for those satisfying Prioritize deleting the relevant segments. This is a necessary pre-screening threshold. For example, a stage might remove paragraphs that only relate to company introductions or irrelevant background. Evidence evaluation metrics include key entities before and after the event, constraints, numerical elements, time elements, and chain evidence fingerprint coverage.

[0060] The cost of each stable evidence fragment is a quantitative indicator of the computational resources required for that fragment to be selected into the large model's generation context. It is calculated by weighting the fragment's token length, the type weight of its modality, the cache hit status of the preprocessed content, and the information density adjustment coefficient. For high-cost, high-initial-screening evidence fragments... Fragments of evidence indicating low initial screening cost When the following conditions are met: and At this point, fragment swapping is performed. For example, this stage might replace a long, lengthy background clause with a single cell that precisely covers the maximum amount. The minimum set of evidence is obtained after the iterative process converges. .in Indicates from Delete .

[0061] In the above embodiments, this application can obtain core evidence from the initial screening evidence fragments by solving the minimum evidence set from the initial screening set, which greatly reduces the context occupation, computational cost and response latency of large model reasoning; the sufficiency constraint ensures that the evidence can completely cover all information elements required by the query to improve the accuracy of the answer. In addition, the simplified minimum evidence set provides an engineering feasibility basis for the subsequent construction of the answer unit support graph and the necessity verification of edge deletion replay, so that the entire evidence verification closed loop can be implemented efficiently under limited computing power.

[0062] In some embodiments, determining the support alignment score between each response unit and each stable evidence fragment in the minimum evidence set, and constructing a support graph for each response unit based on the support alignment scores, includes: For a stable evidence fragment and a response unit in the minimum evidence set, calculate the semantic similarity value, entity similarity value, numerical consistency rate and temporal consistency rate between the stable evidence fragment and the response unit. Perform weighting on the semantic similarity value, entity similarity value, numerical consistency rate and temporal consistency rate to determine the support alignment score between the stable evidence fragment and the response unit. When the support alignment score between the stable evidence fragment and the response unit is greater than the support alignment score threshold, a support edge is established between the stable evidence fragment and the response unit until all support alignment scores are traversed, thus obtaining the response unit support graph.

[0063] For each answer unit With each stable piece of evidence in the minimum evidence set Calculate the support alignment score: .

[0064] Indicates the answer unit With stable evidence fragments The support alignment score, which takes the value of This is used to measure whether a piece of evidence is likely to support the answer unit. The larger, the more it means The more likely it is to become Supporting evidence. Indicates the semantic similarity value. For answering unit The semantic vectors are processed by the same semantic encoder as the fragment embedder. Obtained through text encoding. This represents the entity similarity value, obtained from the Jaccard similarity of the named entity sets of the two entities. The numerical consistency rate is obtained by the matching ratio of the numerical sets extracted from both sources. The time consistency rate is obtained by the matching ratio of the time / date sets extracted from both. Non-negative weights; This is the Sigmoid function.

[0065] when At that time, in the answer unit With stable evidence fragments Establish supporting edges between them. This forms the support threshold for the answer unit. ,in This represents the established set of support edges. Provide the unit support diagram. Used to explicitly express stable evidence supporting each response unit, it serves as the input for the subsequent two-stage edge-elimination replay—only for… The existence of supporting edge response units and stable evidence fragments enables the execution of edge deletion replay, giving necessity verification clear boundaries and objectives.

[0066] In the above embodiments, in the support graph of the answer unit, each answer unit and the stable evidence fragment supporting it are established with explicit support edges in the form of a weighted bipartite graph. This makes the implicit relationship of which sentence in the answer is supported by which piece of evidence computable and traversable. At the same time, by integrating four dimensions of semantic consistency, entity consistency rate, numerical consistency rate and temporal consistency rate to calculate the support alignment score, the risk of mismatches due to semantic similarity but factual inconsistency is greatly reduced. The structure of the support graph of the answer unit directly provides a precise operation object for subsequent edge deletion replay. Necessity verification is performed on the actual support edges in the graph, avoiding exhaustive search on all possible answer unit and evidence fragment pairs. The verification scope is compressed to the actual relevant relation subset, thereby greatly reducing the computational complexity.

[0067] In some embodiments, based on the answer unit support graph, the output proof package for each answer unit includes: For any answer unit in the answer unit support graph, obtain the target stable evidence fragment that has a support edge with the answer unit, and obtain the support evidence set of the answer unit. Based on the support alignment score between the answer unit and the target stable evidence fragment, the necessary pre-screening score corresponding to the target stable evidence fragment, and the bridging degree of the target stable evidence fragment in the answer unit support graph, calculate the upper limit prediction value of the necessity corresponding to the target stable evidence fragment. If the upper limit of necessity prediction is greater than or equal to the upper limit of necessity threshold, the target stable evidence fragment corresponding to the upper limit of necessity prediction is marked as a candidate necessary evidence and enters the edge deletion replay operation. If the upper limit of necessity prediction is less than the upper limit of necessity threshold, the target stable evidence fragment corresponding to the upper limit of necessity prediction is marked as a non-necessary candidate evidence and does not enter the edge deletion replay operation. During the edge deletion replay process, candidate necessary evidence is removed from the minimum evidence set to obtain an updated minimum evidence set. Based on the updated minimum evidence set, the response content corresponding to the target stable evidence fragment is regenerated, and the comparison response unit is obtained. Calculate the text consistency between the response unit corresponding to the target stable evidence fragment and the control response unit, and calculate the necessity score between the target stable evidence fragment and the response unit based on the text consistency. The necessity score is compared with the necessity determination threshold. If the necessity score is higher than or equal to the necessity determination threshold, the target stable evidence fragment is determined to be necessary evidence for the corresponding answer unit, and the target stable evidence fragment is retained in the supporting evidence set of the answer unit. If the necessity score is lower than the necessity determination threshold, the target stable evidence fragment is determined not to be necessary evidence for the corresponding answer unit, and the target stable evidence fragment is deleted from the supporting evidence set of the answer unit. After traversing all the target stable evidence fragments of the answer unit, the final updated supporting evidence set is taken as the minimum necessary evidence subset of the answer unit. Output a proof package based on the minimum necessary subset of evidence for each response unit.

[0068] In this embodiment, the answering unit The supporting evidence set comes directly from the answer unit support diagram. That is, the answer unit Supporting evidence set .

[0069] right Each target stable evidence fragment ,calculate The upper limit of necessity prediction: .

[0070] for and The support alignment score between them; for The necessary pre-screening score indicates deletion. Then query the decrease in coverage of each element. For bridging degree, by in Delete fragment The corresponding vertex answers the unit. The percentage decrease in the number of support paths was obtained. Non-negative weights; This is the upper bound threshold for necessity. If Then If marked as non-essential candidate evidence, full playback will not be performed; Then Marked as necessary candidate evidence, it enters the edge-replay operation, thus significantly reducing the number of full edge-replays. The edge-replay operation, under the premise of fixing the context of all versions of the system, temporarily deletes a stable evidence fragment and regenerates the response, observing whether the response unit disappears or changes, thereby accurately determining whether the stable evidence fragment is an indispensable necessary evidence for the response unit.

[0071] Fixed version context .

[0072] in These are the object version, preprocessed version, embedded model version, indexed version, rearranged version, and hint template version, respectively. (This refers to the necessary evidence for candidates.) Perform edge deletion replay: Necessary evidence for candidates Performing edge deletion replay refers to: within the minimum set of evidence Middle section Remove the evidence set after edge deletion ; In maintaining version context Under strictly unchanged conditions, call the playback function. For query request Regenerate answer "Edge deletion" refers to deleting objects that are part of the answer unit support diagram. The evidence fragment pointed to by the corresponding supporting edge; playback refers to re-executing the generation process under controlled conditions. To delete the segment The overall answer obtained after playback.

[0073] Compare and answer the unit From the replay answer Extracted from the original answer, using the method of segmentation. The exact same segmentation rules are used, and the answer units are compared with the original answer units based on their semantic topic, entity anchoring, and numerical anchoring. Alignment; if There is no longer any existence with The semantically corresponding answer unit will then This is considered an empty response unit. This indicates the deletion of evidence fragments. After passing Regenerated answer Zhongyu The corresponding text is the original answer unit. The control product.

[0074] and Necessity score The text consistency score is calculated based on the text consistency function.

[0075] The text consistency function is: .

[0076] and They can represent two answer units respectively, for example and Or two complete answers, for example and ; The semantic vectors of the texts corresponding to the two response units are encoded using the same encoding model as the fragment embedder; This represents the entity consistency rate (Jaccard similarity). For numerical consistency rate, For time consistency rate; Non-negative weights are defined as follows: ; This is the threshold for determining necessity. If... Then it is determined For answering unit Necessary evidence, and included in the minimum necessary evidence subset. .

[0077] In some embodiments, the output proof package includes, based on the minimum necessary subset of evidence for each response unit: For any response unit, obtain each necessary evidence fragment in the minimum necessary evidence subset corresponding to the response unit, average the support alignment scores between the response unit and each necessary evidence fragment, and calculate the average support alignment score; average the necessity scores corresponding to each necessary evidence fragment in the necessity score vector corresponding to the response unit, and calculate the average necessity score; average the verifiability of the chain evidence fingerprints of each necessary evidence fragment, and obtain the average verification score; obtain the local drift penalty value corresponding to the response unit; and determine the credibility of the response unit based on the average support alignment score, average necessity score, average verification score, and local drift penalty value. A proof signature is generated based on the answer unit identifier, the minimum necessary evidence subset corresponding to the answer unit, the necessity score vector corresponding to the answer unit, the version context, the chain evidence fingerprint of the necessary evidence corresponding to the answer unit, and the credibility. A proof package is generated based on the answer unit identifier, the minimum necessary evidence subset corresponding to the answer unit, the necessity score vector corresponding to the answer unit, the version context, the chain evidence fingerprint of the necessary evidence corresponding to the answer unit, the credibility, and the proof signature.

[0078] In this embodiment, for each answer unit Generate a structured proof package: .

[0079] in To answer the unit identifier; This is the minimum necessary subset of evidence.

[0080] This is the necessity score vector.

[0081] For version context; It is a chain-like set of evidence fingerprints; To assess the credibility of the answer unit; The proof signature for the answer unit. The necessary evidence fragments are stable evidence fragments from the minimum necessary evidence subset.

[0082] Definition of credibility of the response unit , For answering unit Support alignment score between all corresponding target stable evidence fragments The arithmetic mean; for Necessity score of each necessary piece of evidence The arithmetic mean; for Verifiability of chain evidence fingerprints for each necessary piece of evidence The arithmetic mean; For answering unit The local drift penalty term is determined by The chain fingerprint of the evidence relied upon is combined with the current system version context verification result. If all fingerprints match, the value is 0. If there are fingerprint mismatches, the value increases linearly according to the mismatch ratio. The weights are non-negative.

[0083] When the system version is Change to When defining the degree of difference in responses , In version context The original answer below; To change in version context The answer afterward; To ensure consistency between the two texts, the calculation is performed according to the formula in the previous section; The larger the value, the greater the difference between the two generated results. The overall drift is defined as: The drift of each component was estimated separately using single-factor substitution: These are object drift, preprocessing drift, embedding drift, index drift, rearrangement drift, and template drift, respectively; marked with an apostrophe. The version symbol indicates the new version number after the corresponding component has been replaced; the version symbol without an apostrophe indicates that the corresponding component retains its original version number. The final version drift component vector is: By comparison The magnitude of each component can pinpoint the dominant factor causing changes in the response, thereby supporting system-level attribution and regression localization.

[0084] One or more embodiments provided in this application can achieve at least one of the following beneficial effects: First, by unifying anchor tuples and anchor graphs, multimodal unified position representation and unified graph structure calculation are realized; second, excessive segmentation and evidence chain breakage are reduced by using a dual-scale fragment stabilization algorithm; third, the minimum sufficient evidence subset is obtained by topological constraint evidence compression, thereby increasing the effective evidence density and reducing contextual noise; fourth, evidence evaluation is upgraded from relevance to necessity determination by using answer unit support graphs and two-stage edge deletion replay; fifth, the traceability of results is improved by using chained evidence fingerprints and fixed version context; sixth, machine-readable and attributable results are provided for contract review, industrial inspection, and knowledge question answering by providing proof package output; and seventh, the number of complete replays is significantly reduced by necessity upper bound prediction, thereby improving project deployability.

[0085] To better understand the above embodiments, the following example is provided: Input object: Contract text file (PDF), image file of the signature page (PNG), Price Attachment Forms (XLSX). User query What are the triggering conditions and maximum amount of the penalty for breach of contract?

[0086] Step 1: Read in the above three files and record the object identifiers uniformly. Object version Source timestamp and source path. determination ,right determination ,right determination .

[0087] Step 2: [The sentence is incomplete and requires more context.] Extracting sentences This includes the clause number, heading level, and logical position; for Page layout detection to extract stamp area Signature area Related text regions ;right Table parsing extracts header blocks, amount cell blocks, and condition description cell blocks. Generate for each atomic unit Then follow the formula Obtain the standard anchor vector.

[0088] Step 3: Construct an anchor point diagram based on text logical continuity, table row and column adjacency, and image and text layout adjacency. , border rights Calculated using a formula, for example, the stamp area. The contract signing terms text directly below it An edge with a relatively large weight is formed due to its high positional continuity.

[0089] Step 4: Calculate the boundary scores one by one For example, the space between the last sentence of Clause 3.1 and the first sentence of Clause 3.2 This involves establishing candidate boundaries at this point; and calculating the stability of each candidate evidence fragment. For those with insufficient stability, merging, shifting, or bridging are performed to ultimately obtain a set of stable evidence fragments. ,For example For the "penalty triggering conditions clause" segment, This is a fragment of the "Amount Limit Cell".

[0090] Step 5: For each Generate preprocessed signature Content signature Chain evidence fingerprints and the fragment vector , Index slots Write it into the vector index as well.

[0091] Step 6: Query Request Generate query vectors Recall from vector index After sorting by comprehensive score, a three-stage solution is initiated. In the first stage, a greedy algorithm is used to select the input. , Has crossed Phase Two: Try deleting low-impact company profile paragraphs; Phase Three: Replace long, lengthy background clauses with table cells containing specific amounts. After convergence, the minimum subset of evidence is obtained. ,in This is the area for signatures and seals (to assist in verifying the validity of the terms).

[0092] Step 7: Call Received answer text It was then divided into two answer units. For "trigger condition response unit", This is the "Amount Limit Answer Unit".

[0093] Step 8: Calculation and Take the threshold Establish supporting edges above to obtain Typical results are: Depend on support; Depend on support.

[0094] Step 9: [The sentence is incomplete and requires more context to be translated accurately.] The first stage is to calculate and The former is higher than Enter precise playback, the latter is slightly lower but still enters; fixed version context. ,delete get Extract from ,calculate Due to a significant decrease in the numerical consistency rate, ,Will Determined as Necessary evidence; similarly, deletion. get The trigger condition description is missing, but the amount remains unchanged. The threshold was not exceeded, therefore it was not included. .final .

[0095] Step 10: Output ;right Similar output .

[0096] Step 11: When upgrading subsequent OCR models or embedded models, or Replace with the new version number and execute the replay to get ,and Together they form the drift component vector This allows us to pinpoint the primary source of the changes in responses.

[0097] Another example is as follows: Input objects include equipment inspection videos, on-site audio explanations, and maintenance record sheets. Keyframes and shot segments are extracted from the video (video modal atomic units); time slices and corresponding transcribed text are extracted from the audio (audio modal atomic units); maintenance record sheets are extracted from the maintenance number, component name, time field, and handling measures cell blocks (table modal atomic units). For video keyframes, Effective; for audio time slices, Valid; for table cells, Valid. Obtained via modal mapping. Construct anchor point graph Then, after dual-scale segment stabilization, the following was obtained: For the query "time of anomaly, abnormal component, and corresponding maintenance record", a set of candidate evidence is retrieved from the vector index, categorized by... Sort and by Find the minimum sufficient evidence subset After generating the response, it is divided into three response units: abnormal time, abnormal component, and maintenance record. For the maintenance record response unit, the upper limit prediction value of the necessity of each supporting piece of evidence is calculated. Maintenance table segments with high bridging and support alignment scores are entered into precise playback; if deleting the segment results in significant inconsistencies in the maintenance number, component name, or handling measures, ... , included If only a background audio segment is deleted and the answer remains unchanged, it will not be included. The system generates and outputs a proof package for each answer unit. When the keyframe extraction algorithm, audio transcription model, or table parsing rules of the inspection video change, it is obtained through single-factor replacement playback. and drift vector .

[0098] In another aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the anchor graph-based multimodal evidence question-answering method of any embodiment of this application.

[0099] Among them, in the computer program product, the optional implementation form of the program module architecture of the computer program that implements each step of the anchor graph-based multimodal evidence question answering method can be an anchor graph-based multimodal evidence question answering device.

[0100] Please see Figure 4 One embodiment of this application provides a multimodal evidence question-answering device based on an anchor graph, comprising: a building module 41, used to build a vector index including multiple stable evidence fragments; a preliminary screening module 42, used to obtain a query request, generate a query vector based on the query request, and a preliminary screening set including multiple preliminary screening evidence fragments that match the query vector from the vector index; a determination module 43, used to determine a minimum evidence set that satisfies constraints based on the preliminary screening set; a generation module 44, used to generate answer content based on the minimum evidence set and divide the answer content into multiple answer units; the building module 41 is further used to determine the support alignment score between each answer unit and each stable evidence fragment in the minimum evidence set, and build an answer unit support graph based on each support alignment score; and an output module 45, used to output the proof package of each answer unit based on the answer unit support graph.

[0101] It will be understood by those skilled in the art that Figure 4 The structure of the anchor graph-based multimodal evidence question-answering device does not constitute a limitation on anchor graph-based multimodal evidence question-answering devices. Each module can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the controller in the computer device, or stored in software in the memory of the computer device, so that the controller can invoke and execute the operations corresponding to each module. In other embodiments, the anchor graph-based multimodal evidence question-answering device may include more or fewer modules than illustrated.

[0102] Please see Figure 5 In another aspect of this application, a computing device 200 is also provided, including a memory 3011 and a processor 3012. The memory 3011 stores a computer program, and when the computer program is executed by the processor, the processor 3012 performs the steps of the multimodal evidence question-answering method based on anchor graphs provided in any of the above embodiments of this application. The computing device 200 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., a pair of smart glasses or a smartwatch) or similar devices, and robotic devices.

[0103] The processor 3012 is the control center, connecting various data points throughout the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 3011, and calls data stored in the memory 3011 to perform various functions and process data. Optionally, the processor 3012 may include one or more processing cores; preferably, the processor 3012 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 3012.

[0104] The memory 3011 can be used to store software programs and modules. The processor 3012 executes various functional applications and data processing by running the software programs and modules stored in the memory 3011. The memory 3011 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 3011 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 3011 may also include a controller to provide the processor 3012 with access to the memory 3011.

[0105] In another aspect, this application also provides a storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the multimodal evidence question-answering method based on anchor graphs provided in any of the above embodiments of this application.

[0106] Those skilled in the art will understand that all or all data flows in the methods provided in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the flows of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile media. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.

[0107] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal evidence question-answering method based on anchor graphs, characterized in that, include: Establishing a vector index comprising multiple stable evidence fragments includes obtaining the original object of the multimodality and extracting the set of atomic units corresponding to the original object; constructing a standard anchor vector for each atomic unit in the set of atomic units; constructing an anchor graph based on the standard anchor vectors of each atomic unit; and performing fragment stabilization on each atomic unit based on the anchor graph to obtain multiple stable evidence fragments, and generating a chain evidence fingerprint for each stable evidence fragment, and writing it into the vector index. The process of performing fragment stabilization on each atomic unit based on the anchor graph to obtain multiple stable evidence fragments includes: for any two adjacent atomic units that are naturally adjacent, determining the semantic distance, structural jump distance, temporal continuity distance, and anchor vector distance between the two adjacent atomic units; performing a weighted average on the semantic distance, structural jump distance, temporal continuity distance, and anchor vector distance to determine the boundary score between the two adjacent atomic units; when the boundary score is greater than a boundary threshold, establishing a candidate boundary between the two adjacent atomic units; and based on the candidate boundary, performing... Continuous atomic units are grouped to obtain candidate evidence fragments; for any candidate evidence fragment, the semantic cohesion within the candidate evidence fragment, the coverage of the candidate evidence fragment to entities and constraints in the database, the topological continuity of the candidate evidence fragment in the anchor graph, and the evidence chain breakage penalty value corresponding to the candidate evidence fragment are determined; the semantic cohesion, the coverage, the topological continuity, and the evidence chain breakage penalty value are weighted to determine the stability of the candidate evidence fragment; based on the stability of each candidate evidence fragment, fragment stabilization is performed to obtain multiple stable evidence fragments; Obtain a query request, generate a query vector based on the query request, and filter from the vector index to obtain a preliminary set of multiple preliminary evidence fragments that match the query vector; Based on the initial screening set, determine the minimum set of evidence that satisfies the constraints; Based on the minimum set of evidence, generate the answer content and divide the answer content into multiple answer units; Determine the support alignment score between each of the aforementioned response units and each stable evidence fragment in the minimum evidence set, and establish a support graph for the response units based on each of the aforementioned support alignment scores; Based on the answer unit support graph, the proof package for each answer unit is output.

2. The multimodal evidence question-answering method based on anchor graphs as described in claim 1, characterized in that, The standard anchor vector for constructing each atomic unit includes: For any of the atomic units, the atomic attributes of the atomic units are obtained, wherein the atomic attributes include the modality type code corresponding to the atomic unit, the positioning position range corresponding to the atomic unit, the positioning quality score corresponding to the positioning position range, and the preprocessing confidence level corresponding to the atomic unit. Based on the atomic properties of the atomic unit, determine the anchor tuple of the atomic unit; Using the tuple expansion function, the fields in the anchor tuple are concatenated in a preset order to obtain the standard anchor vector.

3. The multimodal evidence question-answering method based on anchor graphs as described in claim 1, characterized in that, The construction of the anchor graph based on the standard anchor vectors of each atomic unit includes: For any two atomic units, based on the standard anchor vectors corresponding to each of the two atomic units, the semantic similarity, positional continuity, temporal continuity, and structural correlation between the two atomic units are calculated. The semantic similarity, positional continuity, temporal continuity, and structural correlation are weighted to determine the edge weights corresponding to the two atomic units. When the edge weight is greater than the edge weight threshold, an edge is established between the two atomic units in the anchor point graph.

4. The multimodal evidence question-answering method based on anchor graphs as described in claim 1, characterized in that, The fragment stabilization includes fragment merging, boundary shifting, and shortest bridging; The step of stabilizing the evidence fragments based on their stability to obtain multiple stable evidence fragments includes: When the stability of two adjacent candidate evidence segments is less than the first stability threshold, the two adjacent candidate evidence segments are merged to obtain the stable evidence segment. When the stability of the adjacent evidence segments of the candidate evidence segment is lower than the second stability threshold, the boundary of the candidate evidence segment is shifted towards the direction of the adjacent evidence segment until the stability of the updated adjacent evidence segment is higher than or equal to the second stability threshold, thus obtaining the stable evidence segment. When the shortest bridging cost between two adjacent candidate evidence fragments is less than the bridging cost threshold, the minimum bridging cost between the candidate evidence fragment and the adjacent evidence fragment on the anchor graph is calculated. When the minimum bridging cost is less than the bridging cost threshold, the shortest bridging is performed, and the atomic units traversed by the bridging path corresponding to the minimum bridging cost are incorporated into the candidate evidence fragment to obtain the stable evidence fragment.

5. The multimodal evidence question-answering method based on anchor graphs as described in claim 1, characterized in that, The determination of the minimum set of evidence satisfying the constraints based on the initial screening set includes: Starting with an empty set as the initial evidence set, calculate the comprehensive score corresponding to each preliminary screening evidence fragment in the preliminary screening set. Based on the comprehensive score corresponding to each preliminary screening evidence fragment, sort the preliminary screening set to obtain an initialized sorted fragment queue. According to the order of the sorted fragment queue, select the initial screening evidence fragment that maximizes the improvement of the sufficiency function value and add it to the current evidence set; Calculate the necessary pre-screening score for each currently screened evidence fragment in the current evidence set. The necessary pre-screening score reflects the degree of decrease in the coverage of each evidence evaluation indicator after deleting the currently screened evidence fragment from the current evidence set. Mark the currently screened evidence fragments whose necessary pre-screening scores are lower than a preset necessary pre-screening threshold as suspected redundant fragments. For a suspected redundant fragment, if the sufficiency function value of the current evidence set is greater than or equal to the sufficiency threshold after deleting the suspected redundant fragment, then the suspected redundant fragment is permanently deleted; if the sufficiency function value of the current evidence set is less than the sufficiency threshold after deleting the suspected redundant fragment, the suspected redundant fragment is retained, and the current evidence set is updated. Within the updated current evidence set, select high-cost preliminary screening evidence fragments with costs higher than the first cost threshold. Obtain low-cost preliminary screening evidence fragments from the sorted fragment queue that were not selected into the updated current evidence set and whose costs are lower than the second cost threshold. Exchange the high-cost preliminary screening evidence fragments with the low-cost preliminary screening evidence fragments. If the sufficiency function value of the current evidence set after the exchange is not lower than the sufficiency threshold and the optimization objective function value is lower than before the exchange, then perform the exchange. Based on the current evidence set and the remaining preliminary screening evidence fragments in the preliminary screening set, update the sorted fragment queue, and continue to select the preliminary screening evidence fragments that maximize the increase in the sufficiency function value according to the order of the sorted fragment queue and add them to the current evidence set until the termination iteration condition is met. The current evidence set after the termination iteration condition is met is taken as the minimum evidence set.

6. The multimodal evidence question-answering method based on anchor graphs as described in claim 1, characterized in that, The step of determining the support alignment score between each of the response units and each stable evidence fragment in the minimum evidence set, and establishing a support graph for the response units based on each of the support alignment scores, includes: For a stable evidence fragment and a response unit in the minimum evidence set, calculate the semantic similarity value, entity similarity value, numerical consistency rate and temporal consistency rate between the stable evidence fragment and the response unit. Perform weighting on the semantic similarity value, entity similarity value, numerical consistency rate and temporal consistency rate to determine the support alignment score between the stable evidence fragment and the response unit. When the support alignment score between the stable evidence fragment and the response unit is greater than the support alignment score threshold, a support edge is established between the stable evidence fragment and the response unit until all support alignment scores are traversed to obtain the support graph of the response unit.

7. The multimodal evidence question-answering method based on anchor graphs as described in claim 5, characterized in that, The proof package for each answer unit, based on the answer unit support graph, includes: For any answer unit in the answer unit support graph, obtain the target stable evidence fragment that has a supporting edge with the answer unit to obtain the support evidence set of the answer unit. Based on the support alignment score between the answer unit and the target stable evidence fragment, the necessary pre-screening score corresponding to the target stable evidence fragment, and the bridging degree of the target stable evidence fragment in the answer unit support graph, calculate the necessary upper limit prediction value corresponding to the target stable evidence fragment. If the predicted value of the upper limit of necessity is greater than or equal to the upper limit threshold of necessity, the target stable evidence fragment corresponding to the predicted value of the upper limit of necessity is marked as a candidate necessary evidence and the edge deletion replay operation is entered. If the predicted value of the upper limit of necessity is less than the upper limit threshold of necessity, the target stable evidence fragment corresponding to the predicted value of the upper limit of necessity is marked as a non-necessary candidate evidence and the edge deletion replay operation is not entered. During the edge-deletion replay process, the candidate necessary evidence is removed from the minimum evidence set to obtain an updated minimum evidence set. Based on the updated minimum evidence set, the response content corresponding to the target stable evidence fragment is regenerated, and a comparison response unit is obtained. Calculate the text consistency between the response unit corresponding to the target stable evidence fragment and the control response unit, and calculate the necessity score between the target stable evidence fragment and the response unit based on the text consistency. The necessity score is compared with the necessity determination threshold. If the necessity score is higher than or equal to the necessity determination threshold, the target stable evidence fragment is determined to be necessary evidence for the corresponding answer unit, and the target stable evidence fragment is retained as a necessary evidence fragment in the supporting evidence set of the answer unit. If the necessity score is lower than the necessity determination threshold, the target stable evidence fragment is determined not to be necessary evidence for the corresponding answer unit, and the target stable evidence fragment is deleted from the supporting evidence set of the answer unit. After traversing all the target stable evidence fragments of the answer unit, the final updated supporting evidence set is taken as the minimum necessary evidence subset of the answer unit. The proof package is output based on the minimum necessary subset of evidence for each of the aforementioned response units.

8. The multimodal evidence question-answering method based on anchor graphs as described in claim 7, characterized in that, The output of the proof package, based on the minimum necessary subset of evidence for each of the aforementioned response units, includes: For any of the aforementioned response units, obtain each necessary evidence fragment in the minimum necessary evidence subset corresponding to the response unit, average the support alignment scores between the response unit and each of the necessary evidence fragments, and calculate the average support alignment score; average the necessity scores corresponding to each necessary evidence fragment in the necessity score vector corresponding to the response unit, and calculate the average necessity score; average the verifiability of the chain evidence fingerprints of each necessary evidence fragment, and obtain the average verification score; obtain the local drift penalty value corresponding to the response unit; and determine the credibility of the response unit based on the average support alignment score, the average necessity score, the average verification score, and the local drift penalty value. Based on the answer unit identifier, the minimum necessary evidence subset corresponding to the answer unit, the necessity score vector corresponding to the answer unit, the version context, the chain evidence fingerprint of the necessary evidence corresponding to the answer unit, and the credibility, a proof signature is generated. The proof package is generated based on the answer unit identifier, the minimum necessary evidence subset corresponding to the answer unit, the necessity score vector corresponding to the answer unit, the version context, the chain evidence fingerprint of the necessary evidence corresponding to the answer unit, the credibility, and the proof signature.

Images