Multimodal question answering method based on evidence faithful mapping

Abstract

The application discloses a multi-modal question answering method based on evidence fidelity mapping, comprising: obtaining a query request, analyzing the query request to obtain a query facet set; obtaining a cross-modal global evidence graph, calculating the support strength between each query facet and each evidence unit, based on each support strength, clipping a cross-modal evidence subgraph from the cross-modal global evidence graph, and generating a super-fusion evidence arrangement matrix; for the invisible evidence units without access rights, performing structure fidelity evidence mapping to generate the corresponding shadow evidence units of the invisible evidence units; based on the shadow evidence units, locally modifying the cross-modal evidence subgraph to obtain a current evidence graph; according to the super-fusion evidence arrangement matrix and the current evidence graph, determining the support evidence set of each candidate answer; based on the support evidence set of each candidate answer, determining the decision score of each candidate answer; and based on the decision score of each candidate answer, determining the target answer of the query request.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a multimodal question-answering method based on evidence-fidelity mapping. Background Technology

[0002] As the application of multimodal big data models and AI infrastructure continues to expand in government affairs, finance, manufacturing, healthcare, energy, contract management, and enterprise knowledge services, users increasingly require systems that can simultaneously process text, images, tables, audio, video, logs, and structured records, and output evidence-based, traceable, and auditable question-and-answer results.

[0003] Existing multimodal question-answering systems typically employ a processing path of "retrieval-assembly-generation" or "knowledge graph-evidence graph-answer selection." While these methods can improve answer accuracy in general public data scenarios, they exhibit the following shortcomings in applications involving multi-level access control, cross-system forensics, and audit trails: When an evidence item does not meet the current user's access permissions, existing systems usually directly delete, mask, or only indicate "restricted results exist," leading to a break in the answer support chain. Users with lower permissions cannot distinguish between "evidence does not exist" and "evidence exists but is restricted." Restricted evidence, after being filtered, typically does not participate in the answer scoring process, making answer selection highly sensitive to permission switching and exhibiting poor stability. Summary of the Invention

[0004] To address the existing technical problems, this application provides a multimodal question-answering method based on evidence fidelity mapping. This method ensures that restricted evidence can still participate in answer scoring, evidence chain continuity, and role consistency interpretation output through shadow form. It not only improves the stability of answers under role switching conditions but also provides an interpretable basis for key evidence identification and authority impact quantification for audit trails.

[0005] Firstly, a multimodal question-answering method based on evidence fidelity mapping is provided, comprising: acquiring a user-initiated query request; parsing the query request from different dimensions to obtain a set of query facets including multiple query facets; acquiring a pre-constructed cross-modal global evidence graph, wherein evidence units are nodes and two evidence units are edges in the cross-modal global evidence graph; calculating the support strength between each query facet and each evidence unit in the cross-modal global evidence graph; based on each support strength, cropping a cross-modal evidence subgraph from the cross-modal global evidence graph; and arranging the support strength between each query facet and each evidence unit in the cross-modal evidence subgraph in order to generate... A hyperconverged evidence orchestration matrix is ​​used; based on the user's permissions, access permissions for each evidence unit are determined; for invisible evidence units without access permissions, structure-fidelity evidence mapping is performed to generate shadow evidence units corresponding to the invisible evidence units; based on the shadow evidence units, the cross-modal evidence subgraph is locally modified to obtain the current evidence graph; based on the hyperconverged evidence orchestration matrix and the current evidence graph, a candidate answer set including multiple candidate answers is generated, and the supporting evidence set for each candidate answer is determined; based on the supporting evidence set for each candidate answer, the decision score for each candidate answer is determined; based on the decision score for each candidate answer, the target answer for the query request is determined.

[0006] This application first performs multi-dimensional facet analysis on user queries, transforming natural language queries into structured representations of different dimensions, thereby providing a refined matching benchmark for subsequent evidence matching. Simultaneously, the system pre-constructs a cross-modal global evidence graph with evidence units as nodes and multimodal associations as edges, decoupling this cross-modal global evidence graph from the query as a foundational asset for offline persistence, allowing any query to reuse the same global graph resource. Based on this, the system calculates the support strength between each query facet and each evidence unit in the cross-modal global evidence graph. It then first trims a cross-modal evidence subgraph relevant to the current query from the cross-modal global evidence graph based on the support strength, and then generates a hyper-fusion evidence orchestration matrix on this subgraph. This compresses the online reasoning scope from the full evidence graph to the query-related subgraph, significantly reducing online computational overhead while preserving the topological structure between evidence. Subsequently, the system performs structure-fidelity evidence mapping on invisible evidence in the cross-modal evidence subgraph according to user permissions to generate shadow evidence units. This maintains the local topological continuity of the evidence chain without disclosing sensitive content. The system also locally modifies the cross-modal evidence subgraph to obtain the current evidence graph. Finally, the system jointly generates candidate answers and their supporting evidence sets based on the hyper-fusion evidence orchestration matrix and the current evidence graph. Based on the supporting evidence set of each candidate answer, the system determines the decision score for each candidate answer and outputs the target answer based on the decision score. This ensures that while low-privilege users cannot access sensitive original text, restricted evidence can still participate in answer scoring, evidence chain continuity, and role-consistent interpretation output through shadow forms. This improves the stability of the answer under role switching conditions and provides an interpretable basis for key evidence identification and quantification of the impact of permissions for audit trails. Attached Figure Description

[0007] Figure 1 This is an application environment diagram of a multimodal question answering method based on evidence fidelity mapping in one embodiment; Figure 2 This is a flowchart of a multimodal question-answering method based on evidence fidelity mapping in one embodiment; Figure 3 This is a schematic diagram illustrating the construction of a cross-modal global evidence graph in one embodiment; Figure 4 This is a comparison diagram before and after performing neighborhood compensation in one embodiment; Figure 5 This is a schematic diagram of a multimodal question-answering device based on evidence fidelity mapping in one embodiment; Figure 6 This is a schematic diagram of a computing device in one embodiment. Detailed Implementation

[0008] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0009] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0010] In the following description, the expression “some embodiments” refers to a subset of all possible embodiments. However, it should be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0011] See Figure 1 In one embodiment, the application environment diagram of the multimodal question-answering method based on evidence fidelity mapping includes multiple terminal devices 100 and a computing device 200. The terminal devices 100 can communicate with the computing device 200. The multimodal question-answering method based on evidence fidelity mapping provided in this embodiment can be applied to the computing device 200, which communicates with a database. The computing device 200 obtains query requests from the terminal devices 100 and outputs the target answer corresponding to the query request.

[0012] Existing multimodal question-answering systems typically employ a processing path of "retrieval-assembly-generation" or "knowledge graph-evidence graph-answer selection." While these methods can improve answer accuracy in general public data scenarios, they still have the following shortcomings in application scenarios involving multi-level access control, cross-system forensics, and audit trails: When an evidence item does not meet the current user's access permissions, the existing system usually deletes, obscures, or only prompts "restricted results exist", which leads to the interruption of the answer support chain. Users with low permissions cannot distinguish between "evidence does not exist" and "evidence exists but is restricted". The explanatory texts, evidence lists, and reasoning paths presented under different roles vary greatly, making it difficult to form a stable and consistent interpretation for each role, which is also not conducive to post-audit. Restricted evidence, once filtered, typically no longer participates in the answer scoring process, making answer selection highly sensitive to changes in permissions and less stable. Existing explanations are mostly retrospective additions, making it difficult to answer audit questions such as "deleting which piece of evidence would change the conclusion" or "whether the conclusion changes after switching to a higher-privilege role." Text segments, image regions, table units, video clips, and log events differ significantly in structural granularity and temporal constraints, making it difficult for existing methods to maintain the topological stability of cross-modal evidence networks under permission constraints.

[0013] Therefore, this application provides a multimodal question-answering method based on evidence fidelity mapping, which enables the system to avoid simply deleting evidence when faced with restricted access. Instead, it maintains the local topology, supporting paths, and interpretation consistency of the evidence network through structure-fidelity evidence mapping, and continues to use the mapped results for answer calculation and audit output.

[0014] Please see Figure 2 This is a flowchart of a multimodal question-answering method based on evidence fidelity mapping provided in an embodiment of this application. The multimodal question-answering method based on evidence fidelity mapping is applied in a computing device and includes the following steps: S10. Obtain the query request initiated by the user, parse the query request from different dimensions, and obtain a set of query facets including multiple query facets.

[0015] A query request is a natural language question or structured query instruction submitted by a user to the system, used to trigger the question-and-answer process. A query request includes the user's query text. User role identifier and session context The user query text is the question posed by the user in natural language. The user role identifier is the permission role of the user who submitted the query within the system. The session context is the preceding information in the current query's dialogue history.

[0016] Query faceting refers to the structured representation obtained by explicitly deconstructing user query text from various dimensions such as entity, modality, time, and task. The core purpose of faceted representation is to transform free-form natural language queries into a set of computable and matchable discrete features, so that subsequent evidence retrieval, matching, arrangement, and scoring all have clear dimensional references, reducing the semantic coverage and dimensional coupling problems caused by single vectorization. Query faceting is obtained by performing entity recognition, modal intent recognition, time constraint extraction, and task type recognition on user queries. Entity recognition refers to identifying named entities from the query text, such as names of people, organizations, product numbers, supplier names, contract numbers, etc., which can be achieved using slot extraction based on large models or dictionary-rule-based matching. Modal intent recognition refers to determining the target modality or combination of modalities indicated by the query. For example, "image display" indicates an image modality, "report data" indicates a table modality, and "in video" indicates a video modality, which can be achieved using text classification models or rule matching based on keyword-modality mapping tables.

[0017] Query faceted set in, Indicates faceting of a solid. Represents modal surface, Representing time-series facets, This represents the task facets. To improve the computability of the facet representation, for each facet... Construct normalized facet vectors, normalized facet vectors in, For faceted coding function, and For trainable parameters, This is the normalization function. Represents an entity facet, used to carry the set of entities extracted from the query. Its value is a tuple set of entity identifiers and their types.

[0018] Example is .

[0019] This represents a modal facet, used to carry the target modal label set. Its value is a subset of the modal labels. An example is shown below. . This represents a temporal facet, used to carry temporal constraints, with values ​​taking the time interval. Time point or time decay coefficient. This represents a task facet, used to carry task type labels. Its values ​​are derived from a predefined set of task categories, for example... .

[0020] To improve the computability of faceted representation, for each query facet... Construct normalized facet vectors, normalized facet vectors ,in, For faceted coding function, and For trainable parameters, This is the normalization function.

[0021] S11. Obtain the pre-constructed cross-modal global evidence graph, where evidence units are nodes and two evidence units are edges.

[0022] In this embodiment, as Figure 3 The diagram shown is a schematic of constructing a cross-modal global evidence graph in one embodiment. Multimodal evidence content is represented as evidence unit structure. Based on multiple dimensions such as semantic similarity, structural similarity, temporal distance, and continuity between two evidence units, the correlation between the two evidence units is evaluated, and a cross-modal global evidence graph is established.

[0023] Evidence content is extracted from at least one data source, and a cross-modal global evidence graph is built from this content. Data sources include document libraries, image libraries, table libraries, video libraries, audio libraries, log libraries, business databases, or knowledge graphs. For example, data sources can be paragraphs, sentence groups, image-based region divisions or target detection boxes, cells, video clips with time windows, semantic segments, event windows, etc. Content from different modalities is segmented into evidence units of uniform granularity. Evidence Unit in, For modal types, For timestamps or time ranges For semantic vectors, For structure vectors, For permission tags. Evidence unit. It can also include , This serves as an identifier for the evidence unit.

[0024] The evidence units for different modalities are obtained by segmenting them as follows: text modality is segmented by paragraph, sentence group or semantic block; image modality is segmented by detection box, page area or image-text alignment area; table modality is segmented by table header, row, column and cell relationship; video modality is segmented by shot segment, time window or keyframe group; audio modality is segmented by semantic segment or speech turn; log modality is segmented by event window and associated transaction identifier.

[0025] Constructing a cross-modal global evidence graph using evidence units as nodes , where the set of nodes , Let be the set of edges. Let be the set of edge weights. For any two evidence units... and Its edge weights are: .

[0026] in Indicates evidence unit and The overall correlation strength between them; They are respectively The semantic vector is output by the multimodal encoder. They are respectively The structure vector contains components such as source location, hierarchy number, page area, shot range, table coordinates, or log field relationships. They are respectively The timestamp or time range. This is a time decay scale parameter, with a positive real number as its unit, consistent with the time distance. The weights are non-negative and satisfy the following conditions: ; The cosine similarity function; For structural similarity functions; It is a time-distance function; This is a similarity function that references continuity.

[0027] semantic similarity Structural similarity ,in, Represented by structure vector The exported set of structure keys, such as a set of string identifiers for document level paths, shot group numbers, table (row, column) coordinates, etc.

[0028] Time distance: .

[0029] They represent The start and end times of the time interval. They represent The start and end times of the time interval. Continuity similarity. ,in, express Source identification, express A collection of external objects that are explicitly referenced or jointly referenced. , To prevent dividing by a small amount. , These represent the weighting coefficients.

[0030] when At the node and Edges are established between them to reduce noisy connections. This represents the edge weight threshold.

[0031] S12. Calculate the support strength between each query facet and each evidence unit in the cross-modal global evidence graph. Based on each support strength, cut out the cross-modal evidence subgraph from the cross-modal global evidence graph. Arrange the support strength between each query facet and each evidence unit in the cross-modal evidence subgraph in order to generate a hyper-fusion evidence orchestration matrix.

[0032] In this embodiment, a hyper-fusion evidence orchestration matrix is ​​constructed between query facets and evidence units. , Matrix elements Indicates query faceting For evidence unit Support strength. Indicates faceting of a given query Under the constraints, the evidence unit The confidence level of an evidence unit that can be used as evidence for that facet. The larger the value, the better the evidence unit meets the entity, modality, time, or task requirements described by that facet. Support strength is the core quantity of the hyperconverged evidence orchestration matrix. This indicates the total number of facets being queried. This represents the total number of evidence units in the cross-modal evidence subgraph.

[0033] Upon receiving a query request, the support strength between the query facet and each evidence unit in the cross-modal global evidence graph is calculated using the following formula from the cross-modal global evidence graph. The middle cropping yields a cross-modal evidence graph relevant to the query request. : in This is the query relevance threshold. express The set of vertices. express The set of edges. express The set of edge weights. Cross-modal global evidence graph. It is a stable asset shared across the entire database, decoupled from queries, and reusable for any query; in the online phase, it only needs to be used in the trimmed version. Performing permission checks, mappings, and answer calculations on the platform can significantly reduce online response time.

[0034] S13. Based on the user's user permissions, determine the access permissions for each evidence unit. For invisible evidence units without access permissions, perform structure-fidelity evidence mapping to generate shadow evidence units corresponding to the invisible evidence units.

[0035] In this embodiment, for each evidence unit Execute the permission determination function. Permission determination function: .

[0036] when When, retain the original evidence unit; when At that time, the structure-preserving evidence mapping operator is executed on it. Generate shadow evidence units . This refers to the user who initiated the query request.

[0037] Structure-fidelity evidence mapping can be performed using the structure-fidelity evidence mapping operator. The function of the structure-fidelity evidence mapping operator is to: [details about the role / role]. For evidence unit Do not delete directly if you do not have access permissions. Instead, This is mapped to a substitute object that does not reveal the original content but retains its structural role, called a shadow evidence unit. Shadow Evidence Unit It is the original evidence unit The structural projection in the constrained view is used to place the original evidence in the evidence diagram; it provides insensitive category, time and structural information to the subsequent answer scoring and interpretation output modules; and it maintains the topological continuity of the evidence chain and the stability of the answer under role switching.

[0038] S14. Based on the shadow evidence unit, locally modify the cross-modal evidence subgraph to obtain the current evidence graph.

[0039] In this embodiment, the current evidence graph refers to a local logical view that is dynamically generated and belongs exclusively to the current user's current question-and-answer session after performing structure-fidelity evidence mapping on the pre-built cross-modal evidence subgraph based on the user role identifier of the current user.

[0040] S15. Based on the hyperconverged evidence arrangement matrix and the current evidence graph, generate a set of candidate answers including multiple candidate answers, and determine the set of supporting evidence for each candidate answer.

[0041] In this embodiment, the supporting evidence set refers to the set of all evidence units that provide factual basis and logical support for the candidate answer.

[0042] S16. Based on the supporting evidence set for each candidate answer, determine the decision score for each candidate answer.

[0043] In this embodiment, the decision score is used as a quantitative indicator to measure the overall suitability of a candidate answer as the final output answer. The decision score is the final score obtained after multi-dimensional evaluation of the candidate answers.

[0044] S17. Based on the decision score of each candidate answer, determine the target answer for the query request.

[0045] In the above embodiments, the user query is first analyzed using multi-dimensional facets, transforming the natural language query into a structured representation of different dimensions, thereby providing a refined matching benchmark for subsequent evidence matching. Simultaneously, the system pre-constructs a cross-modal global evidence graph with evidence units as nodes and multimodal associations as edges, and decouples this cross-modal global evidence graph from the query as a foundational asset for offline persistence, allowing any query to reuse the same global graph resource. Based on this, the system calculates the support strength between each query facet and each evidence unit in the cross-modal global evidence graph. It then first trims a cross-modal evidence subgraph relevant to the current query from the cross-modal global evidence graph based on the support strength, and then generates a hyper-fusion evidence orchestration matrix on this subgraph. This compresses the online reasoning scope from the entire evidence graph to the query-related subgraph, significantly reducing online computational overhead while preserving the topological structure between evidence. Subsequently, the system performs structure-fidelity evidence mapping on invisible evidence in the cross-modal evidence subgraph according to user permissions to generate shadow evidence units. This maintains the local topological continuity of the evidence chain without disclosing sensitive content. The system also locally modifies the cross-modal evidence subgraph to obtain the current evidence graph. Finally, the system jointly generates candidate answers and their supporting evidence sets based on the hyper-fusion evidence orchestration matrix and the current evidence graph. Based on the supporting evidence set of each candidate answer, the system determines the decision score for each candidate answer and outputs the target answer based on the decision score. This ensures that while low-privilege users cannot access sensitive original text, restricted evidence can still participate in answer scoring, evidence chain continuity, and role-consistent interpretation output through shadow forms. This improves the stability of the answer under role switching conditions and provides an interpretable basis for key evidence identification and quantification of the impact of permissions for audit trails.

[0046] In some embodiments, calculating the support strength between each query facet and each evidence unit in the cross-modal evidence subgraph includes: For a query facet and an evidence unit, the semantic matching degree between the query facet and the evidence unit is determined based on the normalized facet vector of the query facet and the semantic vector of the evidence unit. Based on the modal type of the query facet and the modal type of the evidence unit, determine the modal alignment between the query facet and the evidence unit; Based on the structural vectors of the query facets and the evidence units, the structural similarity between the query facets and the evidence units is determined. Based on the temporal information of the query facets and the temporal information of the evidence units, the temporal consistency between the query facets and the evidence units is determined. Evidence units with similarity to the dimension attributes corresponding to the query facets that are higher than the similarity threshold are identified from the cross-modal evidence subgraph as candidate evidence seeds. Candidate evidence seeds with a path length less than or equal to a preset length are added to the evidence seed set. The context path consistency between the query facets and evidence units is determined based on the edge weights between each evidence unit in the evidence seed set. The support strength between query facets and evidence units is determined by weighting semantic matching degree, modal alignment degree, structural similarity, temporal consistency degree and contextual path consistency degree.

[0047] In this embodiment, semantic matching measures the directional consistency between the normalized vector of the query facet and the semantic vector of the evidence unit in the semantic space. Modal alignment measures the degree of matching between the modal intent of the query facet and the actual modal type of the evidence unit. Structural similarity measures the similarity between the structural vectors of different evidence units. Temporal consistency measures the temporal proximity between the temporal constraints in the query facet and the timestamps or time intervals of the evidence units.

[0048] No. The query is directed to the first... Support strength of each unit of evidence: The optional methods for calculating semantic matching degree are as follows: , For the first The semantic vector of each evidence unit. Modal alignment. ,in, For evidence unit Modal type, This is the cross-modal alignment discount factor. The alignment score is the score obtained by passing the modal type vector of the query facet and the modal type vector of the evidence unit through the cross-modal alignment model. This is a preset value, less than 1. To query the modal type of the facets. Time consistency. , To query facets Timing information, For evidence unit The timing information. These are preset coefficients.

[0049] Context path consistency: .

[0050] in, For the set of evidence seeds, Let be the edge weight. to The weights are non-negative and satisfy the following conditions: . , These are evidence units within the evidence seed set. for One of the paths. The calculation of structural consistency is the same as that of structural similarity, and will not be repeated here.

[0051] In the above embodiments, faceted decoupling enables the system to independently monitor the coverage of each query dimension, avoiding matching bias caused by dimension coupling in single vectorization; the introduction of modality alignment and temporal consistency enables fair comparison of cross-modal evidence and evidence with uneven temporal distribution within a unified framework; context path consistency enables evidence without direct semantic matching but indirectly related through graph paths to still obtain effective scores, overcoming the recall limitations of pure semantic retrieval; the four components have clear physical meanings and are mutually orthogonal, providing a fine-grained and interpretable quantitative basis for the generation of the hyper-fusion evidence orchestration matrix, while enabling shadow evidence units to still obtain effective support strength scores through the above dimensions even without original sensitive content, and continue to participate in answer support in low-permission views.

[0052] In some embodiments, for invisible evidence units without access rights, performing structure-fidelity evidence mapping to generate shadow evidence units corresponding to the invisible evidence units includes: Obtain the business category to which the invisible evidence unit belongs, and perform coarse-grained mapping of the business category through the category bucketing function to obtain the category bucket of the shadow evidence unit; Obtain the time information of the invisible evidence unit, and perform coarse-grained mapping of the time information through the time bucketing function to obtain the time bucket of the shadow evidence unit; Obtain the evidence content to which the invisible evidence unit belongs, perform irreversible encoding on the evidence content to obtain the encoded content, and use the encoded content as the irreversible summary of the shadow evidence unit; The reason why the invisible evidence unit is restricted from access is used as the restriction reason code of the shadow evidence unit; The source domain identifier of the invisible evidence unit is used as the source domain summary of the shadow evidence unit; Shadow evidence units are generated based on at least one of the following: category bucket, time bucket, irreversible digest, restriction reason code, and source domain digest.

[0053] In this embodiment, , Category buckets represent invisible evidence units. The corresponding coarse-grained business category, such as "compensation," "contract terms," ​​and "log events," is determined through... Type label executes bucketing function get, . express Fine-grained type tags, For predefined category bucket mapping, fine-grained types are mapped to coarse-grained business category buckets. The purpose is to aggregate fine categories that may uniquely identify a sensitive object upwards to insensitive category buckets.

[0054] For time bucket, it means Time information The coarse-grained time intervals that fall into the range, such as "the last 7 days", "the last 30 days", or "Q1 of a certain year", are determined by a bucketing function. get, . for The timestamp or time range The function for time bucketing, such as bucketing by a sliding window into "last 7 days / last 30 days / last 90 days / earlier", abstracts precise time into time buckets from which the precise occurrence time of the event cannot be deduced.

[0055] It is an irreversible summary. Original content , The irreversible but instance-stable string obtained after triple processing with key hash is used for identity alignment of the same sensitive evidence across sessions.

[0056] .

[0057] This represents the set of character fragments obtained by performing N-gram decomposition on the original content. The MinHash stable compression function compresses a set of high-dimensional fragments into a fixed-length signature. For keyed An irreversible hash function is used to generate a hash of the same... Maintain stability, for different Fingerprints that are distinctive and whose original content cannot be deduced are used for identity alignment of shadow evidence units.

[0058] The reason code is used to restrict access. The value is taken from a predefined set of reason codes, such as "FIN_DENY Insufficient financial privileges" or "PII_DENY Involves personal information". It is output by the permission determination module when access is denied. This is a source domain summary, indicating... The high-level source domain identifier, such as "financial domain," "production domain," or "compliance domain," is determined through... The execution source is divided into buckets.

[0059] The above fields together ensure that low-privilege users have enough information to determine what type, when, and from what source evidence exists here, but cannot deduce the original sensitive content.

[0060] In some embodiments, based on shadow evidence units, local modifications are made to the cross-modal evidence subgraph to obtain the current evidence graph, including: The invisible evidence units in the cross-modal evidence subgraph are replaced with shadow evidence units. The edges of the neighbor nodes visible to the user and their corresponding edge weights in the invisible evidence units are transferred to the shadow evidence units. The edges corresponding to the neighbor nodes invisible to the user are deleted to obtain a temporary evidence graph. Based on the cross-modal evidence subgraph, the first local graph information of the invisible evidence unit is determined, and based on the temporary evidence graph, the second local graph information of the shadow evidence unit is determined. Based on the information from the first local graph and the information from the second local graph, the neighborhood preservation coefficient of the invisible evidence unit is determined. The current evidence graph is determined based on the neighborhood preservation coefficient.

[0061] In this embodiment, the shadow evidence unit The invisible evidence unit placed in the cross-modal evidence subgraph The original location was used to replace the invisible evidence unit. In the cross-modal evidence subgraph, the node roles are as follows: traverse all adjacent edges of the invisible evidence unit; for edges pointing to neighboring nodes visible to the user, transfer the edge along with its original edge weight to the shadow evidence unit, so that the shadow evidence unit inherits all the associations between the invisible evidence unit and the visible evidence; for edges pointing to neighboring nodes invisible to the user, since the invisible neighboring nodes have been removed during the permission pruning process, the edge loses its connection endpoint, so delete the edge directly without inheriting it from the shadow evidence unit; after the above replacement, transfer and deletion operations, a temporary evidence graph is obtained.

[0062] To ensure that shadow evidence units are not simply placeholders, but rather retain the structural role of the original evidence nodes in the local evidence graph, a neighborhood preservation coefficient is defined. The neighborhood preservation coefficient measures the degree to which the local topology of invisible evidence units in a cross-modal evidence subgraph is preserved before and after performing structure-fidelity evidence mapping. In other words, it reflects the extent to which shadow evidence units retain the structural role and connectivity patterns of the original evidence nodes in their local evidence networks.

[0063] Optionally, based on the cross-modal evidence subgraph, determining the first neighborhood information of the invisible evidence unit includes: Based on the cross-modal evidence subgraph, the degree of the first node of the invisible evidence unit is determined, the shortest path and the total number of first shortest paths of each node pair in the cross-modal evidence subgraph are obtained, and the number of first target paths passing through the invisible evidence unit is determined in the shortest paths of each node pair. Based on the total number of first shortest paths and the number of first target paths of each node pair, the proportion of the number of first paths of each node pair is determined. The proportions of the number of first paths of each node pair are accumulated to obtain the first transit importance corresponding to the invisible evidence unit. From the cross-modal evidence subgraph, the first adjacent edge weight vector of the invisible evidence unit is determined. Based on the provisional evidence map, the second partial map information for determining the shadow evidence unit includes: Based on the temporary evidence graph, the degree of the second node of the shadow evidence unit is determined, the shortest path and the total number of second shortest paths of each node pair in the temporary evidence graph are obtained, and the number of second target paths passing through the shadow evidence unit is determined in the shortest path of each node pair. Based on the total number of second shortest paths and the number of second target paths of each node pair, the proportion of the number of second paths of each node pair is determined. The proportions of the number of second paths of each node pair are accumulated to obtain the second transit importance corresponding to the shadow evidence unit. From the temporary evidence graph, the second adjacent edge weight vector of the shadow evidence unit is determined. Based on the information from the first local graph and the second local graph, the neighborhood preservation coefficient of the invisible evidence unit is determined as follows: The differences between the degree of the first node and the degree of the second node, the importance of the first transit point and the importance of the second transit point, and the weight vectors of the first and second adjacent edges are calculated to obtain the neighborhood preservation coefficient. Neighborhood Preservation Coefficient: .

[0064] in The degree of the first node of the invisible evidence unit and This indicates the degree of the second node of the shadow evidence unit. This indicates the importance of the first transit point corresponding to the invisible evidence unit. This indicates the second transit importance corresponding to the shadow evidence unit. The first transit importance measures the importance of the invisible evidence unit as a transit point in the cross-modal evidence subgraph. The second transit importance measures the importance of the invisible evidence unit as a transit point in the temporary evidence graph. ,in, For nodes arrive The number of shortest paths, For the process of The number of entries. The smaller the value, the better the shadow evidence unit is preserved. It serves as a key node in the support path. Let be the weight vector of the first adjacent edge of the invisible evidence unit. Invisible evidence sheet The vector formed by the edge weights between the first-order neighbors and the first-order neighbors, arranged by neighbor index. This is the second adjacent edge weight vector of the shadow evidence unit in the temporary evidence graph. Both... The smaller the distance, the closer the evidence transmission path in the temporary evidence diagram is, which is used to quantify the preservation of the weight distribution of shadow evidence units in the temporary evidence diagram. It is a non-negative parameter.

[0065] Optionally, the current evidence graph is determined based on the neighborhood preservation coefficient, including: When the neighborhood preservation coefficient is greater than or equal to the coefficient threshold, the temporary evidence graph is used as the current evidence graph; When the neighborhood preservation coefficient is less than the coefficient threshold, the following steps are performed: From the cross-modal evidence subgraph, identify neighboring nodes that are directly connected to or connected through an intermediate node to the invisible evidence unit, and filter out the neighboring nodes that are visible to the user to add to the candidate neighbor set; Calculate the edge weights between the shadow evidence unit and each neighbor node in the candidate neighbor set; Sort each edge weight in descending order, filter target neighbor nodes, and prioritize the edge weights corresponding to the target neighbor nodes by a predetermined number. In the temporary evidence graph, establish edges between the shadow evidence unit and each target neighbor node. For the remaining neighbor nodes in the candidate neighbor set that do not belong to the target neighbor node, discard the edges between the remaining neighbor nodes and the shadow evidence unit in the temporary evidence graph. Use the updated temporary evidence graph as the current evidence graph.

[0066] In this embodiment, when the neighborhood preservation coefficient is less than the coefficient threshold, neighborhood compensation is performed on the temporary evidence graph. At that time, explain the shadow evidence unit Local structures and invisible evidence units in the provisional evidence diagram Significant differences will affect the consistency between subsequent answer scoring and explanations. The threshold value is used as the coefficient. Neighborhood compensation is then performed to reconstruct the local support structure without revealing the original content.

[0067] The specific steps of neighborhood compensation include: Step 1: With Centered on the first-order neighborhood of the cross-modal evidence subgraph, take its first-order neighborhood. and second-order neighborhood Step 1: Select neighbor nodes that are visible to the user and add them to the candidate neighbor set; Step 2: Recalculate Each neighbor node in the candidate neighbor set Bridge edge rights between ;Retain the top facets most relevant to the query. Set one adjacent edge, and set the remaining adjacency relations to zero; Step 3: In the shadow evidence unit A bridging edge is inserted between the visible evidence unit and the unit, so that Maintain connectivity with the primary support path. Primary support path This refers to the visible evidence units on the current evidence graph that are strongly correlated with the query facets. For example... Figure 4 The image shown is a comparison diagram before and after performing neighborhood compensation in one embodiment. After performing neighborhood compensation, the chain of evidence is more continuous.

[0068] After obtaining the current evidence graph, the final neighborhood preservation coefficient can be obtained, and the neighborhood preservation coefficient can be used as an attribute of the shadow evidence unit.

[0069] In the above embodiments, by replacing the invisible evidence units in the cross-modal evidence subgraph with shadow evidence units and performing structural compensation, the resulting current evidence graph not only meets the permission compliance requirements, but also maintains the continuity of the evidence chain and the stability of the answer through the placement of shadow evidence units and the compensation of bridging edges; the desensitized structure of the shadow evidence units provides an information basis for interpretable output, and the secure isolation from the global evidence graph ensures audit traceability capabilities.

[0070] In some embodiments, the query facets include task facets, which are used to represent task type labels; Based on the hyperconverged evidence orchestration matrix and the current evidence graph, a candidate answer set including multiple candidate answers is generated, including: Based on the task type label, select the target generation method corresponding to the task type label, and use the target generation method to generate a set of candidate answers including multiple candidate answers based on the hyperconverged evidence orchestration matrix and the current evidence graph.

[0071] In this embodiment, based on task facets The value of is selected from one or more of the following four methods for candidate answer generation: Method G1 is template extraction, which predefines answer templates for task types such as fact verification and cause attribution. Hyperconverged evidence orchestration matrix The strongest evidence unit is filled into the template slot to generate candidate answers. Method G2 is a graph path induction approach. For task types such as process tracing and temporal relationships, it enumerates all evidence units with a length not exceeding the specified value on the current evidence graph, starting from the query facet seed node and ending at the evidence unit that satisfies the task constraints. The supported paths are summarized into a candidate answer for each path. The seed node for the query facet is the evidence node in the current evidence graph that has the strongest direct association with a given query facet. Method G3 is a generative model that arranges the hyper-fusion evidence matrix. China ranked top The evidence units are assembled into context in the form of evidence cards, which are then input into a large language model for controlled generation to obtain a set of candidate answers. Method G4 is a candidate pool retrieval approach. For tasks with known answer candidate pools (such as FAQ databases or case law databases), the similarity is calculated using the query facet vector and the candidate pool representation, and then sorted in descending order, taking the top similarity results. The candidate answers.

[0072] For each candidate answer The binding of the supporting evidence set, as seen in the evidence unit set: .

[0073] Shadow evidence unit set: .

[0074] Supporting evidence set: .

[0075] in, for with candidate answers The local support strength between candidate answers can be determined based on their semantic vectors and... The similarity is calculated from the semantic vectors. To support intensity thresholds; as candidate answers The set of evidence sources directly cited or obtained through path induction during the generation process depends specifically on the generation method: if Generated by method G1, then For all evidence units filled into the template slots; if Generated by method G2, then To constitute all the evidence units on the supporting path that generates this answer; if Generated by method G3, then To splice into the context One unit of evidence; if Generated by method G4, then This is the set of related evidence corresponding to the search results in the candidate pool. This is the set of visible evidence units that the user has access to in the current evidence graph. Indicates from the set of shadow evidence units In the middle, exclude those that simultaneously belong to the set of visible evidence units. The set of elements remaining after the first element. for with candidate answers The local support strength between candidate answers can be determined based on their semantic vectors and... The similarity is calculated from the semantic vectors.

[0076] By applying the aforementioned dual conditions of local support strength threshold and generating source tracing set, we can ensure... Both cover direct generation The evidence unit is further covered by evidence supported by similarity. Other evidence units provide complete input for subsequent calculation of the basic score.

[0077] In some embodiments, determining the decision score for each candidate answer based on the supporting evidence set for each candidate answer includes: For any candidate answer, the candidate answer is evaluated based on the set of supporting evidence, and the user's permissions are met, to obtain the first basic score; For each piece of supporting evidence in the supporting evidence set, the evidence deletion perturbation calculation is performed. After removing the supporting evidence from the supporting evidence set, the second basic score of the candidate answer is recalculated to obtain the second basic score corresponding to each piece of supporting evidence. The difference between the first basic score and the second basic score corresponding to each piece of supporting evidence is taken as the evidence deletion perturbation contribution of each piece of supporting evidence. From the evidence deletion perturbation contributions of each piece of supporting evidence, the largest evidence deletion perturbation contribution is selected as the evidence domain counterfactual contribution. Obtain target permissions higher than user permissions, calculate the third basic score of candidate answers under the target permissions, and use the difference between the first basic score and the third basic score as the role switching perturbation contribution. The total value of the disturbance contribution is determined based on the counterfactual contribution of the evidence domain and the disturbance contribution of role switching; The decision score for each candidate answer is determined based on the initial baseline score and the total value of the disturbance contribution.

[0078] In this embodiment, candidate answers Decision score ,in This indicates the first basic score. The total value contributed to the disturbance. These are preset coefficients.

[0079] The total disturbance contribution refers to the marginal impact of evidence on the final conclusion, evaluated simultaneously across two different disturbance domains: the physical evidence domain and the authority domain. Specifically, in the physical evidence domain, the impact on the answer score is evaluated by performing a counterfactual operation—deleting a unit of evidence from the supporting evidence set—once the evidence is removed. In the authority domain, the impact on the answer score is evaluated by performing a counterfactual operation—switching the user's permissions to a higher level—once the user's permissions are changed. The introduction of dual-domain counterfactual contribution allows the system to directly answer two types of audit questions while providing the answer: which evidence is critical, and whether the conclusion would differ depending on permissions.

[0080] To incorporate interpretation and robustness analysis into the main path of answer selection, the total perturbation contribution includes counterfactual contributions from the evidence domain and perturbation contributions from role switching. Evidence deletion perturbation contribution. The measurement removes a specific unit of evidence while keeping the roles unchanged. back The decrease in baseline scores reflects the necessity of the evidence. Role-switching perturbation contribution. The measurement switches the role from user-level to high-level role while keeping the evidence set unchanged. back The increase in the base rating reflects the character's unlockability. Both affect different counterfactual variables and are therefore independent, requiring separate calculation and weighted fusion. Evidence deletion perturbation contribution. , Indicates deletion The second base score obtained afterward. Role switching disturbance contribution. , This indicates the first basic score. Obtaining the third basic score under the target permissions means that when the target permissions change, it is necessary to redefine the current evidence graph and the set of supporting evidence to obtain the third basic score. Total perturbation contribution value. , These are preset coefficients. This indicates the counterfactual contribution of the evidence domain.

[0081] Optionally, based on the supporting evidence set for the candidate answers, and with the user's permissions, the candidate answers are evaluated to obtain a first basic score, including: For any candidate answer, count the number of query facets covered in the supporting evidence set, and use the ratio of the number of query facets to the total number of query facets as the coverage of the candidate answer; Based on the semantic vectors of each supporting evidence in the supporting evidence set and the semantic vectors of the candidate answers, the local support strength of each supporting evidence for the candidate answer is calculated, and the total local support strength corresponding to the candidate answer is obtained based on the local support strength of each supporting evidence. The continuity contribution value of each supporting evidence in the supporting evidence set is determined. When the supporting evidence is a visible evidence unit, the continuity contribution value of the supporting evidence is the preset continuity contribution value. When the supporting evidence is a shadow evidence unit, the continuity contribution value of the supporting evidence is the neighborhood preservation coefficient of the shadow evidence unit. The continuity contribution values ​​of each supporting evidence in the supporting evidence set are added together and then divided by the total number of supporting evidence to obtain the continuity corresponding to the candidate answer. Calculate the risk penalty for candidate answers. The risk penalty includes at least one or more of the following: time expiration risk, source conflict risk, and permission uncertainty risk. The first basic score is determined based on at least one of the following: coverage, total local support strength, continuity, and risk penalty.

[0082] In this embodiment, the evidence units in the supporting evidence set are supporting evidence. For each candidate answer... The coverage, total local support strength, continuity, and risk penalty are calculated to obtain the first basic score. The calculation process for the second and third basic scores is the same and will not be repeated here.

[0083] Coverage , Indicates candidate answers Coverage of the query faceted set; To query the total number of facets; as candidate answers The set of supporting evidence; To support strength; This is the facet coverage threshold; For indicator functions, the value is [value] when the condition is true. Otherwise The coverage calculation process is as follows: for each query facet... ,Pick The maximum support strength of this facet in the set of supporting evidence. If not less than the threshold If the coverage is effective, then the facet is considered to be effectively covered; the coverage is the proportion of facets that are effectively covered.

[0084] Local support total strength , To support the evidence For candidate answers Local support strength, used to measure Does it directly support The proposition asserted can be calculated based on the similarity between the semantic vectors of the supporting evidence and the semantic vectors of the candidate answers.

[0085] continuity , Indicates candidate answers The continuity of the chain of evidence; for The number of elements supporting the evidence set; That is, the continuity contribution of the visible evidence unit is The continuity contribution of shadow evidence units is their neighborhood preservation coefficient. Continuity is The arithmetic mean of the continuity contributions of all supporting evidence units is used to quantify the current permissions view. The completeness of the support chain.

[0086] First Basic Scoring ,in The risk penalties are characterized by at least the risks of time expiration, source conflict, and authority uncertainty. Adjustable weights. First basic score. Based on the above four dimensions, this serves as a preliminary score for candidate answers without introducing counterfactual analysis, for future reference.

[0087] In some embodiments, the target answer to the query request is determined based on the decision score of each candidate answer, including: Determine the maximum decision score based on the decision score for each candidate answer; When there is only one maximum decision score, the candidate answer corresponding to the maximum decision score is taken as the target answer. When there are multiple maximum decision scores, the target answer is determined based on the following priority order, from high to low: coverage, total local support strength, and continuity.

[0088] In this embodiment, in the candidate answer set Calculate a decision score for each candidate answer. The candidate answer with the highest score is selected as the final output answer. When multiple candidate answers have the same score, the answer is selected based on coverage. Total strength of local support Continuity The order of events is broken.

[0089] When there are multiple candidate answers make To ensure the determinism and reproducibility of the output answer, a lexicographical comparison rule is used to sort parallel candidates. First, candidates with higher coverage are prioritized. If coverage is equal, candidates with higher total local support strength are prioritized. If total local support strength is still equal, candidates with higher continuity are prioritized. This lexicographical processing ensures the determinism of the output result under the same input; avoids answer jitter caused by floating-point calculation errors; prioritizes the breadth of facet coverage of the answer, and then considers the strength of evidence and structural integrity, consistent with the technical principle of prioritizing structural fidelity in this application. If all three levels of comparison are equal, the answer is considered completely equivalent and output in ascending order of candidate answer index.

[0090] Based on the final answer Output at least the following: answer text, a summary of role consistency explanations, and an audit view. (Summary of Role Consistency Explanations) , To establish a visible primary support path, the answer can be obtained from the target by working on the current evidence graph being constructed. Visible evidence supporting the set Find the main path of the longest spanning tree. To support the path for the shadow, the answer can be obtained by addressing the target. Shadow evidence support set According to the neighborhood preservation coefficient It is obtained by connecting the largest to the smallest. Answer the target Supporting evidence set In the process, each evidence unit is calculated and removed one by one. After that, the decrease in the base score of the answer. Then, extract the largest decrease from it. The stability level of the conclusion is derived from the... and The joint classification, in which The contribution of role switching disturbance is expressed by the following formula: when At that time, the system will automatically add a prompt to the role-consistent explanation summary: "The current conclusion may be more supported or adjusted under a high-privilege view." , These represent the threshold values.

[0091] In some embodiments, joint training is used to simultaneously train the parameters of all trainable modules in this invention, including at least one of the following: faceted encoding functions. and its parameters Multimodal encoder (generates semantic vectors) );coefficient ;coefficient Modal alignment model; category bucket and time bucket binning functions; neighborhood preservation weights. and neighborhood compensation coefficient Evidence weight coefficient Weighting of base score Counterfactual weight and final score weighting Joint training is reflected in the system training phase. During online training, when a certain type of restricted evidence leads to prolonged... When the value is too high, the time bucket or category bucket granularity of this type of shadow evidence unit can be automatically increased, or the neighborhood compensation parameter can be adjusted to improve the consistency of conclusions between different roles.

[0092] In another aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal question-answering method based on evidence fidelity mapping according to any embodiment of this application.

[0093] Among them, in the computer program product, the optional implementation form of the program module architecture of the computer program that implements each step of the multimodal question answering method based on evidence fidelity mapping can be a multimodal question answering device based on evidence fidelity mapping.

[0094] Please see Figure 5This application provides an embodiment of a multimodal question-answering device based on evidence fidelity mapping, comprising: an acquisition module 51, configured to acquire a user-initiated query request, parse the query request from different dimensions to obtain a set of query facets including multiple query facets; a construction module 52, configured to acquire a pre-constructed cross-modal global evidence graph, wherein evidence units are nodes and two evidence units are edges in the cross-modal global evidence graph; and a generation module 53, configured to calculate the support strength between each query facet and each evidence unit in the cross-modal global evidence graph, based on each support strength, crop a cross-modal evidence subgraph from the cross-modal global evidence graph, and arrange the support strength between each query facet and each evidence unit in the cross-modal evidence subgraph in order to generate a hyper-fusion evidence orchestration. The generation module 53 is further configured to determine access permissions for each of the evidence units based on the user's user permissions, and to perform structure-fidelity evidence mapping for invisible evidence units without access permissions, thereby generating shadow evidence units corresponding to the invisible evidence units; the generation module 53 is further configured to locally modify the cross-modal evidence subgraph based on the shadow evidence units to obtain the current evidence graph; the determination module 54 is configured to generate a set of candidate answers including multiple candidate answers based on the hyper-fusion evidence arrangement matrix and the current evidence graph, and to determine the supporting evidence set for each candidate answer; the determination module 54 is further configured to determine the decision score for each candidate answer based on the supporting evidence set for each candidate answer; the determination module 54 is further configured to determine the target answer for the query request based on the decision score for each candidate answer.

[0095] It will be understood by those skilled in the art that Figure 5 The structure of the evidence-fidelity mapping-based multimodal question-answering device does not constitute a limitation on the device itself. Each module can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the controller in the computer device, or stored in software in the memory of the computer device, so that the controller can invoke and execute the operations corresponding to each module. In other embodiments, the evidence-fidelity mapping-based multimodal question-answering device may include more or fewer modules than illustrated.

[0096] Please see Figure 6In another aspect of this application, a computing device 200 is also provided, including a memory 3011 and a processor 3012. The memory 3011 stores a computer program, and when the computer program is executed by the processor, the processor 3012 performs the steps of the multimodal question-answering method based on evidence fidelity mapping provided in any of the above embodiments of this application. The computing device 200 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., a pair of smart glasses or a smartwatch) or similar devices, and robotic devices.

[0097] The processor 3012 is the control center, connecting various data points throughout the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 3011, and calls data stored in the memory 3011 to perform various functions and process data. Optionally, the processor 3012 may include one or more processing cores; preferably, the processor 3012 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 3012.

[0098] The memory 3011 can be used to store software programs and modules. The processor 3012 executes various functional applications and data processing by running the software programs and modules stored in the memory 3011. The memory 3011 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 3011 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 3011 may also include a controller to provide the processor 3012 with access to the memory 3011.

[0099] In another aspect, this application also provides a storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the multimodal question-answering method based on evidence fidelity mapping provided in any of the above embodiments of this application.

[0100] Those skilled in the art will understand that all or all data flows in the methods provided in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the flows of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile media. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.

[0101] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal question-answering method based on evidence-fidelity mapping, characterized in that, include: Obtain the query request initiated by the user, parse the query request from different dimensions, and obtain a set of query facets including multiple query facets; Obtain a pre-constructed cross-modal global evidence graph, wherein evidence units are nodes and two evidence units are edges; Calculate the support strength between each query facet and each evidence unit in the cross-modal global evidence graph. Based on each support strength, cut out a cross-modal evidence subgraph from the cross-modal global evidence graph. Arrange the support strength between each query facet and each evidence unit in the cross-modal evidence subgraph in order to generate a hyper-fusion evidence orchestration matrix. Based on the user's user permissions, the access permissions for each of the evidence units are determined. For invisible evidence units without access permissions, structure-fidelity evidence mapping is performed to generate shadow evidence units corresponding to the invisible evidence units. The structure-fidelity evidence mapping maps the invisible evidence units to a substitute object that does not reveal the original content of the invisible evidence units but retains the structural role of the invisible evidence units. Based on the shadow evidence unit, the cross-modal evidence subgraph is locally modified to obtain the current evidence graph; Based on the hyperconverged evidence orchestration matrix and the current evidence graph, a set of candidate answers including multiple candidate answers is generated, and a set of supporting evidence for each candidate answer is determined. Based on the supporting evidence set for each candidate answer, a decision score is determined for each candidate answer; The target answer to the query request is determined based on the decision score of each candidate answer.

2. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 1, characterized in that, The calculation of the support strength between each of the query facets and each of the evidence units in the cross-modal evidence subgraph includes: For a query facet and an evidence unit, the semantic matching degree between the query facet and the evidence unit is determined based on the normalized facet vector of the query facet and the semantic vector of the evidence unit. Based on the modal type of the query facet and the modal type of the evidence unit, determine the modal alignment between the query facet and the evidence unit; Based on the structural vector of the query facet and the structural vector of the evidence unit, the structural similarity between the query facet and the evidence unit is determined. Based on the time sequence information of the query facets and the time sequence information of the evidence units, the time consistency between the query facets and the evidence units is determined. Evidence units with similarity to the dimension attribute corresponding to the query facet higher than the similarity threshold are determined from the cross-modal evidence subgraph as candidate evidence seeds. Candidate evidence seeds with a path length less than or equal to a preset length are added to the evidence seed set. The context path consistency between the query facet and the evidence unit is determined based on the edge weight between each evidence unit in the evidence seed set. The semantic matching degree, modal alignment degree, structural similarity, temporal consistency degree, and contextual path consistency degree are weighted to determine the support strength between the query facet and the evidence unit.

3. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 1, characterized in that, The step of performing structure-fidelity evidence mapping on invisible evidence units for which access is denied, and generating shadow evidence units corresponding to the invisible evidence units, includes: Obtain the business category to which the invisible evidence unit belongs, and perform coarse-grained mapping on the business category through the category bucketing function to obtain the category bucket of the shadow evidence unit; Obtain the time information of the invisible evidence unit, and perform coarse-grained mapping on the time information through the time bucketing function to obtain the time bucket of the shadow evidence unit; Obtain the evidence content to which the invisible evidence unit belongs, perform irreversible encoding on the evidence content to obtain the encoded content, and use the encoded content as the irreversible digest of the shadow evidence unit; The reason why the invisible evidence unit is restricted from access is used as the restriction reason code of the shadow evidence unit; The source domain identifier of the invisible evidence unit is used as the source domain digest of the shadow evidence unit; The shadow evidence unit is generated based on at least one of the following: the category bucket, the time bucket, the irreversible digest, the restriction reason code, and the source domain digest.

4. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 1, characterized in that, The step of locally modifying the cross-modal evidence subgraph based on the shadow evidence unit to obtain the current evidence graph includes: The invisible evidence unit in the cross-modal evidence subgraph is replaced with the shadow evidence unit. The edges of the neighbor nodes visible to the user and their corresponding edge weights in the invisible evidence unit are transferred to the shadow evidence unit. The edges corresponding to the neighbor nodes invisible to the user are deleted to obtain a temporary evidence graph. Based on the cross-modal evidence subgraph, the first local graph information of the invisible evidence unit is determined, and based on the temporary evidence graph, the second local graph information of the shadow evidence unit is determined; Based on the first local graph information and the second local graph information, the neighborhood preservation coefficient of the invisible evidence unit is determined; The current evidence graph is determined based on the neighborhood preservation coefficient.

5. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 4, characterized in that, The determination of the first neighborhood information of the invisible evidence unit based on the cross-modal evidence subgraph includes: Based on the cross-modal evidence subgraph, the degree of the first node of the invisible evidence unit is determined, the shortest path and the total number of first shortest paths of each node pair in the cross-modal evidence subgraph are obtained, and the number of first target paths passing through the invisible evidence unit is determined in the shortest paths of each node pair. Based on the total number of first shortest paths and the number of first target paths of each node pair, the proportion of the number of first paths of each node pair is determined. The proportions of the number of first paths of each node pair are accumulated to obtain the first transit importance corresponding to the invisible evidence unit. From the cross-modal evidence subgraph, the first adjacent edge weight vector of the invisible evidence unit is determined. The determination of the second partial map information of the shadow evidence unit based on the temporary evidence map includes: Based on the temporary evidence graph, the degree of the second node of the shadow evidence unit is determined, the shortest path and the total number of second shortest paths of each node pair in the temporary evidence graph are obtained, and the number of second target paths passing through the shadow evidence unit is determined in the shortest paths of each node pair. Based on the total number of second shortest paths and the number of second target paths of each node pair, the proportion of the number of second paths of each node pair is determined. The proportions of the number of second paths of each node pair are accumulated to obtain the second transit importance of the shadow evidence unit. From the temporary evidence graph, the second adjacency edge weight vector of the shadow evidence unit is determined. The determination of the neighborhood preservation coefficient of the invisible evidence unit based on the first local graph information and the second local graph information includes: The differences between the degree of the first node and the degree of the second node, the importance of the first transit point and the importance of the second transit point, and the weight vectors of the first and second adjacent edges are calculated to obtain the neighborhood preservation coefficient.

6. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 4, characterized in that, Determining the current evidence graph based on the neighborhood preservation coefficient includes: When the neighborhood preservation coefficient is greater than or equal to the coefficient threshold, the temporary evidence graph is used as the current evidence graph; When the neighborhood preservation coefficient is less than the coefficient threshold, the following steps are performed: From the cross-modal evidence subgraph, determine the neighboring nodes that are directly connected to the invisible evidence unit or connected through an intermediate node, and filter out the neighboring nodes that are visible to the user to add to the candidate neighbor set; Calculate the edge weight between the shadow evidence unit and each neighbor node in the candidate neighbor set; Sort the edge weights in descending order to filter target neighbor nodes. The edge weights corresponding to the target neighbor nodes are ranked first by a preset number. In the temporary evidence graph, edges are established between the shadow evidence unit and each target neighbor node. For the remaining neighbor nodes in the candidate neighbor set that do not belong to the target neighbor node, the edges between the remaining neighbor nodes and the shadow evidence unit are discarded in the temporary evidence graph. The updated temporary evidence graph is used as the current evidence graph.

7. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 1, characterized in that, The query facets include task facets, which are used to represent task type labels. The step of generating a candidate answer set including multiple candidate answers based on the hyperconverged evidence orchestration matrix and the current evidence graph includes: Based on the task type label, select the target generation method corresponding to the task type label, and use the target generation method to generate a set of candidate answers including multiple candidate answers based on the hyperconverged evidence orchestration matrix and the current evidence graph.

8. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 1, characterized in that, The determination of the decision score for each candidate answer based on the supporting evidence set for each candidate answer includes: For any of the candidate answers, the candidate answer is evaluated based on the set of supporting evidence for the candidate answer, under the user's permissions, to obtain a first basic score; For each piece of supporting evidence in the supporting evidence set, evidence deletion perturbation calculation is performed. After removing the supporting evidence from the supporting evidence set, the second basic score of the candidate answer is recalculated to obtain the second basic score corresponding to each piece of supporting evidence. The difference between the first basic score and the second basic score corresponding to each piece of supporting evidence is taken as the evidence deletion perturbation contribution of each piece of supporting evidence. From the evidence deletion perturbation contributions of each piece of supporting evidence, the largest evidence deletion perturbation contribution is selected as the evidence domain counterfactual contribution. Obtain target permissions higher than user permissions, calculate the third basic score of the candidate answer under the target permissions, and use the difference between the first basic score and the third basic score as the role switching perturbation contribution. Based on the counterfactual contribution of the evidence domain and the role switching disturbance contribution, the total disturbance contribution is determined; The decision score of the candidate answer is determined based on the first basic score and the total value of the disturbance contribution.

9. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 8, characterized in that, The supporting evidence set based on the candidate answers, under the user's permissions, evaluates the candidate answers to obtain a first basic score, including: For any of the candidate answers, the number of query facets covered by the supporting evidence set is counted, and the ratio of the number of query facets to the total number of query facets is taken as the coverage of the candidate answer. Based on the semantic vectors of each supporting evidence in the supporting evidence set and the semantic vector of the candidate answer, the local support strength of each supporting evidence for the candidate answer is calculated, and the total local support strength corresponding to the candidate answer is obtained based on the local support strength of each supporting evidence. The continuity contribution value of each supporting evidence in the supporting evidence set is determined respectively. When the supporting evidence is a visible evidence unit, the continuity contribution value of the supporting evidence is a preset continuity contribution value. When the supporting evidence is a shadow evidence unit, the continuity contribution value of the supporting evidence is the neighborhood preservation coefficient of the shadow evidence unit. The continuity contribution values ​​of each supporting evidence in the supporting evidence set are added together and divided by the total number of supporting evidence to obtain the continuity corresponding to the candidate answer. Calculate the risk penalty item for the candidate answer, wherein the risk penalty item includes at least one or more of the following: time expiration risk, source conflict risk, and permission uncertainty risk; The first base score is determined based on at least one of the following: coverage, total local support strength, continuity, and risk penalty.

10. The multimodal question-answering method based on evidence-fidelity mapping as described in claim 9, characterized in that, The determination of the target answer for the query request based on the decision score for each candidate answer includes: Determine the maximum decision score based on the decision score for each candidate answer; When there is only one maximum decision score, the candidate answer corresponding to the maximum decision score is taken as the target answer. When there are multiple maximum decision scores, the target answer is determined based on the following priority order, wherein the priority order from high to low is: coverage, total local support strength, and continuity.

Images