Multi-modal evidence chain management method and device, computer device and medium

Abstract

The application discloses a kind of multi-modal evidence chain management and control method, device, computer equipment and medium, comprising: standardization processing is carried out to multi-modal data, obtain standardized data, and multi-modal data includes image data, video data and text data;According to standardized data, feature extraction and low rank projection are carried out, and uniform feature representation is obtained;Causal evidence graph is constructed based on uniform feature representation, and causal relationship network is obtained;Causal reinforcement retrieval is carried out to causal relationship network, and retrieval result is obtained;Restricted decoding and attention consistency control are carried out using retrieval result, and generation content is obtained;Structured evidence chain record is generated based on generation content and retrieval result, and structured evidence chain record is used as traceability output information.The application realizes the promotion of evidence chain quality and credibility, enhances generation controllability and resource utilization rate.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method, apparatus, computer equipment, and medium for managing multimodal evidence chains. Background Technology

[0002] With the rapid development of multimodal large language models, retrieval enhancement generation technology has expanded from the realm of plain text to complex scenarios encompassing multimodal information such as images, videos, audio, and sensor data. In critical business applications such as multimodal question answering, security traceability, industrial quality inspection, and identity verification, systems need to retrieve relevant evidence from massive amounts of heterogeneous data and generate reliable analytical conclusions or decision-making suggestions.

[0003] Existing technologies mainly employ semantic similarity-based retrieval methods, which filter relevant fragments by calculating the distance between the query and multimodal evidence in the vector space, and then directly input the retrieval results into the generative model to generate output content.

[0004] In the process of realizing this invention, the inventors realized that the prior art has at least the following technical problems:

[0005] First, semantic relevance cannot ensure that there is a real causal relationship between evidence fragments. This may result in retrieved evidence that, although relevant to the query, is isolated and lacks a logical chain, making it difficult to support a convincing conclusion.

[0006] Secondly, the alignment between different modal evidence mainly relies on a simple mapping of the feature space, which is prone to alignment deviations in key dimensions such as temporal sequence and spatial location, resulting in inconsistent focus of the generative model when integrating multimodal information.

[0007] Furthermore, the lack of effective constraints on model referencing behavior during the generation phase may lead to the model generating content beyond the scope of the retrieved evidence, affecting the credibility and verifiability of the output. Finally, existing cross-modal attention mechanisms incur significant computational overhead when processing complex data such as high-resolution images and long video sequences, limiting their feasibility for deployment in real-time systems.

[0008] Therefore, there is an urgent need for a multimodal evidence chain management scheme that can effectively construct causal relationships, ensure controllable generation, and take into account computational efficiency, in order to solve the dual dilemma of existing technologies in terms of credibility and excessive resource consumption. Summary of the Invention

[0009] This invention provides a multimodal evidence chain management method, apparatus, computer equipment, and storage medium to improve the credibility of the evidence chain and the efficiency of evidence chain management.

[0010] To address the aforementioned technical problems, embodiments of this application provide a multimodal evidence chain control method, including:

[0011] The multimodal data is standardized to obtain standardized data, which includes image data, video data, and text data.

[0012] Based on the standardized data, feature extraction and low-rank projection are performed to obtain a unified feature representation;

[0013] Based on the unified feature representation, a causal evidence graph is constructed to obtain a causal relationship network;

[0014] A causal reinforcement search was performed on the causal relationship network to obtain the search results;

[0015] The retrieved results are used for restricted decoding and attention consistency control to obtain the generated content;

[0016] A structured chain of evidence record is generated based on the generated content and search results, and the structured chain of evidence record is used as traceability output information.

[0017] Optionally, the step of performing feature extraction and low-rank projection based on the standardized data to obtain a unified feature representation includes:

[0018] Each modal data is encoded separately to obtain encoded features;

[0019] Based on the singular value energy distribution of the encoded features, the rank of the projection matrix is ​​adaptively determined;

[0020] The encoded features are projected onto the low-rank projection matrix of the rank to generate the low-dimensional feature representation, thus obtaining a unified feature representation.

[0021] Optionally, constructing a causal evidence graph based on the unified feature representation to obtain a causal relationship network includes:

[0022] Based on the unified feature representation, calculate the directed causal strength between evidence nodes;

[0023] Based on the directed causal strength, directed edges are constructed between evidence nodes to form a causal relationship network, and the overall causal confidence of the causal relationship network is calculated through path product.

[0024] Optionally, the causal reinforcement retrieval of the causal relationship network to obtain retrieval results includes:

[0025] From the unified feature representation, obtain the unified feature representation of the query evidence and the candidate evidence, and obtain the query evidence feature and the candidate evidence feature respectively;

[0026] Semantic relevance scores are calculated for the query evidence features and candidate evidence features to obtain the semantic relevance score;

[0027] The semantic relevance score is weighted and fused with the overall causal confidence score to generate the final retrieval score. Based on this score, the candidate evidence is rearranged to obtain retrieval results that include causal path information from the causal relationship network.

[0028] Optionally, the step of performing restricted decoding and attention consistency control based on the retrieval results to obtain generated content includes:

[0029] Based on the search results, a limited generated vocabulary is determined, and the optional vocabulary in the decoding process is restricted to the scope of the limited generated vocabulary.

[0030] By using the KL divergence loss function, the attention distribution of the generative model during the generation process is aligned with the attention distribution of the retrieval module on the retrieval results;

[0031] Based on the restricted generated vocabulary and the attention-aligned generation model, restricted decoding is performed to generate the generated content.

[0032] To address the aforementioned technical problems, this application also provides a multimodal evidence chain control device, comprising:

[0033] The data processing module is used to standardize multimodal data to obtain standardized data, wherein the multimodal data includes image data, video data and text data;

[0034] The feature extraction module is used to extract features and perform low-rank projection based on the standardized data to obtain a unified feature representation;

[0035] The evidence graph construction module is used to construct a causal evidence graph based on the unified feature representation to obtain a causal relationship network;

[0036] The causal retrieval module is used to perform causal reinforcement retrieval on the causal relationship network and obtain retrieval results;

[0037] The content generation module is used to perform restricted decoding and attention consistency control on the search results to generate content;

[0038] The evidence chain survival module is used to generate a structured evidence chain record based on the generated content and search results, and to use the structured evidence chain record as traceability output information.

[0039] Optionally, the feature extraction module includes:

[0040] The coding unit is used to encode each modal data separately to obtain the coded features;

[0041] A computational unit is used to adaptively determine the rank of the projection matrix based on the singular value energy distribution of the encoded features;

[0042] The projection unit is used to project the encoded features using the low-rank projection matrix of the rank, generate the low-dimensional feature representation, and obtain a unified feature representation.

[0043] Optionally, the evidence graph construction module includes:

[0044] The causal strength calculation unit is used to calculate the directed causal strength between evidence nodes based on the unified feature representation.

[0045] The causal relationship network construction unit is used to construct directed edges between evidence nodes based on the directed causal strength, form a causal relationship network, and calculate the overall causal confidence of the causal relationship network through path product.

[0046] To address the aforementioned technical problems, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned multimodal evidence chain control method.

[0047] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned multimodal evidence chain control method.

[0048] The multimodal evidence chain management method, apparatus, computer equipment, and storage medium provided in this invention standardize multimodal data (including image, video, and text data) to obtain standardized data. Feature extraction and low-rank projection are performed on the standardized data to obtain a unified feature representation. A causal evidence graph is constructed based on the unified feature representation to obtain a causal relationship network. Causal reinforcement retrieval is performed on the causal relationship network to obtain retrieval results. Restricted decoding and attention consistency control are applied to the retrieval results to obtain generated content. A structured evidence chain record is generated based on the generated content and retrieval results, and this structured evidence chain record is used as traceability output information. By standardizing multimodal data and using unified feature representation, the semantic gap and alignment challenges of heterogeneous data sources are effectively overcome, laying a high-quality data foundation for subsequent processing. Furthermore, the introduction of a causal evidence graph and a causal reinforcement retrieval mechanism breaks through the limitations of traditional methods that rely solely on semantic relevance, ensuring that the retrieved evidence is both semantically relevant and causally coherent, fundamentally improving the logical rigor of the evidence chain. Furthermore, by employing restricted decoding and attention consistency control, the generation process is strictly constrained within the scope of evidence support, significantly reducing the risks of model illusion and unauthorized citation, and enhancing the reliability and credibility of the generated content. Finally, by generating a structured chain of evidence, the entire process from raw data to generated conclusions is traceable and verifiable, providing a complete machine-readable basis for audit review. This improves the quality and credibility of the evidence chain, enhances the controllability of generation, and improves resource utilization. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0051] Figure 2 This is a flowchart of an embodiment of the multimodal evidence chain control method of this application;

[0052] Figure 3 This is a schematic diagram of a structure of an embodiment of the multimodal evidence chain control device according to this application;

[0053] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0055] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 ,like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0058] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0059] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0060] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0061] It should be noted that the multimodal evidence chain management method provided in this application embodiment is executed by the server, and correspondingly, the multimodal evidence chain management device is set in the server.

[0062] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. The terminal devices 101, 102, and 103 in this embodiment can specifically correspond to application systems in actual production.

[0063] Please see Figure 2 , Figure 2 This invention illustrates a multimodal evidence chain control method provided by an embodiment of the present invention, which is applied to... Figure 1 Taking the server-side as an example, the details are as follows:

[0064] S201: Standardize the multimodal data to obtain standardized data. The multimodal data includes image data, video data, and text data.

[0065] Specifically, multimodal data from different sources undergoes standardization processing. The acquired multimodal data includes RGB image data from surveillance cameras, thermal imaging data from infrared sensors, continuous video frame sequences containing temporal information, and text data such as device logs, operation records, and manual annotations. These multimodal data are timestamped and aligned to a unified time standard, ensuring consistency across different sensors in the temporal dimension. Next, quality screening is performed, assessing the quality of image and video data and removing blurry, overexposed, or low-quality frames. Then, image data is uniformly adjusted to a preset resolution, and text data undergoes word segmentation and vectorization. Finally, a sliding window is used for temporal partitioning, and Regions of Interest (ROIs) are extracted from the images in the spatial domain to form a standardized set of evidence units.

[0066] This step aims to transform heterogeneous multimodal data from different sources into a unified, standardized format, laying the foundation for subsequent feature extraction and fusion processing. The specific implementation process includes the following sub-steps:

[0067] First, multimodal data acquisition and input are performed. The system receives raw data from multiple data sources, including: RGB image data acquired from a visible light camera, thermal imaging data acquired from an infrared sensor, video stream data containing continuous temporal information, and text description data from system logs, manually entered data, or automatically generated data. These data differ significantly in format, resolution, and temporal characteristics, requiring unified processing.

[0068] Next, timestamp alignment and synchronization are performed. Since different sensors may have clock discrepancies, the system first assigns a uniform timestamp to all input data, performing time alignment with millisecond-level precision. For video data, keyframes are extracted and bound to their corresponding timestamps; for text logs, the time information is parsed and correlated with the visual data. This step ensures accurate establishment of cross-modal temporal relationships during subsequent processing.

[0069] Then, data quality screening and enhancement are performed. Visual data (images and video frames) undergo quality assessment, using a deep learning-based quality detection model to identify and remove low-quality frames such as blurry, overexposed, underexposed, or containing multiple faces. Text data is cleaned, including removing special characters, correcting spelling errors, and standardizing named entity formats. Simultaneously, image data with varying resolutions is interpolated or downsampled to uniformly adjust to a preset input size (e.g., 112×112 pixels).

[0070] Finally, the evidence units were divided and the output standardized. In the temporal domain, a fixed-duration sliding window (e.g., a 1-second window with a 0.5-second step) was used to segment the continuous data. In the spatial domain, regions of interest were extracted from the image data, such as using the RetinaFace model for face detection and cropping. The processed data was organized into a standardized set of evidence units, each containing a visual data block in a uniform format and a corresponding text description, forming standardized data output that can be directly used for feature extraction.

[0071] Through the above standardized processing flow, the system transforms the original multimodal heterogeneous data into standardized data that is time-aligned, of controllable quality, and in a unified format, providing a reliable data foundation for subsequent feature extraction and evidence chain construction.

[0072] S202: Perform feature extraction and low-rank projection based on standardized data to obtain a unified feature representation.

[0073] In one specific optional implementation, step S202, performing feature extraction and low-rank projection based on standardized data to obtain a unified feature representation includes:

[0074] Each modal data is encoded separately to obtain encoded features;

[0075] Based on the singular value energy distribution of the coding features, the rank of the projection matrix is ​​adaptively determined;

[0076] The encoded features are projected using a low-rank projection matrix of rank to generate a low-dimensional feature representation, thus obtaining a unified feature representation.

[0077] Specifically, this step aims to transform the standardized multimodal data into a unified low-dimensional feature representation, providing high-quality input features for subsequent causal inference. The implementation process includes the following three key steps:

[0078] Multimodal feature encoding is the primary step. For image modal data, the CLIP pre-trained VisionTransformer is used as the image encoder, taking a standardized 112×112 pixel image as input and outputting a 512-dimensional image feature vector. For text modal data, the CLIP text encoder is used to encode the cleaned text, generating a 512-dimensional text feature vector with the same dimension as the image features. For video data, a 3D convolutional neural network is used to extract spatiotemporal features, ensuring the capture of dynamic changes in the video. Encoders for all modalities output feature vectors of the same dimension, creating conditions for subsequent cross-modal alignment.

[0079] Dynamic low-rank projection is the core innovation of this step. To reduce computational complexity and prevent overfitting, low-rank projection processing is performed on the encoded high-dimensional features. Specifically, the covariance matrix of each modality feature matrix is ​​first calculated, and then singular value decomposition is performed to obtain the singular value spectrum. Set energy threshold =0.9, through the formula Adaptive determination of the rank of the projection matrix Next, construct the low-rank projection matrix. ,in and From the results of singular value decomposition, Finally, the high-dimensional features are projected onto the low-dimensional space to obtain... ,in These are the original encoded features. These are low-dimensional features after projection.

[0080] The unified feature representation is generated, and the final feature integration is completed. Multimodal evidence features under the same event are weighted and aggregated, and an attention mechanism is used to calculate the weight coefficients of each piece of evidence. Then, the document-level feature representation is obtained by weighted summation. Simultaneously, a vector index library based on FAISS was constructed, and a mapping table of "evidence → document" was established to support efficient similarity retrieval. The final output unified feature representation not only preserves the semantic information of the original data, but also significantly reduces the feature dimensionality through low-rank projection, thereby improving the efficiency of subsequent processing while ensuring feature quality.

[0081] Through the above processing flow, the system transforms multimodal data into a unified low-dimensional feature representation, providing high-quality feature input for subsequent causal evidence graph construction and causal reinforcement retrieval. At the same time, the dynamic low-rank mechanism effectively balances computational efficiency and feature fidelity.

[0082] S203: Construct a causal evidence graph based on unified feature representation to obtain a causal relationship network.

[0083] In one specific optional implementation, step S203, constructing a causal evidence graph based on a unified feature representation to obtain a causal relationship network, includes:

[0084] Calculate the directed causal strength between evidence nodes based on the unified feature representation;

[0085] Based on the directed causal strength, directed edges are constructed between evidence nodes to form a causal relationship network, and the overall causal confidence of the causal relationship network is calculated by path product.

[0086] Specifically, this step aims to transform the unified feature representation into a graph structure with explicit causal relationships, providing a theoretical foundation for subsequent causal reinforcement retrieval. The specific implementation process includes the following three key steps:

[0087] Directed causality strength calculation is the foundation for constructing a causal evidence graph. For each pair of evidence nodes in the evidence set... Based on its unified feature representation and The causal strength is calculated using a lightweight neural network. Specifically, the feature vectors of the two evidence nodes are first concatenated, then passed through a two-layer MLP network, and finally the output is mapped to the (0,1) interval using a sigmoid activation function. This process can be expressed by the following formula: ,in Indicates an MLP network. This represents the Sigmoid function. In this embodiment, this setting ensures that the weight of each directed edge has a clear probabilistic meaning, representing the causal strength from cause to effect.

[0088] Constructing a causal evidence graph is a crucial step in integrating discrete causal relationships into a structured network. A node set is formed using all evidence nodes {e_i}. The calculated directed causality strength Construct a directed graph by forming a set of directed edges A. Adjacency matrix It fully describes the strength of causal relationships between evidence nodes. During the construction process, time constraints are considered, allowing only nodes with earlier timestamps to point to nodes with later timestamps, ensuring the temporal consistency of causal relationships. For nodes with the same timestamp, bidirectional connections are established based on their semantic relevance.

[0089] The overall causal confidence calculation is a crucial step in the quantitative evaluation of the constructed causal evidence graph. On the set of paths P that satisfy the time constraint, the overall causal confidence of the graph is calculated using path product. The specific formula is as follows: The calculation process traverses all possible causal paths, measures the confidence of a path by multiplying the weights of its edges, and then averages the confidence of all paths to obtain the overall confidence assessment of the entire causal evidence graph. This metric will serve as an important reference in subsequent causal reinforcement retrieval.

[0090] Through the above processing flow, the system transforms discrete evidence nodes into a structured causal evidence graph. This not only establishes the relationships between evidence but, more importantly, establishes directional causal connections, laying a solid foundation for generating an interpretable and reliable chain of evidence. The output causal network... and its overall causal confidence level This will directly serve the subsequent causal reinforcement retrieval module, ensuring that the retrieval results are both semantically relevant and causally consistent.

[0091] S204: Perform causal reinforcement retrieval on the causal relationship network to obtain retrieval results.

[0092] In one specific optional implementation, a causal reinforcement search is performed on the causal relationship network to obtain search results including:

[0093] From the unified feature representation, obtain the unified feature representation of the query evidence and the candidate evidence, and obtain the query evidence features and candidate evidence features respectively;

[0094] Semantic relevance scores are calculated for the query evidence features and candidate evidence features to obtain the semantic relevance score;

[0095] The semantic relevance score is weighted and fused with the overall causal confidence score to generate the final search score. Based on this score, the candidate evidence is rearranged to obtain search results that include causal path information from the causal relationship network.

[0096] Specifically, this step aims to combine traditional semantic retrieval with causal reasoning to achieve evidence retrieval based on causal constraints. The specific implementation process includes the following three key steps:

[0097] Query and evidence feature acquisition are fundamental to the retrieval process. Upon receiving a user query, the same multimodal encoder and low-rank projection matrix as in step S202 are first used to transform the query content into a unified feature representation Zq. Simultaneously, the unified feature representations {z} of all candidate evidence are retrieved from the vector index database established in step S202. d This process ensures that the query and candidate evidence reside in the same feature space, providing a foundation for subsequent similarity calculations. The index is built based on FAISS or Milvus, supporting efficient nearest neighbor search and enabling rapid retrieval of a set of candidate evidence relevant to the query.

[0098] Preferably, the semantic relevance calculation uses cosine similarity to measure the degree of semantic association between the query and candidate evidence. The specific calculation process is as follows: , where · represents the vector dot product, This represents the L2 norm of the vector. The calculation produces a similarity score between 0 and 1, with higher scores indicating stronger semantic relevance. The system also sets a similarity threshold, retaining only candidate evidence with scores above the threshold for further processing, effectively filtering irrelevant evidence and improving retrieval efficiency.

[0099] It should be noted that causal reinforcement rearrangement and result generation are the core of this step. After obtaining the semantic relevance score, it is compared with the overall causal confidence calculated in step S203. Weighted fusion is performed to generate the final search score: .in To balance the parameters, the default setting is 0.7, which can be adjusted according to specific application scenarios. The system reorders candidate evidence in descending order based on the final score, ensuring that evidence with both high semantic relevance and strong causal support is returned first. The search results not only include the evidence content itself, but also related causal path information, including path composition, edge weight strength, and overall confidence level, providing users with a complete causal chain display.

[0100] Through the aforementioned causal reinforcement retrieval mechanism, the system outputs retrieval results that not only maintain the relevance advantage of traditional semantic retrieval, but also significantly improve the logical coherence and interpretability of the evidence through causal constraints, providing high-quality input evidence for subsequent generation control.

[0101] S205: Use the search results for restricted decoding and attention consistency control to obtain generated content.

[0102] In one specific optional implementation, based on the retrieval results, constrained decoding and attention consistency control are performed to obtain the generated content, which includes:

[0103] Based on the search results, a limited vocabulary list is determined, and the optional vocabulary list during the decoding process is restricted to the scope of the limited vocabulary list;

[0104] By using the KL divergence loss function, the attention distribution of the generative model during the generation process is aligned with the attention distribution of the retrieval module on the retrieval results;

[0105] Based on a restricted generative vocabulary and an attention-aligned generative model, restricted decoding is performed to generate content.

[0106] Specifically, this step aims to ensure, through a dual constraint mechanism, that the generated content is strictly based on retrieved evidence, preventing the model from generating illusions or making unauthorized citations. The implementation includes the following three key steps:

[0107] First, constructing a restricted genealogy is the primary measure to control the scope of generation. The system extracts all evidence-supporting keywords and phrases from the search results, including entity names, action descriptions, attribute features, etc., to construct a restricted genealogy V(E). In specific implementation, the retrieved multimodal evidence is first processed into text, extracting noun phrases, verb phrases, and adjective phrases; then, through lemmatization and deduplication, a basic vocabulary is formed; finally, based on word frequency and evidence relevance, only words highly relevant to the retrieved evidence are retained. This vocabulary serves as the vocabulary selection space in the generation process, ensuring that the model can only use evidence-supporting words for content generation.

[0108] Second, attention distribution alignment achieves controllability in the generation process through the KL divergence loss function. During the training phase, the system simultaneously computes two attention distributions: one generated by the retrieval module based on evidence importance. And the attention distribution generated by the generative model during the decoding process. Alignment is achieved by minimizing the KL divergence between the two distributions: In practice, at each decoding time step t, the attention weights of the generative model for all evidence are calculated and compared with the evidence importance weights given by the retrieval module. The parameters of the generative model are then adjusted through gradient backpropagation to ensure that its attention pattern is consistent with the evidence importance.

[0109] Third, the constrained decoding execution applies the constraints of the first two stages to the generation process. To prevent the model from over-reference or creating illusions, the generation stage imposes hard constraints on quotable words / fragments, allowing decoding only from the vocabulary derived from retrieved evidence or from the annotation space aligned to the evidence ROI / table unit. Simultaneously, attention consistency is used to penalize the "say B after seeing A" behavior, ensuring that the language model's attention distribution aligns with the retrieval module's attention distribution on evidence, thus truly implementing the "chain of evidence" in the text output. At each time step of decoding, the system strictly limits the candidate vocabulary to the constrained generated vocabulary. Within the range, evidence-supported cross-entropy loss: In practice, the output probability distribution of the standard Softmax layer is modified to avoid... The probability of words within the specified range is set to zero to ensure that the model can only select from words supported by evidence. At the same time, combined with a generative model trained with attention alignment, the generation process is ensured to be both constrained in word selection and consistent with the importance of evidence in terms of semantic focus.

[0110] Through these three control mechanisms, the system achieves the reliability and verifiability of the generated content, ensuring that each generated fragment is supported by corresponding evidence, significantly improving the credibility of the generated results, and providing technical support for building a credible multimodal evidence chain.

[0111] S206: Generate a structured chain of evidence record based on the generated content and search results, and use the structured chain of evidence record as traceability output information.

[0112] Specifically, this embodiment integrates the generated content with supporting evidence in a structured manner to form a complete, traceable, and verifiable chain of evidence. The specific implementation process includes the following three sections:

[0113] The first section, evidence-conclusion binding and association, forms the foundation for constructing a chain of evidence. This implementation establishes a "conclusion-evidence" correspondence by precisely binding the text content generated in step S205 with the evidence nodes retrieved in step S204. Specifically, each key argument in the generated content is labeled with its corresponding evidence node ID, forming a mapping relationship between arguments and evidence. Simultaneously, the specific role of each evidence node in the causal evidence graph is recorded, distinguishing it as either a "cause" node or an "effect" node, and its contribution score is recorded. For multimodal evidence, the system simultaneously records its source modality type (e.g., image, text, video) and its location information in the original data (e.g., timestamp, spatial coordinates), ensuring the traceability of the evidence.

[0114] The second section, metadata collection and recording, provides complete contextual information for the chain of evidence. The system automatically collects and records causal path scores. and search score These two quantitative indicators assess the quality of the evidence chain from the perspectives of causal consistency and semantic relevance, respectively. Simultaneously, complete timestamp information is recorded, including the time of evidence generation, retrieval execution time, and conclusion generation time, establishing a complete temporal chain. Regarding device metadata, sensor parameters (such as exposure time and gain settings), acquisition environment conditions (such as lighting and weather), and hardware device identifiers are recorded to provide a basis for subsequent data quality assessment. Furthermore, the system records the model version fingerprints used, including version information for the feature extraction model, causal graph construction model, and generation model, ensuring the reproducibility of the experiment.

[0115] The structured record generation and output integrates the aforementioned information into a standardized, traceable output. The system generates structured records conforming to the JSON Schema specification, including core fields such as decision conclusions, confidence levels, evidence chain lists, causal path scores, retrieval scores, timestamp sequences, device metadata, and model versions. In specific implementation, a hierarchical structure is used to organize the data: the top layer contains conclusive information and overall evaluation indicators, the middle layer details the composition and relationships of the evidence chain, and the bottom layer stores access links and metadata for the original data. This structured design meets both machine readability requirements and supports rapid location and understanding during manual review. The final output structured evidence chain records can serve as audit vouchers, decision-making basis, or input for subsequent analysis, achieving full-process traceability and verifiability.

[0116] Furthermore, to meet engineering review and compliance requirements, the conclusions are structurally bound to the chain of evidence: including the evidence nodes used, causal paths, scores, timestamps, sources, shooting / collection parameters, and model version fingerprints; simultaneously, a replayable object storage handle and visualization (evidence heatmap, path graph) are generated. This step elevates "interpretability" from a subjective description to a verifiable and reproducible machine-readable record.

[0117]

[0118] Where E is a list of evidence nodes, including the score and timestamp of each node; and Provides dual quantification of causality and correlation.

[0119] By implementing this step, the system integrates scattered multimodal evidence, causal reasoning processes, and generated conclusions into a unified structured record, forming a complete chain of evidence from raw data to final conclusions, providing reliable technical support for compliance requirements and audit needs in high-risk application scenarios.

[0120] In another optional implementation, the multimodal evidence chain control method further includes:

[0121] The generated content is then bound to the evidence nodes in the search results;

[0122] Record causal path scores, timestamp information, and device metadata;

[0123] Generate a structured record containing the model version fingerprint.

[0124] Specifically, this method aims to systematically integrate the results of previous steps to form a complete and traceable chain of evidence, which includes the following three stages:

[0125] First, the precise binding of evidence and conclusions is the primary task in constructing a credible chain of evidence. The system employs a mapping mechanism based on unique identifiers to establish a precise association between each assertion in the generated content and the specific evidence nodes supporting it. In practice, the generated content is first semantically parsed to identify key assertions and claims; then, attention weights are used to trace back to determine the main evidence nodes upon which each assertion depends; finally, an "assertion-evidence" mapping table is established, recording the assertion content, the corresponding evidence node ID, the support strength score (calculated based on attention weights), and the role (cause / effect) of the evidence node in the causal graph. For example, for the assertion "personnel identity verification passed" in the generated content, the system will bind it to the IR liveness detection evidence node and the RGB face matching evidence node, and record their respective contributions.

[0126] Second: The systematic recording of multi-dimensional metadata provides complete contextual information for the chain of evidence. The system collects and records metadata in three dimensions: firstly, causal quantification indicators, including causal path scores calculated based on path products. And a search score that combines semantics and causality These two indicators assess the quality of the evidence chain from the perspectives of causal consistency and comprehensive relevance, respectively. Secondly, there is time-series information, which establishes a complete timestamp sequence from evidence generation and retrieval to conclusion generation, ensuring traceability throughout the entire process. Finally, there is device and environmental metadata, which records sensor parameters (such as exposure time 6ms, gain 1.2), acquisition conditions, and hardware device identification in detail, providing a basis for data quality assessment and result reproduction.

[0127] Third: Standardized structured record generation ensures that the output format is both machine-readable and human-understandable. The system generates JSON format records conforming to a preset schema, whose structure includes: decision conclusions and confidence levels, detailed evidence chain composition, quantitative evaluation indicators, timestamp sequences, device environment parameters, and model version fingerprints. Regarding model version management, the system records complete version information for feature extraction models, causal graph construction models, and generated models, including model architecture, training data versions, and parameter fingerprints, ensuring complete reproducibility of experiments. This structured record supports rapid parsing and processing by automated systems and facilitates intuitive understanding during manual review, effectively meeting the audit and compliance requirements in engineering scenarios.

[0128] Through this systematic evidence chain generation process, this method achieves full transparency and verifiability from multimodal data input to final conclusion output, providing a solid technical guarantee for high-reliability application scenarios.

[0129] In another optional embodiment of this example, the multimodal evidence chain control method further includes training a causal relationship network, specifically including:

[0130] Construct a causal contrastive learning loss function;

[0131] Combine generation loss, attention consistency loss, and low-rank regularization loss;

[0132] End-to-end joint optimization training is performed using the standardized data.

[0133] Specifically, this step aims to achieve a synergistic improvement in causal reasoning, generative control, and computational efficiency through multi-objective joint optimization. The implementation includes the following three key steps:

[0134] The causal contrastive learning loss function construction focuses on enhancing the model's causal reasoning ability. The system constructs "cause-effect-noise" triplet training samples, where positive sample pairs contain evidence nodes with genuine causal relationships, and negative samples are obtained through random replacement. In specific implementation, the InfoNCE loss function is used: ,in Indicates causal characteristics. Indicates the characteristics of the result. Indicates negative sample features. For cosine similarity calculation, The temperature coefficient is used. This loss function, through a contrastive learning mechanism, enables the model to effectively distinguish between genuine causal relationships and spurious associations, thereby improving the accuracy of causal inference.

[0135] Multi-objective loss function ensemble achieves balanced optimization for different training objectives. The system organically combines four core loss functions: generation loss... Ensuring consistency between generated content and evidence, and the loss of attention consistency. Ensure the generation process aligns with the search focus, and avoid causal contrast loss. Improve causal reasoning ability, low-rank regularization loss Controlling model complexity. The total loss function is expressed as: ,in , , To balance hyperparameters, adjustments are made based on specific task requirements. This ensemble design allows the model to optimize multiple key performance metrics simultaneously during training, avoiding overfitting to a single objective.

[0136] End-to-end joint optimization training enables collaborative learning of parameters throughout the entire process. Standardized multimodal data is used as input, employing an alternating training strategy: first, the causal graph network parameters are fixed, optimizing the generation and retrieval modules; then, the generation and retrieval parameters are fixed, optimizing the causal inference module. The training process uses the AdamW optimizer with a cosine annealing scheduling strategy. In each training batch, the system simultaneously calculates four loss components, obtains the total loss through weighted summation, and then performs backpropagation. This end-to-end training approach ensures that each module can promote and co-evolve, ultimately forming a unified and self-consistent multimodal evidence chain processing system.

[0137] Through this complete training framework, this embodiment achieves a synergistic effect of global optimization while maintaining the professional capabilities of each module, providing a fundamental guarantee for the construction and generation of multimodal evidence chains.

[0138] In this embodiment, multimodal data, including image, video, and text data, is standardized. Feature extraction and low-rank projection are performed on the standardized data to obtain a unified feature representation. A causal evidence graph is constructed based on this unified feature representation, resulting in a causal relationship network. Causal reinforcement retrieval is performed on the causal relationship network to obtain retrieval results. Restricted decoding and attention consistency control are applied to the retrieval results to generate content. A structured evidence chain record is generated based on the generated content and retrieval results, and this record is used as traceable output information. By standardizing the multimodal data and applying a unified feature representation, the semantic gap and alignment challenges of heterogeneous data sources are effectively overcome, laying a high-quality data foundation for subsequent processing. Secondly, the introduction of a causal evidence graph and a causal reinforcement retrieval mechanism overcomes the limitations of traditional methods that rely solely on semantic relevance, ensuring that the retrieved evidence is both semantically relevant and causally coherent, fundamentally improving the logical rigor of the evidence chain. Furthermore, through restricted decoding and attention consistency control, the generation process is strictly constrained within the scope of evidence support, significantly reducing the risks of model illusion and unauthorized citation, and enhancing the reliability and credibility of the generated content. Finally, by generating a structured chain of evidence, the entire process from raw data to generated conclusions is traceable and verifiable, providing a complete machine-readable basis for audit review. This improves the quality and credibility of the chain of evidence, enhances the controllability of its generation, and improves resource utilization.

[0139] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0140] Figure 3 A schematic diagram of a multimodal evidence chain control device is shown, corresponding one-to-one with the multimodal evidence chain control method described in the above embodiments. For example... Figure 3 As shown, the multimodal evidence chain control device includes a data processing module 31, a feature extraction module 32, an evidence graph construction module 33, a causal retrieval module 34, a content generation module 35, and an evidence chain survival module 36. Detailed descriptions of each functional module are as follows:

[0141] Data processing module 31 is used to standardize multimodal data to obtain standardized data. Multimodal data includes image data, video data and text data.

[0142] Feature extraction module 32 is used to extract features and perform low-rank projection based on standardized data to obtain a unified feature representation;

[0143] Evidence graph construction module 33 is used to construct a causal evidence graph based on a unified feature representation to obtain a causal relationship network;

[0144] Causal retrieval module 34 is used to perform causal reinforcement retrieval on the causal relationship network and obtain retrieval results;

[0145] Content generation module 35 is used to perform restricted decoding and attention consistency control on the search results to generate content;

[0146] The evidence chain survival module 36 is used to generate structured evidence chain records based on the generated content and search results, and to use the structured evidence chain records as traceability output information.

[0147] Optionally, the feature extraction module 32 includes:

[0148] The coding unit is used to encode each modal data separately to obtain the coded features;

[0149] A computational unit is used to adaptively determine the rank of the projection matrix based on the singular value energy distribution of the encoded features;

[0150] The projection unit is used to project the encoded features using a low-rank projection matrix of rank to generate a low-dimensional feature representation, thus obtaining a unified feature representation.

[0151] Optionally, the evidence graph construction module 33 includes:

[0152] The causal strength calculation unit is used to calculate the directed causal strength between evidence nodes based on the unified feature representation;

[0153] The causal relationship network construction unit is used to construct directed edges between evidence nodes based on directed causal strength, forming a causal relationship network, and to calculate the overall causal confidence of the causal relationship network through path product.

[0154] Specific limitations regarding the multimodal evidence chain control device can be found in the limitations of the multimodal evidence chain control method described above, and will not be repeated here. Each module in the aforementioned multimodal evidence chain control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0155] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0156] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components connected to the memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0157] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0158] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as the program code of the multimodal evidence chain control method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0159] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run program code stored in the memory 41 or process data, for example, to run program code for a multimodal evidence chain control method.

[0160] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0161] This application also provides another embodiment, namely, providing a computer-readable storage medium storing an interface display program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the multimodal evidence chain control method described above.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0163] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for controlling a multimodal chain of evidence, characterized in that, include: The multimodal data is standardized to obtain standardized data, which includes image data, video data, and text data. Based on the standardized data, feature extraction and low-rank projection are performed to obtain a unified feature representation. The feature extraction and low-rank projection include: encoding each modal data to obtain encoded features, adaptively determining the rank of the projection matrix based on the singular value energy distribution of the encoded features, and projecting the encoded features using the low-rank projection matrix of the rank to generate the unified feature representation. Based on the unified feature representation, a causal evidence graph is dynamically constructed by calculating the directed causal strength between evidence nodes to obtain a causal relationship network; A causal reinforcement retrieval is performed on the causal relationship network to obtain retrieval results. The causal reinforcement retrieval includes: obtaining unified feature representations of query evidence and candidate evidence from the unified feature representation to obtain query evidence features and candidate evidence features respectively; calculating semantic relevance scores for the query evidence features and candidate evidence features to obtain semantic relevance scores; weighting and fusing the semantic relevance scores with the overall causal confidence score calculated based on the path product of the causal relationship network to generate a final retrieval score, and rearranging the candidate evidence accordingly to obtain retrieval results containing causal path information from the causal relationship network. The retrieved results are used to perform restricted decoding and attention consistency control to obtain generated content. The restricted decoding and attention consistency control include: dynamically determining a restricted generation vocabulary based on the retrieved results and limiting the optional vocabulary in the decoding process to the range of the restricted generation vocabulary; aligning the attention distribution of the generation model in the generation process with the attention distribution of the retrieval module on the retrieved results using the KL divergence loss function; and performing restricted decoding based on the restricted generation vocabulary and the attention-aligned generation model to generate the generated content. A structured chain of evidence record is generated based on the generated content and search results, and the structured chain of evidence record is used as traceability output information.

2. The multimodal evidence chain control method as described in claim 1, characterized in that, The construction of the causal evidence graph based on the unified feature representation to obtain the causal relationship network includes: Based on the unified feature representation, calculate the directed causal strength between evidence nodes; Based on the directed causal strength, directed edges are constructed between evidence nodes to form a causal relationship network, and the overall causal confidence of the causal relationship network is calculated through path product.

3. The multimodal evidence chain control method as described in claim 2, characterized in that, The causal reinforcement retrieval performed on the causal relationship network yields the following retrieval results: From the unified feature representation, obtain the unified feature representation of the query evidence and the candidate evidence, and obtain the query evidence feature and the candidate evidence feature respectively; Semantic relevance scores are calculated for the query evidence features and candidate evidence features to obtain the semantic relevance score; The semantic relevance score is weighted and fused with the overall causal confidence score to generate the final retrieval score. Based on this score, the candidate evidence is rearranged to obtain retrieval results that include causal path information from the causal relationship network.

4. A multimodal evidence chain control device, characterized in that, The multimodal evidence chain control method according to any one of claims 1 to 3, wherein the multimodal evidence chain control device comprises: The data processing module is used to standardize multimodal data to obtain standardized data, wherein the multimodal data includes image data, video data and text data; The feature extraction module is used to extract features and perform low-rank projection based on the standardized data to obtain a unified feature representation; The evidence graph construction module is used to construct a causal evidence graph based on the unified feature representation to obtain a causal relationship network; The causal retrieval module is used to perform causal reinforcement retrieval on the causal relationship network and obtain retrieval results; The content generation module is used to perform restricted decoding and attention consistency control on the search results to generate content; The evidence chain survival module is used to generate a structured evidence chain record based on the generated content and search results, and to use the structured evidence chain record as traceability output information.

5. The multimodal evidence chain control device as described in claim 4, characterized in that, The feature extraction module includes: The coding unit is used to encode each modal data separately to obtain the coded features; A computational unit is used to adaptively determine the rank of the projection matrix based on the singular value energy distribution of the encoded features; The projection unit is used to project the encoded features using the low-rank projection matrix of the rank to generate a low-dimensional feature representation and obtain a unified feature representation.

6. The multimodal evidence chain control device as described in claim 4, characterized in that, The evidence graph construction module includes: The causal strength calculation unit is used to calculate the directed causal strength between evidence nodes based on the unified feature representation. The causal relationship network construction unit is used to construct directed edges between evidence nodes based on the directed causal strength, form a causal relationship network, and calculate the overall causal confidence of the causal relationship network through path product.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal evidence chain control method as described in any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal evidence chain control method as described in any one of claims 1 to 3.

Images