Network fraud evidence solidification method and system for multimodal semantic consistency verification

Abstract

This application discloses a method and system for consolidating evidence of network fraud through multimodal semantic consistency verification, comprising: collecting and modally identifying multimodal electronic evidence and generating original data fingerprints; extracting semantic information from each modal of evidence across five dimensions: time, amount, identity, behavior description, and geographical location; constructing a cross-modal evidence semantic association graph with a heterogeneous graph structure including evidence nodes, element nodes, and relational edges, and establishing attribution edges and cross-modal association edges based on element categories using differentiated semantic similarity calculation methods; generating initial feature vectors for each node, inputting the semantic association graph into a graph attention network model for consistency inference, performing consistency, contradiction, and uncertainty classification on cross-modal association edges, and obtaining a global consistency score; and performing differentiated on-chain consolidation operations based on the score to generate consolidation certificates. This invention can automatically discover semantic contradictions between different modalities of evidence, improving the quality of electronic evidence consolidation.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence, blockchain, digital forensics, and multimodal technologies, specifically to a method and system for consolidating evidence of network fraud through multimodal semantic consistency verification. Background Technology

[0002] With the rapid development of internet technology and mobile communications, telecommunications fraud has become a major global security threat. Fraudsters utilize various technologies such as instant messaging, social networks, and virtual numbers to commit fraud, resulting in increasingly complex and diverse types of electronic evidence. This typically includes data in multiple modalities, such as chat logs, voice call recordings, video call clips, webpage screenshots, application interface screenshots, and bank transfer transaction records. In judicial practice, this multimodal electronic evidence must undergo standardized evidence preservation procedures to be securely fixed and preserved before it can be used as a legal basis for conviction and sentencing in the litigation process.

[0003] Existing electronic evidence preservation technologies primarily rely on blockchain-based evidence storage mechanisms. The core idea is to independently calculate a hash digest for each electronic evidence file and store it on the blockchain, leveraging the immutability of blockchain to ensure the data integrity of individual pieces of evidence. However, this approach only verifies whether a single piece of evidence has been tampered with at the data level, without addressing the semantic content-level correlation analysis and consistency verification between multiple pieces of evidence. In the same online fraud case, the key factual elements carried by multiple pieces of evidence from different modalities, including time, amount, identity of individuals, description of behavior, and geographical location, are not systematically verified at the semantic level in the current evidence preservation process.

[0004] Meanwhile, deep learning-driven multimodal forgery detection technology has made progress in scenarios such as deepfake identification and misinformation detection. However, the goal of these technologies is to determine the authenticity of a single content fragment, rather than to verify whether the semantics of multiple pieces of evidence from different modalities in the same case corroborate each other. Existing named entity recognition technology in the judicial field can extract entities and relationships from a single text modality, but it does not involve the construction of cross-modal evidence association structures. Currently, there is a lack of a technical solution that can map evidence elements from different modalities into a structured association graph and perform consistency reasoning based on this graph.

[0005] Furthermore, with the increasing sophistication of AI-generated forgery technology, attackers now possess the ability to simultaneously forge multiple modalities of evidence and ensure their consistency at the single-modal detection level. Suspects might use large language models to generate forged chat logs, use speech synthesis to forge call recordings, and use deepfake technology to forge video call clips. Each forged piece of evidence may pass authenticity verification in single-modal detection, but cross-modal semantic comparison may reveal semantic conflicts such as discrepancies in monetary values, contradictory timestamp logic, and inconsistent names of individuals. However, existing multimodal consistency detection technologies primarily target adversarial attacks on single samples, relying on intermodal feature distance comparisons. They do not employ message propagation reasoning mechanisms on a graph structure, nor do they address the verification of the causal relationships between multiple pieces of evidence.

[0006] Regarding evidence preservation strategies, existing blockchain evidence preservation platforms all adopt a uniform strategy, failing to adjust their on-chain behavior based on the semantic analysis results of the evidence content. When cross-modal semantic conflicts exist in the evidence set, existing technologies cannot automatically detect and mark such anomalies during the evidence preservation process. This may result in semantically contradictory evidence being preserved in a normal state, affecting the reliability of subsequent judicial proceedings. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides a method and system for consolidating evidence of online fraud through multimodal semantic consistency verification. By constructing a cross-modal evidence semantic consistency verification mechanism, semantic contradictions and logical conflicts between different modal evidences are automatically discovered during the evidence consolidation process, thereby improving the quality of consolidation of electronic evidence of online fraud and its judicial credibility.

[0008] One aspect of the present invention provides a method for consolidating evidence of network fraud through multimodal semantic consistency verification, comprising the following steps: Step S1 involves collecting and modally identifying multimodal electronic evidence in online fraud cases, and generating an original data fingerprint for each piece of electronic evidence, including a SHA-256 hash value and a file metadata digest.

[0009] Preferably, the modality recognition in step S1 adopts a dual verification method based on file header feature signature and content decoding verification: the first level is fast recognition based on file header feature signature, which determines the actual format type of the file by reading the first few bytes of the file and matching the known file format magic number; the second level is deep verification based on content decoding verification, which attempts to perform content decoding parsing on the file that has passed the first level of recognition according to the identified format type to verify whether the file content conforms to the encoding specification of the format; when the file extension is inconsistent with the actual detection result, the actual detection result shall prevail.

[0010] Step S2 involves extracting semantic information from each modal electronic evidence across five dimensions: time, amount, identity, behavior, and geolocation, to obtain a structured representation of the evidence elements.

[0011] Preferably, in step S2, for textual evidence, a named entity recognition model fine-tuned with judicial corpus is used to extract five dimensions of judicial elements. The named entity recognition model is based on a pre-trained language model for transfer learning, and the training corpus includes labeled chat log samples from fraud cases. For audio evidence, an automatic speech recognition model is used to transcribe the speech signal into a text sequence before performing named entity recognition processing, and the voiceprint feature vector of the audio is extracted simultaneously. For image evidence, optical character recognition technology is used to extract the text content in the image before performing named entity recognition processing. For chat screenshots, a chat interface parsing model is used to identify the sender and receiver of message bubbles, message timestamps, and message content areas. For transfer voucher screenshots, a financial voucher parsing model is used to extract the transaction amount, transaction time, payee information, payer information, and transaction status. For video evidence, an adaptive sampling strategy based on visual content change detection is used to extract keyframes and perform image semantic extraction processing separately. The audio track is separated and audio semantic extraction processing is performed to extract facial feature vector sequences. For structured data evidence, its structured fields are directly parsed, and the transaction time is standardized and uniformly converted to Coordinated Universal Time (UTC).

[0012] Furthermore, for text-based evidence, the named entity recognition model can identify non-standardized expressions unique to online fraud scenarios, such as "transfer 5k" meaning a transfer of 5000 yuan, and the implicit time-location association in "tomorrow afternoon 3 o'clock at the same place." For audio-based evidence, the automatic speech recognition model adopts an end-to-end encoder-decoder architecture, supports the recognition of Chinese dialects and colloquial expressions, and retains timestamp alignment information in the transcribed text, enabling each sentence segment in the transcribed text to be mapped to the precise time position in the original audio. For image-based evidence, an image content understanding model is also used to analyze the visual semantics of the image, extracting information such as the scene type, interface elements, and key visual objects presented in the image. For chat screenshots, the chat interface parsing model outputs the parsing results in a structured manner. For transfer voucher screenshots, the financial voucher parsing model can also identify additional fields such as transaction notes. For video-based evidence, the keyframe extraction adopts an adaptive sampling strategy based on visual content change detection, extracting representative keyframe sequences from the video stream, and the facial feature vector sequence is used for cross-modal comparison of person identity. For structured data evidence, including bank transfer records and payment platform transaction records, elements such as transaction time, transaction amount, identification of the transacting parties, transaction type, and transaction remarks are extracted. The standardization process can eliminate the interference of time zone differences between different data sources on subsequent consistency comparisons.

[0013] Step S3: Based on the structured evidence element representation obtained in step S2, construct a cross-modal evidence semantic association graph with a heterogeneous graph structure containing evidence nodes, element nodes, and relation edges; wherein, each evidence node establishes a belonging edge with each of its extracted element nodes; for element nodes of the same element category, a differentiated semantic similarity calculation method is adopted according to the element category, and when two element nodes of the same category come from evidence of different modalities and the semantic similarity exceeds the preset association threshold, a cross-modal association edge is established.

[0014] Furthermore, each piece of electronic evidence corresponds to an evidence node, whose attributes include the modality type of the evidence and the original data fingerprint. Each judicial element extracted from each piece of evidence corresponds to an element node, whose attributes include element category, element value, and source evidence identifier. Relationship edges include attribution edges connecting evidence nodes to their contained element nodes, and cross-modal association edges connecting semantically related element nodes in different pieces of evidence. The edge weight of the cross-modal association edge is the corresponding semantic similarity value.

[0015] Preferably, the differentiated semantic similarity calculation method in step S3 includes: for time elements, calculating the difference between standardized time values; for monetary elements, calculating the relative deviation rate between numerical values; for personal identity elements, calculating the edit distance similarity of text names and the cosine similarity of voiceprint or facial feature vectors respectively, and using the weighted average of the two as the semantic similarity of personal identity elements; for behavioral description elements, using a semantically fine-tuned text encoder to calculate the cosine similarity of sentence vectors; and for geographical location elements, calculating the geocoding distance after the place name is normalized.

[0016] Preferably, step S3 further includes introducing a cross-modal semantic alignment mechanism based on contrastive learning: a cross-modal evidence element alignment model is pre-trained, wherein the cross-modal evidence element alignment model adopts a dual-tower encoder architecture, the first encoding tower is a text modality encoder, and the second encoding tower is a general element encoder, which respectively encodes features of elements from different modalities; the contrastive learning loss function is used to shorten the distance between semantically equivalent cross-modal element pairs in the feature space, and to widen the distance between semantically inequivalent element pairs; the contrastive learning loss function adopts InfoNCE (Information Noise-Contrastive Estimation) loss, and for each positive sample pair in the training batch, i.e., semantically equivalent cross-modal element pairs, the cosine similarity of their feature vectors is calculated, and the negative logarithm is taken as the loss value after being normalized by softmax; the training data for the contrastive learning consists of evidence element pairs of historical fraud cases that have been labeled with consistency tags.

[0017] Preferably, step S3 further includes introducing a time-to-causal consistency constraint layer: establishing a partial order relationship of events based on the event timestamps extracted from different evidence; constructing a directed event time graph based on the partial order relationship; using the event nodes extracted from each modality as nodes in the directed graph; establishing directed edges according to the timestamp size relationship and causal logic relationship, with directed edges pointing from earlier events to later events; performing a topological consistency check on the directed event time graph using a directed acyclic graph, detecting whether there are cycles in the directed graph using a depth-first search or an in-degree-based topological sorting algorithm; if a time cycle is detected, it is marked as a time conflict and the time conflict information is included in the global consistency score. Further, the time-to-causal consistency constraint layer can verify whether the event sequences described by different modal evidence satisfy causal time constraints. For example, the event mentioned in a voice call about transferring money tomorrow should precede the actual transaction time in the transfer record, and the message negotiating the transfer amount in the chat log should precede the transaction completion time shown in the screenshot of the transfer voucher.

[0018] Step S4: Generate initial feature vectors for each evidence node and element node in the cross-modal evidence semantic association graph. Input the initial feature vectors of each node and their graph structure adjacency relationships into the graph attention network model. Aggregate the feature information of the evidence node to its associated element node through the belonging edge. Pass messages between different element nodes through cross-modal association edges. Aggregate information through a multi-layer message propagation mechanism. Perform consistency classification on each cross-modal association edge and output three states: consistent, contradictory, and uncertain. Perform weighted aggregation on the consistency classification results of all cross-modal association edges to obtain a global consistency score.

[0019] Furthermore, after multiple rounds of message propagation, the feature representation of each element node is fused with contextual semantic information from its associated nodes.

[0020] Preferably, the global consistency score in step S4 is calculated as follows: the sum of the weights of the cross-modal related edges that are determined to be in a consistent state is divided by the sum of the weights of all cross-modal related edges to obtain a global consistency score between zero and one; when the global consistency score is lower than a preset consistency threshold, it is determined that the evidence set composed of the multimodal electronic evidence collected in step S1 has a cross-modal semantic conflict.

[0021] Preferably, step S4 further includes performing conflict localization and source analysis when semantic conflicts are detected: extracting all cross-modal association edges that are determined to be in a contradictory state, tracing back the source evidence of the two element nodes connected to each contradictory edge, generating a conflict description record, the conflict description record including the conflict element category, the first evidence identifier and its element value, the second evidence identifier and its element value, and the contradiction confidence level; summarizing all conflict description records to form a semantic consistency verification report, the verification report being graded and sorted according to the severity of the conflict, wherein contradictions involving monetary elements and time elements are marked as high-severity conflicts, contradictions involving personal identity elements are marked as medium-severity conflicts, and contradictions involving behavioral description elements and geographical location elements are marked as low-severity conflicts.

[0022] Step S5: Perform differentiated on-chain operations for evidence preservation based on the global consistency score, write the evidence preservation data package into the blockchain, and generate evidence preservation certificates. The evidence preservation data package includes at least the original data fingerprint of each electronic evidence, the structured evidence element representation, the global consistency score, and the semantic consistency verification report.

[0023] Preferably, the differentiated evidence-based on-chain operation in step S5 includes: when the global consistency score is higher than or equal to a preset consistency threshold, marking the evidence set as having passed consistency verification, executing the standard evidence-based on-chain process, and storing the evidence-based data package on the blockchain; when the global consistency score is lower than the consistency threshold, marking the evidence set as having an abnormal consistency verification state, storing the evidence-based data package along with a detailed conflict description record on the blockchain, and sending a verification anomaly prompt to the evidence-based operator. The prompt information includes the specific location of the conflicting elements and a summary of the conflict content for manual review and confirmation.

[0024] Furthermore, the original data fingerprint of each piece of electronic evidence, its corresponding structured evidence elements, global consistency score, and semantic consistency verification report are packaged into a solidified evidence data package. The hash value of the entire solidified evidence data package is then calculated and written to the blockchain. After the solidified evidence is uploaded to the blockchain, a solidified evidence certificate is generated. The solidified evidence certificate includes the blockchain transaction hash, solidified evidence timestamp, evidence list and corresponding hash values, global consistency score, and verification conclusion. The solidified evidence certificate is encapsulated in a verifiable certificate format, supporting independent verification by third parties.

[0025] Another aspect of the present invention provides a network fraud evidence consolidation system for multimodal semantic consistency verification, comprising: The evidence collection and modality recognition module is used to collect and recognize multimodal electronic evidence in online fraud cases, and generate an original data fingerprint for each piece of electronic evidence, including a SHA-256 hash value and a file metadata digest. The multimodal semantic extraction module is used to extract semantic information from each modality of electronic evidence in five dimensions: time element, amount element, person identity element, behavior description element and geographical location element, to obtain a structured representation of evidence elements. The semantic association graph construction module is used to construct a cross-modal evidence semantic association graph with a heterogeneous graph structure, including evidence nodes, element nodes, and relation edges, based on the structured evidence element representation output by the multimodal semantic extraction module. For element nodes of the same element category, a differentiated semantic similarity calculation method is adopted according to the element category. When two element nodes of the same category come from evidence of different modalities and their semantic similarity exceeds a preset association threshold, a cross-modal association edge is established. Among them, each evidence node establishes a belonging edge between itself and each of its extracted element nodes. The consistency reasoning and conflict detection module is used to generate initial feature vectors for each evidence node and element node in the cross-modal evidence semantic association graph. It inputs the initial feature vectors of each node and their graph structure adjacency relationships into the graph attention network model. Through the belonging edges, it aggregates the feature information of the evidence nodes to their associated element nodes. It performs message passing between different element nodes through cross-modal association edges, and aggregates information through a multi-layer message propagation mechanism. It performs consistency classification on each cross-modal association edge, outputting three states: consistent, contradictory, and uncertain. Finally, it performs weighted aggregation of the consistency classification results of all cross-modal association edges to obtain a global consistency score. The evidence-on-chain module is used to perform differentiated evidence-on-chain operations based on the global consistency score, write the evidence data package into the blockchain, and generate evidence certificates. The evidence data package includes at least the original data fingerprint of each electronic evidence, the structured evidence element representation, the global consistency score, and the semantic consistency verification report.

[0026] This invention constructs a cross-modal evidence semantic association graph and performs graph neural network-driven consistency reasoning on this graph, achieving systematic verification of multimodal electronic evidence at the semantic level. This overcomes the shortcomings of existing piece-by-piece hash-based on-chain evidence consolidation technology, which only focuses on data integrity and neglects cross-evidence semantic consistency. By designing differentiated semantic similarity calculation methods for five types of elements of judicial evidence, this invention achieves accurate matching of different types of factual elements, effectively addressing the problem of heterogeneous expressions of the same factual element in different modal evidence. By introducing a temporal-causal consistency constraint layer, this invention achieves automatic verification of the temporal logical relationships between events described by multiple pieces of evidence, enabling the detection of carefully constructed cross-modal temporal contradictions. By executing differentiated evidence consolidation strategies based on consistency verification results, this invention automatically marks semantically abnormal evidence sets during the evidence consolidation process. This allows evidence consolidation personnel to receive timely alerts of cross-modal semantic conflicts and conduct manual review, thereby improving the quality and judicial credibility of electronic evidence consolidation in cases of online fraud. Attached Figure Description

[0027] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0028] Figure 1 This is a schematic diagram of the overall process of the network fraud evidence consolidation method for multimodal semantic consistency verification provided in the embodiments of the present invention.

[0029] Figure 2 This is a schematic diagram of the structure for multimodal semantic extraction and cross-modal evidence semantic association graph construction provided in the embodiments of the present invention.

[0030] Figure 3This is a schematic diagram of the data flow for consistent reasoning and conflict detection based on graph attention networks provided in an embodiment of the present invention.

[0031] Figure 4 This is a structural block diagram of the network fraud evidence consolidation system for multimodal semantic consistency verification provided in this embodiment of the invention.

[0032] Figure 5 This is a visualization diagram of event reasoning in the temporal-causal consistency constraint layer provided in the embodiments of the present invention.

[0033] Figure 6 This is an example diagram of cross-modal evidence semantic conflict tracing provided in an embodiment of the present invention.

[0034] Figure 7 These are heatmaps showing the conflict detection rate distribution among different modal combinations and conflict type distribution diagrams for high-conflict mode pairs provided in embodiments of the present invention. Detailed Implementation

[0035] To make the objectives and technical solutions of this invention clearer, the embodiments of this invention will be further described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this invention and should not be construed as limiting the scope of protection of this invention.

[0036] Example 1 like Figure 1 As shown, this embodiment provides a method for consolidating evidence in online fraud cases using multimodal semantic consistency verification. This method is applied to the consolidation of electronic evidence in online fraud cases. By performing cross-modal semantic consistency verification on multiple pieces of electronic evidence from different modalities, it automatically identifies potential semantic contradictions and logical conflicts between different modalities of evidence during the evidence consolidation process, thereby improving the quality of evidence consolidation and judicial credibility. The method includes the following steps: Step 1: Unified collection and modal recognition of multimodal electronic evidence in online fraud cases.

[0037] In this step, the system receives a set of multimodal electronic evidence to be secured. This set includes text-based evidence, image-based evidence, audio-based evidence, video-based evidence, and structured data evidence. Text-based evidence includes exported chat logs from instant messaging software, SMS records, and email content; image-based evidence includes chat screenshots, transfer receipt screenshots, webpage screenshots, and application interface screenshots; audio-based evidence includes recorded phone calls and voice messages; video-based evidence includes video call recordings and screen recordings; and structured data evidence includes bank transfer records, third-party payment platform transaction records, and other data stored in tabular or structured formats.

[0038] Automatic modality type identification is performed on each piece of input electronic evidence. This automatic identification classifies each piece of evidence into its corresponding modality category based on file format characteristics, MIME (Multipurpose Internet Mail Extensions) type identifiers, and content decoding verification. Specifically, the modality identification process consists of two levels of verification: the first level is rapid identification based on file header signature features. By reading the first few bytes of the file and matching them with known file format magic numbers, the actual format type of the file is determined. For example, the file header signature of a JPEG image file is 0xFFD8FF, the file header signature of a PNG image file is 89504E47, and the file header of a WAV audio file contains a RIFF identifier, etc. The second level is deep verification based on content decoding verification. For files identified through the first level, content decoding is attempted according to the identified format type to verify whether the file content conforms to the encoding specifications of that format. When the file extension is inconsistent with the actual detection results obtained from the first and second level verifications, the actual detection result prevails, thus effectively addressing adversarial scenarios where file extensions are maliciously tampered with. For example, an attacker might change the file extension of an audio file to an image file to try to circumvent certain types of evidence analysis. The aforementioned dual verification mechanism can correctly identify the true modality of the file.

[0039] Furthermore, an original data fingerprint is generated for each piece of electronic evidence. The original data fingerprint includes a SHA-256 hash value and a file metadata digest. The SHA-256 hash value is obtained by performing a SHA-256 hash operation on the complete binary content of the evidence file, serving as the unique identifier and integrity verification benchmark for that evidence file. The file metadata digest includes basic attribute information such as file size, creation time, last modification time, and file format type. The original data fingerprint is generated and recorded immediately upon the evidence entering the system, ensuring that all subsequent processing operations are based on the original evidence data that has been verified for integrity, providing the foundational data for subsequent evidence preservation and blockchain uploading.

[0040] Step two involves performing semantic information extraction for each modality of electronic evidence, targeting judicial elements, to obtain a structured representation of the evidence elements.

[0041] The judicial elements include five dimensions: time element, monetary element, personal identity element, behavioral description element, and geographical location element. The time element refers to time information such as the time of the event or the agreed-upon time in the evidence content; the monetary element refers to numerical information such as the monetary amount or the number of transfers; the personal identity element refers to identity information such as the name, title, and account identifier of the person involved; the behavioral description element refers to semantic information such as the actions and events described; and the geographical location element refers to location information such as the name and address of the place involved. Appropriate semantic extraction methods are used for different modalities of evidence, which are explained below.

[0042] For textual evidence, a Named Entity Recognition (NER) model fine-tuned with judicial corpus is used to extract the aforementioned five dimensions of judicial elements from text such as chat logs and SMS messages. The NER model is based on transfer learning from a pre-trained language model. Specifically, a pre-trained language model with a parameter size between 100 million and 300 million is selected as the base model, and fine-tuning is performed on this model using labeled judicial corpus. The training corpus includes labeled chat log samples from fraud cases, with a preferred data size between 5,000 and 20,000 labeled samples. The labeling method uses the BIO (Beginning-Inside-Outside) sequence labeling system, assigning corresponding entity category labels to time expressions, monetary amounts, names and titles, action descriptions, and location names in the text. The training dataset is divided into training, validation, and test sets in an 8:1:1 ratio. Fine-tuning training was implemented using the PyTorch deep learning framework on a GPU-equipped server. The batch size was optimally set between 16 and 32, the learning rate between 2e-5 and 5e-5, and the number of training epochs between 5 and 15. The AdamW optimizer was used, along with a linear learning rate warm-up and cosine decay strategy. The cross-entropy loss function was employed, and the F1 score was monitored on the validation set. An early stopping mechanism was triggered when the F1 score on the validation set stopped improving after 3 to 5 consecutive epochs. Through this fine-tuning training, the model was able to recognize non-standard expressions unique to online fraud scenarios. For example, it correctly identified the colloquial expression "transferred 5k" as "transferred 5000 yuan," and correctly extracted the implicit time and location association in "tomorrow afternoon 3 o'clock at the usual place" as independent time and location elements.

[0043] For audio evidence, an Automatic Speech Recognition (ASR) model is first used to transcribe the speech signal into a text sequence. Then, the transcribed text undergoes the aforementioned named entity recognition processing. The ASR model employs an end-to-end encoder-decoder architecture. Specifically, the encoder uses a hybrid structure of multi-layer convolutional neural networks and a self-attention mechanism to encode the Mel-spectral features of the input audio layer by layer, extracting high-level semantic representations. The decoder, based on an autoregressive mechanism, progressively generates the corresponding text sequence. This model supports the recognition of Mandarin Chinese and various Chinese dialects, and can handle speech-specific phenomena such as colloquial expressions, interjections, and repetitive expressions. Preferably, timestamp alignment information is retained in the transcribed text, that is, each sentence segment in the transcribed text records the start and end times of the original audio, enabling the subsequently extracted judicial elements to accurately trace back to the precise time position of the original audio, providing a precise time anchor for subsequent cross-modal temporal comparison. The speaker's acoustic features are extracted synchronously using a speaker embedding model based on a deep neural network. This model encodes the speaker's acoustic features in each speech segment into a fixed-dimensional feature vector, preferably 256-dimensional. These voiceprint feature vectors are used for cross-modal identity comparison in subsequent steps, specifically by calculating the cosine similarity of the speaker's voiceprint vectors in different audio segments to determine if they belong to the same speaker.

[0044] For image-based evidence, Optical Character Recognition (OCR) technology is first used to extract the text content from the image, followed by Named Entity Recognition (NAME) processing on the extracted text. The OCR processing must support the recognition of mixed Chinese and English text and be able to handle text recognition tasks with different resolutions, fonts, and backgrounds. Simultaneously, an image content understanding model is used to analyze the visual semantics of the image, extracting information such as the scene type, interface elements, and key visual objects. This image content understanding model is based on a pre-trained visual-language multimodal model and can generate structured semantic descriptions of the image content. For chat screenshots, a dedicated chat interface parsing model is used. This model identifies the sender and receiver of the message by detecting visual features such as the position, color, and arrangement direction of message bubbles. It also locates and identifies the timestamp and message content areas within the message bubbles, outputting the parsing results in a structured format. Each message record includes three fields: sender identifier, message content text, and message timestamp. For screenshots of transfer vouchers, a specialized financial voucher parsing model is used. This model identifies and extracts structured fields such as transaction amount, transaction time, payee information, payer information, and transaction status by locating key areas in the transaction voucher image. The output includes the text value of each field and its position coordinates in the image.

[0045] For video evidence, keyframe extraction is performed first. This keyframe extraction employs an adaptive sampling strategy based on visual content change detection. By calculating the visual difference metric between adjacent video frames, a scene change or content switch is determined when the difference metric exceeds a preset change detection threshold, and keyframes are extracted at the change points. The difference metric can be calculated using methods such as the mean square value of inter-frame pixel differences, the complementary value of inter-frame structural similarity indices, or the cosine distance of inter-frame depth features. This adaptive sampling strategy reduces the amount of redundant frames processed while ensuring no key information is missed. The sampling frequency is reduced for video segments with slow content changes and increased for video segments with rapid content changes. The extracted keyframes undergo the same semantic extraction processing as image evidence, including OCR text extraction, named entity recognition, and image content understanding. Simultaneously, the audio track of the video is separated, and the semantic extraction processing for audio evidence, including speech-to-text, named entity recognition, and voiceprint feature extraction, is performed on the separated audio. Furthermore, the facial feature vector sequence appearing in the video is extracted, and a face recognition model based on deep convolutional neural networks is used to extract a fixed-dimensional face embedding vector for the detected face region in each frame. The preferred feature vector dimension is 512 dimensions, which is used for cross-modal comparison of person identity.

[0046] For structured data evidence, including bank transfer records and payment platform transaction records, their structured fields are directly parsed. Specifically, based on the data storage format (such as CSV, JSON, XML, or database export format), the corresponding parser is used to extract the values ​​of each field. Extracted elements include transaction time, transaction amount, identifiers of the transacting parties, transaction type, and transaction remarks. Transaction time is standardized and uniformly converted to Coordinated Universal Time (UTC). Specifically, based on the time zone information marked on the data source or the system's default time zone, all time values ​​are converted to UTC timestamps to eliminate the interference of time zone differences between different data sources on subsequent cross-modal consistency comparisons. Transaction amounts are uniformly processed in currency; when different transaction records involve different currencies, they are converted according to the exchange rate corresponding to the transaction time and uniformly expressed in the same currency.

[0047] Step 3: Based on the modal evidence elements extracted in Step 2, construct a cross-modal evidence semantic association graph.

[0048] like Figure 2 As shown, the semantic association graph of evidence is a heterogeneous graph structure, denoted as G = (V, E), where V is the set of nodes and E is the set of edges. The node set V contains two types of nodes: evidence nodes and element nodes. Each piece of electronic evidence corresponds to one evidence node, whose node attributes include the modality type of the evidence, the original data fingerprint, and the evidence number. Each judicial element extracted from each piece of evidence corresponds to one element node, whose node attributes include element category (time, amount, person's identity, behavior description, or geographical location), element value (the extracted specific text or numerical value), and source evidence identifier. The edge set E contains two types of edges: evidence-element attribution edges and cross-modal association edges. Evidence-element attribution edges connect each evidence node to each element node extracted from that evidence, representing the attribution relationship. Cross-modal association edges connect semantically related element nodes in different pieces of evidence, representing cross-modal semantic association relationships.

[0049] The relationships between cross-modal feature nodes are established as follows: For feature nodes of the same feature category, the semantic similarity between their feature values ​​is calculated. The calculation method for semantic similarity varies depending on the feature category. Specifically: For time features, both time values ​​are converted to UTC timestamps and the difference is calculated. The time difference is defined as the absolute value of the difference between the two timestamps, in seconds. Then, a Gaussian kernel function is used to map the difference to a similarity value between zero and one, with the formula Sim_time = exp(-delta_t^2 / (2 * sigma_t^2)), where delta_t is the absolute value of the time difference, and sigma_t is the time tolerance parameter, preferably set between 60 and 600 seconds to control the tolerance window for time matching; For monetary features, the relative deviation rate between two monetary values ​​is calculated, with the formula Sim_amount = 1 -|a1 - a2| / max(a1, a1 and a2 are two monetary values, where the relative deviation rate is zero when it is negative and one when the two monetary values ​​are exactly equal. For the identity element, the edit distance similarity of the text name and the cosine similarity of the biometric vector are calculated separately. The text name similarity is calculated using Levenshtein edit distance normalization, and the biometric similarity is calculated using the cosine similarity of the voiceprint feature vector or the face feature vector. The final identity similarity is the weighted average of the text similarity and the biometric similarity. The weight of the text similarity is preferably set to 0. Between 0.2 and 0.4, the weight of biometric similarity is the corresponding complementary value. For behavioral description elements, a semantically fine-tuned text encoder is used to encode the two behavioral description texts into sentence vectors, and the cosine similarity of the sentence vectors is calculated as the semantic similarity of the behavioral description. For geographic location elements, the place names are first standardized by converting non-standard place names into standard administrative division names. Then, the corresponding latitude and longitude coordinates are obtained through a geocoding service, the geographic distance between the two coordinate points is calculated, and the distance is mapped to a similarity value through a Gaussian kernel function. The formula is Sim_geo = exp(-d_geo^2 / (2 * sigma_geo^2)), where d_geo is the geographic distance in kilometers, and sigma_geo is the geographic tolerance parameter, preferably set between 0.5 kilometers and 5 kilometers.

[0050] When two element nodes of the same category originate from evidence of different modalities and their semantic similarity exceeds a preset association threshold theta, a cross-modal association edge is established, with the edge weight being the corresponding semantic similarity value. The association threshold theta is preferably set between 0.5 and 0.8, and can be adjusted according to the balance requirements between association precision and recall in actual application scenarios.

[0051] Preferably, when constructing cross-modal association edges, a cross-modal semantic alignment mechanism based on contrastive learning is introduced to improve the matching ability of heterogeneous expressions of the same factual element in evidence from different modalities. Specifically, a cross-modal evidence element alignment model is pre-trained, and the model adopts a dual-tower encoder architecture. The first tower of the dual-tower encoder is a text modal encoder, which uses a pre-trained language model based on the Transformer architecture to encode textual element values ​​into fixed-dimensional feature vectors; the second tower is a general element encoder, which performs unified feature encoding on element values ​​from different modalities (including speech-transcribed text, OCR-extracted text, structured field values, etc.). The feature vectors output by the two encoders maintain the same dimension, preferably set to 128 or 256 dimensions. The training process employs a contrastive learning loss function, specifically the InfoNCE loss. For each positive sample pair (i.e., semantically equivalent cross-modal element pairs) in the training batch, the cosine similarity of its feature vectors is calculated, and the negative logarithm is taken as the loss value after softmax normalization. The formula is L = -log(exp(sim(z_i, z_j) / tau) / sum_k(exp(sim(z_i, z_k) / tau))), where z_i and z_j are the two feature vectors of the positive sample pair, z_k is the feature vector of all samples in the batch, sim is the cosine similarity function, and tau is a temperature parameter, preferably set between 0.05 and 0.1. By minimizing this loss function, the distance between semantically equivalent cross-modal element pairs in the feature space is shortened, while the distance between semantically inequivalent element pairs is widened. The training data for the contrastive learning consists of historical fraud case evidence element pairs labeled with consistency tags. The training data size is preferably between 10,000 and 50,000 pairs, containing an equal number of positive and negative sample pairs. Training was performed using the PyTorch framework, with the learning rate preferably set between 1e-4 and 5e-4, the batch size preferably set between 64 and 256, and the number of training epochs preferably set between 20 and 50. The Adam optimizer was employed. This alignment model effectively addresses the problem of heterogeneous representations of the same factual element in different modalities of evidence, such as the semantic equivalence between colloquial monetary expressions in speech-transcribed texts and precise numerical values ​​in bank transfer records.

[0052] Furthermore, this step introduces a temporal-causal consistency constraint layer to verify the temporal logical relationships between events described by multimodal evidence. Specifically, during the construction of cross-modal association edges, a temporal logic verification mechanism is introduced for time-related element nodes. First, based on the event timestamps extracted from each piece of evidence, a partial temporal order relationship is established. This partial temporal order relationship determines the chronological order of events based on the behavioral events described in different pieces of evidence and their corresponding time elements. For example, the message event of negotiating a transfer amount in a chat log should precede the transaction completion event shown in the screenshot of the transfer voucher, and the event of transferring money tomorrow mentioned in a voice call should precede the actual transaction event in the transfer record. Second, based on the above partial temporal order relationship, a directed event temporal graph is constructed, with the event nodes extracted from each modality as nodes in the directed graph. Directed edges are established according to the timestamp size relationship and causal logical relationship, with the directed edges pointing from earlier events to later events. Then, a topological consistency check of the directed acyclic graph (DAG) is performed on the event time-series directed graph. This topological consistency check is implemented using a depth-first search or an in-degree-based topological sorting algorithm to detect the existence of cycles in the directed graph. If a time-series cycle is detected, it indicates a temporal contradiction in the event sequences described by multiple pieces of evidence; that is, event A is described as occurring before event B in one piece of evidence, but event B is described as occurring before event A in another piece of evidence, constituting a causal violation. The event nodes involved in the time-series cycle and their corresponding evidence are marked as time-series conflicts, and this conflict information is included in the consistency score of subsequent steps.

[0053] like Figure 5As shown, the temporal-causal consistency constraint layer constructs a directed event temporal graph and performs a directed acyclic graph topological consistency check to discover temporal contradictions in cross-modal evidence. The directed event temporal graph contains event nodes from different modalities, where rectangular nodes represent text modal sources, elliptical nodes represent audio modal sources, diamond nodes represent image modal sources, hexagonal nodes represent structured data modal sources, and octagonal nodes represent video modal sources. Specifically, event t1 is the negotiation of transfer amount in the chat log, event t2 is the confirmation of transfer intention in the voice call, event t3 is the transaction completion event in the transfer voucher, event t4 is the fund arrival event in the transfer transaction record, and event t5 is the discussion of subsequent operations in the video call. Normal temporal edges are represented by solid arrows indicating causal relationships (t1→t2→t3→t4→t5), while conflicting edges are marked with dashed arrows (t3→t1), indicating that the transaction completion time is earlier than the negotiation time. The temporal loop detection region encloses three nodes t1, t2, and t3, forming a loop t1→t2→t3→t1, constituting a causal violation. The timeline below visually displays the actual timestamp positions of each event, clearly showing the time inversion relationship between t1 and t3. This temporal conflict is identified and marked as a temporal conflict using a depth-first search or topological sorting algorithm, providing a basis for subsequent evidence credibility assessment.

[0054] Step four: Perform consistency reasoning and conflict detection on the cross-modal evidence semantic association graph constructed in step three, and generate a semantic consistency verification report.

[0055] like Figure 3As shown, the consistent reasoning is implemented using a Graph Attention Network (GAT) model. First, initial feature vectors are generated for each evidence node and feature node in the cross-modal evidence semantic association graph. The initial feature vector of a feature node is composed of the feature semantic vector encoded by the cross-modal evidence feature alignment model; the initial feature vector of an evidence node is composed of the one-hot encoding of the evidence's modality type and the hash prefix encoding of the original data fingerprint. Then, the initial feature vectors of each node and their graph structure adjacency relationships are input into the GAT model. The GAT model contains multiple layers of graph attention, preferably 2 to 4 layers. Each graph attention layer calculates the attention weights between nodes through a multi-head attention mechanism, achieving message propagation and information aggregation. Specifically, for any edge (i, j) in the graph, the attention coefficient of node i to node j is calculated as follows: First, the feature vectors of node i and node j are linearly transformed respectively. Then, the two transformed vectors are concatenated and input into a single-layer feedforward neural network. The original attention score is obtained by passing through the LeakyReLU activation function. Then, the attention scores of all neighboring nodes of node i are normalized by the softmax function to obtain the final attention coefficient alpha_ij. The updated feature of each node is a weighted aggregation of the features of all its neighboring nodes, with the weights being the corresponding attention coefficients. Preferably, each graph attention layer adopts a multi-head attention mechanism, setting 4 to 8 independent attention heads, and the outputs of each head are fused by concatenation or averaging.

[0056] The graph attention network model is trained using supervised learning. The training data consists of semantic association graphs of historical fraud case evidence labeled with consistency tags. The labeling categories include consistent, contradictory, and uncertain states. The training data size is preferably between 500 and 2000 labeled graph samples. Model training is implemented using the PyTorch framework and its graph neural network extension library, executed on a server equipped with GPUs. The input feature dimension is consistent with the output dimension of the feature encoder. The hidden layer dimension of the graph attention layer is preferably set between 64 and 256, and the output layer uses a three-class softmax layer. The loss function is a weighted cross-entropy loss function, assigning higher class weights to contradictory categories to address the sample imbalance problem. The weight of contradictory categories is preferably set to 2 to 3 times the weight of consistent categories. The optimizer is the Adam optimizer, with a learning rate preferably set between 1e-3 and 5e-3, a batch size preferably set between 16 and 64, and the number of training epochs preferably set between 50 and 200. A cosine annealing learning rate decay strategy is adopted, and the macro-average F1 score of consistent classification is monitored on the validation set. An early stopping mechanism is triggered when the validation set performance no longer improves after 10 to 20 consecutive rounds. For regularization, a Dropout layer is added between the graph attention layers, with the dropout rate preferably set between 0.3 and 0.5. At the same time, an L2 regularization term is added to the loss function, with the weight decay coefficient preferably set between 1e-4 and 1e-3.

[0057] After multiple rounds of message propagation, the graph attention network aggregates the feature information of evidence nodes to their associated feature nodes through belonging edges. Message passing between different feature nodes is achieved through cross-modal association edges, ensuring that the feature representation of each feature node incorporates contextual semantic information from its associated nodes. Based on this, consistency classification is performed on each cross-modal association edge. Specifically, for each cross-modal association edge (i, j), the updated feature vectors of the two feature nodes connected by the edge are concatenated and input into an edge-level classifier. This edge-level classifier is a two-layer fully connected neural network with a hidden layer dimension preferably between 64 and 128, a ReLU activation function, and a three-class softmax output layer, outputting the probability distributions of three states: consistent, contradictory, and uncertain. The state with the highest probability is taken as the consistency classification result for that edge, and the corresponding probability value is used as the classification confidence.

[0058] Based on edge-level consistency classification, a global consistency score is performed. This global consistency score is obtained by weighted aggregation of the consistency classification results of all cross-modal association edges in the evidence semantic association graph. Specifically, the global consistency score formula is: S_global = sum(w_e * I(e)) / sum(w_e), where the summation range is all cross-modal association edges e, w_e is the weight of edge e (i.e., semantic similarity value), and I(e) is an indicator function that takes a value of one when edge e is determined to be in a consistent state, and a value of zero otherwise. S_global is a global consistency score between zero and one; the closer the score is to one, the higher the cross-modal semantic consistency of the evidence set. When S_global is lower than a preset consistency threshold T, the evidence set is determined to have a cross-modal semantic conflict. The consistency threshold T is preferably set between 0.6 and 0.8, and can be adjusted according to the false positive tolerance of the specific application scenario.

[0059] Furthermore, the verification results of the time-series-causal consistency constraint layer in step three are incorporated into the global consistency score. Specifically, when a time-series loop is detected in the event-series directed graph, a penalty term is applied to the global consistency score. The modified global consistency score formula is: S_final = S_global * (1 - lambda * N_conflict / N_total), where N_conflict is the number of detected time-series conflict event pairs, N_total is the total number of event pairs in the event-series directed graph, and lambda is the time-series conflict penalty coefficient, preferably set between 0.1 and 0.3.

[0060] Upon detecting semantic conflicts, conflict localization and source analysis are performed. Specifically, all cross-modal association edges identified as contradictory and all event pairs marked as temporal conflicts are extracted. For each contradictory edge, the source evidence of the two connected element nodes is traced back to generate a conflict description record. The conflict description record includes the conflict element category, the first evidence identifier and its element value, the second evidence identifier and its element value, and the contradiction confidence level. For each temporal conflict event pair, the event description, the corresponding timestamp value, and the source evidence identifier are recorded. All conflict description records are summarized to form a semantic consistency verification report. The verification report is graded and sorted according to the severity of the conflict: conflicts involving monetary and temporal elements are marked as high-severity conflicts, which directly involve the core criminal facts of fraud cases; conflicts involving personal identity elements are marked as medium-severity conflicts, which may affect the identification of criminal suspects; conflicts involving behavioral description and geographical location elements are marked as low-severity conflicts, which, although not directly affecting the determination of core facts, can provide a reference for case analysis. Temporal causal conflicts are marked with corresponding severity levels according to the element categories involved. In terms of evaluation metrics, the accuracy, precision, recall, and F1 score of edge-level consistency classification are used as evaluation metrics for model performance, while the consistency rate between the global consistency score and the consistency conclusion of human annotation is used as an end-to-end evaluation metric.

[0061] like Figure 6 As shown, this invention can automatically detect semantic contradictions between different modal evidence through cross-modal evidence semantic consistency verification. The figure illustrates three typical examples of cross-modal semantic conflicts. The first example is a monetary contradiction: the chat log shows a transfer amount of 50,000 yuan, while the transfer voucher shows a transaction amount of 45,000 yuan. The conflict is marked as 50,000 ≠ 45,000, with a confidence level of 0.91, indicating a numerical discrepancy between the transfer amount mentioned in the chat log and the actual transaction amount shown on the transfer voucher. The second example is a temporal contradiction: the chat log shows a negotiated transfer record with a message time of 15:30, while the transfer voucher shows a transaction completion record with a transaction time of 15:02. This is marked as a temporal causal violation, meaning the negotiation time is later than the transaction time, indicating that the negotiated transfer record was actually recorded after the transaction was completed, constituting a temporal logical contradiction. The third set of conflict examples involves identity discrepancies. The cosine similarity between the speaker's voiceprint feature A extracted from the call recording and the facial feature B extracted from the video call is only 0.42, which is below the threshold for associating identities. This indicates that the speakers considered to be the same person in the two pieces of evidence may not be the same individual. The summary of conflict severity levels shows that monetary and temporal discrepancies are marked as high-severity conflicts, while identity discrepancies are marked as medium-severity conflicts.

[0062] Step 5: Based on the semantic consistency verification result of Step 4, perform the evidence consolidation and on-chain operation.

[0063] In this step, the original data fingerprint of each piece of electronic evidence, its corresponding structured evidence elements, global consistency score, and semantic consistency verification report are packaged to generate a solidified evidence data package. The solidified evidence data package uses a unified data encapsulation format and includes the following components: an evidence list, listing the filenames, modal types, and SHA-256 hash values ​​of all electronic evidence involved in the solidification process; a structured element summary table, recording the five types of judicial elements extracted from each piece of evidence and their specific values ​​in a unified key-value pair format; a global consistency score and the cross-modal correlation edge statistics on which the score calculation is based; and a semantic consistency verification report, containing all conflict description records and their severity levels. The SHA-256 hash value of the entire solidified evidence data package is calculated to obtain a complete summary of the solidified evidence data package, which is then written to the blockchain along with the solidified evidence data package.

[0064] The evidence-on-chain operation executes differentiated evidence-on-chain strategies based on the consistency verification results. When the global consistency score S_final is higher than or equal to the consistency threshold T, the evidence set is marked as having passed consistency verification, and the standard evidence-on-chain process is executed, storing the evidence-on-chain data package on the blockchain. The standard evidence-on-chain process writes the hash value and key summary information of the evidence-on-chain data package into the blockchain's evidence-on-chain contract, generating a corresponding blockchain transaction record. When the global consistency score S_final is lower than the consistency threshold T, the evidence set is marked as having an abnormal consistency verification state. In the abnormal state, the evidence-on-chain data package, along with a detailed conflict description record, is stored on the blockchain. An abnormal flag field is appended to the blockchain transaction record, recording summary information such as conflict type, conflict quantity, and highest severity level. Simultaneously, the system sends a verification anomaly prompt to the evidence-on-chain operator. This prompt includes the specific location of conflict elements and a summary of conflict content, indicating which elements between which pieces of evidence have semantic contradictions, for manual review and confirmation. After manual review, the operator can choose to confirm the evidence-on-chain process or mark it as pending investigation. The corresponding review operation record is also written to the blockchain to ensure the completeness and traceability of the evidence-on-chain process.

[0065] After the evidence is uploaded to the blockchain, an evidence certificate is generated. This certificate includes the blockchain transaction hash, evidence timestamp, evidence list and corresponding hash values, global consistency score, and verification conclusion. The verification conclusion is either a successful consistency check or a consistency check failure; if it is a failure, a conflict digest is attached. Preferably, the evidence certificate is encapsulated in the W3C Verifiable Credentials (VC) format, containing standard fields such as issuer identifier, holder identifier, certificate type, certificate validity period, and digital signature, supporting independent third-party verification. Third-party verification institutions can trace back the complete content of the evidence data packet through the blockchain transaction hash and verify the integrity of the data packet by recalculating the SHA-256 hash value, thereby independently confirming the authenticity of the evidence certificate and the immutability of the evidence data.

[0066] Example 2 In a preferred embodiment of the present invention, the method for constructing a cross-modal evidence semantic association graph is further refined, focusing on the differentiated semantic similarity calculation method for five types of judicial elements, the network architecture and training strategy of the cross-modal semantic alignment model based on contrastive learning, and the complete technical process of heterogeneous node feature encoding and association edge establishment in the semantic association graph.

[0067] For cross-modal semantic similarity calculation of time elements, this implementation method combines multi-granularity time expression normalization with Gaussian kernel similarity mapping. First, time expressions from different modalities are normalized. Textual evidence may contain absolute time expressions (e.g., "March 15, 2025, 2:30 PM") and relative time expressions (e.g., "yesterday," "tomorrow afternoon," "last Friday"). For relative time expressions, parsing is required in conjunction with the creation time of the evidence file or absolute time anchors appearing in the context, converting them into absolute time representations. Audio evidence, after speech transcription, may also contain colloquial time expressions (e.g., "the day after tomorrow," "the day after that," "the beginning of next month"), requiring relative time parsing in conjunction with the recording timestamp of the call recording. Time information in image evidence usually originates from the timestamp area shown in the screenshot, and after OCR extraction, it becomes a highly standardized time string. Time fields in structured data evidence are usually stored in a standard time format and can be directly parsed. All time expressions are normalized and then uniformly converted to UTC timestamps in seconds. The semantic similarity between two time element nodes is calculated using a Gaussian kernel function, with the formula Sim_time = exp(-delta_t^2 / (2 * sigma_t^2)), where delta_t is the absolute value of the difference between the two UTC timestamps, and sigma_t is the time tolerance parameter. The value of the time tolerance parameter needs to be adaptively adjusted according to the time granularity of the event described by the evidence: when both time elements are accurate to the minute level, sigma_t is preferably set between 60 and 300 seconds; when at least one time element is accurate only to the hour or date level, sigma_t is preferably set between 1800 and 7200 seconds. The time granularity is determined by the parsing precision identifier recorded during the time representation normalization process, which records the smallest time unit that the original time representation can be parsed. Through this adaptive time tolerance mechanism, a reasonable correlation can be established between time representations of different precisions, avoiding the failure to associate semantically equivalent time elements due to differences in time precision.

[0068] For cross-modal semantic similarity calculation of monetary amounts, this implementation adopts a multi-level monetary matching strategy. The expression of monetary amounts varies significantly across different modalities of evidence: text chat logs may contain various expressions such as "5000," "five thousand," "5k," and "50 sheets"; voice-transcribed text may contain colloquial expressions such as "five thousand yuan" or "five thousand yuan in total"; image transfer vouchers typically display values ​​accurate to the cent (e.g., "5,000.00 yuan"); and structured transaction records are recorded in a standard numerical format. First, all monetary expressions are normalized by converting Chinese numerals to Arabic numerals, recognizing and processing colloquial unit abbreviations (e.g., "k" for thousand and "w" for ten thousand), and uniformly representing them in the smallest monetary unit (cent). Second, when different monetary records involve different currencies, they are converted according to the exchange rate corresponding to the transaction time, and uniformly represented in the same currency. The formula for calculating the amount similarity is Sim_amount = 1 - |a1 - a2| / max(a1,a2), where a1 and a2 are two normalized amount values. The similarity is one when the two amount values ​​are equal, and approaches zero when the relative deviation exceeds a certain proportion. Furthermore, this implementation introduces a transaction-based tolerance mechanism for amount matching: when an amount value appearing in a single piece of evidence matches the sum of multiple amounts in another piece of evidence (e.g., a chat log mentions "a total of 10,000 was transferred," while the transaction log shows two transfers of 5,000 each), an amount association is established, and the similarity is calculated based on the total amount deviation rate. The maximum number of transactions to be split for transaction matching is preferably limited to between 2 and 5 to avoid excessive splitting leading to false associations.

[0069] For cross-modal semantic similarity calculation of personal identity elements, this implementation adopts a weighted fusion method of text identity similarity and biometric similarity. The calculation of text identity similarity covers the following scenarios: complete name matching, partial name matching (e.g., surname matching only or first name matching only), name-title mapping matching (e.g., the association between "General Manager Zhang" and "Zhang San"), and account identifier matching. The text similarity of names is calculated using Levenshtein edit distance normalization, with the formula Sim_text = 1 - edit_distance(s1, s2) / max(len(s1), len(s2)), where s1 and s2 are two name strings, edit_distance is the edit distance function, and len is the string length function. For title-name mapping matching, a title pattern library is maintained, recording common title composition rules (e.g., "surname + title" "surname + kinship"), etc. The surname component in the title is extracted through pattern matching and then compared with the name. The calculation of biometric similarity includes the cosine similarity of the voiceprint feature vector and the cosine similarity of the face feature vector. The cosine similarity formula is Sim_bio = (v1 dot v2) / (|v1| * |v2|), where v1 and v2 are two feature vectors, dot represents the vector dot product operation, and |v| represents the L2 norm of the vector. When both biometric similarities are available (i.e., the same person has both voiceprint and facial features for comparison), the larger value is taken as the final biometric similarity. The final similarity of the person's identity is the weighted average of the text similarity and the biometric similarity, with the formula Sim_person = alpha * Sim_text + (1 - alpha) * Sim_bio, where alpha is the text weight coefficient, preferably set between 0.2 and 0.4. When only text information is available and no biometric information is available, alpha is set to one, meaning it relies entirely on text similarity. When only biometric information is available and no text information is available, alpha is set to zero, meaning it relies entirely on biometric similarity.

[0070] For cross-modal semantic similarity calculation of behavioral description elements, this implementation uses a semantically fine-tuned text encoder for sentence-level semantic representation and comparison. The text encoder is fine-tuned based on a pre-trained language model for sentence-level semantic task fine-tuning, specifically using a contrastive learning framework for sentence embedding training. The encoder's input is behavioral description text (e.g., "asking the victim to transfer money to a designated account" or "the suspect induces the victim to install remote control software via telephone"), and the output is a fixed-dimensional sentence vector representation, preferably set to 256 or 384 dimensions. The fine-tuning training dataset consists of pairs of behavioral description texts. Positive sample pairs are semantically equivalent behavioral descriptions but with different expressions (e.g., "asking the victim to transfer money" versus "asking the victim to remit money to a designated account"), while negative sample pairs are semantically inequivalent behavioral descriptions. The training data size is preferably between 5000 and 20000 pairs. The semantic similarity of the behavior description is the cosine similarity between the two sentence vectors, and the formula is Sim_action = (u dot v) / (|u| * |v|), where u and v are the sentence vectors of the two behavior description texts, respectively.

[0071] For cross-modal semantic similarity calculation of geographic location elements, this implementation first performs hierarchical normalization processing on place names. Geographical location expressions in different modal evidence may cover multiple levels of precision: provincial-level administrative divisions, municipal-level administrative divisions, district / county-level administrative divisions, street address levels, and informal place names (such as "old place" or "downstairs from the company"). For geographic expressions containing standard administrative division names, normalization mapping is performed through a place name dictionary to convert abbreviations and acronyms into standard full names. For informal place name expressions, semantic parsing is performed in conjunction with contextual information to attempt to associate them with specific geographic locations. After standardization, place names are converted to latitude and longitude coordinates using a geocoding service. The spherical distance between two coordinate points is calculated using the formula: d_geo = R * arccos(sin(lat1) * sin(lat2) + cos(lat1) * cos(lat2) * cos(lon1 - lon2)), where R is the Earth's radius (approximately 6371 km), and lat1, lon1 and lat2, lon2 are the latitude and longitude (in radians) of the two coordinate points, respectively. Geographic similarity is obtained through Gaussian kernel mapping using the formula: Sim_geo = exp(-d_geo^2 / (2 * sigma_geo^2)), where sigma_geo is the geographic tolerance parameter, preferably set between 0.5 km and 5 km. When geocoding is unavailable (e.g., when informal place names cannot be parsed), the calculation degenerates to the edit distance similarity of the place name text.

[0072] Based on the above five types of element similarity calculations, this embodiment further elaborates on the detailed network architecture and training strategy of the cross-modal semantic alignment model based on contrastive learning. The cross-modal evidence element alignment model adopts a dual-tower encoder architecture, with two encoding towers corresponding to element encoding paths from different modalities. The first encoding tower is a text modality encoder, whose base model uses a pre-trained language model with 12 Transformer encoding layers, each containing 12 attention heads, and a hidden layer dimension of 768. Above the output layer of the base model, a fully connected projection layer is added to project the 768-dimensional feature representation onto the target embedding space. The projected feature dimension is preferably set to 256 dimensions. The projection layer uses a combination of layer normalization and ReLU activation function. The second encoding tower is a general element encoder that uniformly encodes element text from different sources such as speech transcription, OCR extraction, and structured fields. Its architecture is the same as the first encoding tower, but its parameters are trained independently to adapt to the different distribution characteristics of text from different sources. The feature vectors output by each encoding tower are L2 normalized and used for similarity calculation.

[0073] Contrastive learning training employs the InfoNCE loss function, specifically in the form L_InfoNCE = -(1 / N) * sum_i(log(exp(sim(z_i^a, z_i^b) / tau) / sum_j(exp(sim(z_i^a, z_j^b) / tau)))), where N is the number of positive sample pairs in the batch, z_i^a and z_i^b are the feature vectors of the i-th positive sample pair after being encoded by two encoding towers, sim is the cosine similarity function, tau is the temperature parameter, and sum_j covers all samples in the batch. The temperature parameter tau controls the smoothness of the similarity distribution. A smaller tau value results in a sharper distribution, strengthening the model's ability to distinguish difficult negative samples, but reducing training stability; a larger tau value results in a smoother distribution, leading to more stable training but reducing discrimination accuracy. The optimal range for tau is 0.05 to 0.1, with an initial value of 0.07, which is kept constant during training. Regarding the construction of training data, positive sample pairs consist of cross-modal feature pairs with known semantic equivalence, while negative sample pairs are automatically generated through intra-batch random pairing. To improve the model's ability to distinguish difficult samples, a difficult negative sample mining strategy is further introduced: in each training batch, the negative sample pair with the highest predicted similarity is assigned a higher sampling weight, making the model pay more attention to easily confused non-equivalent feature pairs during training. The training batch size is preferably set between 128 and 256; a larger batch size can provide richer intra-batch negative samples, which helps improve the effect of contrastive learning. A cosine annealing strategy is adopted for the learning rate, with the initial learning rate preferably set between 1e-4 and 3e-4, and the minimum learning rate being one percent of the initial learning rate. The annealing period is consistent with the total number of training rounds. The optimizer is AdamW, with the weight decay coefficient preferably set between 0.01 and 0.05. The training data size is preferably between 10,000 and 50,000 pairs, with the ratio of positive to negative sample pairs preferably set between 1:3 and 1:5, to simulate the distribution characteristics of non-equivalent element pairs being far greater than equivalent element pairs in real-world scenarios.

[0074] The node features in the heterogeneous graph are initialized as follows: The initial feature vector of the evidence node is formed by concatenating the one-hot encoding of the modality type of the evidence with the hash prefix encoding of the original data fingerprint. The dimension is the number of modality categories plus the hash prefix encoding dimension, preferably 32 to 64 dimensions in total. The initial feature vector of the feature node is formed by the feature semantic vector obtained by the cross-modal evidence feature alignment model encoding, with a dimension of 256 dimensions. When constructing cross-modal association edges, pairwise similarity calculation is first performed on all feature node pairs of the same category. When the number of evidences is M and an average of K features are extracted from each piece of evidence, the order of magnitude of the number of feature pairs to be calculated is O(M^2 * K^2). To reduce computational complexity, for textual features (text portions in behavioral descriptions and personal identities), it is preferable to first use Locality-Sensitive Hashing (LSH) technology to quickly filter candidate pairs, reducing the number of candidate pairs to the order of O(M * K * log(M * K)), and then perform precise semantic similarity calculation on the filtered candidate pairs. The value of the association threshold theta directly affects the connectivity of the graph and the accuracy of consistent inference: too high a theta results in an overly sparse graph, potentially missing true cross-modal associations; too low a theta results in an overly dense graph, introducing too many false associations and increasing the false positive rate of conflict detection. The optimal range for theta is 0.5 to 0.8. In practical deployments, the threshold point corresponding to maximizing the F1 score can be selected by plotting the association precision versus recall curves on the validation set.

[0075] The beneficial effects of this implementation method are as follows. By designing differentiated semantic similarity calculation methods for the five categories of judicial elements, compared to using a uniform similarity metric for all element categories, it is possible to adapt the processing to the semantic characteristics of each element: Gaussian kernel similarity calculation for time elements, combined with an adaptive time tolerance mechanism, can handle reasonable matching between time expressions of different precision levels; the transaction tolerance mechanism for monetary elements can discover the correspondence between transaction transfers and total amount descriptions; the text-biometric feature weighted fusion for personal identity elements can provide supplementary matching basis through voiceprint or facial biometrics when there are large differences between name and title expressions; the sentence-level semantic encoding for behavioral description elements can identify descriptions with different expressions but the same behavioral meaning; and the hierarchical normalization processing for geographical location elements can unify geographical expressions of different precision into calculable coordinate distances. The above differentiated design makes the edge establishment of cross-modal evidence semantic association graph more accurate, reduces the generation of false and omitted association edges, and thus provides a higher quality input graph structure for subsequent consistency reasoning based on graph attention networks. By introducing a cross-modal semantic alignment model based on contrastive learning, a unified semantic representation is established in the cross-modal feature space using a dual-tower encoder architecture and the InfoNCE loss function. This ensures that semantically equivalent factual elements from different modalities have high cosine similarity in the feature space, while semantically inequivalent element pairs maintain a greater distance. This alignment mechanism effectively alleviates the problem of inconsistent feature distributions of the same factual element in multimodal evidence due to modal differences, and improves the accuracy of establishing cross-modal association edges. Locality-sensitive hashing (LSH) is used for rapid candidate pair screening, reducing the computational complexity of element pairs from O(M^2 * K^2) to O(M * K * log(M * K)), making the scheme computationally efficient even on large-scale evidence sets.

[0076] The principle analysis of this implementation method is as follows. The technical rationality of the differentiated semantic similarity calculation method stems from the fact that the five types of judicial elements each have different semantic structures and matching characteristics. The time element is essentially a one-dimensional ordered numerical value, which is suitable for similarity measurement by combining numerical differences with kernel function mapping. The choice of Gaussian kernel function makes the similarity decay exponentially with the time difference, which is in line with the natural semantics of "the closer, the more relevant" in time matching. The amount element is also a numerical type, but in actual fraud scenarios, there are situations where individual transfers are inconsistent with the total amount. Therefore, it is more reasonable to use the relative deviation rate rather than the absolute difference to calculate similarity. The individual transfer tolerance mechanism covers the actual scenario where a single description corresponds to multiple transactions. The difficulty in matching the identity element lies in the fact that the same person may appear in different forms of identification in different modal evidence (such as name, title, account, voiceprint, face). A single-dimensional similarity is difficult to cover all situations. Therefore, a multi-dimensional weighted fusion method is used for comprehensive judgment. The reason for setting the text identity weight alpha between 0.2 and 0.4 is that while matching text names is intuitive, it is easily affected by factors such as changes in titles, homophones, and abbreviations. Biometric features (voiceprints and faces), on the other hand, possess individual uniqueness and cross-modal stability, providing a more reliable basis for identity matching, and therefore are given a higher weight. Behavioral description elements belong to the semantic information in free text form; different expressions can convey the same behavioral meaning. Using semantically fine-tuned sentence vectors for comparison can effectively capture deep semantic similarity beyond surface-level word matching. Matching geographic location elements requires converting textual place names into quantifiable and comparable geographic coordinates. After the Gaussian kernel function maps distance to similarity, its decay rate is controlled by the sigma_geo parameter. The selection of this parameter is based on the fact that in judicial practice, the determination of "the same location" usually allows for a certain range of spatial deviation. The reason why the cross-modal semantic alignment model based on contrastive learning adopts a dual-tower architecture instead of a shared encoder architecture is that although the text elements from different modalities are all converted into text form after semantic extraction, their text style and noise features have systematic differences. Speech-transcribed text contains recognition errors and colloquial features, OCR-extracted text contains recognition noise and typographical residues, and structured field text has a regular format but different information density. The dual-tower architecture allows the two encoder towers to learn to adapt to the text feature distribution of their respective input sources, and achieve alignment after projecting onto a common semantic space. Compared with the single-tower architecture that forces shared parameters, it can better handle the distribution differences of text from different sources. The InfoNCE loss function achieves the learning of distinguishing between positive and negative samples through intra-batch comparison. Its theoretical basis is mutual information maximization, that is, maximizing the mutual information between the feature representations of positive sample pairs, while minimizing the mutual information with negative sample pairs, so that semantically equivalent element pairs cluster in the feature space, and semantically inequivalent element pairs are separated in the feature space.

[0077] Example 3 In another embodiment of the present invention, an alternative technical solution is used to implement the consistency reasoning module based on graph attention network in step four. This embodiment keeps the technical flow of steps one to three and step five unchanged, and replaces the graph attention network model-based scheme with a serialization reasoning method based on cross-modal Transformer for consistency reasoning and conflict detection on the cross-modal evidence semantic association graph in step four.

[0078] Specifically, this implementation transforms the set of cross-modal association edges in the cross-modal evidence semantic association graph constructed in step three into a serialized feature pair representation. For each cross-modal association edge connecting two feature nodes, their respective feature semantic vectors, feature category embeddings, source evidence modality type embeddings, and the semantic similarity value of the edge are concatenated to construct a feature pair feature vector. The feature category embedding consists of learnable embedding vectors corresponding to five categories of judicial elements, preferably with a dimension of 32. The source evidence modality type embedding consists of learnable embedding vectors corresponding to five modality types, preferably with a dimension of 32. The feature vectors of all cross-modal association edges are arranged in a certain order to form a feature pair sequence. The arrangement order is preferably grouped according to feature categories, and within the same category, arranged in descending order of semantic similarity, so that the model first focuses on potential association pairs with high similarity during sequence processing.

[0079] The aforementioned feature pair sequences are input into a cross-modal Transformer encoder for consistent inference. The cross-modal Transformer encoder comprises multiple standard Transformer encoding layers, preferably 4 to 8 layers, each containing a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The multi-head self-attention sub-layer preferably has 8 to 12 attention heads, with the key and value dimensions preferably set to 64 dimensions. The hidden layer dimension of the feedforward neural network sub-layer is preferably four times the input dimension. A learnable classification token (CLS token) is inserted at the beginning of the feature pair sequence to extract a global semantic representation after encoding. Positional encoding information is superimposed on the feature vector of each feature pair before it is input into the Transformer; this positional encoding employs a learnable positional embedding method.

[0080] The cross-modal Transformer encoder models the global dependencies between all feature pairs through a self-attention mechanism. Unlike graph attention networks, which only propagate information within the neighborhood structure of the graph, the Transformer's self-attention mechanism allows direct information interaction between any two feature pairs, thus capturing cross-feature category association patterns. For example, the consistency determination of the monetary element can refer to the consistency status of the time element and the person's identity element in the same evidence set; the Transformer's global attention mechanism can automatically learn this cross-feature category association dependency. After multiple layers of Transformer encoding, the output representation of each feature pair's position has incorporated global contextual information.

[0081] Consistency classification is performed at two levels. At the edge level, the Transformer output representation for each feature pair is connected to a two-layer fully connected classification head. The hidden layer dimension is preferably 128, the activation function is GELU, and the output layer is a three-class softmax layer, outputting the probability distributions of consistent, contradictory, and uncertain states. For global consistency evaluation, the Transformer output representation at the classification label position is connected to another independent fully connected regression head, outputting a global consistency prediction value between zero and one. This global consistency prediction value is directly obtained from the global representation regression of the Transformer, comprehensively considering the consistency state of all feature pairs and their interrelationships. Compared to the weighted summation-based global scoring formula in Example 1, it has stronger nonlinear modeling capabilities and can learn the global consistency level corresponding to different combinations of conflict modes.

[0082] The model training employs a multi-task learning strategy, simultaneously optimizing edge-level consistency classification loss and global consistency regression loss. The total loss function is L_total = L_edge + beta * L_global, where L_edge is the mean of the three-class cross-entropy loss at all feature pair positions, L_global is the mean squared error loss between the globally consistent predicted value and the labeled globally consistent label, and beta is the weight coefficient of the global loss, preferably set between 0.5 and 1.0. The training data is constructed from the same source as the training data of the graph attention network in Example 1, but the data format is converted from a graph structure to a sequence of feature pairs. Regarding training hyperparameters, the learning rate is preferably set between 5e-5 and 2e-4, the batch size is preferably set between 16 and 32, and the number of training epochs is preferably set between 30 and 100. The AdamW optimizer is used, the weight decay coefficient is preferably set between 0.01 and 0.1, and the learning rate adopts a combination strategy of linear warm-up and cosine decay, with the number of warm-up steps preferably set to 5% to 10% of the total training steps. Dropout regularization is applied between the Transformer encoding layers and on the hidden layers of the classification head, with the dropout rate preferably set between 0.1 and 0.3.

[0083] In terms of conflict localization and source analysis, this implementation method, in addition to utilizing the edge-level classification results for conflict localization, can also perform conflict attribution analysis through the attention weights of the Transformer's self-attention layer. Specifically, for element pairs determined to be contradictory, the attention distribution between the position of this element pair and other element pairs in the self-attention weight matrix of each Transformer layer is extracted. This identifies other element pairs that contribute the most to the contradiction determination, thereby revealing the basis for the contradiction determination. For example, when a monetary element pair is determined to be contradictory, attention weight analysis may show that the determination is influenced by the contradictory state of the time element pair in the same evidence combination, indicating that evidence elements in multiple dimensions point to semantic inconsistency in this evidence combination. The above attention weight analysis results are incorporated into the semantic consistency verification report as supplementary information for the conflict description record, providing a more detailed reference for manual review by evidence preservation personnel.

[0084] Compared to the graph attention network model-based scheme in Example 1, the technical difference in this implementation is that: graph attention networks perform local message propagation on the neighborhood structure of the graph, and their information aggregation range is limited by the graph's topology and the number of message propagation layers. Long-range dependencies between distant nodes in the graph need to be gradually transmitted through multiple layers of propagation. In contrast, the Transformer self-attention mechanism used in this implementation can model the global dependencies between all element pairs within a single layer, and is more direct in modeling cross-element category association patterns. Furthermore, this implementation uses a multi-task learning strategy to simultaneously train edge-level classification and global consistency regression, enabling the model to learn the consistency judgment of individual element pairs while also learning direct predictions of global consistency levels. The gradient signals between the two tasks can complement each other. In terms of inference efficiency, when the element pair sequence length is short (e.g., less than 100 element pairs), the computational efficiency of Transformer is comparable to that of graph attention networks; when the element pair sequence length is long, a sparse attention mechanism or block processing strategy can be used to control computational complexity. This implementation method can serve as an alternative technical route to the graph attention network model-based scheme in Example 1. In different application scenarios, a suitable reasoning scheme can be selected based on the size and elements of the evidence set.

[0085] Example 4 like Figure 4 As shown, this embodiment provides a network fraud evidence consolidation system with multimodal semantic consistency verification. This system is used to implement the network fraud evidence consolidation method with multimodal semantic consistency verification described in Embodiment 1. The system includes an evidence collection and modality recognition module, a multimodal semantic extraction module, a semantic association graph construction module, a consistency reasoning and conflict detection module, and an evidence consolidation and on-chain module. Data flow between these modules is achieved through a unified data interface.

[0086] The evidence collection and modality recognition module is used for unified collection and modality recognition of multimodal electronic evidence in online fraud cases. This module receives a set of multimodal electronic evidence to be secured, including text, image, audio, video, and structured data evidence. For each piece of input electronic evidence, the module automatically identifies its modality type, classifying it into the corresponding modality category based on file format features, MIME type identifiers, and content decoding verification. The modality type recognition employs a dual verification method based on file header signature features and content decoding verification. When the file extension differs from the actual detection result, the actual detection result prevails, effectively addressing scenarios where file extensions are maliciously tampered with. The module generates an original data fingerprint for each piece of electronic evidence, including a SHA-256 hash value and a file metadata digest, serving as the foundation for subsequent on-chain evidence securing. After processing, the evidence collection and modality recognition module outputs the evidence data with modality category tags and original data fingerprints to the multimodal semantic extraction module.

[0087] The multimodal semantic extraction module is used to extract semantic information oriented towards judicial elements from electronic evidence of each modality, obtaining structured representations of evidence elements. These judicial elements include five dimensions: time, amount, identity, behavior description, and geographical location. For different modalities of evidence, this module employs appropriate semantic extraction methods. For textual evidence, a named entity recognition model fine-tuned with judicial corpus is used to extract time expressions, monetary amounts, names and titles, action descriptions, and location names from text such as chat logs and text messages. This named entity recognition model is based on a pre-trained language model using transfer learning; the training corpus includes labeled chat log samples from fraud cases, enabling the model to recognize non-standardized expressions unique to online fraud scenarios. For audio evidence, an automatic speech recognition model is first used to transcribe the speech signal into a text sequence. Then, named entity recognition processing is performed on the transcribed text, simultaneously extracting the audio's voiceprint feature vector for subsequent cross-modal identity comparison. The automatic speech recognition model employs an end-to-end encoder-decoder architecture, supporting the recognition of Chinese dialects and colloquial expressions, and retaining timestamp alignment information in the transcribed text. For image-based evidence, optical character recognition (OCR) technology is used to extract text content from the image, followed by named entity recognition (NAME) processing. Simultaneously, an image content understanding model is used to analyze the visual semantics of the image. For chat screenshots, a specialized chat interface parsing model is used to identify the sender and receiver of message bubbles, message timestamps, and message content areas. For transfer voucher screenshots, a specialized financial document parsing model is used to identify and extract structured fields such as transaction amount, transaction time, payee information, payer information, and transaction status. For video evidence, keyframe extraction is first performed. An adaptive sampling strategy based on visual content change detection is used to perform image semantic extraction processing on each extracted keyframe. Simultaneously, the audio track of the video is separated for audio-based semantic extraction processing, further extracting facial feature vector sequences appearing in the video. For structured data evidence, its structured fields are directly parsed to extract elements such as transaction time, transaction amount, identifiers of the transacting parties, transaction type, and transaction remarks. The transaction time is then standardized and converted to Coordinated Universal Time (UTC). The multimodal semantic extraction module outputs the structured element representations of each modality of evidence to the semantic association graph construction module.

[0088] The semantic association graph construction module is used to construct a cross-modal evidence semantic association graph based on the modal evidence elements output by the multimodal semantic extraction module. The evidence semantic association graph is a heterogeneous graph structure, containing three types of elements: evidence nodes, element nodes, and relationship edges. Each piece of electronic evidence corresponds to an evidence node, whose attributes include the modality type of the evidence and the original data fingerprint. Each judicial element extracted from each piece of evidence corresponds to an element node, whose attributes include element category, element value, and source evidence identifier. Relationship edges connect evidence nodes with their contained element nodes, as well as semantically related element nodes in different pieces of evidence. The relationships between cross-modal element nodes are established by calculating the semantic similarity between element values ​​of the same element category. The calculation method for semantic similarity varies depending on the element category: for time elements, the difference between standardized time values ​​is calculated; for monetary elements, the relative deviation rate between numerical values ​​is calculated; for identity elements, the edit distance similarity of the text name and the cosine similarity of the voiceprint or facial feature vectors are calculated, and the weighted average of the two is used as the semantic similarity of the identity element; for behavioral description elements, a semantically fine-tuned text encoder is used to calculate the cosine similarity of sentence vectors; and for geographic location elements, the geocoding distance after the normalization of the place name is calculated. When two element nodes of the same category originate from evidence of different modalities and their semantic similarity exceeds a preset association threshold, a cross-modal association edge is established. Each evidence node establishes a belonging edge between it and each of its extracted element nodes. This module also introduces a cross-modal semantic alignment mechanism based on contrastive learning. A cross-modal evidence element alignment model with a dual-tower encoder architecture is pre-trained. The contrastive learning loss function brings semantically equivalent cross-modal element pairs closer in the feature space and widens the distance between semantically inequivalent element pairs. This module also introduces a temporal-causal consistency constraint layer. Based on the event timestamps extracted from different pieces of evidence, it establishes a temporal partial order relationship, constructs a directed event temporal graph, and performs a topological consistency check on the directed acyclic graph. If a temporal loop or causal violation is detected, it is marked as a temporal conflict. The semantic association graph construction module outputs the completed cross-modal evidence semantic association graph to the consistency reasoning and conflict detection module.

[0089] The consistency reasoning and conflict detection module performs consistency reasoning and conflict detection on the cross-modal evidence semantic association graph output by the semantic association graph construction module, generating a semantic consistency verification report. The consistency reasoning is implemented using a graph attention network model. This module generates initial feature vectors for each evidence node and element node in the cross-modal evidence semantic association graph. It inputs the initial feature vectors of each node and their graph structure adjacency relationships into the graph attention network model. Through the belonging edges, the feature information of the evidence node is aggregated to its associated element nodes. Message passing occurs between different element nodes through cross-modal association edges. The graph attention network model aggregates information on the graph structure through a multi-layer message propagation mechanism, enabling the feature representation of each element node to incorporate contextual semantic information from its associated nodes. After multiple rounds of message propagation, consistency classification is performed on each cross-modal association edge to determine whether the facts represented by the two cross-modal element nodes connected by the edge are semantically consistent. The output of the consistency classification includes three states: consistent, contradictory, and uncertain. Based on edge-level consistency classification, a global consistency score is performed. This global consistency score is obtained by weighted aggregation of the consistency classification results of all cross-modal related edges. The sum of the weights of the cross-modal related edges judged to be in a consistent state is divided by the sum of the weights of all cross-modal related edges, resulting in a global consistency score between zero and one. When the global consistency score is lower than a preset consistency threshold, the evidence set is determined to have a cross-modal semantic conflict. When a semantic conflict is detected, this module performs conflict localization and source tracing analysis, extracting all cross-modal related edges judged to be in a contradictory state. For each contradictory edge, the source evidence of the two element nodes it connects is traced back, generating a conflict description record including the conflict element category, the first evidence identifier and its element value, the second evidence identifier and its element value, and the contradiction confidence level. The record is then sorted according to the severity of the conflict to form a semantic consistency verification report. The consistency reasoning and conflict detection module outputs the global consistency score and the semantic consistency verification report to the evidence on-chain module.

[0090] The evidence-on-chain module is used to perform evidence-on-chain operations based on the semantic consistency verification results output by the consistency reasoning and conflict detection module. This module packages the original data fingerprint of each piece of electronic evidence, the corresponding structured evidence elements, the global consistency score, and the semantic consistency verification report into an evidence-on-chain data package. After calculating the hash value of the entire evidence-on-chain data package, it is written to the blockchain. The evidence-on-chain operation executes differentiated evidence-on-chain strategies based on different consistency verification results: when the global consistency score is higher than or equal to the consistency threshold, the evidence set is marked as having passed consistency verification, the standard evidence-on-chain process is executed, and the evidence-on-chain data package is stored on the blockchain; when the global consistency score is lower than the consistency threshold, the evidence set is marked as having an abnormal consistency verification state, the evidence-on-chain data package, along with a detailed conflict description record, is stored on the blockchain, and a verification anomaly prompt is sent to the evidence-on-chain operator. The prompt information includes the specific location of the conflict element and a summary of the conflict content for manual review and confirmation. After the evidence is uploaded to the blockchain, the module generates an evidence certificate, which includes the blockchain transaction hash, evidence timestamp, evidence list and corresponding hash value, global consistency score and verification conclusion. The evidence certificate is encapsulated in a verifiable certificate format and supports independent verification by third parties.

[0091] In terms of hardware deployment, the system is deployed on a computing server equipped with a general-purpose processor and a graphics processing unit (GPU). The general-purpose processor is used to run basic computing tasks such as file parsing, format verification, and hash calculation in the evidence collection and modality recognition module, as well as blockchain interaction operations in the evidence preservation and on-chain module. The GPU is used to accelerate deep learning model inference in the multimodal semantic extraction module, contrastive learning model calculation in the semantic association graph construction module, and graph attention network model inference in the consistency inference and conflict detection module. The system's storage subsystem includes high-speed solid-state storage devices and large-capacity persistent storage devices. The former is used to cache evidence files to be processed and intermediate calculation results, while the latter is used to persistently store evidence preservation data packets and verification reports. Data transfer between modules is achieved through a unified message queue middleware, using an asynchronous message passing mechanism to support decoupled deployment and elastic scaling between modules. The system provides a standardized application programming interface (API) to support receiving evidence to be preserved and returning evidence preservation certificates via network protocols.

[0092] Example 5 This embodiment demonstrates the implementation process and technical effects of the network fraud evidence consolidation method for multimodal semantic consistency verification described in this invention through a specific application case.

[0093] In the evidence consolidation scenario of a certain online fraud case, the electronic evidence set to be consolidated contained 12 pieces of electronic evidence in 5 modalities, specifically including: 3 pieces of text evidence, namely, exported files of instant messaging software chat logs, SMS log files, and email content files; 2 pieces of image evidence, namely, chat screenshots and transfer voucher screenshots; 2 pieces of audio evidence, namely, two telephone call recordings, with durations of 4 minutes 32 seconds and 7 minutes 15 seconds respectively; 2 pieces of video evidence, namely, a video call recording clip and a screen recording video, with durations of 3 minutes 18 seconds and 5 minutes 42 seconds respectively; and 3 pieces of structured data evidence, namely, two bank transfer records and one transaction record from a third-party payment platform. This case involved multiple fund transfers, with evidence sources scattered across multiple communication channels and payment platforms. There were complex semantic relationships between the different modalities of evidence, and some evidence contained colloquial and non-standardized statements of fact. Conventional piece-by-piece hashing and on-chain consolidation methods could not detect potential semantic contradictions between the evidence.

[0094] According to the method described in this invention, step one is first performed to uniformly collect and perform modal identification on the aforementioned 12 pieces of electronic evidence. The system performs automatic modal type identification on each piece of evidence based on dual verification of file header signature features and content decoding. During the identification process, the file extension of the second piece of audio evidence is an image format extension, but the file header signature verification detects that its actual content is in audio encoding format. The system correctly classifies it as audio evidence based on the content decoding verification analysis result. SHA-256 hash values ​​and file metadata digests are generated for all 12 pieces of evidence as raw data fingerprints.

[0095] Step two then proceeds, extracting semantic information for each modal of evidence based on judicial elements. For the three text-based pieces of evidence, the named entity recognition model, fine-tuned for judicial applications, extracted nine time elements, six monetary elements, five identity elements, eleven behavioral description elements, and three geographical location elements. Notably, the chat logs contained colloquial monetary expressions, which the model correctly identified as their corresponding standard monetary values. For the two audio-based pieces of evidence, the automatic speech recognition model transcribed the speech signals into text sequences, preserving timestamp alignment information. The named entity recognition model then extracted four time elements, three monetary elements, three identity elements, six behavioral description elements, and two geographical location elements from the transcribed text. Simultaneously, 256-dimensional voiceprint feature vectors were extracted from each speaker in both recordings. For the two pieces of image-based evidence, optical character recognition (OCR) technology extracted the text content from the images. A chat interface analysis model identified eight message records from the chat screenshots, including sender identifiers, message content, and message timestamps. A financial document analysis model extracted structured fields such as transaction amount, transaction time, payee information, and payer information from the transfer voucher screenshots. For the two pieces of video-based evidence, an adaptive sampling strategy extracted 42 and 67 keyframes from the two video clips, respectively. Image semantic extraction was performed on the keyframes, and the audio track was separated for speech transcription and voiceprint feature extraction. A 512-dimensional facial feature vector sequence appearing in the videos was also extracted. For the three pieces of structured data-based evidence, their structured fields were directly parsed, extracting elements such as time, amount, and transacting party identifiers from a total of 15 transaction records. All transaction times were uniformly converted to Coordinated Universal Time (UTC). After step two, a total of 127 legal elements were extracted from all 12 pieces of evidence.

[0096] Next, step three is executed: a cross-modal evidence semantic association graph is constructed based on the extracted evidence elements. The constructed heterogeneous graph contains 12 evidence nodes and 127 element nodes. Through differentiated semantic similarity calculation, under the condition that the association threshold is set to 0.65, a total of 83 cross-modal association edges are established, including 21 time element association edges, 18 monetary element association edges, 14 person identity element association edges, 19 behavioral description element association edges, and 11 geographic location element association edges. In the process of association of person identity elements, the cross-modal evidence element alignment model successfully matched the person's title in the chat log with the speaker's identity corresponding to the voiceprint features in the audio recording, and the same person's identity corresponding to the facial feature vector in the video and the voiceprint feature vector in the audio. In the processing of the temporal-causal consistency constraint layer, the system constructs an event temporal directed graph containing 23 event nodes based on the event timestamps extracted from each piece of evidence, and performs a topological consistency check on the directed graph.

[0097] Step four then proceeds, performing consistency reasoning and conflict detection on the constructed cross-modal evidence semantic association graph. The graph attention network model performs consistency classification on each of the 83 cross-modal association edges. The classification results are as follows: 71 edges are classified as consistent, 8 edges as contradictory, and 4 edges as uncertain. Among the 8 contradictory edges, 3 involve monetary elements, 3 involve time elements, and 2 involve personal identity elements. Specifically, among the three monetary discrepancies, the first is a numerical discrepancy between the transfer amount extracted from the chat log text and the transaction amount extracted from the screenshot of the transfer voucher; the former is 50,000 yuan while the latter is 45,000 yuan, with a discrepancy confidence level of 0.91. The second is an inconsistency between the monetary element mentioned in the transcribed text of the first call recording and the monetary element recorded in the bank transfer statement; the former is 30,000 yuan while the latter is 28,000 yuan, with a discrepancy confidence level of 0.87. The third is a difference between the cumulative total transfer amount mentioned in the chat log and the sum of the transaction amounts in the three structured data sets, with a discrepancy confidence level of 0.83. Among the three temporal discrepancies, the first is a temporal loop detected in the event sequence directed graph: the timestamp of the negotiated transfer message in the chat log is later than the transaction completion timestamp shown in the screenshot of the transfer voucher, indicating that the transfer was completed before the negotiation, constituting a causal temporal violation. The other two are discrepancies between the temporal elements of the same event extracted from different modal evidence that exceed the time tolerance range. In the two conflicting pieces of evidence regarding the identities of individuals, the cosine similarity between the voiceprint feature vector of a speaker in the first recorded phone call and the voiceprint feature vector of another person in the video call is only 0.42, which is below the threshold for associating identities. This suggests that the speakers considered to be the same person in the two pieces of evidence may not be the same natural person.

[0098] Based on the above classification results, the global consistency score S_global is calculated to be 0.72. After incorporating the temporal conflict penalty, the corrected global consistency score S_final is calculated to be 0.68. With the consistency threshold T set to 0.75, S_final is lower than T, and the system determines that the evidence set contains cross-modal semantic conflicts. The system classifies and sorts the conflicts according to their severity; three monetary conflicts and three temporal conflicts are marked as high-severity conflicts, and two identity conflicts are marked as medium-severity conflicts, generating a semantic consistency verification report.

[0099] Finally, step five is executed, performing the evidence-based on-chain operation based on the above verification results. Because the global consistency score is below the consistency threshold, the system marks the evidence set as having an abnormal consistency verification status. The system packages the original data fingerprints of 12 pieces of evidence, 127 structured evidence elements, a global consistency score of 0.68, and a verification report containing 8 conflict description records into an evidence-based data package, which is then stored on the blockchain along with the detailed conflict description records. Simultaneously, the system sends a verification anomaly notification to the evidence-based operator, specifying the exact numerical deviation of the amount, the specific timestamp comparison of the temporal discrepancies, and the specific voiceprint similarity value for the mismatched identity, for manual review and confirmation. After the evidence is uploaded to the blockchain, the system generates an evidence certificate containing the blockchain transaction hash, the evidence timestamp, a list of SHA-256 hash values ​​for the 12 pieces of evidence, a global consistency score of 0.68, and the conclusion of the abnormal consistency verification.

[0100] The above application examples demonstrate that the method described in this invention, by performing cross-modal semantic consistency verification on 12 pieces of electronic evidence from 5 modalities, automatically detected 8 semantic contradictions from 127 judicial elements and 83 cross-modal association edges. These contradictions cover three types: numerical discrepancies in monetary amounts, violations of temporal causality, and mismatches in personal identities. These contradictions are automatically marked as abnormal states during the evidence consolidation process, providing prompts for manual review. Compared to evidence consolidation methods that only perform hash integrity verification on a single piece of evidence, this invention can discover contradictory relationships between cross-modal evidence at the semantic level, improving the verification depth and credibility of the evidence consolidation results.

[0101] To verify the performance advantages of the proposed cross-modal association edge consistency classification method based on graph attention networks, ablation experiments and comparative tests were conducted on a multimodal network fraud evidence dataset. The evaluation metrics were the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The testing conditions were as follows: a unified evidence corpus was used, containing five modalities: text, audio, image, video, and structured data. A cross-modal association edge sample set was constructed, with consistent, contradictory, and uncertain edges sampled according to the distribution of actual judicial cases. Comparison methods included: a threshold classification baseline method based on feature distance (AUC=0.73), a GAT method using a unified similarity metric (AUC=0.82), a GAT method removing the contrast learning alignment module (AUC=0.88), and the method of this invention (AUC=0.94). The ROC curve of the method of this invention was above all the comparison methods, with an AUC value of 0.94, representing an improvement of approximately 0.21 compared to the threshold classification method based solely on feature distance, and an improvement of approximately 0.06 compared to the ablation experiment method removing the contrast learning alignment module. The above results demonstrate that the synergistic effect of differentiated cross-modal semantic similarity measurement and contrastive learning alignment mechanism makes a substantial contribution to edge-level consistency classification performance. Both of these technical components are indispensable and have a substantial impact on the overall classification performance.

[0102] To verify the semantic contradiction detection performance of the method of this invention under different modal combinations, for five modalities—text, audio, image, video, and structured data—the proportion of edges judged as contradictory in cross-modal association edges between each modal pair was statistically analyzed, i.e., the conflict detection rate. A 5x5 conflict detection rate matrix was constructed, and conflict type decomposition analysis was performed on modal pairs with high conflict rates. Figure 7 As shown, the matrix rows and columns of the heatmap on the left correspond to five modal types, with colors gradually changing from white to darker; darker colors indicate higher conflict detection rates. Diagonal cells represent same-modal pairs and are not included in cross-modal association statistics. The heatmap shows that the conflict detection rate is highest between images and structured data (0.22), reflecting significant numerical discrepancies between screenshots of transfer vouchers and transaction records. The conflict detection rate is second highest between text and structured data (0.18), and lowest between audio and video pairs (0.06), indicating high consistency when audio and video are from the same source. The horizontally stacked bar chart on the right shows the conflict type composition of the top four conflict-prone modal pairs: the proportions of monetary, temporal, and identity conflicts vary depending on the modal pair. In image and structured data pairs, monetary conflicts account for approximately 55%, making it the primary conflict type, while in text and image pairs, temporal conflicts account for a relatively high proportion of approximately 40%. The above distribution characteristics are consistent with the case data of 3 contradictions in amount, 3 contradictions in time and 2 contradictions in identity of persons identified in the embodiments of the present invention, which verifies the effective identification capability of the cross-modal semantic contradiction detection method for contradictions of different types of evidence.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for consolidating evidence of network fraud through multimodal semantic consistency verification, characterized in that, Includes the following steps: Step S1: Collect and perform modal identification on multimodal electronic evidence in online fraud cases, and generate an original data fingerprint for each piece of electronic evidence, including the SHA-256 hash value and file metadata digest. Step S2: Semantic information extraction is performed on each modality of electronic evidence in five dimensions: time element, amount element, person identity element, behavior description element and geographical location element, to obtain a structured representation of evidence elements; Step S3: Based on the structured evidence element representation obtained in step S2, construct a cross-modal evidence semantic association graph with a heterogeneous graph structure including evidence nodes, element nodes, and relation edges; wherein, each evidence node establishes a belonging edge with each of its extracted element nodes; for element nodes of the same element category, a differentiated semantic similarity calculation method is adopted according to the element category, and when two element nodes of the same category come from evidence of different modalities and the semantic similarity exceeds a preset association threshold, a cross-modal association edge is established; Step S4: Generate initial feature vectors for each evidence node and element node in the cross-modal evidence semantic association graph; input the initial feature vectors of each node and their graph structure adjacency relationships into the graph attention network model; aggregate the feature information of the evidence node to its associated element node through the belonging edge; perform message passing between different element nodes through the cross-modal association edge; perform information aggregation through a multi-layer message propagation mechanism; perform consistency classification on each cross-modal association edge and output three states: consistent, contradictory, and uncertain; perform weighted aggregation on the consistency classification results of all cross-modal association edges to obtain a global consistency score. Step S5: Perform differentiated on-chain evidence storage operations based on the global consistency score, write the evidence storage data package into the blockchain, and generate evidence storage certificates. The evidence storage data package includes at least the original data fingerprint of each electronic evidence, the structured evidence element representation, the global consistency score, and the semantic consistency verification report.

2. The method according to claim 1, characterized in that, The differentiated semantic similarity calculation method described in step S3 includes: for time elements, calculating the difference between standardized time values; for monetary elements, calculating the relative deviation rate between numerical values; for personal identity elements, calculating the edit distance similarity of text names and the cosine similarity of voiceprint or facial feature vectors respectively, and using the weighted average of the two as the semantic similarity of personal identity elements; for behavioral description elements, using a semantically fine-tuned text encoder to calculate the cosine similarity of sentence vectors; and for geographical location elements, calculating the geocoding distance after the place name is normalized.

3. The method according to claim 1, characterized in that, Step S3 further includes introducing a cross-modal semantic alignment mechanism based on contrastive learning: a cross-modal evidence element alignment model is pre-trained, which adopts a dual-tower encoder architecture. The first encoding tower is a text modality encoder, and the second encoding tower is a general element encoder, which encodes features of elements from different modalities respectively. The contrastive learning loss function is used to shorten the distance between semantically equivalent cross-modal element pairs in the feature space and to widen the distance between semantically inequivalent element pairs. The contrastive learning loss function adopts InfoNCE (Information Noise-Contrast)... The loss (Estimation) is calculated for each positive sample pair in the training batch, i.e., semantically equivalent cross-modal element pairs. The cosine similarity of their feature vectors is calculated and normalized by softmax, and the negative logarithm is taken as the loss value. The training data for the contrastive learning consists of evidence element pairs of historical fraud cases labeled with consistency tags. The semantic vector of the element obtained by the cross-modal evidence element alignment model encoding is used as the initial feature vector of the element node in step S4. The one-hot encoding of the modality type of the evidence is concatenated with the hash prefix encoding of the original data fingerprint to form the initial feature vector of the evidence node.

4. The method according to claim 1, characterized in that, The modality recognition described in step S1 adopts a dual verification method based on file header feature signature and content decoding verification: the first level is fast recognition based on file header feature signature, which determines the actual format type of the file by reading the first few bytes of the file and matching the known file format magic number; The second level is deep verification based on content decoding. For files identified by the first level, content decoding and parsing are attempted according to the identified format type to verify whether the file content conforms to the encoding specifications of that format. When the file extension is inconsistent with the actual detection result, the actual detection result shall prevail.

5. The method according to claim 1, characterized in that, Step S3 also includes introducing a time-causal consistency constraint layer: establishing a partial order relationship of event time based on the event timestamps extracted from different evidence, constructing a directed graph of event time based on the partial order relationship of event time, taking the event nodes extracted from each modality as nodes of the directed graph, establishing directed edges according to the timetamp size relationship and causal logic relationship, with the directed edges pointing from earlier events to later events. Perform a topological consistency check of the directed acyclic graph on the event time-series directed graph, and detect whether there are cycles in the directed graph by depth-first search or topological sorting algorithm based on in-degree. If a timing loop is detected, it is marked as a timing conflict and the timing conflict information is included in the global consistency score.

6. The method according to claim 1, characterized in that, The global consistency score in step S4 is calculated as follows: the sum of the weights of the cross-modal related edges that are determined to be in a consistent state is divided by the sum of the weights of all cross-modal related edges to obtain a global consistency score between zero and one; when the global consistency score is lower than a preset consistency threshold, it is determined that the evidence set composed of the multimodal electronic evidence collected in step S1 has a cross-modal semantic conflict.

7. The method according to claim 6, characterized in that, Step S4, upon detecting a semantic conflict, further includes performing conflict localization and source analysis: extracting all cross-modal association edges identified as contradictory, tracing back the source evidence of the two element nodes connected to each contradictory edge, and generating a conflict description record. The conflict description record includes the conflict element category, the first evidence identifier and its element value, the second evidence identifier and its element value, and the contradiction confidence level. All conflict description records are summarized to form a semantic consistency verification report. The verification report is graded and sorted according to the severity of the conflict. Conflicts involving monetary and time elements are marked as high-severity conflicts, conflicts involving personal identity elements are marked as medium-severity conflicts, and conflicts involving behavioral description elements and geographical location elements are marked as low-severity conflicts.

8. The method according to claim 1, characterized in that, The differentiated evidence-based on-chain operation in step S5 includes: when the global consistency score is higher than or equal to a preset consistency threshold, the evidence set is marked as having passed consistency verification, the standard evidence-based process is executed, and the evidence-based data package is stored on the blockchain; when the global consistency score is lower than the consistency threshold, the evidence set is marked as having an abnormal consistency verification state, the evidence-based data package, along with a detailed conflict description record, is stored on the blockchain, and a verification anomaly prompt is sent to the evidence-based operator. The prompt information includes the specific location of the conflicting elements and a summary of the conflict content for manual review and confirmation.

9. A network fraud evidence consolidation system with multimodal semantic consistency verification, characterized in that, include: The evidence collection and modality recognition module is used to collect and recognize multimodal electronic evidence in online fraud cases, and generate an original data fingerprint for each piece of electronic evidence, including a SHA-256 hash value and a file metadata digest. The multimodal semantic extraction module is used to extract semantic information from each modality of electronic evidence in five dimensions: time element, amount element, person identity element, behavior description element and geographical location element, to obtain a structured representation of evidence elements. The semantic association graph construction module is used to construct a cross-modal evidence semantic association graph with a heterogeneous graph structure, including evidence nodes, element nodes, and relation edges, based on the structured evidence element representation output by the multimodal semantic extraction module. For element nodes of the same element category, a differentiated semantic similarity calculation method is adopted according to the element category. When two element nodes of the same category come from evidence of different modalities and their semantic similarity exceeds a preset association threshold, a cross-modal association edge is established. Among them, each evidence node establishes a belonging edge between itself and each of its extracted element nodes. The consistency reasoning and conflict detection module is used to generate initial feature vectors for each evidence node and element node in the cross-modal evidence semantic association graph. It inputs the initial feature vectors of each node and their graph structure adjacency relationships into the graph attention network model. Through the belonging edges, it aggregates the feature information of the evidence nodes to their associated element nodes. It performs message passing between different element nodes through cross-modal association edges, and aggregates information through a multi-layer message propagation mechanism. It performs consistency classification on each cross-modal association edge, outputting three states: consistent, contradictory, and uncertain. Finally, it performs weighted aggregation of the consistency classification results of all cross-modal association edges to obtain a global consistency score. The evidence-on-chain module is used to perform differentiated evidence-on-chain operations based on the global consistency score, write the evidence data package into the blockchain, and generate evidence certificates. The evidence data package includes at least the original data fingerprint of each electronic evidence, the structured evidence element representation, the global consistency score, and the semantic consistency verification report.

Images