Multi-agent deepfake attack detection system for multimodal data

Abstract

The present invention discloses a multi-agent deep fake attack detection system for multimodal data, which specifically relates to the technical field of image analysis and voice-assisted fault judgment of power equipment. The semantic extraction module constructs image and voice semantic vectors, the semantic alignment module calculates alignment scores and coverage ratios, the semantic judgment module identifies semantic inconsistencies, the agent fusion module outputs comprehensive fault judgments based on multi-dimensional scoring, and the explanation and display module realizes risk area visualization and semantic deviation annotation, thereby enhancing the accuracy, interpretability and practicality of fault detection. The present invention realizes structured semantic extraction and alignment judgment of multimodal power operation and maintenance data, improves fault detection accuracy based on dual-indicator consistency evaluation and multi-level conflict identification, introduces multi-agent confidence scoring and dynamic fusion mechanisms to enhance judgment credibility, supports graphic and text linkage explanation output and report generation, and facilitates the review and use of operation and maintenance results in power grid operation and maintenance scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis and voice-assisted fault judgment of power equipment. More specifically, the present invention relates to a multi-agent deep fake attack detection system for multimodal data. Background Art

[0002] With the in-depth application of artificial intelligence technology in power equipment fault diagnosis, voice-assisted inspections, and remote operation and maintenance, intelligent decision-making systems that rely on multimodal data such as images and voice have become an essential component of modern power operation and maintenance processes. When conducting equipment inspections, operators typically verbally describe the fault using images of the power equipment. The system then analyzes the images and text simultaneously to automatically generate reports and provide troubleshooting recommendations.

[0003] However, with the advancement of technologies like image generation and speech synthesis, multimodal data from power equipment is facing the risk of more subtle deep faults. Attackers could create semantic inconsistencies between images and speech by enhancing abnormal areas in images or tampering with fault-diagnosing keywords in speech, thereby misleading the system into outputting false positives that do not correspond to the actual power equipment status. Existing detection methods are often limited to local analysis of image fault traces or audio anomalies, lacking the ability to comprehensively determine multimodal faults for semantic consistency.

[0004] In typical intelligent inspection, auxiliary recording and remote operation and maintenance scenarios, the discrepancy between images and text is often not caused by voice recognition errors or image misdecoding, but by structural tampering at the semantic level. For example, a slight fault feature is implanted in the image, and the voice content simultaneously describes the wrong equipment location or fault conclusion, forming a false result that is reasonable on the surface but contradictory in reality. Such faults are extremely confusing and difficult to identify through a single modality. At the same time, there may be inconsistencies in the perception of faults between different modal detection agents. The system cannot determine the source of the conflict, nor can it clearly point out "which descriptions are inconsistent with which areas", which seriously affects the review efficiency and judgment confidence of the operation and maintenance personnel. Therefore, the present invention proposes a multi-agent deep fake attack detection system for multimodal data in order to solve the above problems. Summary of the Invention

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A multi-agent deepfake attack detection system for multimodal data, including:

[0007] The semantic extraction module performs structural recognition and feature encoding on power equipment images to generate image semantic vectors. It also performs speech-to-text conversion and power terminology analysis on the speech content of operation and maintenance personnel to generate speech semantic vectors and construct semantic expression pairs.

[0008] The semantic alignment module receives the image semantic vector and the speech semantic vector in the semantic expression pair, calculates the semantic alignment score and the semantic coverage ratio through spatial mapping and semantic dimension normalization operations, and forms a first alignment result;

[0009] A semantic determination module determines whether the semantic alignment score and coverage ratio in the first alignment result are lower than a preset consistency threshold. If so, a semantic inconsistency flag is output, and a second determination result is generated together with the semantic expression pair and the semantic alignment score.

[0010] The agent fusion module constructs multiple detection agents, performs anomaly determination correlation analysis on the image semantic vector, speech semantic vector, first alignment result, and second determination result, and generates a score value used to measure the degree of anomaly consistency. This score value serves as the fault confidence score of the agent output. The confidence-weighted fusion is then performed using a weighted model based on the stability and historical accuracy of the agent output to output a comprehensive fault judgment conclusion representing the degree of global anomaly consistency.

[0011] The explanation and display module marks risk areas in the power equipment image based on the aligned abnormal areas involved in the comprehensive fault judgment conclusion, extracts inspection description fragments with semantic deviations in the speech, and generates visual explanation information output for manual interpretation.

[0012] In a preferred embodiment, when generating speech semantic vectors, the semantic extraction module performs power entity extraction and contextual semantic dependency analysis based on the speech-to-text results, constructs a semantic dependency graph through the context window, and uses the nodes in the graph to represent power terms and the edges to represent the semantic paths between terms, and then uses a graph embedding algorithm to generate speech semantic vectors. The graph embedding algorithm uses one of the graph convolutional network method, the attention path propagation method or the structured recursive coding method to ensure that the speech semantic vectors accurately reflect the logical relationship between sentences and the correlation strength of power terms in the structural dimension.

[0013] In a preferred embodiment, the semantic extraction module performs multi-layer feature extraction and region label classification operations when generating an image semantic vector, semantically annotates the power equipment area contained in the image and assigns a structural label. Based on the prior knowledge of the structural layout and connection mode of the power equipment, a preset target detection model is used to identify suspected abnormal areas in the image and determine their boundary positions. After the boundary area is determined, a fusion mechanism is used to combine and model the local image features within the boundary with the spatial structural features of the entire image to form a structural vector. This structural vector is then weighted and merged through an attention mechanism that assigns weights to features of different regions to ultimately generate an image semantic vector.

[0014] The fusion mechanism is implemented by combining one or more of the following three methods:

[0015] First, when there are multiple overlapping bounding boxes of abnormal regions, the region with the largest response value is selected as the main discrimination region, and the other bounding boxes are merged or suppressed;

[0016] Second, different weights are assigned to different channels in the feature map channel;

[0017] Third, the regional location information mask is introduced to mark the location of key areas.

[0018] In a preferred embodiment, when calculating the semantic alignment score, the semantic alignment module uses a distance metric calculation method in a vector space based on the similarity relationship between the image semantic vector and the speech semantic vector, specifically a Euclidean distance calculation, a cosine angle calculation, or one of the matching score functions learned and generated by the mapping network, to generate a first matching score value;

[0019] Comparing all semantic items contained in the speech semantic vector with semantic items in the image semantic vector that can establish an explicit matching relationship, counting the proportion of the number of matching semantic items, and generating a second coverage ratio value;

[0020] During the coverage ratio calculation process, semantic items are segmented and labeled according to their contextual position order to avoid cross-segment mismatching and improve the accuracy of the matching ratio;

[0021] The semantic alignment module compares the first matching score value and the second coverage ratio value with their respective preset thresholds. When any indicator is lower than the corresponding threshold, it is determined that the semantic inconsistency condition is met, and the result is passed to the semantic judgment module as the basis for judging the first alignment result.

[0022] In a preferred embodiment, the semantic determination module classifies the semantic expression pairs and the alignment scores in the second determination result into three consistency levels: complete consistency, partial consistency, and obvious inconsistency. A semantic inconsistency flag is generated only for the "obvious inconsistency" level, and the flag, along with the semantic expression pairs and the semantic alignment score, constitute the second determination result.

[0023] The level judgment adopts a multi-threshold segmentation strategy, and combines the semantic coverage ratio and the structural difference of the semantic dependency graph as auxiliary judgment references in the judgment, so as to enhance the positioning and credible judgment capabilities of inconsistent behaviors.

[0024] In a preferred embodiment, the detection agents in the agent fusion module include four categories: image agents, voice agents, alignment agents, and judgment agents. Each agent independently processes the image semantic vector, voice semantic vector, first alignment result, and second judgment result, and adopts a multidimensional indicator system to perform fault confidence scoring in its respective processing flow. The multidimensional indicator system includes three types of quantitative items: preset structural anomaly score, semantic mutation index, and consistency fluctuation amplitude. In the fusion stage, a dynamic weighting mechanism is used to adjust the agent contribution value according to the agent scoring result. The dynamic weighting mechanism includes a weighting model based on agent output stability, historical behavior performance, or combined confidence factors, which is used to generate a comprehensive fault judgment conclusion and retain the scores of each agent for subsequent calls to the explanation and display module.

[0025] In a preferred embodiment, the confidence-weighted fusion method in the dynamic weighting mechanism constructs a confidence factor based on the stability distribution of the historical scores of each detection agent. The confidence factor is generated by weighted calculation of the inverse function of the standard deviation and the historical average accuracy to form a joint scoring weight for each agent. The fault confidence scores of all agents are weighted and summed according to the joint scoring weight to obtain a comprehensive fault judgment conclusion. Based on this judgment, a one-time voting retention mechanism is used to determine the agent score with the highest confidence in the current round of scoring as the priority reference result, thereby improving the credibility and abnormal sensitivity of the fusion judgment in the case of multi-agent inconsistency.

[0026] In a preferred embodiment, when the explanation and display module generates a visual output of the image risk area, it constructs a fault risk heat map based on the image semantic vector and the abnormal trigger area index contained in the comprehensive fault judgment conclusion, and strengthens the visual response area through the saliency mapping method, and finally superimposes the heat map on the original image in a semi-transparent manner; corresponding to the extraction of the voice inspection description fragment, based on the semantic inconsistency mark and the positioning of key words at high-weight nodes in the voice dependency path, the voice text is trimmed through the fragment boundary calibration mechanism to form an explanation area, forming a visual explanation information output of a combination of pictures and text.

[0027] In a preferred embodiment, the explanation and display module further generates a textual fault verification report, which includes a comprehensive fault judgment conclusion, a description of the fault risk heat map location, a list of abnormal phonetic and semantic terms, and annotation text on the impact on the power fault judgment. Each part of the report structure is accompanied by a source reference index, which comes from the output results of the semantic extraction module, the semantic alignment module, and the agent fusion module respectively. The report can be exported as a standard format text file for manual review and operation and maintenance report archiving, thereby enhancing the integration efficiency and traceability of the detection system.

[0028] In a preferred embodiment, the comprehensive fault judgment conclusion includes at least the following two structured data parts: fault level classification results and abnormal trigger area index information; wherein the fault level classification results are divided into three levels: safe, suspicious, and obvious fault based on the comprehensive score value output by the agent fusion module; the abnormal trigger area index information is used to record the image semantic vector segment index and its corresponding alignment identification segment that are repeatedly judged as obvious fault levels in the multi-agent confidence score. The interpretation and display module locates the position range of the image semantic vector in the original power equipment image through the index information, and combines the speech semantic vector marked as the conflict area in the semantic expression pair to construct a cross-modal mapping path to realize the image hot zone annotation and text keyword marking of the risk area.

[0029] The technical effects and advantages of the present invention are as follows:

[0030] This invention builds a semantic extraction module to establish semantic vector representation structures for both power equipment images and operator speech content, achieving structured modeling of multimodal data within a unified semantic space. Image semantic vectors are generated for power equipment images through structural recognition and feature encoding, while speech semantic vectors are generated for operator speech content through speech-to-text conversion, power terminology parsing, and dependency graph construction. These semantic vectors are then encapsulated as semantic expression pairs and used as input for subsequent analysis. This provides a semantic hierarchical foundation for multimodal consistency judgment and enhances the system's ability to perceive deep associations within modal content.

[0031] Through the collaborative design of a semantic alignment module and a semantic judgment module, this paper establishes a cross-modal consistency judgment mechanism based on the dual indicators of matching score and coverage ratio. Furthermore, it introduces a multi-threshold grading strategy and auxiliary judgment logic based on semantic dependency structure differences, effectively improving the accuracy of semantic conflict judgment. The system can quantitatively grade and judge both subtle inconsistencies and significant conflicts between modal expressions, triggering fault identification output only in cases of significant inconsistency, effectively reducing the false alarm rate. It also provides a clear basis for subsequent explanations and displays, enhancing the interpretability and credibility of fault judgments.

[0032] This invention utilizes a multi-agent scoring mechanism and a dynamic confidence fusion method, assigning independent scoring to four types of agents for image features, speech content, alignment relationships, and decision structures. A confidence-weighted mechanism based on scoring stability and historical accuracy is incorporated into the fusion phase, and anomaly identification sensitivity is enhanced through a single-vote retention strategy. The resulting comprehensive fault judgment conclusion is highly structured and can directly drive the interpretation and display module to annotate fault risk hotspots and trace semantic anomaly keywords. It also supports the output of standardized text reports, effectively supporting the system's practicality and integration in scenarios such as power operation and maintenance audits. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0034] Figure 1 This is a schematic diagram of the multi-agent deep fake attack detection system for multimodal data in the present invention. DETAILED DESCRIPTION

[0035] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] Reference Figure 1 The following examples were obtained:

[0037] Example 1: The present invention is a multi-agent deep fake attack detection system for multimodal data. Focusing on the deep fault attack detection task for multimodal data, a multi-agent collaborative judgment system with semantic understanding and cross-modal alignment as the core is constructed. The system first performs structural recognition and semantic encoding on the image of power equipment through a semantic extraction process, and at the same time performs semantic analysis and vectorization conversion on the voice content of the operation and maintenance personnel to form a semantic expression pair, thereby achieving structural unification of different modalities at the semantic level. Subsequently, through the semantic alignment mechanism, combined with spatial mapping and semantic dimension normalization operations, the matching score and semantic coverage ratio between the image and the voice are calculated to identify potential inconsistent behaviors between modal expressions.

[0038] To address varying degrees of semantic conflict, the system further introduces a consistency grading mechanism and semantic judgment process, implementing multi-layered judgments on alignment results and supplementing assessments based on coverage ratios and differences in semantic dependency structures. The comprehensive judgment results are independently scored by multiple agents, each focusing on image features, speech expression, alignment relationships, and semantic conflicts, and performing fault confidence scoring based on multi-dimensional indicators. Ultimately, a structured, comprehensive fault judgment conclusion is formed through a confidence-weighted fusion approach. Dynamic weighting and confidence retention mechanisms are introduced into the fusion process to enhance the stability, responsiveness, and risk controllability of the judgment results.

[0039] After forming a fault diagnosis conclusion, the system uses an interpretation and display mechanism to annotate and semantically trace the risk areas, and constructs visual output and textual reports. Risk areas are located based on multi-agent confidence score overlays and semantic conflict indexes, displayed as heatmaps in the image, and abnormal segments are extracted in the voice dimension through dependency paths and keyword recognition. Relevant indicators are then output in a standardized report format, providing intuitive, accurate, and traceable fault interpretation assistance for manual interpretation and power assessment. Specifically, this includes the following:

[0040] The semantic extraction module performs structural recognition and feature encoding on power equipment images to generate image semantic vectors. It also performs speech-to-text conversion and power terminology parsing on the operator's speech content to generate speech semantic vectors and construct semantic expression pairs. This module undertakes the fundamental task of preprocessing and abstractly expressing input multimodal data (i.e., images and speech). Specifically, structural recognition of power equipment images involves converting the graphical structures and abnormal areas in two-dimensional / three-dimensional power equipment images into vector representations with semantic labels through image analysis algorithms, making the images comparable and capable of subsequent computational properties. Speech-to-text conversion of operator speech eliminates the unstructured nature of the language modality, and power terminology parsing further ensures the accuracy of the converted text with industry semantics. By constructing semantic expression pairs of "image semantic vectors" and "speech semantic vectors," the abstract migration of multimodal information from the perceptual modality to the semantic modality is completed. This abstract representation serves as the common input source for subsequent alignment and judgment operations and forms the starting point for the entire system logic. This step establishes the semantic expression foundation of images and speech, completing the bridge from raw perception to structured semantics between modalities. It is an irreplaceable preprocessing link for achieving cross-modal fault detection.

[0041] The semantic alignment module receives the image and speech semantic vectors from a semantic expression pair and, through spatial mapping and semantic dimension normalization, calculates a semantic alignment score and semantic coverage ratio to produce a first alignment result. This module receives the "semantic expression pair" output by the semantic extraction module and performs alignment analysis using the image and speech semantic vectors as input. Its core functions include: unifying the two modalities into a comparable semantic space through spatial mapping and semantic dimension normalization; generating a semantic alignment score to quantify the degree of semantic similarity using vector matching algorithms (such as Euclidean distance, cosine similarity, and deep matching functions); and calculating a semantic coverage ratio to assess whether important information in the speech semantics is adequately represented in the image semantics. The module's output, the "first alignment result," not only includes two key metrics but also provides direct data support for semantic consistency assessment. This step, for the first time, enables semantic consistency assessment between speech and image, providing a quantitative basis for subsequent fault determination and a key node in identifying logical deviations between modalities.

[0042] The semantic judgment module determines whether the semantic alignment score and coverage ratio in the first alignment result are lower than the preset consistency threshold. If so, it outputs a semantic inconsistency flag and generates a second judgment result together with the semantic expression pair and the semantic alignment score. This module receives the "first alignment result" from the semantic alignment module and performs logical judgment on it. Specific functions include: comparing the "semantic alignment score" and "semantic coverage ratio" with the preset consistency threshold respectively. If any indicator is lower than the threshold, the system automatically generates a semantic inconsistency flag; further forming a "second judgment result", which includes a combination of "semantic expression pair", "semantic alignment score" and "inconsistency flag" to enhance the contextual integrity of subsequent judgments; in addition, a three-level classification mechanism of "complete consistency, partial consistency, and obvious inconsistency" is introduced to semantically subdivide the consistency status and enhance the explanatory power of model judgment. This step is the core of the transformation from "analysis to judgment" in the entire process. It quantifies the modal deviation into a "fault probability level" through structured judgment criteria, forming the first layer of confirmation logic for deep fault risk.

[0043] The agent fusion module constructs multiple detection agents, each performing anomaly determination correlation analysis on image semantic vectors, speech semantic vectors, the first alignment result, and the second judgment result. Each agent generates a score to measure the degree of anomaly consistency, which serves as the fault confidence score output by the agent. This score is then combined using a confidence-weighted fusion model based on the stability and historical accuracy of the agent outputs, resulting in a comprehensive fault determination conclusion representing the degree of global anomaly consistency. This module incorporates multiple "detection agents," each responsible for data scoring from different perspectives: the image agent processes image semantic vectors; the speech agent processes speech semantic vectors; the alignment agent scores based on the first alignment result; and the judgment agent scores based on the second judgment result. Each agent uses a multi-dimensional indicator system, including a "structural anomaly score," a "semantic mutation index," and a "consistency fluctuation amplitude," to assign a fault confidence score to the input and form a sub-decision. Finally, a confidence-weighted fusion method is used to combine all agent outputs to form a unified comprehensive fault determination conclusion, which includes key information such as "fault level classification" and "anomaly triggering region index." This module is the system's "multi-agent collective judgment mechanism", which improves the system's anti-interference ability and judgment robustness in multi-scenario and multi-modal conflicts through the integration of agent collaboration and confidence mechanism.

[0044] The interpretation and display module marks risk areas within the power equipment image based on the aligned abnormal regions involved in the comprehensive fault judgment conclusion, extracts inspection description segments with semantic deviations from the speech, and generates visual interpretation information output for human interpretation. This module uses the "comprehensive fault judgment conclusion" as its core input and provides visualization and interpretation output: Based on the "abnormal trigger area index," it generates a fault heat map within the original power equipment image; uses a "saliency mapping method" to enhance hot zone response and ensure visual intuition; simultaneously, it extracts high-weight description segments located by "semantically inconsistent markers" on the speech side; forms a combined graphic and text interpretation information output, and further organizes and generates a standard "fault verification report text." This module bridges the gap between model output and operation and maintenance cognition and is a key manifestation of the intuitive assistance provided to operation and maintenance experts by the entire system, enhancing the system's explanatory power, credibility, and power adoptability.

[0045] When generating speech semantic vectors, the semantic extraction module performs power entity extraction and contextual semantic dependency analysis based on the speech-to-text results, constructs a semantic dependency graph through the context window, and uses nodes in the graph to represent power terms and edges to represent semantic paths between terms. Then, a graph embedding algorithm is used to generate speech semantic vectors. The graph embedding algorithm uses one of the graph convolutional network method, the attention path propagation method, or the structured recursive encoding method to ensure that the speech semantic vectors accurately reflect the logical relationship between sentences and the association strength of power terms in the structural dimension.

[0046] In the multi-agent deep fake attack detection system for multimodal data, the semantic extraction module is responsible for extracting structured semantics from the original speech modality. Its goal is to convert the verbal interpretation of the power equipment images by the operation and maintenance personnel into machine-readable and matchable speech semantic vectors to achieve subsequent alignment and judgment operations with the image semantics.

[0047] The module first performs a speech-to-text process based on the original speech input, which uses end-to-end automatic speech recognition technology to convert the speech signal into continuous readable natural language text.

[0048] After obtaining the text results, the module further performs power entity extraction. This process uses pre-trained power language models (such as PowerBERT and SubstationBERT) combined with industry power terminology dictionaries to identify key power concepts in the text. Identification results include, but are not limited to, equipment location (such as "main transformer high-voltage side"), anomaly type (such as "discharge traces" and "insulation aging"), and quantity and parameter values ​​(such as "3 megohms" and "5mm cracks").

[0049] Contextual semantic dependency analysis is then performed. This analysis parses the syntactic structure of the text to clarify the grammatical and semantic relationships between words in the sentence, specifically identifying the subordinate relationships between "descriptive words" and "equipment terms." For example, in the sentence "The image shows insulation aging in the lower part of the main transformer," "existence" is the core of the subject-predicate relationship, "insulation aging" is the described electrical entity, and "lower part of the main transformer" is its structural positional dependency.

[0050] Based on the results of dependency analysis, the system uses a context window mechanism to extract contextual segments around the power equipment entity to form local semantic units, and then constructs a semantic dependency graph in the entire text. The structure of the graph is a directed graph, in which: nodes represent the various types of power terms that have been identified; edges represent the path connections between these terms formed by contextual syntax and semantic dependencies, such as "graph structure → existence → fault". After the graph is constructed, the system introduces a graph embedding algorithm to vectorize the semantic graph. This embedding algorithm is implemented in one of the following three ways:

[0051] Graph Convolutional Network (GCN): This method aggregates the feature vectors of each node's neighbors and updates them, allowing the embedding vector to simultaneously encode local structure and global semantic diffusion information.

[0052] Attention path propagation method: This method is based on the weighted attention allocation mechanism of dependency paths and dynamically selects the path features that contribute most to the node to improve context discrimination;

[0053] Structured recursive encoding: This approach uses a deep recursive network based on a graph structure to propagate node states across the graph, enhancing the ability to retain long-range dependency information. Ultimately, the graph embedding algorithm outputs a set of vectors, which are the speech semantic vectors. This vector not only contains the semantic representation of all power terms in the speech but also preserves implicit relationships between terms, such as structure, dependency, and position.

[0054] This speech semantic vector is then used as input to form a "semantic expression pair" with the image semantic vector for use in the subsequent semantic alignment module. In this way, the system achieves a complete migration of the original unstructured speech content into a structured semantic space, improving the accuracy and reliability of cross-modal alignment analysis.

[0055] Assume that the original speech is: "The operator reported that there is a crack of about 2.8 cm on the lower part of the high-voltage side of the main transformer, which may be a precursor to breakdown." After the above process, the system will extract the following semantic units: node example: "lower part of the high-voltage side of the main transformer" (pictured part); "crack" (description of structural defects); "precursor of breakdown" (nature / potential risk inference); dependency path example: "crack" ← "existence" ← "lower part of the high-voltage side of the main transformer"; "crack" ← "may be" → "precursor of breakdown". After embedding, the system generates a set of semantic vectors to encode the structural relationship and provide them for semantic alignment and matching analysis of the "image semantic vector".

[0056] When generating image semantic vectors, the semantic extraction module performs multi-layer feature extraction and region label classification operations, semantically annotating the power equipment areas contained in the image and assigning structural labels. Based on prior knowledge of the power equipment's structural layout and connection methods, a preset target detection model is used to identify suspected abnormal areas in the image and determine their boundary locations. After the boundary area is determined, a fusion mechanism is used to combine and model the local image features within the boundary with the spatial structural features of the entire image to form a structural vector. This structural vector is then weighted and merged through an attention mechanism that assigns weights to features of different regions to ultimately generate the image semantic vector.

[0057] The fusion mechanism is implemented by combining one or more of the following three methods:

[0058] First, when there are multiple overlapping bounding boxes of abnormal regions, the region with the largest response value is selected as the main discrimination region, and the other bounding boxes are merged or suppressed;

[0059] Second, different weights are assigned to different channels in the feature map channel;

[0060] Third, the regional location information mask is introduced to mark the location of key areas.

[0061] In a multi-agent deepfake attack detection system for multimodal data, one of the core functions of the semantic extraction module is to generate image semantic vectors. This vector is a structured representation that describes the regional features and semantic content of fault diagnosis-relevant regions in power equipment images. It serves as one of the data input foundations for subsequent modules such as semantic alignment, semantic judgment, and agent fusion.

[0062] The generation of image semantic vectors includes the following stages:

[0063] Multi-layer feature extraction and regional label classification: This process uses a multi-layer neural network structure to extract features layer by layer from the input power equipment image, including low-level edge and material texture features and high-level structural semantics and spatial distribution features. Common implementations include ResNet, UNet, or their variants used in industrial image processing.

[0064] During feature extraction, the system performs region labeling, semantically annotating different equipment regions within the image (e.g., main transformer, cable connector, insulator, circuit breaker module, etc.). Each region is assigned an electrical structure label, such as "main transformer high-voltage side," "switchgear compartment," or "cable terminal head," for subsequent device structure location and semantic comparison.

[0065] Object detection algorithm based on power structure priors: Building on the aforementioned multi-layer features, the system further incorporates prior information about the structure of power equipment. This information includes existing knowledge of the physical layout and relationships of power equipment within the image, such as busbar layout, equipment connection methods, and insulation zoning logic. This prior knowledge is embedded into the object detection model through pre-training or structural mapping. Common detection algorithms include YOLOv5, RetinaNet, and variants specifically designed for power grid inspection, which are used to quickly identify potential abnormal areas within the equipment.

[0066] The object detection algorithm identifies and determines the location and boundaries of potentially abnormal areas. This algorithm generates a bounding box for each suspected abnormal area, including coordinates, dimensions, and associated graphical area. This bounding box information is the key physical location foundation for image semantic analysis.

[0067] Regional feature fusion mechanism: After detecting boundaries, the system further encodes the power equipment image through a regional feature fusion mechanism. The core purpose is to fuse local features within the boundary (i.e., subtle texture variations, material heterogeneity, and structural damage patterns within the equipment's potential anomaly area) with overall image features (e.g., equipment structural symmetry, clarity of electrical component arrangement boundaries, and overall grayscale or thermal imaging distribution patterns) to form a structural vector representation with contextual semantic associations. This structural vector integrates the local anomaly features of the equipment image with its overall spatial semantic structure. It serves as a high-level semantic expression unit for subsequent semantic matching, cross-modal alignment, and forgery detection analysis. It serves as a bridge between the "local details + global layout" linkage in image semantic modeling.

[0068] Attention Mechanism Optimization Weighted Merging: After the regional feature fusion mechanism completes the construction of the structure vector, the system introduces an attention mechanism to perform weighted optimization on multiple structure vectors to generate the final image semantic vector. This attention mechanism, a key component of the image semantic expression stage, assigns differentiated weights to the regional features contained in the structure vector, thereby enhancing the model's ability to focus on key abnormal regions and reducing the interference of irrelevant regions on the final judgment results. The attention mechanism in this invention establishes a set of weighted coefficients based on the importance judgment between each structure vector. The calculation of attention weights can be based on feature significance indicators (such as gradient change amplitude, semantic concentration, historical anomaly relevance, etc.) to construct an attention scoring matrix and apply a corresponding numerical weight to each structure vector. The system can adopt channel attention, spatial attention, or joint attention mechanisms to perform significance assessment and different weight adjustments from the channel dimension, spatial dimension, or a combination of the two dimensions.

[0069] Specifically, the channel-wise attention mechanism weights different semantic channels in the image feature map (such as the infrared heat signal channel, edge contour channel, and color shift channel) to emphasize responsiveness to channels directly related to device state variations. The spatial attention mechanism evaluates regional saliency in the image region dimension based on the spatial position of the structure vector, improving the representation accuracy of key areas. The joint attention mechanism fuses channel-wise and spatial-level weight matrices to capture the interactive signals between abnormal expression and structural mutations at a higher level. Finally, the system weights and combines multiple structure vectors according to the weights assigned by the attention mechanism to output an image semantic vector with regional saliency. This image semantic vector not only reflects responsiveness to multiple abnormal regions but also integrates semantic information across the entire spatial distribution, serving as input for subsequent semantic consistency matching and forgery detection with the speech semantic vector. The introduction of the attention mechanism significantly improves the system's sensitivity to high-risk areas and improves the accuracy of semantic representation. It avoids the semantic weakening and judgment ambiguity associated with averaging multiple structure vectors, ensuring that the image representation results are more discriminative in clinical or inspection scenarios.

[0070] After the regional feature fusion mechanism completes the construction of the structural vector, the system introduces an attention mechanism to perform weighted optimization on multiple structural vectors, aiming to generate image semantic vectors with regional saliency. This attention mechanism primarily operates during the semantic expression phase, assigning different weights to the image regions corresponding to each structural vector. This guides the model to automatically focus on important regional features related to potential fault manifestations, thereby enhancing the accuracy and discriminative power of semantic modeling.

[0071] The weight generation process in the attention mechanism is performed by a trainable submodule. During the system training phase, this module uses a supervised learning strategy to fit parameters to historically annotated samples to learn the importance mapping relationship between structure vectors, thereby forming an attention weight generation function. During actual operation, the system dynamically calculates the attention coefficient for each structure vector based on the set of structure vectors in the current input image to guide the subsequent weighting process. The basis for weight generation is a set of feature saliency indicators, mainly including gradient change amplitude, semantic concentration, and historical anomaly relevance. Gradient change amplitude is obtained by calculating the mean or variance of the pixel gradient values ​​on the feature map of a local image region, which measures the intensity of texture abrupt changes or edge structure changes in the region. Semantic concentration is based on the density of the distribution of structure vectors in the semantic embedding space, such as the average Euclidean distance or cluster radius between multiple vectors, and is used to indicate the degree of focus of the semantic expression of the region. Historical anomaly relevance is based on the model's high-frequency recognition of such regions during system training, or the frequency of statistical anomaly levels in the historical confidence score distribution, and is used to measure the risk importance of the region in past samples. These indicators are used as input parameters and input into the weight generation network, and the attention weight matrix is ​​output after multi-layer mapping and normalization operations.

[0072] The implementation of the attention mechanism can include three types of structures: channel attention, spatial attention, or joint attention. Channel attention is weighted based on the importance differences between different semantic channels (such as thermal signal distribution, color deviation channel, and edge clarity); spatial attention implements saliency annotation based on the difference in response intensity of the image region position; and joint attention combines the above two mechanisms to jointly guide the model to enhance regional recognition capabilities in both channel and spatial dimensions. After obtaining the attention weights of all structural vectors, the system performs a weighted merging operation. Specifically, each structural vector is multiplied by its corresponding attention coefficient and then weightedly summed to form a semantically enhanced comprehensive image semantic vector for subsequent matching, alignment, and consistency judgment with the speech semantic vector. This mechanism significantly improves the model's semantic focusing ability in situations with multi-region anomalies, local interference, or visual inconsistency, and avoids the weakening of key features caused by the average fusion of structural information.

[0073] Implementation of regional feature fusion mechanism: The regional feature fusion mechanism used in this system to extract key regional information from power equipment images can be implemented by combining one or more of the following three methods to improve the recognition accuracy and expression ability of abnormal areas:

[0074] 1. Maximum response mechanism for overlapping areas: In fault detection tasks, when the system uses the target detection algorithm to mark multiple abnormal area bounding boxes in the image, and these bounding boxes overlap in space, redundant and ambiguous regional expressions may be generated. This mechanism analyzes the image response intensity corresponding to each bounding box (such as texture anomalies, the intensity of edge gradient changes), selects the area with the largest response value as the main discrimination area, and merges or suppresses other bounding boxes to ensure that the final feature expression is concentrated on the area with the strongest abnormal discrimination power. For example: When there are local ablation features in the transformer bushing area that overlap with background interference, the system can prioritize retaining the bounding box of the area showing typical discharge traces, and exclude interference from repeated or offset areas.

[0075] 2. Feature map channel attention weighting mechanism: Power equipment images usually contain significant features in multiple dimensions, such as hot spot areas, color deviations, structural contours, etc. In the image feature extraction process, each feature map channel corresponds to a visual signal type. In order to enhance attention to abnormal signals, this mechanism assigns learnable weights to all channels separately, guiding the model to automatically focus on the channels that are most relevant to equipment status changes. For example: When the system processes an image of a switch cabinet, the system can identify infrared intensity distribution channels or edge blur change channels related to temperature anomalies in the channel and assign them higher weights, thereby improving the recognition sensitivity of fault signs such as local overheating or structural fatigue.

[0076] 3. Position Information Mask Enhancement Mechanism: This mechanism embeds the position code of the detection area into the overall image representation by introducing explicit spatial position information. This means that the position of the detected key area in the image is marked with a mask that is consistent with the image size. This allows the model to perceive the "location of the target area" during subsequent feature fusion or semantic encoding, thereby enhancing the model's spatial understanding ability. For example, if a bulge in the cable sheath is found in the upper left area of ​​the image, the system marks this area as a high-response location in the position mask. The subsequent fusion module integrates this position information with the visual features to ensure that the model can distinguish that the anomaly belongs to the outer structure rather than background or edge false detections.

[0077] The above three mechanisms are all commonly used feature enhancement methods in this field. They have good adaptability and complementarity in power image processing scenarios. The system can flexibly select them according to the complexity of the scene, model structure and deployment requirements to achieve more accurate, stable and interpretable feature extraction effects for potential abnormal areas in power equipment images.

[0078] Fault type label supervision: During the entire model training phase, the system introduces power fault type labels, assigning classification labels to different types of equipment faults (such as "discharge traces," "insulation damage," and "loose connections"). These labels are used to supervise the distribution consistency of the structure vector along the semantic feature dimension, thereby improving the accuracy and comparability of semantic representation in power operations and maintenance. These supervision labels directly influence the generation of the structure vector, ensuring that the image semantic vector not only possesses spatial structural features but also has the ability to express distinguishable fault categories, enabling accurate cross-modal matching in the subsequent semantic alignment module.

[0079] For example, if the input is a substation inspection image, the system may identify the following structural information: structural label: "lower terminal area of ​​the main transformer"; bounding box information: [x1, y1, x2, y2]; regional features: dark color, surface cracks, and deformation marks; structural vector: combining these local features with the brightness distribution pattern of the entire image to encode a context-dependent expression vector. Through the attention mechanism, this region is given a higher response weight, and the final image semantic vector will include: spatial location, structural label, potential fault type (such as "insulation layer aging"), and feature response strength (for subsequent comparison). This image semantic vector is then paired with the corresponding speech semantic vector to form a cross-modal semantic expression pair, which is used by the "semantic alignment module" for consistency judgment and forgery detection.

[0080] When calculating the semantic alignment score, the semantic alignment module uses a distance metric calculation method in the vector space based on the similarity relationship between the image semantic vector and the speech semantic vector, specifically a Euclidean distance calculation, a cosine angle calculation, or one of the matching score functions learned and generated by the mapping network, to generate a first matching score value;

[0081] Comparing all semantic items contained in the speech semantic vector with semantic items in the image semantic vector that can establish an explicit matching relationship, counting the proportion of the number of matching semantic items, and generating a second coverage ratio value;

[0082] During the coverage ratio calculation process, semantic items are segmented and labeled according to their contextual position order to avoid cross-segment mismatching and improve the accuracy of the matching ratio;

[0083] The semantic alignment module compares the first matching score value and the second coverage ratio value with their respective preset thresholds. When any indicator is lower than the corresponding threshold, it is determined that the semantic inconsistency condition is met, and the result is passed to the semantic judgment module as the basis for judging the first alignment result.

[0084] In this system, the core task of the semantic alignment module is to perform structured similarity analysis and semantic correspondence judgment between the image semantic vectors generated by the semantic extraction module and the speech semantic vectors, thereby forming a cross-semantic consistency assessment result for multimodal content. This result will serve as the decision basis for the downstream "semantic judgment module". The input received by the semantic alignment module is the semantic expression pair generated by the semantic extraction module. This pair consists of two high-dimensional vectors:

[0085] Image semantic vector: A structured vector generated through equipment structure recognition, boundary detection, feature fusion and attention weighting mechanism. It expresses the spatial location, structural characteristics and fault risk attributes of key areas in power equipment images, and is used to support risk analysis and judgment at the image level.

[0086] Speech semantic vector: A graph-structured vector obtained through speech-to-text conversion by operation and maintenance personnel, identification of key power terminology, and semantic graph modeling algorithms (such as graph embedding). It expresses the relationships between fault judgment terms involved in speech recordings and their description logic order, and is used to construct a risk understanding structure under the language modality.

[0087] Because image semantic vectors and speech semantic vectors originate from the image and speech modalities, respectively, there may be some deviation between the two in terms of their original semantic dimensions, structural representations, and vector lengths. Therefore, before further comparative analysis, the semantic alignment module in the system must first perform spatial mapping and semantic dimension normalization to unify the two types of vectors into a comparable embedding space, thereby enabling subsequent semantic consistency determination.

[0088] Spatial mapping: Using linear transformations, projection matrices, or small neural networks, the two vectors are projected into a unified semantic embedding space. Dimension normalization: Normalizing the vector values ​​to a uniform range (e.g., between [0, 1]) ensures that subsequent similarity calculations remain stable within a reasonable numerical domain. This process ensures that subsequent similarity comparisons can be performed within a consistent representation framework.

[0089] After the representation is unified, the system performs a semantic alignment score calculation, which evaluates the overall similarity between the image semantic vector and the speech semantic vector. This process is based on a distance metric in vector space, and can be performed using one of the following three methods:

[0090] Euclidean distance calculation: measures the straight-line distance between two vectors in Euclidean space;

[0091] Cosine angle calculation: measures the cosine value of the angle between two vectors to reflect the degree of similarity in direction;

[0092] Matching score function generated by mapping network learning: The matching score function trained by concatenating two vectors through a neural network (such as MLP).

[0093] All three methods generate a first matching score value, which is used to quantify the overall matching degree of the two semantic vectors. The higher the value, the more similar they are.

[0094] In addition to overall similarity, the system also assesses whether the semantic terms in the speech are adequately covered in the image representation. This metric is called the semantic coverage ratio. The calculation process is as follows: the system extracts each semantic term (i.e., individual power terms such as "main transformer high-voltage side," "grounding anomaly," and "insulation layer aging") from the speech semantic vector. It then compares these terms with semantic terms in the image semantic vector for which explicit matches can be established. The system then counts the number of successfully matched semantic terms and calculates the ratio by the total number of semantic terms in the speech: semantic coverage ratio = number of matched terms ÷ total number of semantic terms. This matching relationship can be achieved through a nearest neighbor search in the semantic embedding space or an exact matching mechanism using term labels. To improve the accuracy of the coverage ratio assessment, the system incorporates a contextual position-ordered segmentation tagging mechanism into the calculation process. Specifically, the speech text is logically divided into paragraph units (e.g., a complete description or a group of related statements) based on context. During the comparison process, semantic matching is performed only within the same paragraph to avoid mixing semantic terms across segments, thereby reducing the risk of falsely high coverage.

[0095] For example, a speech message might read: "Discharge marks are found on the lower part of the main transformer"; an image might read: "Abnormal hot zone at the cable connector." These two segments should be classified as distinct semantic segments. Mixing them might lead to an erroneous judgment of high coverage, even though they are not actually consistent. This mechanism effectively improves the semantic localization accuracy of semantic coverage ratio assessment, significantly reduces the impact of mismatches on the final judgment, and ensures the stability and reliability of forgery detection.

[0096] The system combines the first matching score and the second coverage ratio to form the "first alignment result." Simultaneously, the module compares each metric against its own preset threshold. If any metric falls below the threshold, a semantic inconsistency condition is generated. This structural judgment result is then passed to the semantic judgment module for subsequent grading and fault diagnosis.

[0097] Assume that the image semantic vector contains the following content: equipment location: "high-voltage side of the main transformer"; fault type: "damaged insulation layer"; abnormal size: "about 2.5 cm"; and the corresponding speech semantic vector is expressed as: "burn marks appear in the cable joint area, and there may be a breakdown risk"; then in the alignment analysis: the Euclidean distance is farther → indicating that the overall similarity in the semantic space is lower; the number of matching terms is 0 (because "cable joint" and "burn marks" do not appear in the image expression); the semantic coverage ratio is calculated as: 0 / 3=0; based on the above alignment results, the system determines that there is a significant semantic inconsistency between the image and the speech expression, that is, the risk area described by the speech has no direct correspondence with the actual analysis area of ​​the image, which may be abnormal forged content or cross-device confusion.

[0098] The semantic judgment module classifies the semantic expression pairs and alignment scores in the second judgment result into three levels of consistency: complete consistency, partial consistency, and obvious inconsistency. A semantic inconsistency flag is generated only for the "obvious inconsistency" level, and this flag, together with the semantic expression pairs and semantic alignment scores, constitutes the second judgment result.

[0099] The level judgment adopts a multi-threshold segmentation strategy, and combines the semantic coverage ratio and the structural difference of the semantic dependency graph as auxiliary judgment references in the judgment, so as to enhance the positioning and credible judgment capabilities of inconsistent behaviors.

[0100] In a multi-agent deepfake attack detection system for multimodal data, the semantic judgment module receives the first alignment result from the semantic alignment module. Combining the semantic expression structure and similarity values, it determines the consistency level between the multimodal content and outputs a structured judgment result, the second judgment result. This module bridges the gap between semantic alignment analysis and downstream fault diagnosis, and its functionality directly impacts the stability and credibility of fault diagnosis.

[0101] The semantic judgment module bases its judgment on the "first alignment result," which consists of two key numerical items: a semantic alignment score, which quantifies the similarity between the image semantic vector and the speech semantic vector within a unified semantic space; a higher value indicates closer similarity; and a semantic coverage ratio, which measures the proportion of all terms included in the speech semantics that are matched in the image semantics, used to assess coverage completeness. The input also includes the original semantic expression pair—a combination of the image semantic vector and the speech semantic vector—to provide structural semantic context and assist in judgment.

[0102] The semantic judgment module performs consistency classification based on the above inputs, which is used to classify the semantic alignment status into the following three standard levels:

[0103] Fully consistent: The alignment score reaches a high threshold, and the coverage ratio is close to 100%; the image semantics is highly consistent with the speech description, which is commonly found in accurate consultation records;

[0104] Partially consistent: The alignment score is medium, and there are omissions in the coverage ratio; some descriptions are missing or blurred in the image, which is commonly found in incomplete or abstract expressions;

[0105] Significantly inconsistent: The alignment score is far below the threshold, and the coverage ratio is low; the key information described in the speech is not reflected in the image or there are obvious conflicts; this is the core basis for the system to determine potential failures.

[0106] Only when the judgment level is "significantly inconsistent", the module generates a semantic inconsistency identifier as an explicit signal of the possibility of deep failure. Subsequently, the module combines the following three items into the second judgment result: Semantic inconsistency identifier: Mark that there are serious conflicts in this multimodal content; Semantic expression pair: Provide the original semantic data of the context; Semantic alignment score: Provide a quantitative scoring result for reference by subsequent modules. This result is the core input structure for the proxy fusion module to score and the interpretation module to visualize.

[0107] When the module performs level division, it adopts a multi-threshold segmentation strategy. This strategy uses the following method: Set two segmentation thresholds (for example, T1, T2): If the score ≥ T1, mark it as "fully consistent"; if T2 ≤ score < T1, mark it as "partially consistent"; if the score < T2, mark it as "significantly inconsistent". This strategy can effectively avoid the problem of excessive sensitivity caused by binary judgment, and improve the stability of judgment and the ability to adapt to different data qualities.

[0108] Implementation example: Input data: Image semantic vector: contains "lower wiring area of the main transformer", "aging of the insulation layer", "3 cm crack"; Speech semantic vector: contains "mid-cable joint", "no obvious abnormality"; Judgment process: The semantic alignment score is low (the Euclidean distance is far, and the cosine angle is close to 0); The semantic coverage ratio is 0% (there is no semantic item match between the two); The semantic dependency graph structure is significantly different: there is a clear fault description in the image expression, while the speech indicates "no abnormality"; Judgment result: Consistency level: significantly inconsistent; The output structure is: Semantic inconsistency identifier; The corresponding semantic expression pair; The specific alignment score value; This result can be directly used as an important signal for the system to determine forgery or description errors, and provide a support basis for subsequent risk warning, graphic annotation, and report generation modules.

[0109] The detection agents in the agent fusion module include four categories: image agents, voice agents, alignment agents, and judgment agents. Each agent independently processes the image semantic vector, voice semantic vector, first alignment result, and second judgment result, and adopts a multidimensional indicator system to perform fault confidence scoring in its respective processing flow. The multidimensional indicator system includes three types of quantitative items: preset structural anomaly score, semantic mutation index, and consistency fluctuation amplitude. In the fusion stage, a dynamic weighting mechanism is used to adjust the agent contribution value according to the agent scoring result. The dynamic weighting mechanism includes a weighted model based on agent output stability, historical behavior performance, or combined confidence factors to generate a comprehensive fault judgment conclusion and retain the scores of each agent for subsequent calls to the explanation and display module.

[0110] The confidence-weighted fusion method in the dynamic weighting mechanism constructs a confidence factor based on the stability distribution of the historical scores of each detection agent. The confidence factor is generated by weighted calculation of the inverse function of the standard deviation and the historical average accuracy to form a joint scoring weight for each agent. The fault confidence scores of all agents are weighted and summed according to the joint scoring weight to obtain a comprehensive fault judgment conclusion. Based on this judgment, a one-time voting retention mechanism is used to determine the agent score with the highest confidence in the current round of scoring as the priority reference result, thereby improving the credibility and abnormal sensitivity of the fusion judgment in the case of multi-agent inconsistency.

[0111] In this system, the agent fusion module integrates fault identification results from multiple detection paths, enabling collaborative cross-modal fault risk assessment. It is a key technical step in the transition from "local analysis by path" to "global trusted assessment." Its core goal is to generate a comprehensive fault assessment conclusion with a clear structure, defined risk level, and strong interpretability by fusing information from multiple sources.

[0112] This module builds multiple detection agents (DetectionAgent) for performing fault confidence scoring, which are divided into the following four categories, each of which specializes in processing a data source:

[0113] Image agent: processes image semantic vectors to focus on potential signs of structural failure in image features;

[0114] Speech agent: processes speech semantic vectors and identifies language failure features such as semantic jumps and context breaks;

[0115] Alignment agent: processes the first alignment result, i.e., the semantic matching score and coverage ratio between image and speech, and determines multimodal consistency;

[0116] Decision Agent: This process processes the second decision result, including semantic inconsistency identification and alignment level, and analyzes the severity of the semantic conflict. Each type of agent runs independently and conducts parallel evaluations, forming a multi-source fault decision path.

[0117] Each agent uses a multi-dimensional indicator system consisting of three quantitative items when performing fault confidence scoring:

[0118] The structural anomaly score is used to assess whether the structural features of the region represented by the image semantic vector deviate from the distribution of standard power equipment diagrams, thereby identifying potential structural faults or counterfeit interventions in the image. It is calculated as follows: the geometric boundary outline of the identified abnormal region (i.e., the position and shape characteristics of the detected fault region's border) is extracted from the image semantic vector. This boundary information is then compared with the average shape template of standard power equipment diagrams (e.g., standard transformer terminal structures, cable connector geometry models, etc.). The Hausdorff distance is used as an evaluation metric, measuring the maximum and minimum distances between the region's boundary points and the standard shape template. Finally, this distance is normalized to the range [0, 1] to generate a structural anomaly score. A larger value indicates that the region's geometric form deviates more likely from the normal distribution of the equipment structure. For example, irregular extensions, deformed edges, or repeated structural interference may indicate a localized equipment fault, physical damage, or image manipulation.

[0119] The semantic mutation index measures the degree of jumps between semantic units in a speech semantic vector, reflecting any unusual breaks, jumps, or discontinuous expressions in the semantic logic. This helps identify potential splicing errors, fragment reassembly, or editing artifacts in speech content. The calculation method is as follows: Based on a semantic dependency graph constructed using a graph embedding approach, the dependency path lengths between all nodes (i.e., identified power terms or key descriptors) are calculated. The pair of adjacent key power semantic nodes in the graph with the largest path span (for example, the unusual span between "cable mid-section burn" and "normal operation") is identified. The path length difference between this pair of nodes is calculated and then compared with the average path length of all paths in the dependency graph to generate a normalized index. The formula is: semantic mutation index = maximum path length difference / average path length. A higher index indicates abrupt switches or unreasonable jumps in the logical structure of the speech expression, which is likely due to splicing, editing, or fabrication. Therefore, it is an important indicator for detecting forgery in the speech modality.

[0120] Consistency fluctuation is used to assess the degree of consistency variation in a detection agent's output over multiple rounds of evaluation, identifying instances of significant fluctuation in detection results for faulty samples. This calculation method involves recording the semantic alignment scores obtained for the same semantic expression pair across multiple time slices (or multiple detection rounds); calculating the standard deviation of this sequence to represent its fluctuation range; then calculating the mean of the sequence; and dividing the standard deviation by the mean to obtain the normalized fluctuation result. The formula is: Fluctuation = Standard Deviation / Mean. Larger values ​​indicate unstable detection scores, potentially due to hidden fault characteristics, and serve as a warning indicator of model signal instability.

[0121] After all agents have completed their scoring, the system implements a dynamic weighting mechanism to adjust the fusion contribution of different agents. The input of this mechanism is the agent scoring results, and its weighting rules can be based on any or a combination of the following methods to build a weighted model:

[0122] Agent output stability: agents with less rating fluctuations are more trustworthy;

[0123] Historical performance: Agents with high past judgment accuracy receive greater weight;

[0124] Combined confidence factors: Use statistics (such as variance, confidence interval) to construct joint weight coefficients.

[0125] The result of this mechanism is that each agent is assigned a different weight as its contribution factor for weighted fusion.

[0126] The core method of the dynamic weighting mechanism is the confidence-weighted fusion approach, which specifically performs the following steps: Constructing a confidence factor: Based on each agent's historical scoring results, statistically analyze the stability distribution of the score (such as standard deviation). The inverse of the standard deviation is used as the agent's fluctuation confidence factor; Weighted calculation with historical average accuracy: Multiply the previous step's result by the accuracy rate of the agent's past output results in hitting real faults to generate a joint scoring weight for each agent; Performing a weighted sum: Multiplying each agent's fault confidence score by its joint scoring weight, summing all the results, and generating a final comprehensive fault judgment conclusion with a probabilistic basis for interpretation.

[0127] To improve the credibility of the system's judgments in the event of multi-agent judgment conflicts, the system introduces a one-time voting retention mechanism, namely: among all agent scores in this round, the agent with the highest confidence factor (joint scoring weight) is determined; the score of this agent is regarded as the "main judgment result" and given priority reference weight in the fusion score; this mechanism ensures that when multiple agent outputs conflict, the trusted source dominates, improving the system's robustness and abnormal response capabilities in complex fault environments.

[0128] The fusion result is a structured object called a comprehensive fault judgment conclusion. It not only provides a grading system (e.g., "obvious fault" or "suspicious") but also includes: a weighted unified confidence score; the original scoring records from each agent; a confidence factor ranking and the leading agent index. This structure is directly used in the interpretation and display module to generate fault heatmaps and semantic annotations.

[0129] Assume that the data received by the system is summarized and sorted to obtain the agent score table as shown in Table 1.

[0130] Table 1: Agent Rating Table

[0131] Agent Type Rating value Standard deviation Historical accuracy Joint Weight Image Proxy 0.85 0.10 0.90 9.0×0.90=8.1 Voice Agent 0.40 0.25 0.60 4.0×0.60=2.4 Alignment Agent 0.30 0.15 0.50 6.6×0.50=3.3 Decision Agent 0.75 0.05 0.80 20.0×0.80=16.0

[0132] Based on the data in Table 1, the final composite score = weighted sum / weighted sum = (0.85 × 8.1 + 0.4 × 2.4 + 0.3 × 3.3 + 0.75 × 16.0) / (8.1 + 2.4 + 3.3 + 16.0) = 20.835 / 29.8 ≈ 0.698. The single-vote retention mechanism selects the decision agent (with the highest combined score weight), whose score serves as the preferred reference. Under this mechanism, the decision agent with the highest combined score weight (16.0) is selected from the four agents. The decision agent's score is 0.75, which serves as the subjective guide or preferred reference result. Assuming a score range of ≥0.80 corresponding to a safe level, ≥0.50 and <0.80 corresponding to a suspicious level, and <0.50 corresponding to a significant fault level, the composite score is ≈0.698∈[0.50–0.79], falling into the "suspicious" level. A single decision agent score of 0.75 also falls into the "suspicious" level, resulting in a suspicious level.

[0133] The comprehensive fault judgment conclusion includes at least the following two structural data parts: fault level classification results and abnormal trigger area index information; the fault level classification results are divided into three levels: safe, suspicious, and obvious faults based on the comprehensive score value output by the agent fusion module; the comprehensive score value refers to the forgery confidence scores output by multiple detection agents, which are aggregated through a confidence-weighted fusion mechanism, and is used to characterize the overall judgment strength of the cross-modal fault consistency anomaly in the current semantic expression pair; the abnormal trigger area index information is used to record the image semantic vector segment index and its corresponding alignment identification segment that have been repeatedly judged as obvious fault levels in the multi-agent confidence score. The interpretation and display module locates the position range of the image semantic vector in the original power equipment image through the index information, and combines the speech semantic vector marked as the conflict area in the semantic expression pair to construct a cross-modal mapping path to realize the image hot zone annotation and text keyword marking of the risk area.

[0134] When the explanation and display module generates a visual output of the image risk area, it constructs a fault risk heat map based on the image semantic vector and the abnormal trigger area index contained in the comprehensive fault judgment conclusion, and enhances the visual response area through the saliency mapping method. Finally, the heat map is superimposed on the original image in a semi-transparent manner; corresponding to the extraction of the voice inspection description fragment, the key words are located based on the semantic inconsistency mark and the high-weight nodes in the voice dependency path, and the voice text is trimmed through the fragment boundary calibration mechanism to form an explanation area, forming a visual explanation information output of the image and text combination.

[0135] The explanation and display module further generates a textual fault verification report, which includes a comprehensive fault judgment conclusion, a description of the fault risk heat map location, a list of abnormal phonetic and semantic terms, and annotations on the impact on power fault judgment. Each part of the report structure is accompanied by a source reference index, which comes from the output results of the semantic extraction module, semantic alignment module, and agent fusion module respectively. The report can be exported as a standard format text file for manual review and operation and maintenance report archiving, enhancing the integration efficiency and traceability of the detection system.

[0136] In this system, the interpretation and display module is responsible for outputting the structural analysis results of the fault detection process in a visual and textual form. This aims to provide understandable, traceable, and verifiable fault analysis results for maintenance experts and end users. The main inputs of this module are: comprehensive fault judgment conclusions and related intermediate data (such as image semantic vectors, speech semantic vectors, and semantic expression pairs).

[0137] The comprehensive fault judgment conclusion is output by the agent fusion module and consists of two key structural components:

[0138] Fault level classification results: This refers to mapping the fusion score into three levels: "safe", "suspicious", and "obvious fault". It is used to quickly locate the fault risk level and is derived from the weighted fusion of the confidence scores of each agent;

[0139] Abnormal trigger area index information: refers to the image semantic vector segment index that has been repeatedly judged as "obvious fault level" in multi-agent scoring and its corresponding alignment identifier segment in semantic alignment. This structure is used to locate the specific content segment involved in the fault behavior between the image and the voice.

[0140] Image-side visual output mechanism: The interpretation and display module first performs image dimension positioning processing on the "abnormal trigger area index information." Based on the index information, it searches for the corresponding image semantic vector segment in the original power equipment image. It then locates the image location of the fault risk area using parameters such as "spatial position," "bounding box coordinates," and "structural features" encoded in the image semantic vector. It then constructs a fault risk heat map, generating a response intensity layer for these risk areas in the image.

[0141] Saliency mapping methods (such as Grad-CAM and Layer-CAM) are introduced to enhance the visual focus response of the target area in the heat map. Finally, the heat map is superimposed on the original image in a semi-transparent manner to realize visual risk area prompts in the image layer, which is convenient for experts to intuitively identify.

[0142] In the speech dimension, the system identifies inspection description segments with semantic deviations through the following process: based on the semantic inconsistency mark previously generated by the semantic judgment module; backtracking the semantic dependency graph structure generated by the corresponding speech semantic vector; extracting high-weight nodes from the dependency graph (that is, power terms or verbs that have a judgment influence in the judgment process, such as "appear", "discover", etc.); applying the segment boundary calibration mechanism, with these key nodes as the center, expanding the word window forward and backward by a fixed number to form a syntactically complete explanation area segment; combining these explanation segments into a speech description anomaly list for visual display and textual archiving.

[0143] The image risk area heat map and the speech abnormality keyword fragments are jointly output to form the final visual explanation information output: the suspected tampering area is displayed through a color layer on the image; the abnormal terms in the corresponding description are highlighted in the text; and the "image-text" bidirectional index mapping is established for the structure through semantic expression to support cross-navigation and verification.

[0144] The explanation and display module further organizes the above results into a structured fault detection report, the structure of which includes: comprehensive fault judgment conclusion (including fault level and comprehensive score); fault risk heat map location description (index area coordinates, structural characteristics, response intensity level, etc.); a list of abnormal speech semantic terms (keywords, dependency paths, and sentence segments); annotation text that affects the judgment of power faults. The system generates prompt instructions based on the semantics of the conflicting area, such as "The description in the speech does not match the image area, which may mislead the fault location judgment."

[0145] Each part of the report content is accompanied by a reference index, marking its source module. The image content reference source → image semantic vector output by the semantic extraction module; modal alignment source → semantic item matching result in the semantic alignment module; risk level source → fault level result in the agent fusion module; the report can be automatically exported to a standard format text file (such as PDF, DOC, HL7 annotated report, etc.) for manual review or power operation and maintenance archive archiving.

[0146] Suppose the system detects the following: a fault risk heat map area located on the "high-voltage side of the main transformer," approximately 3cm x 4cm in size; and a corresponding voice description containing "cable joint hot zone, potential breakdown risk." Due to the significant discrepancy in semantic alignment, the system classifies the semantic inconsistency as "significantly inconsistent." The system then annotates the "high-voltage side of the main transformer" area in the image, coloring it dark red to highlight the faulty area; and labels key terms such as "cable joint hot zone" and "potential breakdown risk" in the text. The report outputs: "The description location is inconsistent with the actual fault area in the image, indicating a risk of misjudgment." This report can be provided to operations and maintenance personnel to alert them to possible equipment fault confusion and facilitate further investigation and confirmation.

[0147] Further explanation: To enable the system to visually label suspected deep fault areas in power equipment images, the interpretation and display module combines the image semantic vector with the abnormal trigger area index information contained in the comprehensive fault judgment conclusion to further construct a fault risk heat map that can be intuitively understood by operation and maintenance experts. The areas marked on the heat map are identified as risk areas by the system, and the saliency mapping method is used to improve the pertinence and clarity of the visual response.

[0148] The risk area refers to the image area in the semantic structure of the power equipment image that is detected by the system as having serious semantic inconsistencies with the voice description and is repeatedly judged as having obvious fault levels in the agent scoring results. It specifically comes from the "abnormal trigger area index information" in the comprehensive fault judgment conclusion.

[0149] The index information includes the following: the structure segment ID in the image semantic vector (such as "nth block vector"); the corresponding spatial boundary coordinates (such as coordinates [x1, y1, x2, y2]); the number of times and scores marked as faulty by multiple agents; and the indices of semantic items that failed to align with the speech semantic vector. Therefore, the spatial extent of the risk area, the source of the semantic structure, and the fault judgment can all be clearly recorded in the data.

[0150] The saliency mapping method in this system is a visualization technology used to highlight the areas of interest of the neural network, which is used to help explain why the system determines that a certain area is a fault risk area. The specific implementation method is as follows:

[0151] Feature map reverse gradient calculation: retaining the importance response map used to judge image features in the proxy model, such as the output of the last convolution layer in a convolutional neural network (CNN);

[0152] Backpropagate the fault level determination result and calculate the gradient value of the output score for each channel in the convolution feature map (indicating the sensitivity of the channel to the determination result);

[0153] Channel importance weighting: The average of the channel gradients is used as the weight of each feature map, multiplied by the original feature map, and summed to form a significant response map;

[0154] Heatmap normalization: normalize the saliency map to a min-max value and scale it to [0, 1]. A higher value indicates a higher level of attention paid to the area.

[0155] Fusion with risk area boundaries: This saliency response map is superimposed at the image location corresponding to the abnormal trigger area index to construct a fault heat map.

[0156] Color mapping output: Maps the normalized response values ​​to visual colors, for example using the Jet color map (blue-green-red), where red indicates the strongest response and blue indicates the weakest. This method can be implemented using existing technologies such as Grad-CAM, Layer-CAM, and Score-CAM, with the specific settings determined during system deployment.

[0157] When generating a fault verification report, the system will output the highest response area level in the heat map, the response value distribution map, and the location coordinate description to assist manual analysis. Assume that the system determines that the fault is "obvious fault", and the 5th block in the image semantic vector is marked by multiple detection agents, with spatial coordinates of [x1, y1, x2, y2]. Through the significance mapping method, the maximum response value of this area in the response map is 0.82, and this value falls into the "high response level" interval. The system will overlay this area with red and add a record to the fault verification report: "Area 5 of the image (the terminal area on the high-voltage side of the main transformer) is a high-response fault area, with a significance value of 0.82; This report provides clear fault location information for operation and maintenance personnel to review, which is helpful for further fault confirmation and subsequent maintenance operations.

[0158] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0159] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0160] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A multi-agent deep fake attack detection system for multimodal data, characterized by: include: The semantic extraction module performs structural recognition and feature encoding on power equipment images to generate image semantic vectors. It also performs speech-to-text conversion and power terminology analysis on the speech content of operation and maintenance personnel to generate speech semantic vectors and construct semantic expression pairs. The semantic alignment module receives the image semantic vector and the speech semantic vector in the semantic expression pair, calculates the semantic alignment score and the semantic coverage ratio through spatial mapping and semantic dimension normalization operations, and forms a first alignment result; A semantic determination module determines whether the semantic alignment score and coverage ratio in the first alignment result are lower than a preset consistency threshold. If so, a semantic inconsistency flag is output, and a second determination result is generated together with the semantic expression pair and the semantic alignment score. The agent fusion module constructs multiple detection agents, performs anomaly determination correlation analysis on the image semantic vector, speech semantic vector, first alignment result, and second determination result, and generates a score value used to measure the degree of anomaly consistency. This score value serves as the fault confidence score of the agent output. The confidence-weighted fusion is then performed using a weighted model based on the stability and historical accuracy of the agent output to output a comprehensive fault judgment conclusion representing the degree of global anomaly consistency. The interpretation and display module marks risk areas in the power equipment image based on the alignment of abnormal areas involved in the comprehensive fault judgment conclusion, extracts inspection description segments with semantic deviations in the speech, and generates visual interpretation information output for human interpretation; When calculating the semantic alignment score, the semantic alignment module uses a distance metric calculation method in the vector space based on the similarity relationship between the image semantic vector and the speech semantic vector, specifically a Euclidean distance calculation, a cosine angle calculation, or one of the matching score functions learned and generated by the mapping network, to generate a first matching score value; Comparing all semantic items contained in the speech semantic vector with semantic items that establish an explicit matching relationship in the image semantic vector, counting the proportion of matching semantic items, and generating a second coverage ratio value; During the coverage ratio calculation process, the semantic items are segmented and marked in the order of their contextual positions. The semantic alignment module compares the first matching score value and the second coverage ratio value with their respective preset thresholds. When any indicator is lower than the corresponding threshold, it is determined that the semantic inconsistency condition is met, and the result is passed to the semantic judgment module as the basis for judging the first alignment result.

2. The multi-agent deep fake attack detection system for multimodal data according to claim 1 is characterized in that: When generating speech semantic vectors, the semantic extraction module performs power entity extraction and contextual semantic dependency analysis based on the speech-to-text results, constructs a semantic dependency graph through the context window, and uses nodes in the graph to represent power terms and edges to represent semantic paths between terms. Then, a graph embedding algorithm is used to generate speech semantic vectors. The graph embedding algorithm uses one of the graph convolutional network method, the attention path propagation method, or the structured recursive coding method.

3. The multi-agent deep fake attack detection system for multimodal data according to claim 2 is characterized in that: When generating image semantic vectors, the semantic extraction module performs multi-layer feature extraction and region label classification operations, semantically annotating the power equipment areas contained in the image and assigning structural labels. Based on prior knowledge of the power equipment's structural layout and connection methods, a preset target detection model is used to identify suspected abnormal areas in the image and determine their boundary locations. After the boundary area is determined, a fusion mechanism is used to combine and model the local image features within the boundary with the spatial structural features of the entire image to form a structural vector. This structural vector is then weighted and merged through an attention mechanism that assigns weights to features of different regions to ultimately generate the image semantic vector. The fusion mechanism is implemented by combining one or more of the following three methods: First, when there are multiple overlapping bounding boxes of abnormal regions, the region with the largest response value is selected as the main discrimination region, and the other bounding boxes are merged or suppressed; Second, different weights are assigned to different channels in the feature map channel; Third, the regional location information mask is introduced to mark the location of key areas.

4. The multi-agent deep fake attack detection system for multimodal data according to claim 3 is characterized in that: The semantic judgment module implements consistency level classification for the semantic expression pairs and alignment scores in the second judgment result. According to the preset strategy, the levels are divided into three levels: complete consistency, partial consistency, and obvious inconsistency. A semantic inconsistency mark is only generated for the obvious inconsistency level, and the combination of the mark, the semantic expression pair and the semantic alignment score is used as the second judgment result.

5. The multi-agent deep fake attack detection system for multimodal data according to claim 4 is characterized in that: The detection agents in the agent fusion module include four categories: image agents, speech agents, alignment agents, and judgment agents. Each agent independently processes the image semantic vector, speech semantic vector, first alignment result, and second judgment result, and uses a multidimensional indicator system to perform fault confidence scoring in its respective processing flow. The multidimensional indicator system includes three quantitative items: preset structural anomaly score, semantic mutation index, and consistency fluctuation amplitude. In the fusion stage, a dynamic weighting mechanism is used to adjust the agent contribution value based on the agent scoring results.

6. The multi-agent deep fake attack detection system for multimodal data according to claim 5 is characterized in that: The structural anomaly score is calculated as follows: the geometric boundary outline of the identified abnormal area is extracted from the image semantic vector and then compared with the average shape template in the standard power equipment diagram. The Hausdorff distance is used as the evaluation metric to measure the maximum and minimum distance between the current area boundary point set and the standard shape template. Finally, this distance value is normalized to the range [0, 1] to generate the structural anomaly score. The semantic mutation index is calculated as follows: Based on the semantic dependency graph constructed by graph embedding, the dependency path lengths between all nodes are calculated. The pair of adjacent key power semantic nodes with the largest path span is found in the graph. The path length difference between this pair of nodes is calculated and then compared with the average length of all paths in the entire dependency graph to generate a normalized index. The calculation formula is: semantic mutation index = maximum path length difference ÷ average path length. The consistency fluctuation range is calculated by recording the semantic alignment scores obtained for the same semantic expression pair at multiple time slices to obtain a semantic alignment score sequence. The standard deviation of the sequence is calculated to represent its fluctuation range, and then the average value of the sequence is calculated. The standard deviation is divided by the average value to obtain the normalized fluctuation range. The formula is: consistency fluctuation range = standard deviation / average value. In the fusion stage, a dynamic weighting mechanism is used to adjust the agent contribution value based on the agent scoring results. This means that the input of this mechanism is the agent scoring result, and a different weight is assigned to each agent according to its preset weighting rules as its contribution factor for weighted fusion.

7. The multi-agent deep fake attack detection system for multimodal data according to claim 6 is characterized in that: The core method of the dynamic weighting mechanism is the confidence weighted fusion method, which specifically performs the following steps: Constructing a confidence factor: Based on each agent's historical scoring results, the stability distribution of the statistical score, i.e., the standard deviation, is calculated, and the inverse of the standard deviation is used as the agent's volatility confidence factor; Weighted calculation with historical average accuracy: Multiply the result of the previous step by the accuracy of the corresponding agent's past output results in hitting real faults to generate a joint scoring weight for each agent; Perform weighted summation: Multiply each agent's fault confidence score by its joint score weight, sum all the results, and generate the final comprehensive fault judgment conclusion.

8. The multi-agent deep fake attack detection system for multimodal data according to claim 7 is characterized in that: The comprehensive fault judgment conclusion includes at least the following two structural data parts: fault level classification result and abnormal triggering area index information; The fault level classification results are based on the comprehensive score output by the agent fusion module and are divided into three levels: safe, suspicious, and obvious faults; The comprehensive score value refers to the forgery confidence scores output by multiple detection agents, which are aggregated through a confidence-weighted fusion mechanism; The abnormal trigger region index information is used to record the image semantic vector segment index and its corresponding alignment identification segment that are repeatedly judged as obvious fault levels in the multi-agent confidence score.

Images