Multi-modal semantic federation learning complex working condition equipment fault diagnosis method

Abstract

The invention discloses a multi-modal semantic federation learning complex working condition equipment fault diagnosis method, which comprises the following steps that: a client extracts initial characteristics by using independent encoders of different modals, and realizes intra-modal redundancy removal through abnormal region mapping and spatial pyramid compression; in a cross-modal stage, utilizing a region-guided semantic interaction mechanism to reinforce local anomaly characterization and obtain fusion features; an anomaly scoring network is constructed, anomaly scores are distributed to the marks, and local representation with the maximum information gain is optimized by combining terminal bandwidth constraint and task emergency degree; and the client performs local decoding and prediction, and the server performs federal average aggregation and broadcasts update parameters. According to the method, under the constraint that original data of each client cannot be shared, through federal cooperation of semantic perception, the accuracy rate in a recognition task of multiple public data sets can be improved by 12.69%, the single-round communication overhead can be reduced by 71.04%, the overall efficiency of a complex working condition diagnosis system is remarkably improved, and the method has a wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of complex working condition equipment fault detection, and particularly relates to a multi-modal semantic federated learning complex working condition equipment fault diagnosis method. BACKGROUND

[0002] With the continuous expansion of the scale of intelligent systems and the continuous complexity of the running environment, the number of various complex working condition equipment has increased rapidly, and the safety monitoring demand of its running state has become increasingly urgent. Such equipment is widely distributed in typical automation scenes such as industrial production, energy transmission and distribution, agricultural operation and infrastructure, and is often in complex environments such as high temperature, high humidity, high load or multi-source interference, with harsh running conditions, and is prone to overheating, performance degradation or structural abnormalities and other hidden dangers. If not detected and processed in time, it may cause serious equipment failure and system paralysis. The traditional manual inspection method has problems such as low efficiency, high risk and incomplete coverage in complex working conditions, while intelligent inspection robots have higher flexibility and safety in complex environments and have become a new trend in complex working condition inspection. However, complex working condition equipment often has potential faults such as thermal abnormalities, mechanical fatigue or energy consumption abnormalities, which are characterized by multi-modal sensing features (such as thermal imaging, images, and sound), and require multi-source information fusion for accurate analysis, increasing the diagnosis complexity. At the same time, these robots are widely distributed and are subject to network bandwidth and privacy protection constraints, and cannot upload all raw data centrally. How to realize accurate multi-modal fault diagnosis while ensuring efficient communication and data security has become a core challenge in current complex working condition intelligent inspection.

[0003] Current research on complex working condition equipment fault diagnosis mainly focuses on two aspects: one is single-modal diagnosis method, such as thermal spot recognition based on infrared images, surface defect detection based on image processing, and partial discharge recognition based on audio signals, which has limited accuracy and is difficult to adapt to multi-source complex scenes; the other is multi-modal fusion method, such as using deep neural networks to jointly model images and infrared data, but most of them are based on centralized training and lack consideration of privacy protection and edge deployment. In recent years, federated learning methods have been gradually introduced into complex working condition systems, such as load prediction and complex working condition fault classification, but there is still a blank in multi-modal semantic modeling and communication optimization. Therefore, it is urgent to build a federated learning method framework that can effectively fuse multi-modal information, have semantic perception and reasoning ability, and reduce communication cost while protecting data security.

[0004] Although there are some researches on complex working condition equipment fault diagnosis, most of them focus on federated modeling or simple data level fusion under a single mode. In the actual complex working condition inspection scene, the data faced by the inspection robot not only comes from various sources, but also has inconsistent semantics. How to realize cross-device and cross-modal information fusion, and build a unified and interpretable hazard source detection model while ensuring data privacy has become the core problem that current researches urgently need to break through. First, due to the complex environment of each robot device and the differences in the distribution of modal data, it is difficult for the global model to learn a unified semantic expression. Therefore, multi-modal perception has become a key means to improve fault diagnosis accuracy, and the fusion of visual, thermal imaging and audio modal information has been proven to significantly enhance the robustness and context understanding ability of the model. Then, distributed inspection robots are deployed on multiple edge nodes, and there are problems of privacy leakage and communication burden in uploading original data. Centralized modeling is not feasible. Therefore, in view of the data security and communication constraints, federated learning (FL) technology is introduced into the collaborative modeling between edge devices, which effectively alleviates the risk of data privacy leakage. Finally, in order to reduce the communication overhead of anomaly detection, semantic communication is used to realize data compression, and only the consistent information of the characteristic semantics is uploaded, so as to ensure that the entire federated framework has high efficiency in real scenes.

[0005] Therefore, it is necessary to design a new fault diagnosis framework supporting multi-modal semantic understanding, federated collaborative training and high communication efficiency. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a multi-modal semantic federated learning complex working condition equipment fault diagnosis method in view of the defects in the prior art.

[0007] The technical scheme adopted by the present application to solve its technical problem is: The present application provides a multi-modal semantic federated learning complex working condition equipment fault diagnosis method, which regards each inspection robot for complex working condition equipment as a client, and constructs a regional perception multi-modal semantic federated learning framework through each client and a center server. The method comprises the following steps: Step 1, a regional perception multi-modal semantic fusion mechanism is executed on the client side, and each client respectively extracts initial features by using independent encoders for different modes. After the initial features of each mode are extracted, the intra-modal redundancy is removed through abnormal region mapping and spatial pyramid compression, and the local anomaly representation is strengthened by using region-guided semantic interaction in the cross-modal stage to obtain a compact structure and semantic prominent fusion feature sequence. Step 2: The client performs local semantic communication optimization processing based on the fused feature sequence; an anomaly scoring network is used to calculate an anomaly score for each label in the fused feature sequence, and an incremental scoring mechanism is introduced. Based on the anomaly score and the marginal value of candidate labels on the basis of the current selected label set, the incremental scoring function is used as a guide; and combined with the client's bandwidth constraints and task urgency, a greedy Top-K strategy is used to select key labels, dynamically determine the upper limit of the number of labels uploaded in each round of federation, and obtain the set of labels with the highest task relevance. Step 3: The client inputs the set of tags with the highest task relevance into the detection head for local decoding and prediction; after receiving the latest round of global parameters from the server, the client performs local training to obtain gradient difference parameters; after receiving the gradient difference parameters from all clients, the server performs federated averaging and aggregation; the updated global parameters are then broadcast to all clients to enter the next round of iteration; until training is completed; the trained region-aware multimodal semantic federated learning framework is used for fault diagnosis of equipment under complex working conditions.

[0008] Furthermore, the specific method of step 1 of the present invention is as follows: Step 1.1, Single-modal region-aware feature extraction: Client For a given mode Input image The server-issued initialization modal encoder is used. Extracting multi-scale features ; Step 1.2, Local Region Mapping and Spatial Pyramid Augmentation: Pyramid compression features are obtained through local region mapping and spatial pyramid strategies. ; Step 1.3, Cross-modal region heatmap fusion: Calculate the region attention score for each modality to obtain a multimodal region heatmap. ; Step 1.4, Region-guided cross-modal feature fusion: Based on multimodal region heatmaps soft mask Compress features of each mode Flattened into a labeled sequence The region weighting is achieved through label-by-label multiplication. The weighted multimodal labels are then input into the cross-modal attention module to obtain the fused label sequence. .

[0009] Furthermore, in step 1.1 of this invention, a dedicated encoder for initializing modal modes is used. The specific method for extracting multi-scale features is as follows:

[0010] in, Indicates high-resolution detail features. This represents the local response characteristics at medium resolution. For low-resolution global semantic features, the input image , C The number of channels in an RGB image. C =3, in infrared images C =1, H The height of the image. W The width of the image; In step 1.2, the heatmap of the intramodal region is calculated through shallow convolution and normalization operations:

[0011] in, For Sigmoid or Softmax activation functions, the output is... Indicates the intensity of the anomalous response at each spatial location; Using threshold Divide the area into hotspot and non-hotspot areas:

[0012] in, i Indicates the client number. j This indicates the client's first j One sample; p , q Represents spatial location coordinates, corresponding to indices in the height and width directions. In hotspot areas... Preserve high-resolution features In the cold zone Only low-resolution features are retained. Spatial pyramid compression features are obtained by aligning and fusing resolutions using bilinear interpolation. .

[0013] Furthermore, the specific method of step 1.3 of the present invention is as follows: Calculate the region attention score for each modality:

[0014] in, For global average pooling Activated for ReLU These are learnable parameters; the modal fusion weights are obtained through Softmax regularization.

[0015] This leads to the multimodal region heat map:

[0016] The specific method for step 1.4 is as follows: Depend on Generate a soft mask:

[0017] in, The temperature coefficient controls the steepness of the response distribution. The threshold is set to centralize; features of each mode are compressed. Flattened into a labeled sequence And region weighting is achieved through mark-by-mark multiplication:

[0018] The weighted multimodal labels are input into the cross-modal attention module to obtain the client. Fusion marker sequence for: .

[0019] Furthermore, the specific method of step 2 of the present invention is as follows: Step 2.1, Abnormal Region Identification and Saliency Scoring: Client Labels are obtained through multimodal feature fusion. Then, an anomaly scoring network was used. For each tag Calculate the abnormality score ; Step 2.2, Multimodal Redundancy Suppression and Key Region Optimization: Introducing an Incremental Scoring Function Based on the abnormality score and the currently selected set of tags R Evaluate candidate labels based on t The marginal value; Step 2.3, Semantic Tag Extraction under Dynamic Communication Budget: Considering the Client Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. In the constructed incremental scoring function Under the guidance of this method, a greedy Top-K strategy is used to select the key markers, resulting in k marker information. .

[0020] Furthermore, the incremental scoring function in step 2.2 of this invention... for:

[0021] in, The importance of controlling anomalies β Control redundancy penalties; Indicates candidate tokens With selected marker r Hybrid similarity between modalities; to suppress redundant labels between modalities and within spatial neighborhoods, a hybrid similarity function fusing semantic features and spatial location is introduced. Defined as:

[0022] The first term is semantic cosine similarity, and the second term is spatial Gaussian decay function; parameters Control the weighting ratio of semantic and spatial similarity. The decay scale is Gaussian. Indicates a marker The two-dimensional center coordinates in the original image are obtained by positional encoding mapping; The upper limit of the number of markers in step 2.3 The definition is as follows:

[0023] in, V This represents the average communication cost of a single tag. Due to communication bandwidth constraints, The urgency of the current task. The maximum number of tags allowed by the system; In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is adopted for key label selection; specifically: initializing the selected set. R ← Iteratively select the label with the largest incremental score from the unselected labels. ,Right now:

[0024]

[0025] Add to collection R Repeat the process of selecting and adding items to the set until... R The number of elements in the middle reaches the limit .

[0026] Furthermore, the specific method of step 3 of the present invention is as follows: Step 3.1, Client-side local semantic decoding and training update: Client Obtain the set of tags with the highest task relevance. Then, it is used as the input to the compact semantic representation detection head for local decoding and prediction; the client receives the latest round of global parameters from the server. Then, the gradient difference parameters are obtained through local training. ; Step 3.2, Differential Parameter Calculation and Upload: Only the differential parameters of the fusion module and the detection head are transmitted; Step 3.3, Server Aggregation and Broadcast: The server receives differential parameters from each client. The average aggregation is then performed, and the updated global parameters are broadcast to all clients to enter the next iteration; until the training is completed and a trained multimodal semantic federated learning framework is obtained.

[0027] Furthermore, the specific method of step 3.1 of the present invention is as follows: Client Obtain the set of tags with the highest task relevance:

[0028] The labeled set is used as the input to the compact semantic representation for local decoding and prediction in the detection head. The detection head consists of two parts: a query decoder, which interacts with the input features and the learnable query vector through an attention mechanism to generate predicted query features. The prediction branch outputs each predicted query feature via two parallel branches: ,

[0029] in, Represents class probability, Indicates the normalized bounding box parameters; The client receives the latest round of global parameters from the server. Then, it is synchronized to the local fusion module and the detection head to initialize weights, and local data is used. Perform several rounds of gradient descent updates;

[0030] in, L (•) represents the overall loss of the detection task.

[0031] Furthermore, the specific method of step 3.2 of the present invention is as follows: Transmit only the differential parameters of the fusion module and the detection head :

[0032] Here, fh represents some parameters of the fusion module and the detection head.

[0033] The specific method for step 3.3 is as follows: The federated server receives data from each client. And perform FedAvg aggregation:

[0034] The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0035] This invention provides a multimodal semantic federated learning-based fault diagnosis system for equipment under complex operating conditions, comprising: Several inspection robots, each of which is designed for equipment in complex working conditions, serve as a client. Each client is equipped with multiple types of sensors to collect multimodal data, including images, thermal infrared, and audio. The server constructs a region-aware multimodal semantic federated learning framework with each client and a central server. The server is equipped with a memory and a processor. The memory is used to store executable computer programs. The processor is used to execute the executable computer programs stored in the memory to implement the above-mentioned multimodal semantic federated learning method for diagnosing equipment faults under complex working conditions.

[0036] The beneficial effects of this invention are: 1. This invention constructs a regionally perceptive multimodal semantic federated learning framework for fault diagnosis of equipment under complex operating conditions. Each inspection robot acts as a federated client, integrating multimodal sensors such as image, thermal infrared, and audio to achieve local data acquisition and model training. The clients collaborate to build a global model through a federated mechanism, without sharing raw data, thereby achieving efficient fault diagnosis and privacy protection.

[0037] 2. This invention proposes a region-optimized driven multimodal fusion mechanism to address the problem of insufficient response to small-scale anomalies in traditional global equilibrium modeling. By performing region-aware single-modal feature extraction at the edge and combining it with spatial pyramid compression for fine-grained characterization of local hotspots, the model not only enhances the representation of salient regions but also suppresses the interference of redundant information at the source. Subsequently, in the cross-modal interaction stage, region-guided alignment and fusion are introduced, enabling multimodal representations to form structured complementary representations, thereby obtaining more discriminative multimodal semantic features.

[0038] 3. This invention designs a dynamic communication optimization method based on semantic tagging. After identifying abnormal regions locally and scoring their saliency, only key task tags are retained for uploading. By fusing semantic-spatial diversity constraints, duplicate selections are avoided, ensuring balanced coverage of uploaded information across the global distribution. Combined with a sparse differential update strategy, the system maintains detection accuracy while significantly reducing communication overhead, thus ensuring that the entire federated framework is efficient and privacy-friendly in real-world scenarios. Attached Figure Description

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a system architecture diagram of an embodiment of the present invention.

[0040] Figure 2 This is a flowchart of the method of the present invention.

[0041] Figure 3 This is a comparison of the optimal single-modal and multimodal performance of this invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0043] Example 1 like Figure 1 As shown in the figure, the proposed region-aware multimodal semantic federated learning framework (RS_MFL) for fault diagnosis of equipment under complex operating conditions in this invention is as follows: Figure 1 As shown. Consider a federated learning system for complex inspection scenarios: There is one central server and N inspection robots as clients. Number Each client is equipped with multiple types of sensors, capable of simultaneously acquiring heterogeneous modal data such as visible light (RGB), thermal infrared (IR), and acoustic pulses (m represents the modal category). ), note the client The local multimodal dataset is The goal is to improve fault diagnosis performance for small-area anomalies through semantically aware federated collaboration, while significantly reducing communication overhead, under the constraint that raw data from various clients cannot be centrally shared.

[0044] The system framework consists of three core modules: a region-aware multimodal semantic fusion module, a semantic communication optimization module, and a server-side federated aggregation module.

[0045] First, on the end side The execution region optimization-driven multimodal fusion mechanism allows each client to extract initial features using an independent encoder for different modalities. After extraction, each modal feature undergoes anomaly region mapping and spatial pyramid compression to remove intramodal redundancy. Furthermore, region-guided semantic interaction is used in the cross-modal stage to enhance local anomaly representations, thereby obtaining a compact and semantically prominent fusion feature representation. .

[0046] Then, an anomaly scoring network is constructed based on the fused feature sequences to assign anomaly scores to each label, and this is combined with terminal bandwidth constraints. Based on task urgency, a greedy Top-K strategy is used to select the local representation with the maximum information gain. This significantly reduces communication overhead while ensuring detection performance.

[0047] Finally, the client will The input-side detection head performs local decoding and prediction, and receives the latest global parameters from the cloud in the r-th round. Gradient difference parameters are then obtained through local training. The server performs federated average aggregation after receiving parameter information from all clients. The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0048] Example 2 Based on Embodiment 1, this invention provides the main steps specifically included in the method for implementing the RS_MFL framework of this invention: Step 1: Client For a given mode Input image A modal encoder is used. Extracting multi-scale features Then, pyramid compression features are obtained through local region mapping and a spatial pyramid strategy. Calculate the region attention score for each modality to obtain a multimodal region heatmap. and soft mask Compress features of each mode. Flattened into a labeled sequence ,Will and Region weighting is achieved through label-by-label multiplication. Subsequently, the weighted RGB and TIR labels are input into a cross-modal attention module to obtain a fused label sequence. It not only maintains intermodal complementarity but also highlights the spatial semantics of potential anomalies, laying the foundation for subsequent optimization.

[0049] Step 2: Client based on Communication optimization processing is performed. Specifically, an anomaly scoring network is first used for each label. Calculate the abnormality score Then, an incremental scoring mechanism is introduced, scoring based on the degree of abnormality. And evaluate the marginal value of candidate labels based on the current set of selected labels R. In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is used for key label selection. Among them, It takes into account the communication bandwidth constraints of the end-side devices. and the urgency index of the current task The maximum number of tags uploaded in each round of federated data is dynamically determined, ultimately resulting in the Top-K tags. .

[0050] Step 3: Client Obtain the set of tags with the highest task relevance. Then, these tags are used as compact semantic representations as input to the edge detection head for local decoding and prediction. The edge receives the latest global parameters from the cloud. After (round r), the weights are synchronized to the local fusion module and detection head, and the local data is used. Gradient descent is used to obtain the gradient difference parameters. The differential parameters of the fusion module and the detection head are used. The data is uploaded to the central server, which then performs federated average aggregation after receiving parameters from all clients. The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0051] Repeat the above steps until the termination condition is met, completing the entire training process. The termination condition can be the maximum number of training epochs, the convergence of the loss function, or other user-defined conditions. Finally, the optimal loss function can be obtained according to the following formula:

[0052] in, express The loss of the network.

[0053] After the iteration is completed, the trained model can be obtained and used for fault diagnosis of equipment under complex working conditions.

[0054] In a preferred embodiment of the present invention, the specific method for multimodal fusion of local region optimization in step 1 is as follows: To address the issue of insufficient response of global balanced modeling to minor anomalies (such as localized overheating or minute discharge points in equipment), this invention proposes a region-optimized driven multimodal fusion mechanism. This mechanism effectively enhances the saliency of small-scale anomalies in the fused features through intramodal region-aware feature extraction, spatial pyramid-style hotspot fidelity compression, and cross-modal region-guided fusion, while simultaneously reducing redundant feature transmission and improving communication efficiency.

[0055] Step 1.1: Extraction of Single-Modal Region Perception Features Given from modality Input image The initialization modal encoder is issued by the central server. Extracting multi-scale features:

[0056] in, Indicates high-resolution detail features. This represents the local response characteristics at medium resolution. These are low-resolution global semantic features. C The number of channels in an RGB image. C =3, in infrared images C =1, H The height of the image. W The width of the image.

[0057] Step 1.2: Local Region Mapping and Spatial Pyramid Enhancement To compress redundant regional information while preserving details of abnormal hot spots, embodiments of the present invention introduce a regional heat estimation and pyramid mapping strategy based on medium-resolution features.

[0058] First, the heatmap of the intramodal region is calculated through shallow convolution and normalization operations:

[0059] in, For Sigmoid or Softmax activation functions, the output is... This indicates the intensity of the anomalous response at each spatial location.

[0060] Subsequently, using threshold Divide the area into hotspot and non-hotspot areas:

[0061] in, i Indicates the client number. j This indicates the client's first j One sample; p , qRepresents spatial location coordinates, corresponding to indices in the height and width directions. In hotspot areas... Preserve high-resolution features In the cold zone Only low-resolution features are retained. Spatial pyramid compression features are obtained by aligning and fusing resolutions using bilinear interpolation.

[0062] This operation significantly reduces the number of tags while preserving hotspot details, thereby reducing redundant overhead in subsequent cross-modal processing and communication.

[0063] Step 1.3, Cross-modal region thermal fusion To integrate the complementary responses of different modalities to anomalous regions, the region attention score for each modality is first calculated:

[0064] in, For global average pooling Activated for ReLU These are learnable parameters. The modality fusion weights are then obtained through Softmax regularization.

[0065] The final multimodal region heatmap is obtained:

[0066] Step 1.4: Region-guided cross-modal feature fusion Depend on Generate a soft mask:

[0067] in, The temperature coefficient controls the steepness of the response distribution. The threshold is used for centering. Features of each modality are compressed. Flattened into a labeled sequence And region weighting is achieved through mark-by-mark multiplication:

[0068] Subsequently, the weighted multimodal labels are input into the cross-attention transformation module to obtain the client-side data. Fusion marker sequence for:

[0069] While maintaining modal complementarity, this sequence further highlights the semantic features of anomalous hotspot regions, thereby achieving region-aware-driven high semantic expression.

[0070] Thus, step 1 (region-aware multimodal semantic fusion module) not only completed single-modal encoding and saliency modeling, but also obtained a high-quality labeled representation with regional semantic structure through cross-modal fusion, laying a unified input foundation for subsequent semantic communication optimization and interpretability analysis.

[0071] In a preferred embodiment of the present invention, the specific method for optimizing local semantic communication in step 2 is as follows: To reduce communication overhead and adapt to resource-constrained edge computing environments in small-area anomaly detection tasks, this module structurally optimizes traditional methods that rely on large-scale data transmission and global feature interaction, constructing a simplified semantic delivery mechanism based on local anomaly regions. This mechanism comprises three core steps: anomaly region identification and saliency scoring, locating and quantifying local regions with potential anomaly features; multimodal redundancy suppression and key region selection, filtering the most informative local representations across modalities through semantic diversity constraints; and semantic tag extraction under a dynamic communication budget, adaptively generating and delivering simplified semantic representations based on task relevance and communication cost to support efficient semantic aggregation in federated learning.

[0072] Step 2.1: Anomaly Region Identification and Saliency Scoring Client Labels are obtained through multimodal feature fusion. Then, an anomaly scoring network was used. For each tag Calculate the abnormality score:

[0073] Step 2.2: Multimodal redundancy suppression and key region selection To select key markers with strong representativeness and low redundancy from multimodal features, this invention introduces an incremental scoring mechanism. Based on its abnormality score And, based on the currently selected label set R, evaluate the marginal value of the candidate label t. Its definition is as follows:

[0074] in, Controlling the importance of anomalies, β controls redundancy penalties; Indicates candidate tokens The hybrid similarity between the selected label r and the selected label r. To more effectively suppress redundant labels between modalities and within the spatial neighborhood, a hybrid similarity function that integrates semantic features and spatial location is introduced. Defined as:

[0075] The first term is semantic cosine similarity, which measures the directional consistency between labeled pairs in the feature semantic space, helping to identify semantic redundancy in cross-modal expressions; the second term is spatial Gaussian decay function, used to penalize spatially neighboring labeled regions, effectively dispersing the selection results to enhance coverage; parameters Control the weighting ratio of semantic and spatial similarity. The decay scale is Gaussian. Indicates a marker The two-dimensional center coordinates in the original image can be obtained by positional encoding mapping.

[0076] This design ensures that the selected tag set has semantic diversity and spatial separation, satisfying the discriminative and sparsity requirements of anomaly region tagging representation. This similarity definition not only unifies the semantic and spatial motivations of redundant compression but also makes the subsequent incremental scoring function more interpretable and stable.

[0077] Step 2.3 Semantic Tag Extraction under Dynamic Communication Budget Consider end-side equipment Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. The definition is as follows:

[0078] in, V This represents the average communication cost of a single tag. Due to communication bandwidth constraints, The urgency of the current task. This sets the maximum allowed number of tags for the system. This strategy can effectively control the communication burden while ensuring model performance, achieving resource-adaptive semantic compression.

[0079] In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is adopted for key tag selection. Specifically: First, the selection set R is initialized. Then, iteratively select the label with the largest incremental score from the unselected labels. ,Right now:

[0080]

[0081] The elements will be added to set R, and the above process will be repeated until the number of elements in R reaches its maximum. This greedy strategy ensures that the selected label has the optimal marginal information gain at each step, effectively avoiding redundancy in semantic or spatial dimensions and improving the discriminativeness and compactness of the final representation.

[0082] It is important to note that, considering the Top-K strategy may miss key regions with small scales or low scores during the ranking process, the anomaly-priority retention, hybrid ranking, and dynamic K mechanisms introduced in this embodiment of the invention can achieve high-fidelity semantic transmission and stable detection performance under different task urgency and bandwidth constraints. This process not only significantly reduces the amount of redundant information transmitted, but also ensures that the uploaded features have strong task relevance and semantic coverage through salient region selection, providing high-quality support for subsequent federated model training.

[0083] In a preferred embodiment of the present invention, the specific method for local training and federated aggregation in step 3 is as follows: To further improve communication efficiency and data privacy, this invention employs edge-side semantic encoding and decoding. Unlike cloud-based decoding, local decoding and differential uploading of sparse parameters effectively avoid the direct transmission of high-dimensional features. This process includes three core steps: edge-side local semantic decoding and training update, differential parameter calculation and uploading, and differential parameter calculation and uploading.

[0084] Step 3.1: Local semantic decoding and training update on the device side After completing the key tag filtering, the client Obtain the set of tags with the highest task relevance:

[0085] These tags serve as the input to the compact semantic representation detection head for local decoding and prediction. Generally, the detection head typically consists of two parts: a query decoder, which interacts with the input features and the learnable query vector through an attention mechanism to generate predicted query features. The prediction branch (FFN) outputs each predicted query feature via two parallel branches: ,

[0086] in, Represents class probability, This represents the normalized bounding box parameters.

[0087] The device receives the latest global parameters from the cloud. After (round r), the weights are synchronized to the local fusion module and detection head, and the local data is used. Perform several rounds of gradient descent updates:

[0088] Where L(•) represents the overall loss of the detection task.

[0089] Step 3.2: Calculation and Upload of Differential Parameters To further reduce communication overhead, this invention does not transmit the complete model weights back, but only the differential parameters of the fusion module and the detection head. :

[0090] Here, fh represents some parameters of the fusion module and the detection head.

[0091] It retains only gradient update terms related to key markers, achieving sparse propagation. This maintains the global model's discriminative performance in outlier regions while significantly reducing gradient redundancy in irrelevant regions.

[0092] Step 3.3, Central Aggregation and Broadcasting The federated server receives data from each client. And perform FedAvg aggregation:

[0093] The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0094] Module 1 (the region-aware multimodal semantic fusion module) uses single-modal redundancy suppression to weight and weaken background noise and highly similar regions within a modality, thereby improving the semantic purity of the fused input. Meanwhile, Module 2 (the semantic communication optimization module) uses multimodal redundancy suppression, based on the fused features and combined with communication budget constraints, to select the markers with the greatest information gain across modalities. These two modules operate at different stages: the former ensures fusion quality, while the latter ensures communication efficiency, forming a complementary rather than conflicting optimization link in the feature semantic space and transmission dimension.

[0095] Example 3: Experimental Test Example To verify the effectiveness of the proposed framework RS_MFL, experimental performance was compared with FedAvg, FedRS, and FedOpt on four existing public datasets: MiT-51, UCI-HAR, KU-HAR, and CrisisMMD, focusing on accuracy (Acc) and communication overhead. For the UCI-HAR and KU-HAR datasets, accelerometer data (Accel) and gyroscope data (Gyro) were set as bimodal datasets, as described in Table 1. To accommodate the limitations of computation and communication overhead in practical inspection robots, a lightweight network structure was adopted in the model design. An RNN-based model architecture was used in the video and text modalities, while a Conv-RNN architecture was used for validation in other modalities. The learning rate was set to 0.01, the training epochs were 200, and the number of clients was 5.

[0096] Table 1: Introduction to the dataset

[0097] 1. Comparison of multimodal and single-modal performance A fundamental research question surrounding multimodal learning is how it performs compared to single-modal models. Figure 3 This paper summarizes the performance comparison between unimodal learning (FL) and multimodal learning (FL). Comparisons were made on the UCI-HAR and CrisisMMD datasets, with the unimodal learning model representing the best performing modality. Overall, unimodal learning is observed to offer competitive performance compared to multimodal FL. However, in most cases, multimodal learning still provides additional advantages, especially when dealing with complex tasks and cross-modal information fusion. For example, in the KU-HAR dataset, the proposed RS_MFL achieves a 10.44% improvement in multimodal FL performance compared to unimodal learning.

[0098] 2. Multimodal performance comparison In the experimental comparison, the proposed local region-aware multimodal semantic federated learning method was compared with three classic federated learning algorithms: FedAvg, FedProx, and FedOpt. It is important to emphasize that, to ensure fairness, all three comparison methods employ the same modality fusion mechanism based on a fundamental attention mechanism; that is, the multimodal features are globally interacted on the client side before being uploaded for federated aggregation. The parameters are set to... The dataset is divided using a Dirichlet distribution with a resolution of 0.1 to simulate a non-independent identically distributed (non-IID) dataset with high heterogeneity among clients.

[0099] As shown in Table 3, the method of this invention achieved optimal performance on multiple public datasets. On the MiT-5 action recognition dataset, traditional attention fusion relies on weighted combination of intermodal features, lacking fine-grained modeling of the relevance of local abnormal regions to the task. Therefore, it often fails to guarantee the effectiveness of cross-modal collaboration when redundant features or noisy modalities exist. The method of this invention, however, can retain key action-related segments through local region perception and anomaly labeling, thus achieving a significant performance improvement. On the UCI-HAR and KU-HAR human action recognition datasets, the method of this invention significantly outperforms all comparable methods in the F1 score, even improving the score by 12.69% compared to FedAvg on the KU-HAR dataset. This demonstrates that in cross-client scenarios, semantic representation based on local regions can enhance robust modeling capabilities for key action patterns. On the CrisisMMD multimodal social media emergency dataset, the method of this invention also achieved the best F1 score, exhibiting strong noise resistance and cross-modal generalization ability.

[0100] Comprehensive analysis reveals that the method of this invention effectively focuses on task-related regions and suppresses redundant features through a local region perception mechanism and semantic tag selection based on anomaly scoring and diversity constraints, thereby maintaining optimal performance under different tasks and partition settings.

[0101] Table 2: Performance comparison of different multimodal methods on different datasets (%)

[0102] 3. Comparison of communication overhead To further verify the advantages of the proposed RS-MFL method in terms of communication efficiency, the training batch size B=32 was uniformly set in the experiments to achieve comparable communication overhead statistics between traditional distributed and federated learning methods. Traditional distributed methods require uploading raw data to the cloud, resulting in huge communication overhead and significant privacy and security risks due to centralized data storage. Federated learning methods, on the other hand, avoid uploading raw data through local model updates and parameter sharing, while also reducing communication volume to some extent. A comparative analysis was conducted on two benchmarks: traditional distributed transmission methods and typical federated learning methods.

[0103] As shown in Table 3, for the low-dimensional sensor datasets UCI-HAR and KU-HAR, traditional distributed methods only need to upload one batch of raw data per round, while FedAvg and FedProx require uploading the complete model parameters. Superficially, the communication overhead of traditional distributed methods appears lower than that of federated methods. This is mainly due to factors such as the small sample size of the HAR dataset, its proximity to IID data, and low task complexity. It is important to emphasize that this is only a single-round comparison. Considering the cumulative communication volume and the large-scale multimodal datasets, the communication overhead of traditional distributed methods, due to the continuous uploading of raw data, is still far higher than that of federated learning methods.

[0104] For the high-dimensional multimodal datasets MiT-51 and CrisisMMD, the traditional distributed learning model incurs communication overhead of 48.78 MB and 16.97 MB per round, respectively. In contrast, the proposed RS-MFL, through local anomaly region filtering and semantic tagging compression, achieves a communication overhead of only 14.15 MB per round, saving 71.04% on MiT-51 and 13.03% on CrisisMMD compared to traditional distributed learning. Compared to FedAvg, FedProx, and FedOpt, RS-MFL reduces communication overhead by an average of approximately 48.17%-50.53% across all datasets, with the most significant advantages on high-dimensional visual or image / text modal datasets like CrisisMMD and MiT-51. These results demonstrate that RS-MFL effectively reduces redundant information transmission while preserving key task features, achieving a balance between low communication overhead and high performance. This enhances the practicality and scalability of multimodal federated learning in bandwidth-constrained or real-time-critical application scenarios, making it particularly suitable for complex inspection scenarios such as transformer fault diagnosis.

[0105] Table 3: Comparison of communication overhead (MB) for different algorithms

[0106] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multi-modal semantic federated learning complex working condition equipment fault diagnosis method, characterized in that, Each inspection robot facing complex working condition equipment is taken as a client, and a regional perception multi-modal semantic federated learning framework is constructed through each client and a central server. Step 1, a regional perception multi-modal semantic fusion mechanism is executed on the client side, each client respectively extracts initial features by using independent encoders for different modalities, the initial features of each modality are subjected to in-modal redundancy removal after extraction through abnormal region mapping and spatial pyramid compression, and local abnormal representation is strengthened by using region-guided semantic interaction in the cross-modal stage to obtain a compact structure and semantic prominent fusion feature sequence; Step 2, the client performs local semantic communication optimization processing based on the fusion feature sequence; an abnormality scoring network is used to calculate an abnormality score for each label in the fusion feature sequence, an incremental scoring mechanism is introduced, the marginal value of the candidate label is evaluated based on the abnormality score and the current selected label set, and is used as an incremental scoring function for guidance; and combining the client bandwidth constraint and the task urgency, a greedy Top-K strategy is used for key label optimization to dynamically determine the upper limit of the number of labels uploaded in each round of federation, and a label set with the highest task relevance is obtained; Step 3, the client inputs the obtained label set with the highest task relevance into the side detection head for local decoding and prediction; the client performs local training to obtain gradient difference parameters after receiving the latest round of global parameters from the server; the server aggregates the gradient difference parameters after receiving the gradient difference parameters from all clients; the updated global parameters are broadcast to all clients again, and the next round of iteration is entered; until the training is completed; the complex working condition equipment fault diagnosis is performed through the trained regional perception multi-modal semantic federated learning framework.

2. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 1, characterized in that, The specific method of step 1 is: Step 1.1, single-modal region-aware feature extraction: client For a given input image from a modality , a server-deployed initialization modality-specific encoder is used to extract multi-scale features ; Step 1.2, Local Region Mapping and Spatial Pyramid Enhancement: Pyramid compressed features are obtained by local region mapping and spatial pyramid strategy ; Step 1.3, cross-modal region heat fusion: calculate the region attention score of each modality to obtain a multi-modal region heat map ; Step 1.4, Region-guided cross-modal feature fusion: Based on multimodal region heatmaps soft mask Compress features of each mode Flattened into a labeled sequence The region weighting is achieved through label-by-label multiplication. The weighted multimodal labels are then input into the cross-modal attention module to obtain the fused label sequence. .

3. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 2, characterized in that, In the step 1.1, an initialization modal specific encoder is adopted The specific method for extracting multi-scale features is: wherein, denotes high-resolution detail features, denotes mid-resolution local response features, denotes low-resolution global semantic features, an input image , C denotes the number of channels, in an RGB image C = 3, in an infrared image C = 1, H denotes the height of the image, W denotes the width of the image; In step 1.2, the in-modal region heat map is calculated through shallow convolution and normalization operation: wherein, is a Sigmoid or Softmax activation function, output represents the abnormal response intensity of each spatial position; Utilizing a threshold Dividing hotspot areas and non-hotspot areas: wherein, i represents the client number, j represents the first j sample of this client; p q represents the spatial position coordinates, corresponding to the index of height and width direction; high resolution features are reserved in the hot area , and only low resolution features are reserved in the cold area ; the spatial pyramid compression features are obtained by aligning the resolution and fusing through bilinear interpolation.​​​ 。 4. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 3, characterized in that, The specific method of step 1.3 is: computing regional attention scores for each modality : where, is global average pooling is ReLU activation, are learnable parameters; the modality fusion weights are obtained by Softmax regularization: And then the multi-modal region heat map is obtained: The specific method of step 1.4 is: From Generate soft mask: wherein, is a temperature coefficient, controlling the steepness of the response profile; is a centering threshold; each modality compressed feature is flattened into a sequence of tokens , denotes the flattened spatial number of tokens for the m-th modality and the region weighting is realized by a token-wise multiplication: The weighted multi-modal label is input into a cross-modal attention module to obtain a fusion label sequence of the client :​ 。 5. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 1, characterized in that, The specific method of step 2 is: Step 2.1, abnormal region identification and saliency scoring: client After multi-modal feature fusion, the marked is obtained, and an abnormality scoring network is used to score the abnormality of each marked region ; Step 2.2, Multimodal redundancy suppression and key region preference: Introducing a delta scoring function , based on abnormality score and marginal value of candidate markers based on the current selected marker set R t ​​ Step 2.3, Semantic Tag Extraction under Dynamic Communication Budget: Considering the Client Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. In the constructed incremental scoring function Under the guidance of this method, a greedy Top-K strategy is used to select the key markers, resulting in k marker information. .

6. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 5, characterized in that, The step 2.2 incremental scoring function is: wherein, controlling abnormality importance, β controlling redundancy penalty; representing a candidate marker and a selected marker r between them; to suppress redundant markers within and across modalities and spatial neighborhoods, a hybrid similarity function that fuses semantic features and spatial locations is introduced defined as: Wherein, the first term is semantic cosine similarity, the second term is spatial Gaussian decay function; parameters The weighted proportion of the control semantics and the spatial similarity, Gaussian decay scale; The mark The two-dimensional center position coordinates in the original image are obtained by position encoding mapping. The upper limit of the number of labels in step 2.3 is defined as follows: wherein, V denotes the average communication overhead of a single token, is a communication bandwidth constraint, is an urgency of the current task, an indicator is a maximum allowed number of tokens set for the system; In the constructed incremental scoring function Under the guidance, the greedy Top-K strategy is adopted to perform key label optimization; specifically, an elected set is initialized R ← The label with the maximum current incremental score is iteratively selected from the unselected labels That is: adding the set R , the process of selecting the set is repeated until R the number of elements in the set reaches the upper limit .

7. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 1, characterized in that, The specific method of step 3 is: Step 3.1, Client local semantic decoding and training update: Client Obtain the set of tokens with the highest relevance to the task After that, it is decoded and predicted locally as a compact semantic representation input side detection head; The client receives the latest round of global parameters from the server After that, it performs local training to obtain gradient difference parameters ; Step 3.2, difference parameter calculation and uploading: only the difference parameters of the fusion module and the detection head are transmitted; Step 3.3, Server aggregation and broadcast: the server receives the differential parameters from each client and average aggregation, the updated global parameters are broadcast to all clients, and the next round of iteration is entered; until the trained multi-modal semantic federated learning framework is obtained after completing the training.

8. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 7, characterized in that, The specific method of step 3.1 is: Client Obtain the set of tags that are most relevant to the task: The token set is input into a compact semantic representation input-side detection head for local decoding and prediction, the detection head consists of two parts: a query decoder, which interacts with input features and a learnable query vector through an attention mechanism to generate predicted query features ; a prediction branch, each predicted query feature is output via two parallel branches: ， wherein, represents a class probability, represents a normalized bounding box parameter; The client receives the global parameters from the server for the latest round After that, it synchronizes the local fusion module with the detection head to initialize the weights and uses the local data It performs several rounds of gradient descent updates; wherein, L (•) is the detection task synthesis loss.

9. The multi-modal semantic federated learning complex working condition device fault diagnosis method according to claim 8, characterized in that, The specific method of step 3.2 is: Transmit only fusion module and differential parameters of the detection head : Wherein, fh is part of the parameters of the fusion module and the detection head; The specific method of step 3.3 is: The federal server receives the and performs FedAvg aggregation: The updated global parameters are broadcast to all clients again, and the next round of iteration is entered.

10. A multi-modal semantic federated learning complex working condition equipment fault diagnosis system, characterized in that, It comprises: A plurality of inspection robots, each inspection robot facing complex working condition equipment is taken as a client, each client is equipped with multiple types of sensors for collecting multi-modal data including images, thermal infrared and audio; A server, a regional perception multi-modal semantic federated learning framework is constructed through each client and a central server; the server is provided with a memory and a processor; the memory is used to store executable computer programs; the processor is used to execute the executable computer programs stored in the memory, and the multi-modal semantic federated learning complex working condition equipment fault diagnosis method in any one of claims 1 to 9 is realized.

Images