A multi-modal semantic federated learning complex working condition equipment fault diagnosis method

Abstract

The application discloses a kind of multi-modal semantic federal learning complex working condition equipment fault diagnosis methods, comprising: client utilizes the independent encoder of different modalities to extract initial feature, realizes modal redundancy removal via abnormal area mapping and space pyramid compression, in cross-modal stage utilizes area guided semantic interaction mechanism to strengthen local abnormal representation and obtains fusion feature;Abnormality scoring network is constructed, and abnormality score is assigned to label, and the local representation of maximum information gain is preferred in combination with terminal bandwidth constraint and task urgency degree;Client carries out local decoding and prediction, and server side executes federal average aggregation and broadcasts update parameter.The application is under the constraint that each client original data is not sharable, through semantic perception federal cooperation, in the identification task of multiple public data sets, accuracy can reach 12.69%, single round communication cost can be reduced by 71.04%, significantly improve the overall efficiency of complex working condition diagnosis system, with wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of fault detection technology for equipment under complex operating conditions, and in particular to a multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions. Background Technology

[0002] As intelligent systems continue to expand in scale and their operating environments become increasingly complex, the number of various complex-condition equipment is surging, making the need for safety monitoring of their operational status ever more urgent. These devices are widely distributed in typical automated scenarios such as industrial production, energy transmission and distribution, agricultural operations, and infrastructure. They often operate under harsh conditions, including high temperature, high humidity, high load, or multi-source interference, making them prone to overheating, performance degradation, or structural abnormalities. Failure to detect and address these issues in a timely manner can lead to serious equipment failures and system paralysis. Traditional manual inspection methods suffer from low efficiency, high risk, and incomplete coverage under complex conditions. Intelligent inspection robots, however, offer greater flexibility and safety in complex environments, becoming a new trend in complex-condition inspection. However, complex-condition equipment often exhibits potential faults such as thermal anomalies, mechanical fatigue, or abnormal energy consumption, which manifest as multimodal sensing characteristics (e.g., thermal imaging, images, and sound). This requires multi-source information fusion for accurate analysis, increasing diagnostic complexity. Furthermore, these robots are widely distributed, and limitations such as network bandwidth and privacy protection prevent the centralized uploading of all raw data. How to achieve accurate multimodal fault diagnosis while ensuring efficient communication and data security has become the core challenge in intelligent inspection under complex working conditions.

[0003] Current research on fault diagnosis for equipment operating under complex conditions mainly focuses on two aspects: first, single-modal diagnostic methods, such as hotspot identification based on infrared images, surface defect detection based on image processing, and partial discharge identification based on audio signals. These methods have limited accuracy and are difficult to adapt to complex multi-source scenarios. Second, multi-modal fusion methods, such as using deep neural networks to jointly model images and infrared data, but most of these are based on centralized training and lack consideration for privacy protection and edge deployment. In recent years, federated learning methods have been gradually introduced into complex operating system systems, such as for load forecasting and fault classification under complex conditions, but there is still a gap in multi-modal semantic modeling and communication optimization. Therefore, there is an urgent need to construct a federated learning framework that can effectively fuse multi-modal information, possess semantic awareness and reasoning capabilities, and reduce communication costs while protecting data security.

[0004] While some research has been conducted on fault diagnosis of equipment under complex operating conditions, most studies focus on federated modeling or simple data-level fusion under a single modality. In actual complex inspection scenarios, inspection robots face data from diverse sources with semantic inconsistencies. How to achieve cross-device and cross-modal information fusion and construct a unified and interpretable hazard detection model while ensuring data privacy has become a core issue that current research urgently needs to address. First, due to the complex environments and varying modal data distributions of different robot devices, it is difficult for the global model to learn a unified semantic representation. Therefore, multimodal perception has become a key means to improve the accuracy of fault diagnosis, and fusing information from visual, thermal imaging, and audio modalities has been shown to significantly enhance the robustness and contextual understanding capabilities of the model. Second, distributed inspection robots are deployed on multiple edge nodes, and uploading raw data raises privacy and communication burdens, making centralized modeling impractical. Therefore, to address data security and communication constraints, federated learning (FL) technology has been introduced into collaborative modeling between edge devices, effectively mitigating the risk of data privacy leakage. Finally, to reduce the communication overhead of anomaly detection, semantic communication is used to compress data, and only feature semantic consistency information is uploaded, thereby ensuring that the entire federated framework has high efficiency in real-world scenarios.

[0005] Therefore, it is necessary to design a new fault diagnosis framework that supports multimodal semantic understanding, federated collaborative training, and high communication efficiency. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a multimodal semantic federated learning method for diagnosing equipment faults under complex working conditions, addressing the deficiencies in the existing technology.

[0007] The technical solution adopted by this invention to solve its technical problem is: This invention provides a multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions. Each inspection robot for equipment operating under complex conditions is treated as a client, and a region-aware multimodal semantic federated learning framework is constructed through each client and a central server. The method includes the following steps: Step 1: Execute a region-aware multimodal semantic fusion mechanism on the client side. Each client uses an independent encoder for different modalities to extract initial features. After extraction, the initial features of each modality are redundant within the modality through anomaly region mapping and spatial pyramid compression. In the cross-modal stage, region-guided semantic interaction is used to strengthen the representation of local anomalies, resulting in a fusion feature sequence that is structurally compact and semantically prominent. Step 2: The client performs local semantic communication optimization processing based on the fused feature sequence; an anomaly scoring network is used to calculate an anomaly score for each label in the fused feature sequence, and an incremental scoring mechanism is introduced. Based on the anomaly score and the marginal value of candidate labels on the basis of the current selected label set, the incremental scoring function is used as a guide; and combined with the client's bandwidth constraints and task urgency, a greedy Top-K strategy is used to select key labels, dynamically determine the upper limit of the number of labels uploaded in each round of federation, and obtain the set of labels with the highest task relevance. Step 3: The client inputs the set of tags with the highest task relevance into the detection head for local decoding and prediction; after receiving the latest round of global parameters from the server, the client performs local training to obtain gradient difference parameters; after receiving the gradient difference parameters from all clients, the server performs federated averaging and aggregation; the updated global parameters are then broadcast to all clients to enter the next round of iteration; until training is completed; the trained region-aware multimodal semantic federated learning framework is used for fault diagnosis of equipment under complex working conditions.

[0008] Furthermore, the specific method of step 1 of the present invention is as follows: Step 1.1, Single-modal region-aware feature extraction: Client For a given mode Input image The server-issued initialization modal encoder is used. Extracting multi-scale features ; Step 1.2, Local Region Mapping and Spatial Pyramid Augmentation: Pyramid compression features are obtained through local region mapping and spatial pyramid strategies. ; Step 1.3, Cross-modal region heatmap fusion: Calculate the region attention score for each modality to obtain a multimodal region heatmap. ; Step 1.4, Region-guided cross-modal feature fusion: Based on multimodal region heatmaps soft mask Compress features of each mode Flattened into a labeled sequence The region weighting is achieved through label-by-label multiplication. The weighted multimodal labels are then input into the cross-modal attention module to obtain the fused label sequence. .

[0009] Furthermore, in step 1.1 of this invention, a dedicated encoder for initializing modal modes is used. The specific method for extracting multi-scale features is as follows:

[0010] in, Indicates high-resolution detail features. This represents the local response characteristics at medium resolution. For low-resolution global semantic features, the input image , C The number of channels in an RGB image. C =3, in infrared images C =1, H The height of the image. W The width of the image; In step 1.2, the heatmap of the intramodal region is calculated through shallow convolution and normalization operations:

[0011] in, For Sigmoid or Softmax activation functions, the output is... Indicates the intensity of the anomalous response at each spatial location; Using threshold Divide the area into hotspot and non-hotspot areas:

[0012] in, i Indicates the client number. j This indicates the client's first j One sample; p , q Represents spatial location coordinates, corresponding to indices in the height and width directions. In hotspot areas... Preserve high-resolution features In the cold zone Only low-resolution features are retained. Spatial pyramid compression features are obtained by aligning and fusing resolutions using bilinear interpolation. .

[0013] Furthermore, the specific method of step 1.3 of the present invention is as follows: Calculate the region attention score for each modality:

[0014] in, For global average pooling Activated for ReLU These are learnable parameters; the modal fusion weights are obtained through Softmax regularization.

[0015] This leads to the multimodal region heat map:

[0016] The specific method for step 1.4 is as follows: Depend on Generate a soft mask:

[0017] in, The temperature coefficient controls the steepness of the response distribution. The threshold is set to centralize; features of each mode are compressed. Flattened into a labeled sequence And region weighting is achieved through mark-by-mark multiplication:

[0018] The weighted multimodal labels are input into the cross-modal attention module to obtain the client. Fusion marker sequence for: .

[0019] Furthermore, the specific method of step 2 of the present invention is as follows: Step 2.1, Abnormal Region Identification and Saliency Scoring: Client Labels are obtained through multimodal feature fusion. Then, an anomaly scoring network was used. For each tag Calculate the abnormality score ; Step 2.2, Multimodal Redundancy Suppression and Key Region Optimization: Introducing an Incremental Scoring Function Based on the abnormality score and the currently selected set of tags R Evaluate candidate labels based on t The marginal value; Step 2.3, Semantic Tag Extraction under Dynamic Communication Budget: Considering the Client Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. In the constructed incremental scoring function Under the guidance of this method, a greedy Top-K strategy is used to select the key markers, resulting in k marker information. .

[0020] Furthermore, the incremental scoring function in step 2.2 of this invention... for:

[0021] in, The importance of controlling anomalies β Control redundancy penalties; Indicates candidate tokens With selected marker r Hybrid similarity between modalities; to suppress redundant labels between modalities and within spatial neighborhoods, a hybrid similarity function fusing semantic features and spatial location is introduced. Defined as:

[0022] The first term is semantic cosine similarity, and the second term is spatial Gaussian decay function; parameters Control the weighting ratio of semantic and spatial similarity. The decay scale is Gaussian. Indicates a marker The two-dimensional center coordinates in the original image are obtained by positional encoding mapping; The upper limit of the number of markers in step 2.3 The definition is as follows:

[0023] in, V This represents the average communication cost of a single tag. Due to communication bandwidth constraints, The urgency of the current task. The maximum number of tags allowed by the system; In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is adopted for key label selection; specifically: initializing the selected set. R ← Iteratively select the label with the largest incremental score from the unselected labels. ,Right now:

[0024]

[0025] Calculated Add to collection R Repeat the process of selecting and adding items to the set until... R The number of elements in the middle reaches the limit .

[0026] Furthermore, the specific method of step 3 of the present invention is as follows: Step 3.1, Client-side local semantic decoding and training update: Client Obtain the set of tags with the highest task relevance Then, it is used as the input to the compact semantic representation detection head for local decoding and prediction; the client receives the latest round of global parameters from the server. Then, the gradient difference parameters are obtained through local training. ; Step 3.2, Differential Parameter Calculation and Upload: Only the differential parameters of the fusion module and the detection head are transmitted; Step 3.3, Server Aggregation and Broadcast: The server receives differential parameters from each client. The average aggregation is then performed, and the updated global parameters are broadcast to all clients to enter the next iteration; until the training is completed and a trained multimodal semantic federated learning framework is obtained.

[0027] Furthermore, the specific method of step 3.1 of the present invention is as follows: Client Obtain the set of tags with the highest task relevance:

[0028] The labeled set is used as the input to the compact semantic representation for local decoding and prediction in the detection head. The detection head consists of two parts: a query decoder, which interacts with the input features and the learnable query vector through an attention mechanism to generate predicted query features. The prediction branch outputs each predicted query feature via two parallel branches: ,

[0029] in, Represents class probability, Indicates the normalized bounding box parameters; The client receives the latest round of global parameters from the server. Then, it is synchronized to the local fusion module and the detection head to initialize weights, and local data is used. Perform several rounds of gradient descent updates;

[0030] in, L ( This represents the overall loss from the detection task.

[0031] Furthermore, the specific method of step 3.2 of the present invention is as follows: Transmit only the differential parameters of the fusion module and the detection head :

[0032] Here, fh represents some parameters of the fusion module and the detection head.

[0033] The specific method for step 3.3 is as follows: The federated server receives data from each client. And perform FedAvg aggregation:

[0034] The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0035] This invention provides a multimodal semantic federated learning-based fault diagnosis system for equipment under complex operating conditions, comprising: Several inspection robots, each of which is designed for equipment in complex working conditions, serve as a client. Each client is equipped with multiple types of sensors to collect multimodal data, including images, thermal infrared, and audio. The server constructs a region-aware multimodal semantic federated learning framework with each client and a central server. The server is equipped with a memory and a processor. The memory is used to store executable computer programs. The processor is used to execute the executable computer programs stored in the memory to implement the above-mentioned multimodal semantic federated learning method for diagnosing equipment faults under complex working conditions.

[0036] The beneficial effects of this invention are: 1. This invention constructs a regionally perceptive multimodal semantic federated learning framework for fault diagnosis of equipment under complex operating conditions. Each inspection robot acts as a federated client, integrating multimodal sensors such as image, thermal infrared, and audio to achieve local data acquisition and model training. The clients collaborate to build a global model through a federated mechanism, without sharing raw data, thereby achieving efficient fault diagnosis and privacy protection.

[0037] 2. This invention proposes a region-optimized driven multimodal fusion mechanism to address the problem of insufficient response to small-scale anomalies in traditional global equilibrium modeling. By performing region-aware single-modal feature extraction at the edge and combining it with spatial pyramid compression for fine-grained characterization of local hotspots, the model not only enhances the representation of salient regions but also suppresses the interference of redundant information at the source. Subsequently, in the cross-modal interaction stage, region-guided alignment and fusion are introduced, enabling multimodal representations to form structured complementary representations, thereby obtaining more discriminative multimodal semantic features.

[0038] 3. This invention designs a dynamic communication optimization method based on semantic tagging. After identifying abnormal regions locally and scoring their saliency, only key task tags are retained for uploading. By fusing semantic-spatial diversity constraints, duplicate selections are avoided, ensuring balanced coverage of uploaded information across the global distribution. Combined with a sparse differential update strategy, the system maintains detection accuracy while significantly reducing communication overhead, thus ensuring that the entire federated framework is efficient and privacy-friendly in real-world scenarios. Attached Figure Description

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a system architecture diagram of an embodiment of the present invention.

[0040] Figure 2 This is a flowchart of the method of the present invention.

[0041] Figure 3 This is a comparison of the optimal single-modal and multimodal performance of this invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0043] Example 1 like Figure 1 As shown in the figure, the proposed region-aware multimodal semantic federated learning framework (RS_MFL) for fault diagnosis of equipment under complex operating conditions in this invention is as follows: Figure 1 As shown. Consider a federated learning system for complex inspection scenarios: There is one central server and N inspection robots as clients. Number Each client is equipped with multiple types of sensors, capable of simultaneously acquiring heterogeneous modal data such as visible light (RGB), thermal infrared (IR), and acoustic pulses (m represents the modal category). ), note the client The local multimodal dataset is The goal is to improve fault diagnosis performance for small-area anomalies through semantically aware federated collaboration, while significantly reducing communication overhead, under the constraint that raw data from various clients cannot be centrally shared.

[0044] The system framework consists of three core modules: a region-aware multimodal semantic fusion module, a semantic communication optimization module, and a server-side federated aggregation module.

[0045] First, on the end side The execution region optimization-driven multimodal fusion mechanism allows each client to extract initial features using an independent encoder for different modalities. After extraction, each modal feature undergoes anomaly region mapping and spatial pyramid compression to remove intramodal redundancy. Furthermore, region-guided semantic interaction is used in the cross-modal stage to enhance local anomaly representations, thereby obtaining a compact and semantically prominent fusion feature representation. .

[0046] Then, an anomaly scoring network is constructed based on the fused feature sequences to assign anomaly scores to each label, and this is combined with terminal bandwidth constraints. Based on task urgency, a greedy Top-K strategy is used to select the local representation with the maximum information gain. This significantly reduces communication overhead while ensuring detection performance.

[0047] Finally, the client will The input-side detection head performs local decoding and prediction, and receives the latest global parameters from the cloud in the r-th round. Gradient difference parameters are then obtained through local training. The server performs federated average aggregation after receiving parameter information from all clients. The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0048] Example 2 Based on Embodiment 1, this invention provides the main steps specifically included in the method for implementing the RS_MFL framework of this invention: Step 1: Client For a given mode Input image A modal encoder is used. Extracting multi-scale features Then, pyramid compression features are obtained through local region mapping and a spatial pyramid strategy. Calculate the region attention score for each modality to obtain a multimodal region heatmap. and soft mask Compress features of each mode. Flattened into a labeled sequence ,Will and Region weighting is achieved through label-by-label multiplication. Subsequently, the weighted RGB and TIR labels are input into a cross-modal attention module to obtain a fused label sequence. It not only maintains intermodal complementarity but also highlights the spatial semantics of potential anomalies, laying the foundation for subsequent optimization.

[0049] Step 2: Client based on Communication optimization processing is performed. Specifically, an anomaly scoring network is first used for each label. Calculate the abnormality score Then, an incremental scoring mechanism is introduced, scoring based on the degree of abnormality. And evaluate the marginal value of candidate labels based on the current set of selected labels R. In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is used for key label selection. Among them, It takes into account the communication bandwidth constraints of the end-side devices. and the urgency index of the current task The maximum number of tags uploaded in each round of federated data is dynamically determined, ultimately resulting in the Top-K tags. .

[0050] Step 3: Client Obtain the set of tags with the highest task relevance. Then, these tags are used as compact semantic representations as input to the edge detection head for local decoding and prediction. The edge receives the latest global parameters from the cloud. After (round r), the weights are synchronized to the local fusion module and detection head, and the local data is used. Gradient descent is used to obtain the gradient difference parameters. The differential parameters of the fusion module and the detection head are used. The data is uploaded to the central server, which then performs federated average aggregation after receiving parameters from all clients. The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0051] Repeat the above steps until the termination condition is met, completing the entire training process. The termination condition can be the maximum number of training epochs, the convergence of the loss function, or other user-defined conditions. Finally, the optimal loss function can be obtained according to the following formula:

[0052] in, express The loss of the network.

[0053] After the iteration is completed, the trained model can be obtained and used for fault diagnosis of equipment under complex working conditions.

[0054] In a preferred embodiment of the present invention, the specific method for multimodal fusion of local region optimization in step 1 is as follows: To address the issue of insufficient response of global balanced modeling to minor anomalies (such as localized overheating or minute discharge points in equipment), this invention proposes a region-optimized driven multimodal fusion mechanism. This mechanism effectively enhances the saliency of small-scale anomalies in the fused features through intramodal region-aware feature extraction, spatial pyramid-style hotspot fidelity compression, and cross-modal region-guided fusion, while simultaneously reducing redundant feature transmission and improving communication efficiency.

[0055] Step 1.1: Extraction of Single-Modal Region Perception Features Given from modality Input image The initialization modal encoder is issued by the central server. Extracting multi-scale features:

[0056] in, Indicates high-resolution detail features. This represents the local response characteristics at medium resolution. These are low-resolution global semantic features. C The number of channels in an RGB image. C =3, in infrared images C =1, H The height of the image. W The width of the image.

[0057] Step 1.2: Local Region Mapping and Spatial Pyramid Enhancement To compress redundant regional information while preserving details of abnormal hot spots, embodiments of the present invention introduce a regional heat estimation and pyramid mapping strategy based on medium-resolution features.

[0058] First, the heatmap of the intramodal region is calculated through shallow convolution and normalization operations:

[0059] in, For Sigmoid or Softmax activation functions, the output is... This indicates the intensity of the anomalous response at each spatial location.

[0060] Subsequently, using threshold Divide the area into hotspot and non-hotspot areas:

[0061] in, i Indicates the client number. j This indicates the client's first j One sample; p , qRepresents spatial location coordinates, corresponding to indices in the height and width directions. In hotspot areas... Preserve high-resolution features In the cold zone Only low-resolution features are retained. Spatial pyramid compression features are obtained by aligning and fusing resolutions using bilinear interpolation.

[0062] This operation significantly reduces the number of tags while preserving hotspot details, thereby reducing redundant overhead in subsequent cross-modal processing and communication.

[0063] Step 1.3, Cross-modal region thermal fusion To integrate the complementary responses of different modalities to anomalous regions, the region attention score for each modality is first calculated:

[0064] in, For global average pooling Activated for ReLU These are learnable parameters. The modality fusion weights are then obtained through Softmax regularization.

[0065] The final multimodal region heatmap is obtained:

[0066] Step 1.4: Region-guided cross-modal feature fusion Depend on Generate a soft mask:

[0067] in, The temperature coefficient controls the steepness of the response distribution. The threshold is set as the center. Features of each modality are compressed. Flattened into a labeled sequence And region weighting is achieved through mark-by-mark multiplication:

[0068] Subsequently, the weighted multimodal labels are input into the cross-attention transformation module to obtain the client-side data. Fusion marker sequence for:

[0069] While maintaining modal complementarity, this sequence further highlights the semantic features of anomalous hotspot regions, thereby achieving region-aware-driven high semantic expression.

[0070] Thus, step 1 (region-aware multimodal semantic fusion module) not only completed single-modal encoding and saliency modeling, but also obtained a high-quality labeled representation with regional semantic structure through cross-modal fusion, laying a unified input foundation for subsequent semantic communication optimization and interpretability analysis.

[0071] In a preferred embodiment of the present invention, the specific method for optimizing local semantic communication in step 2 is as follows: To reduce communication overhead and adapt to resource-constrained edge computing environments in small-area anomaly detection tasks, this module structurally optimizes traditional methods that rely on large-scale data transmission and global feature interaction, constructing a simplified semantic delivery mechanism based on local anomaly regions. This mechanism comprises three core steps: anomaly region identification and saliency scoring, locating and quantifying local regions with potential anomaly features; multimodal redundancy suppression and key region selection, filtering the most informative local representations across modalities through semantic diversity constraints; and semantic tag extraction under a dynamic communication budget, adaptively generating and delivering simplified semantic representations based on task relevance and communication cost to support efficient semantic aggregation in federated learning.

[0072] Step 2.1: Anomaly Region Identification and Saliency Scoring Client Labels are obtained through multimodal feature fusion. Then, an anomaly scoring network was used. For each tag Calculate the abnormality score:

[0073] Step 2.2: Multimodal redundancy suppression and key region selection To select key markers with strong representativeness and low redundancy from multimodal features, this invention introduces an incremental scoring mechanism. Based on its abnormality score And, based on the currently selected label set R, evaluate the marginal value of the candidate label t. Its definition is as follows:

[0074] in, Controlling the importance of anomalies, β controls redundancy penalties; Indicates candidate tokens The hybrid similarity between the selected label r and the selected label r. To more effectively suppress redundant labels between modalities and within the spatial neighborhood, a hybrid similarity function that integrates semantic features and spatial location is introduced. Defined as:

[0075] The first term is semantic cosine similarity, which measures the directional consistency between labeled pairs in the feature semantic space, helping to identify semantic redundancy in cross-modal expressions; the second term is spatial Gaussian decay function, used to penalize spatially neighboring labeled regions, effectively dispersing the selection results to enhance coverage; parameters Control the weighting ratio of semantic and spatial similarity. The decay scale is Gaussian. Indicates a marker The two-dimensional center coordinates in the original image can be obtained by positional encoding mapping.

[0076] This design ensures that the selected tag set has semantic diversity and spatial separation, satisfying the discriminative and sparsity requirements of anomaly region tagging representation. This similarity definition not only unifies the semantic and spatial motivations of redundant compression but also makes the subsequent incremental scoring function more interpretable and stable.

[0077] Step 2.3 Semantic Tag Extraction under Dynamic Communication Budget Consider end-side equipment Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. The definition is as follows:

[0078] in, V This represents the average communication cost of a single tag. Due to communication bandwidth constraints, The urgency of the current task. This sets the maximum allowed number of tags for the system. This strategy can effectively control the communication burden while ensuring model performance, achieving resource-adaptive semantic compression.

[0079] In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is adopted for key tag selection. Specifically: First, the selection set R is initialized. Then, iteratively select the label with the largest incremental score from the unselected labels. ,Right now:

[0080]

[0081] Calculated Add elements to set R and repeat the above process until the number of elements in R reaches the upper limit. This greedy strategy ensures that the selected label has the optimal marginal information gain at each step, effectively avoiding redundancy in semantic or spatial dimensions and improving the discriminativeness and compactness of the final representation.

[0082] It is important to note that, considering the Top-K strategy may miss key regions with small scales or low scores during the ranking process, the anomaly-priority retention, hybrid ranking, and dynamic K mechanisms introduced in this embodiment of the invention can achieve high-fidelity semantic transmission and stable detection performance under different task urgency and bandwidth constraints. This process not only significantly reduces the amount of redundant information transmitted, but also ensures that the uploaded features have strong task relevance and semantic coverage through salient region selection, providing high-quality support for subsequent federated model training.

[0083] In a preferred embodiment of the present invention, the specific method for local training and federated aggregation in step 3 is as follows: To further improve communication efficiency and data privacy, this invention employs edge-side semantic encoding and decoding. Unlike cloud-based decoding, local decoding and differential uploading of sparse parameters effectively avoid the direct transmission of high-dimensional features. This process includes three core steps: edge-side local semantic decoding and training update, differential parameter calculation and uploading, and differential parameter calculation and uploading.

[0084] Step 3.1: Local semantic decoding and training update on the device side After completing the key tag filtering, the client Obtain the set of tags with the highest task relevance:

[0085] These tags serve as the input to the compact semantic representation detection head for local decoding and prediction. Generally, the detection head typically consists of two parts: a query decoder, which interacts with the input features and the learnable query vector through an attention mechanism to generate predicted query features. The prediction branch (FFN) outputs each predicted query feature via two parallel branches: ,

[0086] in, Represents class probability, This represents the normalized bounding box parameters.

[0087] The device receives the latest global parameters from the cloud. After (round r), the weights are synchronized to the local fusion module and detection head, and the local data is used. Perform several rounds of gradient descent updates:

[0088] Among them, L( This represents the overall loss from the detection task.

[0089] Step 3.2: Calculation and Upload of Differential Parameters To further reduce communication overhead, this invention does not transmit the complete model weights back, but only the differential parameters of the fusion module and the detection head. :

[0090] Here, fh represents some parameters of the fusion module and the detection head.

[0091] It retains only gradient update terms related to key markers, achieving sparse propagation. This maintains the global model's discriminative performance in outlier regions while significantly reducing gradient redundancy in irrelevant regions.

[0092] Step 3.3, Central Aggregation and Broadcasting The federated server receives data from each client. And perform FedAvg aggregation:

[0093] The updated global parameters are then broadcast to all clients, initiating the next iteration.

[0094] Module 1 (the region-aware multimodal semantic fusion module) uses single-modal redundancy suppression to weight and weaken background noise and highly similar regions within a modality, thereby improving the semantic purity of the fused input. Meanwhile, Module 2 (the semantic communication optimization module) uses multimodal redundancy suppression, based on the fused features and combined with communication budget constraints, to select the markers with the greatest information gain across modalities. These two modules operate at different stages: the former ensures fusion quality, while the latter ensures communication efficiency, forming a complementary rather than conflicting optimization link in the feature semantic space and transmission dimension.

[0095] Example 3: Experimental Test Example To verify the effectiveness of the proposed framework RS_MFL, experimental performance was compared with FedAvg, FedRS, and FedOpt on four existing public datasets: MiT-51, UCI-HAR, KU-HAR, and CrisisMMD, focusing on accuracy (Acc) and communication overhead. For the UCI-HAR and KU-HAR datasets, accelerometer data (Accel) and gyroscope data (Gyro) were set as bimodal datasets, as described in Table 1. To accommodate the limitations of computation and communication overhead in practical inspection robots, a lightweight network structure was adopted in the model design. An RNN-based model architecture was used in the video and text modalities, while a Conv-RNN architecture was used for validation in other modalities. The learning rate was set to 0.01, the training epochs were 200, and the number of clients was 5.

[0096] Table 1: Introduction to the dataset

[0097] 1. Comparison of multimodal and single-modal performance A fundamental research question surrounding multimodal learning is how it performs compared to single-modal models. Figure 3 This paper summarizes the performance comparison between unimodal learning (FL) and multimodal learning (FL). Comparisons were made on the UCI-HAR and CrisisMMD datasets, with the unimodal learning model representing the best performing modality. Overall, unimodal learning is observed to offer competitive performance compared to multimodal FL. However, in most cases, multimodal learning still provides additional advantages, especially when dealing with complex tasks and cross-modal information fusion. For example, in the KU-HAR dataset, the proposed RS_MFL achieves a 10.44% improvement in multimodal FL performance compared to unimodal learning.

[0098] 2. Multimodal performance comparison In the experimental comparison, the proposed local region-aware multimodal semantic federated learning method was compared with three classic federated learning algorithms: FedAvg, FedProx, and FedOpt. It is important to emphasize that, to ensure fairness, all three comparison methods employ the same modality fusion mechanism based on a fundamental attention mechanism; that is, the multimodal features are globally interacted on the client side before being uploaded for federated aggregation. The parameters are set to... The dataset is divided using a Dirichlet distribution with a resolution of 0.1 to simulate a non-independent identically distributed (non-IID) dataset with high heterogeneity among clients.

[0099] As shown in Table 3, the method of this invention achieved optimal performance on multiple public datasets. On the MiT-5 action recognition dataset, traditional attention fusion relies on weighted combination of intermodal features, lacking fine-grained modeling of the relevance of local abnormal regions to the task. Therefore, it often fails to guarantee the effectiveness of cross-modal collaboration when redundant features or noisy modalities exist. The method of this invention, however, can retain key action-related segments through local region perception and anomaly labeling, thus achieving a significant performance improvement. On the UCI-HAR and KU-HAR human action recognition datasets, the method of this invention significantly outperforms all comparable methods in the F1 score, even improving the score by 12.69% compared to FedAvg on the KU-HAR dataset. This demonstrates that in cross-client scenarios, semantic representation based on local regions can enhance robust modeling capabilities for key action patterns. On the CrisisMMD multimodal social media emergency dataset, the method of this invention also achieved the best F1 score, exhibiting strong noise resistance and cross-modal generalization ability.

[0100] Comprehensive analysis reveals that the method of this invention effectively focuses on task-related regions and suppresses redundant features through a local region perception mechanism and semantic tag selection based on anomaly scoring and diversity constraints, thereby maintaining optimal performance under different tasks and partition settings.

[0101] Table 2: Performance comparison of different multimodal methods on different datasets (%)

[0102] 3. Comparison of communication overhead To further verify the advantages of the proposed RS-MFL method in terms of communication efficiency, the training batch size B=32 was uniformly set in the experiments to achieve comparable communication overhead statistics between traditional distributed and federated learning methods. Traditional distributed methods require uploading raw data to the cloud, resulting in huge communication overhead and significant privacy and security risks due to centralized data storage. Federated learning methods, on the other hand, avoid uploading raw data through local model updates and parameter sharing, while also reducing communication volume to some extent. A comparative analysis was conducted on two benchmarks: traditional distributed transmission methods and typical federated learning methods.

[0103] As shown in Table 3, for the low-dimensional sensor datasets UCI-HAR and KU-HAR, traditional distributed methods only need to upload one batch of raw data per round, while FedAvg and FedProx require uploading the complete model parameters. Superficially, the communication overhead of traditional distributed methods appears lower than that of federated methods. This is mainly due to factors such as the small sample size of the HAR dataset, its proximity to IID data, and low task complexity. It is important to emphasize that this is only a single-round comparison. Considering the cumulative communication volume and the large-scale multimodal datasets, the communication overhead of traditional distributed methods, due to the continuous uploading of raw data, is still far higher than that of federated learning methods.

[0104] For the high-dimensional multimodal datasets MiT-51 and CrisisMMD, the traditional distributed learning model incurs communication overhead of 48.78 MB and 16.97 MB per round, respectively. In contrast, the proposed RS-MFL, through local anomaly region filtering and semantic tagging compression, achieves a communication overhead of only 14.15 MB per round, saving 71.04% on MiT-51 and 13.03% on CrisisMMD compared to traditional distributed learning. Compared to FedAvg, FedProx, and FedOpt, RS-MFL reduces communication overhead by an average of approximately 48.17%-50.53% across all datasets, with the most significant advantages on high-dimensional visual or image / text modal datasets like CrisisMMD and MiT-51. These results demonstrate that RS-MFL effectively reduces redundant information transmission while preserving key task features, achieving a balance between low communication overhead and high performance. This enhances the practicality and scalability of multimodal federated learning in bandwidth-constrained or real-time-critical application scenarios, making it particularly suitable for complex inspection scenarios such as transformer fault diagnosis.

[0105] Table 3: Comparison of communication overhead (MB) for different algorithms

[0106] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0107] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions, characterized in that, Each inspection robot, designed for equipment operating in complex conditions, is treated as a client. A region-aware, multimodal semantic federated learning framework is constructed by connecting each client to a central server. This framework includes the following steps: Step 1: Execute a region-aware multimodal semantic fusion mechanism on the client side. Each client extracts initial features using an independent encoder for different modalities. After extraction, the initial features of each modality are enhanced by local region mapping and spatial pyramid enhancement to remove intramodal redundancy. In the cross-modal stage, region-guided semantic interaction is used to strengthen the representation of local anomalies, resulting in a fusion feature sequence that is structurally compact and semantically prominent. Step 2: The client performs local semantic communication optimization processing based on the fused feature sequence; an anomaly scoring network is used to calculate an anomaly score for each tag in the fused feature sequence, and an incremental scoring mechanism is introduced. Based on the anomaly score and the mixed similarity of candidate tags on the basis of the current selected tag set, the mixed similarity is composed of semantic cosine similarity and spatial Gaussian decay function, which serves as the incremental scoring function for guidance; and combined with the client's bandwidth constraints and task urgency, a greedy Top-K strategy is used to select key tags, dynamically determine the upper limit of the number of tags uploaded in each round of federated processing, and obtain the tag set with the highest task relevance. Step 3: The client inputs the set of tags with the highest task relevance into the end-side detection head for local decoding and prediction; after receiving the latest round of global parameters from the server, the client performs local training to obtain gradient difference parameters; after receiving the gradient difference parameters from all clients, the server performs federated averaging and aggregation; the updated global parameters are then broadcast to all clients to enter the next iteration; until training is completed; the trained region-aware multimodal semantic federated learning framework is used for fault diagnosis of equipment under complex working conditions. The specific method for step 1 is as follows: Step 1.1, Single-modal region-aware feature extraction: Client For a given mode Input image The server-issued initialization modal encoder is used. Extracting multi-scale features ; Step 1.2, Local Region Mapping and Spatial Pyramid Augmentation: Pyramid compression features are obtained through local region mapping and spatial pyramid strategies. ; Step 1.3, Cross-modal region heatmap fusion: Calculate the region attention score for each modality to obtain a multimodal region heatmap. ; Step 1.4, Region-guided cross-modal feature fusion: Based on multimodal region heatmaps soft mask Compress features of each mode Flattened into a labeled sequence Region weighting is achieved through label-by-label multiplication. The weighted multimodal labels are then input into the cross-modal attention module to obtain the fused feature sequence. ; In step 1.1, a dedicated modal encoder is used for initialization. The specific method for extracting multi-scale features is as follows: in, Indicates high-resolution detail features. This represents the local response characteristics at medium resolution. For low-resolution global semantic features, the input image , C The number of channels in an RGB image. C =3, in infrared images C =1, H The height of the image. W The width of the image; In step 1.2, the heatmap of the intramodal region is calculated through shallow convolution and normalization operations: in, For Sigmoid or Softmax activation functions, the output is... Indicates the intensity of the anomalous response at each spatial location; Using threshold Divide the area into hotspot and non-hotspot areas: in, i Indicates the client number. j This indicates the client's first j One sample; p , q Represents spatial location coordinates, corresponding to indices in the height and width directions; in hotspot areas. Preserve high-resolution features In the cold zone Only low-resolution features are retained. Spatial pyramid compression features are obtained by aligning and fusing resolutions using bilinear interpolation. ； The specific method for step 1.3 is as follows: Calculate the region attention score for each modality. : in, For global average pooling Activated for ReLU These are learnable parameters; the modal fusion weights are obtained through Softmax regularization. This leads to the multimodal region heat map: The specific method for step 1.4 is as follows: Depend on Generate a soft mask: in, The temperature coefficient controls the steepness of the response distribution. The threshold is set to centralize; features of each mode are compressed. Flattened into a labeled sequence , This represents the number of spatial labels for the m-th mode after flattening, and region weighting is achieved through label-by-label multiplication: The weighted multimodal labels are input into the cross-modal attention module to obtain the client. fusion feature sequence for: 。 2. The multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions according to claim 1, characterized in that, The specific method for step 2 is as follows: Step 2.1, Abnormal Region Identification and Saliency Scoring: Client Labels are obtained through multimodal feature fusion. Then, an anomaly scoring network was used. For each tag Calculate the abnormality score ; Step 2.2, Multimodal Redundancy Suppression and Key Region Optimization: Introducing an Incremental Scoring Function Based on the abnormality score and the currently selected set of tags R Evaluate candidate labels based on t The hybrid similarity is composed of semantic cosine similarity and spatial Gaussian decay function; Step 2.3, Semantic Tag Extraction under Dynamic Communication Budget: Considering the Client Communication bandwidth constraints and the urgency index of the current task Dynamically determine the upper limit of the number of tags uploaded in each round of federated updates. In the constructed incremental scoring function Under the guidance of this method, a greedy Top-K strategy is used to select the key markers, resulting in k marker information. .

3. The multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions according to claim 2, characterized in that, Incremental scoring function in step 2.2 for: in, The importance of controlling anomalies β Control redundancy penalties; Indicates candidate tokens With selected marker r Hybrid similarity between modalities; to suppress redundant labels between modalities and within spatial neighborhoods, a hybrid similarity function fusing semantic features and spatial location is introduced. Defined as: The first term is semantic cosine similarity, and the second term is spatial Gaussian decay function; parameters Control the weighting ratio of semantic and spatial similarity. The decay scale is Gaussian. Indicates a marker The two-dimensional center position coordinates in the original image are obtained by positional encoding mapping; The upper limit of the number of markers in step 2.3 The definition is as follows: in, V This represents the average communication cost of a single tag. Due to communication bandwidth constraints, Assessing the urgency of the current task, the indicator The maximum number of tags allowed by the system; In the constructed incremental scoring function Under guidance, a greedy Top-K strategy is adopted for key label selection; specifically: initializing the selected set. R ← Iteratively select the label with the largest incremental score from the unselected labels. ,Right now: Calculated Add to collection R Repeat the process of selecting and adding items to the set until... R The number of elements in the middle reaches the limit .

4. The multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions according to claim 1, characterized in that, The specific method for step 3 is as follows: Step 3.1, Client-side local semantic decoding and training update: Client Obtain the set of tags with the highest task relevance. Then, it is used as the input of the compact semantic representation to the end-side detection head for local decoding and prediction; The client receives the latest round of global parameters from the server. Then, the gradient difference parameters are obtained through local training. ; Step 3.2, Differential Parameter Calculation and Upload: Only the differential parameters of the fusion module and the detection head are transmitted; Step 3.3, Server Aggregation and Broadcast: The server receives differential parameters from each client. The average aggregation is then performed, and the updated global parameters are broadcast to all clients to enter the next iteration; until the training is completed and a trained multimodal semantic federated learning framework is obtained.

5. The multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions according to claim 4, characterized in that, The specific method for step 3.1 is as follows: Client Obtain the set of tags with the highest task relevance: The labeled set is used as the input to the edge detection head for local decoding and prediction, which consists of two parts: a query decoder and a predictive query feature. The query decoder interacts with the input features and the learnable query vector through an attention mechanism. The prediction branch outputs each predicted query feature via two parallel branches: ， in, Represents class probability, Indicates the normalized bounding box parameters; The client receives the latest round of global parameters from the server. Then, it is synchronized to the local fusion module and the detection head to initialize weights, and local data is used. Perform several rounds of gradient descent updates; in, L ( This represents the overall loss from the detection task.

6. The multimodal semantic federated learning method for fault diagnosis of equipment under complex operating conditions according to claim 5, characterized in that, The specific method for step 3.2 is as follows: Transmit only the differential parameters of the fusion module and the detection head : Where fh represents some parameters of the fusion module and the detection head; The specific method for step 3.3 is as follows: The federated server receives data from each client. And perform FedAvg aggregation: The updated global parameters are then broadcast to all clients, initiating the next iteration.

7. A multimodal semantic federated learning system for diagnosing equipment faults under complex operating conditions, characterized in that, include: Several inspection robots, each of which is designed for equipment in complex working conditions, serve as a client. Each client is equipped with multiple types of sensors to collect multimodal data, including images, thermal infrared, and audio. The server constructs a region-aware multimodal semantic federated learning framework with each client and a central server; the server is equipped with a memory and a processor; the memory is used to store executable computer programs; the processor is used to execute the executable computer programs stored in the memory to implement the multimodal semantic federated learning complex working condition equipment fault diagnosis method as described in any one of claims 1 to 6.

Images