Federated learning methods, apparatuses, devices, storage media, and program products

Abstract

The application discloses a federated learning method, device, equipment, storage medium and program product. The federated learning method comprises the following steps: obtaining learning data of a first modality of a first client and learning data of a second modality of a second client; inputting the learning data of the first modality into a pseudo-modality data generation model to obtain pseudo-modality learning data of the second modality output by the pseudo-modality data generation model; obtaining target learning data of the second modality aligned with the learning data of the first modality according to the similarity between the pseudo-modality learning data of the second modality and the learning data of the second modality; aligning and splicing the learning data of the first modality and the target learning data of the second modality to obtain federated learning data; uploading the federated learning data to a centralized server to perform federated learning, and obtaining a communication operator fault model. The application realizes alignment and fusion of cross-modality data of the client, and improves the accuracy of the obtained communication operator fault model.

Description

Technical Field

[0001] This application belongs to the field of big data technology, specifically relating to a federated learning method, apparatus, device, storage medium, and program product. Background Technology

[0002] Multimodal federated hybrid modeling refers to the joint modeling of multiple data modalities (such as images and text) within a federated learning framework. This aims to achieve collaborative learning across devices and modalities while protecting privacy. Simultaneously, the hybrid modeling approach enhances the model's adaptability to heterogeneous data and improves overall performance. This multimodal federated hybrid modeling can be applied to fault diagnosis in telecommunications operators.

[0003] Existing technologies have made some progress in multimodal federated modeling. For example, they address heterogeneous data fusion issues through temporal processing, text feature extraction, and contrastive learning alignment modules. Furthermore, they introduce nonnegative matrix factorization and label correlation matrices within the federated framework to improve privacy protection and multi-label classification performance in multimodal information fusion. However, these methods suffer from three major drawbacks when applied to the fault diagnosis domain of telecommunications operators: First, they assume clients possess complete multimodal data, but in practice, clients often only have a single modality (e.g., one client only has images, another only has text), leading to prominent modality missing issues and impacting model training performance. Second, in the data isolation environment of federated learning, clients cannot directly share detailed data, making cross-client, cross-modal contrastive learning difficult to implement effectively, thus limiting modality alignment and reducing model generalization ability. Third, global aggregation only allocates weights based on the amount of client data, without considering modality scarcity and the reduction ratio of alignment loss, resulting in unreasonable aggregation strategies and low modeling accuracy. Therefore, effectively addressing client modality missing issues and achieving efficient cross-client, cross-modal alignment in multimodal federated hybrid modeling is a pressing problem that needs to be solved. Summary of the Invention

[0004] This application provides a federated learning method, apparatus, device, storage medium, and program product to address the problem that existing federated learning methods cannot effectively handle modality missing data from clients.

[0005] Firstly, this application provides a federated learning method, the method comprising:

[0006] Acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults;

[0007] The learning data of the first modality is input into the pseudomodal data generation model to obtain the pseudomodal learning data of the second modality output by the pseudomodal data generation model.

[0008] Based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality, target learning data of the second modality that is aligned with the learning data of the first modality is obtained.

[0009] The learning data of the first modality and the target learning data of the second modality are aligned and concatenated to obtain federated learning data. The federated learning data is uploaded to a centralized server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data of the communication operator.

[0010] Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

[0011] Optionally, obtaining target learning data for the second modality aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality includes:

[0012] The first learning data with the highest similarity to the learning data of the second modality is selected from the pseudo-modal learning data of the second modality.

[0013] The first learning data is input into the data alignment model to obtain the predicted scene domain of the first learning data as predicted by the data alignment model.

[0014] Based on the predicted scene domain and the real scene domain corresponding to the learning data of the first modality, the domain alignment accuracy between the learning data of the first modality is determined;

[0015] If the first condition is met, the first learning data is used as the target learning data for the second modality; if the first condition is not met, the target learning data for the second modality is obtained based on the first learning data and the pseudo-modal learning data of the second modality.

[0016] The first condition includes:

[0017] The alignment accuracy of the domain corresponding to the first learning data exceeding a preset proportion is greater than a preset accuracy threshold.

[0018] Optionally, the pseudo-modal data generation model includes a pseudo-modal generator and a pseudo-modal discriminator;

[0019] The method further includes:

[0020] Acquire training data of the first modality of the third client and training data of the second modality of the third client; wherein, the training data is repair data in the field of communication operator faults;

[0021] Feature extraction is performed on the training data of the first modality to obtain the first training data features, and feature extraction is performed on the training data of the second modality to obtain the second training data features;

[0022] Based on the first training data features and the second training data features, N training iterations are performed to obtain the pseudomodal data generation model;

[0023] The nth training iteration includes:

[0024] The first input data is obtained by concatenating the features of the first training data and the first noise information;

[0025] The first input data and the features of the second training data are input into the pseudo-modality generator, and the generator parameters of the pseudo-modality generator are used to generate the first pseudo-modality data based on the first input data and the features of the second training data.

[0026] The first pseudomodal data and the first training data features are input into the pseudomodal discriminator. The discriminator parameters of the pseudomodal discriminator are used to obtain a first discrimination result of the first pseudomodal data and a second discrimination result of the first training data features. The discriminator parameters are updated according to the first discrimination result and the second discrimination result to obtain the updated discriminator parameters. The first discrimination result is used to indicate the recognition result between the first pseudomodal data and the first label, and the second discrimination result is used to indicate the recognition result between the first training data features and the second label. The first label is the label of the pseudomodal data, and the second label is the label of the training data.

[0027] The generator parameters are updated based on the first discrimination result to obtain the updated generator parameters;

[0028] Where N is a positive integer, and n is an integer greater than 0 and less than N;

[0029] When n equals 1, the generator parameters of the pseudo-modal generator are the first preset parameters, and the discriminator parameters of the pseudo-modal discriminator are the second preset parameters; when n is greater than 1, the generator parameters of the pseudo-modal generator are the updated generator parameters corresponding to the (n-1)th training, and the discriminator parameters of the pseudo-modal discriminator are the updated discriminator parameters corresponding to the (n-1)th training.

[0030] Optionally, generating the first pseudo-modal data using the generator parameters of the pseudo-modal generator based on the features of the first input data and the second training data includes:

[0031] The pseudo-modal generator is used to generate initial pseudo-modal data based on the first input data;

[0032] The dimension parameters of the first pseudomodal data in the generator parameters are determined based on the dimension parameters of the second training data features.

[0033] A mapping matrix is ​​generated based on the dimension parameters of the first pseudomodal data and the output dimension parameters in the generator parameters;

[0034] The first pseudomodal data is generated based on the mapping matrix and the initial pseudomodal data.

[0035] Optionally, obtaining a first discrimination result of the first pseudo-modal data and a second discrimination result of the first training data features using the discriminator parameters of the pseudo-modal discriminator includes:

[0036] Obtain the normal distribution information of the features of the first training data;

[0037] The target pseudo-modal data in the first pseudo-modal data is determined based on the normal distribution information;

[0038] The first probability that the target pseudomodal data is pseudomodal data is obtained using the discriminator parameters, and the first discrimination result is obtained based on the first probability;

[0039] The second probability that the first training data features are training data is obtained using the discriminator parameters, and the second discrimination result is obtained based on the second probability.

[0040] Optionally, the method further includes:

[0041] Obtain the training data of the first modality of the fourth client and the training data of the second modality of the fourth client;

[0042] Feature extraction is performed on the training data of the first modality to obtain the third training data feature, and feature extraction is performed on the training data of the second modality to obtain the fourth training data feature;

[0043] Using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality, the model parameters of the initial model for the scene domain used to predict the data are trained to obtain the data alignment model.

[0044] Optionally, the model parameters of the initial model include model parameters of a first layer for semantic abstraction of the data and model parameters of a second layer for aligning the data with the scene domain;

[0045] The step of training the model parameters of the initial model for the scene domain used to predict the data using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality to obtain the data alignment model includes:

[0046] The model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the third training data to obtain the first training prediction scenario domain; and the model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the fourth training data to obtain the second training prediction scenario domain.

[0047] The data alignment model is obtained by training the model parameters of the first partial layer and the second partial layer based on the loss function between the first training prediction scene domain and the scene domain corresponding to the training data of the first modality, and the loss function between the second training prediction scene domain and the scene domain corresponding to the training data of the second modality.

[0048] Optionally, the step of aligning and concatenating the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and uploading the federated learning data to a centralized server for federated learning, includes:

[0049] Gradient information is obtained by aligning and concatenating the learning data of the first modality and the target learning data of the second modality, and the gradient information is uploaded to the centralized server for federated learning.

[0050] The centralized server is used to obtain the sample size, modality number, and alignment loss reduction information of the federated learning data based on the gradient information, and to perform federated learning based on the sample size, modality number, and alignment loss reduction information of the federated learning data.

[0051] Secondly, embodiments of this application also provide a federated learning device, the device comprising:

[0052] The first acquisition module is used to acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults;

[0053] The first processing module is used to input the learning data of the first modality into the pseudo-modal data generation model to obtain the pseudo-modal learning data of the second modality output by the pseudo-modal data generation model.

[0054] The second processing module is used to obtain target learning data of the second modality that is aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality.

[0055] The third processing module is used to align and stitch together the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and upload the federated learning data to the central server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data reported by the communication operator.

[0056] Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

[0057] Thirdly, embodiments of this application also provide a federated learning device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the federated learning method as described in any one of the first aspects.

[0058] Fourthly, embodiments of this application also provide a readable storage medium storing a program that, when executed by a processor, implements the steps of the federated learning method as described in any one of the first aspects.

[0059] Fifthly, embodiments of this application also provide a computer program product, including computer instructions that, when executed by a processor, implement the steps in the federated learning method as described in any one of the first aspects.

[0060] The beneficial effects of this application are:

[0061] The federated learning method provided in this application acquires learning data of a first modality from a first client and learning data of a second modality from a second client. The learning data is fault reporting data from a telecommunications operator. The first modality is either text-based or image-based; if the first modality is text-based, the second modality is image-based; if the first modality is image-based, the second modality is text-based. This indicates that the modal data from both the first and second clients exhibit modality missing information. The learning data of the first modality is input into a pseudo-modal data generation model to obtain a pseudo-modal of the second modality output by the model. The learning data is obtained by aligning the target learning data of the second modality with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality. The learning data of the first modality and the target learning data of the second modality are then aligned and concatenated to obtain federated learning data, thereby realizing the alignment and fusion of cross-modal data of the client. The federated learning data is then uploaded to a centralized server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data of the communication operator, thereby improving the accuracy of the obtained communication operator fault model. Attached Figure Description

[0062] Figure 1 This is a flowchart of the federated learning method provided in the embodiments of this application;

[0063] Figure 2 This is a flowchart illustrating the federated learning method provided in the embodiments of this application;

[0064] Figure 3 This is a schematic diagram of the structure of the federated learning device provided in the embodiments of this application;

[0065] Figure 4 This is a schematic diagram of the structure of the federated learning device provided in the embodiments of this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0068] In the following description, specific details such as particular configurations and components are provided merely to aid in a comprehensive understanding of the embodiments of this application. Therefore, those skilled in the art will understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Furthermore, for clarity and brevity, descriptions of known functions and constructions have been omitted.

[0069] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0070] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0071] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0072] Furthermore, the "or" in this application indicates at least one of the connected objects. For example, "A or B" covers three scenarios: Scenario 1: includes A but excludes B; Scenario 2: includes B but excludes A; Scenario 3: includes both A and B. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0073] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as one in which the sender explicitly informs the receiver of specific information, the operation to be performed, or the requested result, etc.; an indirect instruction can be understood as one in which the receiver determines the corresponding information based on the instruction sent by the sender, or makes a judgment and determines the operation to be performed or the requested result, etc., based on the judgment result.

[0074] Before describing the specific implementation methods of this application, the following will be explained first:

[0075] Multimodal federated learning is a distributed machine learning method that combines a federated learning framework with multimodal data processing. It effectively addresses data silos and privacy protection challenges by enabling joint modeling of multi-source heterogeneous data (such as images, text, voice, and sensor signals) across devices and institutions without sharing the original data. Its applications are widely distributed across fields such as healthcare, intelligent transportation, smart homes, and financial risk control. From a commercial perspective, this technology not only helps enterprises compliantly utilize distributed data to improve the performance of artificial intelligence (AI) models and reduce data sharing risks and compliance costs, but also fosters innovative service models through cross-industry collaboration, such as precision medicine and city-level intelligent decision-making. This opens up new business growth points for industries with strict data privacy requirements and promotes the large-scale deployment of artificial intelligence while protecting user rights.

[0076] To address the problem that existing federated learning methods cannot effectively handle modality missing data from clients, embodiments of this application provide a federated learning method, apparatus, device, storage medium, and program product.

[0077] like Figure 1 As shown in the embodiment of this application, a federated learning method is provided, the method comprising:

[0078] Step 101: Obtain the learning data of the first modality of the first client and the learning data of the second modality of the second client, wherein the learning data is the repair data in the field of communication operator faults.

[0079] Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

[0080] It should be noted that the learning data for the complete modality of the client includes data for both the image modality and the text modality.

[0081] The first client has learning data for the first modality, and the second client has learning data for the second modality. Therefore, it can be assumed that the data from the first client and the second client have modality missing.

[0082] For example, the first client has repair data in text mode and the second client has repair data in image mode. In the process of multimodal federated learning, since the clients cannot directly share the original data, the multimodal data of text and images are missing or cannot be aligned, making it difficult to perform joint modeling. It is necessary to complete and align the cross-modal data in federated learning.

[0083] Step 102: Input the learning data of the first modality into the pseudomodal data generation model to obtain the pseudomodal learning data of the second modality output by the pseudomodal data generation model.

[0084] In this embodiment of the application, taking the first modality's learning data as the text modality's repair data as an example, the text modality's repair data is generated into the pseudo modality data generation model to obtain the pseudo modality image repair data corresponding to the text modality's repair data generated by the pseudo modality data generation model.

[0085] Step 103: Based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality, obtain the target learning data of the second modality that is aligned with the learning data of the first modality.

[0086] The similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality is obtained by cosine similarity matching.

[0087] Step 104: Align and concatenate the learning data of the first modality and the target learning data of the second modality to obtain federated learning data. Upload the federated learning data to a centralized server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data reported by the communication operator.

[0088] Specifically, in this step, taking a federated learning scenario where a telecommunications operator uses multimodal data of fault report text and base station fault images to achieve fault classification as an example, the feature dimension of the first modality's learning data (text-based fault report data) is 768-dimensional, and the feature dimension of the second modality's learning data (image-based fault report data) is 12288-dimensional. The first modality's learning data and the second modality's target learning data are aligned and concatenated to obtain federated learning data. The joint feature dimension of this concatenated federated learning data is 768-dimensional + 12288-dimensional.

[0089] It should be noted that, in this embodiment, the data participating in federated learning includes the learning data of the first modality and the target learning data of the second modality. It may also include the intermediate gradient information obtained by aligning and splicing the learning data of the first modality and the learning data of the second modality of the client to the central server. Optionally, the intermediate gradient information corresponding to the learning data of the first modality and the learning data of the second modality of the client is uploaded to the central server by means of homomorphic encryption.

[0090] The centralized server aggregates the received data and performs federated learning to obtain a fault model for the telecommunications operator. The input of this fault model is the repair data reported by the telecommunications operator, and the output is the fault classification information of the telecommunications operator.

[0091] In some embodiments of this application, obtaining target learning data for the second modality aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality includes:

[0092] The first learning data with the highest similarity to the second modality's learning data is selected from the pseudo-modal learning data of the second modality. This involves performing cosine similarity matching between the pseudo-modal learning data of the second modality and the learning data of the second modality, and using the first learning data with the highest similarity to the second modality's learning data as the aligned image of the first modality's learning data.

[0093] The first learning data is input into the data alignment model to obtain the predicted scene domain of the first learning data predicted by the data alignment model.

[0094] The data alignment model is trained using training data and the corresponding scenario domain. For example, the training data consists of repair reports from a telecommunications operator's fault domain, and the scenario domain is the telecommunications operator's fault domain. Optionally, the training data can also be data from an industrial sector, and the scenario domain can be the industrial sector.

[0095] The input to the data alignment model is the reported fault data, and the output of the data alignment model is the predicted fault domain of the telecommunications operator (i.e., the predicted scenario domain) based on the reported fault data.

[0096] Based on the predicted scene domain and the real scene domain corresponding to the learning data of the first modality, the domain alignment accuracy between the learning data of the first modality is determined. This alignment accuracy can also be understood as alignment precision.

[0097] If the first condition is met, the first learning data is used as the target learning data for the second modality; if the first condition is not met, the target learning data for the second modality is obtained based on the first learning data and the pseudo-modal learning data of the second modality.

[0098] The first condition includes:

[0099] The alignment accuracy of the domain corresponding to the first learning data exceeding a preset proportion is greater than a preset accuracy threshold.

[0100] For example, the preset ratio is 50% and the preset accuracy threshold is 90%.

[0101] It is understandable that a prediction scenario domain can be obtained by using a data alignment model to predict a first set of learning data.

[0102] The process of obtaining the target learning data for the second modality based on the first learning data and the pseudo-modal learning data of the second modality includes: using the pseudo-modal learning data of the second modality to modally supplement the first learning data to obtain the target learning data for the second modality.

[0103] Specifically, if more than 50% of the first learning data corresponds to a domain alignment accuracy greater than 90%, the first learning data is used as the target learning data for the second modality; otherwise, the first learning data is used to supplement the pseudo-modal learning data of the second modality to obtain the target learning data for the second modality.

[0104] In one optional embodiment, the pseudomodal data generation model includes a pseudomodal generator and a pseudomodal discriminator. The pseudomodal generator includes a pseudomodal text data generator and a pseudomodal image data generator. The pseudomodal text data generator generates pseudomodal text data based on image data, and the pseudomodal image data generator generates pseudomodal image data based on text data. The pseudomodal discriminator includes a pseudomodal text data discriminator and a pseudomodal image data discriminator.

[0105] The method further includes:

[0106] The training data for the first modality of the third client and the training data for the second modality of the third client are acquired. The training data is fault reporting data from the telecommunications operator's fault domain. This training data can also be referred to as local raw data.

[0107] In this step, the third client can be understood as a client that simultaneously possesses repair data in both text and image formats.

[0108] Aligned multimodal data from clients possessing full modalities are used as training samples for the pseudomodal data generation model, such as clients that simultaneously possess fault report text and base station fault images. Clients participating in federated learning that have missing modalities preprocess their local raw data, including data cleaning and normalization. Appropriate feature extractors are used to extract features for different modalities.

[0109] Feature extraction is performed on the training data of the first modality to obtain the first training data features, and feature extraction is performed on the training data of the second modality to obtain the second training data features.

[0110] Prior to this, for clients with full modal data text in and images Perform data preprocessing.

[0111] Specifically, the training data for the first modality is cleaned and normalized. Then, domain-adaptive feature extraction is performed on the training data for the first modality to obtain the first training data features. For example, in the case of image-based repair data, the first training data features are image features. Similarly, the training data for the second modality is cleaned and normalized. Then, domain-adaptive feature extraction is performed on the training data for the second modality to obtain the second training data features. For example, in the case of text-based repair data, the second training data features are text features.

[0112] The training data for the first modality consists of original images from Contrastive Language–Image Pre-training (CLIP). Wherein, the ViT function is the CLIP image encoder with a dimension of 64*64*3=12288, and the image features after domain adaptation are shown in Formula 1:

[0113] Formula 1

[0114] in, This represents the domain indicator vector (e.g., a fault domain for a telecommunications operator is set to [1,0], and an industrial domain is set to [0,1]). The dimension is 12288×2, which is a learnable adaptation matrix (d is the feature dimension, k is the number of domain categories).

[0115] The training data for the second modality consists of original text from Contrastive Language–Image Pre-training (CLIP). The Text function is a CLIP image encoder with a dimension of 768. The domain-adapted text features are shown in Formula 2.

[0116] Formula 2

[0117] in, This represents the domain indicator vector (e.g., a fault domain for a telecommunications operator is set to [1,0], and an industrial domain is set to [0,1]). The dimension is 768×2, which is a learnable adaptation matrix (d is the feature dimension, k is the number of domain categories).

[0118] By introducing domain-specific features to enhance the learnable adaptation matrix, feature offset correction for aligning multimodal data across different industry sectors can be achieved.

[0119] The goal of training the pseudo-modal discriminator is to maximize its ability to distinguish between real and generated data, while the goal of training the pseudo-modal generator is to minimize the discriminator's probability of recognizing generated data.

[0120] Based on the first training data features and the second training data features, N training iterations are performed to obtain the pseudomodal data generation model;

[0121] The nth training iteration includes:

[0122] The first training data features and the first noise information are concatenated to obtain the first input data. The first training data features can be the features of a randomly selected batch of training data, and the first noise information is randomly generated noise. The concatenation of the first training data features and the first noise information is used as the input of the pseudo-modal generator.

[0123] The first input data and the features of the second training data are input into the pseudo-modal generator, and the generator parameters of the pseudo-modal generator are used to generate the first pseudo-modal data based on the first input data and the features of the second training data.

[0124] The first pseudomodal data and the first training data features are input into the pseudomodal discriminator. The discriminator parameters of the pseudomodal discriminator are used to obtain a first discrimination result for the first pseudomodal data and a second discrimination result for the first training data features. The discriminator parameters are then updated based on the first and second discrimination results to obtain the updated discriminator parameters. The first discrimination result indicates the recognition result between the first pseudomodal data and the first label, and the second discrimination result indicates the recognition result between the first training data features and the second label. The first label is the label of the pseudomodal data, and the second label is the label of the training data. When n equals 1, the discriminator parameters of the pseudomodal discriminator are the second preset parameters; when n is greater than 1, the discriminator parameters of the pseudomodal discriminator are the updated discriminator parameters corresponding to the (n-1)th training iteration.

[0125] Optionally, pseudomodal data can be represented by label 0, and training data (i.e., real data) can be represented by label 1.

[0126] Specifically, during training, the discrimination results of real samples are made as close as possible to label 1, and the discrimination results of generated pseudomodal data are made as close as possible to label 0. The pseudomodal discriminator parameters are updated through backpropagation.

[0127] The generator parameters are updated based on the first discrimination result to obtain the updated generator parameters. Specifically, when n equals 1, the generator parameters of the pseudo-modal generator are the first preset parameters; when n is greater than 1, the generator parameters of the pseudo-modal generator are the updated generator parameters corresponding to the (n-1)th training iteration.

[0128] Specifically, the generated pseudo-data is input into the pseudo-modal discriminator, and its discrimination result is made as close as possible to label 0. The pseudo-modal generator parameters are then updated through backpropagation.

[0129] Where N is a positive integer, and n is an integer greater than 0 and less than N. Optionally, N is 1000.

[0130] That is, iterate the training process for 1000 rounds until the model converges.

[0131] Subsequently, the final image pseudomodal generator, image pseudomodal discriminator, text pseudomodal generator, and text pseudomodal discriminator are obtained through training.

[0132] It should be noted that in federated learning, when the centralized server aggregates the intermediate gradients from each client, it involves not only clients with full-modal data but also clients with only partial-modal data. Taking a federated learning scenario where a telecommunications operator uses multi-modal data of fault report text and base station fault images for fault classification as an example, some clients only have fault report text, while others only have base station fault images. When the centralized server aggregates the gradients from each client, the fault report text and base station fault images need to be aligned for federated learning computation. This embodiment proposes that the centralized server use the aligned sample data from clients with full-modal data to train a pseudo-modal data generator, thereby generating pseudo-modal data to supplement and align missing modalities.

[0133] Optionally, generating the first pseudo-modal data using the generator parameters of the pseudo-modal generator based on the features of the first input data and the second training data includes:

[0134] The pseudo-modal generator is used to generate initial pseudo-modal data based on the first input data.

[0135] Specifically, the pseudo-modal generator generates virtual modal data based on a noise vector (first noise information) z and existing features (first training data features) F, and a discriminator is used to distinguish between real data and generated data. For example, if a client has image modal data and needs to generate text modal data, a corresponding text pseudo-modal generator needs to be constructed. Text pseudomodal discriminator A client has text modal data and needs to generate image modal data; similarly, a corresponding image pseudomodal generator needs to be built. Text pseudomodal discriminator .

[0136] Pseudomodal generator structure: Employs a multi-layer neural network, with noise vector as input. and text or image vectors The concatenated vector (first input data) is Output virtual data The calculation method is as shown in Formula 3:

[0137] Formula 3

[0138] in, Represents a linear transformation matrix. The activation function is (e.g., the Rectified Linear Unit (ReLU) function). Taking the federated learning scenario where a telecommunications operator uses multimodal data of fault report text and base station fault images to achieve fault classification as an example, the pseudo-modal generator... Input is noise and image feature vectors The concatenated vector is Virtual text (i.e., initial pseudomodal data) is generated through a two-layer network:

[0139] Formula 4

[0140] in, , .

[0141] The dimension parameters of the first pseudomodal data in the generator parameters are determined based on the dimension parameters of the features of the second training data.

[0142] Specifically, the pseudo-modal generator also includes a dynamic dimension adaptation layer. This embodiment addresses the issue of differing dimensions of missing modal data across different clients by using this dynamic dimension adaptation layer to resolve modal heterogeneity. Since the dimensions of missing modal data may differ across different federated learning tasks (e.g., text modality is 768 dimensions, image modality is 12288 dimensions), the dynamic dimension adaptation layer automatically adjusts the generator's output dimensions to ensure that the generated data matches the target modal dimensions.

[0143] During the initialization process of the pseudo-modal generator, the dimension parameters of the target modality are input. The dimensionality parameter of the target mode This refers to the dimension parameter of the second training data feature. For example, the dimension of the data feature of the image modality is 12288.

[0144] A mapping matrix is ​​generated based on the dimension parameters of the first pseudomodal data and the output dimension parameters in the generator parameters.

[0145] Specifically, based on the dimensional parameters of the target modality and the penultimate layer output dimension of the pseudomodal generator Dynamically generate mapping matrix , Among them, the penultimate layer output dimension of the pseudo-modal generator It has 2048 dimensions.

[0146] The first pseudomodal data is generated based on the mapping matrix and the initial pseudomodal data.

[0147] The penultimate layer output of the initial pseudomodal data is obtained through a mapping matrix. The data is converted to the target dimension, and an activation function (Sigmoid function) is selected based on the modality type to generate the first pseudomodal data, as shown in Formula 5:

[0148] Formula 5

[0149] in, For example, the sigmoid function.

[0150] Taking the federated learning scenario where the aforementioned telecommunications operator uses multimodal data of fault report text and base station fault images to achieve fault classification as an example, the target modality... For a 12288-dimensional base station fault image, the output of the penultimate layer of the pseudo-modality generator is... The mapping matrix is ​​dynamically generated with 2048 dimensions. After transformation, it is activated by Sigmoid and the output has 12288 dimensions. By introducing a dynamically generated mapping matrix, it can be adapted to any output dimension vector requirement.

[0151] It should be noted that the pseudomodal discriminator D employs a multi-layer neural network structure, with input data generated by the pseudomodal generator. A domain consistency verification layer is incorporated to address the issue of cross-client data distribution discrepancies. Its core principle lies in the fact that during federated learning, the actual data distributions of each client may differ. By verifying the consistency between the input data and the local data distributions of each client, abnormal multimodal data samples are eliminated, thereby improving the accuracy of federated learning.

[0152] Optionally, obtaining a first discrimination result of the first pseudo-modal data and a second discrimination result of the first training data features using the discriminator parameters of the pseudo-modal discriminator includes:

[0153] Obtain the normal distribution information of the features of the first training data;

[0154] The target pseudo-modal data in the first pseudo-modal data is determined based on the normal distribution information;

[0155] Before the pseudo-modal discriminator outputs the discrimination result, the following is performed first:

[0156] Local domain distribution modeling: During the initialization phase of each client, a normal distribution model of the client's data is trained using local modal data, and the statistical characteristics such as the mean and variance of the client's data are recorded.

[0157] Normal distribution consistency check: Calculate the normal distribution of the input data X (data generated by the pseudo-modal generator) and calculate its consistency with the local distribution. Determine the normal value interval [a, b] according to the following formula 6:

[0158] P(a ≤ X ≤ b) = 0.9 (Formula 6)

[0159] Formula 6 can be used to calculate the boundaries a and b of the normal value range for this client. If X b, then X is determined to be an outlier, is discarded, and will not participate in federated learning updates.

[0160] Taking the federated learning scenario where the above-mentioned telecommunications operator uses multimodal data of fault reporting text and base station fault images to achieve fault classification as an example, each mobile client can calculate the normal distribution model corresponding to its fault reporting text data; similarly, each client can establish a normal distribution model for its base station fault image data, and eliminate abnormal samples outside the distribution interval according to the above formula 6 to obtain the target pseudomodal data.

[0161] The first probability that the target pseudomodal data is pseudomodal data is obtained using the discriminator parameters, and the first discrimination result is obtained based on the first probability;

[0162] The second probability that the first training data features are training data is obtained using the discriminator parameters, and the second discrimination result is obtained based on the second probability.

[0163] Specifically, the first probability that the target pseudo-modal data is pseudo-modal data is output by the pseudo-modal discriminator is calculated as shown in Formula 7:

[0164] Formula 7

[0165] in, It is a linear transformation matrix. The activation function is (e.g., ReLU). Taking the federated learning scenario where a telecommunications operator uses fault report text modal data and base station fault image multimodal data to achieve fault classification as an example, the client input is a pseudo-modal discriminator. The input is a base station fault image. The probability is output through a 2-layer network: ,in , .

[0166] Specifically, the second probability of the first training data feature being output by the pseudo-modal discriminator is calculated as shown in Formula 7 above, and will not be repeated here.

[0167] It should be noted that in a multimodal federated learning scenario, clients may store data of different modalities, such as text and images. For example, client A may only have text samples, client B may only have image data samples, and client C may have both text and image data samples. This embodiment trains a multimodal data alignment model for clients with complete modalities (such as client C, which has both text and images). The data alignment model serves two purposes: first, it improves the accuracy of federated learning; second, it provides correctness verification for subsequent pseudomodal data generated by the centralized server.

[0168] This embodiment provides a training method for the CLIP model. Through hierarchical parameter adjustment and domain adaptation module design, it achieves efficient model adaptation in specific downstream tasks, improving the cross-modal matching performance of text and images. An example is a scenario where a client simultaneously possesses full multimodal data of both telecommunications operator fault report texts and base station fault images, and the CLIP model is fine-tuned to align the report texts and base station fault images.

[0169] In some embodiments of this application, the method further includes:

[0170] Obtain the training data of the first modality of the fourth client and the training data of the second modality of the fourth client;

[0171] Feature extraction is performed on the training data of the first modality to obtain the third training data feature, and feature extraction is performed on the training data of the second modality to obtain the fourth training data feature.

[0172] It should be noted that in this embodiment, the fourth client and the third client mentioned above can be the same client. For the specific acquisition process of the third training data feature and the fourth training data feature, please refer to Formula 1 and Formula 2 above, which will not be repeated here.

[0173] Using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality, the model parameters of the initial model for the scene domain used to predict the data are trained to obtain the data alignment model.

[0174] Understandably, the initial model was the CLIP model.

[0175] Optionally, the model parameters of the initial model include model parameters for a first layer for semantic abstraction of the data and model parameters for a second layer for aligning the data with the scene domain.

[0176] Specifically, the initial model includes an image encoder (i.e., the ViT encoder) and a text encoder (Transformer encoder). The initial model is divided into layers: bottom layer (responsible for general feature extraction), middle layer (for semantic abstraction), and top layer (for modality alignment). The first layer is the middle layer, and the second layer is the top layer.

[0177] Freeze the bottom layer during training. to Adjust the model parameters only for the middle layer. to The model parameters and the top-level model parameters are used. For example, in the above example of fault identification for a telecommunications operator, the parameters of the first 8 layers of the image encoder ViT and the first 6 layers of the text encoder Transformer are frozen, and only the parameters of the remaining layers and the projection layer are fine-tuned.

[0178] The step of training the model parameters of the initial model for the scene domain used to predict the data using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality to obtain the data alignment model includes:

[0179] The model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the third training data to obtain the first training prediction scenario domain; and the model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the fourth training data to obtain the second training prediction scenario domain.

[0180] The data alignment model is obtained by training the model parameters of the first partial layer and the second partial layer based on the loss function between the first training prediction scene domain and the scene domain corresponding to the training data of the first modality, and the loss function between the second training prediction scene domain and the scene domain corresponding to the training data of the second modality.

[0181] Specifically, a bidirectional cross-attention layer is added to the second layer (i.e., the top layer) to enhance the interaction between image and text features. The bidirectional cross-attention layer for images is shown in Equation 8, and the bidirectional cross-attention layer for text is shown in Equation 9.

[0182] Formula 8

[0183] Formula 9

[0184] in, Represents the image features after domain adaptation. This represents the image features output by the bidirectional cross-attention layer. Represents the text features after domain adaptation. This represents the text features output by the bidirectional cross-attention layer.

[0185] Taking the federated learning scenario where a telecommunications operator uses multimodal data of fault report text and base station fault images to achieve fault classification as an example, the number of cross-attention layer heads is 12, and the hidden dimension is 768. Weights are calculated through the aforementioned cross-attention mechanism to enhance the feature association between the fault report text and the base station fault image, thereby improving the accuracy of multimodal data alignment.

[0186] Loss function design:

[0187] Using the CLIP contrastive loss function, with a batch size of 100 and a temperature parameter of 0.08, the above-mentioned unfrozen parameters and neighborhood enhancement parameters are trained to obtain the trained CLIP model, which is the data alignment model.

[0188] In some embodiments of this application, the step of aligning and concatenating the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and uploading the federated learning data to a centralized server for federated learning, includes:

[0189] Gradient information is obtained by aligning and concatenating the learning data of the first modality and the target learning data of the second modality, and then the gradient information is uploaded to the centralized server for federated learning.

[0190] Specifically, clients with full-modal data align and concatenate text and images to obtain intermediate gradients, which are then uploaded to the central server using homomorphic encryption. Clients with only single-modal data use a pseudo-modal generator to generate the missing modal data, perform local alignment, and then encrypt and upload the intermediate gradients.

[0191] The centralized server is used to obtain the sample size, modality number, and alignment loss reduction information of the federated learning data based on the gradient information, and to perform federated learning based on the sample size, modality number, and alignment loss reduction information of the federated learning data.

[0192] A centralized server aggregates the gradient information uploaded by each client. This embodiment comprehensively considers the amount of client sample data, modality scarcity, and the reduction ratio of alignment loss, and uses a federated learning gradient aggregation method based on an aggregation factor to improve the accuracy of the federated learning model. The specific calculation method is shown in Formula 10:

[0193] Formula 10

[0194] in, Indicates the first The normalized value of the client data sample size. Let the client sample size be... If the minimum sample size among all clients is minnum and the maximum sample size is maxnum, then the normalized value is calculated using Formula 11:

[0195] Formula 11

[0196] Modal scarcity The definition is as follows: Let the total number of modes be... (In this embodiment, it is 2, including text and images). If the client has all class modalities, then =1; if only one type of modality is possessed, then =0.5.

[0197] Alignment loss reduction ratio For the full-modal client, the value is 1; for the partial-modal client, the average similarity between the data generated by the pseudo-modal generator and the real aligned sample is calculated, which is the average similarity between the target learning data of the second modality and the learning data of the second modality.

[0198] When participating in federated learning gradient aggregation, the weight of each client's intermediate gradient is determined by the client's aggregation factor, which is the proportion of the client's aggregation factor to the sum of all client aggregation factors.

[0199] The following is combined with Figure 2 The federated learning method provided in the embodiments of this application is described in detail below:

[0200] The data alignment model is trained; a pseudo-modal generator is trained to generate pseudo-modal data to fill in the missing modalities; the pseudo-modal generator is used to achieve cross-client modal data alignment; multi-modal data fusion (alignment and stitching) is performed; and the multi-modal fused data is uploaded to a centralized server to participate in federated learning.

[0201] The federated learning method provided in this application can effectively address modality loss and achieve efficient cross-modal alignment. On the one hand, it completes the supplementation of missing modalities and multimodal data alignment; on the other hand, it improves the accuracy of federated learning through a multi-dimensional weighted gradient fusion mechanism.

[0202] This application firstly enhances the model's adaptability to specific scenario tasks and improves cross-modal matching performance of text and images by adjusting hierarchical parameters and designing a domain adaptation module, based on client samples with complete modal data. Secondly, a missing modality generation model is trained on a centralized server to generate pseudo-modal data to bridge missing modalities, achieving the supplementation and alignment of missing modalities. Finally, during the aggregation process, the centralized server comprehensively considers the amount of client sample data, modality scarcity, and the reduction ratio of alignment loss, and determines the fusion weights of intermediate gradients in a weighted manner. This method achieves both the supplementation of missing modalities and accurate alignment of multimodal data, and improves the accuracy of federated learning through a multi-factor weighted gradient fusion mechanism.

[0203] This application first formats the image data by size and cleans the text by word segmentation and stop word removal. A domain-adaptive embedding layer is then introduced to correct domain offsets between image and text features, improving the model's adaptability in different business scenarios. Second, the CLIP image encoder and text encoder are hierarchically divided into a bottom layer (general feature extraction), a middle layer (semantic abstraction), and a top layer (modal alignment). The bottom layer parameters are frozen, and only the middle and top layer parameters are fine-tuned. Third, a bidirectional cross-attention layer is introduced at the top layer to enhance the interaction between image and text features. Finally, the unfrozen parameters and the domain enhancement parameters are fine-tuned by comparing the loss function. This method can significantly improve the alignment accuracy of multimodal data in different business scenarios.

[0204] During the training of the pseudomodal generator and pseudomodal discriminator, a dynamic mapping matrix is ​​introduced to address the dimensional differences of missing modalities across different clients. A dynamic dimensional adaptation layer is constructed to automatically adjust the generator's output dimension, ensuring that the generated data matches the target modal dimension, thereby resolving the modal heterogeneity problem. When evaluating the generation quality, based on the normal distribution characteristics (mean, variance) of the complete modal client data, outlier data with long tails on both sides are identified and removed, improving the accuracy of pseudomodal data generation.

[0205] This application trains a missing modality data generation model on a centralized server to generate high-quality pseudo-modal data to bridge the missing modalities, thereby compensating for the incomplete modality problem in client-side data. The method introduces a dynamic mapping matrix to construct a dynamic dimension adaptation layer, automatically adjusting the generator's output dimension to ensure that the generated pseudo-modal data matches the target modality dimension. This solves the modality heterogeneity problem and significantly improves the feasibility and flexibility of multimodal federated learning in practical applications.

[0206] Improving Cross-Client Alignment Accuracy: To address the failure of cross-modal contrastive learning caused by the prohibition of detailed data sharing in federated learning, this application proposes a multimodal data alignment fine-tuning method. By standardizing and cleaning image and text data, introducing a domain-adaptive embedding layer, and combining it with the CLIP model's hierarchical structure (bottom-layer freezing, middle-layer fine-tuning, and top-layer enhanced interaction), a bidirectional cross-attention mechanism is introduced at the top layer, significantly enhancing the interaction capability between image and text features, thereby effectively improving the multimodal data alignment accuracy in different business scenarios.

[0207] Optimizing gradient aggregation strategies to improve the modeling accuracy of federated learning: In existing technologies, gradient aggregation weights are usually determined only based on the amount of client data, resulting in low modeling accuracy. This application innovatively proposes an aggregation weight calculation method that integrates sample data volume, modality scarcity, and alignment loss reduction ratio, making the aggregation of gradients among clients more reasonable and effectively improving the overall modeling accuracy and robustness of federated learning. It is especially suitable for practical application scenarios with uneven data distribution and severe modality loss.

[0208] Enhancing the model's domain adaptability: By introducing a domain-adaptive embedding layer, the feature vectors of images and text are corrected for domain offset, enabling the model to better adapt to the differences in data distribution under different business scenarios, thereby improving the model's generalization ability and practical application effect.

[0209] Improving the quality of pseudomodal data generation: In the process of pseudomodal data generation, this application uses the normal distribution characteristics to filter out abnormal data and remove low-quality data distributed in the long tail region, thereby significantly improving the accuracy and reliability of pseudomodal data and providing higher quality data support for subsequent cross-modal alignment.

[0210] like Figure 3 As shown in the embodiments of this application, a federated learning device is also provided, the device comprising:

[0211] The first acquisition module 301 is used to acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults;

[0212] The first processing module 302 is used to input the learning data of the first modality into the pseudo-modal data generation model to obtain the pseudo-modal learning data of the second modality output by the pseudo-modal data generation model.

[0213] The second processing module 303 is used to obtain target learning data of the second modality that is aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality.

[0214] The third processing module 304 is used to align and splice the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and upload the federated learning data to the central server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data of the communication operator.

[0215] Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

[0216] Optionally, the second processing module 303 includes:

[0217] The first processing unit is used to select the first learning data with the highest similarity to the learning data of the second modality from the pseudo-modal learning data of the second modality.

[0218] The second processing unit is used to input the first learning data into the data alignment model to obtain the predicted scene domain of the first learning data predicted by the data alignment model.

[0219] The third processing unit is used to determine the domain alignment accuracy between the learning data of the first modality based on the predicted scene domain and the real scene domain corresponding to the learning data of the first modality.

[0220] The fourth processing unit is configured to, when the first condition is met, use the first learning data as the target learning data of the second modality, and when the first condition is not met, obtain the target learning data of the second modality based on the first learning data and the pseudo-modal learning data of the second modality.

[0221] The first condition includes:

[0222] The alignment accuracy of the domain corresponding to the first learning data exceeding a preset proportion is greater than a preset accuracy threshold.

[0223] Optionally, the pseudo-modal data generation model includes a pseudo-modal generator and a pseudo-modal discriminator;

[0224] The device further includes:

[0225] The second acquisition module is used to acquire the training data of the first mode of the third client and the training data of the second mode of the third client; wherein the training data is repair data in the field of communication operator faults;

[0226] The fourth processing module is used to extract features from the training data of the first modality to obtain first training data features, and to extract features from the training data of the second modality to obtain second training data features.

[0227] The fifth processing module is used to perform N training iterations based on the features of the first training data and the features of the second training data to obtain the pseudomodal data generation model;

[0228] The nth training iteration includes:

[0229] The first input data is obtained by concatenating the features of the first training data and the first noise information;

[0230] The first input data and the features of the second training data are input into the pseudo-modality generator, and the generator parameters of the pseudo-modality generator are used to generate the first pseudo-modality data based on the first input data and the features of the second training data.

[0231] The first pseudomodal data and the first training data features are input into the pseudomodal discriminator. The discriminator parameters of the pseudomodal discriminator are used to obtain a first discrimination result of the first pseudomodal data and a second discrimination result of the first training data features. The discriminator parameters are updated according to the first discrimination result and the second discrimination result to obtain the updated discriminator parameters. The first discrimination result is used to indicate the recognition result between the first pseudomodal data and the first label, and the second discrimination result is used to indicate the recognition result between the first training data features and the second label. The first label is the label of the pseudomodal data, and the second label is the label of the training data.

[0232] The generator parameters are updated based on the first discrimination result to obtain the updated generator parameters;

[0233] Where N is a positive integer, and n is an integer greater than 0 and less than N;

[0234] When n equals 1, the generator parameters of the pseudo-modal generator are the first preset parameters, and the discriminator parameters of the pseudo-modal discriminator are the second preset parameters; when n is greater than 1, the generator parameters of the pseudo-modal generator are the updated generator parameters corresponding to the (n-1)th training, and the discriminator parameters of the pseudo-modal discriminator are the updated discriminator parameters corresponding to the (n-1)th training.

[0235] Optionally, the fifth processing module includes:

[0236] The fifth processing unit is used to generate initial pseudo-modal data based on the first input data using the pseudo-modal generator.

[0237] The sixth processing unit is used to determine the dimension parameters of the first pseudo-modal data in the generator parameters based on the dimension parameters of the features of the second training data.

[0238] The seventh processing unit is used to generate a mapping matrix based on the dimension parameters of the first pseudomodal data and the output dimension parameters in the generator parameters;

[0239] The eighth processing unit is used to generate the first pseudomodal data based on the mapping matrix and the initial pseudomodal data.

[0240] Optionally, the fifth processing module includes:

[0241] The first acquisition unit is used to acquire the normal distribution information of the features of the first training data;

[0242] The ninth processing unit is used to determine the target pseudo-modal data in the first pseudo-modal data based on the normal distribution information;

[0243] The tenth processing unit is used to obtain a first probability that the target pseudomodal data is pseudomodal data using the discriminator parameters, and to obtain the first discrimination result based on the first probability;

[0244] The eleventh processing unit is used to obtain a second probability that the features of the first training data are training data using the discriminator parameters, and to obtain the second discrimination result based on the second probability.

[0245] Optionally, the device further includes:

[0246] The third acquisition module is used to acquire the training data of the first modality of the fourth client and the training data of the second modality of the fourth client;

[0247] The sixth processing module is used to extract features from the training data of the first modality to obtain the third training data features, and to extract features from the training data of the second modality to obtain the fourth training data features.

[0248] The seventh processing module is used to train the model parameters of the initial model for the scene domain used to predict the data by utilizing the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality, so as to obtain the data alignment model.

[0249] Optionally, the model parameters of the initial model include model parameters of a first layer for semantic abstraction of the data and model parameters of a second layer for aligning the data with the scene domain;

[0250] The seventh processing module includes:

[0251] The twelfth processing unit is used to predict the third training data features using the model parameters of the first part layer and the model parameters of the second part layer to obtain a first training prediction scenario domain, and to predict the fourth training data features using the model parameters of the first part layer and the model parameters of the second part layer to obtain a second training prediction scenario domain.

[0252] The thirteenth processing unit is used to train the model parameters of the first partial layer and the model parameters of the second partial layer according to the loss function between the first training prediction scene domain and the scene domain corresponding to the training data of the first modality, and the loss function between the second training prediction scene domain and the scene domain corresponding to the training data of the second modality, to obtain the data alignment model.

[0253] Optionally, the third processing module 304 includes:

[0254] The fourteenth processing unit is used to align and concatenate the learning data of the first modality and the target learning data of the second modality to obtain gradient information, and upload the gradient information to the central server for federated learning.

[0255] The centralized server is used to obtain the sample size, modality number, and alignment loss reduction information of the federated learning data based on the gradient information, and to perform federated learning based on the sample size, modality number, and alignment loss reduction information of the federated learning data.

[0256] It should be noted that the federated learning device provided in this application embodiment is a device capable of executing the above-described federated learning method. Therefore, all embodiments of the above-described federated learning method are applicable to this device and can achieve the same or similar technical effects.

[0257] like Figure 4 As shown in the figure, this application embodiment also provides a federated learning device, including: a processor 401; and a memory 403 connected to the processor 401 via a bus interface 402, the memory 403 being used to store programs and data used by the processor 401 when performing operations, and the processor 401 calling and executing the programs and data stored in the memory 403.

[0258] The transceiver 404 is connected to the bus interface 402 and is used to receive and send data under the control of the processor 401. Specifically, the processor 401 is used to read the program in the memory 403 and to execute the following processes:

[0259] Acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults;

[0260] The learning data of the first modality is input into the pseudomodal data generation model to obtain the pseudomodal learning data of the second modality output by the pseudomodal data generation model.

[0261] Based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality, target learning data of the second modality that is aligned with the learning data of the first modality is obtained.

[0262] The learning data of the first modality and the target learning data of the second modality are aligned and concatenated to obtain federated learning data. The federated learning data is uploaded to a centralized server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data of the communication operator.

[0263] Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

[0264] Optionally, the processor 401 is used to:

[0265] The first learning data with the highest similarity to the learning data of the second modality is selected from the pseudo-modal learning data of the second modality.

[0266] The first learning data is input into the data alignment model to obtain the predicted scene domain of the first learning data as predicted by the data alignment model.

[0267] Based on the predicted scene domain and the real scene domain corresponding to the learning data of the first modality, the domain alignment accuracy between the learning data of the first modality is determined;

[0268] If the first condition is met, the first learning data is used as the target learning data for the second modality; if the first condition is not met, the target learning data for the second modality is obtained based on the first learning data and the pseudo-modal learning data of the second modality.

[0269] The first condition includes:

[0270] The alignment accuracy of the domain corresponding to the first learning data exceeding a preset proportion is greater than a preset accuracy threshold.

[0271] Optionally, the pseudo-modal data generation model includes a pseudo-modal generator and a pseudo-modal discriminator;

[0272] The processor 401 is further configured to:

[0273] Acquire training data of the first modality of the third client and training data of the second modality of the third client; wherein, the training data is repair data in the field of communication operator faults;

[0274] Feature extraction is performed on the training data of the first modality to obtain the first training data features, and feature extraction is performed on the training data of the second modality to obtain the second training data features;

[0275] Based on the first training data features and the second training data features, N training iterations are performed to obtain the pseudomodal data generation model;

[0276] The nth training iteration includes:

[0277] The first input data is obtained by concatenating the features of the first training data and the first noise information;

[0278] The first input data and the features of the second training data are input into the pseudo-modality generator, and the generator parameters of the pseudo-modality generator are used to generate the first pseudo-modality data based on the first input data and the features of the second training data.

[0279] The first pseudomodal data and the first training data features are input into the pseudomodal discriminator. The discriminator parameters of the pseudomodal discriminator are used to obtain a first discrimination result of the first pseudomodal data and a second discrimination result of the first training data features. The discriminator parameters are updated according to the first discrimination result and the second discrimination result to obtain the updated discriminator parameters. The first discrimination result is used to indicate the recognition result between the first pseudomodal data and the first label, and the second discrimination result is used to indicate the recognition result between the first training data features and the second label. The first label is the label of the pseudomodal data, and the second label is the label of the training data.

[0280] The generator parameters are updated based on the first discrimination result to obtain the updated generator parameters;

[0281] Where N is a positive integer, and n is an integer greater than 0 and less than N;

[0282] When n equals 1, the generator parameters of the pseudo-modal generator are the first preset parameters, and the discriminator parameters of the pseudo-modal discriminator are the second preset parameters; when n is greater than 1, the generator parameters of the pseudo-modal generator are the updated generator parameters corresponding to the (n-1)th training, and the discriminator parameters of the pseudo-modal discriminator are the updated discriminator parameters corresponding to the (n-1)th training.

[0283] Optionally, the processor 401 is specifically used for:

[0284] The pseudo-modal generator is used to generate initial pseudo-modal data based on the first input data;

[0285] The dimension parameters of the first pseudomodal data in the generator parameters are determined based on the dimension parameters of the second training data features.

[0286] A mapping matrix is ​​generated based on the dimension parameters of the first pseudomodal data and the output dimension parameters in the generator parameters;

[0287] The first pseudomodal data is generated based on the mapping matrix and the initial pseudomodal data.

[0288] Optionally, the processor 401 is specifically used for:

[0289] Obtain the normal distribution information of the features of the first training data;

[0290] The target pseudo-modal data in the first pseudo-modal data is determined based on the normal distribution information;

[0291] The first probability that the target pseudomodal data is pseudomodal data is obtained using the discriminator parameters, and the first discrimination result is obtained based on the first probability;

[0292] The second probability that the first training data features are training data is obtained using the discriminator parameters, and the second discrimination result is obtained based on the second probability.

[0293] Optionally, the processor 401 is further configured to:

[0294] Obtain the training data of the first modality of the fourth client and the training data of the second modality of the fourth client;

[0295] Feature extraction is performed on the training data of the first modality to obtain the third training data feature, and feature extraction is performed on the training data of the second modality to obtain the fourth training data feature;

[0296] Using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality, the model parameters of the initial model for the scene domain used to predict the data are trained to obtain the data alignment model.

[0297] Optionally, the model parameters of the initial model include model parameters of a first layer for semantic abstraction of the data and model parameters of a second layer for aligning the data with the scene domain;

[0298] The processor 401 is specifically used for:

[0299] The model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the third training data to obtain the first training prediction scenario domain; and the model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the fourth training data to obtain the second training prediction scenario domain.

[0300] The data alignment model is obtained by training the model parameters of the first partial layer and the second partial layer based on the loss function between the first training prediction scene domain and the scene domain corresponding to the training data of the first modality, and the loss function between the second training prediction scene domain and the scene domain corresponding to the training data of the second modality.

[0301] Optionally, the processor 401 is used to:

[0302] Gradient information is obtained by aligning and concatenating the learning data of the first modality and the target learning data of the second modality, and the gradient information is uploaded to the centralized server for federated learning.

[0303] The centralized server is used to obtain the sample size, modality number, and alignment loss reduction information of the federated learning data based on the gradient information, and to perform federated learning based on the sample size, modality number, and alignment loss reduction information of the federated learning data.

[0304] Among them, Figure 4In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 401) and memory (memory 403). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides a user interface 405. A transceiver 404 may be multiple elements, including transmitters and receivers, providing units for communicating with various other devices over a transmission medium. Processor 401 is responsible for managing the bus architecture and general processing, and memory 403 may store data used by processor 401 during operation.

[0305] In addition, specific embodiments of this application also provide a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps in the federated learning method as described above.

[0306] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0307] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0308] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute partial steps of the resource selection method described in the various embodiments of this application, or to execute partial steps of the information transmission method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] A specific embodiment of this application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above-described functionality. Figure 1 The various processes of the method embodiments shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0310] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A federated learning method, characterized in that, The method includes: Acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults; The learning data of the first modality is input into the pseudomodal data generation model to obtain the pseudomodal learning data of the second modality output by the pseudomodal data generation model. Based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality, target learning data of the second modality that is aligned with the learning data of the first modality is obtained. The learning data of the first modality and the target learning data of the second modality are aligned and concatenated to obtain federated learning data. The federated learning data is uploaded to a centralized server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data of the communication operator. Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

2. The method according to claim 1, characterized in that, The step of obtaining target learning data for the second modality aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality includes: The first learning data with the highest similarity to the learning data of the second modality is selected from the pseudo-modal learning data of the second modality. The first learning data is input into the data alignment model to obtain the predicted scene domain of the first learning data as predicted by the data alignment model. Based on the predicted scene domain and the real scene domain corresponding to the learning data of the first modality, the domain alignment accuracy between the learning data of the first modality is determined; If the first condition is met, the first learning data is used as the target learning data for the second modality; if the first condition is not met, the target learning data for the second modality is obtained based on the first learning data and the pseudo-modal learning data of the second modality. The first condition includes: The alignment accuracy of the domain corresponding to the first learning data exceeding a preset proportion is greater than a preset accuracy threshold.

3. The method according to claim 1, characterized in that, The pseudomodal data generation model includes a pseudomodal generator and a pseudomodal discriminator; The method further includes: Acquire training data of the first modality of the third client and training data of the second modality of the third client; wherein, the training data is repair data in the field of communication operator faults; Feature extraction is performed on the training data of the first modality to obtain the first training data features, and feature extraction is performed on the training data of the second modality to obtain the second training data features; Based on the first training data features and the second training data features, N training iterations are performed to obtain the pseudomodal data generation model; The nth training iteration includes: The first input data is obtained by concatenating the features of the first training data and the first noise information; The first input data and the features of the second training data are input into the pseudo-modality generator, and the generator parameters of the pseudo-modality generator are used to generate the first pseudo-modality data based on the first input data and the features of the second training data. The first pseudomodal data and the first training data features are input into the pseudomodal discriminator. The discriminator parameters of the pseudomodal discriminator are used to obtain a first discrimination result of the first pseudomodal data and a second discrimination result of the first training data features. The discriminator parameters are updated according to the first discrimination result and the second discrimination result to obtain the updated discriminator parameters. The first discrimination result is used to indicate the recognition result between the first pseudomodal data and the first label, and the second discrimination result is used to indicate the recognition result between the first training data features and the second label. The first label is the label of the pseudomodal data, and the second label is the label of the training data. The generator parameters are updated based on the first discrimination result to obtain the updated generator parameters; Where N is a positive integer, and n is an integer greater than 0 and less than N; When n equals 1, the generator parameters of the pseudo-modal generator are the first preset parameters, and the discriminator parameters of the pseudo-modal discriminator are the second preset parameters; when n is greater than 1, the generator parameters of the pseudo-modal generator are the updated generator parameters corresponding to the (n-1)th training, and the discriminator parameters of the pseudo-modal discriminator are the updated discriminator parameters corresponding to the (n-1)th training.

4. The method according to claim 3, characterized in that, The step of generating first pseudo-modal data based on the generator parameters of the pseudo-modal generator and the features of the first input data and the second training data includes: The pseudo-modal generator is used to generate initial pseudo-modal data based on the first input data; The dimension parameters of the first pseudomodal data in the generator parameters are determined based on the dimension parameters of the second training data features. A mapping matrix is ​​generated based on the dimension parameters of the first pseudomodal data and the output dimension parameters in the generator parameters; The first pseudomodal data is generated based on the mapping matrix and the initial pseudomodal data.

5. The method according to claim 3, characterized in that, Obtaining a first discrimination result of the first pseudomodal data and a second discrimination result of the first training data features using the discriminator parameters of the pseudomodal discriminator includes: Obtain the normal distribution information of the features of the first training data; The target pseudo-modal data in the first pseudo-modal data is determined based on the normal distribution information; The first probability that the target pseudomodal data is pseudomodal data is obtained using the discriminator parameters, and the first discrimination result is obtained based on the first probability; The second probability that the first training data features are training data is obtained using the discriminator parameters, and the second discrimination result is obtained based on the second probability.

6. The method according to claim 2, characterized in that, The method further includes: Obtain the training data of the first modality of the fourth client and the training data of the second modality of the fourth client; Feature extraction is performed on the training data of the first modality to obtain the third training data feature, and feature extraction is performed on the training data of the second modality to obtain the fourth training data feature; Using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality, the model parameters of the initial model for the scene domain used to predict the data are trained to obtain the data alignment model.

7. The method according to claim 6, characterized in that, The model parameters of the initial model include model parameters for a first layer used for semantic abstraction of the data, and model parameters for a second layer used for alignment of the data and the scene domain; The step of training the model parameters of the initial model for the scene domain used to predict the data using the third training data features, the scene domain corresponding to the training data of the first modality, the fourth training data features, and the scene domain corresponding to the training data of the second modality to obtain the data alignment model includes: The model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the third training data to obtain the first training prediction scenario domain; and the model parameters of the first part layer and the model parameters of the second part layer are used to predict the features of the fourth training data to obtain the second training prediction scenario domain. The data alignment model is obtained by training the model parameters of the first partial layer and the second partial layer based on the loss function between the first training prediction scene domain and the scene domain corresponding to the training data of the first modality, and the loss function between the second training prediction scene domain and the scene domain corresponding to the training data of the second modality.

8. The method according to claim 1, characterized in that, The step of aligning and concatenating the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and uploading the federated learning data to a centralized server for federated learning, includes: Gradient information is obtained by aligning and concatenating the learning data of the first modality and the target learning data of the second modality, and the gradient information is uploaded to the centralized server for federated learning. The centralized server is used to obtain the sample size, modality number, and alignment loss reduction information of the federated learning data based on the gradient information, and to perform federated learning based on the sample size, modality number, and alignment loss reduction information of the federated learning data.

9. A federated learning device, characterized in that, The device includes: The first acquisition module is used to acquire learning data of the first modality of the first client and learning data of the second modality of the second client, wherein the learning data is repair data in the field of communication operator faults; The first processing module is used to input the learning data of the first modality into the pseudo-modal data generation model to obtain the pseudo-modal learning data of the second modality output by the pseudo-modal data generation model. The second processing module is used to obtain target learning data of the second modality that is aligned with the learning data of the first modality based on the similarity between the pseudo-modal learning data of the second modality and the learning data of the second modality. The third processing module is used to align and stitch together the learning data of the first modality and the target learning data of the second modality to obtain federated learning data, and upload the federated learning data to the central server for federated learning to obtain a communication operator fault model. The communication operator fault model is used to predict the fault classification information of the communication operator based on the repair data reported by the communication operator. Wherein, the first modality is a text modality or an image modality; when the first modality is a text modality, the second modality is an image modality; when the first modality is an image modality, the second modality is a text modality.

10. A federated learning device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the federated learning method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the steps of the federated learning method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps in the federated learning method as described in any one of claims 1 to 8.

Images