Federal multi-modal preference learning and alignment method for mode-lacking condition

Abstract

The invention discloses a federal multi-modal preference learning and alignment method for a mode lacking condition, and aims to solve the problem of unstable alignment caused by image-text data mode lacking, preference noise and cross-domain conflict in multi-client privacy protection. According to the method, a modal state is identified and a conditional preference pair is constructed at a client: cross-modal consistency scoring is carried out to screen good and bad responses when images and texts are complete, a visual attribute lexicon is introduced to punishment suppression assume when the texts are lack of texts, and entity existence verification punishment illusion is carried out when the images are lack of texts; noise updating is reduced in combination with sample-level and client-level two-level gating based on prediction uncertainty; and the server side forms multi-preference experts by utilizing low-dimensional preference signature clustering and issues and fuses the multi-preference experts according to similarity routing. According to the method, the consistency and robustness of multi-modal generation are improved in a modal-lacking scene, and negative migration and illusion are reduced.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and distributed learning, and to a training control method for exchanging parameters / statistics between a client and a server, specifically a federated multimodal preference learning and alignment method with missing modality conditions. Background Technology

[0002] With the advancements of multimodal large language models (MLLMs) and visual language models (VLMs) in tasks such as visual question answering, image description, and cross-modal retrieval, multimodal generation systems are gradually entering application scenarios with high reliability requirements, such as healthcare, finance, government affairs, and industry. However, existing multimodal generation and alignment technologies typically face two types of engineering challenges in real-world deployments: (1) missing / incomplete modal inputs (text only or images only, etc.); and (2) generation illusions (outputs that are inconsistent with available modal evidence or lack factual support). In the context of federated learning (FL) under privacy and data compliance constraints, these problems are further compounded, making it difficult for existing technologies to stably achieve multi-agency collaborative alignment. Existing technology 1: Multimodal alignment methods based on preference optimization and their shortcomings.

[0003] To alleviate the illusion problem of multimodal models, some studies have attempted to introduce preference optimization paradigms (such as DPO-type methods) into the multimodal alignment process.

[0004] For example, Fu et al. proposed the preference optimization approach for multimodal illusions (HDPO), pointing out that directly applying DPO to illusion relief in multimodal models has problems such as unstable improvement and inconsistent returns, and attempted to improve it by focusing more on preference optimization for illusions.

[0005] For example, Wu et al. proposed Multimodal Preference Optimization (EMPO) for Entity Centers, pointing out that existing preference alignment methods often focus on "catering to preferences" but ignore explicit alignment of text and image modalities, leading to over-reliance on language priors and inducing illusions.

[0006] Although the above methods offer improvements at the levels of "construction of multimodal preference data" and "design of preference objectives," at least three key shortcomings still exist from the perspective of engineering deployment and federated scenarios:

[0007] 1. The implicit assumption of modal completeness leads to insufficient evidence of preference in scenarios with missing modalities: The above-mentioned preference construction and the default input of the preference target contain sufficient multimodal evidence for discrimination (such as the simultaneous existence of text and images or complete instructions). When the actual input has a missing modality such as "text only" or "image only", the criteria for selecting better and worse responses from the candidate set will be weakened or even invalid, thereby introducing preference label noise and causing the objective function to be biased in the direction of updating model parameters.

[0008] 2. Lack of explicit modeling and training gating mechanisms for uncertainties in missing modalities: When modal information is insufficient, multimodal models often exhibit higher uncertainty. If parameters are still updated with uniform intensity, unreliable samples are easily introduced into the training and amplify preference noise. Existing work focuses more on constructing or aligning preference pairs, and lacks adaptive gating update mechanisms driven by uncertainties in missing modal samples.

[0009] 3. The discussion mainly focuses on centralized training or single-domain settings: Related methods typically rely on centralized or uniform data distribution for preference alignment. However, in federated environments with isolated data from multiple institutions, heterogeneous distributions, and inconsistent modal combinations, the preference construction and optimization processes are difficult to directly apply or achieve stable convergence. [3] Existing technology two: Federated multimodal learning framework and its shortcomings.

[0010] To meet privacy protection requirements, federated learning achieves multi-party collaboration through a paradigm of "client-side local training – server-aggregated updates". For distributed training of multimodal data, existing federated multimodal learning work has proposed personalized aggregation and cross-modal alignment strategies.

[0011] For example, Zhang et al. proposed FedEPA, which employs personalized aggregation weights and unsupervised modality alignment strategies in multimodal federated learning, emphasizing performance improvement under multimodal classification tasks and limited labeling conditions. Furthermore, Xiong et al. proposed the Federated Multimodal Instruction Fine-tuning Task (FedMIT) and the framework Pilot for collaborative fine-tuning of multimodal instruction models in a distributed environment.

[0012] The aforementioned federated multimodal learning framework provides fundamental support for "multi-party collaborative training," "modal alignment," and "personalized aggregation," but it still has significant gaps in the more challenging scenario of "multimodal generation preference alignment and illusion mitigation."

[0013] 1. The task objectives are biased towards representation learning / classification, and do not address the closed-loop design for generating illusions and aligning preferences: For example, FedEPA mainly focuses on feature alignment and improving classification performance, and lacks preference pair construction, preference optimization objectives and illusion suppression mechanisms for generation tasks; therefore, it is difficult to directly solve the problem of inconsistency between generated output and available modal evidence.

[0014] 2. The superposition of missing modalities and heterogeneous preferences leads to aggregation noise problems that are not handled by the system: In federated scenarios, different clients may have differences in the proportion of missing modalities, data quality, and preferences. If fixed aggregation or weighting based solely on sample size is still used, the unreliable updates caused by missing modalities may easily spread to the global model during server aggregation updates, affecting stability.

[0015] 3. Lack of structured modeling and multi-expert / routing mechanism for “preference conflicts”: In multi-organizational scenarios, preferences (such as output detail, compliance terminology, diagnostic style, etc.) may vary significantly, and a single global model often results in a compromise. Existing frameworks aggregate more from the perspective of “data heterogeneity” and do not provide a general mechanism to cluster and decouple “preference heterogeneity” and distribute routes.

[0016] In summary, existing technologies have made progress in multimodal preference alignment and federated multimodal learning, but the following problems still need to be addressed in the combined scenario of "lack of modality + federation + generative preference alignment + illusion mitigation":

[0017] (1) Insufficient evidence for constructing preference pairs under the condition of lack of modality leads to preference label noise and alignment bias;

[0018] (2) When the uncertainty of missing modal samples increases, there is a lack of training gating and aggregation noise reduction mechanisms, which can easily lead to pollution of federated updates;

[0019] (3) The conflict caused by the heterogeneity of preferences of multiple institutions lacks a system solution for cluster decoupling, multi-expert maintenance and routing, making it difficult to balance personalization and global consistency. Summary of the Invention

[0020] The purpose of this invention is to address the issues of missing modality preference alignment and illusion in existing multimodal large-scale model applications, and to propose a federated multimodal preference learning and alignment method under missing modality conditions. This method improves the consistency and robustness of multimodal generation in missing modality scenarios, reduces negative transfer and illusion, reduces the interference of missing modality samples on the federated training process, improves the stability of global model updates, and enhances the scalability of preference modeling and aggregation processes while maintaining federated privacy constraints.

[0021] The technical solution to achieve the objective of this invention is:

[0022] A federated multimodal preference learning and alignment method with missing modality conditions, comprising several local clients and a server, the method comprising the following steps:

[0023] Step 1: Server-side initialization configuration distribution

[0024] Before federated training begins, the central server performs system initialization and distributes global configurations and public resources related to aligning training consistency to each client. The distributed content includes:

[0025] 1.1. Global Model and Reference Strategy Distribution: The server distributes a set of model parameters for federated alignment training to the client. The global model adopts a shared base + expert model structure, while the client's local model is composed of the distributed shared base and an updatable expert model. The expert model is an adjustable parameter module attached to the shared base model, wherein:

[0026] 1.1.1. Shared Base Distribution: The server distributes the shared base model. , Includes the visual encoder and language model backbone, which are shared by all clients throughout the federated training process. To reduce communication overhead, the server only sends out the command once during the initialization phase. The client loads the shared base accordingly; in subsequent federation rounds, θ remains frozen and does not participate in federation aggregation updates.

[0027] 1.1.2. Initial Multi-Expert Model Deployment: The server initializes a set of expert models based on the shared base model. The shared base model and the corresponding expert model are then sent to the client; whereby... One value can be chosen, i.e., single-expert initialization; when When the value is greater than 1, the server can use the same initialization for all experts, for example... and shared base model Together they form the initial multi-expert model set. ;

[0028] The above initialization method ensures that multiple global preference expert models are subsequently generated on the server side based on preference signature clustering, and that the Top-p experts are distributed to the client during the model distribution phase, so that the client can seamlessly connect to the mechanism of local routing fusion.

[0029] 1.1.3 Reference Strategy Composition and round update rules: Reference strategy Compared with the current strategy Using the same structure, but frozen during this round of local training, it is only used for optimizing the loss in conditional preference settings. The calculation; in the first round of federated training, the client uses the policy starting point model, which is composed of the shared base model parameters and the matching initial expert model parameter set issued by the server during the initialization phase, as the reference policy. In round t>1, the client combines the expert model parameters from the previous round sent by the server with the shared base model θ stored locally on the client and kept frozen to form the starting strategy model for this round of training, and then copies it to generate the reference strategy model. Before the start of this round of local training, the parameters of the reference policy model are frozen so that they do not participate in parameter updates during this round of training, but are only used to calculate reference terms in preference optimization;

[0030] 1.2. Distribution of Consistent Random Projection Matrix Configuration: To support the subsequent generation of low-dimensional preference signatures and privacy compression, the server generates a unified random seed during the system initialization phase. The random seed is then distributed to each client. The client generates a random matrix based on the random seed and obtains a fixed random orthogonal projection matrix R through QR decomposition. The random orthogonal projection matrix R maintains fixed matrix parameters in multiple federated training rounds to ensure that the preference signatures generated in different rounds are located in the same low-dimensional representation space, thereby supporting similarity calculation and clustering processing on the server side.

[0031] 1.3. Distribution of Common Rules and Lexicon Related to Modality: The server distributes unified rule configurations for modality missingness determination to the client, namely, mapping null values / damaged / low-quality thresholds to missing conditions, and a visual attribute lexicon for T-only scenarios. This ensures that the client follows a consistent constraint approach when performing modal mask determination and constructing conditional preference pairs;

[0032] 1.4. Training and Gating Hyperparameter Distribution: The server distributes key hyperparameters and threshold ranges to the client. These key hyperparameters and threshold ranges include: preference-optimized temperature coefficient. Sample-level gated temperature parameters Entity existence threshold, penalty coefficient The proportion of inferior sampling locations The range of values ​​for the number of candidate responses N is used to ensure that the behavior of local training and gating updates on each client is consistent and controllable.

[0033] Step 2: Multimodal input state recognition and modal mask generation:

[0034] Configure local multimodal samples on the client ,in, Represents image modal data, This represents text modal data, where the subscript i indicates the i-th sample data, corresponding to the i-th modal mask variable below;

[0035] The client scans the samples using a modal detection operator to generate modal mask variables. Used to identify the modal completeness state of the current sample: ;in This indicates that both image and text modalities exist; This indicates that only the text modality exists, and the image modality is missing; This indicates that only the image modality exists, while the text modality is missing. The modality detection operator can be implemented in any of the following ways or combinations and supports a unified mapping of "null / damaged / low-quality threshold" to missing values: based on whether the data structure field is empty; based on whether the modality encoder output is a zero vector or a null value; based on the modality quality threshold.

[0036] The client stores ( ) for each sample , Triples, and put As a condition variable for all subsequent operators;

[0037] Step 3: Construction of conditional preference pairs based on modality masks:

[0038] The client utilizes the current local model For the sample Generate a set of candidate responses:

[0039] in The number of candidate responses is selected, with a range of values. ;

[0040] And according to the mask Implement different automated screening strategies to construct the optimal response. and poor response Preference data pairs:

[0041] 3.1 Regarding IT data: When The system provides complete text and images instantly, and performs fine-grained cross-modal alignment: it utilizes the CLIP model to compute each answer. With images Visual-semantic consistency score: The candidate responses are sorted in descending order of score to obtain the sequence. The response with the highest score is selected as the preferred response. From the sorted position Take inferior response This forces local models to distinguish between high-quality descriptions and seemingly plausible illusions; among which ;

[0042] 3.2 For T-only data: When For text-only scenarios, apply visual attribute density penalty: load a pre-defined visual attribute lexicon. The lexicon contains visual descriptive terms that cannot be supported by text when visual modalities are lacking, such as: color, shape, size; spatial location, directional relationship; quantity, relative layout; domain-related visual entity descriptive terms; and for each candidate response Construct a set of visual penalty words:

[0043] ( )=

[0044] And calculate the scoring function: ( ) = - Select the response with the highest score as Low response as , obtain The specific methods and steps are described in section 3.1. The method is the same;

[0045] 3.3 For I-only data: When In the case of images only, perform entity existence closed-loop verification: similar to the IT scenario, but introduce an entity penalty term; for candidate responses Entity noun extraction is performed to obtain an entity set. For each entity Construct a prompt template and calculate the probability of the entity's presence in the image. : ,like Below the threshold If this occurs, a point deduction mechanism will be triggered, including an existence threshold. The final scoring function is defined as:

[0046] ( )=

[0047] in This is the penalty coefficient, with a value range of [0.1, 2.0]. This is an indicator function used to determine whether an object in the response is a hallucinatory object. When an object is determined to be a hallucinatory object, the indicator function takes a value of 1, triggering a corresponding penalty; when the object is determined to be a real object in the image, the indicator function takes a value of 0, and no penalty is triggered. This forces the local model to adopt a conservative WYSIWYG strategy when there are no user instructions, thereby suppressing hallucinations. The penalty mechanism is as follows: for candidate responses... Each entity extracted Calculate the entity in the image The probability of existence in ;like < If the entity is identified as a hallucinatory entity, a fixed penalty will be imposed on it in the scoring function. ;like ≥ If no penalty is imposed, the final deduction is based on the number of hallucinatory entities and the penalty coefficient. The product of the two values ​​is subtracted from the basic image-text similarity score.

[0048] choose ( The highest-scoring response is used as ,choose ( Low-scoring responses are considered as , obtain The specific methods and steps are described in section 3.1. The method is the same;

[0049] Preference pairs obtained from construction The conditional preference optimization loss function is used:

[0050]

[0051] in: π(...) is the conditional probability, ref is the reference strategy, β is the temperature coefficient, (...) is the Sigmoid function;

[0052] Step 4: Two-stage gradient gating based on prediction entropy:

[0053] To prevent low-quality gradients from missing modal samples from contaminating the model, a two-level dynamic weighting is implemented:

[0054] 4.1. Sample-level gating

[0055] During local training on the client side, the local model encounters input samples of varying quality. If a sample's lack of modality leads to high uncertainty in the local model's response, it is desirable to reduce the sample's contribution to model updates. The training process refers to performing local training using local preference-aligned training data to update the expert ensemble model parameters.

[0056] First, it is necessary to quantify the uncertainty of the model regarding the current sample; then, calculate the uncertainty of the model regarding the current sample. Token-level prediction entropy:

[0057]

[0058] According to Token-level information entropy Construct a nonlinear weighting function:

[0059]

[0060] The preference loss is weighted, and the weighted loss is backpropagated to update the local expert ensemble model. :

[0061] Automatically "stops" parameter updates for samples with high uncertainty, preventing erroneous preference data from "contaminating" model parameters;

[0062] 4.2. Client-level gating

[0063] In federated learning, some clients may have poor data quality, which can negatively impact the global model. Therefore, it is necessary to reduce the influence of poor-quality client data during global aggregation.

[0064] Client calculates reliability score :

[0065] Client After one round of local training, calculate the average weight of all training samples:

[0066]

[0067] If a client's data consists entirely of high-quality, low-uncertainty data, ≈ 1; if all the data consists of high noise and lacks modalities, ≈0;

[0068] During aggregation, the server assigns a corresponding aggregation weight to each client. The aggregation weight is determined by the sample size of the client. Reliability score corresponding to this client The product of these factors determines the automatic reduction of the contribution of low-quality data nodes. The formula is as follows:

[0069] ;

[0070] Step 5: Client representation based on low-dimensional preference signatures:

[0071] In each round After the local alignment training is completed, the preference signature is calculated based on the updated gradient projection of the local parameters. The implementation method for calculating the local updated parameter difference is as follows:

[0072]

[0073] To protect client privacy (reduce the risk of inferring the original gradient / data from uploaded information) and reduce computational burden, high-dimensional vectors are projected into a low-dimensional space using a fixed random orthogonal matrix. projection:

[0074]

[0075] in The server generates a random seed, and the client uses QR decomposition to generate an orthogonal matrix. ; Issued and fixed during the initialization phase of the federal system;

[0076] To eliminate the difference in gradient magnitude between different clients due to the size of the data, the projected vector is normalized to obtain the preference signature vector:

[0077]

[0078] in Options include 128, 256, or 512. Select based on the number of clients / communication bandwidth; Larger clusters result in more refined clustering but also higher communication efficiency.

[0079] Step Six: Signature-based Clustering and Multi-Expert Aggregation:

[0080] The server receives model parameter updates uploaded by each client. Preference signature vector Reliability score and sample size The server performs clustering on the preference signature vectors, grouping clients with similar preferences into the same cluster. The update amounts for each client within each cluster are then weighted and aggregated based on their reliability scores to generate multiple corresponding global preference expert models. The implementation formula is shown below:

[0081]

[0082] The server determines the number of clusters using an adaptive clustering algorithm based on the distribution characteristics of the preference signature vectors uploaded by each client. Before each round of aggregation, the server calculates the contour coefficients for different cluster numbers and selects the optimal value as the number of expert models for the current round, thereby achieving a balance between personalized adaptation and global consistency.

[0083] Step 7: Experts distribute and integrate with local routing:

[0084] The server establishes a mapping table to record the center of each expert's "preference cluster," which is the cluster's set of all preference signature vectors. The average value, the server based on the preference signature vector Filtering based on distance: The server calculates the preference signature vector most recently uploaded by the client. and The cosine similarity between the centers of each expert cluster is sorted from high to low before distribution. An expert gave client k;

[0085] The client uses the gating coefficient By integrating expert models, a multi-expert ensemble model is obtained. :

[0086] Among them, the gating coefficient Client k initializes a vector of length P. , Let the vector be a real number vector, with initial values ​​set to random small values; this is to ensure that the sum of the weights of all experts is 1. And the weights are positive numbers, for The gate coefficients are obtained by performing a softmax operation:

[0087] .

[0088] Compared with existing technologies, this technical solution has the following technical effects by introducing modality-deficient preference pair construction, uncertainty-driven gradient gating mechanism, and preference signature-based multi-expert aggregation and routing strategy within the federated learning framework:

[0089] (i) Improve the consistency between the generated results and the available modal information under the condition of missing modal input.

[0090] This technical solution introduces modality mask variables during the preference pair construction stage, and adopts differentiated preference discrimination strategies for samples in different modal complete states:

[0091] In the case of missing image modalities, fine-grained visual descriptive words contained in candidate responses are detected and penalized; in the case of missing text modalities, an entity existence verification mechanism is used to constrain candidate responses to describe only the discernible content in the image; in the case of complete images and text, a visual-semantic consistency score is used to distinguish between high-quality descriptions and low-consistency descriptions.

[0092] Through the aforementioned technical features, the model reduces the generation of visual detail descriptions that are not supported by input modality evidence during the lack of modality reasoning process, thereby improving the consistency and interpretability between the generated results and the currently available modality information.

[0093] (ii) Reduce the interference of missing modal samples on the federated training process and improve the stability of global model updates.

[0094] This technical solution introduces a sample-level gating mechanism based on prediction uncertainty during the preference optimization process to quantify the prediction entropy of the model on samples lacking modality, and adjusts the gradient contribution of preference loss accordingly. Meanwhile, in the federated aggregation stage, a reliability score is constructed based on the statistical value of sample weights within the client, and this score is incorporated into the calculation of updated weights in the client.

[0095] By employing the dual gating techniques at both the sample and client levels, the impact of relevant samples and client parameter updates on the global model is reduced in client scenarios with a high proportion of missing modalities or low sample quality. This reduces the risk of noise updates caused by missing modal samples spreading in federated aggregation and improves the convergence stability of the federated training process.

[0096] (III) In scenarios with heterogeneous preferences among multiple institutions, achieve structural decoupling and personalized adaptation of model parameter updates.

[0097] This technical solution constructs a preference signature vector that only reflects the direction of parameter optimization by performing a low-dimensional projection on the local efficient fine-tuning parameter update direction. On the server side, the client is clustered based on the signature vector to form multiple preference expert models. The server sends expert models with high similarity to the preference signature to the client, and the client then performs weighted fusion of multiple expert models through a local routing mechanism.

[0098] Through the aforementioned technical features, in a federated learning environment, different clients can obtain matching model parameter combinations based on their preference features. This reduces the trade-off effect of a single global model in preference conflict scenarios without exposing the original data or complete gradient information, and improves the model's adaptability to the preference needs of different institutions.

[0099] (iv) Enhance the scalability of preference modeling and aggregation processes while maintaining federal privacy constraints.

[0100] This technical solution only uploads the low-dimensional projection representation and statistical information of parameter updates during the preference modeling and aggregation process, without relying on the centralized sharing of original training data, complete gradients, or explicit preference labels; the low-dimensional projection can serve as a compact representation of preference directions for server-side clustering and expert maintenance.

[0101] Through the aforementioned technical features, the method of this technical solution can support scalable deployment in scenarios with an increasing number of clients and more complex modal combinations, while meeting the privacy constraints of federated learning, thus avoiding privacy risks and increased communication overhead caused by centralized preference modeling. Attached Figure Description

[0102] Figure 1 This is a flowchart illustrating an embodiment;

[0103] Figure 2 This is a schematic diagram of the sub-process for constructing conditional preference pairs based on modal masks in the embodiment;

[0104] Figure 3 This is a schematic diagram of a multi-expert federated aggregation and routing architecture based on preference signatures provided in the embodiment. Detailed Implementation

[0105] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0106] Example:

[0107] Reference Figure 1 A federated multimodal preference learning and alignment method with missing modality conditions, comprising several local clients and a server, the method comprising the following steps:

[0108] Step 1: Server-side initialization configuration distribution

[0109] Before federated training begins, the server performs system initialization and distributes global configurations and public resources related to aligning training consistency to each client. The distributed content includes:

[0110] 1.1. Global Model and Reference Strategy Distribution: The server distributes a set of model parameters for federated alignment training to the client. The global model adopts a shared base + expert model structure, while the client's local model is composed of the distributed shared base and an updatable expert model. The expert model is an adjustable parameter module attached to the shared base model, wherein:

[0111] 1.1.1. Shared Base Distribution: The server distributes the shared base model parameters. Among them, the shared base model parameters Includes the visual encoder and language model backbone, which are shared by all clients throughout the federated training process. To reduce communication overhead, the server only sends out the command once during the initialization phase. The client loads the same base accordingly; in subsequent federation rounds, Keep frozen, meaning do not participate in federal aggregate updates;

[0112] 1.1.2. Initial Multi-Expert Model Deployment: The server initializes a set of expert models based on the shared base model. The shared base model and the corresponding expert model are then sent to the client; whereby... One value can be chosen, i.e., single-expert initialization; when When the value is greater than 1, the server can use the same initialization for all experts, for example... and shared base model Together they form the initial multi-expert model set. ;

[0113] The above initialization method ensures that multiple global preference expert models are subsequently generated on the server side based on preference signature clustering, and that the Top-p experts are distributed to the client during the model distribution phase, so that the client can seamlessly connect to the mechanism of local routing fusion.

[0114] 1.1.3 Reference Strategy Composition and round update rules: Reference strategy Compared with the current strategy Using the same structure, but frozen during this round of local training, it is only used for optimizing the loss in conditional preference settings. The calculation; in the first round of federated training, the client uses the policy starting point model, which is composed of the shared base model parameters and the matching initial expert model parameter set issued by the server during the initialization phase, as the reference policy. In round t>1, the client combines the expert model parameters issued by the server in the previous round with the shared base model parameters θ that are stored locally on the client and kept frozen to form the starting strategy model for this round of training, and then copies it to generate the reference strategy model. Before the start of this round of local training, the parameters of the reference policy model are frozen so that they do not participate in parameter updates during this round of training and are only used to calculate reference terms in preference optimization.

[0115] 1.2. Distribution of Consistent Random Projection Matrix Configuration: To support the subsequent generation of low-dimensional preference signatures and privacy compression, the server generates a unified random seed during the system initialization phase. The random seed is then distributed to each client. The client generates a random matrix based on the random seed and obtains a fixed random orthogonal projection matrix R through QR decomposition. The random orthogonal projection matrix R maintains fixed matrix parameters in multiple federated training rounds to ensure that the preference signatures generated in different rounds are located in the same low-dimensional representation space, thereby supporting similarity calculation and clustering processing on the server side.

[0116] 1.3. Distribution of Common Rules and Lexicon Related to Modality: The server distributes unified rule configurations for modality missingness determination to the client, namely, mapping null values / corruption / low quality thresholds to missing conditions, and a visual attribute lexicon for T-only scenarios. This ensures that the client follows a consistent constraint approach when performing modal mask determination and constructing conditional preference pairs;

[0117] 1.4. Training and Gating Hyperparameter Deployment: The server deploys the key hyperparameters and threshold ranges required for this example to the client. These key hyperparameters and threshold ranges include: the preference optimization temperature coefficient. Sample-level gated temperature parameters Entity existence threshold, penalty coefficient The proportion of inferior sampling locations The range of values ​​for the number of candidate responses N is used to ensure that the behavior of local training and gating updates on each client is consistent and controllable.

[0118] Step 2: Multimodal input state recognition and modal mask generation:

[0119] Configure local multimodal samples on the client ,in, Represents image modal data, This represents text modal data, where the subscript i indicates the i-th sample data, corresponding to the i-th modal mask variable below;

[0120] The client scans the samples using a modal detection operator to generate modal mask variables. Used to identify the modal completeness state of the current sample: ;in This indicates that both image and text modalities exist; This indicates that only the text modality exists, and the image modality is missing; This indicates that only the image modality exists, while the text modality is missing. The modality detection operator can be implemented in any of the following ways or combinations and supports a unified mapping of "null / damaged / low-quality threshold" to missing values: based on whether the data structure field is empty; based on whether the modality encoder output is a zero vector or a null value; based on the modality quality threshold.

[0121] The client stores ( ) for each sample , Triples, and put As a condition variable for all subsequent operators;

[0122] Step 3: Construct conditional preference pairs based on modality masks, such as Figure 2 As shown:

[0123] The client utilizes the current local model For the sample Generate a set of candidate responses:

[0124] in The number of candidate responses is selected, with a range of values. ;

[0125] And according to the mask Implement different automated screening strategies to construct the optimal response. and poor response Preference data pairs:

[0126] 3.1 Regarding IT data: When The system provides complete text and images instantly, and performs fine-grained cross-modal alignment: it utilizes the CLIP model to compute each answer. With images Visual-semantic consistency score: The candidate responses are sorted in descending order of score to obtain the sequence. The response with the highest score is selected as the preferred response. From the sorted position Take inferior response This forces local models to distinguish between high-quality descriptions and seemingly plausible illusions; among which ;

[0127] 3.2 For T-only data: When For text-only scenarios, apply visual attribute density penalty: load a pre-defined visual attribute lexicon. The lexicon contains visual descriptive terms that cannot be supported by text when visual modalities are lacking, such as: color, shape, size; spatial location, directional relationship; quantity, relative layout; domain-related visual entity descriptive terms; and for each candidate response Construct a set of visual penalty words:

[0128] ( )= .

[0129] And calculate the scoring function: ( ) = - Select the response with the highest score as Low response as , obtain The specific method is the same as step 3.1;

[0130] 3.3 For I-only data: When In the case of images only, perform entity existence closed-loop verification: similar to the IT scenario, but introduce an entity penalty term; for candidate responses Entity noun extraction is performed to obtain an entity set. For each entity Construct a prompt template and calculate the probability of the entity's presence in the image. : ,like Below the threshold If this occurs, a point deduction mechanism will be triggered, including an existence threshold. The final scoring function is defined as:

[0131] ( )=

[0132] in This is the penalty coefficient, with a value range of [0.1, 2.0]. It is an indicator function that only deducts points for objects determined to be hallucinations; existing objects are not deducted. This forces the local model to adopt a conservative WYSIWYG strategy when there are no user instructions, thereby suppressing hallucinations. The deduction mechanism is as follows: for candidate responses... Each entity extracted Calculate the entity in the image The probability of existence in ;like < If the entity is identified as a hallucinatory entity, a fixed penalty will be imposed on it in the scoring function. ;like ≥ If no penalty is imposed, the final deduction is based on the number of hallucinatory entities and the penalty coefficient. The product of the two values ​​is subtracted from the basic image-text similarity score.

[0133] choose ( The highest-scoring response is used as ,choose ( Low-scoring responses are considered as , obtain The specific method is the same as step 3.1;

[0134] Preference pairs obtained from construction The conditional preference optimization loss function is used:

[0135]

[0136] in: π(...) is the conditional probability, ref is the reference strategy, β is the temperature coefficient, (...) is the Sigmoid function;

[0137] Step 4: Two-stage gradient gating based on prediction entropy:

[0138] To prevent low-quality gradients from missing modal samples from contaminating the model, a two-level dynamic weighting is implemented:

[0139] 4.1. Sample-level gating

[0140] During local training on the client side, the local model encounters input samples of varying quality. If a sample's lack of modality leads to high uncertainty in the local model's response, it is desirable to reduce the sample's contribution to model updates. The training process refers to performing local training using local preference-aligned training data to update the expert ensemble model parameters.

[0141] First, it is necessary to quantify the uncertainty of the model regarding the current sample; then, calculate the uncertainty of the model regarding the current sample. Token-level prediction entropy:

[0142]

[0143] According to Token-level information entropy Construct a nonlinear weighting function:

[0144]

[0145] The preference loss is weighted, and the weighted loss is backpropagated to update the local expert ensemble model. :

[0146] Automatically "stops" parameter updates for samples with high uncertainty, preventing erroneous preference data from "contaminating" model parameters;

[0147] 4.2. Client-level gating

[0148] In federated learning, some clients may have poor data quality, which can negatively impact the global model. Therefore, it is necessary to reduce the influence of poor-quality client data during global aggregation.

[0149] Client calculates reliability score :

[0150] Client After one round of local training, calculate the average weight of all training samples:

[0151]

[0152] If a client's data consists entirely of high-quality, low-uncertainty data, ≈ 1; if all the data consists of high noise and lacks modalities, ≈0;

[0153] The weight of each client during server aggregation. Due to its sample size and The product of these factors determines the automatic reduction of the contribution of low-quality data nodes. The formula is as follows:

[0154] ;

[0155] Step 5: Client representation based on low-dimensional preference signatures:

[0156] In each round After the local alignment training is completed, the preference signature is calculated based on the updated gradient projection of the local parameters. The implementation method for calculating the local updated parameter difference is as follows:

[0157]

[0158] To protect client privacy (reduce the risk of inferring the original gradient / data from uploaded information) and reduce computational burden, high-dimensional vectors are projected into a low-dimensional space using a fixed random orthogonal matrix. projection:

[0159]

[0160] in The server generates a random seed, and the client uses QR decomposition to generate an orthogonal matrix. ; Issued and fixed during the initialization phase of the federal system;

[0161] To eliminate the difference in gradient magnitude between different clients due to the size of the data, the projected vector is normalized to obtain the preference signature vector:

[0162]

[0163] in Options include 128, 256, or 512. Select based on the number of clients / communication bandwidth; Larger clusters result in more refined clustering but also higher communication efficiency.

[0164] Step Six: Signature-based clustering and multi-expert aggregation, such as... Figure 3 As shown:

[0165] The server receives model parameter updates uploaded by each client. Preference signature vector Reliability score and sample size The server performs clustering on the preference signature vectors, grouping clients with similar preferences into the same cluster. The update amounts for each client within each cluster are then weighted and aggregated based on their reliability scores to generate multiple corresponding global preference expert models. The implementation formula is shown below:

[0166]

[0167] The server determines the number of clusters using an adaptive clustering algorithm based on the distribution characteristics of the preference signature vectors uploaded by each client. Before each round of aggregation, the server calculates the contour coefficients for different cluster numbers and selects the optimal value as the number of expert models for the current round, thereby achieving a balance between personalized adaptation and global consistency.

[0168] Step 7: Experts distribute and integrate with local routing:

[0169] The server establishes a mapping table to record the center of each expert's "preference cluster," which is the cluster's set of all preference signature vectors. The average value, the server based on the preference signature vector Filtering by distance: The server calculates the signature of the latest upload from the client. and The cosine similarity between the centers of each expert cluster is sorted from high to low before distribution. An expert gave client k;

[0170] The client uses the gating coefficient Fusion expert model:

[0171] Among them, the gating coefficient Client k initializes a vector of length P. , Let the vector be a real number vector, with initial values ​​set to random small values; this is to ensure that the sum of the weights of all experts is 1. And the weights are positive numbers, for The gate coefficients are obtained by performing a softmax operation:

[0172] .

[0173] This example uses medical imaging to further describe the case:

[0174] 1 System Environment and Parameter Configuration

[0175] (1) System Architecture

[0176] Construct a federated learning system, including: 1 central server; K = 10 client nodes, each corresponding to 10 medical institutions (hospitals).

[0177] The server and client exchange model parameters or their derived information through a secure communication protocol; clients do not directly share data with each other.

[0178] (2) Model configuration

[0179] Base model: Multimodal vision-language model (e.g., LLaVA-Med or a large multimodal model with equivalent functionality);

[0180] Parameter update method: The parameter efficient fine-tuning (PEFT) method is adopted. In this embodiment, the LoRA structure is used. This part is initially distributed to each client by the server.

[0181] (3) Data distribution

[0182] Each client holds local private medical data, including X-ray images and corresponding diagnostic texts or patient complaints;

[0183] The modal completeness ratio of images and text varies across different clients, with some clients exhibiting a high proportion of missing modal samples.

[0184] The data does not leave the client's local machine and is used only for local training.

[0185] Build a system that includes one central server and A federated learning network for individual hospital clients. The base model uses LLaVA-Med, which is distributed and frozen by the server to each client during the initialization phase. Each client holds a different proportion of X-ray and diagnostic report data.

[0186] 2 Modal detection and sample classification

[0187] The client scans each local sample one by one. (Based on the client...) A certain training sample For example:

[0188] The input samples only contained patient complaint text, such as: "The patient has been coughing for three days, accompanied by mild chest tightness," and did not include corresponding X-ray images;

[0189] The modality detection operator determines that the image modality is empty based on the data structure fields and outputs a modality mask: This mask serves as a condition variable for subsequent preference construction and loss calculation.

[0190] 3. Preferences for the construction process

[0191] (1) Candidate response generation

[0192] Response A: "There may be local structural features in the image, such as a local image appearance of about 2cm, which needs to be confirmed with further image data."

[0193] Response B: "No image modality input is currently provided. It is recommended to supplement relevant image examination data to support a more reliable judgment."

[0194] Response C: "Based on the existing textual information, a preliminary analysis can be made, but there is a lack of supporting video evidence, and further video information is needed to corroborate it."

[0195] The candidate responses mentioned above are only used to illustrate the differences in the model's generative behavior under different evidence availability conditions.

[0196] (2) Definition and use of visual attribute lexicon

[0197] In this embodiment, the visual attribute lexicon Defined as: a set of radiological or visually relevant descriptive terms that cannot be reliably inferred from textual descriptions alone in the absence of image modal input.

[0198] The lexicon includes at least one or a combination of the following categories: image structure descriptions; spatial and location descriptions; morphological and quantity descriptions.

[0199] The lexicon can be generated manually, maintained by domain experts, or based on medical text corpora, and can be expanded or tailored according to specific application scenarios.

[0200] (3) Preference discrimination in T-only scenarios

[0201] The system detected response A. Although the expression is logically coherent, it contains multiple [various errors] when the input is only text and no image modality is provided. The descriptive words in the input text that do not appear in the input text are included in the penalty set. .

[0202] Both responses B and C accurately identify the current technical state of "modal missing" and strictly adhere to the strategy of "generating only based on existing evidence", with zero penalty items.

[0203] According to the ranking strategy in this example, response A is determined to be a medium-quality hallucination sample with "leading" properties, denoted as . The highest response score is recorded as C. .

[0204] 4. Uncertainty Gating Training Process

[0205] (1) Calculation of sample-level uncertainty

[0206] Client for samples Perform forward inference to calculate the token-level prediction probability distribution output by the model, and calculate the prediction entropy accordingly:

[0207]

[0208] In this embodiment, if the model exhibits high uncertainty among different candidate diagnoses, the entropy value is high.

[0209] (2) Sample weight calculation

[0210] Construct sample weights based on predicted entropy: ,in For temperature parameters, the selectable range is... .

[0211] When the prediction entropy is high, the sample weight tends to be smaller, thereby reducing the impact of the sample on the model parameter update during backpropagation.

[0212] 5. Client Reliability Assessment and Server Aggregation

[0213] (1) Client reliability score

[0214] After completing one round of local training, the client calculates the average weight of all its samples: .

[0215] (2) Server aggregation

[0216] The server receives the parameter updates uploaded by each client and the corresponding... Then, calculate the aggregate weights as follows:

[0217]

[0218] in This indicates the number of client-side samples.

[0219] This aggregation method reduces the impact of clients with a high proportion of missing modes and significant prediction uncertainty on global updates.

[0220] 6. Preference Signature Generation and Projection

[0221] (1) Parameter difference calculation

[0222] The client calculates the update difference of the local LoRA parameters:

[0223]

[0224] (2) Low-dimensional projection

[0225] To generate the preference signature vector, the client uses a fixed random orthogonal matrix:

[0226]

[0227] in: The dimension of the flattened parametric difference vector; The projection dimension can be selected as 128, 256, or 512.

[0228] The preference signature vector is calculated as follows:

[0229] Then, L2 normalization is performed to obtain the final preference signature vector:

[0230]

[0231] 7. Server-side clustering, multi-expert generation and distribution

[0232] The server clusters the collected preference signature vectors. In this embodiment, for the two typical polarized preferences of 'concise' and 'extensive' in medical image diagnosis tasks, the number of clusters is determined. The default value is 2, which aims to verify the decoupling accuracy and stability of the method in this example with the minimum expert scale.

[0233] Preference Cluster 1: Clients that prefer concise and conservative diagnostic recommendations;

[0234] Preference Cluster Two: Clients that prefer detailed analysis and reasoning.

[0235] The server performs weighted aggregation of the model updates from clients within each cluster to generate the corresponding preference expert model. , .

[0236] 8. Client-side expert routing and model usage

[0237] The server distributes the Top-Matching expert cluster centers to the client based on the similarity between the client's preference signature and the cluster centers. One expert model (in this embodiment) ).

[0238] The client uses the received expert model as model parameters for local inference or subsequent training.

[0239] The innovative aspects of this example are as follows:

[0240] 1. The "counterfactual" preference based on modal mask awareness for automated construction mechanisms

[0241] This example proposes a reverse training mechanism that transforms missing data into hallucination suppression signals. Conditional logic transformation: In this example's federated learning system (including a central server and multiple client nodes), modality completion is not performed; instead, modality mask variables are introduced. This method automatically constructs illusion traps for T-only samples. The construction method is detailed as follows: using a pre-defined visual attribute density operator, it automatically identifies and penalizes responses that still describe visual details even in the absence of an image (defined as...). ), while rewarding conservative responses that honestly acknowledge missing graphs (defined as ).

[0242] Technical effects: This mechanism forces the model to learn the boundaries of visual perception, reduces the probability of generating visual words without evidence, reduces preference label noise, and improves the factuality of the model under incomplete data.

[0243] 2. A Federated Robust Preference Alignment Mechanism with Two-Level Gated Predictive Entropy

[0244] A two-level gated federated preference alignment method is proposed, which explicitly embeds prediction uncertainty into the backpropagation link of the model. By limiting the granularity of entropy calculation and introducing a family of truncated exponential gate functions, it achieves dynamic suppression of noisy preference signals under modality-deficient conditions, thereby improving the stability and robustness of the federated training process. In this example, prediction entropy is not used as a single scalar evaluation metric, but is calculated at at least one of the following granularities: a) Token-level prediction entropy (preferred implementation): as shown in step four; b) Sequence-level prediction entropy (optional implementation): the entire sequence is treated as a random variable, the complete output distribution is estimated, and its entropy value is calculated for uncertainty evaluation of the entire preference sample.

[0245] Technical effects: Effectively suppresses gradient contamination caused by high uncertainty bias samples in modality-deficient scenarios; prevents noisy client updates from being amplified in federated aggregation; and improves overall training stability and convergence consistency.

[0246] 3: Multi-expert maintenance and routing mechanism based on projection preference signature

[0247] A federated multi-expert maintenance and routing method based on low-dimensional preference signatures is proposed. The method uses a uniformly generated and synchronized random orthogonal projection matrix to compress the client preference update and restricts the preference signature to expert clustering and routing decision-making. This achieves structured modeling of preference heterogeneity without directly exposing the details of model update.

[0248] Preference signature generation and projection method: using a fixed random orthogonal projection matrix The gradient of the high-dimensional PEFT parameters is updated. Projection into a low-dimensional signature vector The projection matrix Generated by the server during system initialization, the generation process involves: generating a random matrix based on a unified random seed; obtaining an orthogonal projection matrix through QR decomposition or equivalent orthogonalization; and distributing the random seed or matrix identifier to each client, enabling clients to reproduce the same projection matrix. Furthermore, the projection matrix remains fixed across multiple federated rounds to avoid introducing additional instability. Multi-expert dynamic routing: the server no longer trains a single model but instead uses signature vectors. The clustering results maintain multiple preference experts. During the model delivery phase, the client selects a route based on the similarity between its own preference signature and each expert center, and performs weighted fusion of the selected expert models accordingly. The use of preference signatures is limited: In this example, the preference signature vector satisfies the following limitations: it is only used for: client preference clustering; expert model routing and selection; it is not used for: model parameter restoration; direct reconstruction of local data or gradients.

[0249] Technical benefits: This mechanism reduces the risk of inferring specific update details from uploaded information by using low-dimensional projection and application limitation, while significantly reducing communication overhead. It supports continuous changes in preferences rather than hard clustering; and allows clients with similar preferences to share expert knowledge.

Claims

1. A federated multimodal preference learning and alignment method with missing modality conditions, characterized in that it comprises a system consisting of several local clients and a server, the method comprising the following steps: Step 1: Server-side initialization configuration distribution: Before federated training begins, the server performs system initialization and distributes global configurations and public resources related to aligning training consistency to each client. The distributed content includes: 1.

1. Global Model and Reference Strategy Distribution: The server distributes a set of model parameters for federated alignment training to the client. The global model adopts a shared base + expert model structure, while the client local model is composed of the distributed shared base + updatable expert model. The expert model is an adjustable parameter module attached to the shared base model. 1.

2. Distribution of Consistent Random Projection Matrix Configuration: To support the subsequent generation of low-dimensional preference signatures and privacy compression, the server generates a unified random seed during the system initialization phase. The random seed is then distributed to each client. The client generates a random matrix based on the random seed and obtains a fixed random orthogonal projection matrix R through QR decomposition. The random orthogonal projection matrix R maintains fixed matrix parameters in multiple federated training rounds to ensure that the preference signatures generated in different rounds are located in the same low-dimensional representation space, thereby supporting similarity calculation and clustering processing on the server side. 1.

3. Distribution of Common Rules and Lexicon Related to Modality: The server distributes unified rule configurations for modality missingness determination to the client, namely, mapping null values / damaged / low-quality thresholds to missing conditions, and a visual attribute lexicon for T-only scenarios. This ensures that the client follows a consistent constraint approach when performing modal mask determination and constructing conditional preference pairs; 1.

4. Training and Gating Hyperparameter Distribution: The server distributes hyperparameters and threshold ranges to the client. These hyperparameters and threshold ranges include: preference-optimized temperature coefficient. Sample-level gated temperature parameters Entity existence threshold, penalty coefficient The proportion of inferior sampling locations The range of values ​​for the number of candidate responses N is used to ensure that the behavior of local training and gating updates on each client is consistent and controllable. Step 2: Multimodal input state recognition and modal mask generation: Configure local multimodal samples on the client ,in, Represents image modal data, This represents text modal data, where the subscript i indicates the i-th sample data, corresponding to the i-th modal mask variable; The client scans the samples using a modal detection operator to generate modal mask variables. Used to identify the modal state of the current sample: ;in This indicates that both the image modality and the text modality exist; This indicates that only the text modality exists, and the image modality is missing; This indicates that only the image modality exists, and the text modality is missing; Step 3: Construction of conditional preference pairs based on modality masks: The client utilizes the current local model For the sample Generate a set of candidate responses: in The number of candidate responses is selected, with a range of values. ; And according to the mask Implement different automated screening strategies to construct the optimal response. and poor response Preference data pairs; Step 4: Two-stage gradient gating based on prediction entropy: To prevent low-quality gradients from missing modal samples from contaminating the model, a two-level dynamic weighting is implemented: 4.

1. Sample-level gating; 4.

2. Client-level gating: The client calculates the reliability score. ; During aggregation, the server assigns a corresponding aggregation weight to each client. The aggregation weight is determined by the sample size of the client. Reliability score corresponding to this client The product of the two factors determines the outcome. Step 5: Client representation based on low-dimensional preference signatures: After each round of local alignment training, the preference signature is calculated based on the updated gradient projection of the local parameters, as follows: ， To protect client privacy and reduce computational burden, high-dimensional vectors are projected into a low-dimensional space using a fixed random orthogonal matrix. projection: ， in The server generates a random seed, and the client uses QR decomposition to generate an orthogonal matrix. ; Issued and fixed during the initialization phase of the federal system; The projected vector is normalized to obtain the preference signature vector: ， Where t represents the t-th round; Choose one of 128, 256, or 512. Select based on the number of clients / communication bandwidth; Larger clusters result in more refined clustering but also higher communication efficiency. Step Six: Signature-based Clustering and Multi-Expert Aggregation: The server receives model parameter updates uploaded by each client. Preference signature vector Reliability score and sample size The server performs clustering on the preference signature vectors, grouping clients with similar preferences into the same cluster. The update amounts for each client within each cluster are then weighted and aggregated based on their reliability scores to generate multiple corresponding global preference expert models. The implementation formula is shown below: ； Step 7: Experts distribute and integrate with local routing: The server establishes a mapping table to record the preference cluster center corresponding to each expert, i.e., all preference signature vectors of that cluster. The average value, the server based on the preference signature vector The distance is used to filter and select p experts to be sent to the corresponding clients. The clients receive the expert models and update their local expert models. The client uses the gating coefficient By fusing expert models, a multi-expert ensemble model is obtained. : ， The client processes the input and generates the corresponding candidate set based on the fused expert ensemble model and the frozen base.

2. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, The modal detection operator in step two is implemented using any of the following methods or combinations and supports a unified mapping of null / damaged / low-quality thresholds to missing values: based on whether a data structure field is empty; based on whether the modal encoder output is a zero vector or a null value; Based on modal quality threshold judgment; And the client stores ( ) for each sample. , Triples, and put It serves as the condition variable for all subsequent operators.

3. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, In step three, based on the mask Implement different automated screening strategies to construct the optimal response. and poor response The specific steps for obtaining preference data pairs are as follows: 3.1 Regarding IT data: When The system provides complete text and images instantly, and performs fine-grained cross-modal alignment: it utilizes the CLIP model to compute each answer. With images Visual-semantic consistency score: The candidate responses are sorted in descending order of score to obtain the sequence. The response with the highest score is selected as the preferred response. From the sorted position Take inferior response This forces the local model to distinguish between high-quality descriptions and illusions; among them ; 3.2 For T-only data: When For text-only scenarios, apply visual attribute density penalty: load a pre-defined visual attribute lexicon. The lexicon contains visual descriptive words that cannot be supported by text when visual modalities are lacking; for each candidate response Construct a set of visual penalty words: （ ）= ， And calculate the scoring function: ( ) = - Select the response with the highest score as Low response as , obtain Specific methods and steps 3.1 Obtaining The method is the same; 3.3 For I-only data: When In the case of images only, perform entity existence closed-loop verification: similar to the IT scenario, but introduce an entity penalty term; for candidate responses Entity noun extraction is performed to obtain an entity set. For each entity Construct a prompt template and calculate the probability of the entity's presence in the image. : ,like Below the threshold If this occurs, a point deduction mechanism will be triggered, including an existence threshold. The final scoring function is defined as: ( )= ， in This is the penalty coefficient, with a value range of [0.1, 2.0]. This is an indicator function used to determine whether an object in the response is a hallucinatory object. When an object is determined to be a hallucinatory object, the indicator function takes a value of 1, triggering a corresponding penalty; when the object is determined to be a real object in the image, the indicator function takes a value of 0, and no penalty is triggered. This forces the local model to adopt a conservative WYSIWYG strategy when there are no user instructions, thereby suppressing hallucinations. The penalty mechanism is as follows: for candidate responses... Each entity extracted Calculate the entity in the image The probability of existence in ;like < If the entity is identified as a hallucinatory entity, a fixed penalty will be imposed on it in the scoring function. ;like ≥ If no penalty is imposed, the final deduction is based on the number of hallucinatory entities and the penalty coefficient. The product of the two values ​​is subtracted from the basic image-text similarity score. choose ( The highest-scoring response is used as ,choose ( Low-scoring responses are considered as , obtain Specific methods and steps 3.1 Obtaining The method is the same; Preference pairs obtained from construction The conditional preference optimization loss function is used: ， in: π(...) is the conditional probability, ref is the reference strategy, β is the temperature coefficient, (...) is the Sigmoid function.

4. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, The specific steps for implementing the two-level dynamic weighting in step four are as follows: 4.

1. Sample-level gating During local training on the client side, the local model will encounter input samples of varying quality; If a sample causes high uncertainty in the local model's response due to a lack of modality, it is desirable to reduce the contribution of that sample to the local model update. First, it is necessary to quantify the degree of uncertainty of the local model regarding the current sample; Calculate the local model for the current sample Token-level prediction entropy: ， According to Token-level information entropy Construct a nonlinear weighting function: ， The preference loss is weighted, and the weighted loss is backpropagated to update the local expert ensemble model. : Automatically stopping parameter updates for high-uncertainty samples prevents erroneous preference data from contaminating model parameters; 4.

2. Client-level gating Reduce the influence of poor-quality client data during global aggregation; Client calculates reliability score : Client After one round of local training, calculate the average weight of all training samples: ， If a client's data consists entirely of high-quality, low-uncertainty data, ≈1; If all the data consists of high noise and lacks modal characteristics, ≈0; The weight of each client during server aggregation. The sample size of each client and The product of these factors determines the automatic reduction of the contribution of low-quality data nodes. The formula is as follows: 。 5. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, The specific process of the server performing clustering algorithm processing on the preference signature vector in step six is ​​as follows: The server determines the number of clusters using an adaptive clustering algorithm based on the distribution characteristics of the preference signature vectors uploaded by each client. ; Before each round of aggregation, the server calculates the contour coefficients for different cluster numbers and selects the optimal value as the number of expert models for the current round, thereby achieving a balance between personalized adaptation and global consistency.

6. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, In step seven, the server uses the preference signature vector. The specific process of filtering by distance is as follows: The server calculates the latest signature uploaded by the client. and The cosine similarity between the centers of each expert cluster is sorted from high to low before distribution. An expert gave the client k.

7. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, In step seven, the gating coefficient... Client k initializes a vector of length P. , Let the vector be a real number vector, with initial values ​​set to random small values; this is to ensure that the sum of the weights of all experts is 1. And the weights are positive numbers, for The gate coefficients are obtained by performing a softmax operation: 。 8. The federated multimodal preference learning and alignment method with missing modal conditions according to claim 1, characterized in that, In step 1.1, the global model and reference strategy are distributed as follows: the server distributes the set of model parameters for federated alignment training to the client. The global model adopts a shared base + expert model structure. in: 1.1.

1. Shared Base Distribution: The server distributes the shared base model. , Includes the visual encoder and language model backbone, which are shared by all clients throughout the federated training process. To reduce communication overhead, the server only sends out the command once during the initialization phase. The client loads the shared base accordingly; in subsequent federation rounds, θ remains frozen and does not participate in federation aggregation updates. 1.1.

2. Initial Multi-Expert Model Deployment: The server initializes a set of expert models based on the shared base model. The shared base model and the corresponding expert model are then sent to the client; whereby... One value can be chosen, i.e., single-expert initialization; when When the value is greater than 1, the server uses the same initialization for all experts and shares the base model. Together they form the initial multi-expert model set. ; The above initialization method distributes Top-p experts to the client during the model distribution phase, and the client's local route fusion mechanism is seamlessly integrated. 1.1.3 Reference Strategy Composition and round update rules: Reference strategy Compared with the current strategy Using the same structure, but frozen during this round of local training, it is only used for optimizing the loss in conditional preference settings. The calculation; in the first round of federated training, the client uses the policy starting point model, which is composed of the shared base model parameters and the matching initial expert model parameter set issued by the server during the initialization phase, as the reference policy. In round t>1, the client combines the expert model parameters from the previous round sent by the server with the shared base model θ stored locally on the client and kept frozen to form the starting strategy model for this round of training, and then copies it to generate the reference strategy model. Before the start of this round of local training, the parameters of the reference policy model are frozen so that the parameters of the reference policy model do not participate in parameter updates during this round of training, but are only used to calculate reference terms in preference optimization.

Images