Multi-modal data feature re-entanglement enhancement method for federal body intelligent scene

Abstract

The invention provides a multi-modal data feature re-entanglement enhancement method for a federal body intelligent scene, and the method comprises the steps: training a core feature encoder and a deviation feature encoder respectively through a plurality of clients, and extracting core sample features and deviation sample features; respectively training a core feature generator and a deviation feature generator to fit the distribution of the core sample features and the deviation sample features of the local sample data of the client; respectively calculating distribution difference indexes of fitting sample data generated by fitting the core feature generator and the deviation feature generator of the client, and performing cross fusion on the core feature generators and the deviation feature generators from different clients according to the distribution difference indexes so as to construct a virtual enhanced client, generating enhanced training sample data, and outputting the enhanced training sample data. The target federated model is trained and optimized, and the problems that deviation types are unknown and diversified and a depolarization technology is difficult to take effect in a traditional federated intelligent application environment are solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method for enhancing the re-entanglement of multimodal data features for federated embodied intelligence scenarios. Background Technology

[0002] Federated embodied intelligence learning is a collaborative learning paradigm that trains a shared federated model without compromising the data privacy of client (agent) components. This collaborative learning paradigm is highly compatible with distributed embodied intelligence applications such as the Internet of Things (IoT) and edge computing. In these applications, data is often generated by agent devices deployed in different locations or environments, such as humanoid robots and wheeled autonomous vehicles. However, due to differences in the environment and operating status of these agent devices, the data they generate often exhibits significant heterogeneity and may introduce biased features. For example, a photo taken by a wheeled autonomous vehicle in a dimly lit environment such as a tunnel or corner may have a darker background compared to a photo taken by a wheeled autonomous vehicle in a well-lit location. These darker photos not only contain core features of the target of interest, such as the target's outline or color, but may also contain a large amount of shadowed background with varying brightness, i.e., biased features. Using data containing biased features to train a federated model may seriously mislead it, thereby reducing the model's optimization performance. Traditional debiasing techniques can usually only effectively deal with a single bias feature contained in the sample. However, in federated embodied intelligence applications, each client (agent) may be in a different environment, thus introducing different bias features into their respective local data.

[0003] Due to privacy requirements, information on these biased features cannot be shared in advance in federated embodied intelligence applications. Therefore, under conditions where the types of biased features are diverse and unknown, neural network models are less adaptable to these multimodal data containing biased features, thereby reducing model performance. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios, which solves the problems of unknown and diverse bias types and the ineffectiveness of bias removal techniques in traditional federated embodied intelligence application environments.

[0005] This application provides a method for enhancing multimodal data feature re-entanglement in federated embodied intelligence scenarios, the method comprising: Multiple clients use their local sample data to train core feature encoders and bias feature encoders respectively, so as to extract core sample features and bias sample features from the local sample data through the core feature encoder and bias feature encoder respectively; Multiple clients train a core feature generator and a deviation feature generator respectively based on the core sample features and deviation sample features extracted by the core feature encoder and the deviation feature encoder, so as to fit the distribution of the core sample features and deviation sample features of the client's local sample data respectively. The pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the core feature generator and the deviation feature generator of the client respectively, and cross-merges the core feature generator and the deviation feature generator from different clients according to the distribution difference index to construct at least one virtual enhanced client. Enhanced training sample data is generated based on the at least one virtual enhanced client, and the target federated model is trained and optimized based on the enhanced training sample data.

[0006] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios includes training a core feature encoder, comprising: The parameters of the core feature encoder are optimized by minimizing a first loss function of the core feature encoder; the first loss function is: ; Among them, the The first loss function characterizing the core feature encoder, Characterizing the first cross-entropy loss function, For local sample data, The true label is the core feature; Characterizes the core feature encoder, where These are the parameters of the core feature encoder. The encoder characterizes the core feature encoder for the input The output; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output of .

[0007] In some embodiments, the multimodal data feature reentanglement enhancement method for federated embodied intelligence scenarios includes training a bias feature encoder, comprising: The parameters of the bias feature encoder are optimized by minimizing the second loss function of the bias feature encoder; The second loss function is: ; in, Characterizes the second general cross-entropy loss function; For local sample data, The true label is the core feature; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output of .

[0008] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios, wherein training the core feature generator and the bias feature generator separately includes: The core feature encoder, the core feature generator, and the first discriminator corresponding to the core feature generator are jointly optimized using a first fusion loss function; The deviation feature encoder, the deviation feature generator, and the second discriminator corresponding to the deviation feature generator are jointly optimized using a second fusion loss function. Wherein, the first fusion loss function The second fusion loss function ; Among them, the Characterizing the first fusion loss function, The first loss function characterizing the core feature encoder, Adversarial loss characterizing the core feature generator; Characterizing the second fusion loss function, The second loss function characterizing the bias feature encoder, Adversarial loss for characterization bias feature generator; Characterizes the preset loss reconstruction factor.

[0009] In some embodiments, in the multimodal data feature reentanglement enhancement method for federated embodied intelligence scenarios, the pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the core feature generator and the bias feature generator of the client, including: The pre-configured coordinator determines the mean distribution difference between the two types of generators for each client through the following steps: the two types of generators include a core feature generator and a deviation feature generator. For the current client to be evaluated, the coordinator drives its core feature generator and bias feature generator respectively to generate a first fitted sample dataset corresponding to each of the two types of generators; for each other client, the coordinator drives a generator of the same type to generate a second fitted sample dataset corresponding to each of the two types of generators; the coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to each of the two types of generators; the coordinator arithmetically averages the distribution distances corresponding to the two types of generators to determine the mean distribution difference between the two types of generators for the current client.

[0010] In some embodiments, in the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios, the coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to the two types of generators, including: For each type of generator, the feature distance component is determined based on the difference index between each pair of fitted sample feature vectors that correspond sequentially in the first fitted sample dataset and the second fitted sample dataset. Select all fitting sample feature vectors with the same label from the first fitting sample dataset and the second fitting sample dataset, and calculate the mean of the fitting sample feature vectors with the same label in the first fitting sample dataset and the second fitting sample dataset respectively, to obtain the first average feature vector corresponding to the first fitting sample dataset and the second average feature vector corresponding to the second fitting sample dataset. Calculate the distance between the first average feature vector and the second average feature vector, and determine the label distance component between the first fitted sample dataset and the second fitted sample dataset; Based on the optimal transmission strategy, the feature distance component and the label distance component are fused to form the distribution distance corresponding to this type of generator.

[0011] In some embodiments, the multimodal data feature reentanglement enhancement method for federated embodied intelligence scenarios, wherein the difference index based on each pair of fitted sample feature vectors corresponding sequentially in the first fitted sample dataset and the second fitted sample dataset includes: The sum of the squares of the differences between the feature vectors of each pair of fitted samples in the first fitted sample dataset and the second fitted sample dataset in sequence is calculated as the difference index.

[0012] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios, wherein the step of cross-fusion of core feature generators and bias feature generators from different clients based on the distribution difference index to construct at least one virtual enhanced client includes: Based on the distribution difference index of the core feature generators of all clients, generate the first sorted sequence of the core feature generators; Based on the distribution difference index of the deviation feature generators of all clients, a second sorting sequence of deviation feature generators is generated; At least one source core feature generator is selected from the first sorting sequence; the source core feature generator is a preset number of core feature generators that are sorted first. The fusion window is determined based on the position of the target deviation feature generator, which is from the same client as the selected source core feature generator, in the second sorting sequence; The source core feature generator is fused with at least one deviation feature generator from other clients located within the fusion window to generate a virtual enhanced client.

[0013] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios includes fusing the source core feature generator with at least one bias feature generator from other clients located within the fusion window, comprising: The source core feature generator is fused with at least one bias feature generator from other clients located within the fusion window based on the following formula: ; in, This represents an enhanced client generated by the re-entanglement of the source core feature generator from client i and the bias feature generator from client j; Characterizes the source core feature generator from client i. The characterization comes from the client-side j-bias feature generator. Represents the heavy entanglement coefficient; The joint label representing the enhanced client-generated sample. Characteristic from Core feature labels , Characteristic from Deviation feature label .

[0014] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios further includes: For different application scenarios of federated embodied intelligence, different re-entanglement coefficients λ are set to generate enhanced training sample data with different proportions of bias features.

[0015] This application provides a method for enhancing multimodal data features through re-entanglement in federated embodied intelligence scenarios. The method transforms the need for debiasing into feature enhancement. During federated model optimization, two encoders (a core feature encoder and a bias feature encoder) are trained for each client (agent). These encoders extract core and bias features from the client's (agent's) local sample data, achieving decoupling of local sample features. This process effectively breaks the false associations established by bias features in training samples and solves the problems of inconsistent bias features across clients (agents) and the coordinator's inability to obtain this bias feature information. Subsequently, the decoupled features extracted by the core and bias feature encoders are used to train a core feature generator and a bias feature generator. Feature generators are used to fit core features and bias features extracted from local sample data, respectively. Finally, the coordinator evaluates the core feature generators and bias feature generators of different clients based on the distribution differences of the generated samples, and performs cross-entanglement to fuse them into new enhanced clients (agents). The training samples generated by these enhanced clients (agents) will contain core features and bias features from different original client samples. Adding these new training samples to the optimization process of the federated model not only effectively weakens the negative impact of spurious associations in the original client samples, but also increases the diversity of training samples, further improving the optimization effect of the model. This effectively solves the problem of unknown and diverse bias types and the difficulty in effective debiasing techniques when dealing with biased client data in federated embodied intelligence applications. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This shows a two-dimensional projection of the biased training sample described in an embodiment of this application; Figure 2 The flowchart of the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios described in this application embodiment is shown. Figure 3 This is an architecture diagram of the data feature re-entanglement enhancement method described in the embodiments of this application; Figure 4 This is a diagram of the feature encoder and generator fusion network structure described in the embodiments of this application; Figure 5 This is a schematic diagram of the re-entanglement process of the feature generator described in the embodiments of this application; Figure 6 This is a schematic diagram of the re-entanglement matching strategy of the feature generator described in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0019] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0021] In federated embodied intelligence applications, the supervised multimodal training dataset D typically consists of a series of features f and corresponding labels l, i.e., D = {f, l}. Among these, features f often consist of core features. Sum of deviation characteristics It is formed by fusion, that is , It is the fusion coefficient. and Corresponding to core features Sum of deviation characteristics The labels. Taking the traditional image dataset CMNIST as an example, the labels of the core features. Labels of deviation features These correspond to the numbers and background colors in the samples, respectively. The main purpose of federated model training is to utilize local data from different clients (agents) to help the model improve on core features. and core tags To establish a precise correlation between them. However, when the deviation characteristics... and tags When a more obvious and learnable relationship exists between objects, the model will tend to establish a spurious association between them, leading to model drift and reduced optimization performance. For example, suppose a humanoid robot is deployed in a poorly lit area. In this environment, the robot's camera captures many photos of vehicles against dark backgrounds. When these photos are used to train an object recognition model, they are likely to mislead the model in identifying objects against "dark backgrounds" (where the robot is more likely to be ... ")" and "vehicles ( This establishes a relationship between "night" and "( )," ultimately causing the model to incorrectly assign "night" to "night" during runtime. "pedestrian ()" The character “)” was mistakenly identified as “vehicle ( )”. )".

[0022] To further illustrate the above issues, bias characteristics in the CMNIST dataset... and tags An artificial relationship was established between the samples in the dataset, and the impact of this artificial relationship on the model was observed through a series of experiments. Specifically, the experiments classified the training samples in the dataset according to their respective... The tags are divided into different subsets to simulate local data from different clients (agents). The proportion of these data subsets is... The training samples have the same Labels, the bias characteristics represented by these labels are called consistency bias, and the remaining training samples have randomness. Tags. By adjusting The size can change the bias characteristics in the training set. and The strength of artificial associations between tags. The experiment uses a projection algorithm... Figure 1 The image shows a two-dimensional projection of these client training samples. Green projection points represent bias features in the training samples that are random. ='2', =random color), red projection points represent the proportion of training samples. The samples have consistency bias ( ='2', =blue), the bias characteristics of the remaining training samples are random.

[0023] The distribution of projection points in the experiment shows that when the bias features in the training samples... and There is no clear relationship between the tags, that is When the value is 0, the distribution ranges of the green and red projection points almost completely overlap. With the deviation feature... and Enhanced relationships between tags, i.e. As the value increases from 0.1 to 1.0, the distribution ranges of the green and red projection points begin to decouple from each other until they completely separate. This change indicates a bias characteristic. and The existence of relationships between labels can cause a significant shift in the feature distribution of training samples, thereby reducing model performance.

[0024] Based on this, this application provides a method for enhancing multimodal data feature re-entanglement in federated embodied intelligence scenarios. This method transforms the debiasing requirement into feature enhancement. During the federated model optimization process, two encoders (a core feature encoder and a bias feature encoder) are additionally trained for each client (agent). These encoders extract core and bias features from the client's (agent's) local sample data, achieving decoupling of local sample features. This process effectively breaks the false associations established by bias features in the training samples and solves the problems of inconsistent bias features across clients (agents) and the coordinator's inability to obtain this bias feature information. Subsequently, the decoupled features extracted by the core feature encoder and bias feature encoder are used to train the core feature generator and the bias feature generator. The generators are used to fit the core features and bias features extracted from the local sample data. Finally, the coordinator evaluates the core feature generators and bias feature generators of different clients based on the distribution differences of the generated samples, and performs cross-entanglement to fuse them into new enhanced clients (agents). The training samples generated by these enhanced clients (agents) will contain core features and bias features from different original client samples. Adding these new training samples to the optimization process of the federated model not only effectively weakens the negative impact of spurious associations in the original client samples, but also increases the diversity of training samples, further improving the optimization effect of the model. This effectively solves the problem of unknown and diverse bias types and the difficulty in effective debiasing techniques when dealing with biased client data in federated embodied intelligence applications.

[0025] Please refer to Figure 2 , Figure 2 The flowchart illustrates the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to an embodiment of this application; the method includes the following steps S201-S204: S201. Multiple clients use their local sample data to train the core feature encoder and the bias feature encoder respectively, so as to extract the core sample features and the bias sample features from the local sample data through the core feature encoder and the bias feature encoder respectively. S202. Multiple clients train a core feature generator and a deviation feature generator respectively based on the core sample features and deviation sample features extracted by the core feature encoder and the deviation feature encoder, so as to fit the distribution of the core sample features and deviation sample features of the client's local sample data respectively. S203. The pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the core feature generator and the deviation feature generator of the client respectively, and cross-merges the core feature generator and the deviation feature generator from different clients according to the distribution difference index to construct at least one virtual enhanced client. S204. Generate enhanced training sample data based on the at least one virtual enhanced client, and train and optimize the target federated model based on the enhanced training sample data.

[0026] The client is an embodied intelligent agent deployed in a physical environment. Therefore, the client can also be referred to as an intelligent agent. The embodied intelligent agent includes "service robots, autonomous vehicles, drones, industrial robotic arms, etc."

[0027] The local sample data is the perception data collected by the perception module of the embodied intelligent agent, such as images, point clouds, depth maps or audio data collected by at least one of the intelligent agent's camera, LiDAR, depth sensor or microphone array.

[0028] The target federation model is used to enable the client to perform tasks in the physical world, such as assisting the intelligent agent in performing target recognition, autonomous navigation, robotic arm grasping, or human-computer interaction tasks.

[0029] The client's local sample data can also be referred to as local training samples, local training samples, etc.

[0030] Please refer to Figure 3 , Figure 3 This paper illustrates the overall architecture of the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios described in this application embodiment; as follows: Figure 3 As shown, the method targets a federated embodied intelligence scenario that includes a global coordinator and multiple intelligent agent clients. Similar to the traditional federated learning optimization process, the coordinator first randomly initializes a federated model. The weights are then passed to all clients; each client optimizes using its local training samples. The weight gradients of the model from this round of optimization are then fed back to the coordinator; the coordinator then aggregates the model gradients from all clients and updates the federated model. The weights are then updated and redistributed to all clients; this process is repeated until the federated model is fully implemented. The model optimization has converged.

[0031] The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios described in this application adds the following additional work based on the above-mentioned federated model optimization process. This work is mainly divided into three stages: In the first stage, each client trains two encoders using the same optimization mode as the federated model. and To extract the core features and bias features from the local training samples respectively; among them, the encoder The core feature encoder extracts core sample features from the local sample data. This is a deviation feature encoder that extracts deviation sample features from local sample data.

[0032] In the second stage, each client trains two generators using the decoupled features extracted in the first stage. and This is used to fit the distributions of core features and bias features in the training samples, respectively. The generator... The core feature generator fits the distribution of core sample features of the client's local sample data. This is a bias feature generator that fits the distribution of bias sample features to the client's local sample data.

[0033] In some embodiments, during the optimization process, the method merges the generator and encoder into a hybrid network, thereby improving the optimization efficiency of the two models.

[0034] In the third phase, the coordinator evaluates the generators on different clients. and The generated samples exhibit distributional differences, and the core feature generator from different clients is used to generate features based on predefined rules. Sum of deviation feature generator Cross-fusion is performed to generate virtual enhanced clients. The method uses these enhanced clients to generate diverse training samples to strengthen the optimization of the federated model, thereby improving the target federated model. Performance.

[0035] In step S201, multiple clients use their local sample data to train a core feature encoder and a bias feature encoder respectively, so as to extract core sample features and bias sample features from the local sample data through the core feature encoder and the bias feature encoder respectively.

[0036] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios includes training a core feature encoder, comprising: The parameters of the core feature encoder are optimized by minimizing a first loss function of the core feature encoder; the first loss function is: ; Among them, the The first loss function characterizing the core feature encoder, Characterizing the first cross-entropy loss function, For local sample data, The true label is the core feature; Characterizes the core feature encoder, where These are the parameters of the core feature encoder. The encoder characterizes the core feature encoder for the input The output; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output of .

[0037] In some embodiments, the multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios includes training a core feature encoder and training a bias feature encoder, comprising: The parameters of the bias feature encoder are optimized by minimizing the second loss function of the bias feature encoder; The second loss function is: ; in, Characterizes the second general cross-entropy loss function; For local sample data, The true label is the core feature; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output of .

[0038] The main purpose of federated model optimization is to optimize the core features of the samples. With core feature labels Establish a precise correlation between them. However, when bias features in the training samples... With core feature labels When there is a more learnable relationship between two entities, the model will tend to establish a false relationship between them, which will cause the model to drift and reduce its performance.

[0039] To eliminate this false association, the method described in this application embodiment trains two feature encoders for each client. and To extract the core features from the local samples respectively Sum of deviation characteristics Encoder The design principle is that the weaker the relationship between the sample and the bias feature, the smaller the loss obtained by the encoder, and vice versa. Based on this design principle, the core feature encoder in the method uses the following first loss function based on cross-entropy (CE) during optimization. : ; Deviation Feature Encoder The bias features in the training samples are learned using the following second loss function based on General Cross-Entropy (GCE): .

[0040] In step S202, multiple clients train a core feature generator and a deviation feature generator respectively based on the core sample features and deviation sample features extracted by the core feature encoder and the deviation feature encoder, so as to fit the distribution of the core sample features and deviation sample features of the client's local sample data respectively.

[0041] In other words, each client uses the core sample features extracted by its own core feature encoder to train a core feature generator, so that the core feature generator can learn and simulate the distribution of the client's local core sample features. Each client uses the deviation sample features extracted by its own deviation feature encoder to train a deviation feature generator, so that the deviation feature generator can learn and simulate the distribution of the client's local deviation sample features.

[0042] In some embodiments, training the core feature generator and the bias feature generator separately includes: The core feature encoder, the core feature generator, and the first discriminator corresponding to the core feature generator are jointly optimized using a first fusion loss function; The deviation feature encoder, the deviation feature generator, and the second discriminator corresponding to the deviation feature generator are jointly optimized using a second fusion loss function. Wherein, the first fusion loss function The second fusion loss function ; Among them, the Characterizing the first fusion loss function, The first loss function characterizing the core feature encoder, Adversarial loss characterizing the core feature generator; Characterizing the second fusion loss function, The second loss function characterizing the bias feature encoder, Adversarial loss for characterization bias feature generator; Characterizes the preset loss reconstruction factor.

[0043] Due to the privacy requirements of the federated learning paradigm, the client cannot directly upload features decoupled from local training samples to the coordinator. Furthermore, considering that the number of local training samples on the client may be limited, the method described here focuses on the core feature encoder. Sum of bias feature encoders Then, a Conditional Generative Adversarial Network (CGAN) is connected in series to fit the distributions of core features and bias features decoupled from the participant training samples. Specifically, the encoder... and The core feature generator is responsible for mapping the client's local training samples x to the decoupled feature space. Sum of deviation feature generator Responsible for fitting the distribution of core features and bias features in the sample. and ;in, These are the parameters of the core feature generator. These are the parameters for the deviation feature generator. Core feature generator. It can also be called Deviation Feature Generator It can also be called .

[0044] Core Feature Generator Learning from distribution arrive mapping , where ze = [z, ] is a label composed of random noise z and core features. It is pieced together. Labels for random noise z and core features The joint distribution This represents the true distribution of the core features of local data.

[0045] Similarly, deviation generator Learning from distribution arrive mapping , where zb = [z, ] is composed of random noise z and bias feature labels It is pieced together; Labels for random noise z and bias characteristics The joint distribution This represents the true distribution of local data deviation characteristics.

[0046] Subsequently, the method uses a core feature discriminator. Sum of bias features discriminants To evaluate the core feature generators separately Sum of deviation feature generator The validity of the generated samples.

[0047] Here, the discriminator It can be called Discriminator It can be called , For discriminator The parameters, For discriminator Parameters; In summary, generators , and discriminator , Through feature encoder , Training game, generator , The desired generated samples will allow the discriminator to... , It was mistakenly identified as a real sample by the discriminator. , The system then attempts to identify which samples were synthesized by the generator. This training game process can be formalized by the following formula: ; Where G is the core feature generator Deviation Feature Generator The general term; D is the core feature discriminator. Bias Feature Discriminator The general term; E is the core feature encoder. Deviation Feature Encoder The collective term, where x represents the client's local training samples; z represents random noise. Represents the distribution of training samples. Noise distribution, c corresponds to the core feature label Or deviation feature label . To simplify the training process of the feature encoder and feature generator and reduce optimization complexity, the encoder... , and generator , A fusion network (CN) is formed, as shown in Figure 4. The client's training samples serve as the input to the encoder, outputting decoupled sample features. The labels of the training samples, concatenated with random noise, serve as the input to the generator, outputting learned decoupled sample features. The discriminator simultaneously receives the output features from both the encoder and generator, determining whether the generator's output features originate from the training samples. Specifically, during the optimization process of the fusion network, the method utilizes loss functions... , The encoder is trained to extract more accurate core or biased features of the samples, through a loss function. , The generator is trained so that its output cannot be distinguished by the discriminator as coming from the encoder or the generator.

[0048] The fusion network combines the loss functions of the encoder and generator to form the following reconstruction optimization loss. (First fusion loss function) or (Second fusion loss function) Responsible for training the encoder and generator related to the core features, i.e. , and Similarly, when training the encoder and generator related to bias features, i.e. , and At that time, the fusion network uses reconstruction loss . In the reconstruction loss, ω represents the loss reconstruction factor, which is the generator in the fusion network. , and discriminator , Both use the Mean Squared Error (MSE) loss function.

[0049] .

[0050] In step S203, the pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the core feature generator and the deviation feature generator of the client respectively, and cross-merges the core feature generator and the deviation feature generator from different clients according to the distribution difference index to construct at least one virtual enhanced client.

[0051] In some embodiments, the pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the client's core feature generator and the bias feature generator, including: The pre-configured coordinator determines the mean distribution difference between the two types of generators for each client through the following steps: the two types of generators include a core feature generator and a deviation feature generator. For the current client to be evaluated, the coordinator drives its core feature generator and bias feature generator respectively to generate a first fitted sample dataset corresponding to each of the two types of generators; for each other client, the coordinator drives a generator of the same type to generate a second fitted sample dataset corresponding to each of the two types of generators; the coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to each of the two types of generators; the coordinator arithmetically averages the distribution distances corresponding to the two types of generators to determine the mean distribution difference between the two types of generators for the current client.

[0052] In other words, the distribution difference index is the mean distribution difference between the first fitted sample dataset generated by the generator corresponding to the current client and the second fitted sample dataset generated by the generators of each other client.

[0053] The distribution difference index is used to quantify the rarity of the feature distribution learned by the generator of each client.

[0054] In other words, the pre-configured coordinator drives the core feature generator of the current client to generate a first fitted core feature sample dataset, and drives the core feature generator of each other client to generate a second fitted core feature sample dataset respectively. The coordinator calculates the core feature distribution distance between the first fitted core feature sample dataset and each second fitted core feature sample dataset; The coordinator takes the arithmetic mean of all the calculated core feature distribution distances to determine the mean core feature distribution difference corresponding to the current client's core feature generator. The coordinator drives the deviation feature generator of the current client to generate a first fitted deviation feature sample dataset, and drives the deviation feature generator of each other client to generate a second fitted deviation feature sample dataset respectively. The coordinator calculates the deviation feature distribution distance between the first fitting deviation core feature sample dataset and each second fitting deviation feature sample dataset; The coordinator performs an arithmetic mean of all calculated deviation feature distribution distances to determine the mean deviation feature distribution difference corresponding to the deviation feature generator of the current client.

[0055] In some embodiments, the coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to the two types of generators, including: For each type of generator, the feature distance component is determined based on the difference index between each pair of fitted sample feature vectors that correspond sequentially in the first fitted sample dataset and the second fitted sample dataset. Select all fitting sample feature vectors with the same label from the first fitting sample dataset and the second fitting sample dataset, and calculate the mean of the fitting sample feature vectors with the same label in the first fitting sample dataset and the second fitting sample dataset respectively, to obtain the first average feature vector corresponding to the first fitting sample dataset and the second average feature vector corresponding to the second fitting sample dataset. Calculate the distance between the first average feature vector and the second average feature vector, and determine the label distance component between the first fitted sample dataset and the second fitted sample dataset; Based on the optimal transmission strategy, the feature distance component and the label distance component are fused to form the distribution distance corresponding to this type of generator.

[0056] In some embodiments, the difference index based on each pair of fitted sample feature vectors corresponding sequentially in the first fitted sample dataset and the second fitted sample dataset includes: The sum of the squares of the differences between the feature vectors of each pair of fitted samples in the first fitted sample dataset and the second fitted sample dataset in sequence is calculated as the difference index.

[0057] In some embodiments, the step of cross-fusing core feature generators and deviation feature generators from different clients based on the distribution difference index to construct at least one virtual enhanced client includes: Based on the distribution difference index of the core feature generators of all clients, generate the first sorted sequence of the core feature generators; Based on the distribution difference index of the deviation feature generators of all clients, a second sorting sequence of deviation feature generators is generated; At least one source core feature generator is selected from the first sorting sequence; the source core feature generator is a preset number of core feature generators that are sorted first. The fusion window is determined based on the position of the target deviation feature generator, which is from the same client as the selected source core feature generator, in the second sorting sequence; The source core feature generator is fused with at least one deviation feature generator from other clients located within the fusion window to generate a virtual enhanced client.

[0058] In some embodiments, fusing the source core feature generator with at least one bias feature generator from other clients located within the fusion window includes: The source core feature generator is fused with at least one bias feature generator from other clients located within the fusion window based on the following formula: ; in, This represents an enhanced client generated by the re-entanglement of the source core feature generator from client i and the bias feature generator from client j; Characterizes the source core feature generator from client i. The characterization comes from the client-side j-bias feature generator. Represents the heavy entanglement coefficient; The joint label representing the enhanced client-generated sample. Characteristic from Core feature labels , Characteristic from Deviation feature label .

[0059] exist Figure 2 In the middle, when the coordinator completes all feature generators and After training, the distribution of core features among clients will be evaluated based on the distribution distance between the synthetic samples generated by these generators. Sum of deviation characteristic distribution The differences between them.

[0060] In this embodiment of the application, the following is used Represents the core feature generator from client c Sum of deviation feature generator A general term. Feature generator. The generated sample set An entry consists of a series of features and tags, defined as Where c represents the number of clients, Represents the number of synthesized samples; Characteristic sample set The first in i The feature vector of a synthetic sample, when When generated by the core feature generator, It is a synthesized core feature vector, when When generated by the deviation feature generator, It is a synthetic bias feature vector. The synthetic sample can also be called a fitted sample, etc. for The corresponding tags.

[0061] Sample set It can also be called a fitted sample dataset, a synthetic sample dataset, a synthetic sample set, etc.

[0062] Since features and labels contribute differently to the distribution distance, the feature generators from the two clients will... and The distribution distance vector between synthetic sample sets is defined as follows: ; in, Represents the distribution distance vector. Representative client m generator The generated synthetic sample dataset, Representative client n generator The generated synthetic sample dataset; Represents the feature distance component, Represents the label distance component; represent The synthetic sample feature vector in Characteristic representation The synthetic sample feature vector in represent The sample labels in the sample.

[0063] In some embodiments, the distribution distance between the two given synthetic sample sets is calculated based on the optimal transmission strategy:

[0064] in, S represents the number of samples in the synthetic dataset. for The i-th synthetic sample vector in the data. for The i-th synthetic sample vector in the data. Characterization and The distribution distance between them The order that represents the distance.

[0065] Since the content of the sample labels is discrete, the distance between them cannot be directly calculated. Therefore, the method uses the following formula to convert the sample label distance into the sample feature distance in order to calculate the label distance component. : ； in, , , ; For all labels in dataset D equal to The feature vector of the sample A set; For the labels in dataset D, equal to The number of samples; For the labels in dataset D, equal to The number of samples.

[0066] In summary, the feature generator and Distribution distance between generated datasets Defined as: .

[0067] Because the method trains a core feature generator for each client during the optimization process... A deviation feature generator To comprehensively evaluate the feature distribution differences among these feature generators, the method designs a novel Distribution Difference Mean (DDM) metric, where, Let represent the mean of the distribution distances between the feature generator from client i∈[C] and the feature generators from all other clients, where C represents the total number of clients.

[0068] ； Feature generator representing client i and the feature generator of client j The distribution distance.

[0069] in, The larger the value, the greater the difference between the core features of the sample on client i and the core features of the sample on other clients. The larger the value, the more significant the bias characteristics in the sample; the sample characteristics in both of the above cases are of high value and are more conducive to improving the diversity of the generated samples.

[0070] Finally, the coordinator will generate core features from different clients. Sum of deviation feature generator Cross-fusion is performed to form a new enhanced client. This cross-fusion process (i.e., feature generator re-entanglement) is as follows: Figure 5 As shown. Specifically, the two DDM sequences are obtained by calculating the mean distribution distance of the feature generators on each client. and They are generated by the core feature generators of all participants. Sum of deviation feature generator Sort by the mean of the distribution distance. Core features generated by the core feature generators at the beginning of the sequence are rarer, and correspondingly, in The bias features generated by the earlier bias feature generators in the sequence will be more significant. For example, the background of a traffic camera photo taken under relatively extreme lighting conditions will be brighter or darker. The enhanced client is formed by the cross-entanglement and fusion of feature generators from the two DDM sequences mentioned above. The matching strategy between different types of feature generators during the fusion process is as follows: Figure 6 As shown.

[0071] The fusion process follows these principles: First: The fewer the core features generated by the generator, the better it is for improving the diversity of model training samples. Therefore, this invention prioritizes the use of the rarer features generated by the generator. The core feature generator is selected sequentially at the beginning of the sequence. .

[0072] Second: In real-world applications of federated embodied intelligence, changes in the client's state or environment are often gradual. Therefore, the core feature generator selected in the first step will be... All bias feature generators within a certain fusion window in the sequence are fused, and this fusion window is in... The position in the sequence is determined by a deviation feature generator, which comes from the same client as the core feature generator selected in the first step.

[0073] Third: To control the number of newly generated enhanced clients, the method only uses... The generators at the beginning of the sequence participate in the fusion; these generators are called source generators (SG). The above feature generator fusion process can be defined by the following formula: ； in, This represents an enhanced client generated by the re-entanglement of the core feature generator from client i and the bias feature generator from client j; The joint label representing the enhanced client-generated sample is composed of data from... Core feature labels and from Deviation feature label composition; This represents the heavy entanglement coefficient.

[0074] when At the same time, enhance the client. Fully inherited from the core feature generator Deviation Feature Generator Ignored, this is equivalent to debiasing the samples from client i. When At that time, the enhanced client only inherits from the deviation feature generator. Its generated samples will not contain any core features, which is of little help in optimizing the federated model.

[0075] In some embodiments, the method further includes: generating enhanced training sample data with different proportions of bias features by setting different re-entanglement coefficients λ for different federated embodied intelligence application scenarios.

[0076] The proposed multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios can control the bias components of generated data to a certain extent according to the specific needs of the target application scenario, thereby improving the practicality and flexibility of the method.

[0077] The proposed multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios synthesizes diverse multimodal training samples in federated embodied intelligence applications, thereby enhancing the training and optimization effect of the model. The encoder and generation fusion optimization mode can improve the optimization efficiency of different types of models. By adjusting the proportion of bias features through parameter adjustment, it can adapt to different embodied intelligence application needs and has strong flexibility. The re-entanglement enhancement method reduces the dependence on prior knowledge such as sample type, effectively solving the problems of unknown and diverse bias types and the ineffectiveness of bias removal techniques in traditional federated embodied intelligence application environments.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0079] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0081] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a platform server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0082] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for enhancing the re-entanglement of multimodal data features in federated embodied intelligence scenarios, characterized in that: The method includes: Multiple clients use their local sample data to train core feature encoders and bias feature encoders respectively, so as to extract core sample features and bias sample features from the local sample data through the core feature encoder and bias feature encoder respectively; Multiple clients train a core feature generator and a deviation feature generator respectively based on the core sample features and deviation sample features extracted by the core feature encoder and the deviation feature encoder, so as to fit the distribution of the core sample features and deviation sample features of the client's local sample data respectively. The pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the core feature generator and the deviation feature generator of the client respectively, and cross-merges the core feature generator and the deviation feature generator from different clients according to the distribution difference index to construct at least one virtual enhanced client. Enhanced training sample data is generated based on the at least one virtual enhanced client, and the target federated model is trained and optimized based on the enhanced training sample data.

2. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 1, characterized in that, Training the core feature encoder includes: The parameters of the core feature encoder are optimized by minimizing a first loss function of the core feature encoder; the first loss function is: ； Among them, the The first loss function characterizing the core feature encoder, Characterizing the first cross-entropy loss function, For local sample data, The true label is the core feature; Characterizes the core feature encoder, where These are the parameters of the core feature encoder. The encoder characterizes the core feature encoder for the input The output; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output.

3. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 1, characterized in that, Training the bias feature encoder includes: The parameters of the bias feature encoder are optimized by minimizing the second loss function of the bias feature encoder; The second loss function is: ; in, Characterizes the second general cross-entropy loss function; For local sample data, The true label is the core feature; Characterizing the bias feature encoder, where These are the parameters of the deviation feature encoder. Characteristic encoder for bias features on input The output.

4. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 1, characterized in that, The separate training of the core feature generator and the bias feature generator includes: The core feature encoder, the core feature generator, and the first discriminator corresponding to the core feature generator are jointly optimized using a first fusion loss function; The deviation feature encoder, the deviation feature generator, and the second discriminator corresponding to the deviation feature generator are jointly optimized using a second fusion loss function. Wherein, the first fusion loss function The second fusion loss function ; Among them, the Characterizing the first fusion loss function, The first loss function characterizing the core feature encoder, Adversarial loss characterizing the core feature generator; Characterizing the second fusion loss function, The second loss function characterizing the bias feature encoder, Adversarial loss for characterization bias feature generator; Characterizes the preset loss reconstruction factor.

5. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 1, characterized in that, The pre-configured coordinator calculates the distribution difference index of the fitted sample data generated by the client's core feature generator and the bias feature generator, including: The pre-configured coordinator determines the mean distribution difference between the two types of generators for each client through the following steps: the two types of generators include a core feature generator and a deviation feature generator. For the current client to be evaluated, the coordinator drives its core feature generator and bias feature generator respectively to generate a first fitted sample dataset corresponding to each of the two types of generators; for each other client, the coordinator drives a generator of the same type to generate a second fitted sample dataset corresponding to each of the two types of generators; the coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to each of the two types of generators; the coordinator arithmetically averages the distribution distances corresponding to the two types of generators to determine the mean distribution difference between the two types of generators for the current client.

6. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 5, characterized in that, The coordinator calculates the distribution distance between the first fitted sample dataset and the second fitted sample dataset corresponding to the two types of generators, including: For each type of generator, the feature distance component is determined based on the difference index between each pair of fitted sample feature vectors that correspond sequentially in the first fitted sample dataset and the second fitted sample dataset. Select all fitting sample feature vectors with the same label from the first fitting sample dataset and the second fitting sample dataset, and calculate the mean of the fitting sample feature vectors with the same label in the first fitting sample dataset and the second fitting sample dataset respectively, to obtain the first average feature vector corresponding to the first fitting sample dataset and the second average feature vector corresponding to the second fitting sample dataset. Calculate the distance between the first average feature vector and the second average feature vector, and determine the label distance component between the first fitted sample dataset and the second fitted sample dataset; Based on the optimal transmission strategy, the feature distance component and the label distance component are fused to form the distribution distance corresponding to this type of generator.

7. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 6, characterized in that, The difference index based on each pair of fitted sample feature vectors corresponding sequentially in the first fitted sample dataset and the second fitted sample dataset includes: The sum of the squares of the differences between the feature vectors of each pair of fitted samples in the first fitted sample dataset and the second fitted sample dataset in sequence is calculated as the difference index.

8. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 1, characterized in that, The step of cross-fusing core feature generators and deviation feature generators from different clients based on the distribution difference index to construct at least one virtual enhanced client includes: Based on the distribution difference index of the core feature generators of all clients, generate the first sorted sequence of the core feature generators; Based on the distribution difference index of the deviation feature generators of all clients, a second sorting sequence of deviation feature generators is generated; At least one source core feature generator is selected from the first sorting sequence; the source core feature generator is a preset number of core feature generators that are sorted first; The fusion window is determined based on the position of the target deviation feature generator, which is from the same client as the selected source core feature generator, in the second sorting sequence; The source core feature generator is fused with at least one deviation feature generator from other clients located within the fusion window to generate a virtual enhanced client.

9. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 8, characterized in that, The process of fusing the source core feature generator with at least one bias feature generator from other clients located within the fusion window includes: The source core feature generator is fused with at least one bias feature generator from other clients located within the fusion window based on the following formula: ; in, This represents an enhanced client generated by the re-entanglement of the source core feature generator from client i and the bias feature generator from client j; Characterizes the source core feature generator from client i. The characterization comes from the client-side j-bias feature generator. Represents the heavy entanglement coefficient; The joint label representing the enhanced client-generated sample. Characteristic from Core feature labels , Characteristic from Deviation feature label .

10. The multimodal data feature re-entanglement enhancement method for federated embodied intelligence scenarios according to claim 9, characterized in that, The method further includes: For different application scenarios of federated embodied intelligence, different re-entanglement coefficients λ are set to generate enhanced training sample data with different proportions of bias features.

Images