Robot control method and system based on bidirectional selection hybrid expert mechanism, terminal and storage medium

Abstract

The invention discloses a robot control method and system based on a bidirectional selection hybrid expert mechanism, a terminal and a storage medium. The method comprises the following steps: acquiring a task language instruction and a visual perception image of a target local data set; performing scene object classification according to each task language instruction, and segmenting each visual perception image to obtain a plurality of image tokens; performing token score calculation on the plurality of image tokens, performing expert activation score calculation according to the initial expert, and determining a target expert; obtaining an expert selection vector in each target client, and carrying out model parameter aggregation processing to obtain a federal visual language action model; and obtaining a target task instruction corresponding to the current task, and inputting the target task instruction to the federated visual language action model to enable the target robot to execute the current task. According to the method, federal learning and a two-way selection hybrid expert mechanism are combined, so that the privacy data of the user is protected, and the operation capability of the robot is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a robot control method, system, terminal and computer-readable storage medium based on a two-way selection hybrid expert mechanism. Background Art

[0002] Vision-Language-Action (VLA) models integrate visual perception, language understanding, and robotic control capabilities, significantly improving robotic manipulation capabilities by enabling robots to understand natural language commands and perform complex tasks. However, training these models requires large indoor datasets that capture diverse, user-specific environments, which can lead to privacy concerns.

[0003] Federated learning (FL) provides an effective privacy-preserving solution for training vision-language-action models. Unlike traditional centralized training, federated learning enables distributed model training across multiple clients without transmitting raw data. However, due to the significant task heterogeneity exhibited by robotic manipulation tasks across different clients, typical federated learning methods can only simply aggregate models between tasks, considering only the shared synergistic effects while ignoring the fundamental differences between different tasks. As a result, existing federated learning methods are not suitable for vision-language-action learning. Although many studies have attempted to address this issue through merge-split training or decoupled model aggregation, these methods are still limited to unimodal settings and assume that the input modalities between tasks are unified. The VLA model operates in a multimodal environment and requires joint processing of visual observations, language commands, and robot actions, which significantly increases the complexity of federated training.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a robot control method, system, terminal and computer-readable storage medium based on a two-way selection hybrid expert mechanism, aiming to solve the problem that when using the federated learning method in the existing technology to train the vision-language-action model, it can only simply aggregate the model between tasks, only considering the shared synergistic effect, but ignoring the fundamental differences between different tasks, resulting in the existing federated learning method being unsuitable for the robot's vision-language-action learning, resulting in low robot operation ability.

[0006] To achieve the above object, the present invention provides a robot control method based on a two-way selection hybrid expert mechanism, the robot control method based on a two-way selection hybrid expert mechanism comprising the following steps:

[0007] Acquire target local data sets from multiple target clients, and acquire task language instructions and visual perception images from each of the target local data sets;

[0008] Performing scene object classification processing according to each of the task language instructions, and segmenting each of the visual perception images to obtain a plurality of image tokens;

[0009] Calculating token scores for the plurality of image tokens to determine an initial expert, and calculating expert activation scores based on the initial expert to determine a target expert;

[0010] Obtaining the expert selection vector of the target expert in each target client, and performing model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model;

[0011] Obtain a target task instruction corresponding to the current task, and input the target task instruction into the federated visual language action model so that the target robot performs the current task.

[0012] Optionally, the robot control method based on a two-way selection hybrid expert mechanism, wherein the step of obtaining target local data sets from multiple target clients and obtaining task language instructions and visual perception images from each target local data set, specifically includes:

[0013] Determining a preset number of local tasks, and performing data collection and processing according to the preset number of local tasks to obtain a preset number of local data sets;

[0014] Determining a preset number of target clients, and allocating the preset number of local tasks and the preset number of local data sets to the preset number of target clients, wherein the local tasks and the local data sets correspond to the target clients in a one-to-one manner;

[0015] Target local data sets allocated to the plurality of target clients are obtained, and task language instructions and visual perception images in each target local data set are extracted.

[0016] Optionally, the robot control method based on the two-way selection hybrid expert mechanism, wherein the scene object classification processing according to each of the task language instructions specifically includes:

[0017] Using named entity recognition to extract target objects in each of the task language instructions, and using a YOLOv8 model to extract foreground objects and background objects in each of the visual perception images;

[0018] Determining a pre-trained CLIP model, and calculating a first cosine similarity between the target object and the foreground object and the background object using the pre-trained CLIP model;

[0019] Classifying the scene objects in the visual perception image according to the first cosine similarity to obtain a plurality of target scene objects, wherein the target scene objects include a target object, surrounding objects, and background objects;

[0020] Encoding is performed on the plurality of target scene objects to obtain text embedding.

[0021] Optionally, the robot control method based on a two-way selection hybrid expert mechanism, wherein the segmentation processing of each visual perception image to obtain a plurality of image tokens specifically includes:

[0022] Using a visual encoder to segment each of the visually perceived images to obtain a plurality of initial image tokens;

[0023] Assigning a plurality of the initial image tokens to each of the target scene objects, and calculating a second cosine similarity between the assigned initial image tokens and the text embedding in each of the target scene objects;

[0024] A plurality of image tokens are extracted from the initial image tokens of the target scene object in descending order of the second cosine similarity.

[0025] Optionally, the robot control method based on the two-way selection hybrid expert mechanism, wherein the token score calculation of the plurality of image tokens to determine the initial expert specifically includes:

[0026] Setting a token-side gating module, and calculating token scores of a plurality of image tokens in each target scene object through the token-side gating module to obtain a token score result;

[0027] A plurality of preset experts are determined, and an initial expert from the plurality of preset experts is selected according to the token score result.

[0028] Optionally, the robot control method based on the two-way selection hybrid expert mechanism, wherein the step of calculating the expert activation score based on the initial expert to determine the target expert specifically includes:

[0029] Setting an expert-side gating module, and calculating the expert activation score of the initial expert on the token scoring result through the expert-side gating module to obtain an expert activation score result;

[0030] A target image token is selected from the plurality of image tokens according to the expert activation score result, and a target expert corresponding to the target image token is determined.

[0031] Optionally, the robot control method based on the two-way selection hybrid expert mechanism, wherein the step of obtaining the expert selection vector of the target expert in each target client and performing model parameter aggregation processing based on all the expert selection vectors to obtain the federated visual language action model, specifically includes:

[0032] Obtaining the backbone module of the target expert selected in each target client, and extracting the expert selection vector corresponding to the backbone module;

[0033] Calculating the similarity of the expert selection vectors between the target clients to obtain a similarity result, and obtaining an aggregation weight of each target client according to the similarity result;

[0034] A federated vision-language-action model is constructed based on the aggregated weights of each target client.

[0035] In addition, to achieve the above-mentioned object, the present invention further provides a robot control system based on a two-way selection hybrid expert mechanism, wherein the robot control system based on the two-way selection hybrid expert mechanism includes:

[0036] A local data set allocation module is used to obtain target local data sets from multiple target clients and obtain task language instructions and visual perception images from each target local data set;

[0037] An image token generation module is used to perform scene object classification processing according to each of the task language instructions and segment each of the visual perception images to obtain multiple image tokens;

[0038] a target expert determination module, configured to calculate token scores for the plurality of image tokens to determine an initial expert, and calculate expert activation scores based on the initial expert to determine a target expert;

[0039] a model parameter aggregation module, configured to obtain the expert selection vector of the target expert in each target client, and perform model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model;

[0040] The current task execution module is used to obtain the target task instruction corresponding to the current task and input the target task instruction into the federated visual language action model so that the target robot performs the current task.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a robot control program based on a two-way selection hybrid expert mechanism stored in the memory and runnable on the processor, and when the robot control program based on a two-way selection hybrid expert mechanism is executed by the processor, the steps of the robot control method based on a two-way selection hybrid expert mechanism as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot control program based on a two-way selection hybrid expert mechanism, and when the robot control program based on a two-way selection hybrid expert mechanism is executed by a processor, the steps of the robot control method based on a two-way selection hybrid expert mechanism as described above are implemented.

[0043] In the present invention, target local data sets in multiple target clients are obtained, and task language instructions and visual perception images in each of the target local data sets are obtained; scene object classification processing is performed according to each of the task language instructions, and each of the visual perception images is segmented to obtain multiple image tokens; token scores are calculated for the multiple image tokens to determine the initial expert, and expert activation scores are calculated based on the initial expert to determine the target expert; expert selection vectors of the target experts in each of the target clients are obtained, and model parameter aggregation processing is performed based on all the expert selection vectors to obtain a federated visual language action model; the target task instruction corresponding to the current task is obtained, and the target task instruction is input into the federated visual language action model to enable the target robot to perform the current task. The present invention uses the target local data set on each target client to perform scene object classification and segmentation processing to obtain image tokens, and introduces a two-way selection hybrid expert mechanism to determine the target expert corresponding to each client according to the image token. The server side receives the expert selection vector of the target expert selected by each client, thereby constructing a federated visual language action model. It can effectively improve the training effect of federated learning on the visual language action model, which not only ensures that the user's privacy data is not leaked, but also enables the target robot to smoothly perform various tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a preferred embodiment of the robot control method based on a two-way selection hybrid expert mechanism of the present invention;

[0045] Figure 2 Schematic diagram of the architecture of the federated visual language action model of a preferred embodiment of the robot control method based on the two-way selection hybrid expert mechanism of the present invention;

[0046] Figure 3 Schematic diagram of a command-oriented scene parsing module of a preferred embodiment of the robot control method based on a two-way selection hybrid expert mechanism of the present invention;

[0047] Figure 4 Schematic diagram of a hybrid expert module for bidirectional gating selection in a preferred embodiment of a robot control method based on a bidirectional selection hybrid expert mechanism of the present invention;

[0048] Figure 5 1 is a structural diagram of a preferred embodiment of a robot control system based on a two-way selection hybrid expert mechanism of the present invention;

[0049] Figure 6 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] Vision-Language-Action (VLA) models integrate visual perception, language understanding, and robotic control, significantly improving robotic manipulation capabilities by enabling robots to understand natural language commands and perform complex tasks. However, training these models requires large indoor datasets that capture diverse, user-specific environments, raising serious data privacy concerns.

[0052] Federated learning (FL) provides an effective privacy-preserving solution for training vision-language-action models. Unlike traditional centralized training, federated learning enables distributed model training across multiple clients without transmitting raw data. However, due to the significant task heterogeneity exhibited by robotic manipulation tasks across different clients, typical federated learning methods can only simply aggregate models between tasks, considering only the shared synergistic effects while ignoring the fundamental differences between different tasks. This makes existing federated learning methods unsuitable for vision-language-action learning. Many studies have attempted to address this issue through merge-split training or decoupled model aggregation, but these methods are still limited to unimodal settings and assume that the input modalities between tasks are unified. The VLA model operates in a multimodal environment and requires joint processing of visual observations, language commands, and robot actions, which significantly increases the complexity of federated training.

[0053] To solve the above problems, the present invention proposes a federated vision-language-action learning framework with dual-gated MoE (Mixture of Experts) based on a multimodal environment, which aims to protect privacy while maintaining task completion efficiency. Through this technology, users can train personalized models in local devices (i.e., the target client in the present invention) and aggregate global knowledge through a federated learning framework without exposing the original data, thereby improving the generalization ability and task adaptability of the model. The present invention can be widely used in various embodied intelligent scenarios such as service robots, smart industry, medical rehabilitation, virtual assistants, etc., and provide industrial enterprises, scientific research institutions and end users with a multimodal perception decision-making solution that combines privacy security and task intelligence. Specifically, the present invention can achieve the following goals:

[0054] 1. Implement privacy-preserving task modeling under multimodal perception: Through a federated learning architecture, vision-language-motion data is distributed across clients for local modeling. Only parameters or expert distributions are uploaded, achieving the privacy-preserving goal of not leaking image, semantic, or motion information during training.

[0055] 2. Achieving high efficiency across different tasks: Introducing a dual-gated hybrid expert mechanism, where self-aware experts independently decide their participation and optimize computational efficiency while maintaining task performance through bidirectional token selection.

[0056] The robot control method based on the two-way selection hybrid expert mechanism described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the robot control method based on the two-way selection hybrid expert mechanism includes the following steps:

[0057] Step S10: Acquire target local data sets from multiple target clients, and acquire task language instructions and visual perception images from each target local data set.

[0058] This paper proposes a robot control method based on a two-way selection hybrid expert mechanism. Through the steps of acquiring and processing data sets, determining target experts, obtaining a model through aggregation processing, and inputting instructions into the model to execute tasks, a federated vision-language-action learning framework that introduces a dual-gated hybrid expert mechanism is proposed. This framework not only protects privacy but also takes into account the efficiency of task completion.

[0059] The first step is the data set preparation process: the present invention conducts experiments in both simulated environments and real-world scenarios. The simulated environment uses the MuJoCo engine and the Meta-World benchmark, and the real environment uses the UR3 robotic arm and the RealSense RGB-D camera. In the simulation experiment, data including tasks such as locking doors, cleaning, opening windows, and closing drawers were collected. Each task contains about 200 trajectories, and each trajectory contains 40 to 100 frames of images and language instructions. Each task is assigned to a federated client to simulate heterogeneous data distribution. In the real environment, expert teaching was used to collect demonstration data for tasks such as "opening drawers" and "sorting medicines", and RGB images, camera poses and language instructions were recorded. A total of about 50 demonstration trajectories were collected. The image resolution is 1280×720 and the frequency is 30Hz. The task data is stored in different client devices for building a federated learning environment.

[0060] Specifically, a preset number of local tasks are determined, and data collection and processing are performed based on the preset number of local tasks to obtain a preset number of local data sets. A preset number of target clients are determined, and the preset number of local tasks and the preset number of local data sets are assigned to the preset number of target clients, wherein the local tasks and the local data sets correspond one-to-one to the target clients. Target local data sets assigned to the plurality of target clients are obtained, and task language instructions and visual perception images in each target local data set are extracted.

[0061] This invention's technical approach is centered around the core concepts of multimodal perception guidance, dynamic expert activation, and collaborative optimization of federated training. It aims to build a federated multimodal learning system that strikes a balance between task performance and privacy protection. The proposed model supports end-to-end training and deployment, enabling robust motion prediction and efficient task execution in a variety of embodied intelligence scenarios.

[0062] Regarding the selection of the test platform: This paper uses the MuJoCo engine to simulate the tasks in the Meta-World robot operation benchmark. All tasks are performed using a Sawyer robot with a parallel gripper at the center of a dining table. Visual data (referring to image data collected in the simulated environment) is collected by three RGB cameras with a fixed resolution of 128×128, providing corner perspectives, top perspectives, and shoulder perspectives. For collision detection and dynamic simulation, the official physics engine is used to ensure accurate robot interaction in the simulated environment.

[0063] For the federated training of vision-language-action models under robot operation, this paper aims to train the vision-language-action model in the federated learning (FL) framework, while ensuring privacy and improving the generalization ability of tasks between different clients. Specifically, given N clients C = {C1, C2, ..., C N ,}, N local tasks T={T1,T2,…,T N ,} and N local data sets (i.e., data sets collected from the simulation environment and the real environment) D={D1,D2,…,D N ,}, each client C i Both with a local task T i and a corresponding local dataset D i Wherein, the local data set D i Contains visual perception images o i , task language instructions i , robot proprioception i and the action sequence a performed by the robot i .

[0064] Wherein, the local dataset D i The expression is:

[0065] D i ={(o i ,l i ,s i ,a i ) j |j=1,2,…,M};

[0066] Where M is the local dataset D i The number of samples, j is the local dataset D i The index of each data in .

[0067] To train the federated Vision-Language-Action (VLA) model θ G , the overall federated optimization goal can be expressed as:

[0068]

[0069] Among them, L i is the local dataset D of each client model i The loss function calculated above, f i represents the action sequence predicted by the local client model, w i Represents the weight assigned to each client during the aggregation process.

[0070] like Figure 2As shown in the figure, it is the FedVLA framework. Each client trains its own model locally and communicates with the server in rounds (uploading, sending and updating parameters) until the model converges.

[0071] Step S20: classify the scene objects according to each of the task language instructions, and segment each of the visual perception images to obtain multiple image tokens.

[0072] Algorithm Design in the Present Invention: This paper proposes a Federated Visual Language Action Learning (FVLA) framework for training VLA models on distributed devices while protecting data privacy. The framework includes three key designs: 1. Scene Parsing Object Representation Module: Guided by the task language, the module performs object-level parsing on observed images (i.e., datasets collected in simulated environments and real-world scenes), extracts target objects, background, and contextual information from the images, and encodes them into representational features. 2. Dual-Gated Expert Mixture Mechanism: Dynamically selects a set of candidate experts for each object feature (i.e., the features obtained by the first encoding step). Each input token first passes through a soft router to obtain a set of scores. Each score corresponds to the degree of preference for each expert for that feature token, indicating which experts it prefers. Each expert then self-perceives whether to process the feature. (The expert also has a threshold for participating in training, which is compared with the score calculated for the token above. When the expert threshold is less than the token score, the expert processes the feature token.) This saves computing resources and enables selective activation. 3. Expert-oriented aggregation strategy: When the federated server aggregates model parameters (aggregation here refers to averaging the parameters of each client), it considers the similarity of activated experts in the client and assigns a higher aggregation weight, thereby improving the generalization ability and transmission efficiency between heterogeneous clients.

[0073] The present invention sets a loss function: during the model (i.e., VLA model) training process, the present invention uses language-vision fusion input as a condition (i.e., the input data is in the form of language instructions (text) and visual images) to predict the actions that the robot should perform under the current environmental state, and uses Huber loss as the loss function to measure and back-propagate the deviation between the predicted action and the actual action.

[0074] Specifically, a named entity recognition method is used to extract a target object from each of the task language instructions, and a YOLOv8 model is used to extract foreground objects and background objects from each of the visual perception images. A pre-trained CLIP model is determined, and a first cosine similarity between the target object and the foreground objects and background objects is calculated using the pre-trained CLIP model. Scene objects in the visual perception images are classified based on the first cosine similarity to obtain a plurality of target scene objects, wherein the target scene objects include the target object, surrounding objects, and background objects. The plurality of target scene objects are encoded to obtain a text embedding.

[0075] First, to ensure the robot focuses on task-relevant information while maintaining contextual awareness to avoid collisions, objects in the scene are decoupled according to the specific robot task. To this end, a command-oriented scene parsing module is proposed to represent objects in the task scene.

[0076] like Figure 3 As shown in the figure, it is an instruction-oriented scene parsing module. Given an instruction and an image, named entity recognition is first used to extract the target object from the instruction, and YOLOv8 is used to detect and classify the foreground objects and background objects in the image. Based on the cosine similarity between the instruction and the object name measured by the pre-trained CLIP model, the scene objects are divided into three categories, following the following matching rules: 1. Target objects (TOs): objects that are explicitly mentioned in the instruction and directly involved in the action. 2. Surrounding objects (SOs): foreground objects that are not in TOs, providing spatial and contextual clues. 3. Background objects (BOs): static environmental elements that ensure scene consistency but have no direct impact on the task. The grouped objects are then encoded using the CLIP model. The text encoder H maps the object names of TOs, SOs, and BOs to text embeddings H(TOs)∈R t×D 、H(BOs)∈R t×D and H(SOs)∈R t×D , where t is the number of objects and D is the corresponding feature dimension.

[0077] Furthermore, a visual encoder is used to segment each of the visually perceived images to obtain a plurality of initial image tokens. The plurality of initial image tokens are assigned to each of the target scene objects, and a second cosine similarity is calculated between the initial image token assigned to each target scene object and the text embedding. The plurality of image tokens are extracted from the initial image tokens of the target scene objects in descending order of the second cosine similarity.

[0078] The visual encoder P segments the entire image into P(X)∈R t×D, obtain multiple initial image tokens, calculate the cosine similarity between the multiple initial image tokens and the text embedding, assign appropriate initial image tokens to object groups (i.e., target object, surrounding objects, background objects), and select the first 8 image tokens for each group of objects.

[0079] To enhance the feature representation within a group, each group is processed through a hybrid expert module, generating refined image tokens that are embedded within them. Finally, the enhanced tokens are concatenated with the remaining tokens for subsequent processing. This module enables the model to focus on key objects, thereby improving its operational capabilities for specific tasks. A token first generates a set of candidate expert scores based on its own features and a routing mechanism, indicating which experts it prefers to be processed by. This is then combined with the selection preferences of the previous layer to generate the expert selection distribution of the current layer, from which the top-scoring experts are selected as the candidate set. These experts then independently decide whether to accept the token based on their own judgment. Ultimately, only those experts recognized by both bidirectional gating systems are activated and participate in the output calculation.

[0080] Step S30: Calculate token scores for the plurality of image tokens to determine an initial expert, and calculate expert activation scores based on the initial expert to determine a target expert.

[0081] The present invention uses a dual-gated hybrid expert (DGMoE) for adaptive routing selection. Unlike ordinary hybrid experts, which assign a fixed number of experts to each token, the dual-gated hybrid expert mechanism can automatically determine (by calculating the cosine similarity and selecting the first 8 tokens) the acceptance or rejection instructions for the token allocation under the action of the self-perceiving expert (the expert side is set with a threshold, and the token will get a score when selecting an expert. When this score is greater than the threshold, it means that the expert accepts to process the token, otherwise it is rejected).

[0082] like Figure 4 As shown, each DGMoE layer consists of K self-aware experts E = {E1, E2, ..., E K In order to realize automatic token-driven expert selection, two gating mechanisms are introduced: token-side gating mechanism G t and expert-side gating mechanism G e .

[0083] Specifically, a token-side gating module is provided, and token scores of a plurality of image tokens in each target scene object are calculated by the token-side gating module to obtain a token score result. A plurality of preset experts are determined, and an initial expert from the plurality of preset experts is selected based on the token score result.

[0084] For token-side gating: G tAs a token-side gating module, it consists of a soft routing module and a gated residual module. Soft routing (soft routing is a network structure consisting of an MLP layer and a softmax layer; Figure 4 The input is the feature vector output by the previous layer) first estimates the scores of the hidden inputs, which indicate the token’s preference for different experts. Then the estimated scores of the previous DGMoE layer are compared with the current DGMoE layer (i.e. Figure 4 The token is combined with the scores of the layer where the gated residual module is located in the previous layer to enable it to inherit the selection prior of the most suitable expert in the previous layer.

[0085] Given an input token x, the token-side gate G t The score calculated by combining the soft routing output and the gated residual module at the h layer is:

[0086]

[0087] in, G is the token side gate t The calculated score at level j is, is the trainable weight matrix of soft routing, is a trainable transformation matrix that aggregates scores from the previous and current layers to ensure smooth transition of expertise between layers. ( x) Apply the softmax function to obtain the token to expert selection score s t (x), this score determines the probability distribution over the available experts.

[0088] Furthermore, an expert-side gating module is provided, and the expert-side gating module calculates the expert activation score of the initial expert for the token scoring result to obtain an expert activation score result. A target image token is selected from the plurality of image tokens based on the expert activation score result, and a target expert corresponding to the target image token is determined.

[0089] For expert-side gating: expert-side gating module G e It can automatically accept or reject tokens based on their acceptance threshold (less than or equal to the threshold). The predicted scores are filtered through a learnable threshold, and only experts that match the most relevant tokens are activated.

[0090] On the self-sensing expert side, the expert side gates G e The token score is evaluated and the final expert activation score s is determined by the following calculation method e (x):

[0091] s e (x)=sign(s t (x)-λWe );

[0092] Among them, W e represents a set of trainable, self-aware parameters corresponding to each expert's gating value, and λ represents a scaling factor used to adjust the gating threshold. In this paper, to balance expert selection, λ is preferably set to 0.5, ensuring neither overactivation of experts nor overly strict filtering of tokens. Then, combining the dual gating mechanism on the token side and the expert side, the expert selection function is defined as follows:

[0093]

[0094] The final output token y of the DGMoE layer can be calculated as the weighted sum of the selected experts’ outputs:

[0095]

[0096] Among them, g i (x) is the final decision flag that combines the two gating modules on the token side and the expert side. Its value is [0, 1], which represents whether the expert is selected and whether to participate in the calculation. i (x) is the expert’s network structure parameter.

[0097] Step S40: Obtain the expert selection vector of the target expert in each target client, and perform model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model.

[0098] Regarding the network training and testing process: In both simulation and real-world embodied tasks, the present invention uses a federated learning framework to train the FedVLA model. Each task corresponds to an independent client. The client only performs model training locally and does not upload the original data. The server is only responsible for aggregating model updates from each client, thereby achieving improved cross-task generalization capabilities while ensuring data privacy.

[0099] In order to evaluate the performance of the present invention in simulated and real environments, the validation set loss (i.e., ValidationLoss, which can be calculated by the Huber loss formula) and the task success rate (i.e., Task Success Rate, the calculation process of the task success rate is: the completion of the task is recorded as 1, the failure of the task is recorded as 0, each task is tested 15 times, and the percentage of the number of successes is taken) are selected as core evaluation indicators. Among them, the validation loss is used to measure the degree of fit of the model to the target action, and the task success rate is used to measure the ability of the model to complete the task goal in actual operation. The lower the validation loss and the higher the success rate, the better the model's task understanding ability and action prediction accuracy under multimodal conditions, and the stronger the overall system performance.

[0100] Specifically, a backbone module of the target expert selected in each target client is obtained, and the expert selection vector corresponding to the backbone module is extracted. The similarity of the expert selection vectors between the target clients is calculated to obtain a similarity result, and an aggregation weight for each target client is obtained based on the similarity result. A federated visual language action model is constructed based on the aggregation weight for each target client.

[0101] Knowledge aggregation on the server side in federated learning: To alleviate the performance degradation caused by task heterogeneity, an expert-driven aggregation (EDA) strategy is proposed, which aggregates model parameters based on the similarity of expert selections between different clients.

[0102] like Figure 2 As shown in Figure 2, only the backbone module is selectively aggregated, while the stem module and the head module keep their personalized settings and do not participate in the aggregation process. Given a backbone module with L layers, each layer contains K experts, and each client C i In each round, DGMoE records an expert selection matrix V i ∈R L×K , where the element V (l,k) Indicates the number of times expert k is activated in layer l, then extract client C i The expert selection vector of the lth layer is:

[0103]

[0104] The similarity between clients is calculated based on the expert selection vector to obtain client C i and C j The similarity at the jth layer is as follows:

[0105]

[0106] Using this similarity, for each client C in layer l, i Determine the aggregation weight w i,j :

[0107]

[0108] Where C is the total number of clients.

[0109] This weighting strategy ensures that experts select clients with higher similarity to contribute more to each other's model updates, thus achieving more effective and task-aligned aggregation among task-heterogeneous clients.

[0110] Step S50: Obtain a target task instruction corresponding to the current task, and input the target task instruction into the federated visual language action model to enable the target robot to perform the current task.

[0111] In the specific implementation process, the experiments in this invention are implemented based on the open source PyTorch deep learning framework. The present invention trains and tests the proposed FedVLA model in a simulation environment and a real robot platform to verify its effectiveness and versatility in multimodal embodied tasks. In the simulation environment, the present invention builds a multi-task embodied interaction scenario based on the MuJoCo platform and performs federated learning training. The training process is performed for a total of 1000 rounds. In each round, all clients perform 5 local training cycles based on their own local task data. The batch size is 256 in each cycle, and the Adam optimizer is used. The initial learning rate is set to 5×10 -6 A fixed step-size strategy is used during training to accelerate global convergence while maintaining local model stability. After each round of local training, the client uploads the weight differences of the model's backbone modules to the server, which performs expert-driven aggregation operations to form a shared model.

[0112] In the real-world experiment, the present invention was deployed on the embodied interactive platform of the UR3 six-degree-of-freedom robotic arm, and combined with the multimodal perception system to collect images, language commands, and action execution data in the actual environment. Similar to the simulation experiment, the number of training rounds was also 1000 rounds, and the client performed 5 local updates in each round. The learning rate was set to 2×10 -5 , also using the Adam optimizer. The model backbone utilizes a pretrained HPT (Holistic Pretrained Transformer) model, fine-tuned using the FedVLA framework. During training, the text and image encoders are frozen, and only task-related parts (such as the instruction parsing module and the hybrid expert architecture) are fine-tuned and optimized. All experiments were conducted on a high-performance workstation equipped with two NVIDIA GeForce RTX4090Ti GPUs, a 48-core Intel Xeon CPU, and 128GB of memory. Training was iteratively performed until the loss function converged.

[0113] Beneficial effects of the present invention: The present invention fully considers the synergistic relationship between multimodal perception fusion, privacy protection requirements and embodied task decision-making, and proposes a vision-language-action integrated modeling method based on federated learning. This method introduces a dual-gating mixture of experts mechanism to achieve cross-client knowledge sharing and generalized modeling while ensuring data locality, and is suitable for multi-scenario, multi-task embodied intelligent systems. Experimental results show that the present invention can still achieve action prediction performance comparable to that of a centralized training model without uploading the original data, and has strong task transfer capabilities and model robustness.

[0114] The beneficial effects of the present invention can be summarized as follows:

[0115] 1. Privacy protection: Using a federated learning framework, each client only performs local training and parameter uploads, and the server does not access the original image, language, or motion data, effectively reducing the risk of data leakage. It is suitable for intelligent interactive application scenarios that require privacy isolation.

[0116] 2. Multimodal collaboration and generalization: Through language-guided object aggregation and a dual-gated expert activation mechanism, the model can dynamically perceive task semantics and environmental characteristics, automatically select module aggregation, and significantly improve generalization capabilities under multiple tasks.

[0117] 3. Computing resource utilization: Compared with the traditional hybrid expert structure, the self-perceiving expert sparse activation strategy introduced in this invention can reduce unnecessary expert calculations, lower system resource overhead, and show better energy efficiency on edge devices.

[0118] Further, if Figure 5 As shown, based on the above-mentioned robot control method based on the two-way selection hybrid expert mechanism, the present invention also provides a robot control system based on the two-way selection hybrid expert mechanism, wherein the robot control system based on the two-way selection hybrid expert mechanism includes:

[0119] The local data set allocation module 51 is configured to obtain target local data sets from multiple target clients, and obtain task language instructions and visual perception images from each target local data set;

[0120] An image token generation module 52 is configured to perform scene object classification processing according to each of the task language instructions and segment each of the visual perception images to obtain a plurality of image tokens;

[0121] a target expert determination module 53, configured to calculate token scores for the plurality of image tokens to determine an initial expert, and calculate expert activation scores based on the initial expert to determine a target expert;

[0122] A model parameter aggregation module 54 is configured to obtain the expert selection vector of the target expert in each target client, and perform model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model;

[0123] The current task execution module 55 is used to obtain the target task instruction corresponding to the current task and input the target task instruction into the federated visual language action model to enable the target robot to perform the current task.

[0124] Further, if Figure 6As shown, based on the above-mentioned robot control method and system based on the two-way selection hybrid expert mechanism, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 6 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0125] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a robot control program 40 based on a two-way selection hybrid expert mechanism is stored on the memory 20, and the robot control program 40 based on a two-way selection hybrid expert mechanism can be executed by the processor 10, thereby realizing the robot control method based on a two-way selection hybrid expert mechanism in the present application.

[0126] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the robot control method based on the two-way selection hybrid expert mechanism.

[0127] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display 30 is used to display information on the terminal and to display a visual user interface.

[0128] In one embodiment, when the processor 10 executes the robot control program 40 based on the two-way selection hybrid expert mechanism in the memory 20 , the steps of the robot control method based on the two-way selection hybrid expert mechanism are implemented.

[0129] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot control program based on a two-way selection hybrid expert mechanism, and when the robot control program based on a two-way selection hybrid expert mechanism is executed by a processor, the steps of the robot control method based on a two-way selection hybrid expert mechanism as described above are implemented.

[0130] In summary, the present invention provides a robot control method, system and terminal based on a two-way selection hybrid expert mechanism, the method comprising: obtaining target local data sets from multiple target clients, and obtaining task language instructions and visual perception images in each of the target local data sets; performing scene object classification processing according to each of the task language instructions, and performing segmentation processing on each of the visual perception images to obtain multiple image tokens; performing token score calculation on the multiple image tokens to determine the initial expert, and performing expert activation score calculation based on the initial expert to determine the target expert; obtaining the expert selection vector of the target expert in each of the target clients, and performing model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model; obtaining the target task instruction corresponding to the current task, and inputting the target task instruction into the federated visual language action model to enable the target robot to perform the current task. The present invention uses the target local data set on each target client to perform scene object classification and segmentation processing to obtain image tokens, and introduces a two-way selection hybrid expert mechanism to determine the target expert corresponding to each client according to the image token. The server side receives the expert selection vector of the target expert selected by each client, thereby constructing a federated visual language action model. It can effectively improve the training effect of federated learning on the visual language action model, which not only ensures that the user's privacy data is not leaked, but also enables the target robot to smoothly perform various tasks.

[0131] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0132] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0133] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A robot control method based on a two-way selection hybrid expert mechanism, characterized in that: The robot control method based on the two-way selection hybrid expert mechanism includes: Acquire target local data sets from multiple target clients, and acquire task language instructions and visual perception images from each of the target local data sets; Performing scene object classification processing according to each of the task language instructions, and segmenting each of the visual perception images to obtain a plurality of image tokens; Calculating token scores for the plurality of image tokens to determine an initial expert, and calculating expert activation scores based on the initial expert to determine a target expert; Obtaining the expert selection vector of the target expert in each target client, and performing model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model; Obtain a target task instruction corresponding to the current task, and input the target task instruction into the federated visual language action model so that the target robot performs the current task.

2. The robot control method based on a two-way selection hybrid expert mechanism according to claim 1 is characterized in that: The step of obtaining target local data sets from multiple target clients and obtaining task language instructions and visual perception images from each target local data set specifically includes: Determining a preset number of local tasks, and performing data collection and processing according to the preset number of local tasks to obtain a preset number of local data sets; Determining a preset number of target clients, and allocating the preset number of local tasks and the preset number of local data sets to the preset number of target clients, wherein the local tasks and the local data sets correspond to the target clients in a one-to-one manner; Target local data sets allocated to the plurality of target clients are obtained, and task language instructions and visual perception images in each target local data set are extracted.

3. The robot control method based on a two-way selection hybrid expert mechanism according to claim 1 is characterized in that: The performing scene object classification processing according to each of the task language instructions specifically includes: Using named entity recognition to extract target objects in each of the task language instructions, and using a YOLOv8 model to extract foreground objects and background objects in each of the visual perception images; Determining a pre-trained CLIP model, and calculating a first cosine similarity between the target object and the foreground object and the background object using the pre-trained CLIP model; Classifying the scene objects in the visual perception image according to the first cosine similarity to obtain a plurality of target scene objects, wherein the target scene objects include a target object, surrounding objects, and background objects; Encoding is performed on the plurality of target scene objects to obtain text embedding.

4. The robot control method based on a two-way selection hybrid expert mechanism according to claim 3 is characterized in that: The segmentation process of each visual perception image to obtain a plurality of image tokens specifically includes: Using a visual encoder to segment each of the visually perceived images to obtain a plurality of initial image tokens; Assigning a plurality of the initial image tokens to each of the target scene objects, and calculating a second cosine similarity between the assigned initial image tokens and the text embedding in each of the target scene objects; A plurality of image tokens are extracted from the initial image tokens of the target scene object in descending order of the second cosine similarity.

5. The robot control method based on a two-way selection hybrid expert mechanism according to claim 4 is characterized in that: Calculating token scores for the plurality of image tokens to determine an initial expert specifically includes: Setting a token-side gating module, and calculating token scores of a plurality of image tokens in each target scene object through the token-side gating module to obtain a token score result; A plurality of preset experts are determined, and an initial expert from the plurality of preset experts is selected according to the token score result.

6. The robot control method based on a two-way selection hybrid expert mechanism according to claim 5, characterized in that: Calculating the expert activation score based on the initial expert to determine the target expert specifically includes: Setting an expert-side gating module, and calculating the expert activation score of the initial expert on the token scoring result through the expert-side gating module to obtain an expert activation score result; A target image token is selected from the plurality of image tokens according to the expert activation score result, and a target expert corresponding to the target image token is determined.

7. The robot control method based on a two-way selection hybrid expert mechanism according to claim 5, characterized in that: The obtaining of the expert selection vector of the target expert in each target client and performing model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model specifically includes: Obtaining the backbone module of the target expert selected in each target client, and extracting the expert selection vector corresponding to the backbone module; Calculating the similarity of the expert selection vectors between the target clients to obtain a similarity result, and obtaining an aggregation weight of each target client according to the similarity result; A federated vision-language-action model is constructed based on the aggregated weights of each target client.

8. A robot control system based on a two-way selection hybrid expert mechanism, characterized in that: The robot control system based on the two-way selection hybrid expert mechanism includes: A local data set allocation module is used to obtain target local data sets from multiple target clients and obtain task language instructions and visual perception images from each target local data set; An image token generation module is used to perform scene object classification processing according to each of the task language instructions and segment each of the visual perception images to obtain multiple image tokens; a target expert determination module, configured to calculate token scores for the plurality of image tokens to determine an initial expert, and calculate expert activation scores based on the initial expert to determine a target expert; a model parameter aggregation module, configured to obtain the expert selection vector of the target expert in each target client, and perform model parameter aggregation processing based on all the expert selection vectors to obtain a federated visual language action model; The current task execution module is used to obtain the target task instruction corresponding to the current task and input the target task instruction into the federated visual language action model so that the target robot performs the current task.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a robot control program based on a two-way selection hybrid expert mechanism stored in the memory and runnable on the processor. When the robot control program based on a two-way selection hybrid expert mechanism is executed by the processor, the steps of the robot control method based on a two-way selection hybrid expert mechanism as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a robot control program based on a two-way selection hybrid expert mechanism. When the robot control program based on a two-way selection hybrid expert mechanism is executed by a processor, the steps of the robot control method based on a two-way selection hybrid expert mechanism as described in any one of claims 1 to 7 are implemented.

Images