Robot multi-task migration method and device based on virtual and real big data and medium

Abstract

The invention relates to a robot multi-task migration method and device based on virtual and real big data and a medium. In model training, virtual data and real data are collected, a migration evaluation model is constructed, virtual and real data distribution differences are quantized and reduced through a domain alignment loss function, and enhanced virtual data is generated. The method comprises the following steps: dynamically determining a virtual and real data mixing ratio according to the requirements of subtasks in dimensions such as environmental complexity and sensor precision, constructing an integrated training data set, and inputting a Transform architecture for pre-training to obtain an initial multi-modal large model; a shared feature space is constructed under multi-task setting, adversarial training is introduced, and cross-domain feature alignment is realized by jointly optimizing multi-task loss and domain adaptation loss. And performing fine adjustment on the model by utilizing the aligned multi-task virtual and real data to obtain a vision-language-action large model with cross-task and cross-scene migration capability. Compared with the prior art, the method has the advantages of high adaptability, high efficiency, strong generalization ability and the like.

Description

Technical Field

[0001] This invention relates to the field of robot intelligence, and in particular to a method, device and medium for robot multi-task migration based on virtual and real big data. Background Technology

[0002] With the continuous development of robotics technology, virtual and real environments are becoming complementary in robot training and testing. Virtual environments, through simulation platforms, can generate large amounts of efficient, low-cost, and diverse training data, offering advantages such as scene controllability, flexibility, and large-scale data acquisition. However, virtual data often struggles to fully reproduce the complex physical characteristics of real environments, such as sensor noise, environmental uncertainties, and nonlinear interaction features in dynamic changes, leading to significant distributional differences between virtual and real data. In contrast, real data possesses high fidelity and realistic perceptual characteristics, accurately reflecting specific attributes and multimodal interactions in the physical environment. However, its acquisition process is typically costly, time-consuming, and limited by practical conditions, resulting in a limited sample size. In multi-task learning, practical robot applications often require simultaneous handling of multiple tasks such as visual perception, path planning, and motion control. Multi-task transfer learning aims to enable models to jointly learn and transfer knowledge across multiple related tasks, thereby improving overall learning efficiency and generalization performance. However, traditional single-task independent training methods struggle to achieve effective collaboration and resource sharing between tasks, especially when faced with significant task differences and heterogeneous data sources, often resulting in insufficient adaptability and decreased generalization performance.

[0003] Vision-language-action multimodal models are playing an increasingly important role in robotic intelligent systems. These models integrate information from multiple modalities, including visual perception, language command parsing, and action execution, enabling end-to-end task processing. While multimodal models exhibit greater robustness and adaptability than single-modal models in complex tasks, their training process still faces challenges such as inconsistent distribution of multi-source data, difficulties in cross-modal alignment, and the complexity of multi-task collaborative optimization. Therefore, effectively integrating virtual and real data, reducing the distributional differences between them, especially in scenarios involving mixed virtual and real data training, and constructing an efficient training framework capable of handling multiple tasks, multiple modalities, and cross-environmental challenges are technical problems that need to be solved. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a robot multi-task transfer method, device, and medium based on virtual and real big data. This involves constructing a virtual and real dataset through multi-source acquisition and preprocessing; reducing the distribution differences between virtual and real data using a transfer evaluation model and a domain alignment loss function; dynamically determining the mixing ratio of virtual and real data based on sub-task requirements for pre-training; constructing a shared feature space under multi-task settings; and combining adversarial training and multi-objective joint optimization to achieve feature alignment and knowledge transfer. This effectively improves the model's adaptability, generalization ability, and robustness in practical deployment under cross-environment and multi-task conditions.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] According to one aspect of the present invention, a robot multi-task transfer method based on virtual and real big data is provided. The specific steps include: collecting multimodal real-time state data and new task requirements described in natural language in the task scenario, inputting them into a multi-task transfer vision-language-action big model, using the cross-task knowledge learned in the pre-training and fine-tuning stages in the shared feature space of the multi-task transfer vision-language-action big model to understand the semantics of the new instructions, and associating them with the feature representations of known tasks to achieve knowledge transfer, and outputting a control strategy corresponding to the new task based on the aligned multimodal features.

[0007] The specific steps for training the multi-task transfer vision-language-action large model include:

[0008] S1. Collect virtual data from the simulation platform execution and video generation, and obtain real data through human operation teaching and batch automated collection. Preprocess the virtual data and real data to obtain preprocessed virtual dataset and real dataset.

[0009] S2. Construct a migration evaluation model. Based on the virtual dataset and the real dataset, quantify the distribution difference between the virtual data and the real data through the migration loss function, thereby obtaining the data migration pattern and the enhanced virtual data after inter-domain alignment migration.

[0010] S3. Based on the requirements and characteristics of the sub-tasks, determine the mixing ratio of the migrated virtual data and the preprocessed real data, and construct an integrated training dataset;

[0011] S4. Input the integrated training dataset into the pre-trained model based on the Transformer architecture for training to obtain the initial multimodal large model;

[0012] S5. Under the multi-task setting, a shared feature space is constructed and adversarial training is introduced. The initial multimodal large model is trained using the joint optimization loss function of the multi-task to obtain virtual and real data between the multi-tasks. The initial multimodal large model is fine-tuned using the virtual and real data between the multi-tasks to obtain the vision-language-action large model.

[0013] Furthermore, the virtual data in S1 includes data collected from the robot control strategy executed by the simulation platform, video sequence data generated by the generative artificial intelligence platform, and data from the Internet and digital terminals; the real data includes human teaching data obtained from demonstrations by human operators, as well as data automatically collected through preset robot behavior scripts and sensor recording mechanisms.

[0014] Furthermore, the preprocessing in S1 includes screening noise samples from the raw data, standardizing the data format and annotation information, and uniformly standardizing the camera parameters, lighting conditions, and motion trajectory data.

[0015] Furthermore, the migration loss function in S2 is a differentiable function defined in the feature space and possessing domain alignment properties, used to measure the difference between the virtual data distribution and the real data distribution, and its expression is:

[0016]

[0017] Where φ(·) is the feature extractor, For the distribution of virtual domain data, Let x represent the distribution of real-world domain data, and x be samples from each domain. Let Hilbert space be the corresponding feature regeneration kernel, and ||·|| denote the norm in this space. By optimizing the transfer loss function, the feature distribution of the virtual data converges to the feature distribution of the real data, resulting in the enhanced virtual data after inter-domain alignment.

[0018] Furthermore, the method for determining the mixing ratio in S3 is to dynamically determine the mixing ratio of the transferred virtual data and the preprocessed real data in the integrated training dataset based on the subtask's requirements for at least one dimension of environmental complexity, sensor accuracy, spatiotemporal resolution, and semantic richness.

[0019] Furthermore, in S5, the method for constructing a shared feature space and jointly optimizing the loss function specifically includes: setting a task set, mapping each task in the set to a unified feature representation through a shared feature extractor, and calculating the joint optimization loss function L, expressed as:

[0020] L = L multitask +λL domain ,

[0021] Among them, L multitask For multi-task loss, λ is the weight hyperparameter, and L... domain To counteract domain adaptation loss,

[0022] Through the first optimization process Second optimization process Joint training is performed with the joint optimization loss function to align the distribution of virtual data and real data in the feature space. φ is the parameter in the adversarial domain adaptation loss, and θ and Θ are the parameters in the multi-task loss.

[0023] Furthermore, the multi-task loss is used to enable the model to perform specialized processing and optimization for the characteristics of different tasks while sharing the feature space. The expression is as follows:

[0024]

[0025] Among them, f θ (·) represents the shared feature extractor with parameter θ; For a set of tasks; s t is a virtual dataset related to task t; x is the sampled data. A task-specific independent network layer receives general features from the shared feature extractor and transforms them into a task-specific output suitable for task t, with parameter Θ.

[0026] Furthermore, the adversarial domain adaptation loss is implemented by introducing a domain discriminator, and its expression is:

[0027]

[0028] Among them, D φ It is a binary classifier neural network with parameters φ and f. θ (·) represents a shared feature extractor; s t A virtual dataset related to task t; R t This is the real dataset related to task t.

[0029] According to a second aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described thereon.

[0030] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described thereon.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] (1) Effectively reduce the difference in the distribution of virtual and real data and improve the cross-domain adaptability of the model: By constructing a transfer evaluation model and using the domain alignment loss function, the distribution difference between virtual data and real data is quantified and optimized, and enhanced virtual data with inter-domain alignment is generated. This reduces the data distribution mismatch between the simulation environment and the real environment, enabling the model trained based on virtual data to adapt more effectively to the complex characteristics of the real physical world and improve the cross-domain transfer capability of the model from virtual to real environment.

[0033] (2) Achieve multi-task collaborative optimization and knowledge sharing, and enhance generalization performance: By constructing a shared feature space under the multi-task setting and introducing an adversarial training mechanism, the joint optimization loss function is used to simultaneously optimize the performance loss and domain adaptation loss of multiple tasks, which promotes the effective sharing and transfer of knowledge between different tasks. Furthermore, adversarial domain alignment ensures the consistency of cross-task and cross-domain feature representations, thereby improving the model's collaborative working ability and generalization performance when dealing with a variety of complex tasks.

[0034] (3) Construct an end-to-end efficient training framework to improve the practicality and robustness of actual deployment: From multi-source virtual and real data acquisition, preprocessing, proportional fusion pre-training, to multi-task domain adaptation and final fine-tuning, a complete VLA large model training process is formed. This framework leverages the advantages of large virtual data scale and low cost and high fidelity of real data. Through data-driven approach, the training strategy is dynamically optimized. The final visual-language-action large model not only has high performance on a single task, but can also be smoothly transferred to new scenarios to execute different types of instructions. In actual robot applications, it exhibits higher task success rate and environmental robustness. Attached Figure Description

[0035] Figure 1 A flowchart illustrating the model training method for a robot multi-task transfer method based on virtual and real big data;

[0036] Figure 2 This is a data flow diagram for model training of a robot multi-task migration method based on virtual and real big data. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] This embodiment discloses a robot multi-task transfer method based on virtual and real big data. By collecting and processing virtual and real data, a multimodal large-scale model is pre-trained. The virtual and real data are then transferred to multiple tasks through domain adaptation. Further fine-tuning yields a vision-language-action large-scale model, significantly improving data utilization and achieving better generalization across multiple tasks. Specific steps include: collecting real-time multimodal state data from the task scenario and new task requirements described in natural language, inputting them into the multi-task transfer vision-language-action large-scale model; utilizing the cross-task knowledge learned during pre-training and fine-tuning in the shared feature space of the multi-task transfer vision-language-action large-scale model to understand the semantics of new instructions and associating them with the feature representations of known tasks to achieve knowledge transfer; and outputting a control strategy corresponding to the new task based on the aligned multimodal features.

[0039] like Figure 1 The diagram shows the specific steps involved in training the multi-task transfer vision-language-action large model, including:

[0040] S1. Collect virtual data from the simulation platform execution and video generation, and obtain real data through human operation teaching and batch automated collection. Preprocess the virtual data and real data to obtain preprocessed virtual dataset and real dataset.

[0041] S2. Construct a migration evaluation model. Based on virtual and real datasets, quantify the distribution differences between virtual and real data through the migration loss function to obtain data migration patterns and obtain enhanced virtual data after inter-domain alignment migration.

[0042] S3. Based on the requirements and characteristics of the sub-tasks, determine the mixing ratio of the transferred virtual data and the preprocessed real data, and construct an integrated training dataset;

[0043] S4. Input the integrated training dataset into the pre-trained model based on the Transformer architecture for training to obtain the initial multimodal large model;

[0044] S5. Under the multi-task setting, a shared feature space is constructed and adversarial training is introduced. The initial multimodal large model is trained with the joint optimization loss function of the multi-task to obtain virtual and real data between the multi-tasks. The initial multimodal large model is fine-tuned using the virtual and real data between the multi-tasks to obtain the vision-language-action large model.

[0045] like Figure 2 The diagram shows the data flow in the training method, specifically:

[0046] The virtual data in S1 includes data collected from the robot control strategy executed by the simulation platform, video sequence data generated by the generative artificial intelligence platform, and data from the Internet and digital devices; the real data includes human teaching data obtained from demonstrations by human operators, as well as data automatically collected through preset robot behavior scripts and sensor recording mechanisms. Preprocessing includes noise sample filtering of the raw data, standardization of data format and annotation information, and unified standardization of camera parameters, lighting conditions, and motion trajectory data.

[0047] The robot control strategy is executed using a dedicated simulation platform. Data collected by the simulation platform is denoted as s1. Videos generated by a generative AI platform, primarily demonstrating motion postures, are denoted as s2. Data from the internet and digital sources are denoted as s3. Real-world data is collected through a combination of human teaching and automated batch acquisition, ensuring high reliability in reflecting real physical conditions and operational habits. Human teaching data, denoted as r1, involves experienced operators directly demonstrating the execution of actual tasks, ensuring the naturalness and applicability of the task strategy. Automated batch acquisition utilizes predefined robot behavior scripts and sensor data recording mechanisms to systematically collect large-scale, diverse real-world environment samples, denoted as r2. Data preprocessing involves rigorous preprocessing and cleaning of the raw data (s1, s2, s3, r1, r2), including filtering noisy samples, standardizing data formats and annotation information, and uniformly standardizing camera parameters, lighting conditions, and motion trajectory data. This process is denoted as virtual data S, and real-world data R, ensuring the quality and usability of the final training dataset.

[0048] Construct a migration evaluation model for virtual data S and real data R, denoted as M. T Furthermore, by defining a loss function to represent the migration difference between virtual data and real data, the migration pattern is obtained, thereby ensuring that the calculation result of the feature distribution of the virtual data and real data under a preset distribution difference metric is less than a preset threshold.

[0049] The migration loss function in S2 is a differentiable function defined in the feature space with domain alignment properties. It is used to measure the difference between the virtual data distribution and the real data distribution, and its expression is:

[0050]

[0051] Where φ(·) is the feature extractor, For the distribution of virtual domain data, Let x represent the distribution of real-world domain data, and x be samples from each domain. Let Hilbert space be the corresponding feature regeneration kernel, and ||·|| denote the norm in this space. By optimizing the transfer loss function, the feature distribution of the virtual data converges to the feature distribution of the real data, resulting in enhanced virtual data with inter-domain alignment after transfer. The optimized scheme reduces the distribution of virtual domain data. and real domain data distribution The difference between the virtual data and the real data is such that the calculated result of the feature distribution of the virtual data and the real data under a preset distribution difference metric is less than a preset threshold.

[0052] Migration evaluation model M T The core of this approach lies in quantifying the differences between virtual and real data through a rationally designed loss function. Specifically, a class of differentiable loss functions with domain alignment properties is defined in the feature space to characterize the degree to which the virtual data distribution approximates the real data distribution. By continuously monitoring and analyzing this loss function during training, it is shown that the virtual data gradually adapts to the real environment in terms of feature representation and task performance. The transfer law is that by optimizing the scheme to make the value of the training iteration loss function steadily decrease, the feature distribution of the virtual data gradually converges to the real data distribution, thereby obtaining the transferred virtual data with inter-domain alignment that does not show statistically significant differences, thus enhancing the adaptability of virtual data in real-world application scenarios.

[0053] For specific sub-task requirements, such as when the task is cumbersome or the scenario is complex, the migrated virtual data S ′ Since complete representation is not possible, it is necessary to construct a pre-trained proportional representation scheme M, and then combine it with a certain proportion of real data to construct an ensemble training dataset G. data This approach fully leverages the advantages of large virtual data volume and precise information contained in real data, ensuring efficient utilization while maintaining the generalization of subtask execution. The data is then input into a Transformer-based pre-trained model to initially obtain a multimodal large model m suitable for both virtual and real data. φ (φ is a parameter). In S3, the mixing ratio is determined dynamically based on the subtask's requirements for at least one dimension of environmental complexity, sensor accuracy, spatiotemporal resolution, and semantic richness, in the integrated training dataset. The ratio-optimized virtual and real data are then input into a multimodal pre-trained model with a Transformer-based architecture. The model's advantages in cross-modal feature fusion and multi-task shared representation learning enable it to effectively integrate virtual and real information from different sensor modalities, resulting in a large multimodal model m of virtual and real data. φ This improves the standardization and generalization of feature extraction.

[0054] Transformer-based pre-trained models require an ensemble training dataset G. data As input to the pre-training dataset, the Transformer-based model has advantages in cross-modal feature and virtual-real data fusion, enabling the model to effectively integrate virtual and real information from different sensor modalities, achieving unified feature extraction and improved generalization. By introducing such data fusion and weighting optimization strategies during the pre-training stage, the resulting multimodal large model m... φ It not only demonstrates superior applicability and robustness in downstream subtasks, but also provides a flexible and efficient extension space for subsequent model fine-tuning.

[0055] In S5, the multi-task requirement specifically means that the model needs to simultaneously handle multiple task types and scenarios, such as visual perception, path planning, and motion planning, and achieve cross-domain adaptation and performance improvement in both virtual and real environments. The method for constructing a shared feature space and jointly optimizing the loss function specifically includes: setting a task set, mapping each task in the set to a unified feature representation through a shared feature extractor, and calculating the joint optimization loss function L, expressed as:

[0056] L = L multitask +λL domain ,

[0057] Among them, L multitask For multi-task loss, λ is the weight hyperparameter, and L... domain To counteract domain adaptation loss,

[0058] Through the first optimization process Second optimization process Joint training is performed with the joint optimization loss function to align the distribution of virtual data and real data in the feature space. φ is the parameter in the adversarial domain adaptation loss, and θ and Θ are the parameters in the multi-task loss.

[0059] The multi-task requirement specifically means that the model needs to simultaneously handle multiple task types or multiple task scenarios, and achieve cross-domain adaptation and performance improvement in both virtual and real environments. The domain adaptation method involves constructing a shared feature space within the neural network structure, mapping the input data of each task to a unified latent representation. Based on this, an adversarial training strategy is combined, introducing a game between the domain discriminator and the feature extractor to align the feature distributions of virtual and real data, making it difficult for the network to distinguish the data source domain based on features. The multi-task loss is used to enable the model to perform specialized processing and optimization for the characteristics of different tasks based on the shared feature space. Its expression is:

[0060]

[0061] Among them, f θ(·) represents the shared feature extractor with parameter θ; For a set of tasks; s t is a virtual dataset related to task t; x is the sampled data. An independent network layer specific to task t receives general features from the shared feature extractor and transforms them into a task-specific output suitable for task t, with parameter Θ, to improve the model's adaptability to various tasks.

[0062] Adversarial domain adaptation loss is implemented by introducing a domain discriminator, and its expression is:

[0063]

[0064] Among them, D φ It is a binary classifier neural network with parameters φ and f. θ (·) represents a shared feature extractor; s t A virtual dataset related to task t; R t This is the real dataset related to task t.

[0065] By jointly optimizing the loss L, we can further align the distributions of virtual and real data across multiple tasks, resulting in a virtual-real dataset D across multiple tasks. multi This is to prepare for achieving greater adaptability and generalization capabilities in a variety of complex tasks and cross-domain scenarios.

[0066] The virtual and physical data between multiple tasks will be adapted through domain-adapted D multi Input a large multimodal model m suitable for both virtual and real data φ (φ is a parameter), perform fine-tuning of the model, and freeze the pre-trained large model m. φ The underlying feature layer only updates parameters of certain layers, such as the classification layer, to prevent learning general features from scratch, greatly reducing computation. Simultaneously, the learning rate is dynamically adjusted to ensure more efficient model convergence. Ultimately, this results in a large-scale Vision-Language-Action (VLA) model that enables cross-modal and cross-scene multi-task transfer, denoted as model [model name missing]. VLA This enables the model to not only stably perform specific tasks in a single environment, but also to smoothly migrate to new domains or scenarios to perform different types of tasks, achieving a highly robust, high-performance end-to-end multimodal large model that meets the needs of various complex tasks.

[0067] This embodiment addresses the shortcomings of existing technologies in domain adaptation, such as the reliance on a single virtual data source and insufficient generalization ability. It enhances the model's cross-domain adaptability through multi-source data acquisition and joint preprocessing. This method not only employs conventional domain adaptation techniques to reduce the difference between virtual and real data, but also integrates simulation platforms, generative AI platforms, and internet data at the data source. Furthermore, it innovatively uses quantitative analysis of sub-tasks (such as environmental complexity and sensor accuracy) to determine the optimal ratio of virtual to real data for pre-training in a data-driven manner. This multi-source data fusion preprocessing approach ensures the model has a better generalization foundation before entering adversarial training, thus exhibiting higher stability and robustness in the final cross-domain transfer.

[0068] Furthermore, this embodiment addresses the performance instability issue of existing technologies in jointly optimizing multiple tasks during virtual-to-real migration. While existing technologies can jointly optimize the loss functions of multiple tasks, the collaborative effect between tasks often degrades during Sim-to-Real migration. This application tightly integrates a multi-objective joint optimization strategy with an adversarial training mechanism within a shared feature space. This not only aligns the distribution of virtual and real data but also utilizes the game-theoretic process of the domain discriminator to stabilize the cross-domain performance of different tasks, effectively maintaining the robot's collaborative working ability in real multi-task scenarios.

[0069] This embodiment proposes a complete training method that solves the problem of VLA large models relying on large amounts of real data and exhibiting poor adaptability when directly fine-tuning them in the robotics field. Existing technologies directly fine-tune large VLA models, which are costly and have limited effectiveness. Before the final fine-tuning, this application has fully optimized the model through a series of steps, especially its core aspects: data acquisition and proportional scheme pre-training based on subtask quantization analysis, as well as subsequent transfer evaluation model alignment and multi-task domain adaptation. This complete process of step-by-step optimization enables large models to achieve higher task adaptability with less real data, meeting the actual deployment needs of complex robotic scenarios.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0071] The electronic device of this invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0072] Multiple components in the device are connected to an I / O interface, including: input units such as a keyboard, mouse, etc.; output units such as various types of displays, speakers, etc.; storage units such as disks, optical disks, etc.; and communication units such as network interface cards, modems, wireless transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks. The processing unit performs the various methods and processes described above, such as the method of the present invention. For example, in some embodiments, the method of the present invention may be implemented as a computer software program tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via ROM and / or the communication unit. When the computer program is loaded into RAM and executed by the CPU, one or more steps of the method of the present invention described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute the method of the present invention by any other suitable means (e.g., by means of firmware).

[0073] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0074] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0075] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot multi-task migration method based on virtual and real big data, characterized in that, The specific steps include: collecting multimodal real-time state data and new task requirements described in natural language from the task scenario, inputting them into the multi-task transfer vision-language-action big model, utilizing the cross-task knowledge learned in the pre-training and fine-tuning stages in the shared feature space of the multi-task transfer vision-language-action big model to understand the semantics of the new instructions, and associating them with the feature representations of known tasks to achieve knowledge transfer, and outputting the control strategy corresponding to the new task based on the aligned multimodal features; The specific steps for training the multi-task transfer vision-language-action large model include: S1. Collect virtual data from the simulation platform execution and video generation, and obtain real data through human operation teaching and batch automated collection. Preprocess the virtual data and real data to obtain preprocessed virtual dataset and real dataset. S2. Construct a migration evaluation model. Based on the virtual dataset and the real dataset, quantify the distribution difference between the virtual data and the real data through the migration loss function, thereby obtaining the data migration pattern and the enhanced virtual data after inter-domain alignment migration. S3. Based on the requirements and characteristics of the sub-tasks, determine the mixing ratio of the migrated virtual data and the preprocessed real data, and construct an integrated training dataset; S4. Input the integrated training dataset into the pre-trained model based on the Transformer architecture for training to obtain the initial multimodal large model; S5. Under the multi-task setting, a shared feature space is constructed and adversarial training is introduced. The initial multimodal large model is trained using the joint optimization loss function of the multi-task to obtain virtual and real data between the multi-tasks. The initial multimodal large model is fine-tuned using the virtual and real data between the multi-tasks to obtain the vision-language-action large model.

2. The robot multi-task migration method based on virtual and real big data according to claim 1, characterized in that, The virtual data in S1 includes data collected from the robot control strategy executed by the simulation platform, video sequence data generated by the generative artificial intelligence platform, and data from the Internet and digital terminals; the real data includes human teaching data obtained from demonstrations by human operators, as well as data automatically collected through preset robot behavior scripts and sensor recording mechanisms.

3. The robot multi-task migration method based on virtual and real big data according to claim 1, characterized in that, The preprocessing in S1 includes screening noise samples from the raw data, standardizing the data format and annotation information, and uniformly standardizing the camera parameters, lighting conditions, and motion trajectory data.

4. The robot multi-task migration method based on virtual and real big data according to claim 1, characterized in that, The migration loss function in S2 is a differentiable function defined in the feature space and possessing domain alignment properties. It is used to measure the difference between the virtual data distribution and the real data distribution, and its expression is: Where φ(·) is the feature extractor, For the distribution of virtual domain data, Let x represent the distribution of real-world domain data, and x be samples from each domain. Let Hilbert space be the corresponding feature regeneration kernel, and ||·|| denote the norm in this space. By optimizing the transfer loss function, the feature distribution of the virtual data converges to the feature distribution of the real data, resulting in the enhanced virtual data after inter-domain alignment.

5. The robot multi-task migration method based on virtual and real big data according to claim 1, characterized in that, The method for determining the mixing ratio in S3 is to dynamically determine the mixing ratio of the transferred virtual data and the preprocessed real data in the integrated training dataset based on the subtask's requirements for at least one dimension of environmental complexity, sensor accuracy, spatiotemporal resolution, and semantic richness.

6. The robot multi-task migration method based on virtual and real big data according to claim 1, characterized in that, In step S5, the method for constructing a shared feature space and jointly optimizing the loss function specifically includes: setting a task set, mapping each task in the set to a unified feature representation through a shared feature extractor, and calculating the joint optimization loss function L, expressed as: L=L multitask +λL domain , Among them, L multitask For multi-task loss, λ is the weight hyperparameter, and L... domain To counteract domain adaptation loss, Through the first optimization process Second optimization process Joint training is performed with the joint optimization loss function to align the distribution of virtual data and real data in the feature space. φ is the parameter in the adversarial domain adaptation loss, and θ and Θ are the parameters in the multi-task loss.

7. A robot multi-task migration method based on virtual and real big data according to claim 6, characterized in that, The multi-task loss is used to enable the model to perform specialized processing and optimization for the characteristics of different tasks while sharing the feature space. Its expression is: Among them, f θ (·) represents the shared feature extractor with parameter θ; For a set of tasks; s t is a virtual dataset related to task t; x is the sampled data. A task-specific independent network layer receives general features from the shared feature extractor and transforms them into a task-specific output suitable for task t, with parameter Θ.

8. A robot multi-task migration method based on virtual and real big data according to claim 6, characterized in that, The adversarial domain adaptation loss is implemented by introducing a domain discriminator, and its expression is: Among them, D φ It is a binary classifier neural network with parameters φ and f. θ (·) represents a shared feature extractor; s t A virtual dataset related to task t; R t This is the real dataset related to task t.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images