A simulation-to-real visual servoing dual-arm robot migration imitation learning method

Abstract

The application discloses a simulation-to-reality visual servo dual-arm robot migration imitation learning method, which comprises the following steps: S110, a virtual simulation scene is constructed according to a real task scene, and a connection between the two is established; S120, collected demonstration mechanical arm and execution mechanical arm operation data are replayed and optimized in the virtual simulation scene to construct a final training used data set; S130, a deep imitation learning network is designed and implemented, the input of the deep imitation learning network comprises visual data and demonstration mechanical arm joint motor state data obtained in the real task scene, and the output is predicted execution mechanical arm joint motor state at the next moment; and S140, the deep imitation learning network which has been fully trained and converged is migrated from the virtual simulation scene to a dual-arm robot in the real task scene. The application improves the exploration efficiency and generalization ability of the dual-arm robot learning network.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology and relates to a method for transfer and imitation learning of a visual servo dual-arm robot from simulation to reality. Background Technology

[0002] With the rapid advancement of technology, robotics has become an indispensable part of modern industry and service sectors. Dual-arm robots have attracted particular attention due to their ability to mimic the collaborative work of human arms. They exhibit unique advantages in performing complex tasks, showing outstanding performance in manufacturing, medical surgery, service, and exploration. The research and development of dual-arm robots is of great significance for promoting automation and intelligentization; they can improve production efficiency and service quality, and are key to achieving technological breakthroughs.

[0003] To overcome the shortcomings of traditional control algorithms in adapting to the diversity and complexity of tasks, research on dual-arm robots has begun to focus on deep learning and visual perception technologies. Deep learning technology learns from massive amounts of data to optimize robot behavior, enabling it to better understand and adapt to its working environment. Meanwhile, the development of visual perception technologies, especially visual servo control, has become a key direction for improving the autonomy and adaptability of dual-arm robots. It guides robot movements through visual information, achieving accurate identification and localization of target objects, providing a reliable perceptual foundation for the grasping and collaborative operation of dual-arm robots. Researching how to effectively train deep learning models in simulation environments and successfully transfer them to the real world is crucial for the future development of dual-arm robots.

[0004] The field of dual-arm robotics faces several challenges in the selection and training of deep learning models. First, the lack of labeled data limits the training effectiveness of deep learning models, affecting the robot's learning and adaptation capabilities in complex environments. Second, purely real-world experiments are risky; frequent physical interactions can damage the robot, increasing economic costs and safety risks. Furthermore, the application of visual servo control technology in the real world faces perception challenges, requiring the robot to accurately understand and process visual information for precise control. Simultaneously, selecting appropriate deep learning models and algorithms is also problematic, as different tasks and environments may require different learning strategies and network structures. The accuracy and robustness of visual perception are also critical issues, as visual systems can be affected by various factors such as changes in ambient lighting and object surface characteristics. Solving these problems is crucial for developing a visual servo control technology for dual-arm robots that can be trained in simulation environments and seamlessly transferred to the real world. This will not only drive advancements in robotics technology but also provide strong technical support for future automation and intelligent applications. Summary of the Invention

[0005] This invention provides a transfer learning method for visual servo dual-arm robots from simulation to reality, addressing issues related to environmental perception, training data, and learning methods in related technologies. It aims to utilize simulated scenarios for large-scale data collection linking visual perception and robotic arm motion, driving initial model training based on deep imitation learning. Then, it optimizes the model using a small amount of real data, transferring the model's visual servo motion planning capabilities to a real robotic arm. This effectively accelerates the training time of dual-arm robot control algorithms in real-world scenarios and reduces the time cost and collision damage risk associated with real-world data collection and training.

[0006] The technical solution adopted in this invention is: a transfer and imitation learning method for a visual servoing dual-arm robot from simulation to reality, comprising the following steps:

[0007] S110, construct a virtual simulation scenario based on the real task scenario, and establish the connection between the real task scenario and the virtual simulation scenario;

[0008] S120 replays and optimizes the collected dual-arm robot operation data in a virtual simulation scene to construct the final training dataset;

[0009] S130, Design and implement a deep imitation learning network. This deep imitation learning network adopts an end-to-end strategy and combines variational autoencoder and Transformer architecture. The input of the deep imitation learning network includes visual data obtained from real task scenarios and motor state data of each joint of the taught robotic arm. The output is the predicted state of each joint of the executing robotic arm at the next time step.

[0010] S140 transfers a well-trained and converged deep imitation learning network from a virtual simulation scenario to a dual-arm robot in a real task scenario.

[0011] The beneficial effects of this invention are as follows:

[0012] Compared with existing technologies, this paper presents a solution for visual servo motion planning of dual-arm robots. This solution makes full use of the achievements of semi-physical simulation and deep learning technology, which can accurately simulate and optimize the robot's motion execution in real-world scenarios, achieve efficient and accurate task completion, ensure the quality and safety of operation, and meet the needs of modern intelligent manufacturing and automated production.

[0013] This invention significantly reduces the risk of damage to purely physical experimental platforms and the time consumption of data acquisition in purely simulation experimental platforms by improving the existing training data acquisition process and verification platform for dual-arm robots. At the same time, it ensures the action continuity, representativeness and transferability of the dataset, thereby improving the exploration efficiency and generalization ability of the dual-arm robot learning network. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the physical and simulation scenarios in a vision servo dual-arm robot transfer imitation learning method from simulation to reality according to the present invention.

[0015] Figure 2 This is a diagram illustrating the effect of smooth data generation and playback in a simulation-to-real visual servo dual-arm robot transfer imitation learning method according to the present invention.

[0016] Figure 3 This is a diagram of the deep imitation learning network architecture used in this invention;

[0017] Figure 4 A block description diagram for actions;

[0018] Figure 5 Describe the time groupings in the graph;

[0019] Figure 6 This diagram illustrates the application of a model in the real world to control a dual-arm robot to perform a specific task. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] This invention provides a method for transfer learning from simulation to reality in a visual servoing dual-arm robot, comprising the following steps:

[0023] S110: Construct a virtual simulation scenario based on the real task scenario and establish a connection between the real task scenario and the virtual simulation scenario. Figure 1 This is a schematic diagram of the physical and simulation scenarios in a visual servoing dual-arm robot transfer and imitation learning method according to the present invention. Figure 1As shown, data is collected from the left data acquisition area in conjunction with the middle virtual simulation scene to provide training data for the application of the actual scene model on the right. This step is fundamental to realizing the visual servo motion planning of the dual-arm robot. The robotic arms in the middle virtual simulation scene and the real task scene on the right are called the execution robotic arms, while the robotic arm of the dual-arm robot in the real task scene on the left is called the teaching robotic arm. Specifically, this step includes the following key components:

[0024] S110-1, Obtain the 3D models of all objects in the real task scenario and configure them as a virtual simulation scene. For example... Figure 1 As shown, the core task is to accurately obtain the 3D models of all objects in the real task scene based on the real task scene on the right, and configure them as... Figure 1 The virtual simulation environment shown in the middle.

[0025] In one implementation, 3D models of all objects can be obtained through various methods, including: automating 3D modeling using algorithms and software to automatically construct 3D models based on text, photographs, and video information from multiple angles. Automated 3D modeling technologies include structured light scanning, stereo vision, and Structure from Motion (SFM). Structured light scanning acquires the 3D shape of an object by projecting a specific light pattern and analyzing its deformation on the object's surface; stereo vision reconstructs the 3D structure of an object by analyzing the differences between images captured by two or more cameras; and SFM infers the 3D structure and motion information of an object by analyzing image sequences from different viewpoints. These automated methods can significantly improve the efficiency and accuracy of model acquisition, and are particularly suitable for modeling large-scale scenes. Algorithms and software used may include ContextCapture and Agisoft.

[0026] Besides automated 3D modeling technologies, professional modeling software such as AutoCAD, 3ds Max, and Maya can be used, allowing professionals to manually construct 3D models based on blueprints or photographs. This method is particularly important in scenarios requiring high precision and intricate detail. For example, in architecture and engineering, engineers and designers use software like AutoCAD to precisely construct 3D models of buildings based on detailed engineering drawings and design specifications; while in film, animation, and game development, artists utilize software like 3ds Max and Maya to design and create complex character, prop, and scene models according to creative and visual effects requirements. Manual modeling fully leverages human creativity and professional skills, enabling highly customized model designs.

[0027] In addition to the above, virtual simulation scenarios can also be built using physics engines such as Mujoco (Multi-Joint dynamics with Contact). Mujoco is a powerful physics simulation software capable of accurately simulating complex physical interactions and dynamic behaviors, making it ideal for robot simulation. It provides rich API interfaces, facilitating easy interaction and control with external programs.

[0028] These 3D models will be used to construct an accurate virtual simulation environment that simulates the physical properties and appearances of objects in the real world. In this simulation environment, the physical properties of objects, such as mass, density, and elasticity, need to be accurately simulated to ensure the reliability and validity of the simulation results. For example, in robot manipulation simulations, the weight and coefficient of friction of an object affect the robot's strategy and effectiveness in grasping and moving objects; in car crash simulations, the strength and stiffness of the car body materials determine the deformation and damage during a collision. Simultaneously, the appearance details of objects, such as color, texture, and gloss, also need to be meticulously reproduced to enhance the realism and immersion of the simulation. These accurate 3D models provide a solid foundation for virtual simulation, enabling it to effectively simulate and predict various scenarios and events in the real world.

[0029] S110-2 establishes the connection between the teaching robot in the real-world task scenario and the execution robot in the virtual simulation scenario. The core of this process is ensuring that the virtual simulation environment accurately simulates the movements of the teaching robot in the real world, and that the virtual execution robot accurately replicates these movements. This connection is typically established by combining advanced simulation software and programming languages.

[0030] Simulation software can use a physics engine such as Mujoco (Multi-Joint dynamics with Contact).

[0031] Step S110-2 may include: S110-2a, using the robot arm's software development kit (SDK), Python implements real-time acquisition and synchronization of motion data of the taught robot arm in the real task scenario. Python, as a widely used programming language, has powerful functionality and flexibility in the field of robot simulation and control. The SDK typically provides rich functions and interfaces, capable of acquiring state information such as the robot arm's position, velocity, and acceleration, as well as various commands and parameters to control the robot arm's movement. S110-2b, Python transmits the acquired motion data to the Mujoco virtual simulation environment. In one embodiment, during the simulation, the Python script can continuously acquire motion data from the taught robot arm's SDK and then transmit this data to the Mujoco simulation environment. S110-2c, Mujoco drives the execution robot arm in the virtual world to perform corresponding operations based on the received motion data. In this way, the execution robot arm can accurately replicate the movements of the taught robot arm in the real world, achieving close connection and synchronization between the two. By combining Mujoco with Python, we can not only effectively simulate and verify the motion control strategies of the teachable robotic arm, but also conduct various experiments and tests in a virtual environment, thereby reducing real-world risks and costs. Furthermore, this method can provide strong support and assistance for the optimized design, path planning, and task execution of the teachable robotic arm.

[0032] Optionally, besides using a real-world robotic arm to perform the teaching task, a general and simple method can also be adopted: using a handheld gripper in conjunction with a SLAM (Simultaneous Localization and Mapping) algorithm or an infrared motion capture environment to complete the teaching task. In this case, the handheld gripper in conjunction with the SLAM algorithm or the infrared motion capture environment can be regarded as the teaching robotic arm in the real task scenario mentioned in step S110-2. In this case, step S110-2 may include: S110-2a', tracking the pose of the handheld gripper using a SLAM algorithm, or by installing multiple infrared markers on the handheld gripper and using an infrared camera to capture the motion trajectory of the infrared markers, thereby achieving the tracking of the handheld gripper's pose; S110-2b', transmitting the captured pose of the handheld gripper to the robotic arm implemented in the Mujoco end-effector version, and using Mujoco to accurately simulate the movement and control behavior of the robotic arm based on the pose; S110-2c', achieving precise control of the robotic arm and task execution by operating the end-effector in a virtual simulation scenario.

[0033] The core of this alternative implementation lies in acquiring precise end-effector pose and gripper opening / closing values, and directly manipulating the robotic arm based on this data using an end-effector controller version implemented in Mujoco. When using a handheld gripper, the operator can manually control the movement and opening / closing of the gripper to simulate the end-effector actions of the robotic arm. SLAM algorithms play a crucial role in this process, enabling real-time localization and mapping in unknown environments, providing accurate spatial position information for the handheld gripper. Through SLAM algorithms, the system can accurately track the gripper's position and orientation, maintaining high positioning accuracy even in complex or dynamically changing environments. Furthermore, SLAM algorithms can be combined with machine vision technology to further improve the perception of gripper pose, enabling it to better adapt to different task requirements and scene changes. Infrared motion capture environments achieve precise tracking of the gripper pose by installing multiple infrared markers on the gripper and using an infrared camera to capture the motion trajectories of these markers. Infrared motion capture technology has advantages such as high precision, high sampling rate, and low latency, enabling real-time capture of subtle gripper movements and providing accurate input data for the simulation system. Meanwhile, the infrared motion capture environment can be combined with other sensors and devices, such as gyroscopes and accelerometers, to further enhance the perception and understanding of the gripper's motion state. After acquiring precise end-effector pose and gripper opening / closing values, this data is directly transmitted to the robotic arm's end-effector controller version implemented in Mujoco. Mujoco, as a powerful physical simulation software, can accurately simulate the robotic arm's motion and control behavior based on this data. By operating the end-effector controller in the simulation environment, precise control and task execution of the robotic arm can be achieved, thereby verifying and optimizing control strategies and improving the robotic arm's performance and reliability. Furthermore, this method can provide important references and guidance for the design, planning, and application of robotic arms, promoting the further development of robotics technology.

[0034] S110-3, after establishing the connection between the teaching robot in the real-world task scenario and the execution robot in the virtual simulation scenario, the operator uses the teaching robot to precisely control the execution robot in the virtual simulation environment to perform a series of specific tasks. The key to this process is the real-time and accurate acquisition of motion data from each joint of the teaching robot, including parameters such as position and velocity. This data is transmitted in real-time to the simulation system via sensors and a data acquisition system, serving as control signals to drive the corresponding joints of the execution robot, enabling it to perform the same actions and tasks as the teaching robot in the simulation scenario. In this way, the operator can intuitively operate the teaching robot in the real world, while the simulation system can simulate the actions and behaviors of the execution robot in real time. This not only provides the operator with an intuitive feedback and control interface but also allows for the testing and verification of various task execution strategies and operational techniques in a safe simulation environment. During the simulation, the system records and stores the joint data of the execution robot, which contains rich task execution information and motion characteristics. After subsequent data processing and optimization steps, this data will be used to train and optimize the artificial intelligence model. By accumulating a large amount of task execution data in a simulation environment, the generalization ability and execution efficiency of artificial intelligence models in real-world environments can be improved.

[0035] S120 involves replaying and optimizing the collected data from the teaching and execution robotic arms in a virtual simulation environment to construct the final training dataset. This step is crucial for ensuring the accuracy and effectiveness of the robot's motion data, and includes the following sub-steps:

[0036] In S120-1, the collected operational data of the teaching and execution robotic arms are replayed in a virtual simulation scenario. This process accurately simulates the robot's behavior in a real-world environment. Through data replay, every detail of the robot's actions during task execution can be observed in detail, including its response to visual information, the accuracy of motion planning, and the coherence and smoothness of task execution. This observation not only covers the robot's performance at different task stages but also reveals its adaptability and flexibility under complex environmental conditions. Analysis of the replayed data allows for a comprehensive evaluation of the robot's performance and efficiency, identifying potential problems and shortcomings. Ultimately, the data replay process helps determine the feasibility of the collected data for the task, i.e., whether the data can truly and accurately reflect the robot's operational capabilities and task execution effects in a real-world environment. This is crucial for subsequent model training and algorithm optimization, ensuring that the developed robot system can efficiently and reliably complete various tasks in practical applications. This first step of data validation lays a solid foundation for further data optimization.

[0037] S120-2 automatically selects key actions from the operational data of the teaching and execution manipulators in a virtual simulation scenario using a key action selection algorithm. This algorithm analyzes the changes in motor angle values ​​of each axis of the teaching manipulator to automatically identify key action nodes during task execution. Specifically, the algorithm monitors the motion trajectory and joint changes of the teaching manipulator in real time during task execution, accurately locating the action moments that have a decisive impact on task completion based on characteristics such as the rate and amplitude of motor angle changes and their coordination with other joints. By automatically selecting key action points, this algorithm significantly improves the efficiency and accuracy of subsequent manual selection. It eliminates the need to review the entire operation frame by frame, allowing direct focus on these key action nodes for more in-depth analysis and evaluation. This not only saves significant manpower and time costs but also makes the selection of key actions more objective and scientific, laying a solid foundation for subsequent data annotation, model training, and algorithm optimization. In practical applications, the key action selection algorithm can effectively improve the intelligence level of robot task execution and promote the widespread application of robotics technology in complex task scenarios. The key action selection algorithm can be implemented using the idea of ​​inter-frame difference in motion.

[0038] Specifically, the key action selection algorithm works as follows: First, the first action node in the operation process is selected as the key node. Then, by judging the direction of change of the motor angle value of the current key node's action through adjacent action nodes, if the angle value decreases, a sliding window method is used to find a local minimum point as the key point. After finding the local minimum, a local maximum point is found as the key point, and this process is repeated until the last node, which is then selected as the key point. If the angle value increases, a sliding window method is used to find a local maximum point as the key point. After finding the local maximum, a local minimum point is found as the key point, and this process is repeated until the last node, which is then selected as the key point. This method of dynamically selecting key points based on the direction of change of motor angle value can accurately capture the key action nodes in the operation process, thus providing strong support for subsequent analysis and application.

[0039] S120-3, based on the key actions selected in step S120-2, continues to manually select key actions. This process ensures that the core features of the task operation are preserved. Although the key action selection algorithm has automatically identified a series of key action nodes, manual review is still essential to ensure that the selection of these nodes is completely reasonable and meets the actual needs of the task. This step includes: examining each key action node, and evaluating whether it accurately captures the core aspects of the task in conjunction with the specific goals and operational procedures of the task. By manually selecting key actions, the smoothed and optimized action data can be finely adjusted and supplemented, ensuring that it retains the core features of the task operation while also meeting the needs of subsequent model training and algorithm optimization. This step not only improves the quality and usability of the data but also provides a strong guarantee for the robot to execute tasks accurately and efficiently in practical applications. The key action selection method combining manual and algorithmic approaches fully leverages the advantages of each, making the data verification process more comprehensive and reliable.

[0040] S120-4, utilizing S120-3, generates smooth operational data for the motors of each joint of the robotic arm by manually selecting updated key motion nodes. The effect of smooth data generation and playback is as follows: Figure 2 As shown. This process involves in-depth processing of the raw data, which may include: effectively eliminating excessive jitter noise and outliers in key action nodes by employing data smoothing algorithms and filtering techniques. These noises and outliers may originate from factors such as sensor errors, environmental interference, or instability of the mechanical system. If not processed, they will directly affect the accuracy and efficiency of model training, and thus the performance of the robot in practical applications. After smoothing, the motion trajectory of the robotic arm in the simulation scene becomes smoother and more natural, and the transitions of joint movements are also smoother and more coherent. Playing back these smoothed operation data in the simulation environment allows for a direct observation of the entire process of the robotic arm performing the task, including its response speed to task commands, the accuracy of its movements, and its adaptability in complex environments. This provides an important basis for evaluating the task performance of the robotic arm and helps to identify and solve potential problems in a timely manner. Data smoothing algorithms can use moving average or exponential average methods, and filtering techniques can use median filtering.

[0041] Step S120 may include: to ensure that the smoothed data can complete the task execution completely and accurately, repeating steps S120-2 to S120-4. In each iteration, the selection of key actions and the effect of data smoothing are meticulously checked and optimized until the marked key actions can fully achieve task execution. The checking and optimization can involve further selecting key action nodes based on the previously selected key action nodes, or reselecting key action nodes. This iterative process not only improves the quality and reliability of the data but also lays a solid foundation for the smooth progress of subsequent model training, enabling the trained model to better adapt to the task requirements in the real-world environment and improve the robot's execution efficiency and accuracy in practical applications.

[0042] S120-5: Construct a dataset based on visual data acquired from the virtual simulation scene and smoothed data generated from the key motion nodes selected in step S120-3. By organically combining the visual data acquired from the virtual simulation scene and the smoothed data generated from the key motion nodes selected in step S120-3, a dataset containing all the necessary information for robot operation can be created, providing comprehensive and accurate data support for the training of artificial intelligence models. Visual data provides rich environmental perception information for the dataset, including scene images observed by the robot during task execution, features such as the shape, color, and texture of objects, as well as the relative position and interaction between the robot and the environment. This information is crucial for the model to understand the task scene, identify target objects, and plan reasonable operational strategies. The smoothed data generated from the manually optimized key motion nodes provides precise motion execution information for the dataset, including the motion trajectory, speed, and other parameters of each joint of the teaching robotic arm, as well as the coordination relationships and timing information between actions. This data helps the model accurately learn and master the robot's motion patterns and techniques during task execution, improving the model's control accuracy and execution efficiency. By constructing such a comprehensive dataset, a solid data foundation can be provided for the training of artificial intelligence models. During training, the model can fully utilize various information in the dataset for deep learning and optimization, thereby acquiring stronger perception, decision-making, and execution capabilities. This plays a crucial role in improving the intelligence level of robots in practical applications, enabling them to complete various complex tasks more flexibly and efficiently. Furthermore, high-quality datasets can also promote innovation and development in robotics, providing abundant resources and inspiration for future research and applications.

[0043] S130, designed and implemented a deep imitation learning network. This network employs an end-to-end strategy, combining a variational autoencoder (VAE) and a Transformer architecture to achieve precise control of a dual-arm robot, such as... Figure 3As shown. The network input includes visual data acquired from the real task scenario and the state data of each joint motor of the taught robotic arm. The output is the predicted state of each joint motor of the executing robotic arm at the next time step. By calculating the loss between the actual value and the predicted value, the network is trained based on the loss and the number of training epochs until convergence. During training, the model is periodically validated in a virtual simulation scenario to obtain the task completion rate, which helps to determine whether the network has converged. The visual data can come from any type of visual sensor. Here, we take RGB images from multiple perspectives as an example, which includes the following sub-steps:

[0044] S130-1, with a Transformer encoder at its core, constructs a feature extraction framework as a variational autoencoder. The encoder's input data encompasses the joint motor states of the dual-arm robot at a specific time t, including joint position information, and a series of action sequences from time t to t+k, such as... Figure 4 As shown, these action sequences contain detailed changes in the robot's target joint positions over a period of time. Furthermore, a [CLS] marker is added to the encoder input; this special marker acts as an information aggregator in the sequence, helping the model better understand and integrate the contextual information of the entire action sequence. To enable the Transformer encoder to effectively process this input data, the joint position information and action sequences are first mapped to a high-dimensional space through a linear embedding layer. This mapping process transforms the original low-dimensional data into an embedding-dimensional representation suitable for Transformer model operation, allowing the model to capture more complex and subtle feature relationships. In the embedding dimension, the Transformer encoder leverages its self-attention mechanism to deeply learn the complex patterns and implicit features of each joint's motion when the robot performs fine manipulation tasks. These features encompass various aspects such as the coordination of joint movements, temporal relationships, and interactions with other joints. Finally, the encoder encodes the rich feature information extracted from the input data into a latent representation vector, which is then projected onto a formal variable z through a linear layer. This formal variable z not only contains the static features of the robot's actions but also implies the dynamic changes in the actions and potential execution strategies, providing crucial guidance for subsequent decoding and action generation. Through this step, the deep imitation learning network can effectively capture and understand the robot's action features in complex tasks, laying a solid foundation for achieving accurate action imitation and generation.

[0045] The S130-2 VAE decoder design integrates formal variables z, the motor states of each joint of the dual-arm robot at time t, and RGB images from multiple perspectives as visual data input to achieve accurate prediction and generation of robot movements. First, the input RGB images undergo deep feature extraction via a ResNet18 network, converting rich visual information into feature maps. This process captures key information such as object shape, color, and texture, providing crucial visual support for subsequent movement prediction. To preserve spatial information in the feature maps, the ResNet18 network employs sinusoidal position embedding technology, integrating positional information into the feature maps. Thus, the feature maps not only contain image content but also preserve the spatial relationships between pixels, enabling the model to better understand the spatial layout and relationships of objects. Next, the feature maps are flattened and mapped to a high-dimensional hidden dimension space. Simultaneously, the motor states of each joint of the dual-arm robot at time t are projected through a linear layer. Then, the mapping result of the feature maps is concatenated with the linear layer projection results of the formal variables z and the motor states, serving as the input vector for the Transformer encoder. The formal variable z carries the stylistic features and dynamic laws of the robot's actions, while the joint motor states provide specific execution information for the current action. These data together constitute a comprehensive high-dimensional feature representation. This embedded data is fed into the Transformer encoder, which uses a self-attention mechanism to deeply capture the deep information and complex relationships within the data. The encoder can understand the interactions and influences between different features, as well as their importance in action generation. Next, the data output by the Transformer encoder is passed to the Transformer decoder. The decoder uses a cross-attention mechanism, combining the encoder's output and input data, to predict the robot's action sequence over a future period, and combines these actions to weightedly obtain the joint motor states at time t+1, such as... Figure 5 As shown, this prediction process not only considers the current action state and visual information, but also fully incorporates the action rules and strategies implied by the formal variable z. This allows the prediction results to accurately reflect the robot's action execution in the next moment, providing strong support for achieving coherent and natural action imitation and generation. Through this step, deep imitation learning networks can effectively combine visual information, action state, and style features to achieve accurate prediction and control of robot actions.

[0046] S130-3 designs termination conditions for model training to ensure that the deep imitation learning network can fully learn and master the complex mapping relationship from visual data and taught robotic arm joint motor state data to the predicted execution robotic arm joint motor state at the next time step. First, the key stopping condition is that the end-to-end motion generation loss function decreases to a preset target threshold. This threshold represents the minimum standard the model should achieve in motion prediction accuracy. When the loss function falls below this threshold, it indicates that the model can generate the expected motion sequence relatively accurately, with the prediction error controlled within an acceptable range, thus meeting the basic requirements of the model in motion generation. Second, exceeding a set value in the total number of alternating iterations is also an important stopping condition. During training, the model needs to continuously optimize parameters and adjust the network structure through a large number of iterations to better adapt to the training data and task requirements. Setting a reasonable upper limit on the number of iterations can prevent the model from falling into overtraining, while ensuring that the model has enough time and opportunity to learn and absorb key features and patterns in the data. When the number of training steps reaches or exceeds this set value, it means that the model has been sufficiently trained and has good generalization ability and stability. Finally, verifying the model's convergence through task completion rate in a pure simulation environment is another crucial stopping condition. The simulation environment provides a safe and controllable testing platform, enabling comprehensive evaluation and verification of the model's performance without increasing risks and costs in the real world. Task completion rate is a crucial indicator of a model's performance in practical applications, reflecting its success rate and efficiency in task execution. When the model's task completion rate in the simulation environment reaches a high level and tends to stabilize, it indicates that the model has the ability to perform tasks in a real-world environment, achieving the goal of deep imitation learning. Through this stopping condition design that comprehensively considers the loss function, training steps, and task completion rate, the network can effectively learn the complex mapping relationship from visual data and taught robotic arm joint state data to the predicted execution robotic arm joint state at the next time step. This enables robots to perform fine manipulation tasks even under low-precision hardware conditions, greatly expanding the application scope and potential of robotics technology.

[0047] The end-to-end action generation loss function is described below. The end-to-end action generation loss function includes action prediction loss, cross-entropy loss, and KL divergence loss, as shown in the following formulas:

[0048] ,

[0049] ,

[0050] ,

[0051] ,

[0052] The motion prediction loss is used to measure the difference between the joint motor states predicted by the VAE decoder and the actual joint motor states; here, the mean squared error is chosen. ,in This is the actual state of the joint motor. It is a predicted joint motor state. It's the joint motor number. This refers to the number of joint motors; the cross-attention mechanism loss is used to measure the rationality of the VAE decoder's attention weights on the input data during prediction, and here cross-entropy loss is used. To optimize attention weights and ensure the model can focus more accurately on key information, It is the actual attention weight. It is the predicted attention weight. It is the number of data points. It's about the number of heads to focus on. It is the data point number. It is the attention head number; KL divergence loss This is used to ensure that the distribution of the latent representation vectors approximates the pre-defined prior distribution, which helps the model learn a smoother and more continuous latent space. and These are the mean and standard deviation of the latent representation vector, respectively. It is the dimension of the latent representation vector. This is the dimension number of the latent representation vector. Finally, the comprehensive loss function combines all the above loss functions to form an end-to-end action generation loss function, ensuring the model performs well in multiple aspects. The comprehensive loss function is constructed using a weighted sum. ,in and These are hyperparameters used to balance the contributions of different loss functions.

[0053] S140 involves transferring the fully trained and converged deep imitation learning network from a virtual simulation scenario to a real-world dual-arm robot. This step is crucial for verifying the model's ability to control the dual-arm robot to perform specific tasks in the real world, and for confirming the correctness of the strategy. Figure 6 As shown. The following are the detailed operating steps:

[0054] S140-1, the deep imitation learning network, completed all necessary training and testing processes in a virtual simulation environment and achieved the expected performance metrics. This means the model needs to demonstrate superior visual servo motion planning capabilities in the simulation environment, accurately generating and executing motion planning instructions to guide the robotic arm to complete various complex operational tasks. Furthermore, the model needs to undergo extensive simulated task testing in the simulation environment to verify its adaptability and stability under different task scenarios. These simulated tasks may include various types such as object grasping and placement, precision assembly, path planning and obstacle avoidance, etc. Performance in these tasks allows for a comprehensive evaluation of the model's overall performance. In determining model convergence, close attention must be paid to the decreasing trend of the model's loss function, the changes in accuracy on the training and validation sets, and the model's actual performance in the simulation tasks. When the model's loss function steadily decreases and tends to plateau, the accuracy on the training and validation sets reaches a high level with little difference, and the model's execution in the simulation tasks is stable and reliable, successfully completing all task objectives, the model can be considered to have converged and possess the basic conditions for migration from the simulation environment to the real-world application environment. At this point, the model has strong generalization ability and robustness, and can still maintain good performance when facing complex situations and uncertainties in the real environment, laying a solid foundation for subsequent practical application migration.

[0055] S140-2, Hardware deployment is a crucial step in successfully applying an AI model to a practical dual-arm robot system. First, the trained deep imitation learning network needs to be integrated into the robot's control software to ensure it can run and perform tasks in the actual hardware environment. This typically involves appropriate optimization and tuning of the model to adapt to the robot system's hardware resource limitations and real-time requirements. Next, hardware interface configuration is critical. The deep imitation learning network must accurately receive actual visual data from the robot's sensors. This includes interfacing with visual sensors such as cameras and depth sensors to transmit the image data acquired by the sensors to the deep imitation learning network for processing in real time. Simultaneously, necessary preprocessing of the sensor data is required, such as image denoising, resizing, and color space conversion, to meet the input data requirements of the deep imitation learning network. Furthermore, the deep imitation learning network needs to be able to send generated control commands to the robot's actuators, such as motor drivers and servo controllers. This involves interfacing with the robot's motion control system to ensure that the model's output control signals can be accurately parsed and executed. Detailed configuration of the control signal format, transmission protocol, and timing is required to achieve efficient collaboration between the model and the robot's actuators. During hardware deployment, thorough testing of the entire system's stability and reliability is also required. Through multiple trial runs and debugging in a real-world environment, potential hardware interface issues, data transmission delays, and control command execution errors are identified and resolved promptly. This ensures the AI ​​model can operate stably and accurately perform various tasks within the actual dual-arm robot system. Through this series of hardware deployment efforts, the AI ​​model is successfully migrated from the simulation environment to real-world applications, providing strong support for the robot's intelligent and automated capabilities in complex tasks.

[0056] S140-3, Real-world scenario testing, verifies the performance of deep imitation learning networks in real-world task scenarios. A dual-arm robot is deployed in a real-world working environment to perform a series of specific tasks similar to those in the simulation environment, allowing for direct comparison and evaluation of the deep imitation learning network's performance. These tasks typically involve complex operations such as object grasping, handling, and assembly—common key task types for robots in real-world applications. In grasping tasks, the robot needs to accurately identify the position and shape of the target object, plan a suitable grasping path and posture, and then control the robotic arm and grippers to precisely grasp the object. Handling tasks require the robot to move the object quickly and smoothly to a designated location along a predetermined path while maintaining object stability. Assembly tasks are more complex; the robot must not only accurately identify and grasp each assembly component but also accurately place the components in their corresponding positions according to the assembly sequence and requirements, and perform operations such as fastening and connecting. By performing these tasks in real-world scenarios, the robot's performance in a real-world environment can be observed and recorded, including the accuracy, efficiency, stability, and ability to handle unexpected situations. Simultaneously, a large amount of real-world operational data can be collected for subsequent model optimization and improvement. Real-world scenario testing can not only verify the training results of the model in the simulation environment, but also reveal the problems and shortcomings that the model may have in practical applications, providing important reference for the further improvement and enhancement of the model.

[0057] S140-4 utilizes a deep imitation learning network to generate visual servo motion planning instructions. Based on real-time visual feedback, the deep imitation learning network generates precise visual servo motion planning instructions. These instructions guide the dual-arm robot on how to adjust its movements to adapt to environmental changes and task requirements, thereby accurately completing the specified task. The model needs to comprehensively consider factors such as the relative position between the robot and the target object, the object's posture changes, and obstacles in the surrounding environment to generate reasonable motion trajectories and action sequences, ensuring the robot can smoothly and accurately execute the task. The execution and monitoring phase follows, executing the generated motion planning instructions in a real-world scenario. This requires real-time monitoring of the robot's movements, closely monitoring its trajectory, speed, acceleration, and interaction with the environment. The purpose of monitoring is to ensure that the robot's movements conform to the expected plan and can execute the task within a safe range. Through monitoring, potential anomalies can be detected and addressed promptly, such as excessive robot movement deviations or the risk of collisions with obstacles, thereby ensuring the smooth progress of the task and the safe operation of the equipment. Performance verification is a crucial step in evaluating the model's effectiveness in practical applications. By comparing the robot's performance in actual tasks with its performance in a simulation environment, the correctness and feasibility of the model are comprehensively verified. This includes evaluating the accuracy of task completion, such as the success rate of object grasping and the precision of assembly; evaluating the efficiency of task execution, such as the time required to complete the task and the robot's movement speed; and evaluating the safety of robot operation, such as whether dangerous collisions with the environment are avoided and whether stability is maintained during operation. Through performance verification, the advantages and disadvantages of the model in practical applications can be quantified, providing an important basis for subsequent model optimization and improvement, and ensuring that the artificial intelligence model can reliably and efficiently guide the dual-arm robot to complete various tasks in real-world scenarios.

[0058] S140-5 verifies the correctness of the deep imitation learning network by comparing the robot's performance in actual tasks with its performance in a simulation environment. Based on the task execution results in the real-world scenario, analyze the model's performance and make necessary adjustments and optimizations. Requires in-depth analysis of the model's performance metrics, such as task completion accuracy, efficiency, stability, and adaptability to environmental changes, based on detailed data collected and observed during the dual-arm robot's actual task execution. By comparing the gap between actual performance and expected goals, identify potential shortcomings of the model in practical applications, such as decreased perception accuracy in complex environments, insufficient flexibility in motion planning, and slow response to unexpected situations. Address these identified problems with necessary adjustments and optimizations. This may include fine-tuning model parameters, such as adjusting the neural network weights, learning rate, and regularization terms, to better fit the actual data and task requirements, improving the model's prediction accuracy and generalization ability. Simultaneously, improvements may be required in robot control strategies. For example, optimizing motion planning algorithms to generate smoother, more efficient trajectories; enhancing the robot's perception fusion capabilities to more accurately integrate information from different sensors; or introducing more advanced decision-making mechanisms to better handle complex and ever-changing task scenarios and unforeseen circumstances. Furthermore, the model's architecture and functionality can be expanded and upgraded based on feedback and needs from practical applications. For instance, adding support for new task types allows the robot to perform more diverse operations; or improving the model's real-time performance enables faster responses to environmental changes and task commands, meeting the speed requirements of real-world applications. Through continuous result analysis and optimization, the artificial intelligence model will be continuously improved and enhanced, ultimately achieving efficient, stable, and intelligent operation of dual-arm robots in real-world scenarios.

[0059] In summary, firstly, a semi-physical simulation platform was used to execute an experimental task of a visual servoing dual-arm robot, acquiring visual and robotic arm data for the scene task. Secondly, the collected dual-arm robot operation data was replayed, key actions were selected, and action smoothing was optimized in a pure simulation experimental environment to obtain accurate and feasible data for the dual-arm robot in the simulation experimental scene, and to construct the dataset required for model training. Then, based on the constructed dataset, an artificial intelligence model was trained in a visual servoing deep imitation learning network. The training effect of the artificial intelligence model was evaluated by the loss function during the model training process and the accuracy of the simulation scene verification task, so as to obtain an artificial intelligence model with good generalization ability. Finally, the trained artificial intelligence model was transferred to a dual-arm robot in a real scene to realize the generation and execution of visual servoing motion planning instructions for specific tasks.

[0060] First, a virtual simulation scenario is constructed based on the real-world task scenario. A real-world teaching robotic arm or gripper is used to control a virtual robotic arm to perform the task, acquiring data from each joint of the robotic arm in the virtual world and data from visual sensors during task execution. Optionally, when constructing the virtual simulation scenario based on the real-world task scenario, the method for obtaining the 3D model includes: algorithms and software for automatically constructing 3D models based on text, photos, and videos of real objects from multiple angles, or using modeling software to autonomously construct the 3D model.

[0061] Secondly, the collected dual-arm robot operation data was replayed in a pure simulation experimental environment to observe the task completion. A key action selection algorithm was used to automatically select key action nodes based on the changes in the motor angle values ​​of each axis of the robotic arm. Then, smooth data generated by the updated key action nodes was replayed in the pure simulation experimental environment. Key action nodes were manually selected, and then smooth data generated by the manually optimized key action nodes was replayed in the pure simulation experimental environment to ensure the rationality of the actions for the task. The manual selection and replay of data were repeated until it was ensured that the actions generated by the selected key action nodes could complete the task. Accurate and feasible data of the dual-arm robot in the simulation experimental scenario were obtained. Combined with visual data obtained in the pure simulation experimental environment, the dataset required for model training was constructed.

[0062] Next, based on an end-to-end strategy, a deep imitation learning network is designed by combining a variational autoencoder (VAE) and a Transformer architecture. The network takes visual data and the motor state data of each joint of the robotic arm as input, and outputs the predicted motor state of each joint of the robotic arm at the next time step. The loss between the actual and predicted values ​​is calculated, and the network is trained according to the loss and training epochs until convergence. The model is validated in a pure simulation environment at regular intervals to obtain the task completion rate to help determine whether the network has converged. Furthermore, the deep imitation learning design includes: combining the contextual feature acquisition characteristics of the Transformer architecture to design a VAE encoder and VAE decoder using a VAE architecture. The VAE encoder is primarily a Transformer encoder, whose input includes the motor state (joint position) of each joint of the dual-arm robot at time t and the action sequence (target joint position) from time t to t+k. The encoder input also includes a special [CLS] tag, which is used to aggregate sequence information. In addition, the joint positions and action sequences are mapped to a high-dimensional space through a linear embedding layer so that the Transformer encoder can process them. In this way, the encoder learns the complex patterns and implicit features of joint movements when the robot performs fine manipulation tasks, and encodes these features into a style variable z for the robotic arm operation. The VAE decoder receives the style variable z, the motor states of each joint of the dual-arm robot at time t, and RGB images from multiple perspectives as input visual data. This visual data can originate from any type of visual sensor; RGB images from multiple perspectives are used as an example here. The image data is processed through a ResNet18 network and sinusoidal position embedding to extract spatial features and map joint positions to a high-dimensional embedding space. This embedded data, along with the style variable z, is fed into the Transformer encoder to capture deep information. Subsequently, the data is passed to the Transformer decoder, which uses a cross-attention mechanism to predict the motor states of each joint of the dual-arm robot at time t+1. The final stopping condition for model training is that the end-to-end motion generation loss function decreases to a preset target threshold, or the total number of alternating iterations exceeds a set value, and the convergence of the model is verified in a pure simulation environment through task completion rate.

[0063] Finally, the converged artificial intelligence model was transferred to a dual-arm robot in a real-world scenario to generate and execute visual servo motion planning instructions for specific tasks, verifying the correctness of the model in controlling the dual-arm robot to perform tasks in a real-world scenario.

[0064] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality, characterized in that, Includes the following steps: S110, construct a virtual simulation scenario based on the real task scenario, and establish the connection between the real task scenario and the virtual simulation scenario; S120 replays and optimizes the operation data of the teaching robot in the real task scenario and the execution robot in the virtual simulation scenario in the virtual simulation scenario to construct the dataset used for final training. S130, Design and implement a deep imitation learning network. This deep imitation learning network adopts an end-to-end strategy and combines variational autoencoder and Transformer architecture. The input of the deep imitation learning network includes visual data obtained from real task scenarios and motor state data of each joint of the taught robotic arm. The output is the predicted motor state of each joint of the executing robotic arm in the virtual simulation scenario at the next time step. S140 transfers a well-trained and converged deep imitation learning network from a virtual simulation scenario to a dual-arm robot in a real task scenario. S120 includes the following steps: S120-1, in a virtual simulation scenario, replays the collected data of the teaching robot arm and the execution robot arm operation; S120-2, in the operation data of the teaching robot arm in the real task scenario and the execution robot arm in the virtual simulation scenario, the key action selection algorithm automatically selects the key action nodes. S120-3, Based on the key action nodes selected in step S120-2, continue to manually select key action nodes; S120-4, using step S120-3, the updated key motion nodes are manually selected to generate smooth operation data for the motors of each joint of the teaching robot arm; S120-5, Based on the visual data obtained from the virtual simulation scene and the smooth operation data of each joint motor of the teaching robot generated in step S120-4, construct a dataset.

2. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 1, characterized in that, Step S110 includes the following steps: S110-1: Obtain the 3D models of all objects in the real task scene and configure them as a virtual simulation scene; S110-2, Establish the connection between the teaching robotic arm in the real task scenario and the execution robotic arm in the virtual simulation scenario; S110-3 After establishing the connection between the teaching robot arm in the real task scenario and the execution robot arm in the virtual simulation scenario, the operator uses the teaching robot arm to precisely control the execution robot arm in the virtual simulation environment to perform a series of specific tasks.

3. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 2, characterized in that, Methods for obtaining 3D models of all objects include: using 3D modeling automation technology, which utilizes algorithms and software to automatically construct 3D models based on text, photos, and video information from multiple angles of the object; 3D modeling automation technology includes one of structured light scanning, stereo vision, and motion reconstruction structures; or using professional modeling software for professionals to manually construct 3D models based on blueprints or photos of the object; or using the Mujoco physics engine to build virtual simulation scenes.

4. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 2, characterized in that, Step S110-2 includes the following steps: S110-2a, through the software development kit (SDK) of the robotic arm, uses Python to achieve real-time acquisition and synchronization of motion data of the taught robotic arm in real task scenarios; S110-2b, Python transmits the acquired motion data to the Mujoco virtual simulation scene; S110-2c, Mujoco drives the robotic arm in the virtual simulation scene to perform corresponding operations based on the received motion data.

5. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 1, characterized in that, In step S120-2, the process of the key action selection algorithm includes: first, selecting the first action node in the operation process as the key action node; then, judging the direction of change of the motor angle value when the current key action node is in motion by the adjacent action nodes; if the angle value decreases, using the sliding window method to find the local minimum point as the key action node; after finding the local minimum, finding the local maximum point as the key action node; repeating this process until the last action node, and then using the last action node as the key action node; if the angle value increases, using the sliding window method to find the local maximum point as the key action node; after finding the local maximum, finding the local minimum point as the key action node; repeating this process until the last action node, and then using the last action node as the key action node.

6. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 1, characterized in that, Step S130 includes the following steps: S130-1, with the Transformer encoder as the core, constructs a feature extraction framework as a variational autoencoder. The input data of the variational autoencoder covers the motor states of each joint of the dual-arm robot at a specific time t, including joint position information, and a series of action sequences from time t to t+k. The joint position information and action sequences are first mapped to a high-dimensional space through a linear embedding layer. This mapping process transforms the original low-dimensional data into an embedding dimension representation suitable for the operation of the Transformer encoder. In the embedding dimension, the Transformer encoder learns the complex patterns and implicit features of the joint motion of the robot when performing fine manipulation tasks. Finally, the Transformer encoder encodes the rich feature information extracted from the input data into a latent representation vector, and then projects it into a formal variable z through a linear layer. S130-2, the VAE decoder design integrates formal variables z, the motor states of each joint of the dual-arm robot at time t, and RGB images from multiple perspectives as visual data input: First, the input RGB images are processed by a ResNet18 network for deep feature extraction, converting the rich visual information in the images into feature maps; then, the feature maps are flattened and mapped to a high-dimensional hidden dimension space. Simultaneously, the motor states of each joint of the dual-arm robot at time t are projected through a linear layer. Then, the mapping result of the feature map is concatenated with the linear layer projection results of the formal variables z and the motor states, serving as the input vector of the Transformer encoder. Next, the data output by the Transformer encoder is passed to the Transformer decoder. The Transformer decoder uses a cross-attention mechanism, combining the output and input data of the Transformer encoder, to predict the robot's action sequence over a future period, and uses the action sequence to weightedly obtain the motor states of each joint at time t+1. S130-3, design the training termination condition to ensure that the deep imitation learning network can fully learn and master the mapping relationship from visual data and taught robot arm joint motor state data to the predicted virtual simulation scene of the executing robot arm joint motor state at the next moment.

7. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 6, characterized in that, The training termination condition in step S130-3 includes the end-to-end action generation loss function decreasing to a preset target threshold. The end-to-end action generation loss function includes action prediction loss, cross-entropy loss, and KL divergence loss, as shown in the following formula: ， ， ， ， Action prediction loss This is used to measure the difference between the joint motor states predicted by the VAE decoder and the actual joint motor states, where This is the actual state of the joint motor. It is a predicted joint motor state. It's the joint motor number. It refers to the number of joint motors; cross-entropy loss. This is used to measure the reasonableness of the attention weights applied to the input data by the VAE decoder during the prediction process, whereby... It is the actual attention weight. It is the predicted attention weight. It is the number of data points. It's about the number of heads to focus on. It is the data point number. It refers to the attention head number; KL divergence loss. This is used to ensure that the distribution of the latent representation vectors approximates a pre-defined prior distribution, where and These are the mean and standard deviation of the latent representation vector, respectively. It is the number of dimensions of the latent representation vector. This refers to the dimension number of the latent representation vector. Ultimately, the comprehensive loss function combines the action prediction loss, cross-entropy loss, and KL divergence loss to form an end-to-end action generation loss function, using a weighted sum to construct the comprehensive loss function. ,in and These are hyperparameters used to balance the contributions of different loss functions.

8. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 1, characterized in that, Step S140 includes the following steps: S140-1, the deep imitation learning network completes the training and testing process in a virtual simulation scenario and converges; S140-2, Hardware Deployment: Integrate the trained deep imitation learning network into the robot's control software to ensure that the deep imitation learning network can run and perform tasks in a real hardware environment; S140-3, Real-world scenario testing, verifying the performance of deep imitation learning networks in real-world task scenarios: Deploying a dual-arm robot in a real working environment to perform a series of specific tasks similar to those in a virtual simulation environment, so as to directly compare and evaluate the performance of deep imitation learning networks. S140-4 utilizes a deep imitation learning network to generate visual servo motion planning instructions; S140-5 verifies the correctness of the deep imitation learning network by comparing the robot's performance in actual tasks with its performance in a virtual simulation environment.

9. The method for transfer and imitation learning of a visual servoing dual-arm robot from simulation to reality as described in claim 8, characterized in that, Step S140-5 includes: analyzing the performance metrics of the deep imitation learning network based on the detailed data collected and observed performance of the dual-arm robot during actual task execution, including the accuracy, efficiency, stability of task completion, and adaptability to environmental changes. By comparing the actual performance with the expected goals, we can identify the potential shortcomings of deep imitation learning networks in practical applications and make adjustments and optimizations, including adjusting the weights, learning rate, and regularization terms of the deep imitation learning network.

Images