A Robotic Arm Joint Trajectory Tracking Method Based on Virtual-Real Fusion
By fusing virtual environment models with scene images, and utilizing neural networks and triangulation, the problem of dependence on precise kinematic models and low calibration efficiency in robotic arm joint trajectory tracking was solved, achieving efficient and accurate joint trajectory tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA JIAOTONG UNIVERSITY
- Filing Date
- 2026-06-09
- Publication Date
- 2026-07-10
Smart Images
Figure CN122353632A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for tracking the joint trajectory of a robotic arm based on virtual-real fusion. Background Technology
[0002] With the accelerated advancement of industrial intelligence, robots have gradually developed into core and ubiquitous equipment in modern intelligent manufacturing systems. As a key execution carrier in intelligent manufacturing, the strategic importance of robot technology is becoming increasingly prominent, while also facing both opportunities and challenges in technological breakthroughs and scenario adaptation. As the core carrier for achieving human-robot collaborative operations, the core requirement of robots is not only to complete high-precision operational tasks, but also to ensure safe and natural interaction with humans and the working environment. The primary prerequisite for achieving safe and reliable human-robot collaborative operations is the accurate acquisition of the spatial trajectory of each joint of the robot, as well as its relative positional relationship with the surrounding environment.
[0003] Current mainstream robotic arm joint trajectory tracking methods suffer from significant technical bottlenecks: the accuracy of traditional trajectory tracking schemes heavily relies on the construction of precise robot kinematic models, and model errors are easily transmitted directly to the trajectory tracking results. Traditional vision-based spatial positioning schemes require a cumbersome hand-eye calibration process to establish a rigid transformation relationship between the camera coordinate system and the robot's base coordinate system, resulting in low calibration efficiency and susceptibility to environmental interference. Although many researchers have used intelligent algorithms such as neural networks to optimize traditional methods, these schemes still rely on large-scale field dataset collection and complex annotation work. The gap between accurately labeled data in simulation and reality, along with poor generalization ability, not only increases the cost of algorithm development but also limits their widespread application in lightweight communication devices and resource-constrained scenarios.
[0004] Therefore, it is necessary to provide a new method for tracking the motion trajectory of robotic arms to improve the efficiency of accurately tracking the motion trajectory of robotic arms. Summary of the Invention
[0005] The purpose of this invention is to provide a method for tracking the joint trajectory of a robotic arm based on virtual-real fusion, so as to improve the efficiency of accurately tracking the motion trajectory of the robotic arm.
[0006] The present invention provides a robotic arm joint trajectory tracking method based on virtual-real fusion, comprising: simulating robotic arm operation motion using a simulation environment model and acquiring simulation images; acquiring scene images of the robotic arm operation scenario and real images of the robotic arm operating in the actual scenario; performing image segmentation on the simulation images to obtain virtual robotic arm images; fusing the virtual robotic arm images with the scene images to generate a virtual-real fusion image; iteratively training a neural network for segmentation tasks based on the virtual-real fusion image to obtain a robotic arm target segmentation model; applying the robotic arm target segmentation model to segment the real images to obtain real robotic arm images; generating key point annotation data corresponding to the robotic arm joint centers based on the simulation images; constructing a key point detection neural network based on the heatmap method and performing segmentation based on the virtual robotic arm images and key point annotation data. A keypoint detection neural network model was trained and used to process real robotic arm images to obtain heatmaps corresponding to each keypoint. Peak detection was performed on each heatmap to obtain the pixel coordinates of each keypoint in the real image. Two real images from different perspectives were acquired and processed by the keypoint detection neural network model to obtain the first and second heatmaps corresponding to different perspectives. For each keypoint of the robotic arm, the spatial coordinates of the keypoint were solved by triangulation based on the coordinates of the keypoint in the first and second heatmaps. A world coordinate system was established, and the spatial coordinates of the keypoints were converted into three-dimensional coordinates in the world coordinate system. The joint trajectory of the robotic arm was tracked based on the three-dimensional coordinate sequence of each keypoint of the robotic arm in the world coordinate system obtained in continuous time sequence.
[0007] The beneficial effects of the robotic arm joint trajectory tracking method based on virtual-real fusion provided by this invention are: it can get rid of the dependence on precise kinematic models, avoid tedious hand-eye calibration, and does not require on-site image collection and annotation during processing, which can improve processing efficiency and has strong generalization ability, and can quickly and accurately realize robotic arm motion trajectory tracking.
[0008] The process of fusing virtual robotic arm images with scene images to generate a virtual-real fusion image includes: performing pixel-by-pixel conditional judgments on the virtual robotic arm image and the scene image to retain the robotic arm pixels in the simulation image, and replacing the remaining pixels in the simulation image with random background; at the same time, copying the original robotic arm label information in the simulation image to keep the position information unchanged.
[0009] The process of segmenting a simulated image to obtain a virtual robotic arm image includes: constructing a first neural network; dividing a portion of the acquired simulated image into training samples to train the first neural network to obtain a virtual image segmentation model; and applying the virtual image segmentation model to segment the simulated image to obtain a virtual robotic arm image.
[0010] The iterative training of a neural network based on virtual-real fusion images to obtain a target segmentation model for a robotic arm includes: constructing a second neural network with the same architecture as the first neural network; transferring the convolutional layer weights in the virtual image segmentation model to the corresponding layers of the second neural network for parameter initialization; using the virtual-real fusion images as input samples to perform iterative training of the parameter-initialized second neural network for image segmentation until the loss output of the iterative training does not decrease for multiple consecutive rounds, thus obtaining the target segmentation model for the robotic arm; wherein each round of iterative training includes: sampling a batch of virtual-real fusion images, performing image segmentation processing to obtain a prediction mask and loss value, updating the network parameters through backpropagation, and outputting the updated network parameters and the current batch loss after the current round of iterative training.
[0011] Based on the simulation image, key point annotation data corresponding to the joint center of the robotic arm is generated, including: using the rotation joint center of the robotic arm as the key point of the robotic arm; the key point annotation data includes the normalized coordinates of the bounding box of the target area of the robotic arm, the pixel-level segmentation mask of the overall area of the robotic arm, the normalized coordinates of the key points and their visibility status.
[0012] A keypoint detection neural network based on heatmap method is constructed and trained using virtual robotic arm images and keypoint annotation data as supervision signals. The process includes: constructing a keypoint detection neural network based on heatmap method, using the virtual robotic arm image as input and the corresponding keypoint annotation data as supervision signals; calculating the probability response value of any pixel in the virtual robotic arm image belonging to a keypoint for each keypoint in the virtual robotic arm image, thereby generating a predicted heatmap corresponding to the keypoint, and measuring the difference between the predicted heatmap and the actual heatmap; wherein the probability response value is the Euclidean distance calculated based on the coordinates of the pixel in the virtual robotic arm image and the corresponding keypoint annotation data; and defining the loss function as follows. ,in, Indicates the total number of key points. Indicates the first Predictive heatmaps for key points Indicates the first Real heat maps of key points The Frobenius norm is represented by the keypoint detection neural network model, which is obtained when the loss function converges.
[0013] For each keypoint of the robotic arm, the spatial coordinates of the keypoint are determined using triangulation based on its coordinates in the first and second heatmaps. This includes defining the camera that captures the real image corresponding to the first heatmap as the first camera, and the camera that captures the real image corresponding to the second heatmap as the second camera. The intrinsic parameters of the first camera are determined through camera calibration. The internal parameters of the second camera are The transformation matrix of the second camera relative to the first camera is: ;No. The coordinates of the key points in the first heat map are as follows: The coordinates in the second heat map are , , The relationship between the spatial coordinates of the key points and the key points satisfies the following formula: , ,in, and Represents the distance coefficient. Indicates the first Spatial coordinates of key points; construct a system of linear equations by simultaneously solving the two equations. ,in, , , These represent the first, second, and third row vectors of the first camera intrinsic parameter matrix, respectively. , , Let represent the first, second, and third row vectors of the second camera intrinsic parameter matrix, respectively. Solve the system of linear equations to obtain the spatial coordinates of the key points.
[0014] Establish a world coordinate system and convert the spatial coordinates of the key points obtained from the solution into three-dimensional coordinates in the world coordinate system. This includes selecting the base joint of the robotic arm as the origin of the world coordinate system. And establish a world coordinate system; based on the transformation relationship Convert spatial coordinates to three-dimensional coordinates in the world coordinate system. Indicates the first The three-dimensional coordinates of the key points in the world coordinate system Indicates the first Spatial coordinates of key points.
[0015] The joint trajectory of the robotic arm is tracked based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system obtained under continuous time sequence, including: calculating the displacement trajectory, velocity change and motion posture of each key point of the robotic arm based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system. Attached Figure Description
[0016] Figure 1 A flowchart illustrating a robotic arm joint trajectory tracking method based on virtual-real fusion, provided for an embodiment of the present invention; Figure 2 This is a schematic diagram of image enhancement based on virtual-real fusion of virtual robotic arm image and scene image provided in an embodiment of the present invention; Figure 3A schematic diagram of target segmentation for a robotic arm based on transfer learning, provided for an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the key point definition of a robotic arm provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of key point detection based on robotic arm images provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the error variation curve between the actual and predicted values of key point detection provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the triangulation principle based on key point pixel coordinates provided in an embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but do not exclude other elements or objects.
[0018] This embodiment provides a method for tracking the joint trajectory of a robotic arm based on virtual-real fusion. See also Figures 1 to 5 The method includes: S101: Use a simulation environment model to simulate the operation and movement of the robotic arm and collect simulation images, including scene images of the robotic arm's operation scenario and real images of the robotic arm operating in the actual scenario.
[0019] In one possible embodiment, simulating the robotic arm's operational motion and acquiring simulation images using a simulation environment model includes: constructing a simulation environment model based on a digital engine, manipulating the robotic arm in the simulation environment model to mimic actual operational motion, and acquiring RGB images of different postures using a virtual external camera to obtain simulation images.
[0020] Acquiring scene images includes capturing images of the actual working environment of the robotic arm from different perspectives, while acquiring real images includes capturing images of the robotic arm performing operations in the actual working environment.
[0021] S102: Perform image segmentation on the simulation image to obtain a virtual robotic arm image. Fuse the virtual robotic arm image with the scene image to generate a virtual-real fusion image. Iteratively train the neural network for segmentation tasks based on the virtual-real fusion image to obtain a robotic arm target segmentation model. Apply the robotic arm target segmentation model to segment the real image to obtain a real robotic arm image.
[0022] In one possible embodiment, the process of segmenting a simulated image to obtain a virtual robotic arm image includes: constructing a first neural network; dividing a portion of the acquired simulated image into training samples to train the first neural network to obtain a virtual image segmentation model; and applying the virtual image segmentation model to segment the simulated image to obtain a virtual robotic arm image.
[0023] The iterative training of a neural network based on virtual-real fusion images to obtain a target segmentation model for a robotic arm includes: constructing a second neural network with the same architecture as the first neural network; transferring the convolutional layer weights in the virtual image segmentation model to the corresponding layers of the second neural network for parameter initialization; using virtual-real fusion images as input samples to perform iterative training of the parameter-initialized second neural network for image segmentation until the loss output of the iterative training does not decrease for multiple consecutive rounds, thus obtaining the target segmentation model for the robotic arm; wherein, each round of iterative training includes: sampling a batch of virtual-real fusion images, performing image segmentation processing to obtain a prediction mask and loss value, updating the network parameters through backpropagation, and outputting the updated network parameters and the current batch loss after the current round of iterative training.
[0024] Training virtual image segmentation models and robotic arm target segmentation models using the same network architecture allows for transfer learning without constructing new neural networks, resulting in different segmentation models. In one specific embodiment, this same network architecture is the CSAK-YOLOv8 neural network architecture.
[0025] In one specific embodiment, a CSAK-YOLOv8 neural network is constructed. The acquired simulation images are input into the neural network for training. The loss function is defined as a bounding box regression loss function, specifically satisfying the following formula: ,in, The intersection-union ratio (IU) is an indicator that measures the degree of overlap between two bounding boxes. This represents the Euclidean distance between the center point of the predicted bounding box and the center point of the ground truth bounding box; This represents the diagonal length of the minimum bounding rectangle between the predicted bounding box and the true bounding box. This represents the Euclidean distance between the top-left corner of the ground truth bounding box and the predicted bounding box. This represents the Euclidean distance between the bottom right corner of the ground truth bounding box and the predicted bounding box. Indicates the width of the input image; This represents the height of the input image; training is complete when the loss function converges, generating a virtual image segmentation model. Loss function The design effectively solves the problem of failure when the aspect ratio is the same but the actual size is different or the center point coincides by simultaneously considering the distance between the top left and bottom right corners of the predicted bounding box and the real bounding box, and combining the input image size as a reference, thereby improving the convergence speed and positioning accuracy.
[0026] See Figure 2 and Figure 3 The virtual image segmentation model is applied to perform image segmentation on the simulation images not used for model training to obtain a virtual robotic arm image containing only the target region (robotic arm).
[0027] The acquired scene images are scaled uniformly to ensure they match the virtual image size, guaranteeing accurate foreground label information for the robotic arm during image fusion. Then, the virtual robotic arm image and the scene image are randomly fused to generate a fused virtual-real image. The specific fusion process includes: based on the pixel-level mask of the robotic arm extracted through a virtual segmentation model, pixel-by-pixel conditional judgment is used to retain robotic arm pixels in the virtual image, replacing the remaining pixels with random background. Simultaneously, the original robotic arm label information is directly copied to maintain its positional information. The fusion process can be expressed by the formula: , Represents a fused image of reality and virtuality; Represents a virtual robotic arm image; This is a binary mask representing the pixel positions within the robotic arm region. Its value is: within the robotic arm region In the background area ; This represents a scaled-down scene image; This represents a pixel-by-pixel multiplication operation. The above-mentioned random identical fusion processing, by randomly selecting different fusion positions, can generate diverse virtual-real fusion image samples while keeping the robotic arm label information unchanged, effectively expanding the training dataset and improving the model's adaptability to real-world scenes.
[0028] A CSAK-YOLOv8 neural network was constructed, using a virtual image segmentation model as the pre-trained feature extraction backbone. The convolutional layer weights from the virtual image segmentation model were transferred to the corresponding layers in the newly constructed CSAK-YOLOv8 neural network for parameter initialization. Then, iterative training for image segmentation was performed using fused virtual and real images as input samples until the loss output from the iterative training did not decrease for multiple consecutive rounds, thus obtaining the target segmentation model for the robotic arm. The specific processing for each round of iterative training included: sampling a batch of fused virtual and real images, obtaining the prediction mask and loss value through forward propagation, updating the network parameters through backpropagation, and finally outputting the updated model parameters and the current batch loss.
[0029] In one possible embodiment, the training can be terminated when the loss of the iterative training output of the image segmentation task using the virtual-real fusion image as input sample does not decrease for 20 consecutive rounds, thus obtaining the target segmentation model for the robotic arm.
[0030] The training of the robotic arm target segmentation model uses the parameters of a virtual image segmentation model as pre-training parameters. It leverages the general visual features inherent in the virtual image segmentation model, such as edges, textures, and simple geometric shapes, enabling rapid adaptation to the representation learning of the robotic arm target region and gradually enhancing its ability to discriminate the foreground of the simulated robotic arm under real-world background interference. Furthermore, a progressive domain adaptation training strategy is employed to optimize the neural network in image domains from different sources: the first domain is the pure simulated image domain, i.e., the virtual domain, where training is completed within the virtual image segmentation model, achieving an idealized feature extraction capability for the robotic arm; the second domain is the virtual-real fusion image domain, i.e., the fusion domain, where the parameters trained by the virtual image segmentation model are used as the initial parameters for further optimization training in the fusion domain, ultimately yielding the final robotic arm target segmentation model. Through this progressive optimization process, the segmentation network gradually transitions from an ideal simulation environment to a more realistic fusion environment, ultimately constructing a high-performance and robust robotic arm target segmentation model. This progressive domain adaptation training can be formally represented as: ,in, Indicates the optimal network parameters; and These represent the simulated image and its corresponding segmentation label; and These represent the fused virtual and real images and their corresponding labels; The parameter is Neural networks; The loss function representing a neural network; This represents the inter-domain balance coefficient, used to adjust the contribution weight of virtual domain and fused domain losses to network optimization.
[0031] The final robotic arm target segmentation model can be directly transferred and deployed to real robotic arm operation scenarios to segment real images and obtain real robotic arm images without the need for additional fine-tuning or re-annotation for specific scenarios.
[0032] In practical applications of target segmentation models for robotic arms, real-time acquired images are used. As input, output the corresponding real robotic arm image. The expression is as follows: Threshold segmentation is used to accurately extract the target region of the robotic arm contour in the real environment from real images.
[0033] S103: Generate key point annotation data corresponding to the joint center of the robotic arm based on the simulation image, construct a key point detection neural network based on the heat map method, and train it based on the virtual robotic arm image and key point annotation data to obtain the key point detection neural network model. The key point detection neural network model is used to process the real robotic arm image to obtain the heat map corresponding to each key point, and perform peak detection on each heat map to obtain the pixel coordinates of each key point in the real image.
[0034] In one possible embodiment, a keypoint detection neural network based on heatmap method is constructed and trained using virtual robotic arm images and keypoint annotation data as supervision signals to obtain a keypoint detection neural network model. This includes: constructing a keypoint detection neural network based on heatmap method, using the virtual robotic arm image as input and the corresponding keypoint annotation data as supervision signals; calculating the probability response value of any pixel in the virtual robotic arm image belonging to a keypoint for each keypoint in the virtual robotic arm image, thereby generating a predicted heatmap corresponding to the keypoint, and measuring the difference between the predicted heatmap and the actual heatmap; wherein the probability response value is the Euclidean distance calculated based on the coordinates of the pixel in the virtual robotic arm image and the corresponding keypoint annotation data; and defining the loss function as... ,in, Indicates the total number of key points. Indicates the first Predictive heatmaps for key points Indicates the first Real heat maps of key points The Frobenius norm is represented by the keypoint detection neural network model, which is obtained when the loss function converges.
[0035] In one specific embodiment, simulation images are acquired in batches and virtual robotic arm images are segmented in the simulation environment model, and key point annotation data for the corresponding virtual robotic arm images are generated simultaneously. See the appendix to the specification. Figure 4 The key points specifically refer to the centers of the robotic arm's rotational joints. Key point annotation data includes the normalized coordinates of the bounding box of the target area, the pixel-level segmentation mask of the entire robotic arm area, the normalized coordinates of the key points, and their visibility status. Through a virtual camera imaging mechanism, the precise pixel coordinates of each key point of the robotic arm in the image can be directly obtained. This virtual data acquisition method does not rely on the actual kinematic model parameters of the robotic arm, nor does it require cumbersome hand-eye calibration operations, solving the problems of complex modeling and difficult calibration in traditional key point detection methods.
[0036] Then, a keypoint detection neural network based on heatmap method is constructed, using the keypoint annotation data corresponding to the virtual robotic arm image as input and training with the supervision signal. For each keypoint in the virtual robotic arm image, the probability response value of any pixel in the virtual robotic arm image belonging to the keypoint is calculated. The probability response value is defined as the Euclidean distance calculated based on the coordinates of the pixel in the virtual robotic arm image and the keypoint annotation data corresponding to the keypoint, thereby generating the predicted heatmap corresponding to the keypoint. The generation method of the predicted heatmap can be formally represented as: ,in, Indicates the first A key point is in the image pixel coordinates. The probability response value at that point, Indicates the first The coordinates of each key point in the key point annotation data and the image pixel coordinates The Euclidean distance between them This represents the standard deviation of the diffusion degree of the response region in the heatmap. In this way, the network uses the heatmap as the learning target, transforming the keypoint detection problem into a pixel-level probabilistic map regression problem. During training, the keypoint detection neural network learns the mapping relationship from the input image to the multi-channel heatmap output, with each channel corresponding to the probability distribution map of a keypoint. This mapping relationship can be expressed as: ,in, Indicates the first Predictive heatmaps for key points The parameter is Key point detection neural network, The input image represents the keypoint detection neural network. The goal of the keypoint detection neural network is to minimize the difference between the predicted heatmap and the actual heatmap, and its loss function can be defined as: ,in, The Frobenius norm is used to measure the pixel-level error between the predicted and actual heatmaps. Training is complete when the loss function converges, resulting in the keypoint detection neural network model. The parameters of the keypoint detection neural network are... The heatmap is obtained by iteratively optimizing the pixel mean square error loss function between the predicted and actual heatmaps using backpropagation and the AdamW optimizer. The heatmap output by the keypoint detection neural network exhibits a peak response at the actual location of the keypoint, and the response value decays in a Gaussian distribution as the pixel moves away from the keypoint location.
[0037] See the instruction manual appendix Figure 5 By employing a transfer learning strategy, the keypoint detection neural network model is directly applied to real robotic arm images obtained from segmenting real images. Through forward inference calculations, the output is generated based on a Gaussian kernel function. Each keypoint has a heatmap, meaning each keypoint corresponds to a probability response map. By performing peak detection on each heatmap—that is, finding the pixel location corresponding to the maximum probability response in the heatmap—the precise pixel coordinates of each keypoint in the real image can be obtained. The precise pixel coordinates of keypoints in the real image are calculated according to the following formula: ,in, Indicates the first The precise pixel coordinates of each key point in the real image. Indicates the first A key point is the image pixel coordinates in predicting the heatmap. The probability response value at the location. The key point detection method described above, which is trained on virtual data and deployed to real-world scenarios through transfer learning, significantly reduces the dependence of model deployment on real labeled data while ensuring detection accuracy, achieving efficient and low-cost key point identification in target regions.
[0038] In one specific embodiment, the keypoint detection neural network is based on the CS-CoordUNet neural network architecture. The processing performed by the keypoint detection neural network is to map the input image into a keypoint heatmap with multiple channels, where each channel corresponds to the probability response distribution of a keypoint.
[0039] In one possible embodiment, the predicted values output by the keypoint detection neural network model are denormalized, and the predicted values are compared with the actual values obtained from the model's test set. A schematic diagram illustrating the comparison between the actual and predicted values is provided below. Figure 6 As shown in Table 1, the average pixel error between predicted and actual values, the percentage of correct keypoints within 5 pixels, and the percentage of correct keypoints within 6 pixels are as follows. The lowest average pixel error between predicted and actual values is 2.84187, the highest percentage of correct keypoints within 5 pixels is 94.59060%, and the highest percentage of correct keypoints within 6 pixels is 96.18437%, indicating that the model prediction is accurate and the implementation effect is ideal.
[0040] Table 1. Comparison of Model Prediction Performance Evaluation Indicators S104: Acquire real images from two different perspectives and apply a key point detection neural network model to obtain the first and second heat maps corresponding to different perspectives. For each key point of the robotic arm, the spatial coordinates of the key point are solved by triangulation based on the coordinates of the key point in the first and second heat maps. A world coordinate system is established, and the spatial coordinates of the key points are converted into three-dimensional coordinates in the world coordinate system. The joint trajectory of the robotic arm is tracked based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system obtained in continuous time sequence.
[0041] In one possible embodiment, a first camera and a second camera are used to acquire real images from different perspectives, treating the images acquired by the first and second cameras as images acquired by two cameras in a binocular vision system. The real image acquired by the first camera is segmented to obtain a real robotic arm image, and a keypoint detection neural network model is applied to generate a first heatmap. Similarly, the real image acquired by the second camera is segmented to obtain a real robotic arm image, and a keypoint detection neural network model is applied to generate a second heatmap. The spatial coordinates of key points are then calculated based on the first and second heatmaps using the principle of triangulation.
[0042] Specifically, the intrinsic parameters of the first camera are determined through camera calibration. The internal parameters of the second camera are Define the first camera as the left camera and the measurement coordinate system as { L The transformation matrix of the second camera relative to the first camera is: Therefore, the key points in space can be determined according to the following formula. The coordinates satisfy: , ,in, Indicates the first The coordinates of the key points in the first heatmap Indicates the first The coordinates of the key points in the second heatmap and Represents the distance coefficient. Indicates the first The space of the key points. Solving the above two equations simultaneously, the solution for the first key point is obtained using triangulation. The spatial three-dimensional coordinates of each key point. Specifically, after eliminating the depth coefficient from the two equations, a system of linear equations is constructed. ,in, , , These represent the first, second, and third row vectors of the first camera intrinsic parameter matrix, respectively. , , Let represent the first, second, and third row vectors of the second camera intrinsic parameter matrix, respectively. By solving the above system of equations, the first... Three-dimensional spatial coordinates of key points .
[0043] After obtaining the spatial three-dimensional coordinates of all nodes of the robotic arm, a unified reference coordinate system needs to be established to facilitate the description of the robotic arm's motion state and trajectory tracking. The base joint of the robotic arm is selected as the origin of the world coordinate system, denoted as [reference coordinate system name missing]. And establish a world coordinate system, based on the transformation relationship Convert spatial coordinates to three-dimensional coordinates in the world coordinate system. Indicates the first The three-dimensional coordinates of the key points in the world coordinate system Indicates the first The spatial coordinates of the key points. Among them, the base joint of the robotic arm refers to the first joint of the robotic arm, that is, the joint where the movable part of the robotic arm connects to the base.
[0044] By transforming coordinates, the relative positions of each key point of the robotic arm in the world coordinate system can be accurately determined. Based on the three-dimensional coordinate sequence of each key point in the world coordinate system obtained in continuous time, the displacement trajectory, velocity change, and motion posture of each joint can be further calculated, thereby realizing complete tracking of the robotic arm joint trajectory.
[0045] In one possible embodiment, the first camera and the second camera may not be on the same straight line, but it is better if they are on the same straight line. A preferred arrangement of the first camera (left camera) and the second camera (right camera) is shown in the appendix to the specification. Figure 7 As shown, the horizontal alignment of the first and second cameras (parallel optical axes and horizontal baselines) can greatly simplify the stereo matching algorithm and improve speed and accuracy.
[0046] This invention provides a robotic arm joint trajectory tracking method based on virtual-real fusion. It segments a virtual robotic arm image from a simulated image and fuses it with a scene image to form a fused image highly similar to the real environment. Transfer learning is then performed, and the fused image is input into a neural network for training. The trained model then segments real robotic arm images, eliminating the need for on-site image acquisition and annotation, thus improving efficiency and generalization ability. Addressing the shortcomings of existing robotic arm pose estimation methods—requiring complex kinematic models, complex hand-eye calibration, precise labeled data, and poor generalization—this invention uses a virtual camera to obtain the pixel coordinates of the robotic arm joint centers in the image, generating keypoint annotations. Transfer learning is then used to detect keypoints on the segmented real robotic arm using the trained model. Finally, triangulation is combined to obtain the joint spatial position, achieving joint trajectory tracking.
[0047] The virtual-real fusion-based robotic arm joint trajectory tracking method demonstrates significant generalization ability, exhibiting more comprehensive performance compared to the limitations of traditional posture methods. It possesses significant technical advantages in eliminating reliance on precise kinematic models, avoiding tedious hand-eye calibration, and improving efficiency and generalization capabilities. This has important theoretical and engineering significance for enhancing core robot performance and expanding application scenarios. It not only helps promote the high-performance operation of robots in complex human-robot collaborative scenarios but also provides key theoretical support and feasible practical paradigms for the development of collaborative robot technology for future intelligent manufacturing.
[0048] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0049] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0050] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.
[0051] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.
Claims
1. A method for tracking the joint trajectory of a robotic arm based on virtual-real fusion, characterized in that, include: The simulation environment model is used to simulate the operation and movement of the robotic arm and to collect simulation images, as well as scene images of the robotic arm operation scenario and real images of the robotic arm operating in the actual scenario. The simulated image is segmented to obtain a virtual robotic arm image. The virtual robotic arm image is fused with the scene image to generate a virtual-real fusion image. The neural network is iteratively trained based on the virtual-real fusion image to obtain a robotic arm target segmentation model. The robotic arm target segmentation model is applied to segment the real image to obtain a real robotic arm image. Based on the simulated image, key point annotation data corresponding to the joint center of the robotic arm is generated. A key point detection neural network based on the heat map method is constructed and trained based on the virtual robotic arm image and key point annotation data to obtain a key point detection neural network model. The key point detection neural network model is used to process the real robotic arm image to obtain the heat map corresponding to each key point. Peak detection is performed on each heat map to obtain the pixel coordinates of each key point in the real image. Two real images from different perspectives are acquired and processed using the key point detection neural network model to obtain a first heatmap and a second heatmap corresponding to different perspectives. For each key point of the robotic arm, the spatial coordinates of the key point are solved by triangulation based on the coordinates of the key point in the first and second heatmaps. A world coordinate system is established, and the spatial coordinates of the key points are converted into three-dimensional coordinates in the world coordinate system. The joint trajectory of the robotic arm is tracked based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system obtained in continuous time sequence.
2. The method according to claim 1, characterized in that, The process of fusing the virtual robotic arm image with the scene image to generate a virtual-real fusion image includes: The virtual robotic arm image and the scene image are subjected to pixel-by-pixel conditional judgment to retain the robotic arm pixels in the simulation image, and the remaining pixels in the simulation image are replaced with random background. Simultaneously, the original robotic arm label information in the simulation image is copied to maintain the position information.
3. The method according to claim 1, characterized in that, Image segmentation of the simulated image to obtain a virtual robotic arm image includes: A first neural network is constructed. Partial images are segmented from the collected simulation images and used as training samples to train the first neural network to obtain a virtual image segmentation model. The virtual image segmentation model is then applied to segment the simulation images to obtain virtual robotic arm images.
4. The method according to claim 3, characterized in that, Iterative training of the neural network based on the virtual-real fusion image to obtain a target segmentation model for the robotic arm includes: Construct a second neural network with the same architecture as the first neural network, and transfer the convolutional layer weights in the virtual image segmentation model to the corresponding layer of the second neural network for parameter initialization; The virtual-real fusion image is used as an input sample to perform image segmentation iterative training on the second neural network after parameter initialization, until the loss output of the iterative training does not decrease for multiple consecutive rounds, thus obtaining the target segmentation model of the robotic arm. Each round of iterative training includes: sampling a batch of virtual-real fusion images, performing image segmentation processing to obtain the prediction mask and loss value, updating the network parameters through backpropagation, and outputting the updated network parameters and the current batch loss after the current round of iterative training.
5. The method according to claim 1, characterized in that, Based on the simulated image, key point annotation data corresponding to the center of the robotic arm joint is generated, including: The center of the rotary joint of the robotic arm is taken as the key point of the robotic arm; The key point annotation data includes the normalized coordinates of the bounding box of the target area of the robotic arm, the pixel-level segmentation mask of the overall area of the robotic arm, the normalized coordinates of the key points, and their visibility status.
6. The method according to claim 1, characterized in that, A keypoint detection neural network based on heatmap method is constructed and trained using virtual robotic arm images and keypoint annotation data as supervision signals to obtain the keypoint detection neural network model, including: A key point detection neural network based on heatmap method is constructed, with the key point annotation data corresponding to the virtual robotic arm image as the input and the supervision signal. A keypoint detection neural network based on heatmap method calculates the probability response value of any pixel in the virtual robotic arm image belonging to a keypoint for each keypoint in the virtual robotic arm image, thereby generating a predicted heatmap corresponding to the keypoint and measuring the difference between the predicted heatmap and the real heatmap of the keypoint; wherein, the probability response value is the Euclidean distance calculated based on the coordinates of the pixel in the virtual robotic arm image and the keypoint annotation data corresponding to the keypoint. Define the loss function as ,in, Indicates the total number of key points. Indicates the first Predictive heatmaps for key points Indicates the first Real heat maps of key points The Frobenius norm is represented by the keypoint detection neural network model, which is obtained when the loss function converges.
7. The method according to claim 1, characterized in that, For each key point of the robotic arm, the spatial coordinates of the key point are solved using triangulation based on the coordinates of the key point in the first and second heatmaps, including: The camera that captures the real image corresponding to the first heatmap is defined as the first camera, and the camera that captures the real image corresponding to the second heatmap is defined as the second camera. The intrinsic parameters of the first camera are determined through camera calibration. The internal parameters of the second camera are The transformation matrix of the second camera relative to the first camera is: ; No. The coordinates of the key points in the first heat map are as follows: The coordinates in the second heat map are , , The relationship between the spatial coordinates of the key points and the key points satisfies the following formula: , ,in, and Represents the distance coefficient. Indicates the first Spatial coordinates of key points; Construct a system of linear equations by combining the two equations. ,in, , , These represent the first, second, and third row vectors of the first camera intrinsic parameter matrix, respectively. , , Let represent the first, second, and third row vectors of the second camera intrinsic parameter matrix, respectively. Solve the system of linear equations to obtain the spatial coordinates of the key points.
8. The method according to claim 1, characterized in that, Establish a world coordinate system and convert the spatial coordinates of the key points obtained from the solution into three-dimensional coordinates in the world coordinate system, including: The base joint of the robotic arm is selected as the origin of the world coordinate system. And establish a world coordinate system; According to the transformation relationship Convert spatial coordinates to three-dimensional coordinates in the world coordinate system. Indicates the first The three-dimensional coordinates of the key points in the world coordinate system Indicates the first Spatial coordinates of key points.
9. The method according to claim 1, characterized in that, The joint trajectory of the robotic arm is tracked based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system obtained under continuous time sequence, including: Based on the three-dimensional coordinate sequence of each key point of the robotic arm in the world coordinate system, the displacement trajectory, velocity change and motion posture of each key point are calculated.