Workpiece pose control method and system based on digital twinning and visual servoing

Abstract

This invention relates to the field of industrial automation and robot control technology, proposing a workpiece pose control method and system based on digital twins and visual servoing. By defining 3D geometric key points of the workpiece, a training dataset with 6D pose ground truth and 2D pixel annotations is generated using randomized sampling rendering, enabling virtual-to-real transfer training of the perception model. After acquiring real-time images of the workpiece, a lightweight detection network is used to extract the pixel coordinates of key points. The rotation matrix and translation vector are calculated using projection geometry and nonlinear optimization algorithms, and real-time absolute depth information is extracted. The rotation matrix guides the robotic arm to adjust its posture, and the real-time depth is substituted into the image Jacobian matrix to construct a composite matrix. Based on the feature error vector, an adaptive visual servo control law is used to calculate the robotic arm speed command and perform dynamic correction until the grasping threshold is met. This invention effectively improves the fault tolerance and real-time performance of workpiece grasping in industrial settings.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and robot control technology, and in particular to a workpiece pose control method and system based on digital twin and visual servoing. Background Technology

[0002] In industrial automated production lines, traditional vision positioning systems primarily rely on two-dimensional planar coordinates to guide robotic arms in performing grasping or assembly operations. This makes it difficult to acquire workpiece posture information in three-dimensional space, leading to grasping failures or even equipment collisions when the workpiece tilts, flips, or undergoes other spatial pose changes. Furthermore, deep learning-based visual perception models face severe data bottlenecks in the initial deployment phase. Existing technologies typically require stopping the production line to collect a large number of physical images and relying on manual pixel-level annotation, resulting in lengthy and costly data preparation cycles. Directly debugging control logic in a real environment also carries a high risk of equipment damage. In addition, existing control architectures often employ open-loop execution modes or position-based visual servo strategies. If there are even slight calibration deviations between the camera and the robotic arm, or if the workpiece undergoes minor displacement during the robotic arm's movement, the system lacks the ability to dynamically correct the trajectory during the approximation process, resulting in limited final positioning accuracy and a low overall fault tolerance. Summary of the Invention

[0003] To address the aforementioned shortcomings, the present invention aims to propose a workpiece pose control method and system based on digital twin and visual servoing. By integrating digital twin simulation and visual servo closed-loop control technologies, it constructs a data generation and six-degree-of-freedom pose calculation architecture for virtual-real migration, enabling precise perception of workpiece spatial pose and dynamic correction of robotic arm trajectory. This solves the technical problems of traditional two-dimensional vision lacking spatial pose information, difficulties in cold-starting deep learning models for physical production lines, and poor anti-interference capability of open-loop control.

[0004] To achieve this objective, the present invention adopts the following technical solution: A workpiece pose control method based on digital twin and visual servoing includes: In a digital twin simulation environment, several three-dimensional geometric key points are defined based on the geometric features of the workpiece to be processed; A training dataset with 6D pose ground truth and 2D pixel coordinate annotations is automatically generated by randomized sampling and rendering, and the key point detection model is trained using the training dataset. Acquire a real-time image of the workpiece to be processed, and use the trained key point detection model to extract the two-dimensional pixel coordinates of each key point in the real-time image; Combining the three-dimensional geometric key points and camera intrinsic parameters, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, and the real-time absolute depth information of the workpiece is extracted from the translation vector. The robotic arm end effector is guided to rotate to a preset posture according to the rotation matrix, and the real-time absolute depth information is substituted into the image Jacobian matrix to construct the synthetic image Jacobian matrix. Based on the error vector between the current feature point and the desired feature point in the real-time image, the speed command of the robotic arm is calculated using the visual servo control law, and the robotic arm is driven to perform dynamic correction until the error meets the grasping threshold.

[0005] Preferably, the training dataset automatically generated by randomized sampling and rendering includes 6D pose ground truth and 2D pixel coordinate annotations, comprising: The virtual camera is controlled by parametric scripts to perform randomized sampling around the virtual workpiece, and the lighting intensity, shooting distance, and camera shooting angle of the virtual scene are randomly changed. Extract the absolute 6D pose truth value of the virtual workpiece in the world coordinate system, and project the three-dimensional geometric key points onto the two-dimensional image plane using the intrinsic parameter matrix of the virtual camera. Render and export the RGB image, and simultaneously generate a label file including the class confidence score and the two-dimensional pixel coordinates to obtain the training dataset.

[0006] Preferably, extracting the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model includes: The backbone network of the key point detection model is replaced with GhostModule. Core features are generated by convolutional operations, and similar features are generated by linear operations. During training, channel pruning is used to remove redundant features, and the trained key point detection model is quantized using INT8. The acquired real-time image is input into the key point detection model, multi-scale features are extracted using the backbone network, and the category confidence of the workpiece and the first two-dimensional pixel coordinates of each key point are output by the prediction head. A Kalman filter is connected in series at the back end of the key point detection model. A state vector is constructed using the two-dimensional pixel coordinates and their rate of change. The first two-dimensional pixel coordinates are then optimally estimated to obtain smooth second two-dimensional pixel coordinates.

[0007] Preferably, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, including: Establish a perspective projection model of a pinhole camera, satisfying the following relation: ; in, and These represent the x-coordinate and y-coordinate of the pixel at the observation point in the real-time image, respectively. Indicates the scale factor. This represents the camera intrinsic parameter matrix. Let represent the rotation matrix to be solved. Denotes the translation vector to be solved. , , These represent the three-dimensional spatial coordinates of the workpiece in the local coordinate system; Construct the reprojection error objective function The following relation is satisfied: ; in, The key point detection model detects the first... Two-dimensional pixel coordinate observations of key points This represents the total number of key points. This indicates the perspective projection parameters of the pinhole camera. Indicates the preset first Three-dimensional geometric key points; The reprojection error objective function is solved iteratively using a nonlinear optimization algorithm. In each iteration, the pose update increment is calculated using the normal equation. The normal equation satisfies the following relation: ; in, This indicates that the reprojection error is related to the pose parameters. Jacobian matrix, This represents the damping factor used to adjust the convergence rate. Represents the identity matrix. This represents the pose update increment. The reprojection error parameter represents the current iteration round; When the average reprojection error, pose update increment, or number of iterations meets the preset joint termination condition, the optimal rotation matrix and translation vector are output, and the components are extracted from the translation vector as the real-time absolute depth information of the workpiece.

[0008] Preferably, the calculation of the robot arm's speed command using a visual servo control law includes: Substitute the real-time absolute depth information into the image Jacobian matrix of a single feature point. In the middle, the following relation is satisfied: ; in, This indicates the real-time absolute depth information. and These represent the x and y coordinates of the feature points in the normalized image coordinate system, respectively. The Jacobian matrices of several key points are stacked vertically to construct the composite image Jacobian matrix. And its pseudo-inverse matrix is ​​solved by singular value decomposition. The following relation is satisfied: ; in, The pseudo-inverse matrix of the Jacobian matrix of the synthesized image is represented by the following matrix: This represents the composite image Jacobian matrix, which is formed by stacking the Jacobian matrices of the images at each keypoint. The transpose of the Jacobian matrix of the synthesized image is represented by . Based on the error vector between the current feature point and the desired feature point in the real-time image, and the adaptive proportional control gain, the speed command is calculated. The following relation is satisfied: ; in, Indicates the speed command of the robotic arm. This represents the adaptive proportional control gain based on the error norm. This represents the image feature error vector obtained by subtracting the current feature point position from the desired feature point position; The adaptive proportional control gain Satisfying the relation: ; in, and These represent the preset maximum and minimum gain coefficients, respectively. This represents the preset convergence rate parameter. The norm of the image feature error vector is represented.

[0009] Preferably, it also includes adaptive feedback and active learning mechanisms: Real-time monitoring of the confidence score output by the keypoint detection model, construction of a spatial joint confidence evaluation model, and calculation of the obtained spatial joint confidence. Satisfying the relation: ; in, This indicates the confidence level for assessing the global existence of the workpiece. Indicates the first Local pose confidence at key points Indicates the total number of key points. , , These represent the preset weight coefficients corresponding to global existence, overall pose quality, and local defect penalty, respectively, and satisfy the normalization condition. The spatial joint confidence score is time-series filtered using a moving exponential weighted moving average algorithm to obtain the final smoothed confidence score. The following relation is satisfied: ; in, express The final smooth confidence level at any given moment. This represents the final smooth confidence level at the previous time step. This represents the preset time smoothing factor; When the final smooth confidence level is continuously lower than the safety threshold, the visual blind zone of the long-tail edge condition is determined to be triggered, the observation update of the key point detection model is suspended, and the Kalman filter outputs the predicted feature point coordinates based on the kinematic prior model to maintain visual servo closed-loop control. Real-time images located in the visual blind spot are automatically captured as difficult examples and mixed into the training dataset. The key point detection model is then fine-tuned by freezing the backbone network and incrementally training with a reduced learning rate.

[0010] A workpiece pose control system based on digital twin and vision servoing includes: The space definition module is used to define several three-dimensional geometric key points based on the geometric features of the workpiece to be processed in a digital twin simulation environment. The data evolution module is used to automatically render and generate a training dataset with 6D pose ground truth and two-dimensional pixel coordinate annotations through randomized sampling, and to train the key point detection model using the training dataset. The perception extraction module is used to acquire real-time images of the workpiece to be processed and to extract the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model. The pose calculation module is used to combine the three-dimensional geometric key points and camera intrinsic parameters, use projection relationships and nonlinear optimization algorithms to calculate the rotation matrix and translation vector of the workpiece relative to the camera, and extract the real-time absolute depth information of the workpiece from the translation vector. The mapping construction module is used to guide the end effector of the robotic arm to rotate to a preset posture according to the rotation matrix, and to substitute the real-time absolute depth information into the image Jacobian matrix to construct the synthetic image Jacobian matrix. The closed-loop control module is used to calculate the speed command of the robotic arm based on the error vector between the current feature point and the desired feature point in the real-time image, and drive the robotic arm to perform dynamic correction until the error meets the grasping threshold.

[0011] One of the above technical solutions has the following advantages or beneficial effects: This invention defines the three-dimensional geometric key points of a workpiece in a digital twin simulation environment to establish a spatial structural benchmark. Then, it utilizes randomized sampling to automatically render and generate a synthetic training dataset with six-DOF pose ground truth and two-dimensional pixel coordinate annotations. This allows the key point detection model to complete training and cold start without relying on real physical data acquisition. In the real-time control phase, the trained model extracts the two-dimensional pixel coordinates of key points in the workpiece image. Combined with preset three-dimensional geometric key points and camera intrinsic parameters, it uses projection relationships and nonlinear optimization algorithms to inversely calculate the rotation matrix and translation vector of the workpiece relative to the camera. This elevates the two-dimensional image information to three-dimensional spatial pose perception. Simultaneously, it extracts real-time absolute depth information from the translation vector to overcome the inherent limitations of monocular vision in depth perception. There are bottlenecks. Based on this, a rotation matrix is ​​used to guide the end effector of the robotic arm to rotate to a preset position that matches the workpiece posture. Real-time depth information is then substituted into the image Jacobian matrix to construct a synthetic image Jacobian matrix, establishing a precise mapping relationship between the image feature space and the robotic arm motion space. Finally, based on the error vector between the current feature point and the desired feature point in the real-time image, the speed command of the robotic arm is calculated through a visual servo control law and driven to perform dynamic correction until the error meets the grasping threshold. This transforms the traditional open-loop control into a closed-loop control based on image feature error, enabling the system to perceive and correct trajectory deviations in real time as the robotic arm approaches the workpiece. This significantly reduces the dependence on absolute calibration accuracy and improves the robustness and motion accuracy during the grasping process of complex posture workpieces. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0013] Figure 1 This is a flowchart of a workpiece pose control method based on digital twin and visual servoing provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the workpiece pose control system based on digital twin and visual servoing provided in an embodiment of the present invention; Figure 3 This is a hardware deployment and software node flow diagram of the workpiece pose control method based on digital twin and visual servo provided in the embodiments of the present invention; Figure 4 This is a logic diagram of the virtual-to-real migration data generation and long-tail working condition active learning mechanism provided in the embodiments of the present invention; Figure 5This is a flowchart of the visual servo dynamic correction execution process based on the image Jacobian matrix provided in an embodiment of the present invention. Detailed Implementation

[0014] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0015] In this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0016] Workpiece pose control methods based on digital twins and vision servoing, such as Figure 1 As shown in a preferred embodiment of the present invention, the workpiece pose control method based on digital twin and visual servoing described in the present invention can be applied to closed-loop control consisting of a perception layer, a calculation layer, and an execution layer, such as... Figure 3As shown, the perception layer consists of an industrial camera, photoelectric sensors or limit switches, an industrial PLC, and an edge computing industrial control computer. The industrial camera continuously acquires real-time RGB video streams of the workpiece to be processed. The photoelectric sensors or limit switches monitor whether the workpiece has reached the designated station. After confirming that the workpiece is in place, the industrial PLC sends a hardware trigger command to the computing layer via TCP / IP protocol. Simultaneously, the edge computing industrial control computer in the perception layer performs front-end processing on the acquired video stream. The computing layer is deployed in an edge computing industrial control computer based on a ROS environment and includes an image preprocessing module, a YOLOv8-Pose keypoint detection node, a PnP6D pose calculation node, an IBVS visual servo control node, and a PnP visual servo control node. After receiving the trigger command and image data from the perception layer, this layer preprocesses the image using the image preprocessing module, and then... The YOLOv8-Pose keypoint detection node extracts the 2D pixel coordinates of key points on the workpiece. Then, the PnP6D pose calculation node combines the 3D geometric key points and camera intrinsic parameters to calculate the rotation matrix and translation vector. Subsequently, the IBVS and PnP visual servo control nodes calculate the visual servo control law based on the error between the current feature point and the desired feature point, generating a six-degree-of-freedom speed command. The execution layer consists of a robotic arm controller and a six-degree-of-freedom robotic arm. The robotic arm controller receives the Twist speed command issued by the computing layer at a frequency of 30Hz through the ROS topic mechanism, driving the joint motors of the six-degree-of-freedom robotic arm to perform dynamic correction. At the same time, the physical movement of the robotic arm end changes the camera perspective, and this perspective change is fed back to the industrial camera in the perception layer, forming a closed-loop control loop from image acquisition and pose calculation to robotic arm execution.

[0017] The workpiece pose control method includes the following steps: S1: In the digital twin simulation environment, several three-dimensional geometric key points are defined based on the geometric features of the workpiece to be processed; It should be noted that a digital twin simulation environment refers to a virtual environment in which the mapping and simulation of a physical production line are completed in a virtual space. This virtual environment is used for pre-simulating and generating data with absolute truth values. In step S1, this environment provides a virtual mapping space consistent with the geometry of the real production line, allowing subsequent data generation to be independent of the real physical hardware. The workpiece to be processed refers to the physical object on the production line that needs to be grasped or assembled by the robotic arm. Its geometric shape directly determines the selection strategy for key points. Three-dimensional geometric key points refer to semantically significant three-dimensional spatial coordinate points defined in the workpiece's local coordinate system. These points follow non-collinear and non-coplanar constraints and are used to establish the correspondence between the two-dimensional image and the three-dimensional model in subsequent pose calculations. For example, for an aluminum alloy gearbox housing, the centers of its four mounting bolt holes can be selected as three-dimensional geometric key points. Geometric features refer to the shape attributes of the workpiece surface that can be distinguished and identified, such as corner points, the centers of circular holes, and edge intersections. These features have high visual saliency in the image, facilitating stable detection by deep learning models.

[0018] Understandably, by pre-defining 3D geometric key points in a digital twin simulation environment to establish a spatial structural benchmark for the workpiece, a 3D spatial reference can be provided for key point detection in subsequent 2D images. Utilizing the known precise geometric information in the virtual environment, the spatial morphology of the workpiece is abstracted into a set of discrete and semantically clear feature points. When the visual algorithm identifies the 2D projection positions of these points in the real image, the spatial pose of the workpiece can be inversely deduced through the correspondence between 3D and 2D points, avoiding the instability in pose calculation caused by the lack of prior 3D structural information in traditional methods.

[0019] S2: Automatically generate a training dataset with 6D pose ground truth and two-dimensional pixel coordinate annotations through randomized sampling, and use the training dataset to train the key point detection model. It should be noted that randomized sampling refers to a data acquisition strategy in a virtual environment where a virtual camera is controlled by parametric scripts to perform multi-degree-of-freedom random transformations around a virtual workpiece. This strategy simulates the variable imaging conditions in a real production line by randomly changing the light intensity, shooting distance, and camera angle, thereby enhancing the model's generalization ability. The 6D pose ground truth refers to the six degrees of freedom parameters describing the absolute position and spatial orientation of the virtual workpiece in three-dimensional space, including the three-dimensional coordinates X, Y, and Z and three rotation angles Roll, Pitch, and Yaw. This ground truth is directly output by the digital twin environment and possesses absolute accuracy unattainable in the physical world. Two-dimensional pixel coordinate annotation refers to the pixel position marking information corresponding to key points on the image plane. This annotation is automatically generated by projecting the three-dimensional geometric key points onto the two-dimensional image plane according to the virtual camera's intrinsic parameter matrix, without manual intervention. The training dataset refers to the collection of synthetic images and their corresponding annotations used for training the deep learning model; its data volume can be infinitely expanded through parametric scripts. The key point detection model refers to a neural network built on a deep learning architecture for extracting the pixel coordinates of target key points from images. In this embodiment, YOLOv8-Pose is used as the baseline architecture.

[0020] Understandably, by using randomized sampling to automatically generate precisely labeled synthetic training data in a virtual environment, and then using this dataset to train a keypoint detection model, the challenges of cold-starting deep learning models on physical production lines and the high cost of labeling real data can be addressed. The digital twin environment can precisely control the relative pose of the virtual camera and the workpiece, and automatically record the true 6D pose at each sampling moment. By projecting a pinhole camera model, 3D keypoints are mapped to 2D pixel coordinates, thereby generating batches of perfectly labeled training samples. This allows the model to be trained without real physical data. When the model is deployed to a real production line, it can directly and quickly locate keypoints on the workpiece in real-time images, reducing the preparation cycle of the visual perception model from weeks of downtime on the real production line to hours in the virtual environment. This also avoids the risk of equipment damage caused by debugging control logic in the real environment.

[0021] S3: Obtain a real-time image of the workpiece to be processed, and use the trained key point detection model to extract the two-dimensional pixel coordinates of each key point in the real-time image; It should be noted that real-time images refer to images of the actual physical environment captured by industrial cameras deployed on the production line after the workpiece has reached its position. These images reflect the workpiece's true visual state at the current moment and serve as the information source for subsequent pose calculations. Two-dimensional pixel coordinates refer to the pixel positions of keypoints on the image plane. Typically, the origin is the upper left corner of the image, with the horizontal axis being the u-axis and the vertical axis the v-axis. This coordinate system compresses three-dimensional spatial information into a two-dimensional planar representation, acting as a bridge between the real image and the three-dimensional geometric model. The keypoint detection model, trained and deployed to the computation layer in step S3, receives real-time images as input and outputs the pixel coordinate positions of each keypoint in the image through forward inference.

[0022] Understandably, real-time images of the workpiece to be processed are acquired using industrial cameras, and the two-dimensional pixel coordinates of each keypoint are extracted from these images using a trained keypoint detection model. This transforms real-time visual information from the physical production line into two-dimensional observation data that can be used for pose calculation. A deep learning model extracts multi-scale features from the image through multi-layer convolutional operations and regresses the pixel coordinates of the keypoints at the prediction head. After sufficient training with digital twin synthetic data, the model possesses the ability to recognize semantic feature points of the workpiece in realistic and complex backgrounds, thus providing a stable two-dimensional input for pose calculation and avoiding the interruption of pose calculation caused by the failure of traditional template matching methods under changes in lighting or background interference.

[0023] S4: Combining the three-dimensional geometric key points and camera intrinsic parameters, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationship and nonlinear optimization algorithm, and the real-time absolute depth information of the workpiece is extracted from the translation vector. It should be noted that camera intrinsic parameters refer to the internal parameter matrix describing the focal length and optical center position of the industrial camera. This typically includes focal lengths Fx and Fy and optical centers Cx and Cy. This matrix is ​​obtained beforehand during the camera calibration process and is used in step S4 to establish the projection relationship between 3D spatial points and 2D pixel coordinates. The projection relationship refers to the geometric transformation relationship of mapping 3D spatial points to the 2D image plane through the pinhole camera model. This relationship follows the principle of perspective projection, i.e., the 3D points are transformed by rotation and translation, and then projected onto the image plane through focal length scaling and optical center translation. The nonlinear optimization algorithm refers to a numerical calculation method that minimizes the objective function iteratively to approximate the optimal solution. In step S4, the EPnP algorithm is used to provide initial pose estimation, followed by iterative refinement using the Levenberg-Marquardt algorithm to minimize reprojection errors. The rotation matrix is ​​a 3x3 orthogonal matrix describing the spatial attitude of the workpiece coordinate system relative to the camera coordinate system. This matrix contains three rotational degrees of freedom, determining the workpiece's tilt, flip, and other attitude states. The translation vector is a three-dimensional vector describing the spatial position of the workpiece coordinate system origin relative to the camera coordinate system origin. This vector contains three translational degrees of freedom. Real-time absolute depth information refers to the real-time distance of the workpiece along the optical axis in the camera coordinate system, i.e., the Z component of the translation vector. This parameter is a key input for the subsequent construction of the image Jacobian matrix, used to overcome the inherent deficiency of monocular vision in lacking depth information.

[0024] It is understandable that by combining the preset three-dimensional geometric key points, camera intrinsic parameters, and the two-dimensional pixel coordinates obtained in step S3, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, and real-time absolute depth information is extracted from the translation vector, thereby realizing the upgrade of two-dimensional image observation to three-dimensional spatial pose perception. The pinhole camera model establishes the perspective projection relationship between 3D spatial points and 2D image points. When the correspondence between 3D model points and 2D image points is known, the pose calculation problem is transformed into finding the optimal rotation matrix and translation vector that minimizes the error between the 2D coordinates of the 3D point after projection according to the pose and the observed coordinates. The initial pose is quickly calculated using the EPnP algorithm with geometric constraints, and the reprojection error is iteratively optimized using the Levenberg-Marquardt algorithm to finally obtain accurate six-degree-of-freedom pose parameters. At the same time, the real-time absolute depth information extracted from the translation vector directly reflects the real-time distance between the workpiece and the camera. This depth information is dynamically updated with each frame of the video stream, providing continuous and accurate depth parameters for visual servo control. This overcomes the problem of increased system complexity and cost caused by the lack of real-time depth in traditional monocular image Jacobian matrices, which can only use fixed estimates or additional depth sensors.

[0025] S5: Guide the end effector of the robotic arm to rotate to a preset posture according to the rotation matrix, and substitute the real-time absolute depth information into the image Jacobian matrix to construct the synthetic image Jacobian matrix; It should be noted that the end effector of the robotic arm refers to the installation position of the terminal actuator when the industrial robotic arm performs gripping or assembly operations. In step S5, the posture of this position is adjusted by the rotation matrix to ensure that the relative posture between the end effector and the workpiece meets the gripping requirements. The preset posture refers to the alignment orientation of the end effector relative to the target posture of the workpiece. It usually requires the end effector to be parallel to the workpiece surface or at a specific angle. This posture is directly determined by the rotation matrix calculated in step S4, so that the robotic arm completes the posture alignment before approaching the workpiece. The image Jacobian matrix is ​​a two-dimensional multi-six-dimensional matrix that describes the mapping relationship between the moving speed of image feature points in a two-dimensional plane and the camera space velocity. This matrix establishes a differential mapping between the image space and the camera motion space, and its elements are determined by the normalized coordinates and depth information of the feature points. The synthetic image Jacobian matrix is ​​a high-dimensional matrix formed by vertically stacking the image Jacobian matrices corresponding to multiple key points. In this embodiment, four two-dimensional multi-six-dimensional matrices corresponding to four key points are stacked to form an eight-dimensional matrix. This overdetermined structure makes the system extremely robust to the detection noise of a single feature point.

[0026] Understandably, by guiding the robotic arm's end effector to rotate to a preset posture according to the rotation matrix, and substituting real-time absolute depth information into the image Jacobian matrix to construct a synthetic image Jacobian matrix, a precise mapping relationship from image feature errors to robotic arm speed commands is established. The rotation matrix completely describes the spatial posture of the workpiece relative to the camera. By extracting the posture information from this matrix and converting it into rotation commands for each joint of the robotic arm, the end effector can be pre-adjusted to an orientation matching the workpiece posture before approaching it, avoiding physical interference caused by posture mismatch during translational approach. At the same time, the depth parameter in the image Jacobian matrix is ​​dynamically substituted by the absolute depth calculated in real time in step S4, rather than using a fixed estimate, ensuring that the mapping relationship between the image feature point movement speed and the camera spatial speed remains accurate throughout the entire process of the robotic arm approaching from a far distance to a near distance. The synthetic matrix formed by vertically stacking the Jacobian matrices corresponding to multiple key points further enhances the stability of the system under feature point detection noise, thus providing a reliable mathematical basis for the visual servo control law.

[0027] S6: Based on the error vector between the current feature point and the desired feature point in the real-time image, calculate the speed command of the robotic arm using the visual servo control law and drive the robotic arm to perform dynamic correction until the error meets the grasping threshold.

[0028] It should be noted that the desired feature point refers to the pixel coordinate position of the workpiece key point on the image plane when the end effector of the robotic arm reaches the target pose. This position is usually preset in the system initialization stage according to the grasping process requirements. The error vector is the difference vector between the current feature point pixel coordinates and the desired feature point pixel coordinates. This vector reflects the degree of deviation between the current state and the target state of the robotic arm, and its direction indicates the direction of motion that the robotic arm needs to adjust. The visual servo control law is a mathematical control strategy that calculates the robotic arm speed command based on the image feature error. In step S6, a proportional control form based on a pseudo-inverse matrix is ​​adopted. By mapping the image spatial error to the camera spatial velocity and then converting it to the robotic arm joint velocity, error-driven closed-loop control is achieved. The speed command refers to the six-degree-of-freedom motion speed command sent by the computing layer to the robotic arm joint driver, including three linear velocity components and three angular velocity components. This command is issued in real time at a specific frequency through the ROS topic mechanism. Dynamic correction refers to the control behavior of the robotic arm continuously correcting its motion trajectory based on real-time image feedback during the approach to the workpiece. This behavior is different from the fixed trajectory execution mode of traditional open-loop control. The gripping threshold refers to the image feature error judgment standard for determining whether the end effector of the robotic arm is aligned with the workpiece and can trigger a physical gripping action. When the root mean square error is lower than this threshold, the system triggers a gripping signal.

[0029] Understandably, by using the error vector between the current feature point and the desired feature point in the real-time image, the speed command of the robotic arm is calculated using a visual servo control law, and the robotic arm is driven to perform dynamic correction until the error meets the grasping threshold. This transforms traditional open-loop control into closed-loop control based on image feature errors. The feature error on the image plane directly reflects the relative deviation between the robotic arm's end effector and the target pose. Through the pseudo-inverse operation of the image Jacobian matrix, this two-dimensional error is mapped to a six-dimensional camera space velocity, which is then converted into drive commands for each joint of the robotic arm. This allows the robotic arm to perceive and correct trajectory deviations in real time as it approaches the workpiece. The adaptive proportional control gain is dynamically adjusted according to the error norm, using high gain for rapid approach at long distances and low gain for smooth alignment at close distances, thus avoiding overshoot. When the error converges to below the grasping threshold, the system determines that alignment is complete and triggers physical execution. The above process relies entirely on the feature error on the image plane for dynamic fine-tuning, abandoning the direct dependence on absolute three-dimensional coordinates. This gives the system a natural ability to compensate for calibration errors between the camera and the robotic arm, as well as the small displacements of the workpiece during the robotic arm's movement.

[0030] Preferably, the training dataset automatically generated by randomized sampling and rendering includes 6D pose ground truth and 2D pixel coordinate annotations, comprising: The virtual camera is controlled by parametric scripts to perform randomized sampling around the virtual workpiece, and the lighting intensity, shooting distance, and camera shooting angle of the virtual scene are randomly changed. Extract the absolute 6D pose truth value of the virtual workpiece in the world coordinate system, and project the three-dimensional geometric key points onto the two-dimensional image plane using the intrinsic parameter matrix of the virtual camera. Render and export the RGB image, and simultaneously generate a label file including the class confidence score and the two-dimensional pixel coordinates to obtain the training dataset.

[0031] It should be noted that randomized sampling refers to a data acquisition strategy in a virtual environment where a virtual camera is controlled by parametric scripts to perform multi-degree-of-freedom random transformations around a virtual workpiece. This strategy simulates the varying imaging conditions in a real production line by randomly changing the light intensity, shooting distance, and camera angle, thereby enhancing the model's generalization ability. Parametric scripts are automated control programs written in code within a digital twin environment, used to precisely control the virtual camera's pose, lighting parameters, and rendering process, achieving full automation and repeatability of the data generation process. Light intensity refers to the radiant flux density of light sources in the virtual scene, measured in lux, directly affecting the contrast and shadow distribution of the rendered image. Randomly changing this parameter allows the training data to cover different lighting conditions in the production line. Shooting distance refers to the spatial distance from the optical center of the virtual camera to the geometric center of the virtual workpiece. Randomly changing this parameter allows the model to learn the workpiece's appearance features at different scales. Shooting angle refers to the optical axis direction and observation angle of the virtual camera relative to the virtual workpiece, described by pitch and yaw angles. Randomly changing this parameter allows the model to adapt to multi-angle observation requirements. The 6D pose ground truth refers to the six degrees of freedom parameters describing the absolute position and spatial orientation of the virtual workpiece in three-dimensional space, including the three-dimensional coordinates X, Y, and Z and three rotation angles Roll, Pitch, and Yaw. This ground truth is directly output by the digital twin environment and has absolute accuracy that cannot be obtained in the physical world. The intrinsic parameter matrix refers to the internal parameter matrix describing the focal length and optical center position of the virtual camera, including the focal lengths Fx and Fy and the optical centers Cx and Cy, used to establish the projection mapping relationship between three-dimensional spatial points and two-dimensional pixel coordinates. Projection refers to the mathematical transformation process of mapping geometric points in three-dimensional space to a two-dimensional image plane according to the perspective model of a pinhole camera, achieving dimensionality reduction from three-dimensional to two-dimensional through matrix multiplication. RGB image refers to a color digital image composed of red, green, and blue color channels, used as the input data format for deep learning models. Class confidence refers to the degree of certainty of the deep learning model that the target workpiece exists in the image, expressed as a probability value, used to filter valid detection results. The annotation file is a text file that stores the pixel coordinates and category information of key points in an image in a structured format. It is generated synchronously with the RGB image and constitutes the label part of the training data pairs.

[0032] Understandably, by controlling a virtual camera to perform randomized sampling around a virtual workpiece using parametric scripts, and randomly varying the light intensity, shooting distance, and shooting angle, the absolute 6D pose ground truth of the virtual workpiece in the world coordinate system is extracted. The 3D geometric keypoints are projected onto a 2D image plane using the virtual camera's intrinsic parameter matrix, rendering and exporting RGB images. Simultaneously, annotation files containing class confidence scores and 2D pixel coordinates are generated, enabling the automatic generation of a synthetic training dataset with precise annotations. The digital twin environment can precisely control the relative pose of the virtual camera and the workpiece, and automatically record the 6D pose ground truth at each sampling moment. By mapping 3D keypoints to 2D pixel coordinates through a pinhole camera projection model, perfectly annotated training samples are generated in batches. This allows the keypoint detection model to be trained without real physical data. When the model is deployed to a real production line, it can directly and quickly locate keypoints in real-time images, significantly shortening the preparation cycle of the visual perception model from the lengthy state of relying on real production line downtime to a highly efficient state in the virtual environment. Simultaneously, it avoids the risk of equipment damage caused by debugging control logic in the real environment.

[0033] like Figure 4 As shown, the complete closed-loop architecture of virtual-real migration data generation and active learning in physical space is demonstrated. The virtual space data generation process on the left takes the physical production line CAD model as input, imports it into the digital twin simulation environment, and then performs domain randomization processing to randomly transform lighting, materials, and viewpoints, rendering diverse virtual scenes. Simultaneously, the true 3D absolute pose of the virtual workpiece in the virtual coordinate system is extracted, and the 3D geometric key points are mapped to 2D pixel coordinates through pinhole camera projection model, automatically generating a synthetic training dataset with perfect annotations for training YOLOv8-Po. The SE keypoint detection model, in the real-time operation and active learning process in the physical space on the right, the trained model is deployed to the industrial control computer at the edge of the physical production line to perform real-time inference. At the same time, the confidence of the output results is checked. When the confidence is greater than or equal to 0.6, the keypoint coordinates are output normally and the model enters the subsequent pose calculation stage. When the confidence is less than 0.6, the long-tail edge condition is triggered. The system automatically captures the abnormal frame image. After manual or large model assistance to review and supplement 2D annotations, the difficult example samples are sent back to supplement the synthetic training dataset. The robustness of the model is continuously evolved through incremental training.

[0034] Preferably, extracting the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model includes: The backbone network of the key point detection model is replaced with GhostModule. Core features are generated by convolutional operations, and similar features are generated by linear operations. During training, channel pruning is used to remove redundant features, and the trained key point detection model is quantized using INT8. The acquired real-time image is input into the key point detection model, multi-scale features are extracted using the backbone network, and the category confidence of the workpiece and the first two-dimensional pixel coordinates of each key point are output by the prediction head. A Kalman filter is connected in series at the back end of the key point detection model. A state vector is constructed using the two-dimensional pixel coordinates and their rate of change. The first two-dimensional pixel coordinates are then optimally estimated to obtain smooth second two-dimensional pixel coordinates.

[0035] It's important to note that GhostModule is a lightweight convolutional neural network module. It generates core feature maps using a small number of regular convolutions, then generates similar feature maps through inexpensive linear transformations. The two are then concatenated as the final output. This significantly reduces the number of parameters and computations while maintaining feature extraction capabilities, adapting to the computing power limitations of edge control machines. Core features refer to the basic semantic feature maps extracted from the input image by regular convolution operations. These maps contain low-level visual information such as workpiece edges and textures, serving as the source for subsequent similar feature generation. Similar features are derived feature maps obtained by linear transformations of core features. They are highly correlated with core features but contain complementary information, used to expand the richness of feature representation without adding a large number of convolutional operations. Channel pruning refers to removing redundant or low-contribution channels after model training based on the importance evaluation metrics of each feature channel. This compresses the model size and reduces inference computation while maintaining model accuracy. INT8 quantization is a post-processing technique that converts model weights and activation values ​​from 32-bit floating-point numbers to 8-bit integer representations. This reduces model storage footprint and memory bandwidth requirements during inference, enabling the model to run efficiently on resource-constrained edge devices. Multi-scale features refer to the set of feature maps with different spatial resolutions and receptive fields extracted by the backbone network at different levels of convolutional layers. Low-level features retain fine spatial details, while high-level features contain abstract semantic information. The fusion of the two improves the accuracy of keypoint detection. The prediction head refers to the functional module at the end of the neural network, responsible for converting the feature maps extracted by the backbone network into specific task outputs. In this embodiment, it outputs the workpiece category confidence score and the two-dimensional pixel coordinates of each keypoint. The first two-dimensional pixel coordinates refer to the original pixel coordinates output by the keypoint detection network through direct forward inference. They may be subject to jitter or deviation due to the influence of light flicker and mechanical vibration noise in industrial environments. The Kalman filter is an algorithm that uses the state equation of a linear system to optimally estimate the system state through system input and output observation data. In this embodiment, a state vector is constructed using two-dimensional coordinates and their rate of change to perform temporal smoothing on the original detection results. The state vector refers to a set of variables describing the current state of the system. In this embodiment, it includes the two-dimensional pixel coordinates of the keypoints. and its first rate of change The Kalman filter integrates historical observations with current measurements through a temporal update mechanism. The rate of change refers to the time derivative of pixel coordinates between adjacent frames, reflecting the motion velocity of feature points on the image plane and serving as a crucial basis for Kalman filter state prediction. Optimal estimation processing involves the Kalman filter calculating the posterior state estimate through a two-step prediction-update process, combining the system's kinematic model with the statistical characteristics of observation noise, and outputting statistically optimal smoothed coordinates. The second two-dimensional pixel coordinates refer to the final pixel coordinates after Kalman filter smoothing, avoiding interference from industrial noise and possessing temporal consistency and stability, directly inputting them into subsequent pose calculation steps.

[0036] Understandably, by replacing the backbone network of the keypoint detection model with a GhostModule, core features are generated using convolutional operations, followed by similar features generated through linear operations. During training, channel pruning is employed to remove redundant features, and the trained model undergoes INT8 quantization. Real-time images are input into this lightweight model to extract multi-scale features and output the first two-dimensional pixel coordinates. A Kalman filter is then connected in series at the model's backend to construct a state vector using the two-dimensional pixel coordinates and their rate of change for optimal estimation, resulting in smooth second-dimensional pixel coordinates. This achieves keypoint detection and noise suppression for edge computing. The GhostModule and channel pruning significantly reduce the number of model parameters and computational load, while INT8 quantization further compresses the model's bit width, enabling the model to meet the high real-time requirements of industrial sites on edge control computers. The Kalman filter eliminates coordinate jitter caused by light flicker and mechanical vibration by fusing temporal information, outputting hysteresis-free optimal estimates and avoiding the instability caused by single-frame detection noise being directly input into the pose calculation stage.

[0037] Preferably, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, including: Establish a perspective projection model of a pinhole camera, satisfying the following relation: ; in, and These represent the x-coordinate and y-coordinate of the pixel at the observation point in the real-time image, respectively. Indicates the scale factor. This represents the camera intrinsic parameter matrix. Let represent the rotation matrix to be solved. Denotes the translation vector to be solved. , , These represent the three-dimensional spatial coordinates of the workpiece in the local coordinate system; Construct the reprojection error objective function The following relation is satisfied: ; in, The key point detection model detects the first... Two-dimensional pixel coordinate observations of key points This represents the total number of key points. This indicates the perspective projection parameters of the pinhole camera. Indicates the preset first Three-dimensional geometric key points; The reprojection error objective function is solved iteratively using a nonlinear optimization algorithm. In each iteration, the pose update increment is calculated using the normal equation. The normal equation satisfies the following relation: ; in, This indicates that the reprojection error is related to the pose parameters. Jacobian matrix, This represents the damping factor used to adjust the convergence rate. Represents the identity matrix. This represents the pose update increment. The reprojection error parameter represents the current iteration round; When the average reprojection error, pose update increment, or number of iterations meets the preset joint termination condition, the optimal rotation matrix and translation vector are output, and the components are extracted from the translation vector as the real-time absolute depth information of the workpiece.

[0038] It should be noted that the pinhole camera perspective projection model is a mathematical model describing the mapping of three-dimensional points to a two-dimensional image plane through the ideal pinhole imaging principle. It expresses the perspective transformation relationship from three-dimensional to two-dimensional in matrix form, serving as a bridge connecting the three-dimensional geometry of a workpiece and the two-dimensional observation of the image. (Scale factor) This refers to the scaling factor in the perspective projection equation, reflecting the scaling relationship between the depth of a point in three-dimensional space and its pixel coordinates, ensuring the mathematical consistency of the projection equation in homogeneous coordinates. Camera intrinsic parameter matrix. It refers to the focal length He Guangxin The resulting 3×3 upper triangular matrix describes the camera's internal optical parameters and, after calibration, is used as a known constant in projection calculations. Rotation matrix. This refers to a 3×3 orthogonal matrix describing the spatial attitude of the workpiece coordinate system relative to the camera coordinate system. It contains three rotational degrees of freedom and determines the workpiece's tilt, flip, and other attitude states. Translation vector. This refers to a 3×1 vector describing the spatial position of the workpiece coordinate system origin relative to the camera coordinate system origin, containing three translational degrees of freedom, with its Z component representing the real-time absolute depth. The three-dimensional spatial coordinates X, Y, and Z refer to the three-dimensional coordinate values ​​of the workpiece's geometric key points in the workpiece's local coordinate system, predefined and derived in step S1 within the digital twin environment. The reprojection error objective function... Pose parameters are a mathematical function that measures the difference between the two-dimensional coordinates of a 3D point projected according to its current pose parameters and the observed coordinates detected by the network. Defined as half the sum of squares of the L2 norm, pose parameters are the object to be minimized by optimization algorithms. It refers to the rotational component Translation components The 6×1 vector represents the six-degree-of-freedom pose of the workpiece to be solved. The Jacobian matrix J refers to the reprojection error objective function with respect to the pose parameters. The first-order partial derivative matrix describes the sensitivity of the error to changes in pose parameters and is used to construct the normal equations. The damping factor μ is a parameter used in the Levenberg-Marquardt algorithm to adjust the iteration step size, adaptively switching between gradient descent and the Gauss-Newton method to ensure convergence stability. The identity matrix I is a square matrix with diagonal elements of 1 and all others of 0, multiplied by the damping factor in the normal equations to improve the matrix condition number. Pose update increment. This refers to the pose parameter correction calculated in each iteration, obtained by solving the normal equation, and used to update the current pose estimate. Reprojection error parameter. This refers to the residual vector between the projected coordinates and observed coordinates of each keypoint in the current iteration, and is the direct driving force for pose optimization. The joint termination condition is an iteration stopping criterion composed of a convergence accuracy threshold, a parameter increment threshold, and a maximum number of iterations, ensuring that the algorithm achieves a balance between accuracy, stability, and real-time performance.

[0039] Understandably, by establishing a perspective projection model of a pinhole camera, a reprojection error objective function is constructed. A nonlinear optimization algorithm is then used to iteratively solve the objective function. In each iteration, the pose update increment is calculated using normal equations. When the average reprojection error, pose update increment, or number of iterations meets a preset joint termination condition, the optimal rotation matrix and translation vector are output, and the optimal rotation matrix and translation vector are extracted from the translation vector. The component, as real-time absolute depth information, enables accurate 3D pose calculation based on 2D observation. A pinhole camera model establishes the perspective projection relationship between 3D spatial points and 2D image points. When the correspondence between 3D model points and 2D image points is known, the pose calculation problem is transformed into finding the optimal rotation matrix and translation vector that minimizes the error between the 2D coordinates of the 3D point projected according to this pose and the observed coordinates. The reprojection error is iteratively optimized using a nonlinear optimization algorithm, ultimately yielding accurate six-degree-of-freedom pose parameters. Simultaneously, the real-time absolute depth information extracted from the translation vector directly reflects the real-time distance between the workpiece and the camera. This depth information is dynamically updated with each frame of the video stream, providing continuous and accurate depth parameters for visual servo control. This overcomes the increased system complexity and cost caused by the lack of real-time depth in traditional monocular image Jacobian matrices, which necessitates the use of fixed estimates or additional depth sensors.

[0040] Preferably, the calculation of the robot arm's speed command using a visual servo control law includes: Substitute the real-time absolute depth information into the image Jacobian matrix of a single feature point. In the middle, the following relation is satisfied: ; in, This indicates the real-time absolute depth information. and These represent the x and y coordinates of the feature points in the normalized image coordinate system, respectively. The Jacobian matrices of several key points are stacked vertically to construct the composite image Jacobian matrix. And its pseudo-inverse matrix is ​​solved by singular value decomposition. The following relation is satisfied: ; in, The pseudo-inverse matrix of the Jacobian matrix of the synthesized image is represented by the following matrix: This represents the composite image Jacobian matrix, which is formed by stacking the Jacobian matrices of the images at each keypoint. The transpose of the Jacobian matrix of the synthesized image is represented by . Based on the error vector between the current feature point and the desired feature point in the real-time image, and the adaptive proportional control gain, the speed command is calculated. The following relation is satisfied: ; in, Indicates the speed command of the robotic arm. This represents the adaptive proportional control gain based on the error norm. This represents the image feature error vector obtained by subtracting the current feature point position from the desired feature point position; The adaptive proportional control gain Satisfying the relation: ; in, and These represent the preset maximum and minimum gain coefficients, respectively. This represents the preset convergence rate parameter. The norm of the image feature error vector is represented.

[0041] It should be noted that real-time absolute depth information This refers to the translation vector obtained from step S4. of The real-time distance of the extracted workpiece along the optical axis in the camera coordinate system is dynamically updated with each frame of the video stream, directly determining the value of the depth-related elements in the image's Jacobian matrix. The normalized image coordinate system refers to a two-dimensional coordinate system with the camera's optical center as the origin and the focal length as the normalization factor, converting the original pixel coordinates into dimensionless relative coordinates. This simplifies the mathematical expression of the Jacobian matrix for an image. (x-axis) This refers to the horizontal position of a feature point in the normalized image coordinate system, determined by... The calculated value reflects the horizontal offset of the feature point relative to the optical center. (Vertical axis) This refers to the vertical position of a feature point in the normalized image coordinate system, determined by... The calculated value reflects the vertical offset of feature points relative to the optical center. Image Jacobian matrix. Jacobian matrix refers to a 2×6 matrix that describes the mapping relationship between the moving velocity of a single feature point in a two-dimensional plane and the camera space velocity, establishing a differential mapping between image space and camera motion space. (This refers to the Jacobian matrix of the synthesized image.) This refers to a high-dimensional matrix formed by vertically stacking the image Jacobian matrices corresponding to multiple key points. In this embodiment, four 2×6 matrices are stacked to form an 8×6 matrix, constituting an overdetermined system of equations to enhance noise resistance. (Pseudo-inverse matrix) This refers to the generalized inverse matrix of the Jacobian matrix of the synthesized image, which is solved through singular value decomposition or analytical formulas to map high-dimensional image feature errors to six-dimensional camera spatial velocity. Transpose matrix This refers to the row and column interchange matrix of the synthesized image Jacobian matrix, which is used in analytical methods to find the pseudo-inverse. The multiplication forms the coefficient matrix of the normal equation. (Velocity command) This refers to the six-degree-of-freedom motion velocity commands sent from the computational layer to the robotic arm's joint actuators, including three linear velocity components and three angular velocity components, which are transmitted in real-time at a specific frequency through the ROS topic mechanism. Image feature error vector. This refers to the difference vector between the current feature point pixel coordinates and the desired feature point pixel coordinates, reflecting the degree of deviation between the current state and the target state of the robotic arm. Its direction indicates the direction of motion adjustment that the robotic arm needs to make. Adaptive proportional control gain. This refers to the control gain coefficient, which is dynamically adjusted based on the error norm. It takes a larger value for large errors at long distances to achieve rapid approximation, and attenuates to a smaller value for small errors at short distances to achieve smooth alignment. Maximum gain coefficient. This refers to the preset adaptive gain upper limit, which determines the maximum movement speed of the robotic arm at long distances, preventing the approximation process from being too slow. Minimum gain coefficient. This refers to the preset adaptive gain lower limit, which determines the minimum adjustment speed of the robotic arm at close range, ensuring the stability of the fine alignment process. Convergence rate parameter. It refers to the parameter that controls the curvature of the adaptive gain function as a function of the error norm, determining the gain from... Towards The speed of the transition. Error norm. It refers to the Euclidean length of the image feature error vector, which comprehensively reflects the overall deviation of all feature points and serves as the input variable for adaptive gain calculation.

[0042] Understandably, by substituting real-time absolute depth information into the image Jacobian matrix of a single feature point, and vertically stacking the image Jacobian matrices corresponding to multiple key points to construct a synthetic image Jacobian matrix, and solving its pseudo-inverse matrix through singular value decomposition, the robot's speed command is calculated based on the error vector between the current feature point and the desired feature point in the real-time image, as well as the adaptive proportional control gain, thus achieving six-degree-of-freedom closed-loop control based on image feature errors. The dynamic incorporation of real-time depth information ensures that the mapping relationship between the moving speed of image feature points and the camera's spatial velocity remains accurate throughout the robot's approach from a distant to a near distance. The synthetic matrix formed by vertically stacking the Jacobian matrices corresponding to multiple key points constitutes an overdetermined system of equations, giving the system strong robustness to detection noise of individual feature points. The adaptive proportional control gain is dynamically adjusted according to the error norm, achieving near-zero speed at long distances. High-gain, fast, large-step approximation, decays exponentially at close range. This enables compliant grasping with anti-overshoot capability. The speed command maps the two-dimensional image error to the six-dimensional camera space velocity through pseudo-inverse operation, and then converts it into the drive command of each joint of the robotic arm. This allows the robotic arm to perceive and correct trajectory deviations in real time as it approaches the workpiece, eliminating the direct dependence on absolute three-dimensional coordinates and reducing the system's sensitivity to calibration errors.

[0043] like Figure 5As shown, the execution flow of visual servo closed-loop control based on image feature error is illustrated. After the system starts and receives the trigger signal sent by the industrial PLC, the industrial camera first acquires a real-time image of the current workstation, and the YOLO network extracts the current feature point position of the workpiece from the image. At the same time, the system loads the pre-stored expected feature point positions. Calculate the current image feature error Then, it checks whether the error e is less than the preset grasping threshold. If the condition is met, it determines that the robotic arm has aligned with the target and triggers the grasping or processing command, ending the current work cycle. If the error does not meet the threshold condition, it enters the dynamic correction process, sequentially extracting the current camera pose parameters Z obtained by PnP, constructing the image Jacobian matrix Ls corresponding to each feature point, calculating its generalized inverse matrix, and then applying the visual servo control law. The system calculates the six-degree-of-freedom speed command at the end of the robotic arm and sends the speed command to the robotic arm controller in real time via a Twist message through a ROS topic. This drives the six-degree-of-freedom robotic arm to make fine adjustments. After the physical movement of the robotic arm changes the camera's perspective, the system re-acquires the image and iterates the above error calculation and correction process to form a closed-loop control loop based on image feature errors until the error converges to within the grasping threshold.

[0044] Preferably, it also includes adaptive feedback and active learning mechanisms: Real-time monitoring of the confidence score output by the keypoint detection model, construction of a spatial joint confidence evaluation model, and calculation of the obtained spatial joint confidence. Satisfying the relation: ; in, This indicates the confidence level for assessing the global existence of the workpiece. Indicates the first Local pose confidence at key points Indicates the total number of key points. , , These represent the preset weight coefficients corresponding to global existence, overall pose quality, and local defect penalty, respectively, and satisfy the normalization condition. The spatial joint confidence score is time-series filtered using a moving exponential weighted moving average algorithm to obtain the final smoothed confidence score. The following relation is satisfied: ; in, express The final smooth confidence level at any given moment. This represents the final smooth confidence level at the previous time step. This represents the preset time smoothing factor; When the final smooth confidence level is continuously lower than the safety threshold, the visual blind zone of the long-tail edge condition is determined to be triggered, the observation update of the key point detection model is suspended, and the Kalman filter outputs the predicted feature point coordinates based on the kinematic prior model to maintain visual servo closed-loop control. Real-time images located in the visual blind spot are automatically captured as difficult examples and mixed into the training dataset. The key point detection model is then fine-tuned by freezing the backbone network and incrementally training with a reduced learning rate.

[0045] It should be noted that the confidence score refers to the deterministic measure of the keypoint detection model's detection results for each keypoint in an image. It is output as a probability value and reflects the model's self-assessment of the current detection quality. Spatial joint confidence score. This refers to a comprehensive confidence index that integrates three dimensions: global presence, overall pose quality, and local defect penalty, to evaluate the reliability of the detection results in the current frame from a spatial distribution perspective. Global presence assessment confidence. Local pose confidence refers to the overall probability of the model determining whether a target workpiece exists in an image, reflecting the integrity and detectability of the workpiece within the field of view. This refers to the individual certainty measure of the model's detection result for the i-th keypoint, reflecting the clarity and discernibility of that keypoint under the current image conditions. Weight coefficients , , These refer to preset weighting factors corresponding to global existence, overall attitude quality, and local defect penalties, respectively, satisfying the normalization condition. By adjusting the proportions of these three factors, the evaluation model can be adapted to production scenarios with different accuracy requirements. The moving average weighted average algorithm is a time-series filtering method that weights and fuses historical confidence scores with the current spatial joint confidence score to output a smooth final confidence score, suppressing the interference of abnormal fluctuations in single frames on system decision-making. Final smoothed confidence score. This refers to the confidence output after time-series filtering, which comprehensively reflects the detection quality at the current and recent historical moments, serving as the decision-making basis for determining whether a visual blind spot has been triggered. Time smoothing factor. This refers to the parameter that controls the proportion of historical confidence in the filtering process; it is typically set to 0.8, or larger values. The value makes the output more dependent on historical states, enhancing the ability to suppress transient noise. The safety threshold refers to the lower limit of confidence in determining whether the vision inspection system is in a reliable working state. When the final smoothed confidence level consistently falls below this threshold, the system determines that the current environment exceeds the model's reliable detection range. The long-tail edge visual blind zone refers to the visual inspection failure area caused by abnormally strong light, workpiece oil stains, or severe occlusion at the edge of the production line. Within this area, the deep learning model cannot output reliable keypoint coordinates. Observation update refers to the process by which the Kalman filter corrects its state estimate using new detection results at each time step. Suspending observation update means that the filter no longer trusts external detection results and instead relies entirely on internal predictions. The kinematic prior model is a mathematical model describing the motion of feature points on the image plane. Based on the assumption of uniform velocity or uniform acceleration, it predicts the feature point position at the next time step, maintaining the coordinate output required for closed-loop control during the visual blind zone. Hard examples refer to abnormal frame images automatically captured during the visual blind zone. These images contain extreme cases that are difficult for the model to handle in real production lines and are valuable training resources for improving model robustness. Incremental training refers to a learning method that uses new samples to fine-tune local parameters based on an existing pre-trained model. By freezing the backbone network and reducing the learning rate, it prevents the model from forgetting existing knowledge when adapting to new data.

[0046] Understandably, by monitoring the confidence score output by the keypoint detection model in real time, a spatial joint confidence evaluation model is constructed and the spatial joint confidence is calculated. The spatial joint confidence is then filtered temporally using a moving average algorithm with a sliding exponential weighted average to obtain the final smooth confidence. When the final smooth confidence continuously falls below the safety threshold, a visual blind zone in the long-tail edge condition is triggered, and the observation update of the keypoint detection model is suspended. The Kalman filter outputs the predicted feature point coordinates based on the kinematic prior model to maintain visual servo closed-loop control. At the same time, real-time images in the visual blind zone are automatically captured as hard examples and mixed into the training dataset. The keypoint detection model is fine-tuned by freezing the backbone network and incremental training with a reduced learning rate, thereby achieving a dynamic self-healing cycle during production line operation. Spatial joint confidence comprehensively evaluates detection quality from three dimensions: global existence, overall posture quality, and local defect penalty, avoiding misjudgment based on a single indicator. Temporal filtering suppresses decision jitter caused by instantaneous noise by fusing historical information. During visual blind spots, the Kalman filter continuously outputs predicted coordinates based on kinematic priors, ensuring smooth and uninterrupted robotic arm movements. Incremental training of difficult samples adopts a strategy of freezing the backbone network and using a very small learning rate to prevent catastrophic forgetting when the model adapts to new working conditions, thus achieving robust autonomous and continuous evolution of the system.

[0047] Workpiece pose control system based on digital twin and vision servo, such as Figure 2 As shown, it includes: The space definition module is used to define several three-dimensional geometric key points based on the geometric features of the workpiece to be processed in a digital twin simulation environment. The data evolution module is used to automatically render and generate a training dataset with 6D pose ground truth and two-dimensional pixel coordinate annotations through randomized sampling, and to train the key point detection model using the training dataset. The perception extraction module is used to acquire real-time images of the workpiece to be processed and to extract the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model. The pose calculation module is used to combine the three-dimensional geometric key points and camera intrinsic parameters, use projection relationships and nonlinear optimization algorithms to calculate the rotation matrix and translation vector of the workpiece relative to the camera, and extract the real-time absolute depth information of the workpiece from the translation vector. The mapping construction module is used to guide the end effector of the robotic arm to rotate to a preset posture according to the rotation matrix, and to substitute the real-time absolute depth information into the image Jacobian matrix to construct the synthetic image Jacobian matrix. The closed-loop control module is used to calculate the speed command of the robotic arm based on the error vector between the current feature point and the desired feature point in the real-time image, and drive the robotic arm to perform dynamic correction until the error meets the grasping threshold.

[0048] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0049] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A workpiece pose control method based on digital twin and visual servoing, characterized in that, include: In a digital twin simulation environment, several three-dimensional geometric key points are defined based on the geometric features of the workpiece to be processed; A training dataset with 6D pose ground truth and 2D pixel coordinate annotations is automatically generated by randomized sampling and rendering, and the key point detection model is trained using the training dataset. Acquire a real-time image of the workpiece to be processed, and use the trained key point detection model to extract the two-dimensional pixel coordinates of each key point in the real-time image; Combining the three-dimensional geometric key points and camera intrinsic parameters, the rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, and the real-time absolute depth information of the workpiece is extracted from the translation vector. The robotic arm end effector is guided to rotate to a preset posture according to the rotation matrix, and the real-time absolute depth information is substituted into the image Jacobian matrix to construct the synthetic image Jacobian matrix. Based on the error vector between the current feature point and the desired feature point in the real-time image, the speed command of the robotic arm is calculated using the visual servo control law, and the robotic arm is driven to perform dynamic correction until the error meets the grasping threshold.

2. The workpiece pose control method based on digital twin and visual servoing according to claim 1, characterized in that, The training dataset, automatically generated through randomized sampling and rendering, includes 6D pose ground truth values ​​and 2D pixel coordinate annotations. The virtual camera is controlled by parametric scripts to perform randomized sampling around the virtual workpiece, and the lighting intensity, shooting distance, and camera shooting angle of the virtual scene are randomly changed. Extract the absolute 6D pose truth value of the virtual workpiece in the world coordinate system, and project the three-dimensional geometric key points onto the two-dimensional image plane using the intrinsic parameter matrix of the virtual camera. Render and export the RGB image, and simultaneously generate a label file including the class confidence score and the two-dimensional pixel coordinates to obtain the training dataset.

3. The workpiece pose control method based on digital twin and visual servoing according to claim 1, characterized in that, Extracting the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model includes: The backbone network of the key point detection model is replaced with GhostModule. Core features are generated by convolutional operations, and similar features are generated by linear operations. During training, channel pruning is used to remove redundant features, and the trained key point detection model is quantized using INT8. The acquired real-time image is input into the key point detection model, multi-scale features are extracted using the backbone network, and the category confidence of the workpiece and the first two-dimensional pixel coordinates of each key point are output by the prediction head. A Kalman filter is connected in series at the back end of the key point detection model. A state vector is constructed using the two-dimensional pixel coordinates and their rate of change. The first two-dimensional pixel coordinates are then optimally estimated to obtain smooth second two-dimensional pixel coordinates.

4. The workpiece pose control method based on digital twin and visual servoing of claim 1, wherein, The rotation matrix and translation vector of the workpiece relative to the camera are calculated using projection relationships and nonlinear optimization algorithms, including: Establish a perspective projection model of a pinhole camera, satisfying the following relation: ； in, and These represent the x-coordinate and y-coordinate of the pixel at the observation point in the real-time image, respectively. Indicates the scale factor. Represents the camera intrinsic parameter matrix. Let represent the rotation matrix to be solved. Denotes the translation vector to be solved. , , These represent the three-dimensional spatial coordinates of the workpiece in the local coordinate system; Constructing a reprojection error objective function satisfying the relationship: ； in, The key point detection model detects the first... Two-dimensional pixel coordinate observations of key points This represents the total number of key points. This indicates the perspective projection parameters of the pinhole camera. Indicates the preset first Three-dimensional geometric key points; The reprojection error objective function is solved iteratively using a nonlinear optimization algorithm. In each iteration, the pose update increment is calculated using the normal equation. The normal equation satisfies the following relation: ； in, This indicates that the reprojection error is related to the pose parameters. Jacobian matrix, This represents the damping factor used to adjust the convergence rate. Represents the identity matrix. This represents the pose update increment. The reprojection error parameter represents the current iteration round; When the average reprojection error, pose update increment, or number of iterations meets the preset joint termination condition, the optimal rotation matrix and translation vector are output, and the components are extracted from the translation vector as the real-time absolute depth information of the workpiece.

5. The workpiece pose control method based on digital twin and visual servoing of claim 1, wherein, The speed commands for calculating the robotic arm using visual servo control laws include: substituting the real-time absolute depth information into an image Jacobian matrix of a single feature point where the relationship is satisfied: ； wherein, denotes the real-time absolute depth information, and denote the horizontal and vertical coordinates of the feature point in the normalized image coordinate system, respectively. The image Jacobian matrices corresponding to several key points are stacked longitudinally to construct a synthetic image Jacobian matrix and its pseudo-inverse matrix is solved by singular value decomposition , satisfying the relationship ； wherein, denotes a pseudo-inverse matrix of the composite image Jacobian matrix, denotes a composite image Jacobian matrix composed of the image Jacobian matrices of the respective key points, denotes a transpose matrix of the composite image Jacobian matrix; calculating a velocity command based on the error vector of the current feature point in the real-time image and the desired feature point and an adaptive proportional control gain , satisfying a relationship: ； in, Indicates the speed command of the robotic arm. This represents the adaptive proportional control gain based on the error norm. This represents the image feature error vector obtained by subtracting the current feature point position from the desired feature point position; the adaptive proportional control gain satisfies the relationship: ； wherein, and respectively represent a preset maximum gain coefficient and a minimum gain coefficient, represents a preset convergence rate parameter, represents a norm of the image feature error vector.

6. The workpiece pose control method based on digital twin and visual servoing of claim 1, wherein, It also includes adaptive feedback and active learning mechanisms: Real-time monitoring of the confidence score output by the key point detection model, constructing a spatial joint confidence evaluation model, and calculating the obtained spatial joint confidence satisfies the relationship ； in, This indicates the confidence level for assessing the global existence of the workpiece. Indicates the first Local pose confidence at key points Indicates the total number of key points. , , These represent the preset weight coefficients corresponding to global existence, overall pose quality, and local defect penalty, respectively, and satisfy the normalization condition. temporally filtering the spatial joint confidence using a sliding exponential moving average algorithm to obtain a final smoothed confidence satisfying the relationship: ； wherein, denotes the final smoothed confidence at the time instant, denotes the final smoothed confidence at the previous time instant, denotes a preset time smoothing factor; When the final smooth confidence level is continuously lower than the safety threshold, the visual blind zone of the long-tail edge condition is determined to be triggered, the observation update of the key point detection model is suspended, and the Kalman filter outputs the predicted feature point coordinates based on the kinematic prior model to maintain visual servo closed-loop control. Real-time images located in the visual blind spot are automatically captured as difficult examples and mixed into the training dataset. The key point detection model is then fine-tuned by freezing the backbone network and incrementally training with a reduced learning rate.

7. A workpiece pose control system based on digital twinning and visual servoing, characterized in that, include: The space definition module is used to define several three-dimensional geometric key points based on the geometric features of the workpiece to be processed in a digital twin simulation environment. The data evolution module is used to automatically render and generate a training dataset with 6D pose ground truth and two-dimensional pixel coordinate annotations through randomized sampling, and to train the key point detection model using the training dataset. The perception extraction module is used to acquire real-time images of the workpiece to be processed and to extract the two-dimensional pixel coordinates of each key point in the real-time image using the trained key point detection model. The pose calculation module is used to combine the three-dimensional geometric key points and camera intrinsic parameters, use projection relationships and nonlinear optimization algorithms to calculate the rotation matrix and translation vector of the workpiece relative to the camera, and extract the real-time absolute depth information of the workpiece from the translation vector. The mapping construction module is used to guide the end effector of the robotic arm to rotate to a preset posture according to the rotation matrix, and to substitute the real-time absolute depth information into the image Jacobian matrix to construct the synthetic image Jacobian matrix. The closed-loop control module is used to calculate the speed command of the robotic arm based on the error vector between the current feature point and the desired feature point in the real-time image, and drive the robotic arm to perform dynamic correction until the error meets the grasping threshold.

Images