An industrial robot vision tracking method, apparatus, device and medium

Abstract

This application discloses a visual tracking method, apparatus, device, and medium for industrial robots, relating to the field of industrial robot technology. The method includes: acquiring a continuous image stream and global motion data of the industrial robot's end effector camera; performing optical flow tracing on the current frame and the previous frame to obtain feature point displacement vectors, and calculating mechanical vibration energy indices based on these vectors; adaptively adjusting a smoothing factor based on the vibration indices, filtering the global motion data to obtain a stable image transformation matrix, and performing a geometric transformation on the current frame image to generate a stable image; inputting the stable image into a Transformer architecture detection model, and obtaining the detection result through cross-attention calculation; extracting the target feature vector of the current frame based on the target bounding box, performing dual checks of occlusion gating and consistency gating, executing a corresponding memory update strategy, and outputting the target tracking result. This method can achieve accurate tracking.

Description

Technical Field

[0001] This application relates to the field of industrial robot technology, and in particular to an industrial robot vision tracking method, device, equipment and medium. Background Technology

[0002] With the development of intelligent manufacturing, robot dynamic grasping has become an important part of industrial automation. It relies on visual perception and motion control to accurately grasp moving targets. However, problems such as image jitter and end-to-end latency in industrial scenarios place extremely high demands on grasping accuracy.

[0003] Traditional robot vision grasping solutions mostly adopt a serial process of image stabilization, detection, tracking, and control, with each module operating independently. Although some solutions introduce feedforward control, they cannot simultaneously overcome image jitter and control lag, thus failing to achieve accurate tracking. Summary of the Invention

[0004] This application provides a visual tracking method, device, equipment, and medium for industrial robots, which can achieve accurate tracking.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a visual tracking method for industrial robots, including: Acquire a continuous image stream captured by the end-effector camera of the industrial robot, as well as global motion data of the end-effector; wherein, the continuous image stream includes the current frame image and the previous frame image; Optical flow tracing is performed on the current frame image and the previous frame image to obtain the feature point displacement vector of the current frame image relative to the previous frame image; Calculate the mechanical vibration energy index based on the displacement vector of the characteristic point; The smoothing factor is adjusted based on the mechanical vibration energy index, and the global motion data is adaptively filtered based on the adjusted smoothing factor to obtain the image stabilization transformation matrix. A stable image is obtained by performing a geometric transformation on the current frame image using a stable image transformation matrix; The stabilized image is input into the detection model based on the Transformer architecture, and cross-attention calculation is performed to obtain the detection result; the detection result includes the target bounding box; Based on the target bounding box in the detection results, extract the target feature vector of the current frame from the feature map of the stabilized image; The target feature vector of the current frame is double-checked by occlusion gating and consistency gating. The corresponding memory update strategy is executed according to the double check result to obtain the target tracking result.

[0006] In some possible implementations, occlusion gating and consistency gating are used to perform dual verification on the target feature vector of the current frame. Based on the dual verification result, the corresponding memory update strategy is executed to obtain the target tracking result, including: When the occlusion gating check fails but the consistency gating check passes, the normal update strategy is executed, and the target feature vector of the current frame is written into the short-term memory queue to obtain the target tracking result. When both the occlusion gating check and the consistency gating check fail, a soft reset strategy is executed, the short-term memory queue is discarded, and the target feature vector is re-extracted from the feature map of the stabilized image based on the target bounding box in the detection result, as the initial feature of the new short-term memory queue, to obtain the target tracking result; When the occlusion gating verification passes, the memory freeze strategy is executed, which prohibits the writing of the target feature vector of the current frame into the short-term memory queue and does not update the long-term memory; at the same time, the memory features of the previous frame are reused to predict the position and obtain the target tracking result. Among them, passing the occlusion gating check indicates that the target in the current frame is occluded, and passing the consistency gating check indicates that the target features in the current frame are consistent with the target baseline features in the long-term memory.

[0007] In some possible implementations, the occlusion gating check passes, including: The occlusion score is calculated by performing an occlusion score on the current frame target feature vector; When the occlusion score is greater than the first preset threshold, the occlusion gating verification is deemed to have passed. If the occlusion score is not greater than the first preset threshold, the occlusion gating verification is deemed to have failed.

[0008] In some possible implementations, the consistency gating check passes, including: Retrieve target baseline features from long-term memory; The similarity score is obtained by calculating the similarity between the target feature vector of the current frame and the target baseline features; When the similarity score is not less than the second preset threshold, the consistency gating check is deemed to have passed. When the similarity score is less than the second preset threshold, the consistency gating check is deemed to have failed.

[0009] In some possible implementations, the mechanical vibration energy index is calculated based on the displacement vector of the characteristic point, including: The global motion model is obtained by fitting the displacement vectors of feature points using the random sampling consensus algorithm; Based on the global motion model and the displacement vector of each feature point, calculate the motion residual vector of each feature point; The weighted variance of all motion residual vectors is calculated to obtain the residual field variance; where the residual field variance is used to characterize the degree of non-rigid distortion of the image caused by mechanical vibration. The global velocity vector in the global motion data is differentiated to obtain the global acceleration; the global acceleration is used to characterize the impact strength of the robotic arm. The weighted global acceleration is obtained by multiplying the global acceleration by the weighting coefficients; the weighting coefficients are determined based on the shutter speed. The sum of the residual field variance and the weighted global acceleration is determined as the mechanical vibration energy index.

[0010] In some possible implementations, the stabilized image is input into a Transformer-based detection model, cross-attention is calculated, and the detection results are obtained, including: The stabilized image is input into the detection model, and the backbone network in the detection model performs multi-scale feature extraction on the stabilized image to obtain a multi-scale feature map. The detection model includes a backbone network, a sinusoidal position coding layer, a learnable object query vector, and a cross-attention decoder. By adding sinusoidal position coding to the multi-scale feature map through a sinusoidal position coding layer, a feature sequence carrying spatial position information is obtained. The learnable object query vector and feature sequence are input into the cross-attention decoder. After multiple layers of cross-attention interaction, the detection result is output. The detection result includes multiple prediction results, each of which includes a bounding box and a class probability. The multiple prediction results are sparse and unique outputs obtained by training one-to-one bipartite graph matching, without the need for non-maximum suppression post-processing.

[0011] Among the possible implementations are: The target tracking results are transformed to obtain the current three-dimensional pose in the industrial robot base coordinate system. The total latency of the entire link is determined by the image acquisition time, processing completion time, and average communication time. Construct a target motion state vector including position, velocity, and acceleration based on the current three-dimensional pose; The target motion state vector is filtered using a Kalman filter to obtain the estimated velocity and estimated acceleration. Based on the kinematic model, the predicted pose is calculated according to the current three-dimensional pose, estimated velocity, estimated acceleration and total end-to-end delay; The predicted pose is sent to the robot controller so that the robot controller can generate drive commands based on the predicted pose.

[0012] Secondly, this application provides an industrial robot vision tracking device, comprising: The acquisition module is used to acquire a continuous image stream captured by the end-effector camera of the industrial robot, as well as global motion data of the end-effector of the industrial robot; wherein, the continuous image stream includes the current frame image and the previous frame image; The calculation module performs optical flow tracing on the current frame image and the previous frame image to obtain the feature point displacement vectors of the current frame image relative to the previous frame image; calculates the mechanical vibration energy index based on the feature point displacement vectors; adjusts the smoothing factor based on the mechanical vibration energy index, and performs adaptive filtering on the global motion data based on the adjusted smoothing factor to obtain the image stabilization transformation matrix; performs a geometric transformation on the current frame image using the image stabilization transformation matrix to obtain a stabilized image; inputs the stabilized image into a detection model based on the Transformer architecture for cross-attention calculation to obtain the detection result; the detection result includes the target bounding box; and extracts the target feature vector of the current frame from the feature map of the stabilized image based on the target bounding box in the detection result. The tracking module is used to perform dual verification on the target feature vector of the current frame through occlusion gating and consistency gating, and execute the corresponding memory update strategy according to the dual verification result to obtain the target tracking result.

[0013] Thirdly, this application provides a computing device, including a memory and a processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0015] Fifthly, this application provides a computer program product comprising one or more computer instructions, wherein when the computer instructions are executed by a computer, the computer performs the method as described in any one of the first aspects.

[0016] As can be seen from the above technical solution, this application has at least the following beneficial effects: In this application, excellent results are achieved across multiple dimensions through the organic integration of physical image stabilization, Transformer detection, dual-gated memory update, and predictive feedforward control. Regarding visual image stabilization, a mechanical vibration energy field model is constructed, and vibration energy is quantified in real-time using optical flow residual variance and global acceleration. A Boltzmann distribution-based attenuation function is used to dynamically adjust the smoothing factor, achieving an adaptive optimal balance between stabilization and dynamic response. During high-energy vibrations, the smoothing factor automatically approaches zero, freezing the visual input-dependent historical trajectory and eliminating input-end feature noise at its source. During stable motion, the smoothing factor recovers to its baseline value, allowing the system to quickly respond to target changes. Compared to traditional mean filtering or electronic image stabilization methods, this scheme eliminates the need for multi-frame accumulation or complex feature matching, avoiding phase delay issues and enabling the visual servoing system to maintain stable convergence under millisecond-level response requirements.

[0017] In object detection, an end-to-end detection model based on the Transformer architecture is adopted, utilizing a global attention mechanism to replace the local convolutions of traditional CNNs, thereby improving the detection capability of objects in vibration-blurred images. Through learnable object query vectors and one-to-one bipartite graph matching training, the model output is naturally sparse and unique, eliminating the need for non-maximum suppression post-processing and fundamentally avoiding the inherent defect of traditional detectors in densely stacked scenes that incorrectly suppress adjacent real objects. In industrial scenarios with closely arranged precision workpieces, the false negative rate of stacked objects is reduced, the bounding box localization error is less than 1 pixel, and the end-to-end output ensures deterministic inference latency.

[0018] In target tracking, a streaming memory segmentation network with a dual-gating mechanism is introduced, employing occlusion gating and consistency gating as two logical valves to intelligently manage feature updates. When occlusion is detected, the system automatically executes a memory freeze strategy, prohibiting the writing of occluded features into the memory bank to prevent memory contamination. When feature drift is detected, the system executes a soft reset strategy, discarding contaminated short-term memories and re-initializing them. Compared to the passive weighted updates of existing algorithms such as DeepSORT, this application endows the system with the ability to maintain the target identifier unchanged for a long time even under large-area occlusion, solving the problems of frequent identifier switching and tracking chain breakage in strong occlusion environments.

[0019] Regarding dynamic tracking accuracy, a predictive feedforward control model incorporating end-to-end delay monitoring was constructed. This model measures the total delay from image acquisition to industrial robot execution in real time and calculates the target's predicted pose at the execution moment based on a kinematic model. By mapping the controlled target from the past to the future, the hysteresis error caused by the inherent system delay is effectively reduced. In high-speed operation scenarios with conveyor belt speeds reaching 1 meter per second, the absolute positioning error of dynamic grasping is reduced, achieving zero-hysteresis accurate interception and grasping of high-speed moving targets.

[0020] In terms of overall system performance, a closed-loop system covering all time and space from perception to control is formed through the collaborative design of four modules: physical image stabilization, target detection, memory tracking, and predictive control. The joint training loss function incorporates image stabilization quality, detection accuracy, memory robustness, and control accuracy into a unified optimization objective, ensuring that each module promotes the others rather than operating independently. This method outperforms traditional schemes in four dimensions: vibration resistance, occlusion resistance, dynamic accuracy, and dense detection. Detection frame center bounce is reduced, and the target maintains a very high capture success rate even with high occlusion ratios. This strongly supports applications in high-end industrial manufacturing fields, such as precision component assembly and precision electronic manufacturing, which have stringent requirements for accuracy and robustness. Ultimately, accurate tracking is achieved.

[0021] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0022] Figure 1 An application environment diagram of an industrial robot visual tracking method provided in this application embodiment; Figure 2 A flowchart illustrating an industrial robot vision tracking method provided in this application embodiment; Figure 3 A structural diagram of an industrial robot vision tracking device provided in this application embodiment; Figure 4 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0023] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0024] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: Industrial robot vision tracking is a core technology for achieving automated grasping, assembly, and dynamic operations in high-end intelligent manufacturing. It uses a camera mounted on the robot's end effector to acquire image information, combining visual perception and motion control to achieve target localization and tracking. In an eye-to-hand architecture, the camera moves synchronously with the robotic arm, flexibly covering the work area, and has been widely applied in precision assembly, assembly line sorting, and electronics manufacturing. Currently, mainstream vision tracking solutions typically employ a combined process of image stabilization, target detection, feature matching, and Kalman filtering tracking. First, the shaky image is smoothed, then the target is located using a detection model, and finally, the tracking state is maintained based on a motion model. Traditional solutions suffer from image distortion and feature failure under mechanical vibration interference, leading to poor tracking accuracy.

[0026] In view of this, embodiments of this application provide a visual tracking method for industrial robots. To make the technical solution of this application clearer and easier to understand, the application scenarios of the technical solution of this application are described below with reference to the accompanying drawings. Figure 1 As shown, this figure is an application environment diagram provided by an embodiment of this application.

[0027] In this application environment, server 104 acts as the execution entity, and server 104 establishes a real-time data interaction channel with terminal 102. Terminal 102 is used to display visual tracking images, parameter configurations, and status feedback. Server 104 receives start / stop commands and parameter settings from terminal 102, and simultaneously transmits visual image stabilization results, target tracking position, and system operating status back to terminal 102 for display, realizing human-machine collaborative monitoring and intervention.

[0028] To make the technical solution of this application clearer and easier to understand, the following describes an industrial robot visual tracking method provided by an embodiment of this application, in conjunction with the above application scenarios. Figure 2 As shown, this figure is a flowchart illustrating an industrial robot vision tracking method provided in an embodiment of this application. The industrial robot vision tracking method includes: S201. Acquire the continuous image stream captured by the end-effector camera of the industrial robot, as well as the global motion data of the end-effector of the industrial robot.

[0029] The continuous image stream includes the current frame image and the previous frame image.

[0030] Industrial robots refer to automated operating equipment used in high-end industrial manufacturing, with vision sensors rigidly connected to their end flanges.

[0031] An end-effector camera is a vision sensor that is rigidly connected to the end flange of an industrial robot and is used to acquire a continuous stream of images.

[0032] A continuous image stream refers to a sequence of multiple frames of images that are continuously arranged over time and captured by the end-effector camera of an industrial robot at a preset frame rate.

[0033] Global motion data refers to the global motion velocity vector of the end effector of an industrial robot, which is used to characterize the active motion trajectory of the robotic arm.

[0034] For example, during this step, the industrial robot's end-effector camera acquires a continuous stream of images at a preset frame rate and transmits the acquired continuous image stream to the server. The server obtains the current frame image at the current moment and the previous frame image from the continuous image stream. Simultaneously, the server reads the global motion velocity vector of the robot's end-effector from the industrial robot's control system. This global motion velocity vector represents the active motion speed and direction of the robotic arm at the current moment. The server uses the current frame image, the previous frame image, and the global motion velocity vector as input data for subsequent steps.

[0035] S202. Perform optical flow tracing on the current frame image and the previous frame image to obtain the feature point displacement vector of the current frame image relative to the previous frame image.

[0036] Optical flow tracing refers to a technique that determines the correspondence between pixels in adjacent frames by calculating the motion information of pixels in the time domain in an image sequence.

[0037] Feature points refer to sparse corner points with texture features extracted from an image.

[0038] The feature point displacement vector is the displacement vector between the position of the same feature point in the current frame image and its position in the previous frame image. It is used to characterize the direction and distance of the feature point's movement on the image plane.

[0039] For example, the server utilizes an improved Lucas-Kanade pyramid optical flow algorithm to construct an image pyramid between the current frame and the previous frame. At the top of the pyramid, large displacements are roughly estimated, and then refined layer by layer downwards. The server tracks the corresponding positions of each feature point in the previous frame in the current frame and calculates the optical flow vector for each feature point by minimizing the photometric error within the local window. The server performs a difference operation between the position of each feature point in the previous frame and its position in the current frame to obtain the feature point displacement vector, which contains information about the magnitude and direction of the displacement. The server uses the feature point displacement vectors of all feature points as output data for subsequent calculations of mechanical vibration energy indicators.

[0040] For example, using the improved Lucas-Kanade pyramid optical flow algorithm, the previous frame image... Sparse corner feature set in As the tracking target, in the current frame image of the image to be stabilized. Find the corresponding position of each feature point in the data to obtain the sparse corner feature set corresponding to the current frame. Then, based on the position of each feature point in the previous frame and its corresponding position in the current frame, such as the position coordinates of the i-th sparse corner feature in the previous frame image. The coordinates of the corresponding position of the i-th sparse corner feature in the current frame image The displacement vector of each feature point is calculated, such as Let represent the displacement vector of the i-th feature point. The pyramid structure ensures tracking stability even under large displacements caused by rapid robot movement. Here, n is the total number of sparse corner feature points, which is the set of sparse corner feature points in the previous frame. The set of sparse corner features in the current frame image The total number of feature points included is a positive integer.

[0041] S203. Calculate the mechanical vibration energy index based on the displacement vector of the characteristic point.

[0042] One possible approach involves fitting the displacement vectors of feature points using a random sampling consensus algorithm to obtain a global motion model; calculating the motion residual vector for each feature point based on the global motion model and the displacement vectors of each feature point; calculating the weighted variance of all motion residual vectors to obtain the residual field variance; differentiating the global motion velocity vector in the global motion data to obtain the global acceleration; multiplying the global acceleration by the weighting coefficients to obtain the weighted global acceleration; and determining the sum of the residual field variance and the weighted global acceleration as the mechanical vibration energy index.

[0043] Among them, the random sampling consensus algorithm is an iterative method for estimating mathematical model parameters through random sampling and consensus verification, used to fit robust model parameters from data containing outliers.

[0044] A global motion model is a homography transformation matrix or affine transformation model obtained by fitting the displacement vectors of feature points. It is used to characterize the global motion trend of the camera, i.e., the active motion trajectory of the robot.

[0045] The motion residual vector is the difference vector between the feature point displacement vector and the theoretical displacement vector calculated through the global motion model. It is used to characterize the degree of non-rigid distortion in a local part of the image.

[0046] The residual field variance refers to the weighted variance of all motion residual vectors, used to characterize the degree of non-rigid distortion of an image caused by mechanical vibration.

[0047] Global acceleration refers to the acceleration value obtained by performing time differentiation on the global motion velocity vector, and is used to characterize the impact strength of the robotic arm.

[0048] The weighting coefficient is determined based on the shutter exposure time. The longer the exposure time, the greater the impact of motion blur on the residual, and the weighting coefficient increases accordingly.

[0049] The mechanical vibration energy index is the sum of the residual field variance and the weighted global acceleration, used to comprehensively characterize the severity of mechanical vibration at the current moment.

[0050] For example, in performing this step, the server first obtains the feature point displacement vectors of each feature point. The server uses a random sample consensus algorithm to iteratively fit the feature point displacement vectors. Each time, it randomly selects the smallest sample set to estimate the global motion model parameters, and then calculates the consistency between all feature point displacement vectors and the model. Feature points that meet a preset error threshold are retained as inliers. After multiple iterations, the model with the most inliers is selected as the final global motion model. Based on the fitted global motion model and the feature point displacement vectors of each feature point, the server calculates the motion residual vector for each feature point. Specifically, it subtracts the theoretical displacement vector calculated for the feature point's position using the global motion model from the feature point displacement vector. The server performs a weighted variance calculation on the motion residual vectors of all feature points. That is, it first calculates the square of the magnitude of each motion residual vector, then sums all the squared values ​​and divides by the total number of feature points to obtain the residual field variance. The larger the residual field variance value, the more severe the non-rigid distortion of the image caused by mechanical vibration. The server performs time differentiation on the global motion velocity vector in the global motion data, that is, calculates the rate of change of the global motion velocity vector per unit time to obtain the global acceleration. The larger the global acceleration value, the greater the impact intensity of the robotic arm. The server determines the weighting coefficients based on the camera's shutter exposure time. The longer the shutter exposure time, the greater the impact of motion blur on the residual, and the larger the weighting coefficients are set accordingly. The server multiplies the global acceleration by the weighting coefficients to obtain the weighted global acceleration. The server adds the residual field variance to the weighted global acceleration, and the sum is determined as the mechanical vibration energy index at the current moment. This index is used to characterize the severity of the current mechanical vibration and serves as the input parameter for subsequent adaptive filtering.

[0051] For example, the RANSAC (Random Sample Consensus) algorithm is used to fit the displacement vector of the feature points to obtain a global motion model (which can be a global homography transformation matrix or an affine transformation model Hglobal). This model represents the global motion trend of the camera (i.e., the robot's active motion trajectory).

[0052] The motion residual vector for each feature point can be calculated as follows:

[0053] in, Indicates the first The motion residual vector of each feature point; Represents the global motion model; Indicates the first frame of the previous image The position coordinates of each feature point.

[0054] In an eye-to-hand architecture, high-frequency mechanical vibrations can cause a rolling shutter effect in the CMOS sensor, resulting in a jelly-like distortion in the image. This distortion manifests as high variance in the residual field. Furthermore, a model of the mechanical vibration energy field is constructed, defining a mechanical vibration energy index for the current moment. This index integrates the non-rigid distortion of the image with abrupt changes in global acceleration, and can be expressed as:

[0055] in, This is the mechanical vibration energy index at the current time t; The variance of the residual field is the weighted variance of all motion residual vectors. The larger this value is, the more severe the non-rigid distortion of the image caused by vibration. n is the total number of sparse corner feature points. This is the global velocity vector. This is the global acceleration, which is obtained by differentiating the global motion velocity vector in the global motion data, and reflects the impact intensity of the robotic arm's sudden stop / start. The weighting coefficient is adjusted according to the camera's shutter speed (the longer the exposure time, the greater the impact of motion blur on the residual, and the weight should be adjusted accordingly).

[0056] S204. Adjust the smoothing factor according to the mechanical vibration energy index, and perform adaptive filtering on the global motion data based on the adjusted smoothing factor to obtain the image stabilization transformation matrix.

[0057] The smoothing factor is a parameter in the exponential moving average filter used to control the weighting of historical data and current observation data. The larger the smoothing factor, the greater the weight of the current observation data; the smaller the smoothing factor, the greater the weight of the historical data.

[0058] Adaptive filtering is a filtering method that dynamically adjusts filter parameters based on the characteristics of the input signal, so that the filter can maintain optimal filtering performance under different operating conditions.

[0059] The image stabilization transformation matrix is ​​a transformation matrix generated by adaptive filtering of global motion data for image geometric correction. It is used to perform inverse geometric transformation on the original image to eliminate high-frequency jitter and distortion.

[0060] For example, in performing this step, the server first acquires the mechanical vibration energy index. The server dynamically adjusts the smoothing factor based on a Boltzmann distribution-based decay function. Specifically, the server multiplies the baseline smoothing factor by an exponential decay term, which uses the ratio of the mechanical vibration energy index to a preset vibration energy threshold as the exponential parameter, scaled by a preset decay coefficient, and then takes a negative exponent. When the mechanical vibration energy index is extremely low, the exponential term approaches 1, and the adjusted smoothing factor is close to the baseline smoothing factor. The system highly trusts the current observation data, ensuring extremely low phase delay. When the mechanical vibration energy index is high, the exponential term rapidly approaches 0, and the adjusted smoothing factor approaches 0. The system automatically freezes the visual input, relying entirely on historical inertial trajectories. The server applies the adjusted smoothing factor to an exponential moving average filter, adaptively filtering the global motion velocity vector in the global motion data to obtain filtered global motion parameters. The server constructs a stable image transformation matrix based on the filtered global motion parameters. This stable image transformation matrix contains the inverse geometric transformation information required to eliminate high-frequency jitter. The server uses the generated stable image transformation matrix as output data for geometric correction of the original image in subsequent steps.

[0061] For example, to achieve the optimal balance between image stabilization and dynamic response delay, a decay function based on the Boltzmann distribution is designed to dynamically adjust the smoothing factor of the exponential moving average filter, which can be expressed as:

[0062] in, This represents the smoothing factor adjusted at the current time t; A baseline smoothing factor (e.g., 0.8) is used for low-latency tracking in a steady state; The preset vibration energy threshold; The empirical attenuation coefficient is a preset positive constant.

[0063] The physical logic of this formula is that when the mechanical vibration energy index is extremely low (uniform and stable motion). The system highly trusts the current observations, ensuring extremely low phase delay, enabling the robot to respond quickly to changes in the target; when high-energy vibrations are detected, i.e. (e.g., during sudden arrest tremor) the exponential term makes The smoothing factor decays rapidly, the system automatically freezes the visual input, and relies entirely on historical inertial trajectories, thus shielding oscillation noise and motion blur at the source.

[0064] Furthermore, the adjusted smoothing factor is applied to the exponential moving average filter to adaptively filter the global motion velocity vector in the global motion data; using the adaptively filtered global motion velocity vector, a stable motion model to describe the global stable motion is fitted. Such as affine transformation models or homography transformation models; furthermore, for stable motion models Perform matrix inversion to generate an inverse geometric transformation matrix, which serves as the image stabilization transformation matrix. Among them, the image stabilization transformation matrix It contains all the translation, rotation, scaling, and cropping parameters required for inverse geometric compensation of the original image, used to map pixels in the original image coordinate system to a stable image coordinate system that eliminates high-frequency jitter.

[0065] S205. Perform a geometric transformation on the current frame image using the image stabilization transformation matrix to obtain a stabilized image.

[0066] Geometric transformation refers to the process of mapping and resampling the pixel positions in an image according to a preset transformation matrix, which is used to correct distortion and jitter in the image.

[0067] A stabilized image is the output image obtained by applying a stabilization transformation matrix to the original image for geometric transformation. In this process, high-frequency jitter and non-rigid distortions are eliminated, and the image remains stable and clear.

[0068] For example, in performing this step, the server first obtains the image stabilization transformation matrix and the current frame image. The server applies the image stabilization transformation matrix to each pixel of the current frame image. Specifically, for each target pixel position in the output image, the server calculates the source pixel coordinates corresponding to that position in the original current frame image through the inverse transformation of the image stabilization transformation matrix. The server performs interpolation processing on the calculated source pixel coordinates, using bilinear or bicubic interpolation methods to calculate the pixel value at that position to generate a smooth and continuous output image. The server iterates through all output pixel positions, completing the inverse geometric transformation of the entire image to generate a stabilized image that eliminates high-frequency jitter and non-rigid distortion. The server uses the stabilized image as output data for target detection processing in subsequent steps.

[0069] For example, the image stabilization transformation matrix It contains the coordinate mapping from the original image (i.e., the current frame image to be stabilized) to the stabilized image. By applying this matrix to each pixel of the original input image, the position of each pixel in the stabilized image can be calculated, thereby generating a new image that eliminates high-frequency jitter and distortion, i.e., the stabilized image. .

[0070] S206. Input the stabilized image into the detection model based on the Transformer architecture, perform cross-attention calculation, and obtain the detection result.

[0071] The detection results include the target bounding box.

[0072] One possible approach involves inputting a stabilized image into a detection model, where a backbone network within the detection model extracts multi-scale features from the stabilized image to obtain a multi-scale feature map. The detection model includes a backbone network, a sinusoidal positional encoding layer, a learnable object query vector, and a cross-attention decoder. The sinusoidal positional encoding layer adds sinusoidal positional encoding to the multi-scale feature map, resulting in a feature sequence carrying spatial location information. The learnable object query vector and the feature sequence are input into the cross-attention decoder, and after multiple layers of cross-attention interaction, a detection result is output. This detection result includes multiple prediction results, each containing a bounding box and a class probability. These multiple prediction results are sparse and unique outputs obtained through one-to-one bipartite graph matching training, eliminating the need for non-maximum suppression post-processing.

[0073] Among them, the Transformer architecture is a neural network architecture based on the self-attention mechanism, which captures long-distance dependencies between features through the global receptive field.

[0074] A detection model refers to a target detection network built on the Transformer architecture, used to identify and locate target objects from an input image.

[0075] Cross-attention computation refers to the process in the Transformer decoder where the learnable object query vector and the image feature sequence interact through an attention mechanism.

[0076] The target bounding box refers to the coordinate information of the target object's position marked by a rectangle in an image. It usually includes the center coordinates, width, and height of the rectangle.

[0077] The backbone network refers to the deep learning network in the detection model used to extract multi-scale features of the image, such as DINOv2 as the backbone network.

[0078] Multi-scale feature maps refer to a set of feature maps with different resolutions extracted from different layers of the backbone network, used to capture detailed and contextual information of the target.

[0079] Sine position coding is a method that encodes position information into a feature sequence in the form of sine and cosine functions to preserve the spatial structure information of features in an image.

[0080] Learnable object query vectors refer to the set of vectors learned through backpropagation during training, used to aggregate target information in feature maps.

[0081] The cross-attention decoder is a decoder module in the Transformer architecture that enables the object query vector to interact with the image feature sequence through a multi-layer cross-attention mechanism.

[0082] Sparse unique output means that the prediction results output by the detection model are sparse in space and each target corresponds to only one prediction result, so there is no need to perform post-processing to remove duplicates through non-maximum suppression.

[0083] For example, the DINOv2 backbone network in the detection model performs multi-scale feature extraction on the stabilized image. It progressively extracts feature maps at different scales through multiple Transformer modules. Shallow feature maps retain high-resolution detail information, while deep feature maps capture low-resolution semantic information, forming a multi-scale feature map set. The server processes the multi-scale feature maps through a sinusoidal positional encoding layer. First, the multi-scale feature maps are flattened into a one-dimensional feature sequence. Then, a sinusoidal positional encoding is added to each position in the sequence, enabling the feature sequence to carry spatial location information. The server introduces a set of learnable object query vectors, which are optimized during training to learn prior knowledge of different target categories. The server inputs the learnable object query vectors and the feature sequence carrying spatial location information into a cross-attention decoder. In each layer of the decoder, the object query vector serves as the query value, and the feature sequence serves as the key and value. The cross-attention mechanism calculates the association weight between the query vector and each position in the feature sequence, and aggregates the feature information based on the weights. After multiple layers of cross-attention interaction, each object query vector gradually focuses on a specific target region in the image. The server directly outputs multiple predictions from the cross-attention decoder, each including the target bounding box and its corresponding class probability. Since the detection model is trained using one-to-one bipartite graph matching, each object query vector is assigned a unique matching ground truth target during training. Therefore, the multiple predictions output by the model are naturally sparse and unique, and can be used directly as detection results without non-maximum suppression post-processing. The server uses the detection results containing the target bounding box as output data for feature extraction and tracking processing in subsequent steps.

[0084] For example, stabilized images The input is a Transformer-based detection model, which includes a DINOv2 backbone network, a sinusoidal positional encoding layer, a learnable object query vector, and a cross-attention decoder. In the backbone network feature extraction stage, DINOv2 can be used as the backbone network to extract multi-scale features from the stabilized image, resulting in multi-scale feature maps. DINOv2 is a large-scale pre-trained visual Transformer based on self-supervised learning. Compared to traditional CNNs, it is more robust to changes in lighting and weakly textured objects, and its patch-based attention mechanism is naturally more noise-resistant than convolutional kernels.

[0085] In the location encoding and feature sequence generation stage, a sinusoidal location encoding layer is used to add sinusoidal location codes to the multi-scale feature map to obtain a feature sequence carrying spatial location information. Sinusoidal location encoding is used to preserve the structural information of image features in space.

[0086] In the cross-attention decoding stage, the learnable object query vector and the feature sequence carrying spatial location information are input into the cross-attention decoder. After multiple layers of cross-attention interaction, the object query vector aggregates target information in the feature sequence, outputting multiple prediction results. Each prediction result includes a bounding box and a class probability.

[0087] In the output phase, since the detection model is trained using one-to-one bipartite graph matching, the multiple predictions output are naturally sparse and unique, eliminating the need for non-maximum suppression post-processing. This effectively avoids the inherent defect of non-maximum suppression algorithms in densely stacked scenarios such as precision industrial workpieces, which erroneously removes adjacent real workpieces, thus maximizing recall.

[0088] S207. Based on the target bounding box in the detection results, extract the target feature vector of the current frame from the feature map of the stabilized image.

[0089] In this context, the feature map refers to the feature representation with channel and spatial dimensions output by the backbone network in the detection model, where each spatial location corresponds to a local region in the image.

[0090] The target feature vector is a vector representation extracted from the region corresponding to the target bounding box in the feature map, used to characterize the appearance and semantic information of the target. It is usually obtained through region of interest alignment operation.

[0091] The target feature vector of the current frame refers to the target feature vector extracted from the feature map of the stabilized image of the current frame, which is used to match and update the historical features in the memory bank.

[0092] For example, the server simultaneously acquires multi-scale feature maps of the stabilized image output by the backbone network in the detection model. These feature maps are intermediate results generated during feature extraction. Based on the coordinates of the target bounding box, the server determines the corresponding region of interest (ROI) on the feature map. Specifically, the server maps the coordinates of the target bounding box from the original image resolution to the feature map resolution, obtaining the target region on the feature map. The server performs ROI alignment, performing bilinear interpolation sampling on the features within the target region to eliminate quantization errors caused by rounding the target bounding box coordinates, resulting in a fixed-size feature map patch. The server performs global average pooling or flattening operations on the feature map patch in the spatial dimension, compressing the two-dimensional spatial features into a one-dimensional feature vector, obtaining the target feature vector for the current frame. This target feature vector for the current frame integrates the appearance and semantic information of the target in the stabilized image, and is used for the dual-gating verification and memory update strategy in subsequent steps.

[0093] For example, using the target bounding box Perform mask slicing on the feature map to extract the target feature vector of the current frame. .

[0094] S208. Perform dual verification on the target feature vector of the current frame through occlusion gating and consistency gating, and execute the corresponding memory update strategy according to the dual verification result to obtain the target tracking result.

[0095] One possible approach is as follows: when the occlusion gating check fails but the consistency gating check passes, a normal update strategy is executed, writing the target feature vector of the current frame into the short-term memory queue to obtain the target tracking result; when both the occlusion gating check and the consistency gating check fail, a soft reset strategy is executed, discarding the short-term memory queue and re-extracting the target feature vector from the feature map of the stabilized image based on the target bounding box in the detection result, using it as the initial feature of the new short-term memory queue to obtain the target tracking result; when the occlusion gating check passes, a memory freeze strategy is executed, prohibiting the writing of the target feature vector of the current frame into the short-term memory queue and not updating the long-term memory; simultaneously, the memory features of the previous frame are reused for position prediction to obtain the target tracking result.

[0096] Among them, passing the occlusion gating check indicates that the target in the current frame is occluded, and passing the consistency gating check indicates that the target features in the current frame are consistent with the target baseline features in the long-term memory.

[0097] Occlusion gating is a logical verification mechanism used to determine whether the target in the current frame is occluded. It determines whether to allow the current frame features to be written into the memory bank by comparing the occlusion score output by the occlusion prediction head with a preset threshold.

[0098] Consistency gating is a logical verification mechanism used to determine whether the target features of the current frame are consistent with the baseline features in the long-term memory. It determines whether the features have drifted by calculating the cosine similarity between the two and comparing it with a preset threshold.

[0099] Dual verification refers to the process of verifying the target feature vector of the current frame by passing through two logical gates: occlusion gating and consistency gating. Only features that pass both gates can be written into the memory bank normally.

[0100] Short-term memory queues are memory containers that use a first-in-first-out (FIFO) structure to store the most recent high-resolution feature maps or feature vectors, and are used to capture the local deformation and instantaneous motion of a target.

[0101] Long-term memory refers to a memory container that stores the initial features of a target when it first appears, as well as the high-confidence keyframe features selected subsequently, serving as the target's identity benchmark.

[0102] The memory freeze strategy is a memory update strategy that is executed when severe occlusion is detected. It prohibits writing the current frame features into the memory bank and forces the reuse of the memory features of the previous frame for position prediction.

[0103] The soft reset strategy refers to the memory update strategy executed when feature drift is detected but no occlusion occurs, which discards the short-term memory queue and reinitializes the features with the current detection results.

[0104] The normal update strategy refers to the memory update strategy executed when occlusion gating is not triggered and consistency gating is passed, which pushes the target feature vector of the current frame into the short-term memory queue to maintain the freshness of the features.

[0105] The target tracking result refers to the target location and identity information determined in the current frame, including the target bounding box coordinates and tracking identifier.

[0106] For example, the server performs dual verification on the current frame target feature vector. For occlusion gating verification, the server inputs the current frame target feature vector into an occlusion prediction head, which is a lightweight multilayer perceptron network, and outputs an occlusion score. The occlusion score, between 0 and 1, represents the probability that the current frame target is occluded. The server compares the occlusion score with a first preset threshold. If the occlusion score is greater than the first preset threshold, the occlusion gating verification is deemed to have passed, indicating that the current frame target is in a severely occluded state, and the current feature vector is highly likely to contain occlusion information. If the occlusion score is not greater than the first preset threshold, the occlusion gating verification is deemed to have failed. For consistency gating verification, the server retrieves the baseline features corresponding to the target from the long-term memory, calculates the cosine similarity between the target feature vector of the current frame and the baseline features, and compares the calculated similarity score with a second preset threshold. If the similarity score is not less than the second preset threshold, the consistency gating verification is deemed to have passed, indicating that the target features of the current frame are consistent with the target baseline features and no feature drift has occurred. If the similarity score is less than the second preset threshold, the consistency gating verification is deemed to have failed, indicating that the target features of the current frame have drifted and may have been falsely detected as background or interference.

[0107] The server executes the corresponding memory update strategy based on the results of the dual verification. When the occlusion gating verification fails but the consistency gating verification passes, it indicates that the target is not occluded and the features are consistent. The server executes the normal update strategy, pushing the target feature vector of the current frame into the short-term memory queue to maintain the freshness of the features, and determining the target position based on the target bounding box in the detection result of the current frame, outputting the target tracking result. When both the occlusion gating verification and the consistency gating verification fail, it indicates that the target is not occluded but the features have drifted, which is judged as a tracking error or a label jump. The server executes the soft reset strategy, discarding all features in the current short-term memory queue, re-extracting the target feature vector from the feature map of the stabilized image based on the target bounding box in the detection result, using the re-extracted feature vector as the initial feature of the new short-term memory queue, and outputting the target tracking result based on the current detection result. When the occlusion gating verification passes, it indicates that the target is severely occluded. The server executes the memory freeze policy, which prohibits writing the target feature vector of the current frame into the memory bank and forces the reuse of the memory features of the previous frame for position prediction. That is, the target position and features of the previous frame are obtained from the short-term memory queue, the target position of the current frame is predicted based on the motion model, and the predicted target tracking result is output.

[0108] Optionally, passing the occlusion gating verification includes: calculating the occlusion score by performing an occlusion score calculation on the target feature vector of the current frame; determining that the occlusion gating verification passes when the occlusion score is greater than a first preset threshold; and determining that the occlusion gating verification fails when the occlusion score is not greater than the first preset threshold.

[0109] For example, the server inputs the current frame target feature vector into an occlusion prediction head, which is a lightweight multilayer perceptron network. After forward propagation, it outputs an occlusion score between 0 and 1, where a higher score indicates a greater probability that the current frame target is occluded. The server obtains a preset first threshold, which can be configured according to the application scenario, for example, set to 0.7. The server compares the calculated occlusion score with the first preset threshold. When the occlusion score is greater than the first preset threshold, the server determines that the current frame target is severely occluded, and the current feature is highly likely to contain information about the occlusion, thus the occlusion gating verification passes. When the occlusion score is not greater than the first preset threshold, the server determines that the current frame target is not severely occluded, and the current feature can be used for memory updates, thus the occlusion gating verification fails.

[0110] Optionally, passing the consistency gating check includes: retrieving the target baseline features from the long-term memory; calculating the similarity between the target feature vector of the current frame and the target baseline features to obtain a similarity score; determining that the consistency gating check passes when the similarity score is not less than a second preset threshold; and determining that the consistency gating check fails when the similarity score is less than the second preset threshold.

[0111] For example, during consistency gating verification, the server first retrieves the baseline features corresponding to the target from the long-term memory. These baseline features store the initial features when the target first appears and the subsequently selected high-confidence keyframe features, serving as the target's identity baseline. The server obtains the target feature vector of the current frame and calculates the similarity between the current frame target feature vector and the target baseline features. Specifically, it uses the cosine similarity calculation method, calculating the cosine value of the angle between the two feature vectors as the similarity score. The closer the similarity score is to 1, the more similar the two features are. The server obtains a preset second threshold, which can be configured according to the application scenario, for example, set to 0.6. The server compares the calculated similarity score with the second preset threshold. When the similarity score is not less than the second preset threshold, the server determines that the current frame target features are consistent with the target baseline features in the long-term memory, and no feature drift has occurred; therefore, the consistency gating verification is deemed successful. When the similarity score is less than the second preset threshold, the server determines that the current frame target features have drifted, and background or interference may have been falsely detected; therefore, the consistency gating verification is deemed unsuccessful.

[0112] It should be noted that when the occlusion gating verification passes, regardless of the consistency gating verification result, the memory freeze strategy is directly executed, prohibiting feature writing and reusing the memory features of the previous frame, avoiding occlusion from polluting the memory library and ensuring tracking continuity.

[0113] Optionally, in terms of memory bank structure design, the short-term memory queue adopts a FIFO (First-In-First-Out) structure to store the most recently accessed data. High-resolution feature maps of frames (e.g., 6 frames) are used to capture the target's local deformation and instantaneous motion. Long-term memory is used to store the initial features when the target first appears and the high-confidence keyframe features selected subsequently, serving as the target's identity benchmark.

[0114] Regarding dual verification, the target feature vector of the current frame is... Before being written to the short-term memory queue, it must be verified by two logic gates: occlusion gating and consistency gating, to prevent contamination of the memory.

[0115] The specific process of occlusion gating verification is as follows: it can be implemented using a lightweight occlusion prediction head, which is a multilayer perceptron network, and outputs an occlusion score. .like (e.g., 0.7) If the target in the current frame is determined to be severely occluded, the current feature is very likely to contain information about the occluder, and the occlusion gating verification passes. If the occlusion score is not greater than the first preset threshold, the target in the current frame is determined not to be severely occluded, and the current feature can be used for memory update, and the occlusion gating verification fails.

[0116] The specific process of consistency gating verification is as follows: retrieve the target baseline features from the long-term memory. Calculate the target baseline features and the target feature vector of the current frame. Spatial consistency (i.e., cosine similarity) between them:

[0117] in, The similarity between the target features in the current frame and the baseline features in long-term memory; It is the L2 norm (Euclidean norm) of a vector, used to calculate the magnitude of a vector and normalize the similarity calculation.

[0118] like (e.g., 0.6), it is determined that the target features of the current frame have drifted, and background or interference may be falsely detected. In this case, the consistency gating check fails; if If the target features in the current frame are consistent with the target baseline features in the long-term memory and no feature drift has occurred, then the consistency gating check passes.

[0119] Regarding the memory update strategy, the corresponding memory update strategy is executed based on the occlusion gating verification result and the consistency gating verification result. When the occlusion gating verification fails but the consistency gating verification passes, it means that the target is not occluded and the features are consistent. The normal update strategy is then executed, writing the target feature vector of the current frame into the short-term memory queue to maintain the freshness of the features. The target position is then determined based on the target bounding box in the detection result of the current frame, and the target tracking result is obtained.

[0120] When both the occlusion gating check and the consistency gating check fail, it indicates that the target is not occluded but the features have drifted. This is judged as a tracking error or a label jump. A soft reset strategy is executed, the short-term memory queue is discarded, and the target feature vector is re-extracted from the feature map of the stabilized image based on the target bounding box in the detection result. This re-extracted feature vector is used as the initial feature of the new short-term memory queue to obtain the target tracking result.

[0121] When the occlusion gating check passes, it indicates that the target is severely occluded. A memory freeze strategy is then implemented, prohibiting the writing of the current frame's target feature vector to the short-term memory queue and preventing updates to the long-term memory. Simultaneously, the memory features from the previous frame are reused for position prediction to obtain the target tracking result. This strategy prevents the writing of occlusion features into memory, thus enabling immediate target retrieval after the occlusion ends.

[0122] Furthermore, the target tracking results are transformed to obtain the current 3D pose in the industrial robot base coordinate system; the total end-to-end delay is determined by the image acquisition time, processing completion time, and average communication time; a target motion state vector including position, velocity, and acceleration is constructed based on the current 3D pose; the target motion state vector is filtered using a Kalman filter to obtain the estimated velocity and estimated acceleration; based on the kinematic model, the predicted pose is calculated according to the current 3D pose, estimated velocity, estimated acceleration, and total end-to-end delay; the predicted pose is sent to the industrial robot controller so that the industrial robot controller can generate drive commands based on the predicted pose.

[0123] Coordinate system transformation refers to the process of converting the coordinates of the target in the image pixel coordinate system into the three-dimensional pose in the industrial robot base coordinate system, and the mapping between different coordinate systems is achieved through the hand-eye calibration matrix.

[0124] Three-dimensional pose refers to the position and orientation information of a target in three-dimensional space, including the translation along the three coordinate axes and the rotation angle around the three coordinate axes.

[0125] Total end-to-end latency refers to the total time delay between the moment of image acquisition and the moment of robot execution, including image acquisition time, processing time, communication time, and servo motor response lag time.

[0126] The target motion state vector is a vector that describes the current motion state of the target, containing information in three dimensions: position, velocity, and acceleration.

[0127] The Kalman filter is a recursive state estimation algorithm that estimates the true state of a system by filtering noisy observation data.

[0128] Estimated velocity refers to the target velocity estimate obtained by filtering the target motion state vector using a Kalman filter.

[0129] Estimated acceleration refers to the target acceleration estimate obtained by filtering the target's motion state vector using a Kalman filter.

[0130] A kinematic model is a mathematical model that describes the laws of motion of an object in space. This application uses a quadratic kinematic model that includes position, velocity, and acceleration.

[0131] Predicted pose refers to the target pose of the robot at the moment of action execution, which is calculated by a kinematic model based on the target pose at the current moment, the estimated velocity, the estimated acceleration, and the total latency of the entire link.

[0132] For example, in performing this step, the server first acquires the target tracking result, which includes the centroid coordinates and principal axis angles of the target in the image pixel coordinate system. The server calculates the centroid and principal axis angles in the pixel coordinate system based on the target segmentation mask, and combines this with a pre-calibrated hand-eye calibration matrix to convert the pixel coordinates into a 3D pose in the robot base coordinate system, obtaining the current 3D pose. The server records the timestamps of each stage of the system in real time, acquiring the image acquisition time, the image processing completion time, and the average communication time. The server acquires the response hysteresis constant of the robot servo motors, which is obtained through system identification. The server calculates the total delay from observation to execution, adding the difference between the image acquisition time and the processing completion time, the average communication time, and the servo motor response hysteresis constant to determine the total end-to-end delay. The server constructs a target motion state vector based on the current 3D pose, which includes the target's current position, velocity, and acceleration information. The server inputs the target motion state vector into a Kalman filter. The Kalman filter recursively filters the state vector through two steps: prediction and update. In the prediction step, the state at the next moment is predicted based on the motion model. In the update step, the prediction result is corrected by incorporating observations. Finally, the filtered estimated velocity and estimated acceleration are output. The server calculates the predicted pose based on the kinematic model. Specifically, based on the current 3D pose, estimated velocity, estimated acceleration, and total end-to-end delay, the target pose at the moment of robot action execution is calculated using a kinematics formula containing second-order terms. That is, the current pose is added to the product of velocity and total delay, plus the product of half the acceleration and the square of the total delay, to obtain the predicted pose. This predicted pose maps the control target from the past moment to the future moment, effectively offsetting the lag caused by the total end-to-end delay. The server sends the calculated predicted pose to the robot controller. Based on the received predicted pose, the robot controller uses an inverse kinematics algorithm to generate feedforward velocity commands to drive the joint motors, controlling the robot's end effector to move to the predicted pose to perform grasping or assembly operations, achieving zero-lag precise tracking of high-speed moving targets.

[0133] For example, visual perception results are converted into robot control commands, and lag is eliminated in the time domain. Regarding coordinate system transformation, the target tracking results are transformed to obtain the current 3D pose in the robot's base coordinate system. Specifically, the centroid and principal axis angles in the pixel coordinate system are calculated based on the target segmentation mask, and then converted into a 3D pose in the robot's base coordinate system using the hand-eye calibration matrix.

[0134] In terms of end-to-end latency monitoring, the total end-to-end latency is determined by the image acquisition time, processing completion time, and average communication time. The system records the timestamp of each link in real time, including the image acquisition time, processing completion time, and average communication time, and obtains the response hysteresis constant of the robot's servo motor. The total end-to-end latency is determined by adding the difference between the image acquisition time and the processing completion time, the average communication time, and the servo motor response hysteresis constant.

[0135] In Kalman state observation, a target motion state vector, including position, velocity, and acceleration, is constructed based on the current 3D pose. A Kalman filter is then used to filter this vector to obtain estimated velocity and estimated acceleration. The Kalman filter recursively filters the state vector through two steps: prediction and update. In the prediction step, the state at the next moment is predicted based on the motion model. In the update step, the prediction result is corrected by incorporating observations, ultimately outputting estimated velocity and estimated acceleration.

[0136] In terms of predictive feedforward compensation, based on the kinematic model, the predicted pose is calculated according to the current 3D pose, estimated velocity, estimated acceleration, and total end-to-end delay. Specifically, the target pose at the moment of robot action execution is calculated using a kinematic formula containing second-order terms, which is the current pose plus the product of the estimated velocity and the total end-to-end delay, plus half of the estimated acceleration plus the square of the total end-to-end delay, to obtain the predicted pose. This step maps the control target from the past moment to the future moment, effectively offsetting the lag caused by the total end-to-end delay.

[0137] In terms of instruction generation, the predicted pose is sent to the robot controller, enabling the robot controller to generate drive instructions based on the predicted pose. Specifically, based on the received predicted pose, the robot controller uses an inverse kinematics algorithm to generate feedforward velocity instructions to drive the joint motors, controlling the robot's end effector to move to the predicted pose to perform grasping or assembly operations, achieving zero-lag, precise tracking of high-speed moving targets.

[0138] To enable the four modules of physical image stabilization, target detection, memory segmentation, and predictive control to work together, this application designs a joint training loss that incorporates factors such as image stabilization quality, detection accuracy, memory robustness, and control accuracy into the optimization objective.

[0139] The image stabilization reconstruction loss is used to constrain the difference between the stabilized image and the vibration-free reference image, employing a combination of pixel-level L1 loss and perceptual loss. The pixel-level L1 loss calculates the average of the sum of the absolute differences of each pixel between the stabilized image and the vibration-free reference image. The perceptual loss is introduced into a pre-trained visual network to calculate the differences between the stabilized image and the vibration-free reference image in the intermediate feature layers of the pre-trained visual network. By setting balanced weights, both pixel-level accuracy and visual perceptual consistency can be achieved.

[0140] The matching loss employs bipartite graph matching loss to supervise the end-to-end prediction of the Transformer detector, ensuring the accuracy of bounding box localization. This loss includes bounding box regression loss and class cross-entropy loss. The bounding box regression loss can use GIoU loss, while the class cross-entropy loss is used to constrain the accuracy of class prediction.

[0141] The memory consistency loss is used to supervise the effectiveness of the dual-gating mechanism, ensuring the purity and stability of the memory. This loss includes occlusion gating loss and consistency loss. The occlusion gating loss uses cross-entropy loss to supervise the classification accuracy of the occluded prediction head, where the occluded ground truth label is 0 or 1. The consistency loss is used to constrain the similarity between the current feature and the long-term benchmark feature when there is no occlusion.

[0142] The control compensation loss is used to monitor the accuracy of predictive feedforward control, constrain the difference between the predicted pose and the actual target pose, and calculate the mean square error between the predicted pose and the target's true pose at the time of command execution.

[0143] The total loss is calculated using a weighted summation method, combining the image stabilization reconstruction loss, detection matching loss, memory consistency loss, and control compensation loss. By adjusting the weights of each component, a suitable trade-off can be found among multiple objectives such as image stabilization quality, detection accuracy, memory robustness, and control accuracy, enabling the entire process to be optimized collaboratively rather than operating in isolation.

[0144] Optionally, in order to convert the visual perception results into robot control commands and eliminate lag in the time domain, a coordinate system transformation is performed, that is, the centroid in the pixel coordinate system is calculated based on the target segmentation mask. and spindle angle Combined with hand-eye calibration matrix The target information in the pixel coordinate system is converted into a three-dimensional pose in the robot base coordinate system. .

[0145] Furthermore, Kalman state observation is performed to construct a target motion state vector containing position, velocity, and acceleration. ,in, It is the matrix / vector transpose operator. A Kalman filter is used to smooth observation noise, outputting the estimated velocity of the target. and estimated acceleration .in, The target's current 3D pose, i.e., the 3D pose in the robot's base coordinate system, includes position components. ) and attitude components ( , which corresponds to the position state in the formula; The first derivative of the pose vector with respect to time is the velocity vector of the target at the current moment, which is used to characterize the instantaneous rate of change of the target's pose and includes linear velocity and angular velocity components. It is the second derivative of the pose vector with respect to time, i.e., the acceleration vector of the target at the current moment, used to characterize the instantaneous rate of change of the target's velocity, including linear acceleration and angular acceleration components.

[0146] Furthermore, end-to-end latency monitoring is performed, with the system recording the timestamps of each step in real time, including the image acquisition time. Processing completion time and average communication time Based on the above time information, the total latency from image perception to robot execution is calculated using the following formula:

[0147] in, Total end-to-end latency; This is the response hysteresis constant of the robot's servo motor, which is usually identified by the system.

[0148] Perform predictive feedforward compensation operations, and calculate the robot's action execution time based on the kinematic model (i.e., The corresponding target predicted pose The calculation formula is as follows:

[0149] in, It is the current moment when the processing is completed and the instruction is issued.

[0150] This step maps the control objective from historical observation moments to future execution moments, offsetting... The resulting delay.

[0151] Furthermore, the instruction generation operation is performed to predict the pose. The command is sent to the robot controller, which generates feedforward speed commands to drive the joint motors based on the predicted pose, enabling zero-lag dynamic grasping of the moving target.

[0152] To enable the four modules of physical image stabilization, target detection, memory tracking, and predictive control to work together, a joint training loss is constructed. The image stabilization quality, detection accuracy, memory robustness, and control accuracy are uniformly incorporated into the optimization objective to achieve end-to-end optimization of the overall system.

[0153] Image Stabilization Reconstruction Loss To constrain the difference between the stabilized image and the vibration-free reference image, a weighted fusion of pixel-level L1 loss and perceptual loss is used, and the calculation formula is as follows:

[0154] in, This is a balancing weighting coefficient for pixel-level L1 loss, used to adjust the proportion of pixel-level precision in the total loss; It is a pixel-level L1 loss used to constrain the direct difference between the stabilized image and the reference image in the pixel space; The balancing weight coefficient for perceptual loss is used to adjust the proportion of perceptual feature consistency in the total loss. The perceptual loss is used to constrain the consistency between the stabilized image and the reference image in the deep visual feature space.

[0155] The formula for calculating pixel-level L1 loss is:

[0156] in, Image height, The image width is used to average the total pixel error and eliminate the influence of image size on the loss value. The pixel value at pixel coordinates (i,j) of the stabilized image represents the output image pixel value after image stabilization. is the pixel value of the vibration-free reference image at pixel coordinates (i,j), representing the pixel of the ideal, jitter-free baseline image.

[0157] Perceptual loss is introduced into pre-trained visual networks The formula for extracting deep feature differences is as follows:

[0158] Where C is the number of feature map channels, H′ is the feature map height, and W′ is the feature map width, which are used to average the total feature error; This is a feature extraction function for a pre-trained visual network, used to extract deep visual features from the input image. It typically uses intermediate layer features from pre-trained models such as ResNet. The input data for the stabilized image at feature channel c and feature coordinates (i,j) is used to extract deep features of the stabilized image. The input data for the vibration-free reference image at feature channel c and feature coordinates (i,j) is used to extract the deep reference features of the reference image.

[0159] The detection matching loss employs bipartite graph matching loss supervision based on the Transformer architecture detection model to achieve end-to-end prediction optimization, ensuring the accuracy of target bounding box localization. The calculation formula is as follows:

[0160] in, To detect matching loss, it is used to supervise the end-to-end prediction of the Transformer detector and constrain the target bounding box localization accuracy and class classification accuracy. This is the bounding box regression loss function, used to calculate the deviation between the predicted bounding box and the true bounding box. Common loss functions in this field, such as GIoU loss, can be used. The predicted bounding box for the j-th target represents the target's location and size as output by the detector. The true bounding box of the j-th target represents the true location and size of the labeled target, serving as the benchmark for regression supervision; This is the category cross-entropy loss function, used to calculate the classification bias between the predicted category probability and the true category label; Let be the predicted class probability of the j-th target, and represent the target class confidence score output by the detector. The true category label for the j-th target represents the true category of the labeled target and serves as the benchmark for classification supervision.

[0161] The memory consistency loss is used to supervise the effectiveness of occlusion gating and consistency gating, ensuring the purity and stability of the features in the memory bank. The calculation formula is as follows:

[0162] in, This is used to assess memory consistency loss and monitor the effectiveness of the dual-gating mechanism, ensuring the purity and stability of memory features. The balancing weighting coefficient for occlusion gating loss is used to adjust the proportion of occlusion classification accuracy in the total loss; To mitigate the occlusion gating loss, this method is used to monitor the classification accuracy of the occluded prediction head and filter out invalid occluded features. This is the balancing weight coefficient for consistency loss, used to adjust the proportion of feature similarity constraints in the total loss; This is a consistency loss used to constrain the similarity between the current feature and the long-term benchmark feature in unoccluded scenarios.

[0163] Obstruction of gate control loss The cross-entropy loss is used to supervise the output accuracy of the occlusion prediction head. The calculation formula is as follows:

[0164] in, To obscure the real label, the value is 0 (no obscuring) or 1 (obscuring), which serves as the benchmark for classification supervision; is the natural logarithm function, used to calculate the log-likelihood of the predicted probability; The occlusion score is the output of the occlusion prediction head, with a value range of [0,1], representing the predicted probability that the target is occluded. For unobstructed realism tags, and Complementary, used to supervise the classification accuracy in unobstructed scenes; For unobstructed prediction scores, and Complementary means that the predicted probability of the target being unobstructed is represented.

[0165] Consistency loss The formula used to constrain the similarity between current frame features and long-term memory baseline features in unoccluded scenes is as follows:

[0166] in, It is a function for maximizing the value; The similarity threshold is a preset baseline similarity lower limit used to constrain the minimum requirement for feature similarity. The similarity between the target features in the current frame and the baseline features in long-term memory is usually measured using common metrics in the field, such as cosine similarity.

[0167] The control compensation loss is used to monitor the accuracy of the predictive feedforward control and constrain the deviation between the predicted pose and the true pose. The calculation formula is as follows:

[0168] in, To control compensation losses, and to monitor the accuracy of predictive feedforward control, the deviation between the predicted pose and the actual pose at the time of instruction execution is quantified. The total number of training samples; The target pose predicted by the system in the m-th training sample is the target pose predicted by the system, which is the target pose of the feedforward control. The true pose of the target at the moment of instruction execution in the m-th training sample is used as the baseline ground truth for control accuracy supervision.

[0169] Finally, a weighted fusion joint loss is used to train the entire system end-to-end. The total loss is calculated using the following formula:

[0170] in, The total joint loss of the system is used for end-to-end training of the robot vision control system, and to comprehensively optimize the performance of the four major modules. The balancing weight coefficient for image stabilization reconstruction loss is used to adjust the proportion of the visual image stabilization module in the overall optimization objective; The balancing weight coefficients for detecting matching loss are used to adjust the proportion of the target detection module in the overall optimization objective; This is a balancing weighting coefficient for memory consistency loss, used to adjust the proportion of the memory tracking module in the overall optimization objective; The balancing weighting coefficients for compensating for losses are used to adjust the proportion of the predictive control module in the overall optimization objective. By adjusting the weighting coefficients, an optimal trade-off can be achieved between image stabilization quality, detection accuracy, memory robustness, and control accuracy, enabling coordinated optimization across the entire process.

[0171] Based on the above, the industrial robot visual tracking method achieves excellent results in multiple dimensions through the organic integration of physical image stabilization, Transformer detection, dual-gated memory update, and predictive feedforward control. In terms of visual image stabilization, a mechanical vibration energy field model is constructed, and vibration energy is quantified in real time by combining optical flow residual variance and global acceleration. A Boltzmann distribution-based attenuation function is used to dynamically adjust the smoothing factor, achieving an adaptive optimal balance between stabilization and dynamic response. During high-energy vibrations, the smoothing factor automatically approaches zero, freezing the visual input-dependent historical trajectory and eliminating input-end feature noise at its source. During stable motion, the smoothing factor recovers to the baseline value, allowing the system to quickly respond to target changes. Compared to traditional mean filtering or electronic image stabilization methods, this scheme eliminates the need for multi-frame accumulation or complex feature matching, avoiding phase delay issues and enabling the visual servo system to maintain stable convergence under millisecond-level response requirements.

[0172] In object detection, an end-to-end detection model based on the Transformer architecture is adopted, utilizing a global attention mechanism to replace the local convolutions of traditional CNNs, thereby improving the detection capability of objects in vibration-blurred images. Through learnable object query vectors and one-to-one bipartite graph matching training, the model output is naturally sparse and unique, eliminating the need for non-maximum suppression post-processing and fundamentally avoiding the inherent defect of traditional detectors in densely stacked scenes that incorrectly suppress adjacent real objects. In industrial scenarios with closely arranged precision workpieces, the false negative rate of stacked objects is reduced, the bounding box localization error is less than 1 pixel, and the end-to-end output ensures deterministic inference latency.

[0173] In target tracking, a streaming memory segmentation network with a dual-gating mechanism is introduced, employing occlusion gating and consistency gating as two logical valves to intelligently manage feature updates. When occlusion is detected, the system automatically executes a memory freeze strategy, prohibiting the writing of occluded features into the memory bank to prevent memory contamination. When feature drift is detected, the system executes a soft reset strategy, discarding contaminated short-term memories and re-initializing them. Compared to the passive weighted updates of existing algorithms such as DeepSORT, this application endows the system with the ability to maintain the target identifier unchanged for a long time even under large-area occlusion, solving the problems of frequent identifier switching and tracking chain breakage in strong occlusion environments.

[0174] Regarding dynamic tracking accuracy, a predictive feedforward control model incorporating end-to-end delay monitoring was constructed. This model measures the total delay from image acquisition to industrial robot execution in real time and calculates the target's predicted pose at the execution moment based on a kinematic model. By mapping the controlled target from the past to the future, the hysteresis error caused by the inherent system delay is effectively reduced. In high-speed operation scenarios with conveyor belt speeds reaching 1 meter per second, the absolute positioning error of dynamic grasping is reduced, achieving zero-hysteresis accurate interception and grasping of high-speed moving targets.

[0175] In terms of overall system performance, a closed-loop system covering all time and space from perception to control is formed through the collaborative design of four modules: physical image stabilization, target detection, memory tracking, and predictive control. The joint training loss function incorporates image stabilization quality, detection accuracy, memory robustness, and control accuracy into a unified optimization objective, ensuring that each module promotes the others rather than operating independently. This method outperforms traditional schemes in four dimensions: vibration resistance, occlusion resistance, dynamic accuracy, and dense detection. Detection frame center bounce is reduced, and the target maintains a very high capture success rate even with high occlusion ratios. This strongly supports applications in high-end industrial manufacturing fields, such as precision component assembly and precision electronic manufacturing, which have stringent requirements for accuracy and robustness. Ultimately, accurate tracking is achieved.

[0176] The above text combined Figures 1 to 2 The industrial robot visual tracking method provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0177] This application also provides an industrial robot vision tracking device, such as... Figure 3 As shown in the figure, this is a schematic diagram of an industrial robot vision tracking device provided in an embodiment of this application. The device includes: The acquisition module 301 is used to acquire a continuous image stream captured by the end-effector camera of the industrial robot, as well as global motion data of the end-effector of the industrial robot; wherein, the continuous image stream includes the current frame image and the previous frame image; The calculation module 302 is used to perform optical flow tracing on the current frame image and the previous frame image to obtain the feature point displacement vector of the current frame image relative to the previous frame image; calculate the mechanical vibration energy index based on the feature point displacement vector; adjust the smoothing factor based on the mechanical vibration energy index, and perform adaptive filtering on the global motion data based on the adjusted smoothing factor to obtain the image stabilization transformation matrix; perform geometric transformation on the current frame image using the image stabilization transformation matrix to obtain the image stabilization; input the image stabilization image into the detection model based on the Transformer architecture, perform cross-attention calculation, and obtain the detection result; wherein, the detection result includes the target bounding box; extract the target feature vector of the current frame from the feature map of the image stabilization based on the target bounding box in the detection result; The tracking module 303 is used to perform dual verification on the target feature vector of the current frame through occlusion gating and consistency gating, and execute the corresponding memory update strategy according to the dual verification result to obtain the target tracking result.

[0178] In some possible implementations, the tracing module 303 is specifically used for: When the occlusion gating check fails but the consistency gating check passes, a normal update strategy is executed, writing the target feature vector of the current frame into the short-term memory queue to obtain the target tracking result. When both the occlusion gating check and the consistency gating check fail, a soft reset strategy is executed, discarding the short-term memory queue and re-extracting the target feature vector from the feature map of the stabilized image based on the target bounding box in the detection result, using it as the initial feature of the new short-term memory queue to obtain the target tracking result. When the occlusion gating check passes, a memory freeze strategy is executed, prohibiting the writing of the target feature vector of the current frame into the short-term memory queue and not updating the long-term memory. At the same time, the memory features of the previous frame are reused for position prediction to obtain the target tracking result. Here, passing the occlusion gating check indicates that the target in the current frame is occluded, and passing the consistency gating check indicates that the target features of the current frame are consistent with the target baseline features in the long-term memory.

[0179] In some possible implementations, the tracing module 303 is specifically used for: The occlusion score is calculated from the target feature vector of the current frame. When the occlusion score is greater than the first preset threshold, the occlusion gating verification is deemed to have passed. When the occlusion score is not greater than the first preset threshold, the occlusion gating verification is deemed to have failed.

[0180] In some possible implementations, the tracing module 303 is specifically used for: Retrieve target baseline features from long-term memory; The similarity score is obtained by calculating the similarity between the target feature vector of the current frame and the target baseline features; When the similarity score is not less than the second preset threshold, the consistency gating check is deemed to have passed. When the similarity score is less than the second preset threshold, the consistency gating check is deemed to have failed.

[0181] In some possible implementations, the computing module 302 is specifically used for: The global motion model is obtained by fitting the displacement vectors of feature points using the random sampling consensus algorithm; Based on the global motion model and the displacement vector of each feature point, the motion residual vector of each feature point is calculated; the weighted variance of all motion residual vectors is calculated to obtain the residual field variance; the residual field variance is used to characterize the degree of non-rigid distortion of the image caused by mechanical vibration; the global motion velocity vector in the global motion data is differentiated to obtain the global acceleration; the global acceleration is used to characterize the impact intensity of the robotic arm; the global acceleration is multiplied by the weighting coefficients to obtain the weighted global acceleration; the weighting coefficients are determined according to the shutter exposure time; the sum of the residual field variance and the weighted global acceleration is determined as the mechanical vibration energy index.

[0182] In some possible implementations, the computing module 302 is specifically used for: A stabilized image is input into a detection model. The backbone network of the detection model extracts multi-scale features from the stabilized image to obtain a multi-scale feature map. The detection model includes a backbone network, a sinusoidal position encoding layer, a learnable object query vector, and a cross-attention decoder. The sinusoidal position encoding layer adds sinusoidal position encoding to the multi-scale feature map to obtain a feature sequence carrying spatial location information. The learnable object query vector and the feature sequence are input into the cross-attention decoder. After multiple layers of cross-attention interaction, the detection result is output. The detection result includes multiple prediction results, each of which includes a bounding box and a class probability. The multiple prediction results are sparse and unique outputs obtained through one-to-one bipartite graph matching training, without the need for non-maximum suppression post-processing.

[0183] Among some possible implementations, industrial robot vision tracking devices also include: The prediction module performs coordinate transformation on the target tracking results to obtain the current 3D pose in the industrial robot base coordinate system; determines the total end-to-end latency by using the image acquisition time, processing completion time, and average communication time; constructs a target motion state vector including position, velocity, and acceleration based on the current 3D pose; filters the target motion state vector using a Kalman filter to obtain estimated velocity and estimated acceleration; calculates the predicted pose based on the kinematic model, according to the current 3D pose, estimated velocity, estimated acceleration, and total end-to-end latency; and sends the predicted pose to the industrial robot controller so that the industrial robot controller can generate drive commands based on the predicted pose.

[0184] The industrial robot vision tracking device according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the industrial robot vision tracking device are respectively for implementing Figure 2 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0185] This application also provides a computing device. For example... Figure 4 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 400 includes a bus 401, a processor 402, a communication interface 403, and a memory 404. The processor 402, the memory 404, and the communication interface 403 communicate with each other via the bus 401.

[0186] Bus 401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0187] Processor 402 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0188] Communication interface 403 is used for communication with external devices.

[0189] Memory 404 may include volatile memory, such as random access memory (RAM). Memory 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0190] The memory 404 stores executable code, and the processor 402 executes the executable code to perform the aforementioned industrial robot vision tracking method.

[0191] Specifically, in achieving Figure 3 In the case of the illustrated embodiment, and Figure 3 When the modules or units of the industrial robot vision tracking device described in the embodiments are implemented through software, the following steps are performed: Figure 3 The software or program code required for the functions of each module / unit can be partially or entirely stored in memory 404. Processor 402 executes the program code corresponding to each unit stored in memory 404 to execute the aforementioned industrial robot vision tracking method.

[0192] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned industrial robot vision tracking method.

[0193] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0194] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0195] When the computer program product is executed by a computer, the computer performs any of the aforementioned industrial robot vision tracking methods. The computer program product can be a software installation package; when any of the aforementioned industrial robot vision tracking methods needs to be used, the computer program product can be downloaded and executed on the computer.

[0196] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0197] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A visual tracking method for industrial robots, characterized in that, The method includes: Acquire a continuous image stream captured by the end-effector camera of an industrial robot, as well as global motion data of the end-effector of the industrial robot; wherein the continuous image stream includes the current frame image and the previous frame image; Optical flow tracing is performed on the current frame image and the previous frame image to obtain the feature point displacement vector of the current frame image relative to the previous frame image; Calculate the mechanical vibration energy index based on the displacement vector of the characteristic point; The smoothing factor is adjusted according to the mechanical vibration energy index, and the global motion data is adaptively filtered based on the adjusted smoothing factor to obtain the image stabilization transformation matrix. A stable image is obtained by performing a geometric transformation on the current frame image using the image stabilization transformation matrix. The stabilized image is input into a detection model based on the Transformer architecture, and cross-attention calculation is performed to obtain the detection result; wherein, the detection result includes the target bounding box; Based on the target bounding box in the detection result, extract the target feature vector of the current frame from the feature map of the stabilized image; The target feature vector of the current frame is double-checked by occlusion gating and consistency gating. The corresponding memory update strategy is executed according to the double-check result to obtain the target tracking result.

2. The method according to claim 1, characterized in that, The process involves performing dual verification on the target feature vector of the current frame using occlusion gating and consistency gating, and executing the corresponding memory update strategy based on the dual verification result to obtain the target tracking result, including: When the occlusion gating check fails but the consistency gating check passes, the normal update strategy is executed, and the target feature vector of the current frame is written into the short-term memory queue to obtain the target tracking result. When the occlusion gating check fails and the consistency gating check fails, a soft reset strategy is executed, the short-term memory queue is discarded, and the target feature vector is re-extracted from the feature map of the stabilized image based on the target bounding box in the detection result, as the initial feature of the new short-term memory queue, to obtain the target tracking result; When the occlusion gating verification passes, the memory freeze strategy is executed, which prohibits the writing of the target feature vector of the current frame into the short-term memory queue and does not update the long-term memory; at the same time, the memory features of the previous frame are reused to perform position prediction and obtain the target tracking result. Wherein, passing the occlusion gating check indicates that the target in the current frame is occluded, and passing the consistency gating check indicates that the target features in the current frame are consistent with the target reference features in the long-term memory.

3. The method according to claim 2, characterized in that, The obstruction gating verification passed, including: The occlusion score is calculated by performing an occlusion score on the current frame target feature vector; When the occlusion score is greater than the first preset threshold, the occlusion gating verification is deemed to have passed. When the occlusion score is not greater than the first preset threshold, the occlusion gating verification is determined to fail.

4. The method according to claim 2, characterized in that, The consistency gating verification passed, including: Retrieve target baseline features from long-term memory; The similarity score is obtained by calculating the similarity between the current frame target feature vector and the target baseline feature. When the similarity score is not less than the second preset threshold, the consistency gating check is deemed to have passed. When the similarity score is less than the second preset threshold, the consistency gating check is deemed to have failed.

5. The method according to claim 1, characterized in that, The step of calculating the mechanical vibration energy index based on the displacement vector of the feature point includes: The displacement vectors of the feature points are fitted using a random sampling consensus algorithm to obtain a global motion model; Based on the global motion model and the displacement vector of each feature point, calculate the motion residual vector of each feature point; The weighted variance of all motion residual vectors is calculated to obtain the residual field variance; wherein, the residual field variance is used to characterize the degree of non-rigid distortion of the image caused by mechanical vibration; The global motion velocity vector in the global motion data is differentiated to obtain the global acceleration; wherein, the global acceleration is used to characterize the impact strength of the robotic arm. The global acceleration is multiplied by a weighting coefficient to obtain the weighted global acceleration; wherein the weighting coefficient is determined based on the shutter exposure time. The sum of the residual field variance and the weighted global acceleration is determined as the mechanical vibration energy index.

6. The method according to claim 1, characterized in that, The step of inputting the stabilized image into a detection model based on the Transformer architecture, performing cross-attention calculation, and obtaining the detection result includes: The stabilized image is input into the detection model, and multi-scale feature extraction is performed on the stabilized image through the backbone network in the detection model to obtain a multi-scale feature map; wherein, the detection model includes a backbone network, a sinusoidal position coding layer, a learnable object query vector, and a cross-attention decoder; The sinusoidal position coding layer adds sinusoidal position coding to the multi-scale feature map to obtain a feature sequence carrying spatial position information; The learnable object query vector and the feature sequence are input into the cross-attention decoder. After multiple layers of cross-attention interaction, the detection result is output. The detection result includes multiple prediction results, each of which includes a bounding box and a class probability. The multiple prediction results are sparse and unique outputs obtained by training one-to-one bipartite graph matching, without the need for non-maximum suppression post-processing.

7. The method according to claim 1, characterized in that, The method further includes: The target tracking results are transformed to obtain the current three-dimensional pose in the industrial robot base coordinate system. The total latency of the entire link is determined by the image acquisition time, processing completion time, and average communication time. Construct a target motion state vector including position, velocity, and acceleration based on the current three-dimensional pose. The target motion state vector is filtered using a Kalman filter to obtain the estimated velocity and estimated acceleration; Based on the kinematic model, the predicted pose is calculated according to the current three-dimensional pose, the estimated velocity, the estimated acceleration, and the total end-to-end delay; The predicted pose is sent to the industrial robot controller so that the industrial robot controller can generate drive commands based on the predicted pose.

8. A visual tracking device for industrial robots, characterized in that, The device includes: The acquisition module is used to acquire a continuous image stream captured by the end-effector camera of the industrial robot, as well as global motion data of the end-effector of the industrial robot; wherein, the continuous image stream includes the current frame image and the previous frame image; The calculation module is used to perform optical flow tracing on the current frame image and the previous frame image to obtain the feature point displacement vector of the current frame image relative to the previous frame image; calculate the mechanical vibration energy index based on the feature point displacement vector; adjust the smoothing factor based on the mechanical vibration energy index, and perform adaptive filtering on the global motion data based on the adjusted smoothing factor to obtain a stable image transformation matrix; perform a geometric transformation on the current frame image using the stable image transformation matrix to obtain a stable image; input the stable image into a detection model based on the Transformer architecture to perform cross-attention calculation to obtain a detection result; wherein, the detection result includes a target bounding box; and extract the target feature vector of the current frame from the feature map of the stable image based on the target bounding box in the detection result. The tracking module is used to perform dual verification on the target feature vector of the current frame through occlusion gating and consistency gating, and execute the corresponding memory update strategy according to the dual verification result to obtain the target tracking result.

9. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 7.

Images