Time sequence comparison visual servo method based on pre-training visual model

Abstract

The invention provides a time sequence comparison visual servo method based on a pre-training visual model, and relates to the technical field of visual servo, and the method comprises the steps: obtaining a current image at a current moment and an expected target image, and obtaining an image sequence at a historical moment from an image cache region; inputting the current image, the target image and the historical image sequence into a pre-trained visual basic model for feature coding to obtain a corresponding image coding vector; performing time sequence comparison calculation on the image coding vector by using a plurality of continuous self-attention layers so as to capture dynamic association information in the image sequence; carrying out attention aggregation on the image coding vectors subjected to time sequence comparison calculation to obtain aggregated coding vectors; and on the basis of the aggregated coding vector, predicting the six-axis speed of the camera at the current moment, including a direction vector and a modulus length, and multiplying the normalized direction vector and the modulus length to synthesize a final speed command for controlling the camera or the robot to complete a visual servo task.

Description

Technical Field

[0001] This invention relates to the field of visual servoing technology, and specifically to a temporal contrastive visual servoing method based on a pre-trained visual model. Background Technology

[0002] Visual servoing is defined as a method of controlling a robot using visual features extracted from images. As a robot control technology, its error function is defined as the error between the visual features at the robot's or camera's current position and the desired position. The goal of visual control is to adjust this error and eventually bring it to zero. The integration of visual sensors enables robot systems to operate outside of unstructured industrial environments, and this technology has shown great potential in a wide range of applications, including industrial automation, visual navigation, and drones.

[0003] With the development of technologies such as deep learning and computer hardware, visual servoing technology is undergoing a process of development from traditional methods to the integration of cutting-edge technologies such as deep learning in the past two decades, which have seen a surge in hardware computing power.

[0004] Traditional methods typically extract the correspondence between key feature points in the current image and the desired image, and derive speed control directly based on key point errors (image-based visual servoing, IBVS) or estimated relative pose transformations (position-based visual servoing, PBVS). These methods achieve high servoing accuracy, but suffer from a series of problems such as small convergence regions and inaccurate IBVS or object model and camera intrinsic parameters (PBVS). These limitations often hinder performance in complex and ever-changing real-world scenarios.

[0005] With the development of deep learning, data-driven image servoing has become a new research hotspot. Numerous studies have already constructed deep neural networks to automatically learn rich and effective feature representations and strategies from large datasets, significantly improving the performance and adaptability of visual servoing systems in specific environments.

[0006] However, most current visual servoing models based on deep neural networks are limited by the amount of data they contain, resulting in relatively simple network designs and an inability to learn broader, more general strategies. Fortunately, the rapid development of computer vision has brought numerous opportunities, leading to the emergence of many general-purpose vision models in recent years. These models can be pre-trained on large-scale image datasets and require only fine-tuning or even direct deployment to complete downstream tasks such as feature extraction, depth estimation, and point tracking. Combined with subsequent network design, these networks can be effectively transferred to visual servoing tasks.

[0007] Therefore, this design proposes an innovative framework that collects a large amount of synthetic data through a high-fidelity simulator and leverages a pre-trained visual model and a deep neural network contrast module to achieve powerful scene generalization capabilities. The method utilizes the prior knowledge of large-scale pre-training and diverse synthetic data to address the problem of existing visual servoing models being overly specialized in specific scenes. Summary of the Invention

[0008] This invention presents a temporal contrastive visual servoing method based on a pre-trained visual model. By constructing a large synthetic dataset and incorporating prior knowledge from a pre-trained visual network model and the design of a downstream temporal contrastive network, it overcomes the shortcomings of traditional methods, such as limited scenario applicability and weak versatility. The specific technical solution is as follows: A temporal contrastive visual servoing method based on a pre-trained visual model includes the following steps: S1. Obtain the current image and the desired target image at the current moment, and obtain the historical image sequence from the image buffer, wherein the image buffer stores historical images from the past H moments; S2. Parallelize the current image, target image, and historical image sequence and input them into the pre-trained visual base model for feature encoding to obtain the corresponding image encoding vector; S3. The encoding vectors corresponding to the current image, target image, and historical image sequence are concatenated into a temporal feature sequence, which is then input into a temporal comparison network composed of multiple continuous self-attention layers for calculation. For the encoding vector at any time in the temporal feature sequence, the self-attention mechanism is used to compare it across time with the encoding vectors at all other times to calculate the dependency relationship. S4. Perform attention aggregation on the encoding vector obtained by time-comparison calculation to obtain the aggregated encoding vector; S5. Based on the aggregated encoding vector, predict the 6-axis velocity of the camera at the current moment, realize the prior transformation and transfer of the large-scale visual model based on the temporal contrast network design, form end-to-end control from image input to control output, and control the camera or robot to complete the visual servoing task.

[0009] Furthermore, prior to step S1, a dataset generation step is also included: Based on a high-fidelity simulator, import industrial-grade robot models and object models to build a basic visual servoing environment; Triple randomization configuration is performed on the servo target, servo initial pose, and servo environment; The simulation process is encapsulated through a GPU-accelerated parallel computing framework, enabling synchronized data acquisition across multiple scenarios.

[0010] Furthermore, the initial pose of the servo is randomized, including: A hemispherical shell is constructed with the center of the target object group as the center; random sampling points are taken on the hemispherical shell as the position of the camera; the orientation of the camera is calculated with the center of the sphere as the focus, thereby determining the initial pose of the servo.

[0011] Further, in step S3, the temporal comparison calculation is used, specifically including: inputting the image encoding vector into multiple consecutive self-attention layers; in the self-attention layer, for any part of the encoding vector, attention calculation is performed based on the query vector, key vector and value vector to capture the dependency relationship between image features at different times.

[0012] Furthermore, all the above images are input in parallel into a pre-trained visual base model for feature encoding to obtain the encoding vector corresponding to each image, and the image encodings at different times are concatenated to construct a temporal feature sequence; For any given servo process, obtain the current image at each time step. and target image At any moment A buffer of length H can be cached. Zhang's historical observation images It is input into the pre-trained visual base model along with the current target image pair. Encode to obtain Output encoding of the image at each time step : ; The image encoding outputs at each time step are spliced ​​together to introduce temporal modeling.

[0013] Furthermore, the encoded sequence is input into a temporal contrast network consisting of multiple consecutive self-attention layers. Through a query-key-value attention mechanism, the contrast dependency between the image and the target image features at different times is calculated to realize a temporal-based feature contrast mode. Output encoding for each image use Comparative calculations were performed on consecutive self-attention layers, targeting The i-th part : ; Then, combine the softmax calculation to obtain Q for the i-th part. i K i V i Vectors are derived from attention calculation results : ; in, This represents the dimension of the encoded vector.

[0014] Furthermore, after computation using a time-series self-attention contrastive network, the final result is... Perform attention aggregation to generate aggregated encoding vectors. : ; in, yes The i-th part The weighted score.

[0015] Furthermore, linear layer projection is used to predict the aggregated encoded vectors. Direction of target camera speed at all times and mold length Finally, the direction vector is normalized and multiplied with the magnitude to synthesize the final target camera velocity. : .

[0016] To address the shortcomings of existing technologies, this invention aims to solve the following core technical problems: 1. This invention proposes a high-fidelity dataset generation framework based on a simulator. It utilizes the open-source YCB dataset object model and industrial robot configuration model to build a basic environment, achieving triple randomization of the servo target, pose, and environment. Furthermore, the computation process is encapsulated as a GPU-accelerated parallel workflow, overcoming the bottleneck of low efficiency in traditional data acquisition and enabling the rapid generation of large-scale, high-quality servo datasets, thus solving the core problem of insufficient training data in visual servoing tasks.

[0017] 2. This invention designs an end-to-end visual servo control network that integrates a pre-trained visual model. It designs a temporal processing module with a historical image buffer and performs temporal comparison calculations on the image encoding at each time step through multiple sets of continuous self-attention layers to capture dynamic correlation information in the image sequence and enhance the model's perception of pose change trends during servoing. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a schematic diagram of random sampling of the initial servo pose on a hemispherical shell in an embodiment of the present invention; Figure 2This is a schematic diagram of a performance comparison experiment conducted in a simulation environment in an embodiment of the present invention; wherein, (a) is the image at the initial moment, (b) is the image at time step T=10, (c) is the image at time step T=30, (d) is the image at time step T=50, (e) is the image at the final moment, and (f) is the target image. Figure 3 This is a schematic diagram illustrating the distinction between simple and difficult scenarios in a real-world environment in an embodiment of the present invention, wherein (a) is a difficult scenario and (b) is a simple scenario. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] Example 1 This invention presents a temporal contrastive visual servoing method based on a pre-trained visual model. By constructing a large synthetic dataset and incorporating prior knowledge from a pre-trained visual network model and the design of a downstream temporal contrastive network, it overcomes the shortcomings of traditional methods, such as limited scenario applicability and weak versatility. Specifically: First, high-quality servo datasets are generated using a high-fidelity simulator and its parallel framework, overcoming the problem of scarce datasets in conventional work. By introducing robot and object models, multimodal information, including RGB, depth, and object segmentation, is obtained after randomizing servo configurations in a single scene. Parallel encapsulation enables efficient parallel data acquisition across multiple scenes, including: [1] Using the NVIDIA IsaacSim simulator, the model from the open-source YCB dataset (from YCBBenchmarks–ObjectandModelSet|Benchmarkingforroboticmanipulation) was imported into the simulation environment along with the industrial robot configuration model to build the basic visual servo environment.

[0022] [2] Randomization of servo scene configuration, including servo target randomization, servo pose randomization and servo environment randomization.

[0023] [3] Among them, the servo pose sampling process is as follows: Figure 1 As shown, a radius of 1 is constructed around the center of the object group. A hemispherical shell, in which The servo depth is also randomly selected and randomized by adjusting the angle between the radius and the z-axis on the spherical shell. and the angle between the projection of the target point onto the xy plane and the x-axis. Randomly select target point location The camera pose is calculated with the center of the sphere as the camera center and the direction of alignment.

[0024] [4] By leveraging NVIDIA’s open-source IsaacLab, the simulation environment and computation process are encapsulated into a highly parallel workflow to fully utilize GPU acceleration capabilities and improve servo data acquisition efficiency.

[0025] Second, a pre-trained visual model DINOv2 (an open-source general-purpose visual foundation model from the field of computer vision, used for general feature extraction) is introduced. Combined with a temporal contrastive network design, the prior knowledge of large-scale visual models is transferred to visual servoing tasks to predict the output 6-axis camera speed. ,in, For three-axis velocities, Using three-axis angular velocities, an end-to-end vision servoing network is formed. Specifically: [1] Taking any servo process as an example, the current image can be obtained at each moment. and target image At any moment A buffer of length H can be cached. open( Historical observation images , the current image Target image and historical observation images Composition of image set Input pre-trained visual base model Encode to obtain Output encoding of the image at each time step : ; The image encoding outputs at each time step are spliced ​​together to introduce temporal modeling.

[0026] [2] Output encoding for each image use Comparative calculations were performed on consecutive self-attention layers to... The i-th part For example, firstly, the query vector Q, key vector K, and value vector V are calculated using linear layer projection: ; Then, combine the softmax calculation to obtain Q for the i-th part. i K i V i Vectors are derived from attention calculation results ; in, This represents the dimension of the key vector K.

[0027] [3] After computation by a time-series self-attention contrastive network, the final result is obtained. Attention aggregation: ; in, yes The i-th part The weight scores are derived by the network's own gradient optimization.

[0028] [4] Predict the aggregated low-dimensional coding vectors using linear layer projection. Direction of target camera speed at all times and mold length Finally, the direction vector is normalized and multiplied with the magnitude to synthesize the final target camera velocity. : ;

[0029] Specifically, the overall process is divided into two phases: the training phase and the inference phase.

[0030] First, a large amount of high-quality dataset is collected through a simulator to train the end-to-end servo network. Then, by leveraging the open weights of the pre-trained visual base model, which contains large-scale and highly diverse prior data, it is transferred to the visual servoing task using the dataset proposed in this design.

[0031] During the deployment phase, regardless of whether it is a simulation environment or a real environment, the camera samples each scene and inputs them in chronological order. At each control step, the current image and target image pair are input, and the model outputs the required 6-axis velocity of the camera target for the next control step in real time to complete the servo task.

[0032] Example 2 The servo algorithm of this invention can be applied to precise pose control scenarios involving robots and cameras. Through the refinement of deep point clouds and real-time pose adjustment, it solves the problem of high-precision tracking and operation when the target pose changes dynamically in fields such as industrial operation, aircraft and mobile robots.

[0033] In specific embodiments, in industrial machinery operation scenarios, the present invention can be applied to tasks such as robotic arm assembly and material sorting: On the assembly line, when a part is displaced due to the vibration of the conveyor belt, the camera collects the depth information of the part, extracts key pose features through algorithms, and the servo system drives the end effector of the robotic arm to correct the position in real time, such as adjusting the translation error to within 3cm and the rotation error to within 3°, to ensure that the bolts and screw holes are accurately aligned and the gripper stably grasps irregular parts.

[0034] Meanwhile, this invention can be used in mobile robots. For scenarios such as unmanned aerial vehicle cabin inspection and autonomous docking of mobile robots, the algorithm can assist the aircraft in perceiving the position and posture of target equipment (such as pipe interfaces and charging piles) through the airborne camera. The aircraft, mobile robot and other vehicles can use the underlying servo system to dynamically adjust the camera speed according to the algorithm output to ensure stable hovering within the target range, while maintaining continuous alignment of the camera with the key detection area.

[0035] This invention enhances the generalization and versatility of visual servoing models by leveraging a large prior and theoretical simulation dataset generation process based on pre-trained visual models, thereby obtaining an infinite amount of data. By designing a temporal contrastive attention network, the prior of the visual model is transformed into the visual servoing task, resulting in performance superior to previous methods.

[0036] Example 3 The demonstration is divided into two parts: a simulated environment and a real-world environment. In both cases, the servo task is performed in a scene with randomly sampled initial and target poses using a zero-shot method. The pose acquisition method is as follows: Figure 1 As shown, a hemispherical shell is constructed with the center of the object group as the center. Points are randomly selected on the shell as positions, and the pose is calculated with the center of the shell as the camera center and the aiming direction.

[0037] Images captured by the camera are used as network input to directly predict the speed of the robot's end effector. Evaluation criteria include success rate (SR), translational error (TE), rotational error (RE), and time step (TS). This design scheme is referred to as TCT-VS (Time Contrast Transformer Visual Serving).

[0038] In the simulation environment, as mentioned above, such as Figure 2The image shows the servo process of this method from the initial time to the final time. Images at time steps T=10, 30, and 50, and the final time, are shown, along with the corresponding target image for reference. (a) is the image at the initial time, (b) is the image at time step T=10, (c) is the image at time step T=30, (d) is the image at time step T=50, (e) is the image at the final time, and (f) is the target image. Comparative analysis with the CNS and IBVS methods is performed in this environment. The servo task is considered complete when the final rotation angle is less than 3° and the translation accuracy is controlled within 3 cm. The comparative experimental evaluation results in the simulation environment are shown in Table 1.

[0039] Table 1. Evaluation results of comparative experiments in the simulation environment. As can be seen from the results, the model of this invention integrates a large amount of prior knowledge from the pre-trained visual model, successfully avoids the limitations of scene changes, and achieves the current state-of-the-art performance. The strong model prior enables it to achieve better results in terms of translation error and convergence speed.

[0040] In real-world environments, comparisons and evaluations were conducted with CNS and IBVS methods. Scenes were divided into two difficulty levels: easy and difficult. In easy scenes, multiple objects were introduced to assess image correlation, reducing the difficulty of model comparison and simplifying the servo control process. In the more challenging difficult scenes, the number of objects was reduced, increasing the difficulty of the servo task. Figure 3 As shown in Table 2, (a) represents a difficult scenario and (b) represents a simple scenario. Fifty initial poses and target poses were randomly sampled within each level. The evaluation results for the real-world environment are shown in Table 2.

[0041] Table 2. Assessment Results of the Real-World Environment The results show that the method of this invention achieves superior performance, improving the success rate by approximately 50% compared to the traditional approach (IBVS), especially in challenging scenarios. Leveraging extensive prior knowledge from visual models, the model of this invention possesses inherent robustness and generalization ability for unknown scenarios, and improves the convergence speed of servo tasks—a finding consistent with simulation results. Simultaneously, the model maintains high accuracy and enhances robustness to varying degrees of external noise, achieving smoother motion trajectories.

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1.A time-series contrastive visual servoing method based on a pre-trained visual model, characterized in that, The method comprises the following steps: S1, obtaining a current image at a current time and a target image, and obtaining a historical image sequence from an image cache area, wherein the image cache area stores historical images at H past times; S2, parallelizing the current image, the target image and the historical image sequence and inputting them into a pre-trained visual basic model for feature encoding to obtain corresponding image encoding vectors; S3, splicing the encoding vectors corresponding to the current image, the target image and the historical image sequence into a time sequence feature sequence, inputting the time sequence feature sequence into a time sequence comparison network composed of multiple continuous self-attention layers for calculation, and comparing any time point encoding vector in the time sequence feature sequence with all other time point encoding vectors through a self-attention mechanism to calculate a dependency relationship; S4, performing attention aggregation on the encoding vectors obtained through the time point comparison calculation to obtain aggregated encoding vectors; S5, predicting the 6-axis speed of the camera at the current time based on the aggregated encoding vectors, realizing large-scale visual model prior transformation based on the time sequence comparison network design, forming an end-to-end control from image input to control output, and controlling the camera or robot to complete the visual servo task. 2.The pre-trained visual model based time-sequential contrastive visual servoing method of claim 1, wherein, Before step S1, a data set generation step is further included: Based on a high-fidelity simulator, import industrial robot models and object models to construct a visual servoing basic environment; Triply randomize the servo target, the servo initial pose and the servo environment; Encapsulate the simulation process through a GPU-accelerated parallel computing framework to realize multi-scene synchronized data acquisition. 3.The pre-trained visual model based time-sequential contrastive visual servoing method of claim 2, wherein, Randomly configuring the servo initial pose includes: Taking the center of the target object group as the center of a spherical shell, randomly sampling points on the spherical shell as the positions of the cameras, and calculating the orientations of the cameras with the center as the focal point to determine the servo initial pose. 4.The pre-trained visual model based temporal contrastive visual servoing method of claim 1, wherein, In step S3, the time sequence comparison calculation includes: inputting the image encoding vectors into multiple continuous self-attention layers; in the self-attention layer, for any part of the encoding vector, attention calculation is performed based on the query vector, the key vector and the value vector to capture the dependency relationship between the features of images at different times. 5.The pre-trained visual model based temporal contrastive visual servoing method of claim 1, wherein, All the images are parallelly input into the pre-trained visual basic model for feature encoding to obtain the encoding vectors corresponding to the images, and the image encodings at different times are spliced to construct a time sequence feature sequence; For any servo process, the current image at each time is obtained and the target image At time , the cache area with length H can cache historical observation images The current image , the target image and the historical observation images constitute an image set Input the pre-trained visual base model for encoding to obtain the output encoding of the image corresponding to time : ； The image encoding outputs at different times are spliced to introduce time sequence modeling. 6.The pre-trained visual model based temporal contrastive visual servoing method of claim 5, wherein, The encoding sequence is input into a time sequence comparison network composed of multiple continuous self-attention layers, the comparison dependency relationship between the features of images at different times and the target image is calculated through the query-key-value attention mechanism, and a time sequence-based feature comparison mode is realized; Encoding the output for each image Using Contrastive computation using The i-th part in the middle The query vector Q, the key vector K and the value vector V are calculated using a linear layer ； Q computed for the i-th part in conjunction with softmax computation i , K i , V i vector taking the self-attention computation result : ； wherein, denotes the dimension of the encoding vector. 7.The pre-trained visual model based temporal contrastive visual servoing method of claim 6, wherein, After the passage of the self-attention contrastive network calculation, the final aggregation encoding vector is generated by performing attention aggregation on the obtained : ； wherein is the weight fraction of the i-th portion in the i-th portion. 8.The pre-trained visual model based temporal contrastive visual servoing method of claim 7, wherein, Linear layer projection is used to predict the aggregated encoded vectors The direction of the target camera velocity at the moment And the module length Finally, the direction vector is normalized and multiplied by the module length to synthesize the final target camera velocity : 。

Images