A time-series contrastive visual servoing method based on a pre-trained visual model

Abstract

The application provides a time sequence comparison visual servo method based on a pre-trained visual model, relates to the technical field of visual servo, acquires a current image and a desired target image at a current moment, and acquires an image sequence at a historical moment from an image cache area; the current image, the target image and the historical image sequence are input into a pre-trained visual basic model for feature coding to obtain corresponding image coding vectors; a plurality of continuous self-attention layers are used to perform time sequence comparison calculation on the image coding vectors to capture dynamic correlation information in the image sequence; the image coding vectors subjected to the time sequence comparison calculation are subjected to attention aggregation to obtain aggregated coding vectors; based on the aggregated coding vectors, the 6-axis speed of the camera at the current moment is predicted, including a direction vector and a module length, and a final speed command is synthesized by multiplying the normalized direction vector by the module length, which is used for controlling the camera or the robot to complete a visual servo task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual servoing, and particularly relates to a time series contrast visual servoing method based on a pre-trained visual model. BACKGROUND

[0002] Visual servoing is defined as a method of using visual features extracted from images to control robots. As a robot control technology, its error function is defined as the error of visual features at the current position and the desired position of the robot or camera, and the purpose of visual control is to adjust this error and eventually make it zero. The integration of visual sensors enables robot systems to work outside unstructured industrial environments, and this technology has shown great potential in widely used fields such as industrial automation, visual navigation, and unmanned aerial vehicles.

[0003] With the development of deep learning, computer hardware and other technologies, in the past two decades of hardware computing power flying, visual servoing technology is experiencing a process of developing from traditional methods to integrating cutting-edge technologies such as deep learning.

[0004] Traditional methods generally extract the correspondence relationship between key feature points between the current image and the desired image, and directly derive the velocity control according to the key point error (image-based visual servoing, IBVS) or estimate the relative pose transformation (position-based visual servoing, PBVS). These methods achieve high servoing accuracy, but have a series of problems such as small convergence domain and error correspondence (IBVS) or inaccurate object model and camera internal parameters (PBVS), which often limit the performance in the face of complex and variable actual scenes.

[0005] With the development of deep learning, data-driven image servoing has become a new research hotspot. A large number of studies have constructed deep neural networks to automatically learn rich and effective feature representations and strategies from large data sets, significantly improving the performance and adaptability of visual servoing systems in specific environments.

[0006] However, most of the current deep neural network-based visual servoing models are limited by the data size, and the network design is relatively simple, and the model cannot learn more extensive and general strategies. Fortunately, the rapid development of computer vision has brought a lot of opportunities, and in recent years, a large number of general visual models have appeared, which are pre-trained on large-scale image datasets, and only need to be fine-tuned or directly deployed to complete feature extraction, depth estimation, point tracking and other downstream tasks. Combined with subsequent network design, such networks can be effectively transferred to visual servoing tasks.

[0007] Therefore, the design proposes an innovative framework to realize strong scene generalization capability through a high-fidelity simulator to collect a large amount of synthetic data, and a pre-trained visual model and a deep neural network contrast module design. SUMMARY

[0008] The application is based on a pre-trained visual model time series contrast visual servoing method, which breaks through the defects of single scene and weak generality of traditional methods by constructing a large synthetic data set, introducing pre-trained visual network model prior and downstream time series contrast network design. The specific technical solutions are as follows:

[0009] A pre-trained visual model time series contrast visual servoing method, comprising the following steps:

[0010] S1, obtaining a current image and a desired target image at a current time, and obtaining a historical image sequence from an image cache area, wherein the image cache area stores historical images at H past times;

[0011] S2, parallelizing the current image, target image and historical image sequence to input a pre-trained visual basic model for feature encoding to obtain corresponding image encoding vectors;

[0012] S3, the current image, target image, historical image sequence corresponding encoding vector is spliced into time series feature sequence, input into time series contrast network composed of multi-layer continuous self-attention layer for calculation, any time encoding vector in time series feature sequence is compared with all other time encoding vectors through self-attention mechanism to calculate the dependency relationship;

[0013] S4, attention aggregation is performed on the encoding vectors obtained through time series contrast calculation to obtain aggregated encoding vectors;

[0014] S5, based on the aggregated encoding vectors, predicting the 6-axis speed of the camera at the current time, realizing large-scale visual model prior transfer based on time series contrast network design, forming an end-to-end control from image input to control output, and controlling the camera or robot to complete the visual servoing task.

[0015] Further, before step S1, the data set generation step is further included:

[0016] Based on a high-fidelity simulator, import industrial robot models and object models to construct a visual servoing basic environment;

[0017] Triply randomize the servo target, servo initial pose and servo environment;

[0018] The simulation process is encapsulated by a parallel computing framework accelerated by a GPU to realize synchronized data acquisition in multiple scenes.

[0019] Further, the servo initial pose is randomly configured, including:

[0020] A half-sphere shell is constructed with the center of the target object group as the sphere center, and a random sampling point on the half-sphere shell is used as the position of the camera; the orientation of the camera is calculated with the sphere center as the focal point, thereby determining the servo initial pose.

[0021] Further, in step S3, a time sequence comparison calculation is used, specifically including: inputting the image encoding vector into a plurality of continuous self-attention layers; in the self-attention layer, for any part of the encoding vector, attention calculation is performed based on the query vector, the key vector and the value vector to capture the dependency relationship between the image features at different times.

[0022] Further, all the images are input in parallel into a pre-trained visual base model for feature encoding to obtain the encoding vectors corresponding to each image, and the image encodings at different times are spliced to construct a time sequence feature sequence;

[0023] For any servo process, the current image and the target image are obtained at each time , and the buffer area with a length of H can buffer historical observation images , which are input in parallel with the current target image into a pre-trained visual base model for encoding to obtain the output encoding of the image corresponding to the time :

[0024] ;

[0025] The image encoding outputs at each time are spliced to introduce time sequence modeling.

[0026] Further, the encoding sequence is input into a time sequence comparison network composed of a plurality of continuous self-attention layers, and the comparison dependency relationship between the images at different times and the target image features is calculated through a query-key-value attention mechanism to realize a feature comparison mode based on time sequence;

[0027] The image output encoding is calculated using continuous self-attention layers, and for the i-th part in :

[0028] ;

[0029] Q calculated for the i-th part i , K i , V i vector to obtain the self-attention calculation result :

[0030] ;

[0031] wherein, represents the dimension of the encoding vector.

[0032] Further, after the time-series self-attention contrast network calculation, the final aggregated encoding vector is obtained by attention aggregation to generate the aggregated encoding vector :

[0033] ;

[0034] wherein, is the weight score of the i-th part in the aggregated encoding vector.

[0035] Further, the aggregated encoding vector is projected using a linear layer to predict the direction and the length of the target camera speed at the current moment, respectively, and finally the direction vector is normalized and multiplied by the length to synthesize the final target camera speed :

[0036] .

[0037] In view of the defects of the prior art, the present application needs to solve the following core technical problems:

[0038] 1. The present application proposes a high-fidelity dataset generation framework based on a simulator, which builds a basic environment with the help of an open-source YCB dataset object model and an industrial robot configuration model, realizes triple randomization of servo targets, poses, and environments, and encapsulates the calculation process as a GPU-accelerated parallel workflow, breaking through the bottleneck of low efficiency of traditional data acquisition, realizing fast generation of large-scale high-quality servo datasets, and solving the core problem of lack of training data in visual servoing tasks.

[0039] 2. The present application designs an end-to-end visual servoing control network that integrates a pre-trained visual model, designs a time-series processing module containing a history image buffer, performs time-series contrast calculation on image encoding at each moment through multiple groups of continuous self-attention layers, captures dynamic correlation information in the image sequence, and strengthens the model's perception of the pose change trend in the servo process. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 A schematic diagram of random sampling of the initial pose of the servo in the hemispherical shell in the embodiments of the present application;

[0042] Figure 2 A scene schematic diagram for performance comparison experiment in the simulation environment in the embodiments of the present application; wherein, (a) is an initial time image, (b) is a T=10 time step image, (c) is a T=30 time step image, (d) is a T=50 time step image, (e) is a final time image, and (f) is a target image;

[0043] Figure 3 A schematic diagram of simple scene and difficult scene distinguished in the real world environment in the embodiments of the present application, wherein (a) is a difficult scene, and (b) is a simple scene. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0045] Embodiment 1

[0046] The time series contrast visual servo method based on the pre-trained visual model of the present application breaks through the defects of single scene and weak generality of traditional methods by constructing a large synthetic data set, introducing pre-trained visual network model prior and downstream time series contrast network design. The specific implementation is as follows:

[0047] First, high-quality servo data set generation is realized by means of high-fidelity simulator and its parallel framework, which breaks the problem of lack of conventional working data set. By introducing robot and object model, multi-modal information including RGB, depth, object segmentation, etc. is obtained after randomizing the servo configuration in a single scene, and multi-scene efficient data acquisition is realized through parallel encapsulation, including:

[0048] [1] Using the NVIDIA IsaacSim simulator, the model from the open-source YCB dataset (from YCBBenchmarks–ObjectandModelSet|Benchmarkingforroboticmanipulation) was imported into the simulation environment along with the industrial robot configuration model to build the basic visual servo environment.

[0049] [2] Randomization of servo scene configuration, including servo target randomization, servo pose randomization and servo environment randomization.

[0050] [3] Among them, the servo pose sampling process is as follows: Figure 1 As shown, a radius of 1 is constructed around the center of the object group. A hemispherical shell, in which The servo depth is also randomly selected and randomized by adjusting the angle between the radius and the z-axis on the spherical shell. and the angle between the projection of the target point onto the xy plane and the x-axis. Randomly select target point location The camera pose is calculated with the center of the sphere as the camera center and the alignment direction.

[0051] [4] By leveraging NVIDIA’s open-source IsaacLab, the simulation environment and computation process are encapsulated into a highly parallel workflow to fully utilize GPU acceleration capabilities and improve servo data acquisition efficiency.

[0052] Second, a pre-trained visual model DINOv2 (an open-source general-purpose visual foundation model from the field of computer vision, used for general feature extraction) is introduced. Combined with a temporal contrastive network design, the prior knowledge of large-scale visual models is transferred to visual servoing tasks to predict the output 6-axis camera speed. ,in, For three-axis velocities, Using three-axis angular velocities, an end-to-end vision servoing network is formed. Specifically:

[0053] [1] Taking any servo process as an example, the current image can be obtained at each moment. and target image At any moment A buffer of length H can be cached. open( Historical observation images , the current image Target image and historical observation images Composition of image set Input pre-trained visual base model Encode to obtain Output encoding of the image at each time step :

[0054] ;

[0055] The image encoding outputs at each time step are spliced ​​together to introduce temporal modeling.

[0056] [2] Output encoding for each image use Comparative calculations were performed on consecutive self-attention layers to... The i-th part For example, firstly, the query vector Q, key vector K, and value vector V are calculated using linear layer projection:

[0057] ;

[0058] Then, combine the softmax calculation to obtain Q for the i-th part. i K i V i Vectors are derived from attention calculation results

[0059] ;

[0060] in, This represents the dimension of the key vector K.

[0061] [3] After computation by a time-series self-attention contrastive network, the final result is obtained. Attention aggregation:

[0062] ;

[0063] in, yes The i-th part The weight scores are derived by the network's own gradient optimization.

[0064] [4] Predict the aggregated low-dimensional coding vectors using linear layer projection. Direction of target camera speed at all times and mold length Finally, the direction vector is normalized and multiplied with the magnitude to synthesize the final target camera velocity. :

[0065] ;

[0066] Specifically, the overall process is divided into two phases: the training phase and the inference phase.

[0067] First, a large number of high-quality data sets are collected through the simulator to train the end-to-end servo network, and the open weights of the pre-trained visual base model are used, which contains large-scale high-diversity data priori, which is migrated to the visual servo task through the data set proposed in the design.

[0068] In the deployment stage, whether in a simulation environment or a real environment, the camera observations are sampled for each scene and input in chronological order, and the current and target image pairs are input every control step, and the model outputs the required camera target 6-axis speed for the next control step in real time to complete the servo task.

[0069] Embodiment 2

[0070] The servo algorithm of the application can be applied to precise pose control scenes of robots and cameras, and through the refinement of depth point clouds and real-time pose adjustment, the high-precision tracking and operation problems in the fields of industrial operation, aircraft and mobile robots, etc. are solved when the target pose changes dynamically.

[0071] In specific embodiments, in the industrial mechanical operation scene, the application can be applied to tasks such as mechanical arm assembly and material sorting: when the parts are offset in pose due to the vibration of the conveyor belt on the assembly line, the camera collects the depth information of the parts, extracts the key pose features through the algorithm, and the servo system drives the end effector of the mechanical arm to correct the position in real time, such as adjusting the translation error to within 3 cm and the rotation error to within 3°, to ensure accurate alignment of the bolt and the screw hole and stable grasping of the special-shaped parts by the gripper.

[0072] At the same time, the application can be used for mobile robots, and for scenes such as unmanned aerial vehicle cabin inspection and mobile robot autonomous docking, the algorithm can assist the aircraft to perceive the pose of the target device (such as a pipeline interface or a charging pile) through the on-board camera, and the aircraft and mobile robot can dynamically adjust the camera speed according to the algorithm output by the underlying servo system, to ensure stable hovering within the target range while maintaining the continuous alignment of the camera to the key detection area.

[0073] The application generates a simulation data set through a large number of priori and theoretical pre-trained visual models, enhances the generalization and universality of the visual servo model, and converts the priori of the visual model to the visual servo task through the design of the timing comparison attention network, which has better performance than previous methods.

[0074] Embodiment 3

[0075] The effect display is divided into two parts: simulation environment and real world environment. In both cases, the servo task is performed in a zero-sample manner in a scene with randomly sampled initial pose and target pose, as shown in Figure 1 Fig. 1, a semispherical shell is constructed with the center of the object group, and points are randomly selected on the shell as positions, and the pose is calculated with the camera center as the center of the sphere.

[0076] Images captured by the camera are used as network input to directly predict the speed of the robot's end effector. Evaluation criteria include success rate (SR), translational error (TE), rotational error (RE), and time step (TS). This design scheme is referred to as TCT-VS (Time Contrast Transformer Visual Serving).

[0077] In the simulation environment, as mentioned above, such as Figure 2 The image shows the servo process of this method from the initial time to the final time. Images at time steps T=10, 30, and 50, and the final time, are shown, along with the corresponding target image for reference. (a) is the image at the initial time, (b) is the image at time step T=10, (c) is the image at time step T=30, (d) is the image at time step T=50, (e) is the image at the final time, and (f) is the target image. Comparative analysis with the CNS and IBVS methods is performed in this environment. The servo task is considered complete when the final rotation angle is less than 3° and the translation accuracy is controlled within 3 cm. The comparative experimental evaluation results in the simulation environment are shown in Table 1.

[0078] Table 1. Evaluation results of comparative experiments in the simulation environment.

[0079]

[0080] As can be seen from the results, the model of this invention integrates a large amount of prior knowledge from the pre-trained visual model, successfully avoids the limitations of scene changes, and achieves the current state-of-the-art performance. The strong model prior enables it to achieve better results in terms of translation error and convergence speed.

[0081] In real-world environments, comparisons and evaluations were conducted with CNS and IBVS methods. Scenes were divided into two difficulty levels: easy and difficult. In easy scenes, multiple objects were introduced to assess image correlation, reducing the difficulty of model comparison and simplifying the servo control process. In the more challenging difficult scenes, the number of objects was reduced, increasing the difficulty of the servo task. Figure 3 As shown in Table 2, (a) represents a difficult scenario and (b) represents a simple scenario. Fifty initial poses and target poses were randomly sampled within each level. The evaluation results for the real-world environment are shown in Table 2.

[0082] Table 2. Assessment Results of the Real-World Environment

[0083]

[0084] The results show that the method has better performance, and the success rate is improved by about 50% compared with the traditional scheme (IBVS), especially in difficult scenes. The model has innate robustness and generalization ability to unknown scenes with the help of a large number of priors of the visual model, and improves the convergence speed of the servo task - this finding is consistent with the simulation results. At the same time, the model maintains high precision and enhances the robustness to different degrees of external noise, achieving smoother motion trajectories.

[0085] The above examples are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1.A time-series contrastive visual servoing method based on a pre-trained visual model, characterized in that, The method comprises the following steps: S1, obtaining a current image at a current time and a target image, and obtaining a historical image sequence from an image cache area, wherein the image cache area stores historical images at H past times; S2, parallelizing the current image, the target image and the historical image sequence to input a pre-trained visual base model for feature encoding to obtain corresponding image encoding vectors; S3, splicing the image encoding vectors corresponding to the current image, the target image and the historical image sequence into a time sequence feature sequence, inputting the time sequence contrast network composed of multiple continuous self-attention layers for calculation, and performing attention calculation on the encoding vectors at any time in the time sequence feature sequence based on a query vector, a key vector and a value vector, and capturing the dependency between image features at different times through the self-attention mechanism and cross-time comparison with all encoding vectors at other times; For any servo process, at each time, obtain the current image and the target image , at time , the cache area with length H can cache historical observation images , the current image , the target image and the historical observation images constitute an image set input the pre-trained visual base model for encoding, and obtain the output encoding of the image corresponding to time : ； The image encoding outputs at each time are spliced to introduce time sequence modeling; The encoding sequence is input into the time sequence contrast network composed of multiple continuous self-attention layers, the comparison dependency between different time images and the target image features is calculated through the query-key-value attention mechanism, and the time sequence-based feature comparison mode is realized; Encoding the output for each image Using Contrastive computation using The i-th part in the middle The query vector Q, the key vector K, and the value vector V are calculated using a linear layer projection Linear: ； Q computed for the i-th part in conjunction with softmax computation i , K i , V i vector taking the self-attention computation result : ； wherein, represents the dimension of the encoding vector; S4, performing attention aggregation on the encoding vectors obtained through the time sequence comparison calculation to obtain aggregated encoding vectors; S5, based on the aggregated encoding vectors, predicting the 6-axis speed of the camera at the current time, realizing the large-scale visual model prior transformation migration based on the time sequence contrast network design, forming an end-to-end control from image input to control output, and controlling the camera or robot to complete the visual servo task. 2.The pre-trained visual model based temporal contrastive visual servoing method of claim 1, wherein, Before step S1, a data set generation step is further included: Based on a high-fidelity simulator, import industrial robot models and object models to build a visual servoing basic environment; Triply randomize the servo target, servo initial pose and servo environment; Encapsulate the simulation process through a GPU-accelerated parallel computing framework to realize multi-scenario synchronized data acquisition. 3.The pre-trained visual model based time-sequential contrastive visual servoing method of claim 2, wherein, Randomly configuring the servo initial pose comprises: Taking the center of the target object group as the center of a spherical shell, randomly sampling points on the spherical shell as the positions of the camera, and calculating the orientation of the camera with the center as the focal point to determine the servo initial pose. 4.The pre-trained visual model based temporal contrastive visual servoing method of claim 1, wherein, After the passage of the self-attention contrastive network calculation, the final aggregation encoding vector is generated by performing attention aggregation on the obtained : ； wherein is the weight fraction of the i-th portion in the i-th portion. 5.The pre-trained visual model based temporal contrastive visual servoing method of claim 4, wherein, Linear layer projection is used to predict the aggregated encoded vectors The direction of the target camera velocity at the moment And the module length Finally, the direction vector is normalized and multiplied by the module length to synthesize the final target camera velocity : 。

Images