Rope-driven flexible mechanical arm target tracking method and system based on visual servo

Abstract

The invention discloses a rope-driven flexible mechanical arm target tracking method and system based on visual servo, and the method comprises the steps: carrying out the real-time tracking of a target through a mixed visual module, and continuously outputting the bounding box information and tracking confidence score of the target; according to the method, visual errors are used as guidance, Jacobi is driven by learned data to serve as an accurate conversion model, and a tracking confidence score is introduced to dynamically adjust the response degree of control, so that an accurate and smooth motor driving instruction is solved. The system comprises a mixed target tracking module, a kinematics and jacobian identification module and a self-adaptive control module. By using the method, the tracking precision of the mechanical arm is improved by utilizing a vision and control cooperation mechanism. The method can be widely applied to the field of mechanical control.

Description

Technical Field

[0001] This invention relates to the field of mechanical control, and in particular to a target tracking method and system for a cable-driven flexible robotic arm based on visual servoing. Background Technology

[0002] Existing technologies face dual bottlenecks in target tracking for vision-based tethered flexible robotic arms, stemming from both perception and control. At the visual perception level, mainstream tracking algorithms struggle to balance real-time performance and robustness, either failing due to model drift or target occlusion, or being computationally too demanding to meet the high-frequency requirements of robot control. At the robotic arm control level, the inherent non-rigid characteristics of tethered flexible arms (such as rope elasticity and arm deformation) lead to severe inaccuracies in their kinematic models. Traditional control methods struggle to achieve precise positioning and lack the ability to perceive and adjust external contact forces, resulting in rigid and unsafe physical interactions.

[0003] More importantly, existing systems generally lack deep collaborative mechanisms when integrating vision and control. Most solutions remain at the level of simple command transmission, where the control system blindly executes the target position provided by the vision module without dynamically adjusting its motion strategy based on the "confidence" or "quality" of vision tracking. This decoupled design leads to unstable and overreacting reactions when faced with uncertain visual information, resulting in low overall robustness and intelligence, making it difficult to adapt to complex and unstructured real-world environments. Summary of the Invention

[0004] In view of this, in order to solve the technical problem of low tracking accuracy caused by the lack of a vision and control coordination mechanism in existing robotic arm tracking control methods, the present invention proposes a target tracking method for a cable-driven flexible robotic arm based on vision servoing. The method includes the following steps: First, a hybrid vision module integrating YOLOv8 and KCF intelligently tracks the target in real time, continuously outputting the target's bounding box information and tracking confidence score. This bounding box is used to construct the error signal for the visual servo control law. Simultaneously, addressing the core challenge of flexible robots lacking accurate analytical kinematic models due to their high nonlinearity, this solution employs a transfer learning strategy. A data-driven kinematic model that accurately reflects the true motion characteristics of the flexible arm is learned through a neural network. This offline-trained model can generate a data-driven Jacobian matrix in real time during system runtime using online automatic differentiation technology. Finally, the control system collaboratively utilizes these three core pieces of information within a unified control law: guided by visual error, using the learned data-driven Jacobian as an accurate transformation model, and introducing the tracking confidence score to dynamically adjust the control response, thereby calculating precise and smooth motor drive commands to achieve stable target tracking.

[0005] Based on the above scheme, this invention provides a target tracking method and system for a rope-driven flexible robotic arm based on visual servoing. At the perception level, this invention employs a hybrid vision algorithm that combines high-precision detection and high-speed tracking. Through confidence-based intelligent scheduling, it provides the system with stable, continuous, and high-quality target pose data, fundamentally solving the tracking interruption problem caused by target occlusion or rapid movement. This robust visual information is then input into a data-driven control core. This core learns a Jacobian matrix that accurately reflects the true physical characteristics of the robotic arm through a neural network, thus bypassing the inaccuracy problem of traditional physical modeling methods when facing complex nonlinear factors such as rope elasticity and friction. In addition, this invention establishes an intelligent collaborative mechanism between perception quality and control strategy, which can evaluate the decline in visual tracking quality caused by arm vibration in real time and adjust the aggressiveness of motion control online accordingly to actively suppress system vibration. In summary, this invention patent constructs a logically self-consistent closed-loop system by tightly coupling reliable visual perception, accurate data-driven models, and adaptive collaborative control, fundamentally improving the overall performance of visual servoing technology in controlling such flexible robotic arms. Attached Figure Description

[0006] Figure 1 This is a flowchart of the steps of a target tracking method for a rope-driven flexible robotic arm based on visual servoing according to the present invention; Figure 2 This is a schematic diagram of the hybrid tracking process in a specific embodiment of the present invention; Figure 3 This is a schematic diagram of the servo control process in a specific embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the application and control process of the adaptive function in a specific embodiment of the present invention. Detailed Implementation

[0007] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0008] It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0009] It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this application are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0010] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "a," and / or "the" are not specifically singular and may include the plural. Generally, the terms "comprising" and "including" only indicate the inclusion of expressly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0011] In the description of the embodiments of this application, "a plurality of" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0012] Furthermore, flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Additionally, other operations can be added to these processes, or one or more steps can be removed from them.

[0013] Reference Figure 1This is a flowchart illustrating an optional example of the target tracking method for a cable-driven flexible robotic arm based on visual servoing proposed in this invention. This method can be applied to computer devices, and the target tracking method proposed in this embodiment may include, but is not limited to, the following steps: Step S1: Using YOLOv8 as a global target detector and scale-adaptive KCF as a high-speed inter-frame tracker, target recognition is performed based on the acquired images to generate target location information and visual confidence scores. Step S2: Extract features from the bounding box in the target location information, and combine it with the kinematic model to perform inverse kinematics instruction calculation to generate control instructions; Step S3: Introduce an adaptive function and adjust the control command based on the visual confidence score to obtain the final control command.

[0014] In some feasible embodiments, step S1 specifically includes: YOLOv8 is used as the global object detector. During tracking task initialization or when the high-speed tracking module fails, the entire image frame is scanned to output the target's bounding box information with high accuracy. Additionally, a scale-adaptive KCF is employed as a high-speed inter-frame tracker. KCF utilizes the cyclic matrix characteristics of the region surrounding the target to perform matrix operations in the Fourier frequency domain, improving computational speed. The goal is to learn a filter w such that the filtered response f(x) of the target image patch x approximates a desired Gaussian response. Its solution in the dual space is: The ^ symbol represents the Fourier transform. It is the complex conjugate of x. λ represents element-wise multiplication, α is the regularization coefficient, and α is the filter coefficient in the dual space. When tracking a new frame image z, the response map R is calculated using the following formula; the point with the strongest response is the new target location: It is the kernel correlation between the base sample x and the candidate sample z.

[0015] This high-speed tracker is coordinated with the YOLOv8 detector through a confidence-based intelligent scheduling mechanism. The complete process is described in [reference needed]. Figure 2 .

[0016] Initialization Phase. The system first calls YOLOv8 to perform global detection on the first input frame image. If the predetermined target is successfully identified, its bounding box B0 is recorded, and the corresponding feature map (such as HOG features) is extracted as the initial training sample x0 for KCF. Simultaneously, a Gaussian function with a peak centered at B0 is generated as the expected response output y. Substituting these data into the ridge regression formula: The initial correlation filter model can then be trained, and the system immediately enters a high-speed tracking state.

[0017] High-speed tracking and response calculation stage. For the subsequent t-th frame image, the system extracts candidate image patch z near the target location in the previous frame. To calculate its similarity with the target model, this embodiment first calculates the kernel correlation between the two. And obtain the frequency domain representation of the response map in the Fourier domain. : Transform it back into the spatial domain using an inverse Fourier transform to obtain the response map R. t The coordinates of the point with the largest response value in the diagram represent the new center position of the target. To achieve scale adaptation, the system will revolve around the new center point at a series of scale factors s ∈ {a1, a2, ..., a...} n Extract image patches and calculate responses, then select the scale s that maximizes the response peak. max As the current optimal metric.

[0018] In some feasible embodiments, step S1 further includes: Confidence assessment and intelligent scheduling phase.

[0019] When the KCF filter finds the best match for the target model within the search region, it produces a sharp peak in the response map R. If the target is occluded or deformed, the peak value will significantly decrease, flatten, or even exhibit multiple cluttered secondary peaks. The sidelobe response ratio (PSR) can be used as a metric to quantify this "peak sharpness." PSR is defined as the relationship between the maximum value of the response peak and the statistical characteristics of the sidelobe region. Assuming the response map is R(i, j), its maximum response value (peak value) is... : Define a main lobe region (e.g., an 11×11 window) around the peak point, and define the remaining area outside this main lobe region (or a ring-shaped region around the peak point) as the side lobe region. Then calculate the mean μ of all response values ​​within the side lobe regions. s and standard deviation σ sThe confidence score C is represented by the PSR value: Among them, molecules ( The denominator σ represents the "absolute height" of the peak above the sidelobe background. s This represents the "noise level" or "flatness" of the sidelobe background.

[0020] Three-stage intelligent scheduling mechanism: The intelligent scheduling mechanism sets two key thresholds: a high confidence threshold T. H (e.g., 8.0) and low confidence threshold T L (For example, 4.5). These two values ​​are usually calibrated offline based on factors such as lighting and target characteristics in the actual scene. The confidence score C is calculated in frame t. t Then, the system executes the following three-stage logical judgment: High confidence level (C t > T H The system determines that the current tracking result is highly reliable, indicating that the target is clear and unobstructed. At this point, the system will fully adopt the new position and scale given by the KCF tracker as the final output for this frame. Simultaneously, the system considers this a high-quality training sample and will use a relatively high learning rate η. h The tracking model is smoothly updated to quickly adapt to the normal evolution of the target's appearance. The update process applies simultaneously to the appearance model x and the filter coefficients α, as shown in the following formula: Medium confidence level (T) L < C t < T H The result indicates that the system has basically successfully captured the target, but the result has some uncertainty, which may stem from slight motion blur or partial occlusion. In this case, to ensure the continuity of tracking, the system will temporarily accept the localization result given by KCF. However, to prevent the model from being "contaminated" by low-quality, noisy samples and causing subsequent tracking drift, the system will adopt a conservative model update strategy, using a value much smaller than η. h learning rate η l Update the model, or even skip the model update step entirely in this frame, to maintain the stability of the current model.

[0021] Low confidence level (C) t < T LThe system determines it has entered a low-confidence state, which usually means that tracking has experienced severe drift or complete failure. At this point, the system will immediately trigger a re-detection mechanism. It will first discard the unreliable results given by the current KCF tracker and immediately call the YOLOv8 global detector to search the current complete image frame. If YOLOv8 successfully relocates the target, the system will use its detection bounding box as a reference to forcibly correct and completely reset the state and model of the KCF tracker, essentially restarting tracking with a completely new, high-quality sample. If YOLOv8 also fails to find the target, it is determined that the target is temporarily lost, and the system will continue to call YOLOv8 for a global search in subsequent frames until the target is found.

[0022] In some feasible embodiments, step S2 specifically includes: The core of this step is to leverage the powerful nonlinear fitting capabilities of neural networks to directly learn the kinematic laws of the robotic arm from experimental data, and to generate a high-precision Jacobian matrix in real time using automatic differentiation technology, providing an accurate mathematical model for visual servo control.

[0023] Phase 1: Learning the offline forward kinematics model.

[0024] The goal at this stage is to train a neural network f. NN This allows it to accurately simulate the mapping relationship from the driving space (motor angle q) to the task space (endpoint pose x), i.e., x = f NN (q).

[0025] An efficient "simulation-to-reality" transfer learning strategy is employed to construct the kinematic model, which consists of two stages: simulation pre-training and real-data fine-tuning. First, massive amounts of ideal training data are generated in a dynamic simulation environment, such as motor angles q and end-effector poses x with no measurement noise and perfectly aligned timestamps. Based on this, the neural network is thoroughly pre-trained to initially grasp the basic motion laws of the robotic arm. Subsequently, real-world data is collected on a physical prototype to fine-tune the pre-trained model, ensuring it accurately adapts to the physical characteristics of the real world. To ensure good generalization ability of the collected real-world data, two drive motors are controlled, causing the end-effector to move along a pre-designed spatial spiral or compound sine trajectory, ensuring sufficient and uniform data coverage of its core workspace. During this process, a fixed, calibrated global camera is used to synchronously record a large number of training data pairs. Each data pair (q...) i , x i Includes the drive angle vector measured by the motor encoder: And the seven-dimensional pose (3 positions + 4 pose parameters) of the robotic arm's end effector in the base coordinate system, measured by an external camera system. Finally, after data aggregation, a large-scale dataset for fine-tuning can be obtained: The acquired raw data needs to be preprocessed. This includes low-pass filtering of high-frequency noise and precise alignment of the drive command q with the true pose value x. true To avoid features of different scales dominating network training, we normalize all input and output data to a similar numerical range (e.g., [-1, 1]).

[0026] Neural Network Training: This scheme uses a multilayer perceptron (MLP) with multiple hidden layers as the fitting network. For example, it can be configured with a structure containing 4 hidden layers, each with 256 neurons. The number of neurons in the input layer corresponds to the number of driving motors (2), and the number of neurons in the output layer corresponds to the pose description dimension (the position of the terminal p∈R³ and the quaternion q∈R). 4 The hidden layers use non-linear activation functions such as ReLU to fit complex mapping relationships.

[0027] The training process for this network is divided into two core phases.

[0028] Phase 1 (Simulation Pre-training): First, the MLP network described above is thoroughly trained using a massive amount of ideal data generated in a simulation environment. The training objective of this network is to find an optimal set of network parameters. (i.e., all weights and biases) to minimize the error between the network's predicted pose and the actual measured pose. This error is expressed through a composite loss function. Quantization is performed, and it is composed of a weighted average of position error and attitude error: Among them, position error Mean square error is used: Attitude error The dot product of quaternions is used to measure the angular difference between attitudes: After this stage is completed, we obtain a pre-trained model that has mastered the ideal motion law of the robotic arm.

[0029] The second stage (fine-tuning with real data): This stage no longer starts with random weights, but instead loads a pre-trained network model. We use a relatively small real dataset D collected from the physical prototype to fine-tune the model. We also divide the real dataset D into training, validation, and test sets. The training process still uses the Adam optimizer, and the network parameters are continuously updated using gradient descent on the real training set. This is to minimize the aforementioned loss function. However, the key difference is that the fine-tuning stage usually uses a learning rate much smaller than that used in the pre-training stage. The purpose is to allow the model to accurately adapt to subtle differences introduced by factors such as friction and elasticity in the real world without destroying the general knowledge already learned.

[0030] After training is completed, the test set that was never used in training will be used to objectively evaluate the accuracy and generalization ability of the final model. Its core evaluation metrics include not only the loss function value, but also the mean absolute error that has clear meaning in the physical world, such as end position error (unit: millimeter) and attitude error (unit: degree).

[0031] The fine-tuned neural network is itself the final kinematic model. The task of the kinematic model is to describe the mapping from the motor angle q to the end-effector pose x, i.e., x = f(q). The fine-tuned neural network f... NN It is the concrete, computable entity of this mapping function f.

[0032] The fine-tuning process is as follows: First, the network near the input layer is "frozen," as these lower-level networks have learned general basic features from simulations. By setting their parameters to non-trainable, these general rules learned from simulations are effectively preserved, allowing only higher-level networks to learn the subtle differences brought about by real data. Subsequently, all layers can be unfrozen for overall fine-tuning. Simultaneously, a learning rate much smaller than that in the pre-training stage is set, typically 1% to 10% of the original rate. The aim is to finely adjust the model weights to adapt to the characteristics of real data without destroying existing knowledge, avoiding model oscillations caused by an excessively large learning rate that could lead to the loss of existing basic rules.

[0033] The training process is then restarted using the prepared real dataset, with the same objective function as during pre-training: minimizing the composite error between the predicted pose and the actual measured pose. Since real datasets are typically small, models are more prone to overfitting, so close monitoring of the validation set loss is crucial. An "early stopping" strategy can be used to avoid this problem: training is terminated early when the model's performance on the validation set stops improving for several consecutive epochs, and the best-performing model at that point is saved as the final optimized version.

[0034] Phase 2: Online Jacobian Matrix Generation and Visual Servo Control.

[0035] In a control cycle, the process begins with a visual perception step, such as... Figure 3 As shown, the system first calls the YOLOv8 and KCF hybrid tracking module designed in this invention to process the current frame image acquired by the camera. Utilizing an intelligent scheduling mechanism, this module can efficiently and stably output the real-time bounding box B of the target. t Subsequently, the system extracts the visual feature vector s∈R at the current time from the four corner points (tl, tr, br, bl) of the bounding box. 8 : The system then calculates the current feature s and the target feature s preset at the center of the image. d The 8-dimensional error vector between them is e = s - s d To translate this image error into precise motion of the end effector, we follow an image-based vision servo control framework. At the heart of this framework is the image Jacobian matrix L. s ∈R 8 It establishes the six-dimensional velocity v of the end camera. c With the speed of change of image features Linear relationship between them: To ensure that the error e converges exponentially quickly, the desired camera motion speed v c Set as: Where λ is the control gain. This is the pseudo-inverse of the image Jacobian matrix. The desired terminal velocity v is calculated. c The next step is to use the data learned offline to drive the robot's Jacobian matrix J. NN This converts the speed command in the task space into the motor speed command in the drive space. In the current motor state q... t Below, the system uses a pre-trained forward kinematic neural network f NN (q) Perform automatic differentiation to calculate the Jacobian matrix in real time, which accurately reflects the differential motion relationship of the flexible arm: By taking the pseudo-inverse of the Jacobian matrix driven by this data, we can inversely solve for the final angular velocity command vector driving the two motors: This represents the current driving angle vector of the motor; the superscript + indicates taking the pseudo-inverse of the matrix; This indicates the six-dimensional velocity of the end-point camera. Indicates control gain; Represents the Jacobian matrix of the image Seeking the pseudo-inverse; This represents the current visual feature vector extracted from the image in real time. This represents a pre-defined target visual feature vector.

[0036] The pseudo-inverse J⁺ is used here instead of the standard inverse because this system is a typical overdetermined system (using two motors to control the end effector with six degrees of freedom). The physical significance of finding the pseudo-inverse lies in finding a set of velocities that ensure the actual velocity produced by the end effector matches the desired velocity. The motor speed command that minimizes the sum of squares of the errors between them This refers to the "optimal approximate solution" that can be achieved under the current constraints.

[0037] Finally, the calculated motor speed command The data is sent to the underlying servo driver for execution. Through this entire process, this solution uses an accurate data-driven Jacobian matrix J learned from real data. NN This replaces the analytical Jacobian matrix, which has significant errors in traditional methods. This data-driven Jacobian, obtained through automatic differentiation, not only accurately fits the complex nonlinearities of the system but sometimes even provides better numerical conditions than the Jacobian derived analytically, thus exhibiting greater stability near singular points. Simultaneously, it allows visual servo control to maintain high precision and robustness even when facing highly nonlinear flexible arms.

[0038] In some feasible embodiments, step S3 specifically includes: In the visual processing stage of the main control loop, the feature points s of the target and the confidence score C quantified by the peak-to-sidelobe ratio are obtained. t In classical visual servo control theory, the choice of control gain λ is a crucial trade-off between fast convergence and system stability. For rigid robots, a relatively fixed, large gain can be chosen to achieve a fast response. However, for flexible, oscillating cable-driven robotic arms, a fixed high gain almost inevitably leads to system overshoot and continuous oscillation, while a fixed low gain makes tracking performance too sluggish. The core innovation of this module lies in replacing this fixed gain λ with a value dependent on the visual confidence score C. t The adaptive function λ(C) t This control process refers to... Figure 4 .

[0039] This embodiment uses the Sigmoid function to achieve this adaptive relationship: Where, λ max and λ min These are the preset maximum and minimum control gains; C mid is the center point of the confidence level; k is the slope of the Sigmoid function, which controls the drasticness of its change.

[0040] The key parameters in the aforementioned adaptive relationship need to be determined through a combination of offline system calibration and empirical testing.

[0041] Maximum / Minimum Gain , : The value of is determined by the highest response speed that does not cause significant oscillations in the system under ideal visual conditions. It is then set to a small positive value that can ensure slow and stable error correction.

[0042] Confidence center This value is obtained through statistical analysis of a large number of PSR confidence scores in Module 1. For example, it can be set near the 25th percentile of the PSR score in a typical tracking scenario as an empirical boundary for the transition from "reliable" to "suspicious".

[0043] slope This parameter controls the smoothness of gain switching. A smaller value indicates a smoother transition. A higher value will result in a smoother transition and more stable system performance; a larger value... The value brings a faster switching characteristic closer to that of a "switch". Its final value needs to be weighed according to the specific requirements of response speed and stability in the actual application.

[0044] When visual quality is good (C) t > C mid ), σ s The output of the function approaches 1, at which point λ(C) t Approaching λ max The system tracks with maximum efficiency.

[0045] When visual quality is poor (C) t < C mid ), σ s The output of the function approaches 0, at which point λ(C) t Approaching λ min The system moves in a gentle and conservative manner to avoid violent movements caused by incorrect visual information.

[0046] When visual quality is in the intermediate range, the gain transitions smoothly.

[0047] Substituting this adaptive gain into our final control formula, we obtain the new motor speed command. : This mechanism forms an active damping closed loop in actual operation. The visual confidence score C is evaluated in real time. t It adaptively adjusts the aggressiveness of the control law, thereby improving the stability of the system.

[0048] Furthermore, the adaptive gain adjustment mechanism proposed in this scheme has good scalability. In the current description, we treat the gain λ as a scalar, scaling all components of the six-dimensional velocity at the end proportionally. For more refined control requirements, it can be easily extended to a diagonal gain matrix: This allows us to design independent adaptive gain functions for the translational and rotational motions of the end effector, and even for each individual axis of motion. For example, in some tasks, we might want to maintain a fast response for rotation even if visual confidence decreases, while only reducing the response speed for translation. This decoupled adaptive adjustment strategy enables more complex control performance.

[0049] A target tracking system for a cable-driven flexible robotic arm based on vision servoing includes: The hybrid target tracking module is used to perform step S1; The kinematics and Jacobi identification module is used to perform step S2; The adaptive control module is used to execute step S3.

[0050] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0051] A variety of target tracking devices for cable-driven flexible robotic arms based on vision servoing: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the visual servo-based rope-driven flexible robotic arm target tracking method described above.

[0052] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0053] A storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement a visual servo-based cable-driven flexible robotic arm target tracking method as described above.

[0054] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0055] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for target tracking of a rope-driven flexible manipulator based on visual servoing, characterized in that, The method comprises the following steps: Based on the global target detector and the inter-frame tracker, target recognition is performed according to the collected images to generate target position information and a visual confidence score; Feature extraction is performed on the target position information, and inverse kinematics instruction solving is performed in combination with a kinematics model to generate a control instruction; An adaptive function is introduced, and the control instruction is adjusted in combination with the visual confidence score to obtain a final control instruction.

2. The method of claim 1, wherein, The step of generating target position information and a visual confidence score based on the global target detector and the inter-frame tracker according to the collected images specifically comprises: A global detector is called to perform global detection on an input first image to identify a first bounding box; A corresponding feature map is generated based on the first bounding box as an initial training sample of an inter-frame tracker, and the inter-frame tracker is initialized; A response map is calculated according to a subsequent frame image; Target position information is generated according to the coordinates of the point with the maximum response value in the response map; A peak side lobe ratio is calculated according to the response map to obtain a visual confidence score. 3.The target tracking method of the vision servo based rope-driven flexible manipulator according to claim 1, wherein, The step of generating target position information and a visual confidence score based on the collected images also comprises: When the visual confidence score is greater than a first threshold, the target position information output by the inter-frame tracker is taken as the final output of the frame image, and the model of the inter-frame tracker is updated at a first learning rate; When the visual confidence score is less than the first threshold and greater than a second threshold, the target position information output by the inter-frame tracker is taken as the final output of the frame image, and the model updating step is skipped; When the visual confidence score is less than the second threshold, the global detector is called again to perform target detection, and the inter-frame tracker is reset.

4. The method of claim 1, wherein, The step of obtaining the kinematics model specifically comprises: Ideal training data is generated based on a dynamics simulation environment; A pre-training model is obtained by pre-training a fitting network based on the ideal training data; Real data is collected from a physical prototype; The final kinematics model is obtained by fine-tuning the pre-training model based on the real data.

5. The method of claim 1, wherein, The step of performing feature extraction on the target position information and performing inverse kinematics instruction solving in combination with the kinematics model to generate a control instruction specifically comprises: Feature extraction is performed on the target position information to generate a current frame visual feature vector; An error vector between the current frame visual feature vector and a target feature preset at the center of the image is calculated; The kinematics model is automatically integrated based on a current motor state to generate a Jacobian matrix; The pseudo-inverse of the Jacobian matrix is calculated in combination with the error vector to obtain a control instruction.

6. The method of claim 5, wherein, The formula of the control instruction is as follows: wherein, a control command representing an angular velocity; a data-driven Jacobian matrix calculated in real time; a current driving angle vector of the motor; a superscript + represents a pseudo-inverse of a matrix; a six-dimensional velocity of the end camera, a control gain; a pseudo-inverse of the image Jacobian matrix a pseudo-inverse of the image Jacobian matrix a current visual feature vector extracted in real time from the image; a target visual feature vector set in advance.

7. The method of claim 1, wherein, The adaptive function is as follows: Where, λ max With λ min Indicates the preset maximum and minimum control gain; C mid The center point of the confidence level is represented; k represents the preset slope, which controls the drasticness of its change. This represents the visual confidence score. 8.A vision servoing based rope-driven flexible manipulator target tracking system, characterized in that, It comprises: A hybrid target tracking module that generates target position information and a visual confidence score based on a global target detector and an inter-frame tracker according to collected images; A kinematics and Jacobian identification module that performs feature extraction on the target position information and performs inverse kinematics instruction solving in combination with a kinematics model to generate a control instruction; and A control instruction adjustment module that introduces an adaptive function and adjusts the control instruction in combination with the visual confidence score to obtain a final control instruction. An adaptive control module is introduced to adjust the control command by incorporating an adaptive function and the visual confidence score to obtain a final control command.

9. A vision servoing based rope-driven flexible manipulator target tracking device, comprising: The method comprises the steps of: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, so that the at least one processor implements a target tracking method of a rope-driven flexible manipulator based on visual servoing according to any one of claims 1-7.

Images