Vision servo manipulator motion control method based on natural features

Abstract

The present application relates to the technical field of visual servo manipulator motion control, and specifically to a method for visual servo manipulator motion control based on natural features, the method comprising: using a visual servo manipulator to acquire a target image in real time, obtaining an original image and a real-time image; determining the matching anomaly coefficient of each acquired real-time image; obtaining the Jacobian matrix of each acquired real-time image using a group-based Broyden iteration algorithm, specifically: determining the influence weight of each acquisition on each acquisition; using the historical data step size of each iteration of the iterative algorithm as an unknown quantity, obtaining the step size adjustment coefficient of the iterative algorithm at each acquisition, constructing an objective function, and solving the unknown quantity using an optimization algorithm; determining the movement speed of each joint of the visual servo manipulator based on the Jacobian matrix, and performing motion control on the visual servo manipulator. The present application improves the accuracy of motion control of the visual servo manipulator.

Description

Technical Field

[0001] The present application relates to the technical field of visual servoing robot arm motion control, and in particular to a visual servoing robot arm motion control method based on natural features. Background Art

[0002] With the rapid development of intelligent manufacturing, the demand for robotic arms in various application scenarios is increasing. Traditional robotic arm control methods often rely on precise calibration processes, which are not only time-consuming but also difficult to achieve high-precision operation in complex and dynamically changing environments. Natural feature-based visual servoing robotic arm motion control avoids the tedious calibration steps by directly utilizing natural features in the environment for position estimation and motion control. It is particularly suitable for unstructured and dynamically changing environments.

[0003] In uncalibrated visual servoing tasks, nonlinear optimization methods are required to estimate the Jacobian matrix to achieve precise control of the robotic arm. However, nonlinear optimization methods rely to a certain extent on secant models. The Broyden algorithm, as a type of secant model, is very sensitive to noise and relies only on data from a single control cycle adjacent to the current moment. It is easily affected by fluctuations in data within a single cycle, resulting in inaccurate estimation results.

[0004] The swarm-based Broyden (BP) iteration algorithm uses multiple historical data points for iterative updates. However, the quality of the iteration results is related to the number of selected historical data points. As the number of historical data points increases, the computational burden increases significantly, making it difficult to adapt to the real-time control of the robot arm. At the same time, abnormal fluctuations in the selected historical data will still affect the effectiveness of the swarm-based Broyden (BP) iteration algorithm, reducing the quality of the Jacobian matrix and affecting the control accuracy of the robot arm. Therefore, how to select high-quality historical data for swarm-based Broyden (BP) iteration to balance the control accuracy and computational efficiency of the visual servo robot arm has become a technical challenge that needs to be solved urgently. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides a visual servoing robot arm motion control method based on natural features to solve the existing problems.

[0006] The natural feature-based visual servo robot arm motion control method of this application adopts the following technical solutions:

[0007] One embodiment of the present application provides a natural feature-based visual servoing robot arm motion control method, the method comprising the following steps:

[0008] The target image is captured in real time using a visual servo robot arm. The first captured image is recorded as the original image, and the target area in the original image is selected. All images except the original image are recorded as real-time images.

[0009] Perform feature matching on the real-time image and the original image to obtain a set of matching points, and remove abnormal points in the matching point set to obtain an inlier set; classify all feature points in the real-time image that are located in the inlier set based on the grayscale discreteness of the pixels in the neighborhood of each feature point in the real-time image that are located in the inlier set;

[0010] Analyze the difference between the matching point set and the interior point set in the real-time images collected each time, as well as the difference in the number of feature points of different types, and determine the matching anomaly coefficient of the real-time images collected each time in combination with the grayscale discreteness;

[0011] The Jacobian matrix of each collected real-time image is obtained using the Broyden iterative algorithm based on the population, specifically:

[0012] Determine the influence weight of each acquisition on the previous acquisition based on the matching anomaly coefficient of each acquisition and the real-time image of each acquisition, as well as the interval between each acquisition;

[0013] The historical data step length of each iteration of the iterative algorithm is used as an unknown quantity, and the step length adjustment coefficient of the iterative algorithm at each acquisition is obtained by combining the matching anomaly coefficient and the influence weight. The objective function is constructed by combining the step length adjustment coefficient of the previous acquisition, and the unknown quantity is solved by using an optimization algorithm;

[0014] The motion speed of each joint of the visual servoing robot arm is determined based on the Jacobian matrix, and the motion of the visual servoing robot arm is controlled.

[0015] In one embodiment, classifying all feature points in the inlier set in the real-time image includes:

[0016] A threshold segmentation algorithm is used to obtain a segmentation threshold of the grayscale discreteness of pixels in the neighborhood of all feature points in the inlier set in the real-time image, and all feature points in the inlier set in the real-time image are divided into two categories based on the segmentation threshold.

[0017] In one embodiment, the process of determining the matching anomaly coefficient is as follows:

[0018] For each acquired real-time image, calculate the ratio of the cardinality of the matching point set to the cardinality of the inlier set, recorded as the first ratio; calculate the mean of the grayscale dispersion of the pixels in the neighborhood of all feature points located in the inlier set in the real-time image; calculate the proportion of feature points whose grayscale dispersion is less than the segmentation threshold among all feature points located in the inlier set in the real-time image;

[0019] The matching anomaly coefficient is determined based on the first ratio, the mean, and the proportion, wherein the matching anomaly coefficient is positively correlated with the first ratio and the proportion, and negatively correlated with the mean.

[0020] In one embodiment, the ratio of the proportion to the mean is calculated and recorded as a second ratio, and combined with the first ratio to obtain the matching anomaly coefficient.

[0021] In one embodiment, the matching anomaly coefficient is the product of the first ratio and the second ratio.

[0022] In one embodiment, the process of determining the influence weight is as follows:

[0023] Calculate the sum of the matching anomaly coefficients of the real-time images acquired in each acquisition and all previous acquisitions, calculate the ratio of the matching anomaly coefficient of the real-time image acquired in any acquisition before each acquisition to the sum, record it as a third ratio, and calculate the difference between the sequence number of each acquisition and the sequence number of any acquisition;

[0024] The difference is used as the exponent of an exponential function with a preset time decay factor as the base, and the influence weight of any previous acquisition in each acquisition is determined by combining the third ratio and the calculation result of the exponential function.

[0025] In one embodiment, the difference between 1 and the third ratio is calculated and recorded as the first difference, and the influence weight of any previous collection during each collection is the product of the first difference and the calculation result of the exponential function.

[0026] In one embodiment, the step size adjustment coefficient is expressed as:

[0027] Where, is the step size adjustment coefficient when using the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the mean value of the matching anomaly coefficient of the real-time images collected before i times, is the historical data step length of the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the influence weight of the nth acquisition before the i-th acquisition of real-time image, where, when i=2, .

[0028] In one embodiment, the objective function is expressed as Where, For the The objective function when acquiring real-time images is: is the step size adjustment coefficient when the Broyden iterative algorithm based on population is used for the real-time image acquisition for the i-1th time. In one embodiment, the objective function is minimized when solving the unknown quantity.

[0029] This application has at least the following beneficial effects:

[0030] The present application uses a visual servoing robot to capture the target image in real time, records the first captured image as the original image, and selects the target area in the original image; records all images except the original image as real-time images; performs feature matching between the real-time image and the original image to obtain a matching point set, and removes abnormal points in the matching point set to obtain an inner point set; classifies all feature points in the real-time image that are located in the matching point set based on the grayscale discreteness of the pixel points in the neighborhood of each feature point in the matching point set in the real-time image; reduces the influence of environmental noise on the feature point extraction in the real-time image, and improves the accuracy of complex scenes. Adaptability, improve the robustness of natural feature matching; analyze the difference between the matching point set and the internal point set in the real-time images collected each time, as well as the difference in the number of feature points of different types, and determine the matching anomaly coefficient of the real-time images collected each time in combination with the grayscale discreteness; the determination of the matching anomaly coefficient enhances the self-diagnosis ability of the matching quality, and can quantify the comprehensive impact of feature drift and noise interference to dynamically adjust subsequent control parameters; the Jacobian matrix of the real-time images collected each time is obtained using the Broyden iterative algorithm based on the population, specifically: based on the real-time images of each collection and each previous collection The matching anomaly coefficient of the image and the interval between each acquisition are used to determine the influence weight of each previous acquisition at each acquisition; the influence weight quantifies the influence of historical data on the current robotic arm control, reflects the reference value of historical data, and helps to improve the reliability of historical data step determination in the Broyden iterative algorithm based on the group; at the same time, key historical data are screened by the influence weight, redundant calculation amount is reduced, and computing resource allocation is optimized; the historical data step at each iteration of the iterative algorithm is used as an unknown quantity, and the step adjustment coefficient of the iterative algorithm at each acquisition is obtained by combining the matching anomaly coefficient and the influence weight. The objective function is constructed by combining the step adjustment coefficient of the previous acquisition, and the unknown quantity is solved by using the optimization algorithm; by taking into account both noise interference and historical data quality, the optimal historical data step at each iteration of the Broyden iterative algorithm based on the group is obtained, which improves the accuracy of historical data step determination and helps to improve the quality of the obtained Jacobian matrix. The motion speed of each joint of the visual servoing robotic arm is determined based on the Jacobian matrix, and the visual servoing robotic arm is motion controlled, thereby achieving a balance between computational efficiency and accuracy in the motion control process of the visual servoing robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 A flowchart of the steps of the visual servoing robot arm motion control method based on natural features provided in this application;

[0033] Figure 2 Get the flow graph for the Jacobian matrix;

[0034] Figure 3 Determine the flow chart for the optimal historical data step size. DETAILED DESCRIPTION

[0035] In order to further illustrate the technical means and effects adopted by this application to achieve the predetermined invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features and effects of the natural feature-based visual servoing robot arm motion control method proposed in this application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics of one or more embodiments may be combined in any suitable form.

[0036] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0037] The specific scheme of the visual servoing robot arm motion control method based on natural features provided by this application is described in detail below with reference to the accompanying drawings.

[0038] An embodiment of the present application provides a method for controlling the motion of a visual servoing manipulator based on natural features. Specifically, the following method for controlling the motion of a visual servoing manipulator based on natural features is provided. Figure 1 , the method comprises the following steps:

[0039] Step S001: Use a visual servo robot arm to capture a target image in real time, record the first captured image as the original image, and select the target area in the original image; record all images except the original image as real-time images.

[0040] The visual servoing robot arm based on natural features includes the following structures: a drone body, a robot arm base is provided on the drone body, and in this embodiment, a 6-DOF robot arm is installed on the robot arm base. A servo is provided at each joint of the robot arm, and the maximum speed of the joint is , a gripper is provided at the end of the robotic arm; the network camera of the visual servo system is installed in an "eye-in-hand" manner, that is, the network camera is installed at the end of the robotic arm and moves with the movement of the robotic arm. The image resolution of the network camera used in this embodiment is 640*480, and the maximum frame rate is 120 frames. It mainly performs visual servoing and position estimation on the acquired real-time images.

[0041] This embodiment uses a visual servoing robotic arm to capture images containing the target in real time, records the first captured image as the original image, and uses a rectangular frame to manually frame the area where the target is located in the original image. Then, when the drone robotic arm is used for visual servo detection in the future, the network camera is used to obtain real-time images, that is, all images captured except the original image are recorded as real-time images.

[0042] Step S002: perform feature matching on the real-time image and the original image to obtain a matching point set, and remove abnormal points in the matching point set to obtain an inlier set; based on the grayscale discreteness of the pixel points in the neighborhood of each feature point in the real-time image located in the inlier set, classify all feature points in the real-time image located in the inlier set.

[0043] Traditional robotic arm visual servo systems are highly dependent on the accuracy of system calibration. Furthermore, system calibration is only effective under the same calibration conditions; recalibration is required when conditions change. To avoid the inefficiency caused by complex system calibration, natural feature-based visual servo robotic arm motion control eliminates the need for pre-calibration. Instead, the system is matched based on the natural features of the image, such as its own point features, line features, and geometric features. Consequently, higher requirements are placed on the quality of the captured images. Furthermore, images acquired in natural environments inevitably introduce significant environmental noise. To mitigate the impact of noise on the Broyden algorithm's iterative acquisition of the Jacobian matrix, an improved population-based Broyden (BP) iteration algorithm is currently commonly used. However, as the step size of the selected historical data increases in the population-based Broyden (BP) iteration algorithm, the real-time requirements of visual servoing are seriously affected. Furthermore, abnormal fluctuations in the historical data still affect the Jacobian matrix. The control of the manipulator based on natural features is a kind of uncalibrated visual servoing. The objective function of the image-based visual servoing (IBVS) in the uncalibrated visual servoing is constructed by image information. The control structure is relatively simple, so it is more suitable for visual servoing based on natural features. The essence of IBVS control is to make the error between the original image and the target in the real-time image converge to 0, that is, the objective function is ,in is the error, Target features in real-time images, is the target feature in the original image. When the target function is determined, an open-loop system can be constructed ,in is the derivative of the objective function, is the Jacobian matrix, is the camera's movement speed. As the input of the robot controller, in order to ensure the error can converge to 0 exponentially, that is, , we can get ,in for The generalized inverse of is the gain constant, used to control the convergence speed, The interval range is [0.1,1]. In this embodiment Based on the above analysis, we can know that a key point of IBVS control is the error The structure of is the difference between the target features in the original image and the real-time image, so the accuracy of feature point matching in the two images is very important for subsequent errors. The construction of is crucial. In the image matching process, when the difference between pixels in the same image is smaller, it is more difficult to achieve accurate matching for subsequent image matching, thus affecting the matching accuracy.

[0044] Taking the real-time image collected for the i-th time as an example, this embodiment combines the original image with the The real-time image captured is used as input and the ORB feature matching algorithm is used to output a set of keypoints and a set of matching points. The obtained keypoint set and the corresponding matching point set are then used as input again, and the RANSAC algorithm is used as the homography model type to output a set of inliers. Both the ORB feature matching algorithm and the RANSAC algorithm are well-known technologies, and the specific processes will not be repeated here.

[0045] For each matching point pair in the inlier set, the feature point in the real-time image is taken. neighborhood, and calculate The discrete degree of the grayscale values of all pixels in the neighborhood, in this embodiment =5, the implementer can set it according to the actual situation, and this embodiment does not limit it here.

[0046] The discrete degree of all feature points obtained in the real-time image collected for the i-th time is used as input, Otsu threshold segmentation is adopted, and a segmentation threshold is output. The segmentation threshold is used to divide all feature points in the internal point set in the real-time image into two categories, that is, those greater than the segmentation threshold are classified into one category, and those less than the segmentation threshold are classified into the other category. Among them, Otsu threshold segmentation is a well-known technology and will not be described in detail.

[0047] It should be noted that the calculation method of the degree of dispersion described in this embodiment is variance. The implementer can choose other existing feasible methods for calculating the degree of dispersion according to actual conditions, such as standard deviation, coefficient of variation, etc., and this embodiment does not limit this.

[0048] Based on the above analysis, this embodiment analyzes the difference between the matching point set and the inlier point set in each collected real-time image, as well as the difference in the number of feature points of different types. Combined with the degree of dispersion, the matching anomaly coefficient of each collected real-time image is determined to measure the matching quality of the image and further to measure the noise level in the collected image. The specific calculation formula is:

[0049] : It is The matching anomaly coefficient of the real-time image acquired, 、 They are The cardinality of the internal point set and the cardinality of the matching point set after the real-time image collected is matched with the original image. It is The mean value of the discrete degree of all feature points in the real-time image collected, It is The proportion of feature points with a discrete degree less than the segmentation threshold in the real-time image collected this time among all feature points in the inlier set in the real-time image. Recorded as the first ratio, Recorded as the second ratio.

[0050] It should be understood that after image matching, a higher proportion of inliers in the matching point set indicates better matching quality, fewer mismatched point pairs, and thus a smaller matching anomaly coefficient. The degree of discreteness reflects the grayscale difference between pixels in the neighborhood of a feature point. The greater the grayscale difference, the greater the degree of discreteness. Since smaller grayscale differences between pixels in an image may affect the determination of inliers, the greater the grayscale difference between pixels, the greater the number of inliers with larger grayscale differences, and the corresponding noise affecting matching quality is relatively less, resulting in a smaller matching anomaly coefficient. Conversely, when more noise affects matching quality, the corresponding matching anomaly coefficient is larger.

[0051] Step S003, using the Broyden iterative algorithm based on the population to obtain the Jacobian matrix of the real-time images collected at each time, specifically: (1) based on the matching anomaly coefficient of the real-time images of each acquisition and each previous acquisition, as well as the interval between each acquisition and each acquisition, determine the influence weight of each acquisition before it.

[0052] In the swarm-based Broyden (BP) iterative algorithm, the historical data step size has a significant impact on the iteration quality and iteration efficiency. When the quality of the historical data within the selected step size range is poor, the selected historical data is not enough to ensure the quality of the Jacobian matrix generated by the iteration, which makes the accuracy of the subsequent robotic arm control insufficient. Therefore, it is necessary to expand the step size to ensure that there is enough historical data to participate in the iteration, thereby improving the quality of the Jacobian matrix; when the degree of variation of the historical data within the selected step size range is small, there is no need for too much historical data to participate in the iteration, so a smaller historical data step size can be selected to reduce the amount of data involved in the iteration, thereby improving the iteration efficiency and ensuring real-time performance. The Jacobian matrix acquisition flow chart is as follows: Figure 2 shown.

[0053] It should be noted that the historical data step size refers to the number of historical data points involved in the iteration of the swarm-based Broyden (BP) iterative algorithm. In this embodiment, the historical data step size corresponds to the number of real-time image acquisitions, that is, each real-time image acquisition corresponds to a corresponding amount of historical data. When performing motion control on a visual servoing manipulator, it is necessary to acquire a real-time image, use the swarm-based Broyden (BP) iterative algorithm, obtain the Jacobian matrix, determine the manipulator's motion velocity, and complete motion control. That is, each real-time image acquisition corresponds to a motion control of the visual servoing manipulator.

[0054] Therefore, choosing an appropriate historical data step size is crucial. Based on the above analysis, the key factor affecting iteration quality lies in the variability of historical data. For data with more severe variability, its influence on the iterative Jacobian matrix generation should be reduced. Furthermore, since the target to be grasped by the robotic arm may change over time, for example, objects left outdoors for a long time may exhibit characteristic differences from the initial captured image due to the effects of ultraviolet light or oxidation. In this case, the longer the span of historical data from the current real-time image acquisition, the smaller the corresponding influence weight should be. Finally, in cases of severe data variability, the step size should be increased to ensure sufficient historical data is included and improve the quality of the generated Jacobian matrix. Conversely, for historical data with less variability, its influence on the iterative Jacobian matrix generation should be increased, while the step size should be reduced to ensure operational efficiency. Based on the above analysis, the influence weight of each acquisition relative to its previous real-time image acquisition is determined based on the matching anomaly coefficient of each acquisition and its previous real-time image acquisition, as well as the interval between each acquisition. Taking the Jacobian matrix obtained using the population-based Broyden iteration algorithm for the i-th real-time image acquisition as an example, the specific expression is:

[0055] Where, It is When the real-time image is collected for the first time, the The impact weight of the collection, 、 They are sequence The matching anomaly coefficient of the real-time image collected, i and j are the serial numbers of the collection times, It is a preset time decay factor, which is used to measure the degree of reduction of the impact weight as the span of acquisition times increases. The value range of , when the value is larger, the influence of the acquisition time factor is smaller. In this embodiment , the implementer can set it according to the actual situation, and this embodiment does not limit it. The third ratio is recorded as Recorded as the first difference.

[0056] It should be understood that as the acquisition time span increases, the impact of historical data should be relatively small; at the same time, as the data anomaly is more serious and there is more noise affecting the image matching quality, the corresponding data impact should also be smaller, thereby ensuring the quality of the iteratively generated Jacobian matrix. Therefore, the corresponding impact weight is also smaller.

[0057] (2) The historical data step size of each iteration of the population-based Broyden (BP) iterative algorithm is taken as an unknown quantity. The step size adjustment coefficient of the population-based Broyden (BP) iterative algorithm at each acquisition is obtained by combining the matching anomaly coefficient and the influence weight. The objective function is constructed by combining the step size adjustment coefficient of the previous acquisition and the optimization algorithm is used to solve the unknown quantity.

[0058] Furthermore, in order to obtain the optimal historical data step size of the population-based Broyden (BP) iterative algorithm during iteration, this embodiment constructs an objective function and uses an optimization algorithm to solve it. First, the optimal historical data step size is used as an unknown quantity, and the step size adjustment coefficient of the population-based Broyden (BP) iterative algorithm during each acquisition is obtained. The specific calculation method is:

[0059] Where, is the step size adjustment coefficient when using the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the mean value of the matching anomaly coefficient of the real-time images collected before i times, is the historical data step length of the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the influence weight of the nth acquisition before the i-th acquisition of real-time image, where, when i=2, .

[0060] Then, the objective function is constructed as follows: , where For the The objective function when acquiring real-time images is: The step size adjustment coefficient is used when the Broyden iterative algorithm based on the population is used to collect the real-time image for the i-1th time. The optimization algorithm is used to obtain the optimal solution of the objective function, so that the objective function is minimized, and the optimal historical data step size is obtained when the Jacobian matrix is obtained using the Broyden iterative algorithm based on the population for the real-time image collected for the i-th time. The optimal historical data step size determination flow chart is as follows: Figure 3 It should be noted that the optimization algorithm used in this embodiment is the gradient descent method, and the implementer can choose other feasible optimization algorithms, such as Newton's method, conjugate gradient method, etc., which are not limited in this embodiment.

[0061] It should be understood that the more noise the historical data adjacent to the current real-time image acquisition contains, the smaller the corresponding historical data influence weight will be. Therefore, when the historical data step size remains unchanged, the step size adjustment coefficient will also be smaller. In order to minimize the objective function, more historical data is needed, that is, the historical data step size is smaller. It also ensures the quality of the Jacobian matrix obtained by using the Broyden iterative algorithm based on the population.

[0062] Each time the visual servoing manipulator needs to perform motion control, that is, each time a real-time image is captured, taking the i-th time as an example, the historical data step size of the i-th swarm-based Broyden iteration algorithm is obtained, and the historical data is used as the input of the swarm-based Broyden iteration algorithm to estimate the Jacobian matrix. The change in each joint angle of the visual servoing manipulator and the change in the position coordinates of the end effector are obtained for each capture relative to the previous capture. In this embodiment, the input of the i-th swarm-based Broyden iteration algorithm, that is, the historical data, is the change in the joint angle and the change in the position coordinates of the end effector for all times before the i-th capture.

[0063] Step S004 : determining the motion speed of each joint of the visual servoing manipulator based on the Jacobian matrix, and performing motion control on the visual servoing manipulator.

[0064] After obtaining the Jacobian matrix, the dynamic Gauss-Newton method is used to obtain the global optimal solution, that is, the optimal camera motion speed The Broyden iterative algorithm based on the population and the dynamic Gauss-Newton method are both existing well-known technologies, and the specific processes are not described in detail.

[0065] Since the network camera adopts the "eye-in-hand" installation method, the optimal camera movement speed As input to the robot controller, inverse kinematics modeling is used to transform the motion of the robot's endpiece into the motion speed of each joint, thereby achieving motion control of the visual servo robot. Inverse kinematics modeling is a well-known technique in the field of robot control and will not be described in detail here.

[0066] It should be noted that the order in which the embodiments of the present application are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0067] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0068] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Modifications to the technical solutions described in the aforementioned embodiments, or equivalent replacements of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A visual servo robot arm motion control method based on natural features, characterized in that: The method comprises the following steps: The target image is captured in real time using a visual servo robot arm. The first captured image is recorded as the original image, and the target area in the original image is selected. All images except the original image are recorded as real-time images. Perform feature matching on the real-time image and the original image to obtain a set of matching points, and remove abnormal points in the matching point set to obtain an inlier set; classify all feature points in the real-time image that are located in the inlier set based on the grayscale discreteness of the pixels in the neighborhood of each feature point in the real-time image that are located in the inlier set; Analyze the difference between the matching point set and the interior point set in the real-time images collected each time, as well as the difference in the number of feature points of different types, and determine the matching anomaly coefficient of the real-time images collected each time in combination with the grayscale discreteness; The Jacobian matrix of each collected real-time image is obtained using the Broyden iterative algorithm based on the population, specifically: Determine the influence weight of each acquisition on the previous acquisition based on the matching anomaly coefficient of each acquisition and the real-time image of each acquisition, as well as the interval between each acquisition; The historical data step length of each iteration of the iterative algorithm is used as an unknown quantity, and the step length adjustment coefficient of the iterative algorithm at each acquisition is obtained by combining the matching anomaly coefficient and the influence weight. The objective function is constructed by combining the step length adjustment coefficient of the previous acquisition, and the unknown quantity is solved by using an optimization algorithm; The motion speed of each joint of the visual servoing robot arm is determined based on the Jacobian matrix, and the motion of the visual servoing robot arm is controlled.

2. The natural feature-based visual servoing robot arm motion control method according to claim 1, characterized in that: The classifying of all feature points in the interior point set in the real-time image includes: A threshold segmentation algorithm is used to obtain a segmentation threshold of the grayscale discreteness of pixels in the neighborhood of all feature points in the inlier set in the real-time image, and all feature points in the inlier set in the real-time image are divided into two categories based on the segmentation threshold.

3. The natural feature-based visual servoing robot arm motion control method according to claim 2, characterized in that: The process of determining the matching anomaly coefficient is as follows: For each acquired real-time image, calculate the ratio of the cardinality of the matching point set to the cardinality of the inlier set, recorded as the first ratio; calculate the mean of the grayscale dispersion of the pixels in the neighborhood of all feature points located in the inlier set in the real-time image; calculate the proportion of feature points whose grayscale dispersion is less than the segmentation threshold among all feature points located in the inlier set in the real-time image; The matching anomaly coefficient is determined based on the first ratio, the mean, and the proportion, wherein the matching anomaly coefficient is positively correlated with the first ratio and the proportion, and negatively correlated with the mean.

4. The natural feature-based visual servoing robot arm motion control method according to claim 3, characterized in that: The ratio of the proportion to the mean is calculated and recorded as a second ratio. The matching anomaly coefficient is obtained by combining the first ratio.

5. The natural feature-based visual servoing robot arm motion control method according to claim 4, characterized in that: The matching anomaly coefficient is the product of the first ratio and the second ratio.

6. The natural feature-based visual servoing robot arm motion control method according to claim 1, characterized in that: The process of determining the impact weight is as follows: Calculate the sum of the matching anomaly coefficients of the real-time images acquired in each acquisition and all previous acquisitions, calculate the ratio of the matching anomaly coefficient of the real-time image acquired in any acquisition before each acquisition to the sum, record it as a third ratio, and calculate the difference between the sequence number of each acquisition and the sequence number of any acquisition; The difference is used as the exponent of an exponential function with a preset time decay factor as the base, and the influence weight of any previous acquisition in each acquisition is determined by combining the third ratio and the calculation result of the exponential function.

7. The natural feature-based visual servoing robot arm motion control method according to claim 6, characterized in that: The difference between 1 and the third ratio is calculated and recorded as the first difference. The influence weight of any previous collection in each collection is the product of the first difference and the calculation result of the exponential function.

8. The natural feature-based visual servoing robot arm motion control method according to claim 1, wherein: The expression of the step size adjustment coefficient is: Where, is the step size adjustment coefficient when using the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the mean value of the matching anomaly coefficient of the real-time images collected before i times, is the historical data step length of the Broyden iterative algorithm based on population when collecting real-time images for the i-th time, is the influence weight of the nth acquisition before the i-th acquisition of real-time image, where, when i=2, .

9. The natural feature-based visual servoing robot arm motion control method according to claim 1, wherein: The expression of the objective function is Where, For the The objective function when acquiring real-time images is: is the step size adjustment coefficient when using the population-based Broyden iterative algorithm for the i-1th real-time image acquisition.

10. The natural feature-based visual servoing robot arm motion control method according to claim 1, wherein: The objective function is minimized when solving the unknown quantity.

Images