Visual servo control method and device for robot
By extracting and matching real-time contour feature point sets and calculating control speed, high-precision robot servo control was achieved in complex environments such as lighting changes and object occlusion, solving the problems of insufficient robustness and accuracy of traditional visual servoing methods.
Patent Information
- Application Number
- CN202610219623.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-14
- Publication Date
- 2026-05-19
Smart Images
Figure CN122058356A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and more specifically, to a visual servo control method and apparatus for robots. Background Technology
[0002] Visual servoing control is a key technology in the field of robotics automation, allowing robots to adjust their movements based on visual feedback to achieve specified task objectives. The core of traditional visual servoing methods lies in using image feature points captured by visual sensors as control references. However, traditional visual servoing methods have limitations when encountering non-ideal conditions, such as changes in lighting intensity, partial occlusion of the target object, sensor noise, or drastic changes in viewing angle. When these factors affect image quality, fixed feature points become difficult to accurately identify, prone to mismatch, or even lost, leading to a decline in visual servoing control performance and limiting its application in complex industrial environments.
[0003] In traditional technologies, fixed feature points can become unreliable in real-world industrial environments due to variations in lighting conditions, especially under extreme or unstable conditions. These feature points may disappear completely or introduce significant errors. Sensor noise further degrades the feature point recognition and matching process, leading to the accumulation of errors in servo control and reducing the overall system performance. Furthermore, for targets with little or uniform texture, fixed feature point-based methods may simply be unable to find a sufficient number of feature points for matching and control, thus limiting the application of visual servoing technology in a wider range of scenarios.
[0004] There is currently no effective solution to the above problems. Summary of the Invention
[0005] This invention provides a visual servo control method and apparatus for robots, which at least solves the technical problems of insufficient robustness and accuracy of traditional visual servo control methods in complex environments such as changes in lighting and object occlusion.
[0006] According to one aspect of the present invention, a visual servo control method for a robot is provided, comprising: acquiring a real-time target image captured by a camera of a target robot in a current pose, wherein the camera is mounted on the target robot; extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes a plurality of first feature points of the contour of the target object in the real-time target image; filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs, wherein the desired contour feature point set includes a plurality of second feature points of the contour of the target object in a desired target image captured by the camera of the target robot in a desired pose, and the set of matching point pairs includes a plurality of feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set; calculating a control speed based on the set of matching point pairs; controlling the target robot to move for a preset time based on the control speed and then stopping; repeating the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
[0007] Optionally, extracting a real-time contour feature point set from the real-time target image includes: denoising the real-time target image to obtain a denoised real-time target image, wherein the denoising process includes grayscale conversion and Gaussian filtering; extracting the real-time target contour from the denoised real-time target image using an edge detection algorithm; and uniformly sampling the real-time target contour based on a preset sampling interval to obtain a real-time contour feature point set.
[0008] Optionally, the steps for determining the desired contour feature point set include: acquiring an image of the desired target object captured by a camera in the desired pose of the target robot; performing denoising processing on the desired target object image to obtain a denoised desired target object image, wherein the denoising processing includes grayscale conversion and Gaussian filtering; extracting the desired target object contour from the denoised desired target object image using an edge detection algorithm; and performing uniform sampling on the desired target object contour based on a preset sampling interval to obtain the desired contour feature point set.
[0009] Optionally, based on a preset set of desired contour feature points, a set of matching point pairs is determined by filtering the set of real-time contour feature points. This includes: mapping multiple first feature points to multiple second feature points one-to-one to obtain multiple initial point pairs; calculating the similarity between two feature points in the multiple initial point pairs based on the feature information of the first and second feature points respectively, to obtain multiple similarities, wherein the feature information includes grayscale distribution, gradient direction, and geometric context information; sorting the corresponding multiple initial point pairs according to the order of multiple similarities from largest to smallest to obtain the initial point pair sorting result; retaining the first preset proportion of initial point pairs in the initial point pair sorting result to obtain multiple feature point pairs, which serve as the set of matching point pairs.
[0010] Optionally, the control speed is calculated based on the set of matching point pairs, including: calculating the error between two feature points in multiple feature point pairs to obtain multiple feature errors, forming an image error vector; calculating the image Jacobian matrix corresponding to multiple feature point pairs; and calculating the control speed based on the image error vector and the image Jacobian matrix.
[0011] Optionally, the image Jacobian matrix corresponding to multiple feature point pairs is calculated, including: determining all feature points in the multiple feature point pairs as multiple image feature points; for each image feature point, calculating a row vector of the image Jacobian matrix based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature point, to obtain multiple row vectors; and constructing the image Jacobian matrix based on the multiple row vectors.
[0012] According to another aspect of the present invention, a visual servo control device for a robot is also provided, comprising: an acquisition module for acquiring a real-time target image captured by a camera of a target robot in a current pose, wherein the camera is mounted on the target robot; an extraction module for extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target object in the real-time target image; a filtering module for filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs, wherein the desired contour feature point set includes multiple second feature points of the contour of the target object in a desired target image captured by a camera of the target robot in a desired pose, and the set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set; a calculation module for calculating a control speed based on the set of matching point pairs; a control module for controlling the target robot to move for a preset time based on the control speed and then stopping; and a repetition module for repeating the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
[0013] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, the device where the non-volatile storage medium is located is controlled to execute any of the above-described visual servo control methods for robots.
[0014] According to another aspect of the present invention, a computer device is also provided, the computer device including a processor, the processor being configured to run a program, wherein the program, when running, executes any of the above-described visual servo control methods for a robot.
[0015] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described visual servo control methods for robots.
[0016] In this embodiment of the invention, a visual servo control method for robots is employed. This involves acquiring a real-time target image captured by a camera on the target robot in its current pose; extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the target object's contour in the real-time target image; and filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs. The desired contour feature point set includes multiple second feature points of the target object's contour in the desired target image captured by the camera in the desired pose of the target robot, and the set of matching point pairs includes real... Multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set are identified; based on the set of matching point pairs, the control speed is calculated; based on the control speed, the target robot is controlled to move for a preset time and then stopped; the above steps are repeated until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thus completing the servo control of the target robot. This achieves the goal of guiding the robot to the desired pose in complex environments, thereby realizing the technical effect of ensuring high precision and stability of robot operation under various adverse conditions. It also solves the technical problem of insufficient robustness and precision of traditional visual servo control methods in complex environments such as changes in lighting and object occlusion. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:
[0018] Figure 1 A hardware block diagram of a computer terminal for implementing a visual servo control method for robots is shown.
[0019] Figure 2 This is a flowchart illustrating a visual servo control method for a robot according to an embodiment of the present invention.
[0020] Figure 3 This is a structural block diagram of a visual servo control device for a robot provided according to an embodiment of the present invention. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0023] According to an embodiment of the present invention, a visual servo control method for a robot is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0024] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a visual servo control method for robots is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0025] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0026] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the visual servo control method for robots in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned application program for the visual servo control method for robots. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0027] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.
[0028] Figure 2 This is a flowchart illustrating a visual servo control method for a robot according to an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:
[0029] Step S201: Obtain a real-time image of the target object captured by the camera in the current pose of the target robot, wherein the camera is set on the target robot.
[0030] In this step, the target robot refers to an industrial robot equipped with a visual servo system, while the camera can be any imaging device suitable for robot vision applications, such as an RGB camera or a depth camera. Its mounting method is typically "eye-to-hand" or "eye-in-hand." In this embodiment, the camera adopts an "eye-to-hand" mounting method. During the task execution, the target robot needs to use the camera fixed to the robot to capture images of the target object in real time.
[0031] Specifically, to achieve visual servo control, the robot must update its visual perception in each control loop, that is, acquire an image of the target object at the current moment through the camera. This process is continuous and dynamic. In industrial applications, factors such as changes in lighting, occlusion of the target object, or changes in the robot's perspective may exist, all of which will affect the extraction of feature points and the subsequent matching process. Therefore, the real-time performance and reliability of the camera are crucial for the visual servo system to maintain high robustness and control accuracy.
[0032] Step S202: Extract a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target in the real-time target image.
[0033] In this step, noise is processed based on the real-time image of the target object acquired under the current pose of the target robot. An edge detection algorithm (such as the Canny algorithm) is used to extract the contour information of the target object, and contour feature points are sampled on this basis to form a dense set of two-dimensional feature coordinates, namely the real-time contour feature point set. The real-time contour feature point set consists of multiple first feature points. These first feature points not only represent the boundary shape of the target object, but also provide the necessary data foundation for the visual servo system to perform real-time matching and control.
[0034] Step S203: Based on the preset expected contour feature point set, filter the real-time contour feature point set to determine the matching point pair set. The expected contour feature point set includes multiple second feature points of the contour of the target object in the expected target object image captured by the camera under the expected pose of the target robot. The matching point pair set includes multiple feature point pairs that match each other in the real-time contour feature point set and the expected contour feature point set.
[0035] In this step, firstly, when the target robot is in the desired pose, an image of the desired target object is captured. A desired contour feature point set is extracted from this image, containing multiple second feature points reflecting the target object's contour, serving as a standard reference for visual servo control. Subsequently, the real-time contour feature point set obtained from the captured target object image is meticulously compared with the aforementioned standard reference. This process automatically selects first and second feature point pairs with similarity exceeding a preset threshold by calculating the similarity (or matching score) between each pair of first and second feature points and combining this with the local appearance and geometric context information of the feature points. This forms a set of matching point pairs. This dynamic matching and high-score selection strategy ensures that even when the target object's surface texture changes, lighting conditions are unstable, or occlusion exists, the most stable and reliable feature point pairs can still be selected from the real-time contour feature point set for subsequent servo control calculations.
[0036] Step S204: Calculate the control speed based on the set of matching point pairs.
[0037] In this step, an interaction matrix, namely the image Jacobian matrix, is constructed to correlate the image spatial changes of feature points with the actual physical spatial movement of the robot's end effector. The construction of the interaction matrix L is based on information from the set of matching point pairs, including but not limited to the point's position, depth (or distance), camera intrinsic parameters, and hand-eye matrix. This information together constitutes a mapping used to characterize the relationship between the image error e and the desired robot motion speed v.
[0038] To solve for the control velocity *v*, an image-based feature servo control law is employed. This law involves least squares, gradient descent, or other optimization algorithms, aiming to find a velocity vector that enables the robot to move along an optimal path in the direction that minimizes image errors. Based on the calculated image Jacobian matrix and image errors, the control velocity can be obtained using the calculation methods in the feature servo control law.
[0039] Step S205: Based on the control speed, control the target robot to move for a preset time and then stop.
[0040] In this step, the calculated control speed command is mapped to the physical control speed of the robot's end effector through hand-eye matrix conversion. This physical control speed guides the robot to move at a pre-calculated speed, thereby adjusting its pose in physical space to achieve a target state corresponding to the desired contour feature point set. In actual operation, the robot moves a specific distance at the controlled speed according to a preset time, adjusting its pose. The selection of the preset time considers system response time, control accuracy requirements, and the robot's motion characteristics, ensuring accurate execution of speed commands while adapting to rapidly changing environments, and avoiding problems such as discontinuous control or excessive oscillations due to excessively short time intervals.
[0041] Step S206: Repeat the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thus completing the servo control of the target robot.
[0042] In this step, after each control loop, the deviation between the real-time contour feature point set and the desired contour feature point set is checked again to see if it meets the stopping condition. If the current error is still greater than the preset threshold, it indicates that the target robot's pose adjustment has not yet reached the desired state, and the system will automatically return to the real-time image acquisition step to restart a closed-loop control process. The preset threshold is set based on the specific servo task requirements and the characteristics of the target robot, reflecting both a certain positioning accuracy requirement and a balance between system stability and efficiency. When the error between the real-time contour feature point set and the desired contour feature point set is less than this threshold, it means that the target robot has reached the desired pose state, and the servo control has reached the expected accuracy level.
[0043] Through the above steps, the goal of guiding the robot to the desired pose in complex environments is achieved, thereby realizing the technical effect of ensuring high precision and stability of robot operation under various adverse conditions. This solves the technical problem of insufficient robustness and precision of traditional visual servo control methods in complex environments such as changes in lighting and object occlusion.
[0044] As an optional embodiment, extracting a real-time contour feature point set from a real-time target image includes: denoising the real-time target image to obtain a denoised real-time target image, wherein the denoising process includes grayscale conversion and Gaussian filtering; extracting the real-time target contour from the denoised real-time target image using an edge detection algorithm; and uniformly sampling the real-time target contour based on a preset sampling interval to obtain a real-time contour feature point set.
[0045] Optionally, firstly, the real-time target image is denoised by converting it to grayscale and applying Gaussian filtering to reduce random noise and texture details in the original image, thus enhancing the performance of the edge detection algorithm. Grayscale conversion simplifies color information, allowing the algorithm to focus on structure and contour recognition; while Gaussian filtering smooths the image, reducing high-frequency noise interference and providing a clearer and more stable background for subsequent feature extraction. Secondly, the Canny edge detection algorithm is used to extract the target contour in the denoised real-time target image, and the Rosenfeld skeleton thinning algorithm is used to optimize the contour lines. Uniform sampling is performed along the real-time target contour at a certain sampling interval to generate a series of dense first feature points, forming the real-time contour feature point set P_c. The selection of the sampling interval needs to balance accuracy and computational efficiency, ensuring that the feature points cover the contour sufficiently and uniformly while avoiding redundant calculations caused by too many feature points, thus achieving a good balance between real-time performance and robustness.
[0046] As an optional embodiment, the steps for determining the desired contour feature point set include: acquiring an image of the desired target object captured by a camera in the desired pose of the target robot; performing denoising processing on the desired target object image to obtain a denoised desired target object image, wherein the denoising processing includes grayscale conversion and Gaussian filtering; extracting the desired target object contour from the denoised desired target object image using an edge detection algorithm; and performing uniform sampling on the desired target object contour based on a preset sampling interval to obtain the desired contour feature point set.
[0047] Optionally, firstly, an image of the desired target object captured by the camera in the desired pose of the target robot is acquired. This image represents the final visual state that the system expects to achieve. Image denoising is achieved by converting the image to grayscale and applying Gaussian filtering, reducing color variations and local noise to obtain a clearer and smoother image. Secondly, an edge detection algorithm is used to calculate the gradient intensity of pixels in the denoised image, filtering out edge pixels and identifying and extracting the contour lines of the desired target object. Finally, uniform sampling is performed on the extracted contour of the desired target object at a preset sampling interval to generate a series of second feature points that represent the shape and edge information of the target object. These feature points constitute the desired contour feature point set P_d. The setting of the sampling interval must balance the integrity of the contour information and computational efficiency, ensuring that the extracted feature point set not only fully describes the contour of the target object but also effectively reduces computational load and improves the overall response speed of the system.
[0048] As an optional embodiment, based on a preset expected contour feature point set, a set of matching point pairs is determined by filtering the real-time contour feature point set, including: mapping multiple first feature points to multiple second feature points one-to-one to obtain multiple initial point pairs; calculating the similarity between two feature points in the multiple initial point pairs based on the feature information of the first and second feature points respectively, to obtain multiple similarities, wherein the feature information includes grayscale distribution, gradient direction and geometric context information; sorting the corresponding multiple initial point pairs according to the order of multiple similarities from largest to smallest to obtain the initial point pair sorting result; retaining the first preset proportion of initial point pairs in the initial point pair sorting result to obtain multiple feature point pairs as the matching point pair set.
[0049] Optionally, based on the real-time contour feature point set and the desired contour feature point set, corresponding first and second feature points in the two sets are selected for matching to obtain multiple initial point pairs. Subsequently, the similarity of each initial point pair is calculated based on feature information such as the grayscale distribution, gradient direction, and geometric context information of the first and second feature points. By comparing the above feature information, it can be determined whether the feature points in the real-time image match the feature points in the desired image; a higher similarity indicates a closer correspondence in visual features. Furthermore, the calculated similarity is used as a sorting criterion, and all initial point pairs are arranged in descending order of similarity.
[0050] Based on the sorting results, a certain proportion of initial point pairs with the highest similarity are selected to form a matching point pair set. The preset proportion is set to ensure the high quality and reliability of the selected feature points in servo control, while also providing sufficient redundancy for the algorithm to handle potential lighting changes, target object occlusion, and other situations. In this optional embodiment, the initial point pairs with the highest similarity are selected to form the matching point pair set M_high.
[0051] As an optional embodiment, the control speed is calculated based on the set of matching point pairs, including: calculating the error between two feature points in multiple feature point pairs to obtain multiple feature errors, forming an image error vector; calculating the image Jacobian matrix corresponding to multiple feature point pairs; and calculating the control speed based on the image error vector and the image Jacobian matrix.
[0052] Optionally, firstly, an error assessment is performed on the set of matching point pairs. The difference between the real-time contour feature points and the desired contour feature points in each of the multiple feature point pairs is calculated. This difference is reflected in the difference in coordinates between the two feature points in the image coordinate system, thus obtaining a series of feature errors describing the contour deviation. These feature errors constitute the image error vector, used to reflect the overall deviation between the current image features and the target features.
[0053] Secondly, for the set of matching point pairs, an image Jacobian matrix is constructed using image information and robot geometric parameters. Specifically, for each pair of matching feature points, the row vector corresponding to each feature point is calculated based on their image position and estimated depth information, combined with the camera's intrinsic parameters and the hand-eye matrix. Then, the multiple row vectors of all matching point pairs are arranged or combined into a matrix, which is the image Jacobian matrix of the entire image.
[0054] Finally, based on the image error vector and the image Jacobian matrix, an appropriate control law, such as the least squares method or gradient descent method, is used to calculate the control speed that the robot's end effector should possess. The formula for calculating the control speed is as follows:
[0055]
[0056] Where λ is the positive constant gain, Let L be the pseudo-inverse, and e be the image error between the current feature and the desired feature.
[0057] As an optional embodiment, calculating the image Jacobian matrix corresponding to multiple feature point pairs includes: determining all feature points in multiple feature point pairs as multiple image feature points; for each image feature point, calculating a row vector of the image Jacobian matrix based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature point, thus obtaining multiple row vectors; and constructing the image Jacobian matrix based on the multiple row vectors.
[0058] Optionally, based on the selected set of matching point pairs, each pair of feature points is identified and marked, and the first feature point in the real-time contour and the second feature point in the desired contour are summarized into the image feature point list. For each image feature point, a row vector of the image Jacobian matrix is calculated based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature point, resulting in multiple row vectors. Here, the camera's intrinsic parameters refer to inherent optical characteristics such as lens focal length and optical center coordinates, while the hand-eye matrix describes the transformation relationship between the camera's spatial position and orientation relative to the robot's end effector. These parameters are combined with the depth information of each feature point to calculate the row vector of each image feature point. Finally, based on the row vectors corresponding to all image feature points, they are vertically concatenated according to certain rules to form a matrix, namely the image Jacobian matrix. This matrix is used to characterize the direct mapping relationship between image spatial error and robot motion spatial velocity.
[0059] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.
[0060] Through the above description of the embodiments, those skilled in the art can clearly understand that the visual servo control method for robots according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0061] According to embodiments of the present invention, an apparatus for implementing the above-described visual servo control method for robots is also provided. Figure 3 This is a structural block diagram of a visual servo control device for a robot according to an embodiment of the present invention, such as... Figure 3 As shown, the device includes: an acquisition module 31, an extraction module 32, a filtering module 33, a calculation module 34, a control module 35, and a repeating module 36. The device will be described below.
[0062] The acquisition module 31 is used to acquire real-time target object images captured by the camera of the target robot in its current pose, wherein the camera is set on the target robot.
[0063] The extraction module 32, connected to the acquisition module 31, is used to extract a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target in the real-time target image.
[0064] The filtering module 33, connected to the extraction module 32, is used to filter the real-time contour feature point set based on the preset expected contour feature point set to determine the matching point pair set. The expected contour feature point set includes multiple second feature points of the contour of the target object in the image of the target object captured by the camera under the expected pose of the target robot. The matching point pair set includes multiple feature point pairs that match each other in the real-time contour feature point set and the expected contour feature point set.
[0065] The calculation module 34, connected to the filtering module 33, is used to calculate the control speed based on the set of matching point pairs.
[0066] The control module 35, connected to the calculation module 34, is used to control the target robot to stop after a preset time based on the control speed.
[0067] The repeat module 36, connected to the control module 35, is used to repeat the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
[0068] It should be noted that the acquisition module 31, extraction module 32, filtering module 33, calculation module 34, control module 35, and repetition module 36 mentioned above correspond to steps S201 to S206 in the embodiments. Multiple modules implement the same instances and application scenarios as their corresponding steps, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in the embodiments.
[0069] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.
[0070] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the visual servo control method and device for robots in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned visual servo control method for robots. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0071] The processor can access information and application programs stored in the memory via a transmission device to execute the following steps: acquiring a real-time target image captured by a camera on the target robot in its current pose; extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the target object's contour in the real-time target image; filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs, wherein the desired contour feature point set includes multiple second feature points of the target object's contour in the desired target image captured by the camera in the desired pose of the target robot, and the set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set; calculating the control speed based on the set of matching point pairs; controlling the target robot to move for a preset time based on the control speed and then stopping; repeating the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thus completing the servo control of the target robot.
[0072] Optionally, the processor may also execute program code for the following steps: extracting a real-time contour feature point set from a real-time target image, including: denoising the real-time target image to obtain a denoised real-time target image, wherein the denoising process includes grayscale conversion and Gaussian filtering; extracting the real-time target contour from the denoised real-time target image using an edge detection algorithm; and uniformly sampling the real-time target contour based on a preset sampling interval to obtain a real-time contour feature point set.
[0073] Optionally, the processor may also execute program code for the following steps: the step of determining the desired contour feature point set includes: acquiring the desired target image captured by the camera of the target robot in the desired pose; performing denoising processing on the desired target image to obtain a denoised desired target image, wherein the denoising processing includes grayscale conversion and Gaussian filtering; extracting the desired target contour in the denoised desired target image using an edge detection algorithm; and performing uniform sampling on the desired target contour based on a preset sampling interval to obtain the desired contour feature point set.
[0074] Optionally, the processor may also execute program code for the following steps: based on a preset expected contour feature point set, filtering is performed in the real-time contour feature point set to determine a set of matching point pairs, including: mapping multiple first feature points to multiple second feature points one-to-one to obtain multiple initial point pairs; based on the feature information of the first and second feature points respectively, the similarity between two feature points in the multiple initial point pairs is calculated to obtain multiple similarities, wherein the feature information includes grayscale distribution, gradient direction and geometric context information; the corresponding multiple initial point pairs are sorted in descending order of multiple similarities to obtain an initial point pair sorting result; the first preset proportion of initial point pairs in the initial point pair sorting result are retained to obtain multiple feature point pairs as a set of matching point pairs.
[0075] Optionally, the processor may also execute program code that performs the following steps: calculating the control speed based on the set of matching point pairs, including: calculating the error between two feature points in multiple feature point pairs respectively to obtain multiple feature errors and forming an image error vector; calculating the image Jacobian matrix corresponding to multiple feature point pairs; and calculating the control speed based on the image error vector and the image Jacobian matrix.
[0076] Optionally, the processor may also execute program code that performs the following steps: calculating the image Jacobian matrix corresponding to multiple feature point pairs, including: determining all feature points in the multiple feature point pairs as multiple image feature points; for each image feature point, calculating a row vector of the image Jacobian matrix based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature points, to obtain multiple row vectors; and constructing the image Jacobian matrix based on the multiple row vectors.
[0077] The present invention provides a method for visual servo control of a robot. By acquiring real-time target object images captured by a camera on the target robot in its current pose, the system extracts a real-time contour feature point set from the real-time target object image. This set includes multiple first feature points representing the contour of the target object in the real-time target object image. Based on a preset desired contour feature point set, the system filters the real-time contour feature point set to determine a set of matching point pairs. The desired contour feature point set includes multiple second feature points representing the contour of the target object in the desired target object image captured by the camera in the desired pose. The set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set. Based on the set of matching point pairs, the system calculates a control speed. Based on the control speed, the system controls the target robot to move for a preset time and then stops. The above steps are repeated until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thus completing the servo control of the target robot. This achieves the goal of guiding the robot to the desired pose in complex environments, thereby solving the technical problems of insufficient robustness and accuracy of traditional visual servo control methods in complex environments such as lighting changes and object occlusion.
[0078] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0079] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the visual servo control method for robots provided in the above embodiments.
[0080] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.
[0081] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring a real-time target image captured by a camera on the target robot in its current pose, wherein the camera is mounted on the target robot; extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the target object's contour in the real-time target image; filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs, wherein the desired contour feature point set includes multiple second feature points of the target object's contour in the desired target image captured by the camera in the desired pose of the target robot, and the set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set; calculating a control speed based on the set of matching point pairs; controlling the target robot to move for a preset time based on the control speed and then stopping; repeating the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
[0082] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: extracting a real-time contour feature point set from a real-time target image, including: denoising the real-time target image to obtain a denoised real-time target image, wherein the denoising process includes grayscale conversion and Gaussian filtering; extracting the real-time target contour from the denoised real-time target image using an edge detection algorithm; and uniformly sampling the real-time target contour based on a preset sampling interval to obtain a real-time contour feature point set.
[0083] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: the step of determining the desired contour feature point set includes: acquiring a desired target image captured by a camera of the target robot in a desired pose; performing denoising processing on the desired target image to obtain a denoised desired target image, wherein the denoising processing includes grayscale conversion and Gaussian filtering; extracting the desired target contour in the denoised desired target image using an edge detection algorithm; and performing uniform sampling on the desired target contour based on a preset sampling interval to obtain the desired contour feature point set.
[0084] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: based on a preset expected contour feature point set, filtering is performed in the real-time contour feature point set to determine a set of matching point pairs, including: mapping multiple first feature points to multiple second feature points one-to-one to obtain multiple initial point pairs; based on the feature information of the first feature points and the second feature points respectively, the similarity between two feature points in the multiple initial point pairs is calculated to obtain multiple similarities, wherein the feature information includes grayscale distribution, gradient direction and geometric context information; the multiple initial point pairs are sorted in descending order of multiple similarities to obtain an initial point pair sorting result; the first preset proportion of initial point pairs in the initial point pair sorting result are retained to obtain multiple feature point pairs as a set of matching point pairs.
[0085] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: calculating control speed based on the set of matching point pairs, including: calculating the error between two feature points in multiple feature point pairs respectively to obtain multiple feature errors, forming an image error vector; calculating the image Jacobian matrix corresponding to multiple feature point pairs; and calculating control speed based on the image error vector and the image Jacobian matrix.
[0086] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: calculating the image Jacobian matrix corresponding to multiple feature point pairs, including: determining all feature points in the multiple feature point pairs as multiple image feature points; for each image feature point, calculating a row vector of the image Jacobian matrix based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature point, to obtain multiple row vectors; and constructing the image Jacobian matrix based on the multiple row vectors.
[0087] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can perform the following: acquiring a real-time target image captured by a camera on the target robot in its current pose, wherein the camera is mounted on the target robot; extracting a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target object in the real-time target image; filtering the real-time contour feature point set based on a preset desired contour feature point set to determine a set of matching point pairs, wherein the desired contour feature point set includes multiple second feature points of the contour of the target object in the desired target image captured by the camera in the desired pose of the target robot, and the set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the desired contour feature point set; calculating a control speed based on the set of matching point pairs; controlling the target robot to move for a preset time based on the control speed and then stopping; repeating the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
[0088] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0089] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0091] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0092] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0094] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A visual servo control method for robots, characterized in that, include: Acquire real-time target object images captured by a camera on the target robot in its current pose; A real-time contour feature point set is extracted from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target in the real-time target image; Based on a preset set of desired contour feature points, the set of real-time contour feature points is filtered to determine a set of matching point pairs. The set of desired contour feature points includes multiple second feature points of the contour of the target object in the image of the target object captured by the camera in the desired pose of the target robot. The set of matching point pairs includes multiple feature point pairs that match each other in the set of real-time contour feature points and the set of desired contour feature points. Calculate the control speed based on the set of matching point pairs; Based on the control speed, the target robot stops moving after a preset time. Repeat the above steps until the error between the real-time contour feature point set and the desired contour feature point set is less than a preset threshold, thus completing the servo control of the target robot.
2. The method according to claim 1, characterized in that, The step of extracting a real-time contour feature point set from the real-time target image includes: The real-time target image is denoised to obtain a denoised real-time target image, wherein the denoising process includes grayscale conversion and Gaussian filtering. The real-time target object contour is extracted from the denoised real-time target object image using an edge detection algorithm. Based on a preset sampling interval, uniform sampling is performed on the real-time target object contour to obtain the real-time contour feature point set.
3. The method according to claim 1, characterized in that, The steps for determining the desired contour feature point set include: Acquire the image of the desired target object captured by the camera of the target robot in the desired pose; The desired target image is denoised to obtain a denoised desired target image, wherein the denoising process includes grayscale conversion and Gaussian filtering; The outline of the desired target object is extracted from the denoised image of the desired target object using an edge detection algorithm. Based on a preset sampling interval, uniform sampling is performed on the desired target object contour to obtain the desired contour feature point set.
4. The method according to claim 1, characterized in that, The process of filtering the real-time contour feature point set based on a preset expected contour feature point set to determine a set of matching point pairs includes: By assigning each of the plurality of first feature points to one of the plurality of second feature points, a plurality of initial point pairs are obtained; Based on the feature information of the first feature point and the second feature point respectively, the similarity between two feature points in the plurality of initial point pairs is calculated to obtain multiple similarity values. The feature information includes gray-level distribution, gradient direction and geometric context information. The initial point pairs are sorted according to the multiple similarities from largest to smallest to obtain the initial point pair sorting result; The initial point pairs that are retained in the initial point pair sorting results are a predetermined proportion, and the plurality of feature point pairs are obtained as the matching point pair set.
5. The method according to claim 1, characterized in that, The calculation of the control speed based on the set of matching point pairs includes: The error between two feature points in each of the multiple feature point pairs is calculated to obtain multiple feature errors, which are then combined to form an image error vector. Calculate the image Jacobian matrix corresponding to the plurality of feature point pairs; The control speed is calculated based on the image error vector and the image Jacobian matrix.
6. The method according to claim 5, characterized in that, The calculation of the image Jacobian matrix corresponding to the plurality of feature point pairs includes: All feature points in the plurality of feature point pairs are identified as multiple image feature points; For each image feature point, based on the camera's intrinsic parameters, the target robot's hand-eye matrix, and the depth information of the image feature point, a row vector of the image Jacobian matrix is calculated to obtain multiple row vectors; The image Jacobian matrix is constructed based on the plurality of row vectors.
7. A visual servo control device for a robot, characterized in that, include: An acquisition module is used to acquire real-time images of the target object captured by a camera in the current pose of the target robot, wherein the camera is mounted on the target robot; An extraction module is used to extract a real-time contour feature point set from the real-time target image, wherein the real-time contour feature point set includes multiple first feature points of the contour of the target in the real-time target image; The filtering module is used to filter the real-time contour feature point set based on a preset expected contour feature point set to determine a set of matching point pairs. The expected contour feature point set includes multiple second feature points of the contour of the target object in the expected target object image captured by the camera under the expected pose of the target robot. The set of matching point pairs includes multiple feature point pairs that match each other in the real-time contour feature point set and the expected contour feature point set. The calculation module is used to calculate the control speed based on the set of matching point pairs; The control module is used to control the target robot to stop moving after a preset time based on the control speed; The repeat module is used to repeat the above steps until the error between the real-time contour feature point set and the expected contour feature point set is less than a preset threshold, thereby completing the servo control of the target robot.
8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, the device containing the non-volatile storage medium is controlled to perform the visual servo control method for a robot as described in any one of claims 1 to 6.
9. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the visual servo control method for a robot as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the visual servo control method for a robot as described in any one of claims 1 to 6.