Large-size rotating ring-oriented array type multi-view camera uncalibrated visual servo method

Abstract

The invention discloses an array type multi-view camera uncalibrated visual servo method for a large-size rotating ring. The method is characterized by comprising the following steps: 1) manually controlling a robot to acquire an expected image at a reference pose; 2) selecting a proper sampling method to plan a task working space of the robot to obtain a tail end movement speed # imgabs0 # of the robot in a movement process and an image moment set # imgabs1 # of a corresponding feedback image; 3) training a neural network model by using the collected data to obtain a trained robot tail end movement prediction model; 4) calculating an image moment set delta m of the feedback image, inputting the image moment set delta m into the robot end motion prediction model, and outputting a robot end six-degree-of-freedom motion speed ve; and 5) sending the obtained robot end speed to a robot joint controller to drive the robot to move so as to achieve the purpose of visual servo target tracking. According to the method, multi-view fusion visual servo operation for the large-size rotating ring can be effectively realized, and the method has relatively high operation precision and robustness.

Description

Technical Field

[0001] The invention relates to a servo control technology, in particular to a robot visual servo control technology, specifically to an array-type multi-eye camera non-calibrated visual servo (tracking) method for a large-size rotating ring. Background Art

[0002] In the production process of aerospace products, large-sized rotating rings are often assembled and docked, and the clearance is small and the fitting time is short. The existing domestic rotating ring docking is mainly done by operators manually, mainly relying on the assembly experience of the staff. The assembly method is inefficient and prone to collisions and scratches. It is urgent to propose a robot visual servo method for the position and posture of large-sized rotating rings.

[0003] With the continuous development of visual servo technology in recent years, many scholars have conducted extensive research on uncalibrated visual servoing, but most of them focus on the visual servoing of monocular cameras. When facing large-size rotating rings, monocular cameras are difficult to meet the requirements of large field of view and high precision at the same time. To address the above problems, this paper proposes an array camera visual servoing method for large-size rotating rings. Summary of the invention

[0004] The present invention aims to solve the problem that the monocular visual servoing target imaging field is small and the long-distance imaging target is unclear when facing a large-sized target, and an array multi-camera uncalibrated visual servoing method for a large-sized rotating ring is invented. The present invention can effectively realize the multi-view fusion visual servoing operation for a large-sized rotating ring, and has high operation accuracy and robustness.

[0005] The technical solution of the present invention is:

[0006] An array multi-camera calibration-free visual servoing method for a large-size rotating ring is characterized in that it includes the following steps:

[0007] S1: Manually control the robot to the reference position to collect the desired image;

[0008] S2: Select an appropriate sampling method to plan the robot's task workspace and obtain the robot's terminal motion speed during the motion process And the corresponding image moment set of the feedback image

[0009] S3: Use the collected data to train the neural network model to obtain a trained robot terminal motion prediction model;

[0010] S4: Calculate the image moment set Δm of the feedback image and input it into the robot terminal motion prediction model, and output the robot terminal six-degree-of-freedom motion speed v e;

[0011] S5: The obtained robot terminal velocity is sent to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking.

[0012] The step S2 selects a suitable sampling method to plan the robot's task workspace and obtain the end motion speed of the robot during the motion process. And the corresponding image moment set of the feedback image The following steps are involved:

[0013] S2.1: According to the relative position relationship between the camera field of view and the robot end, the size of 30×30×50mm is selected 3 , a rectangular parallelepiped area with attitude angles of 10°, 4° and 3° is used as the sampling space. According to the LHS method, 20 position coordinates and 15 attitude angles are randomly generated in the sampling space, and then they are arranged and combined to obtain the expected attitude of 300 sampling points;

[0014] S2.2: Manually control the robot to the reference position to collect the desired image and extract the image moment set from it Represents the image moment feature of the reference image, which is a constant. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order geometric moment feature is: m ij (t) = ∫∫ R(t) x i y i f(x,y)dxdy, f(x,y) is the image point p x = gray value of (x, y), for p x ∈R(t), then f(x,y)=1, otherwise f(x,y)=0. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order central moment feature is: μ ij (t) = ∫∫ R(t) (xx g ) i (yy g ) i dxdy,(x g ,y g ) represents the coordinates of the center point of the image, x g =m 10 / m 00 ,y g =m 01 / m 00 .

[0015] S2.3: Move from a certain sampling point to the reference position. During this process, 10 clear feedback images and their corresponding six-freedom motion speeds of the robot end are randomly collected. Extract image moment sets from the above 10 feedback images And calculate

[0016] S2.4: Finally, the input and output pair.

[0017] The step S3 uses the collected data to train the neural network model to obtain a trained robot terminal motion prediction model, which includes the following steps:

[0018] S3.1: Obtain 3000 sets of training samples according to the method in S2, and use 100, 200, ..., 2900, and 3000 sets of samples to train the neural network respectively, study the effect of the number of samples on the prediction bias of the neural network, and obtain the optimal number of training samples;

[0019] S3.2: Divide the data set into training set and test set in a ratio of 10:1, and use genetic particle swarm algorithm to train the neural network to obtain the best neural network model.

[0020] In step S4, the image moment set Δm of the feedback image is calculated and input into the robot terminal motion prediction model, and the six-degree-of-freedom motion speed v of the robot terminal is output. e , including the following steps:

[0021] S4.1: During the servo task, the camera collects the feedback image and calculates the feature difference set Δm=[Δm 00 ,Δm 01 ,Δm 10 ,Δμ 11 ,Δμ 12 ,Δμ 21 ,Δμ 02 ,Δμ 20 ,Δμ 03 ,Δμ 30 ];

[0022] S4.2: Input Δm into the trained neural network model and output the six-degree-of-freedom motion speed of the robot end

[0023] The step S5 sends the obtained robot terminal velocity to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking, including the following steps:

[0024] S5.1: Set the robot's end six-degree-of-freedom motion speed v e Send to robot joint controller;

[0025] S5.2: Achieve stable motion of the robot through PID algorithm.

[0026] The beneficial effects of the present invention are:

[0027] The present invention realizes multi-view fusion visual servoing operation for large-size targets and has high operation accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the system structure of the present invention.

[0029] Figure 2 Schematic diagram of key features of edge arc segments extracted in step S2 of an embodiment of the present invention.

[0030] Figure 3 Schematic diagram of multi-camera calibration involved in step S3 of an embodiment of the present invention.

[0031] Figure 4 Schematic diagram of the robot terminal motion prediction model involved in step S4 of an embodiment of the present invention.

[0032] Figure 5 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] An array multi-camera calibration-free visual servoing method for a large-size rotating ring comprises the following steps:

[0035] S1: Manually control the robot to the reference position to collect the desired image;

[0036] S2: Select an appropriate sampling method to plan the robot's task workspace and obtain the robot's terminal motion speed during the motion process And the corresponding image moment set of the feedback image

[0037] S3: Use the collected data to train the neural network model to obtain a trained robot terminal motion prediction model;

[0038] S4: Calculate the image moment set Δm of the feedback image and input it into the robot terminal motion prediction model, and output the robot terminal six-degree-of-freedom motion speed v e ;

[0039] S5: The obtained robot terminal velocity is sent to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking.

[0040] In step S1, the multi-eye vision system collects images of a large-size rotating ring. The overall structure diagram of the vision system is as follows: Figure 1 As shown in the figure, the number of cameras in the multi-camera vision system is determined by the size of the rotating ring and the field of view of a single camera. The overall field of view of the multi-camera vision system covers the arc features of the rotating ring as much as possible. This system takes a multi-camera vision system composed of four cameras as an example. The four cameras are evenly distributed around the rotating ring, as shown in the figure. Figure 1 shown.

[0041] Step S2 selects a suitable sampling method to plan the robot's task workspace, such as Figure 2 , get the terminal motion speed of the robot during the motion process And the corresponding image moment set of the feedback image The following steps are involved:

[0042] S2.1: According to the relative position relationship between the camera field of view and the robot end, the size of 30×30×50mm is selected 3 , a rectangular parallelepiped area with attitude angles of 10°, 4° and 3° is used as the sampling space. According to the LHS method, 20 position coordinates and 15 attitude angles are randomly generated in the sampling space, and then they are arranged and combined to obtain the expected attitude of 300 sampling points;

[0043] S2.2: Manually control the robot to the reference position to collect the desired image and extract the image moment set from it Represents the image moment feature of the reference image, which is a constant. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order geometric moment feature is: m ij (t) = ∫∫ R(t) x i y i f(x,y)dxdy, f(x,y) is the image point p x = gray value of (x, y), for p x ∈R(t), then f(x,y)=1, otherwise f(x,y)=0. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order central moment feature is: μ ij (t) = ∫∫ R(t) (xx g ) i (yy g ) i dxdy,(x g ,y g) represents the coordinates of the center point of the image, x g =m 10 / m 00 ,y g =m 01 / m 00 .

[0044] S2.3: Move from a certain sampling point to the reference position. During this process, 10 clear feedback images and their corresponding six-freedom motion speeds of the robot end are randomly collected. Extract image moment sets from the above 10 feedback images And calculate

[0045] S2.4: Finally, the input and output pair.

[0046] In step S3, the collected data is used to train the neural network model to obtain a trained robot terminal motion prediction model, such as Figure 3 As shown, the following steps are included:

[0047] S3.1: Obtain 3000 sets of training samples according to the method in S2, and use 100, 200, ..., 2900, and 3000 sets of samples to train the neural network respectively, study the effect of the number of samples on the prediction bias of the neural network, and obtain the optimal number of training samples;

[0048] S3.2: Divide the data set into training set and test set in a ratio of 10:1, and use genetic particle swarm algorithm to train the neural network to obtain the best neural network model.

[0049] In step S4, the image moment set Δm of the feedback image is calculated and input into the robot terminal motion prediction model, and the robot terminal six-degree-of-freedom motion speed v is output. e ,like Figure 4 As shown, the following steps are included:

[0050] S4.1: During the servo task, the camera collects the feedback image and calculates the feature difference set Δm=[Δm 00 ,Δm 01 ,Δm 10 ,Δμ 11 ,Δμ 12 ,Δμ 21 ,Δμ 02 ,Δμ 20 ,Δμ 03 ,Δμ 30 ];

[0051] S4.2: Input Δm into the trained neural network model and output the six-degree-of-freedom motion speed of the robot end

[0052] In step S5, the obtained robot terminal velocity is sent to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking, including the following steps:

[0053] S5.1: Set the robot's end six-degree-of-freedom motion speed v e Send to robot joint controller;

[0054] S5.2: Achieve stable motion of the robot through PID algorithm.

[0055] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

[0056] The present invention does not involve parts that are the same as the prior art or can be implemented by using the prior art.

Claims

1. A calibration-free visual servoing method for array multi-cameras of large-size rotating rings, characterized in that: The following steps are involved: S1: Manually control the robot to the reference position to collect the desired image; S2: Plan the robot's task workspace and obtain the robot's terminal motion speed during the motion process And the corresponding image moment set of the feedback image S3: Use the collected data to train the neural network model to obtain a trained robot terminal motion prediction model; S4: During the servo task, the image moment set Δm of the feedback image is calculated and input into the robot end motion prediction model, and the six-degree-of-freedom motion speed v of the robot end is output. e ; S5: The obtained robot terminal velocity is sent to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking.

2. The method according to claim 1, characterized in that In step S2, the sampling method is selected to plan the robot's task workspace and obtain the end motion speed of the robot during the motion process. And the corresponding image moment set of the feedback image The following steps are involved: S2.1: According to the relative position relationship between the camera field of view and the robot end, select a size of 30×30×50 mm 3 , a rectangular parallelepiped area with attitude angles of 10°, 4° and 3° is used as the sampling space; according to the LHS method, 20 position coordinates and 15 attitude angles are randomly generated in the sampling space, and then they are arranged and combined to obtain the expected attitudes of 300 sampling points; S2.2: Manually control the robot to the reference position to collect the desired image and extract the image moment set from it Represents the image moment feature of the reference image, which is a constant. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order geometric moment feature is: m ij (t) = ∫∫ R(t) x i y i f(x,y)dxdy, f(x,y) is the image point p x = gray value of (x, y), for p x ∈R(t), then f(x,y)=1, otherwise f(x,y)=0. At time t, the projection area of ​​the observed object in the camera image plane is represented by R(t), and its i+j-order central moment feature is: μ ij (t) = ∫∫ R(t) (xx g ) i (yy g ) i dxdy,(x g ,y g ) represents the coordinates of the center point of the image, x g =m 10 / m 00 ,y g =m 01 / m 00 . S2.3: Move from a certain sampling point to the reference position. During this process, 10 clear feedback images and their corresponding six-freedom motion speeds of the robot end are randomly collected. Extract image moment sets from the above 10 feedback images And calculate S2.4: Finally, the input and output Paired, as a dataset for neural network models in S3.

3. The method according to claim 1, characterized in that The step S3 uses the collected data to train the neural network model to obtain a trained robot terminal motion prediction model, which includes the following steps: S3.1: Obtain 3000 sets of training samples according to the method in step S2, and use 100, 200, ..., 2900, and 3000 sets of samples to train the neural network respectively, study the effect of the number of samples on the prediction bias of the neural network, and obtain the optimal number of training samples; S3.2: Divide the data set into training set and test set in a ratio of 10:1, and use genetic particle swarm algorithm to train the neural network to obtain the best neural network model.

4. The method according to claim 1, characterized in that: In step S4, the image moment set Δm of the feedback image is calculated and input into the robot terminal motion prediction model, and the six-degree-of-freedom motion speed v of the robot terminal is output. e , including the following steps: S4.1: During the servo task, the camera collects the feedback image and calculates the feature difference set Δm=[Δm 00 ,Δm 01 ,Δm 10 ,Δμ 11 ,Δμ 12 ,Δμ 21 ,Δμ 02 ,Δμ 20 ,Δμ 03 ,Δμ 30 ]; S4.2: Input Δm into the trained neural network model and output the six-degree-of-freedom motion speed of the robot end 5. The method according to claim 1, characterized in that The step S5 sends the obtained robot terminal velocity to the robot joint controller to drive the robot to move, so as to achieve the purpose of visual servo target tracking, including the following steps: S5.1: Set the robot's end six-degree-of-freedom motion speed v e Send to robot joint controller; S5.2: Achieve stable motion of the robot through PID algorithm.

Images