Eye-in-hand robot positioning system and method based on rotating cascade prism

Abstract

The present invention relates to an eye-in-hand robot positioning system and method based on a rotating cascade prism, the system comprising a visual system, a robot and a host computer; the host computer is connected to the robot and the visual system respectively; the robot and the visual system are connected; the visual system comprises a prism drive component, a rotating cascade prism and a camera; the host computer is used to receive data collected by the optical imaging component, and perform data processing, calculate the three-dimensional coordinates of the target, and thus issue instructions to the robot to realize the target tracking function. The present invention introduces a rotating cascade prism multi-view imaging method, which effectively expands the imaging field of view without any movement of the camera body, simplifies the target calibration process, improves the calibration accuracy and stability, and realizes the robot's three-dimensional perception and target positioning. Compared with the prior art, the present invention has the advantages of a large imaging field of view, a large robot visual working space and high imaging efficiency.

Description

Technical Field

[0001] The invention relates to the field of machine vision, and in particular to an eye-in-hand robot positioning system and method based on a rotating cascade prism. Background Art

[0002] A robot is an automatic, position-controllable, programmable, multifunctional mechanical device with several axes that can process various materials, parts, tools and special devices with the help of programmable operations to perform various tasks. Due to its unique operational flexibility, it has been widely used in industrial assembly, safety and explosion-proof and other fields. Machine vision is an important direction for the development of robots. In many application scenarios, visual guidance is required to control the movement of robots, mainly including control chips, drive circuits, robots, cameras and other parts. Autonomous grasping means that without human intervention, the visual robot system obtains the position of the target object through the camera, uses software to process the data collected by the visual system, and then drives the robot to complete the tracking or grasping task of the target object. At present, there are two main visual guidance methods in the industry: eye-outside hand and eye-in-hand. The eye-in-hand method has become the main development direction because of its good integration and small device size. However, this traditional eye-in-hand method still has many limitations, such as the small field of view of the visual system when the eye is on the hand, easy loss of target position, slow positioning speed, low positioning accuracy and other problems.

[0003] For example, the Chinese patent application with application number CN202111556272.6 discloses a target tracking robotic arm method, system, device and storage medium. This invention can significantly reduce the load of the robotic arm, leave more load space, and improve the actual load capacity of the robotic arm. However, the defect of this invention is that the imaging field of view is small and the tracking marker on the robot must always be within the field of view of the visual system.

[0004] The Chinese patent application number CN202210116083.5 discloses a universal robot eye on hand calibration method and system, which can obtain the calibration results of the robot eye on the hand, solves the problem of non-convergence of the conventional method for SCARA robot, and the steps are simpler. However, this method requires controlling the robot to shoot a series of calibration plate image groups along the robot base coordinate system, the system calibration is cumbersome, and the imaging efficiency is low.

[0005] The Chinese patent application number CN201911019411.4 discloses a monocular industrial robot with visual tracking. This solution uses a monocular vision structure with an eye on the hand, and installs a tracking camera directly below the end effector to observe and measure the position of the moving workpiece relative to the actuator in real time, but it still does not solve the problem of small imaging field of view.

[0006] In summary, the research on eye-in-hand robots has defects such as small imaging field of view, small robot visual working space and low imaging efficiency. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an eye-in-hand robot positioning system and method based on a rotating cascade prism. The present invention realizes a new eye-in-hand robot vision solution, expands the imaging field of view, improves the imaging resolution, efficiency and integration, and realizes the robot's three-dimensional perception and target positioning.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] An eye-in-hand robot positioning system based on a rotating cascade prism, comprising a visual system, a robot and a host computer;

[0010] The host computer is connected to the robot and the visual system respectively; the robot and the visual system are connected;

[0011] The visual system includes an optical imaging component and a prism driving component;

[0012] The optical imaging assembly includes a rotating cascade prism and a camera;

[0013] The rotating cascade prism is used to change the visual axis direction of the camera and expand the imaging field of view of the camera;

[0014] The host computer is used to receive the data collected by the optical imaging component, perform data processing, calculate the three-dimensional coordinates of the target, and issue instructions to the robot based on the three-dimensional coordinates to track the target.

[0015] Further, the prism drive assembly includes a motor housing and a prism motor;

[0016] The prism motor is mounted on the motor housing.

[0017] Furthermore, the rotating cascade prism includes a prism, and the rotating cascade prism is one of a rotating single prism, a rotating double prism, a rotating triangular prism and a rotating polygonal prism.

[0018] Furthermore, a driving end of the prism motor is provided with a mounting hole, the prism is mounted in the mounting hole, and the prism motor is used to drive the prism to rotate.

[0019] Furthermore, the host computer is electrically connected to the robot and the visual system respectively; and the robot and the visual system are rigidly connected.

[0020] An eye-in-hand robot positioning method based on a rotating cascade prism is used to control an eye-in-hand robot positioning system based on a rotating cascade prism as described above, comprising the following steps:

[0021] S1. Determine the target to be tracked;

[0022] S2. Establish a coordinate system and calibrate the initial position and parameters of the system, including calibration of camera internal parameters, calibration of robot hand-eye relationship, calibration of rotating cascade prism, and calibration of relative position and posture of camera and rotating cascade prism;

[0023] S3, changing the rotation angle of the rotating cascade prism through the prism driving component, driving the camera visual axis to rotate, and obtaining the target image of the corresponding viewpoint at the angle by the camera, repeating this step to obtain multiple target images of different viewpoints, combining the multiple target images of different viewpoints into a target image sequence, and transmitting the target image sequence data to the host computer;

[0024] S4, based on the target image sequence, restore the three-dimensional information of the target image, and determine the three-dimensional coordinates of the target position in the robot coordinate system in combination with the coordinate system transfer conversion relationship;

[0025] S5. According to the three-dimensional coordinates of the position of the target in the robot coordinate system, the host computer calculates the rotation angle of each joint angle of the robot and controls the robot to approach the target.

[0026] Furthermore, step S2 includes the following sub-steps:

[0027] S21, establish a coordinate system, including pixel coordinate system uv, image coordinate system xy, normalized image coordinate system x n -y n , actual camera coordinate system X C -Y C -Z C , Rotate the cascade prism coordinate system X P -Y P -Z P 、Robot coordinate system X R -Y R -Z R and the world coordinate system X W -Y W -Z W ;

[0028] S22, calibrating the structural parameters of the prism by a laser interference method, the imaging plane of the camera is kept parallel to the prism plane side by a self-collimation method, and the optical axis of the camera is perpendicular to the center of the prism plane side;

[0029] S23, using Zhang Zhengyou calibration method to determine the internal parameters of the camera, wherein the internal parameters include the principal point pixel position, the lens equivalent focal length and the tilt factor;

[0030] S24, robot hand-eye relationship calibration: fix the camera at the end of the robot, carry the camera to multiple different positions by the robot, and let the camera collect images of the fixed calibration object at multiple different positions. According to the calibration object information observed in the camera coordinate system and the robot coordinate system when the robot moves to each position, construct the relative posture change matrix A of the camera and the relative posture change matrix B of the robot end effector, and obtain the hand-eye transformation matrix X to be solved by solving the basic hand-eye equation AX=XB;

[0031] S25. Use the transmission method to calibrate the main cross-section position of the rotating cascade prism; install the rotating cascade prism in front of the camera, and introduce a plane target fixed relative to the robot to calibrate the relative position of the camera and the rotating cascade prism.

[0032] Furthermore, step S3 includes the following sub-steps:

[0033] S31, sequentially obtaining the corresponding camera imaging angles generated by the rotating cascade prism at different rotation angles; using the rotational motion of the rotating cascade prism to change the camera's visual axis direction, converting the change in the imaging visual axis direction generated by the rotating cascade prism into a change in the camera's own position;

[0034] S32, according to the relative position of the target in the camera field of view, keep the robot and the camera fixed, use the prism drive assembly to drive the rotating cascade prism to perform synchronous rotation at a given step angle δ, thereby changing the prism rotation angle in sequence, so that the target is always within the camera's field of view, use the camera to respectively collect target images from different viewpoints, and combine multiple target images from different viewpoints into a target image sequence;

[0035] S33, the minimum step distance δ of the synchronous rotation movement of the rotating cascade prism min It is expressed as:

[0036]

[0037] Where f is the focal length of the camera lens, μ is the actual physical size of the camera pixel, z is the axial distance between the camera and the target, n is the refractive index of the prism material, α is the wedge angle of the prism, and d min is the minimum axial spacing between rotating cascade prisms.

[0038] Furthermore, step S4 includes the following sub-steps:

[0039] S41. Based on the geometric optics method and the image perspective transformation model, reverse tracing is performed according to the collected target image sequence information to obtain the exact solution of the target image projection light, that is, the pitch angle ρ and azimuth angle of the light relative to the system optical axis. and the intersection coordinates P with the exit surface of the rotating cascade prism k ;

[0040] S42, let the incident light vector of prism x be A r0 , the refraction vector of the incident surface is A rx , according to the law of vector refraction, we can find the outgoing light vector A r(x+1) , whose expressions are:

[0041] A r0 =(sinβ r1 sinγ r1 ,cosβ r1 ,sinβ r1 cosγ r1 ) T =(x r0 ,y r0 ,z r0 ) T

[0042]

[0043]

[0044] Where n represents the refractive index of the prism material, N xx represents the normal vector of the incident surface of the prism x, N x(x+1) represents the normal vector of the prism x+1 output surface; β r1 is the angle between the incident light and the positive direction of the Y axis, γ r1 is the angle between the projection of the incident light in the XOZ plane and the positive direction of the Z axis;

[0045] S43, according to the outgoing light vector A r(x+1) , and obtain the elevation angle ρ and azimuth angle of the system optical axis

[0046] ρ = arccos(z r(x+1) )

[0047]

[0048] Outgoing light vector A r(x+1) The intersection point with the exit surface is the prism exit point, denoted by P k =(X Pk ,Y Pk ,Z Pk );

[0049] S44, according to the prism exit point P k =(X Pk ,Y Pk ,Z Pk ), find the three-dimensional coordinates of the target point in the camera coordinate system to satisfy P C =(X C ,Y C ,Z C ) meets the following conditions:

[0050]

[0051] Where m>=2, the least square method is used to solve the above equation to obtain:

[0052] P C =(C T C) -1 C T D

[0053] Where C is the coefficient matrix on the left side of equation (4), and D is the column vector on the right side of equation (4);

[0054] S45, combining the coordinate system transfer conversion relationship, according to the hand-eye transformation matrix X, the three-dimensional coordinates P of the target point in the camera coordinate system C =(X C ,Y C ,Z C ) is converted to the robot coordinate system and recorded as P R =P C X=(X R ,Y R ,Z R ).

[0055] Furthermore, step S5 includes the following sub-steps:

[0056] S51, the transformation relationship between each joint sub-coordinate system and the tool sub-coordinate system of the robot coordinate system satisfies the multi-body kinematic model;

[0057] S52. Based on the multi-body kinematics model, calculate the corresponding joint rotation angles of the robot;

[0058] S53, controlling the robot to approach the target through the host computer according to the rotation angles of each joint of the robot.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention introduces the rotating cascade prism multi-view imaging method into the robot visual positioning system for the first time. The imaging field of the camera is changed by rotating the rotating cascade prism. The imaging field of the camera is effectively expanded without any movement of the camera body and without introducing reflective elements that are sensitive to error disturbances.

[0061] 2. The present invention uses a camera to obtain the multi-dimensional dynamic posture changes of the tracking target, obtains a multi-viewpoint, high-resolution image sequence of the target, and then uses a host computer to perform a three-dimensional reconstruction method on the obtained multi-view target image sequence with rich parallax information to obtain the three-dimensional data information of the target to be measured. After the host computer obtains the target posture information, it can convert the posture under the camera coordinate system to the robot coordinate system, and guide the robot to quickly approach the target. The tracking target can be tracked in real time.

[0062] 3. The present invention utilizes a rotating cascade prism to expand the imaging field of view, and the target object is not easy to leave the field of view of the visual system, thereby ensuring that the visual system has high reliability in this process, and can effectively expand the field of view of a single fixed camera and improve the robot's visible working space.

[0063] 4. The present invention establishes a real-time association between the rotating cascade prism variable visual axis imaging and the robot posture, thereby quickly and accurately realizing target positioning and perception, with high imaging efficiency and good flexibility.

[0064] 5. Compared with the traditional hand-eye robot system, the present invention has high integration and compact structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of an eye-in-hand robot positioning system based on rotating cascade prisms provided in Example 1 of the present invention;

[0066] Figure 2 Schematic diagram of the incident and outgoing vectors of a prism;

[0067] Figure 3 This is a schematic diagram of the structure of a visual system of an eye-in-hand robot positioning system based on a rotating cascade prism provided in Example 1 of the present invention;

[0068] Figure 3 The corresponding relationship between the numbers in the figure and the components is: 1-robot, 2-host computer, 3-visual system, 4-first-stage prism, 5-second-stage prism, 6-camera, 7-camera bracket, 8-motor housing, 9-prism motor;

[0069] Figure 4 Schematic diagram of the imaging boresight principle based on cascaded prisms;

[0070] Figure 4The corresponding relationship between the numbers and the components in the figure is: A-equivalent camera, B-main camera, C-rotating double prism:;

[0071] Figure 5 Schematic diagram of various coordinate systems in Example 2 of the present invention.

[0072] Figure 6 This is an appearance diagram of the four-degree-of-freedom robot involved in Example 2 of the present invention.

[0073] Figure 7 This is a schematic diagram of the inverse kinematics solution of a four-degree-of-freedom robot based on the geometric method involved in Example 2 of the present invention. DETAILED DESCRIPTION

[0074] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0075] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0077] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are the directions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0078] Example 1

[0079] like Figure 1As shown, an eye-in-hand robot positioning system based on a rotating cascade prism includes a visual system 3, a robot 1, and a host computer 2; the host computer 2 is connected to the robot 1 and the visual system 3 respectively; the robot 1 is connected to the visual system 3. In this embodiment, the host computer 2 is electrically connected to the robot 1 and the visual system 3 respectively; the robot 1 and the visual system 3 are rigidly connected.

[0080] The visual system 3 includes an optical imaging component and a prism driving component; the optical imaging component includes a rotating cascade prism and a camera; the rotating cascade prism is used to change the visual axis direction of the camera, thereby expanding the imaging field of view, and by converting the visual axis direction change brought by the rotating cascade prism into the self-position change of the virtual imaging unit, the multi-degree-of-freedom dynamic position adjustment of the imaging unit is equivalent to the traditional hand-eye system capturing target image information from multiple different viewpoints. It is possible to obtain a multi-viewpoint, high-resolution image sequence of the target, and combined with the sequence image, the imaging field of view of the restricted scene is expanded, and the imaging resolution, efficiency and integration are improved.

[0081] The host computer is used to receive the data collected by the camera in the optical imaging component, and perform data processing such as coordinate calculation and coordinate conversion to calculate the three-dimensional coordinates of the target, thereby issuing instructions to the robot 1 to realize the target tracking function.

[0082] The rotating cascade prism includes a combination of various numbers of prisms, including a rotating single prism, a rotating double prism, a rotating triangular prism, and a rotating multi-prism.

[0083] like Figure 3 As shown, the rotating cascade prism assembly in this embodiment provides an eye-in-hand robot system by taking a rotating double prism as an example, and the optical imaging assembly includes a first-stage prism 4, a second-stage prism 5 and a camera 6. The wedge surface of the first-stage prism 4 and the wedge surface of the second-stage prism 5 are opposite and the two-stage prism planes are coaxial, and the camera 6 and the second-stage prism 5 are located on both sides of the first-stage prism 4. The visual axis of the camera 6, the optical axis of the first-stage prism 4 and the optical axis of the second-stage prism 5 remain coaxial.

[0084] The host computer 2 is connected to the robot 1 and the visual system 3 respectively. The host computer 2 restores the three-dimensional coordinates of the target according to the image sequence taken by the visual system 3, and controls the robot to quickly guide and track.

[0085] One end of the prism is a plane and the other end is an inclined plane, and the cross section is a wedge shape. When the plane of the prism is the incident plane of light, it is a front arrangement, and when the inclined surface of the prism is the incident plane of light, it is a reverse arrangement.

[0086] The imaging target surface of the camera is parallel to the plane sides of the two achromatic prisms. The internal parameters of the camera such as focal length, resolution and detection band can be adjusted according to changes in specific application scenarios.

[0087] The visual system controls the rotation of the two-stage prism to expand the imaging field of view and capture target image sequences from multiple perspectives; the host computer is used to receive visual system data and restore the target's three-dimensional information; the host computer calculates the position and posture of the target object in the robot's coordinate system to guide the robot to quickly track the target.

[0088] As an optional embodiment, the prism drive assembly includes a motor housing 8 and a prism motor 9. The prism motor 9 is mounted on the motor housing 8. The driving end of the prism motor 9 is provided with a mounting hole, and the prism is mounted in the mounting hole, that is, the driving end of the prism motor 9 is connected to the first-stage prism 4 and the second-stage prism 5, and is used to drive the first-stage prism 4 and the second-stage prism 5 to rotate. The camera 6 is mounted on the motor housing 8 through the camera bracket 7. The driving end of the prism motor 9 is provided with a mounting hole, and the first-stage prism 4 and the second-stage prism 5 are mounted in the mounting hole.

[0089] A more optimal implementation can be obtained by arbitrarily combining the above implementations. An optimal implementation is described in detail below.

[0090] like Figure 1 and Figure 3 As shown, the eye-in-hand robot system based on rotating cascade prisms of this embodiment includes a robot 1, a host computer 2, a visual system 3, a first-stage prism 4, a second-stage prism 5, a camera 6, a camera bracket 7, a motor housing 8 and a prism motor 9.

[0091] The camera 6 includes an image sensor and a lens. The target surface size and pixel size of the image sensor, the focal length and depth of field of the lens and other parameters are determined by the range of the target scene. The detection band of the image sensor is determined by the properties of the target scene, which is the visible light band or the infrared light band.

[0092] In this embodiment, the camera 6 is an industrial camera, and a mounting hole is provided on the right end surface of the motor housing 9 , and the camera is vertically fixed to the right end surface of the motor housing 8 through a camera bracket 8 .

[0093] The prism motor 9 is provided with a mounting hole, in which the first-stage prism 4 and the second-stage prism 5 are mounted. The first-stage prism 4 is arranged between the second-stage prism 5 and the camera 6, and the visual axis of the camera 6, the optical axis of the first-stage prism 4 and the optical axis of the second-stage prism 5 are kept coaxial, and the imaging visual axis of the camera 6 is changed by controlling the rotation angle of the rotating cascade prism.

[0094] This embodiment adds a prism drive assembly to the rotating cascade prism device, and can adjust the camera's imaging angle of view arbitrarily through the synchronous rotation of the rotating cascade prism, thereby collecting a multi-view target image sequence with rich parallax information, which can effectively improve the field of view and resolution of three-dimensional computational imaging. Compared with traditional camera imaging devices, its imaging field of view is expanded.

[0095] This embodiment introduces an optical imaging component and a host computer in front of the camera, which can first capture a wide range of target multi-view target image sequences through the field of view expansion and primary imaging of the optical imaging component. Then, the host computer performs a three-dimensional reconstruction method on the multi-view target image sequence with rich parallax information to obtain the three-dimensional data information of the target to be measured. After obtaining the target posture information, the host computer can convert the posture of the camera coordinate system to the robot coordinate system to guide the robot to quickly approach the target. Compared with the existing eye-in-hand robot, the eye-in-hand robot positioning system based on the rotating cascade prism of this embodiment does not require the camera body to perform any form of movement, nor does it introduce reflective elements that are sensitive to error disturbances. It can simultaneously meet the performance requirements of structural compactness, imaging field of view, image resolution, imaging efficiency and flexibility.

[0096] Example 2

[0097] This embodiment provides an eye-in-hand robot positioning method based on a rotating cascade prism, which is used to control an eye-in-hand robot positioning system based on a rotating cascade prism described in Embodiment 1. The method comprises the following steps:

[0098] The parameters of each component of the system in Example 1 are described as follows:

[0099] Camera 6: The horizontal and vertical field of view angles of the camera to be selected are φ ch =23.39° and φ cv =17.65°, the image resolution is 1600pixel(H)×1200pixel(V), the pixel size is 4.4μm(H)×4.4μm(V), and the lens focal length is f=16mm.

[0100] Rotating cascade prism assembly: The rotating cascade prism assembly in this embodiment is a rotating double prism, and the two prisms are exactly the same. Figure 4 As shown, the parameters of the double prism are selected as α=20°, refractive index n=1.517, thin end thickness d0=5mm, and the spacing between the two prisms is d1=30mm. The rotational motion accuracy is better than 0.01°.

[0101] Robot 1: A 4-DOF robot is proposed. Figure 6 As shown in the figure, the robot has five connecting rods, namely chassis, slewing arm, upper arm, lower arm and clamp, which are connected to each other through a rotation joint, with a total of four rotation joints. It can be seen that except for the joint connecting the slewing body and the chassis, the other three joints are parallel to each other. Therefore, when the robot performs a grasping operation, the clamp can only be perpendicular to the rotation axis of joint 2.

[0102] Host computer 2: It has image acquisition software, rotating prism position control software, target 3D information algorithm and robot angle algorithm. It uses the imaging visual axis adjustment inverse algorithm to calculate the prism angle and sends the speed and angle information to the rotating prism controller; it stores the target image acquired by the camera and calculates the target 3D information; it solves the robot joint angle according to the robot's multi-body kinematic model algorithm and guides the robot to track the target.

[0103] The specific steps are as follows:

[0104] S1 establishes a suitable coordinate system and calibrates the initial position and parameters of the system: calibration of camera internal parameters, calibration of robot hand-eye relationship, calibration of rotating cascade prism, and calibration of relative position and posture of camera and cascade prism.

[0105] S2 changes the rotation angle of the cascade prism through the prism driving component to capture target image sequences from different viewpoints and transmits the data to the host computer.

[0106] S3 combines the theoretical model of the cascade prism inverse problem to restore the three-dimensional information of the target and determine the three-dimensional coordinates of the target position in the robot coordinate system in combination with the coordinate system transfer transformation relationship.

[0107] The S4 host computer converts the three-dimensional coordinates of the target position in the robot coordinate system into the rotation angles of each joint angle of the robot and guides the robot to quickly approach the target.

[0108] Step S1 specifically includes the following sub-steps:

[0109] S11. Establish a suitable coordinate system, including pixel coordinate system uv, image coordinate system xy, and normalized image coordinate system x n -y n , actual camera coordinate system X C -Y C -Z C , rotating biprism coordinate system X P -Y P -Z P 、Robot coordinate system X R -Y R -Z R 、World coordinate system X W -Y W -Z W Etc. Figure 6 shown.

[0110] S12. In order to simplify the system calibration process, the following assumptions are made: the structural parameters of the prism are calibrated by laser interference method; there is no need to consider error factors such as wedge angle, refractive index and thin end thickness; the imaging plane of the actual camera is kept parallel to the prism plane side by the self-collimation method, and its optical axis points vertically to the center of the prism plane side.

[0111] S13, camera internal parameter calibration: Zhang Zhengyou calibration method is used to determine the camera's internal parameters such as the principal point pixel position, lens equivalent focal length, and tilt factor.

[0112] S14, robot hand-eye relationship calibration: fix the camera separately at the end of the robot, use the robot-mounted camera to collect images of fixed calibration objects at multiple different positions, and construct the relative posture change matrix A of the camera and the relative posture change matrix B of the robot end effector according to the calibration object information observed in the camera coordinate system and the robot coordinate system when the robot moves to various positions, and obtain the hand-eye transformation matrix X to be solved by solving the basic hand-eye equation AX=XB.

[0113] S15. Use the transmission method to calibrate the main cross-section position of the rotating biprism; install a rotating biprism in front of the camera, and introduce a plane target fixed relative to the robot to calibrate the relative position of the camera and the biprism.

[0114] Step S2 specifically includes:

[0115] S21, sequentially obtain the imaging viewing angles generated by the rotating cascade prisms at different rotation angles; use the independent rotational motion of the rotating double prisms around the optical axis to accurately and smoothly change the visual axis direction of the imaging unit, such as Figure 4 As shown, the change in the visual axis direction brought about by the rotating biprism is equivalent to being converted into a change in the camera's own position.

[0116] S22. According to the relative position of the imaging target in the field of view of the system, while keeping the robot and the camera fixed, use the rotating mechanism to drive the rotating cascade prism to perform synchronous rotation at a given step angle δ to change the prism angle in turn, so that the target point is always within the field of view of the camera, and use the camera to collect multi-view element image sequences of the target.

[0117] S23. The given step angle of the synchronous rotational motion of the rotating cascade prism should take into account the limitation of the imaging resolution of the camera. The minimum step angle δ of the synchronous rotational motion is determined accordingly. min It is expressed as:

[0118]

[0119] Where f is the focal length of the camera lens, μ is the actual physical size of the pixel, z is the axial distance between the camera and the target, n is the refractive index of the prism material, α is the wedge angle of the prism, and d min is the minimum axial spacing between cascaded prisms.

[0120] Step S3 specifically includes:

[0121] S31. Based on the geometric optics method and the image perspective transformation model, reverse tracing can be performed according to the collected image information to obtain the exact solution of the imaging projection light, that is, the pitch angle ρ and azimuth angle of the light relative to the system optical axis and the coordinates of its intersection with the exit surface of the biprism k .

[0122] S32, let the incident light vector of prism x be A r0 , the refraction vector of the incident surface is A rx , according to the law of vector refraction, we can find the outgoing light vector A r(x+1) , whose expressions are:

[0123] A r0 =(sinβ r1 sinγ r1 ,cosβ r1 ,sinβ r1 cosγ r1 ) T =(x r0 ,y r0 ,z r0 ) T

[0124]

[0125]

[0126] Where n represents the refractive index of the prism material, N xx represents the normal vector of the incident surface of the prism x, N x(x+1) represents the normal vector of the prism x+1 output surface, x=1,3,5…; β r1 is the angle between the incident light and the positive direction of the Y axis, γ r1 It is the angle between the projection of the incident light in the XOZ plane and the positive direction of the Z axis.

[0127] Take a single prism as an example, that is, x = 1, and the incident light vector of the prism is A r0 , the refraction vector of the incident surface is A r1 According to the law of vector refraction, the outgoing light vector A can be obtained r2

[0128] A r0 =(sinβ r1 sinγ r1 ,cosβ r1 ,sinβ r1 cosγ r1 ) T =(x r0 ,y r0 ,z r0 ) T

[0129]

[0130]

[0131] According to the outgoing light vector A r2 , the elevation angle ρ and azimuth angle of the optical axis of the system can be obtained

[0132] ρ = arccos(z r2 )

[0133]

[0134] In this embodiment, the rotating cascade prism is a double prism. According to the law of vector refraction, the outgoing light vector A can be obtained. rf :

[0135]

[0136] S33, according to the outgoing light vector A r(x+1) , and obtain the elevation angle ρ and azimuth angle of the system optical axis

[0137] ρ = arccos(z r(x+1) )

[0138]

[0139] Outgoing light vector A r(x+1) The intersection point with the exit surface is the prism exit point;

[0140] In this embodiment, for the dual prism, the elevation angle ρ and the azimuth angle of the optical axis of the system are obtained. for:

[0141] ρ = arccos(z rf )

[0142]

[0143] Outgoing light vector A rf The intersection point with the exit surface is the prism exit point P k =(XPk,YPk,Z Pk ).

[0144] S34, according to the prism exit point P k =(X Pk ,Y Pk ,Z Pk ), combined with the ray tracing principle and the variable visual axis projection model, the three-dimensional coordinates of the target point in the camera coordinate system can be obtained to satisfy P C =(XC ,Y C ,Z C )satisfy:

[0145]

[0146] Since there is always noise in real data, the overdetermined equations given by equation (4) may not have an exact solution, so the least squares method is used to solve it:

[0147] P C =(C T C) -1 C T D (5)

[0148] Where C is the coefficient matrix on the left side of equation (4), and D is the column vector on the right side of equation (4).

[0149] S35, combining the coordinate system transfer conversion relationship, the target pose coordinate P under the camera coordinate system C =(X C ,Y C ,Z C ) is converted to the robot coordinate system P R =P C X=(X R ,Y R ,Z R ).

[0150] Step S4 specifically includes:

[0151] S41. The transformation relationship between each joint sub-coordinate system and the tool sub-coordinate system of the robot coordinate system satisfies the multi-body kinematic model.

[0152] S42. According to the multi-body kinematic model of the robot, calculate the corresponding joint rotation angles of the robot.

[0153] like Figure 6 As shown in the figure, since the robot chassis and the rotating body are both stationary, for the convenience of calculation, the rotation axis of joint 2 is used as the Y coordinate system of the robot. R The robot coordinate system is established by using the geometric method. Then the inverse kinematics analysis is performed on the robot. Figure 7 As shown, let the lengths of the upper arm, lower arm and fixture be L1, L2 and L3 respectively, and the rotation angles of the four joints be θ1, θ2, θ3 and θ4 respectively. R =(X R ,Y R ,Z R ) and solve the rotation angles of the four joints in reverse.

[0154] S43. Generally, when the robot is holding the target object, the fixture L3 is mostly straight up and down, that is, when θ4 = 0°, we can get:

[0155]

[0156]

[0157] S44, the host computer controls the robot to quickly approach the target according to the rotation angles of the robot's joint angles obtained in step S43.

[0158] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. An eye-in-hand robot positioning method based on rotating cascade prisms, characterized in that: Used to control an eye-in-hand robot positioning system based on a rotating cascade prism, the eye-in-hand robot positioning system based on a rotating cascade prism comprising a visual system (3), a robot (1) and a host computer (2); The host computer (2) is connected to the robot (1) and the visual system (3) respectively; the robot (1) and the visual system (3) are connected; The visual system (3) comprises an optical imaging component and a prism driving component; The optical imaging assembly includes a rotating cascade prism and a camera; The rotating cascade prism is used to change the visual axis direction of the camera and expand the imaging field of view of the camera; The host computer is used to receive data collected by the optical imaging component, perform data processing, calculate the three-dimensional coordinates of the target, and issue instructions to the robot (1) based on the three-dimensional coordinates to track the target; The method comprises the following steps: S1. Determine the target to be tracked; S2. Establish a coordinate system and calibrate the initial position and parameters of the system, including calibration of camera internal parameters, calibration of robot hand-eye relationship, calibration of rotating cascade prism, and calibration of relative position and posture of camera and rotating cascade prism; S3, changing the rotation angle of the rotating cascade prism through the prism driving component, driving the camera visual axis to rotate, and obtaining the target image of the corresponding viewpoint at the angle by the camera, repeating this step to obtain multiple target images of different viewpoints, combining the multiple target images of different viewpoints into a target image sequence, and transmitting the target image sequence data to the host computer; S4, based on the target image sequence, restore the three-dimensional information of the target image, and determine the three-dimensional coordinates of the target position in the robot coordinate system in combination with the coordinate system transfer conversion relationship; S5. According to the three-dimensional coordinates of the position of the target in the robot coordinate system, the host computer calculates the rotation angle of each joint angle of the robot and controls the robot to approach the target; Step S4 includes the following sub-steps: S41. Based on the geometric optics method and the image perspective transformation model, reverse tracing is performed according to the collected target image sequence information to obtain the exact solution of the target image projection light, that is, the pitch angle ρ and azimuth angle of the light relative to the system optical axis and the intersection coordinates P with the exit surface of the rotating cascade prism k ; S42, let the incident light vector of prism x be A r0 , the refraction vector of the incident surface is A rx , according to the law of vector refraction, we can find the outgoing light vector A r(x+1) , whose expressions are: A r0 =(sinβ r1 sinγ r1 ,cosβ r1 ,sinβ r1 cosγ r1 ) T =(x r0 ,y r0 ,z r0 ) T Where n represents the refractive index of the prism material, N xx represents the normal vector of the incident surface of the prism x, N x(x+1) represents the normal vector of the prism x+1 output surface; β r1 is the angle between the incident light and the positive direction of the Y axis, γ r1 is the angle between the projection of the incident light in the XOZ plane and the positive direction of the Z axis; S43, according to the outgoing light vector A r(x+1) , and obtain the elevation angle ρ and azimuth angle of the system optical axis ρ=arccos(z r(x+1) ) Outgoing light vector A r(x+1) The intersection point with the exit surface is the prism exit point, denoted by P k =(X Pk ,Y Pk ,Z Pk ); S44, according to the prism exit point P k =(X Pk ,Y Pk ,Z Pk ), find the three-dimensional coordinates of the target point in the camera coordinate system to satisfy P C =(X C ,Y C ,Z C ) meets the following conditions: Where m>=2, the least square method is used to solve the above equation to obtain: P C =(C T C) -1 C T D Where C is the coefficient matrix on the left side of equation (4), and D is the column vector on the right side of equation (4); S45, combining the coordinate system transfer conversion relationship, according to the hand-eye transformation matrix X, the three-dimensional coordinates P of the target point in the camera coordinate system C =(X C ,Y C ,Z C ) is converted to the robot coordinate system and is denoted as P R =P C X=(X R ,Y R ,Z R ).

2. The method according to claim 1, characterized in that The prism drive assembly comprises a motor housing (8) and a prism motor (9); The prism motor (9) is mounted on the motor housing (8).

3. The method according to claim 2, characterized in that The rotating cascade prism includes a prism, and the rotating cascade prism is one of a rotating single prism, a rotating double prism, a rotating triangular prism and a rotating multi-prism.

4. The method according to claim 3, characterized in that The driving end of the prism motor (9) is provided with a mounting hole, the prism is mounted in the mounting hole, and the prism motor (9) is used to drive the prism to rotate.

5. The method according to claim 1, characterized in that The host computer (2) is electrically connected to the robot (1) and the visual system (3) respectively; the robot (1) and the visual system (3) are rigidly connected.

6. The method according to claim 1, characterized in that Step S2 includes the following sub-steps: S21, establish a coordinate system, including pixel coordinate system uv, image coordinate system xy, normalized image coordinate system x n -y n , actual camera coordinate system X C -Y C -Z C , Rotate the cascade prism coordinate system X P -Y P -Z P 、Robot coordinate system X R -Y R -Z R and the world coordinate system X W -Y W -Z W ; S22, calibrating the structural parameters of the prism by a laser interference method, the imaging plane of the camera is kept parallel to the prism plane side by a self-collimation method, and the optical axis of the camera is perpendicular to the center of the prism plane side; S23, using Zhang Zhengyou calibration method to determine the internal parameters of the camera, wherein the internal parameters include the principal point pixel position, the lens equivalent focal length and the tilt factor; S24, robot hand-eye relationship calibration: fix the camera at the end of the robot, carry the camera to multiple different positions by the robot, and let the camera collect images of the fixed calibration object at multiple different positions. According to the calibration object information observed in the camera coordinate system and the robot coordinate system when the robot moves to each position, construct the relative posture change matrix A of the camera and the relative posture change matrix B of the robot end effector, and obtain the hand-eye transformation matrix X to be solved by solving the basic hand-eye equation AX=XB; S25. Use the transmission method to calibrate the main cross-section position of the rotating cascade prism; install the rotating cascade prism in front of the camera, and introduce a plane target fixed relative to the robot to calibrate the relative position of the camera and the rotating cascade prism.

7. The method according to claim 1, characterized in that Step S3 includes the following sub-steps: S31, sequentially obtaining the corresponding camera imaging angles generated by the rotating cascade prism at different rotation angles; using the rotational motion of the rotating cascade prism to change the camera's visual axis direction, and converting the change in the imaging visual axis direction generated by the rotating cascade prism into a change in the camera's own position; S32, according to the relative position of the target in the camera field of view, keep the robot and the camera fixed, use the prism drive assembly to drive the rotating cascade prism to perform synchronous rotation at a given step angle δ, thereby changing the prism rotation angle in sequence, so that the target is always within the camera's field of view, use the camera to respectively collect target images from different viewpoints, and combine multiple target images from different viewpoints into a target image sequence; S33, the minimum step distance δ of the synchronous rotation movement of the rotating cascade prism min It is expressed as: Where f is the focal length of the camera lens, μ is the actual physical size of the camera pixel, z is the axial distance between the camera and the target, n is the refractive index of the prism material, α is the wedge angle of the prism, and d min is the minimum axial spacing between rotating cascade prisms.

8. The method according to claim 1, characterized in that Step S5 includes the following sub-steps: S51, the transformation relationship between each joint sub-coordinate system and the tool sub-coordinate system of the robot coordinate system satisfies the multi-body kinematic model; S52. Based on the multi-body kinematics model, calculate the corresponding joint rotation angles of the robot; S53, controlling the robot to approach the target through the host computer according to the rotation angles of each joint of the robot.

Images