Industrial robot workpiece positioning method and device based on multi-camera information fusion

Abstract

The application discloses a kind of industrial robot workpiece positioning method and device based on multi-camera information fusion, and the method comprises: obtaining the real-time image of multi-camera acquisition workpiece is preprocessed to obtain target image;Identify the optical mark point in the target image, and calculate the single workpiece pose of single camera coordinate system;Based on the single workpiece pose combined with the corresponding single camera weight is weighted to obtain the global pose of workpiece;Based on the global pose of workpiece combined with the current pose of robot is calculated to control robot to workpiece positioning follow, including generating expected speed and output joint torque.The application solves the precision, anti-interference and real-time problem of traditional positioning method by multi-camera cooperation, adaptive information fusion and advanced image processing technology, and provides high reliability, high flexibility workpiece follow solution for industrial robot.

Description

Technical Field

[0001] This invention relates to the field of industrial robot control technology, and more specifically, to an industrial robot workpiece positioning method and apparatus based on multi-camera information fusion. Background Technology

[0002] In modern intelligent manufacturing and automated production processes, industrial robots are widely used in operations such as workpiece handling, assembly, and welding. In these applications, robots need to acquire the workpiece's position and orientation information accurately in real time to achieve precise operation. Traditional position detection methods mainly rely on mechanical positioning, electromagnetic tracking, or monocular vision systems, but these methods have the following problems: 1. Mechanical positioning systems have complex structures, poor flexibility, and are difficult to adapt to the positioning requirements of different workpieces; 2. Electromagnetic tracking systems are susceptible to environmental electromagnetic interference and have a limited measurement range; 3. Monocular vision systems are susceptible to occlusion and motion blur occurs when the workpiece moves rapidly, leading to a decrease in positioning accuracy; 4. Most existing optical tracking systems use the principle of binocular vision. When one of the cameras is blocked, the system will not work properly.

[0003] Furthermore, in industrial production environments, workpieces are often in a dynamic state of change, and traditional positioning methods are difficult to meet the requirements of real-time high-precision tracking. Especially in applications with high precision requirements such as welding and assembly, the positioning accuracy and stability of existing technologies often cannot meet production requirements. Summary of the Invention

[0004] The purpose of this invention is to provide an industrial robot workpiece positioning method and apparatus based on multi-camera information fusion, which can solve the problems of insufficient accuracy, poor anti-interference ability, and susceptibility to occlusion in existing industrial robot workpiece positioning technologies.

[0005] The first aspect of this invention provides a method for industrial robot workpiece positioning based on multi-camera information fusion, comprising the following steps: The target image is obtained by preprocessing real-time images of the workpiece captured by multiple cameras. Identify optical marker points in the target image and calculate the pose of a single workpiece in a single-camera coordinate system; The global pose of the workpiece is obtained by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weight. The robot performs calculations based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece, including generating the desired speed and outputting joint torque.

[0006] In this solution, the step of acquiring real-time images of the workpiece from multiple cameras and preprocessing them to obtain the target image specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein the optical markers are arranged on the surface of the workpiece, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and edge enhancement based on bilateral filtering.

[0007] In this solution, the process of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical markers in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece.

[0008] In this scheme, the step of obtaining the global pose of the workpiece by weighted calculation based on the pose of a single workpiece and the corresponding single-camera weights specifically includes: Obtain the individual workpiece pose and measurement variance for each camera in the current multi-camera setup; Calculate the single-camera weights based on the measurement variance; The global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the single workpiece and the weight of the single camera, wherein the covariance confidence of the global pose of the workpiece is the highest.

[0009] In this solution, the calculation based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece includes generating the desired velocity and outputting joint torque, specifically including: The pose error is calculated based on the global pose of the workpiece and the current pose of the robot, and the desired speed is output accordingly. The joint torque is output based on the speed error calculated between the global pose of the workpiece and the current pose of the robot. Based on the desired speed and joint torque, the joint angular velocity is obtained, and joint control commands are output to control the robot to position and follow the workpiece.

[0010] A second aspect of the present invention also provides an industrial robot workpiece positioning system based on multi-camera information fusion, comprising a memory and a processor. The memory includes a program for an industrial robot workpiece positioning method based on multi-camera information fusion. When the processor executes the program for the industrial robot workpiece positioning method based on multi-camera information fusion, it performs the following steps: The target image is obtained by preprocessing real-time images of the workpiece captured by multiple cameras. Identify optical marker points in the target image and calculate the pose of a single workpiece in a single-camera coordinate system; The global pose of the workpiece is obtained by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weight. The robot performs calculations based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece, including generating the desired speed and outputting joint torque.

[0011] In this solution, the step of acquiring real-time images of the workpiece from multiple cameras and preprocessing them to obtain the target image specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein the optical markers are arranged on the surface of the workpiece, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and edge enhancement based on bilateral filtering.

[0012] In this solution, the process of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical markers in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece.

[0013] In this scheme, the step of obtaining the global pose of the workpiece by weighted calculation based on the pose of a single workpiece and the corresponding single-camera weights specifically includes: Obtain the individual workpiece pose and measurement variance for each camera in the current multi-camera setup; Calculate the single-camera weights based on the measurement variance; The global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the single workpiece and the weight of the single camera, wherein the covariance confidence of the global pose of the workpiece is the highest.

[0014] In this solution, the calculation based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece includes generating the desired velocity and outputting joint torque, specifically including: The pose error is calculated based on the global pose of the workpiece and the current pose of the robot, and the desired speed is output accordingly. The joint torque is output based on the speed error calculated between the global pose of the workpiece and the current pose of the robot. Based on the desired speed and joint torque, the joint angular velocity is obtained, and joint control commands are output to control the robot to position and follow the workpiece.

[0015] A third aspect of the present invention provides a computer-readable storage medium comprising a machine program for a method of locating an industrial robot workpiece based on multi-camera information fusion, wherein when the program is executed by a processor, it implements the steps of the method of locating an industrial robot workpiece based on multi-camera information fusion as described in any of the preceding claims.

[0016] A fourth aspect of the present invention provides an industrial robot workpiece positioning device based on multi-camera information fusion, applicable to any of the industrial robot workpiece positioning methods based on multi-camera information fusion described in any one of the claims, the device comprising: The system comprises a vision acquisition unit, a workpiece, an image processing unit, an information fusion unit, and a robot control unit. The workpiece surface is provided with multiple optical markers to provide positioning features for the image; The vision acquisition unit includes multiple industrial cameras, which are used to simultaneously acquire workpiece images from different angles; The image processing unit is communicatively connected to an industrial camera and is used to preprocess the workpiece image and identify the optical marker points to obtain the pose of a single workpiece. The information fusion calculation unit is communicatively connected to the image processing unit and is used to perform weighted fusion calculation based on the pose of a single workpiece to obtain the global pose of the workpiece. The robot control unit is communicatively connected to the information fusion computing unit and is used to position and follow the workpiece by adjusting the robot joint operation under the position-velocity dual closed-loop control strategy in conjunction with the global pose adjustment of the workpiece.

[0017] This invention discloses an industrial robot workpiece positioning method and device based on multi-camera information fusion. Through multi-camera collaborative work and advanced information fusion algorithms, it achieves high-precision and high-stability real-time tracking of workpieces. The specific beneficial effects are as follows: 1. Improved accuracy, with a positioning accuracy of 1.89±0.24mm, which is more than 30% higher than the traditional method; 2. Enhanced anti-interference capabilities: Utilizing a multi-camera redundancy design, the obstruction or failure of a single camera does not affect system operation. 3. Excellent real-time performance, supporting 60fps image processing and adapting to tracking of high-speed moving workpieces at 0.5m / s; 4. Improved adaptability, with flexible placement of marker points and multi-camera configuration, suitable for different industrial scenarios; 5. Reduced costs: Using ordinary industrial cameras, the cost is lower than that of commercial optical tracking systems. Attached Figure Description

[0018] Figure 1 A flowchart of an industrial robot workpiece positioning method based on multi-camera information fusion according to the present invention is shown; Figure 2 A schematic diagram of the multi-camera acquisition arrangement is shown for an industrial robot workpiece positioning method based on multi-camera information fusion according to the present invention. Figure 3 A schematic diagram of the arrangement of optical marker points in an industrial robot workpiece positioning method based on multi-camera information fusion according to the present invention is shown. Figure 4 A block diagram of an industrial robot workpiece positioning system based on multi-camera information fusion according to the present invention is shown. Figure 5 A block diagram of an industrial robot workpiece positioning device based on multi-camera information fusion according to the present invention is shown. Detailed Implementation

[0019] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0021] Figure 1 A flowchart of an industrial robot workpiece positioning method based on multi-camera information fusion, as described in this application, is shown.

[0022] like Figure 1 As shown, this application discloses a method for industrial robot workpiece positioning based on multi-camera information fusion, including the following steps: S102, acquire real-time images of the workpiece captured by multiple cameras and perform preprocessing to obtain the target image; S104, Identify optical marker points in the target image and calculate the pose of a single workpiece in the single-camera coordinate system; S106, The global pose of the workpiece is obtained by weighted calculation based on the pose of the single workpiece and the corresponding single camera weight. S108, calculate based on the global pose of the workpiece and the current pose of the robot to control the robot to position and follow the workpiece, including generating the desired speed and outputting joint torque.

[0023] It should be noted that this embodiment specifically describes how to perform real-time positioning and tracking of the workpiece, including acquiring real-time images of the workpiece from multiple cameras, such as... Figure 2 The diagram shows a multi-camera setup. In practical applications, a vision system is typically composed of 3-6 industrial cameras. The camera placement principle is as follows: 1. The viewpoint is evenly distributed around the workpiece, covering all possible movement ranges of the workpiece; 2. The cameras have a certain degree of overlapping field of view, ensuring that multiple cameras can see the workpiece at any given time; 3. The installation height and angle should avoid mutual obstruction; 4. Select the appropriate lens focal length based on the working distance; 5. Camera technical specifications: resolution of at least 2 megapixels; frame rate ≥ 30fps; interface is GigE or USB 3.0; and supports external trigger synchronization.

[0024] Furthermore, optical markers are arranged on the surface of the workpiece, such as... Figure 3 The diagram shown illustrates the arrangement of optical marker points, where... Figure 3 A1, A2, A3, A4, A5, and A6 correspond to six optical marker points arranged on the workpiece surface, i.e., Aruco marker numbers, used to locate the workpiece's pose in three-dimensional space. Accordingly, the marker point arrangement principle is as follows: 1. Distribute the markings evenly on the workpiece surface to ensure a sufficient number of markings are visible from different angles; 2. Arranged at a key feature location on the workpiece for easy positioning during subsequent operations; 3. The number should be no less than 4 to ensure the accuracy of pose calculation; 4. Avoid areas that may be obstructed by the operating tools; The process involves preprocessing the real-time image to obtain the target image, then identifying optical markers based on the target image to calculate the pose of a single workpiece in a single-camera coordinate system. Subsequently, the poses of each workpiece in a multi-camera system are weighted and calculated using the corresponding single-camera weights to obtain the global workpiece pose. Finally, the global workpiece pose is combined with the robot's current pose to control the robot's positioning and following of the workpiece. This includes generating the desired velocity and outputting joint torques, details of which are provided in the subsequent specification.

[0025] According to an embodiment of the present invention, the step of obtaining the target image by preprocessing the real-time images of the workpiece acquired by multiple cameras specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein the workpiece surface is provided with optical markers, the size of which ranges from 4mm×4mm to 10mm×10mm, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and edge enhancement based on bilateral filtering.

[0026] It should be noted that, in this embodiment, the optical markers are arranged on the surface of the workpiece. Accordingly, the optical markers specifically include Acuro markers, the size of which is generally "4mm×4mm to 10mm×10mm". The full name of the Acuro marker is "Augmented Reality University of Cordoba", which was developed by the computer vision research team of the University of Cordoba in Spain and is mainly used for fast pose estimation in augmented reality applications.

[0027] Furthermore, in this embodiment, during the image acquisition process of the camera on the workpiece, real-time images of the workpiece are acquired simultaneously by multiple cameras. The acquired real-time images are then preprocessed to obtain the target image. The preprocessing includes denoising and edge enhancement based on bilateral filtering. The bilateral filtering calculation formula is as follows: ; in, Let n be the pixel value of point n in the filtered image. Let m be the pixel value of the neighboring point m. The weights are the bilateral filtering weights. After bilateral filtering, a clear image with denoised pixels is obtained, which can be used for subsequent optical marker recognition. n is a normalization constant used to ensure that the pixel values ​​after filtering are still within the effective grayscale range. Since bilateral filtering applies a weighted average operation to each pixel, the normalization constant n can ensure that all weights add up to 1 (or the filtering result still maintains brightness consistency).

[0028] According to an embodiment of the present invention, the step of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical markers in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece.

[0029] It should be noted that, in this embodiment, Acuro markers in the target image are detected to extract corner pixel coordinates. The two-dimensional image coordinates of the current optical marker point are obtained based on the coordinates of the four corner points. The preset three-dimensional layout corresponds to 3D world coordinates. Then, the pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error. The rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece. Minimizing the reprojection error is a technical means that can be selected by those skilled in the art. In this embodiment, it is applied to solve the pose of a single workpiece. The specific process is not described here.

[0030] According to an embodiment of the present invention, the step of obtaining the global pose of the workpiece by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weights specifically includes: Obtain the individual workpiece pose and measurement variance for each camera in the current multi-camera setup; Calculate the single-camera weights based on the measurement variance; The global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the single workpiece and the weight of the single camera, wherein the covariance confidence of the global pose of the workpiece is the highest.

[0031] It should be noted that, in this embodiment, the measurement variance corresponding to a single camera is obtained as follows: ; in, Let be the measurement variance of the i-th camera, where i is the camera number and k is the measurement time. Furthermore, calculate the single-camera weights: ; in, Let i be the weight of the i-th camera. Let be the measurement variance of the i-th camera, and N be the number of cameras; Finally, the global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the individual workpiece and the weight of the single camera: ; in, This represents the final estimated position of the workpiece, corresponding to the workpiece's global pose. This refers to the global pose estimate of the workpiece, that is, the pose result after final fusion calculation. Let i be the weight of the i-th camera. Let be the measurement value of the i-th camera at time k.

[0032] According to an embodiment of the present invention, the step of calculating the robot's positioning and following of the workpiece based on the workpiece's global pose and the robot's current pose includes generating a desired velocity and outputting joint torque, specifically including: The pose error is calculated based on the global pose of the workpiece and the current pose of the robot, and the desired speed is output accordingly. The joint torque is output based on the speed error calculated between the global pose of the workpiece and the current pose of the robot. Based on the desired speed and joint torque, the joint angular velocity is obtained, and joint control commands are output to control the robot to position and follow the workpiece.

[0033] It should be noted that, in this embodiment, a position-velocity dual closed-loop control strategy is specifically implemented for robot control. Specifically, the pose error is calculated based on the global pose of the workpiece and the current pose of the robot to output the desired velocity. The desired velocity is also output based on the error between the global pose of the workpiece and the current pose. Furthermore, the velocity error is calculated based on the global pose of the workpiece and the current pose of the robot to output the joint torque. Finally, the joint angular velocity is obtained based on the desired velocity and the joint torque, and joint control commands (such as motor torque or pulse signals) are output. Based on these corresponding joint control commands, the robot is controlled to position and follow the workpiece.

[0034] Figure 4 A block diagram of an industrial robot workpiece positioning system based on multi-camera information fusion according to the present invention is shown.

[0035] like Figure 4As shown, this invention discloses an industrial robot workpiece positioning system based on multi-camera information fusion, including a memory and a processor. The memory includes a program for an industrial robot workpiece positioning method based on multi-camera information fusion. When the processor executes the program for the industrial robot workpiece positioning method based on multi-camera information fusion, it performs the following steps: The target image is obtained by preprocessing real-time images of the workpiece captured by multiple cameras. Identify optical marker points in the target image and calculate the pose of a single workpiece in a single-camera coordinate system; The global pose of the workpiece is obtained by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weight. The robot performs calculations based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece, including generating the desired speed and outputting joint torque.

[0036] It should be noted that this embodiment specifically describes how to perform real-time positioning and tracking of the workpiece, including acquiring real-time images of the workpiece from multiple cameras, such as... Figure 2 The diagram shows a multi-camera setup. In practical applications, a vision system is typically composed of 3-6 industrial cameras. The camera placement principle is as follows: 1. The viewpoint is evenly distributed around the workpiece, covering all possible movement ranges of the workpiece; 2. The cameras have a certain degree of overlapping field of view, ensuring that multiple cameras can see the workpiece at any given time; 3. The installation height and angle should avoid mutual obstruction; 4. Select the appropriate lens focal length based on the working distance; 5. Camera technical specifications: resolution of at least 2 megapixels; frame rate ≥ 30fps; interface is GigE or USB 3.0; and supports external trigger synchronization.

[0037] Furthermore, optical markers are arranged on the surface of the workpiece, such as... Figure 3 The diagram shown illustrates the arrangement of optical marker points, where... Figure 3 A1, A2, A3, A4, A5, and A6 correspond to six optical marker points arranged on the workpiece surface, i.e., Aruco marker numbers, used to locate the workpiece's pose in three-dimensional space. Accordingly, the marker point arrangement principle is as follows: 1. Distribute the markings evenly on the workpiece surface to ensure a sufficient number of markings are visible from different angles; 2. Arranged at a key feature location on the workpiece for easy positioning during subsequent operations; 3. The number should be no less than 4 to ensure the accuracy of pose calculation; 4. Avoid areas that may be obstructed by the operating tools; The process involves preprocessing the real-time image to obtain the target image, then identifying optical markers based on the target image to calculate the pose of a single workpiece in a single-camera coordinate system. Subsequently, the poses of each workpiece in a multi-camera system are weighted and calculated using the corresponding single-camera weights to obtain the global workpiece pose. Finally, the global workpiece pose is combined with the robot's current pose to control the robot's positioning and following of the workpiece. This includes generating the desired velocity and outputting joint torques, details of which are provided in the subsequent specification.

[0038] According to an embodiment of the present invention, the step of obtaining the target image by preprocessing the real-time images of the workpiece acquired by multiple cameras specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein the workpiece surface is provided with optical markers, the size of which ranges from 4mm×4mm to 10mm×10mm, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and edge enhancement based on bilateral filtering.

[0039] It should be noted that, in this embodiment, the optical markers are arranged on the surface of the workpiece. Accordingly, the optical markers specifically include Acuro markers, the size of which is generally "4mm×4mm to 10mm×10mm". The full name of the Acuro marker is "Augmented Reality University of Cordoba", which was developed by the computer vision research team of the University of Cordoba in Spain and is mainly used for fast pose estimation in augmented reality applications.

[0040] Furthermore, in this embodiment, during the image acquisition process of the camera on the workpiece, real-time images of the workpiece are acquired simultaneously by multiple cameras. The acquired real-time images are then preprocessed to obtain the target image. The preprocessing includes denoising and edge enhancement based on bilateral filtering. The bilateral filtering calculation formula is as follows: ; in, Let n be the pixel value of point n in the filtered image. Let m be the pixel value of the neighboring point m. The weights are the bilateral filtering weights. After bilateral filtering, a clear image with denoised pixels is obtained, which can be used for subsequent optical marker recognition. n is a normalization constant used to ensure that the pixel values ​​after filtering are still within the effective grayscale range. Since bilateral filtering applies a weighted average operation to each pixel, the normalization constant n can ensure that all weights add up to 1 (or the filtering result still maintains brightness consistency).

[0041] According to an embodiment of the present invention, the step of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical markers in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece.

[0042] It should be noted that, in this embodiment, Acuro markers in the target image are detected to extract corner pixel coordinates. The two-dimensional image coordinates of the current optical marker point are obtained based on the coordinates of the four corner points. The preset three-dimensional layout corresponds to 3D world coordinates. Then, the pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error. The rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece. Minimizing the reprojection error is a technical means that can be selected by those skilled in the art. In this embodiment, it is applied to solve the pose of a single workpiece. The specific process is not described here.

[0043] According to an embodiment of the present invention, the step of obtaining the global pose of the workpiece by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weights specifically includes: Obtain the individual workpiece pose and measurement variance for each camera in the current multi-camera setup; Calculate the single-camera weights based on the measurement variance; The global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the single workpiece and the weight of the single camera, wherein the covariance confidence of the global pose of the workpiece is the highest.

[0044] It should be noted that, in this embodiment, the measurement variance corresponding to a single camera is obtained as follows: ; in, Let be the measurement variance of the i-th camera, where i is the camera number and k is the measurement time. Furthermore, calculate the single-camera weights: ; in, Let i be the weight of the i-th camera. Let be the measurement variance of the i-th camera, and N be the number of cameras; Finally, the global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the individual workpiece and the weight of the single camera: ; in, This represents the final estimated position of the workpiece, corresponding to the workpiece's global pose. This refers to the global pose estimate of the workpiece, that is, the pose result after final fusion calculation. Let i be the weight of the i-th camera. Let be the measurement value of the i-th camera at time k.

[0045] According to an embodiment of the present invention, the step of calculating the robot's positioning and following of the workpiece based on the workpiece's global pose and the robot's current pose includes generating a desired velocity and outputting joint torque, specifically including: The pose error is calculated based on the global pose of the workpiece and the current pose of the robot, and the desired speed is output accordingly. The joint torque is output based on the speed error calculated between the global pose of the workpiece and the current pose of the robot. Based on the desired speed and joint torque, the joint angular velocity is obtained, and joint control commands are output to control the robot to position and follow the workpiece.

[0046] It should be noted that, in this embodiment, a position-velocity dual closed-loop control strategy is specifically implemented for robot control. Specifically, the pose error is calculated based on the global pose of the workpiece and the current pose of the robot to output the desired velocity. The desired velocity is also output based on the error between the global pose of the workpiece and the current pose. Furthermore, the velocity error is calculated based on the global pose of the workpiece and the current pose of the robot to output the joint torque. Finally, the joint angular velocity is obtained based on the desired velocity and the joint torque, and joint control commands (such as motor torque or pulse signals) are output. Based on these corresponding joint control commands, the robot is controlled to position and follow the workpiece.

[0047] A third aspect of the present invention provides a computer-readable storage medium comprising a program for an industrial robot workpiece positioning method based on multi-camera information fusion, wherein when the program is executed by a processor, it implements the steps of the industrial robot workpiece positioning method based on multi-camera information fusion as described in any of the preceding claims.

[0048] A fourth aspect of the present invention provides an industrial robot workpiece positioning device based on multi-camera information fusion, applicable to any of the industrial robot workpiece positioning methods based on multi-camera information fusion described in any one of the claims, the device comprising: The system comprises a vision acquisition unit, a workpiece, an image processing unit, an information fusion unit, and a robot control unit. The workpiece surface is provided with multiple optical markers to provide positioning features for the image; The vision acquisition unit includes multiple industrial cameras, which are used to simultaneously acquire workpiece images from different angles; The image processing unit is communicatively connected to an industrial camera and is used to preprocess the workpiece image and identify the optical marker points to obtain the pose of a single workpiece. The information fusion calculation unit is communicatively connected to the image processing unit and is used to perform weighted fusion calculation based on the pose of a single workpiece to obtain the global pose of the workpiece. The robot control unit is communicatively connected to the information fusion computing unit and is used to position and follow the workpiece by adjusting the robot joint operation under the position-velocity dual closed-loop control strategy in conjunction with the global pose adjustment of the workpiece.

[0049] It should be noted that, in this embodiment, as Figure 5 The diagram shows a block diagram of an industrial robot workpiece positioning device based on multi-camera information fusion. The device includes an acquisition stage, a processing stage, and a control stage. The acquisition stage involves a vision acquisition unit and the workpiece, the processing stage involves an image processing unit and an information fusion unit, and the control stage involves a robot control unit. Specifically, the workpiece surface is equipped with multiple optical markers to provide positioning features for the image, facilitating positioning during the processing stage. The vision acquisition unit includes multiple industrial cameras to simultaneously acquire workpiece images from different angles, ensuring that the workpiece can be photographed from multiple angles to cover all viewing angles.

[0050] Furthermore, in the processing stage, the image is specifically processed to obtain pose information. Specifically, the image processing unit is communicatively connected to the industrial camera to preprocess the workpiece image and identify the optical markers to obtain the pose of a single workpiece. The information fusion calculation unit is communicatively connected to the image processing unit to perform weighted fusion calculation based on the pose of a single workpiece to obtain the global pose of the workpiece.

[0051] Furthermore, in the control phase, the robot joints are controlled. The robot control unit is communicatively connected to the information fusion computing unit and is used to position and follow the workpiece by adjusting the robot joint operation under the position-velocity dual closed-loop control strategy in conjunction with the global pose adjustment of the workpiece.

[0052] Specifically, taking workpiece handling on an automotive welding production line as an example, in application, this invention evenly arranges six 5mm×5mm Aruco markers on the workpiece to be handled, and four 5-megapixel industrial cameras are positioned around the workpiece at a frame rate of 60fps. The image processing unit performs bilateral filtering (σ_q=10, σ_k=5) on each frame, where σ_q represents the spatial domain standard deviation, corresponding to the influence of the spatial distance between pixels. A larger value indicates that the influence on more distant neighboring pixels is also included in the calculation, i.e., a larger filtering range. In this embodiment, σ_q... =10 indicates that distant neighboring pixels in the image are also used for weighted averaging to enhance global smoothness; represents the standard deviation of pixel intensity (color) domain, corresponding to the control of the influence of pixel grayscale / color difference. The smaller the value, the more likely that only pixels with similar colors will participate in smoothing, which can better preserve image edges. In this embodiment, σ_k=5 is to avoid blurring the edges, improve the recognition accuracy of workpiece markers, and identify optical markers to calculate the pose of a single workpiece. The AWPM (Adaptive Weighted Positioning Method) algorithm is used to fuse the data of four cameras to calculate the global pose of the workpiece, and then combined with the robot's current pose to control the welding robot to follow the workpiece movement in real time. The system positioning accuracy reached "1.92±0.26mm" in the test, which meets the welding process requirements, and can still maintain stable tracking when the workpiece moves at a speed of "0.5m / s".

[0053] This invention discloses an industrial robot workpiece positioning method and device based on multi-camera information fusion. By using multi-camera collaboration, adaptive information fusion and advanced image processing technology, it effectively solves the problems of accuracy, anti-interference and real-time performance of traditional positioning methods, and provides industrial robots with a highly reliable and flexible workpiece following solution.

[0054] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0055] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0056] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0057] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A method for workpiece positioning in an industrial robot based on multi-camera information fusion, characterized in that, Includes the following steps: The target image is obtained by preprocessing real-time images of the workpiece captured by multiple cameras. Identify optical marker points in the target image and calculate the pose of a single workpiece in a single-camera coordinate system; The global pose of the workpiece is obtained by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weight. The robot is controlled to position and follow the workpiece based on the workpiece's global pose and the robot's current pose, including generating the desired speed and outputting joint torque. The process of acquiring real-time images of the workpiece from multiple cameras and preprocessing them to obtain the target image specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein the optical markers are arranged on the surface of the workpiece, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and / or edge enhancement based on bilateral filtering; The process of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical marker points in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece. The process of obtaining the global pose of the workpiece by weighting the pose of the individual workpiece with the corresponding single-camera weights specifically includes: Obtain the pose and measurement variance of each individual workpiece corresponding to each camera in the current multi-camera setup; Calculate the single-camera weights based on the measurement variance; The global pose of the workpiece is obtained by weighted fusion calculation based on the pose of the single workpiece and the weight of the single camera, wherein the covariance confidence of the global pose of the workpiece is the highest.

2. The industrial robot workpiece positioning method based on multi-camera information fusion according to claim 1, characterized in that, The calculation based on the workpiece's global pose and the robot's current pose to control the robot's positioning and following of the workpiece includes generating the desired velocity and outputting joint torque, specifically including: The pose error is calculated based on the global pose of the workpiece and the current pose of the robot, and the desired speed is output accordingly. The joint torque is output based on the speed error calculated between the global pose of the workpiece and the current pose of the robot. Based on the desired speed and joint torque, the joint angular velocity is obtained, and joint control commands are output to control the robot to position and follow the workpiece.

3. An industrial robot workpiece positioning system based on multi-camera information fusion, characterized in that, The system includes a memory and a processor. The memory includes a program for an industrial robot workpiece positioning method based on multi-camera information fusion according to any one of claims 1-2. When the program for the industrial robot workpiece positioning method based on multi-camera information fusion is executed by the processor, it performs the following steps: The target image is obtained by preprocessing real-time images of the workpiece captured by multiple cameras. Identify optical marker points in the target image and calculate the pose of a single workpiece in a single-camera coordinate system; The global pose of the workpiece is obtained by weighted calculation based on the pose of the individual workpiece and the corresponding single camera weight. The robot performs calculations based on the global pose of the workpiece and the current pose of the robot to control the robot to locate and follow the workpiece, including generating the desired speed and outputting joint torque.

4. The industrial robot workpiece positioning system based on multi-camera information fusion according to claim 3, characterized in that, The process of acquiring real-time images of the workpiece from multiple cameras and preprocessing them to obtain the target image specifically includes: Acquire real-time images of a workpiece simultaneously captured by multiple cameras, wherein optical markers are arranged on the surface of the workpiece, the size of the optical markers ranging from 4mm×4mm to 10mm×10mm, and the acquired real-time images cover all viewpoints of the current workpiece. The target image is obtained by preprocessing the acquired real-time image, wherein the preprocessing process includes denoising and edge enhancement based on bilateral filtering.

5. The industrial robot workpiece positioning system based on multi-camera information fusion according to claim 4, characterized in that, The process of identifying optical marker points in the target image and calculating the pose of a single workpiece in a single-camera coordinate system specifically includes: Detect optical marker points in the target image and extract corner pixel coordinates; The two-dimensional image coordinates of the current optical marker point are obtained based on the corner pixel coordinates; The pose of a single workpiece in the current single-camera coordinate system is calculated based on the two-dimensional image coordinates and the preset three-dimensional layout. The rotation matrix of the current workpiece pose is solved by minimizing the reprojection error, and the rotation matrix is ​​converted into Euler angles or quaternions to obtain the pose of the single workpiece.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a program for an industrial robot workpiece positioning method based on multi-camera information fusion. When the program is executed by a processor, it implements the steps of the industrial robot workpiece positioning method based on multi-camera information fusion as described in any one of claims 1 to 2.

7. An industrial robot workpiece positioning device based on multi-camera information fusion, characterized in that, An industrial robot workpiece positioning method based on multi-camera information fusion, applicable to any one of claims 1 to 2, wherein the apparatus comprises: The system comprises a vision acquisition unit, a workpiece, an image processing unit, an information fusion unit, and a robot control unit. Multiple optical markers are arranged on the surface of the workpiece to provide positioning features for the image; The vision acquisition unit includes multiple industrial cameras, which are used to simultaneously acquire workpiece images from different angles; The image processing unit is communicatively connected to an industrial camera and is used to preprocess the workpiece image and identify the optical marker points to obtain the pose of a single workpiece. The information fusion unit is communicatively connected to the image processing unit and is used to perform weighted fusion calculations based on the pose of a single workpiece to obtain the global pose of the workpiece. The robot control unit is communicatively connected to the information fusion unit and is used to position and follow the workpiece by adjusting the robot joint operation under the position-velocity dual closed-loop control strategy in conjunction with the global pose adjustment of the workpiece.

Images