Industrial robot grabbing and positioning system based on computer vision

Abstract

The invention discloses an industrial robot grabbing and positioning system based on computer vision, and relates to the technical field of grabbing and positioning, and the system comprises a projection calculation module which is used for determining a roller direction unit vector of a roller based on a calibration plate image, and constructing a projection data pair based on the calibration plate image and the roller direction unit vector; the data acquisition module is used for generating a one-dimensional cosine curve based on the projection data pair and the candidate angular frequency set of the protection window; the image correction module is used for performing coordinate correction on the pixel coordinates of the calibration plate image based on the one-dimensional cosine curve to obtain a corrected image; the key point extraction module is used for carrying out geometric feature extraction on the correction image to obtain a key point set; and the pose optimization module is used for determining the optimal pose of the calibration plate relative to the camera based on the internal reference matrix of the camera and the three-dimensional reference point set and the key point set of the calibration plate. The grabbing stability and reliability of the robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of grasping positioning, and in particular to an industrial robot grasping positioning system based on computer vision. BACKGROUND

[0002] In modern industrial production processes, industrial robots as core automation tools are widely used in precise grasping, positioning, assembly and other tasks. In complex production environments, robots need to accurately identify and grasp objects, especially under the influence of optical media such as protective windows. How to accurately correct the errors caused by periodic optical distortion is the key to ensuring the efficient operation of robots.

[0003] In the prior art, robot grasping positioning mainly relies on a vision system. However, in a complex environment, periodic errors of images such as ripple effects caused by protective windows can lead to reduced positioning accuracy, thereby affecting the grasping effect of the robot. Many traditional image correction methods cannot solve the image distortion caused by periodic optical effects, especially in high-precision industrial grasping tasks, accurate image correction cannot be achieved, resulting in limited grasping accuracy of the robot. SUMMARY

[0004] The purpose of the present application is to solve the problem of image distortion caused by periodic optical effects in the prior art, and to provide an industrial robot grasping positioning system based on computer vision.

[0005] In order to solve the problems existing in the prior art, the present application adopts the following technical solutions: The industrial robot grasping positioning system based on computer vision comprises: A projection calculation module is configured to determine a roll direction unit vector of a roll based on a calibration plate image, and to construct a projection data pair based on the calibration plate image and the roll direction unit vector. A data acquisition module is configured to generate a one-dimensional cosine curve based on the projection data pair and a candidate angular frequency set of the protective window. An image correction module is configured to perform coordinate correction on pixel coordinates of the calibration plate image based on the one-dimensional cosine curve, and to obtain a corrected image. A key point extraction module is configured to perform geometric feature extraction on the corrected image, and to obtain a key point set. A pose optimization module is configured to determine an optimal pose of the calibration plate relative to the camera based on an intrinsic matrix of the camera, a three-dimensional reference point set of the calibration plate, and the key point set. A grasping pose module is configured to determine a grasping pose of the industrial robot in a base coordinate system based on the optimal pose and a rigid transformation matrix of the industrial robot.

[0006] Preferably, determining the roll direction unit vector of the roll based on the calibration plate image comprises: The calibration board was photographed through the protective window using a camera to obtain an image of the calibration board; Based on calibration board images and image recognition technology, conventional camera calibration is performed on the camera's inherent intrinsic parameters to obtain the reprojection residuals of calibration points in the calibration board images; Set multiple different angles and directions; The projection value is obtained by performing a vector dot product operation on the reprojection residual and the unit vector of the angular direction. All projection values ​​are collected sequentially to obtain a one-dimensional residual sequence in the angular direction; Perform a Fourier transform on the one-dimensional residual sequence to obtain the one-dimensional frequency domain spectrum in the angular direction; Calculate the spectral energy of a one-dimensional frequency domain spectrum; The maximum spectral energy is obtained by filtering all spectral energies. The unit vector in the angular direction corresponding to the maximum spectral energy is determined as the roll direction unit vector of the roll.

[0007] Preferably, a projection data pair is constructed based on the calibration plate image and the roll direction unit vector, including: Determine the axial direction of the rolls; Based on the pixel coordinates of the calibration point in the calibration plate image and the roller direction unit vector, calculate the first projection value of the pixel coordinates in the axial direction. The second projection value is obtained by performing a vector dot product operation on the reprojection residual and the roll direction unit vector; Construct projection data pairs based on the first projection value and the second projection value.

[0008] Preferably, generating a one-dimensional cosine curve based on the projection data pair and the candidate angular frequency set of the protective window includes: A set of candidate angular frequencies for the protective window is generated based on the process error period range of the protective window. For each candidate angular frequency in the candidate angular frequency set, the periodogram energy value of the candidate angular frequency and the projected data pair is calculated using the periodogram method. From all the periodic chart energy values, select the maximum energy value; The candidate angular frequency corresponding to the maximum energy value is taken as the target angular frequency; All projected data pairs are compiled into a data pair sequence; Based on the target angular frequency, and by fitting the data pair sequence using the least squares method, a one-dimensional cosine curve is obtained.

[0009] Preferably, the pixel coordinates of the calibration board image are corrected based on a one-dimensional cosine curve to obtain a correction image, including: Parameters are extracted from the one-dimensional cosine curve to obtain the displacement amplitude and phase; The correction coordinates are calculated based on the displacement amplitude, phase, target angular frequency, and pixel coordinates of the calibration plate image; A calibration map is generated based on all the calibration coordinates.

[0010] Preferably, geometric features are extracted from the calibration image to obtain a set of key points, including: Edge gradient detection is performed on the correction image to obtain the edge region; Extract sub-pixel key points in the edge region; All sub-pixel keypoints are gathered into a keypoint set.

[0011] Preferably, determining the optimal pose of the calibration board relative to the camera based on the camera's intrinsic parameter matrix, the set of three-dimensional reference points of the calibration board, and the set of key points includes: The objective function of the calibration board is constructed based on the camera's intrinsic parameter matrix, the set of 3D reference points and key points of the calibration board; Construct the constraints for the objective function; Under constraints, the parameter set of the objective function is optimized to obtain the minimum value of the objective function; The optimal pose of the calibration board relative to the camera is determined by minimizing the objective function. The optimal pose includes the optimal rotation matrix and the optimal translation vector.

[0012] Preferably, determining the grasping pose of the industrial robot in the base coordinate system based on the optimal pose and the rigid body transformation matrix of the industrial robot includes: Obtain the rigid body transformation matrix from the camera to the industrial machine; By performing matrix multiplication on the rotation matrix of the rigid body transformation matrix and the optimal rotation matrix of the optimal pose, the rotation components of the industrial robot in the base coordinate system are obtained. Multiply the rotation matrix of the rigid body transformation matrix and the optimal translation vector of the optimal pose to obtain the translation vector after the rigid body transformation and rotation. Adding the translation vector of the rigid body transformation matrix to the translation vector after rigid body transformation and rotation yields the translation components of the industrial robot in the base coordinate system. Determine the complete pose of the industrial robot in the base coordinate system based on the rotation and translation components; A pose mapping transformation is performed on the complete pose of the object and the grasping reference of the calibration plate to obtain the grasping pose of the industrial robot in the base coordinate system.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. In this invention, a calibration board image and an image recognition method are used to identify and correct periodic optical errors caused by the protective window. By analyzing the reprojection residual in the calibration board image and combining it with image processing techniques such as Fourier transform, the source of periodic error in the image can be effectively identified, especially the ripple effect caused by the surface of the protective window. This allows for precise correction of positioning errors caused by optical distortion, improving the positioning and grasping accuracy of industrial robots in complex environments and solving the problem of the inability to accurately correct periodic optical distortion in existing technologies.

[0014] 2. In this invention, the pixel coordinate correction process of the calibration board image is optimized by constructing projection data pairs and fitting a one-dimensional cosine curve using the least squares method. By correcting the coordinates of the calibration board image, the periodic distortion caused by the protective window can be effectively eliminated, so that the actual features of the image can more accurately reflect the true position of the object. This avoids the problem of inaccurate positioning caused by distortion error in traditional methods, and provides industrial robots with more accurate grasping pose and pose optimization capabilities.

[0015] 3. In this invention, by combining the camera's intrinsic parameter matrix, the three-dimensional reference point set and key point set of the calibration board, the optimal pose of the calibration board is calculated through an optimization algorithm, which further improves the robot's grasping accuracy in complex environments. This not only improves the image correction accuracy but also ensures that the industrial robot can accurately locate the grasping target based on the optimized pose, thereby improving the stability and reliability of the robot's grasping. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a functional block diagram of an industrial robot grasping and positioning system based on computer vision, provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0018] Example: This example provides a computer vision-based industrial robot grasping and positioning system. See [link to example]. Figure 1 Specifically, including: The projection calculation module is used to determine the roll direction unit vector of the roll based on the calibration plate image, and to construct projection data pairs based on the calibration plate image and the roll direction unit vector. In an embodiment of the present invention, determining the roll direction unit vector of the roll based on the calibration plate image includes: The calibration board was photographed through the protective window using a camera to obtain an image of the calibration board; Specifically, an industrial camera is first selected as the image acquisition device. Then, a bracket with three-dimensional adjustment capabilities is used to securely mount the industrial camera on the outside of the protective window. The bracket is adjusted so that the central axis of the industrial camera lens is perpendicular to the plane of the protective window, and the lens faces inwards. A high-precision vacuum suction clamp is used to horizontally fix the calibration board on a precision work platform inside the protective window, ensuring that the geometric center of the calibration board and the optical axis of the industrial camera lens are on the same vertical line. Simultaneously, a lint-free cloth and a special cleaning agent are used to clean the surface of the calibration board. The shooting parameters are adjusted using the industrial camera's control software, setting the exposure time to the range of 50 to 100 milliseconds and the gain value to within 20 dB. The industrial camera is then controlled to start the shooting program at a preset shooting frequency of 5 frames per second, continuously acquiring images of the calibration board 10 times through the protective window. An image sharpness evaluation algorithm is used to evaluate each of the 10 acquired images, and the image with the highest sharpness score and no obvious geometric distortion is selected as the calibration board image for subsequent processing steps.

[0019] Based on calibration board images and image recognition technology, conventional camera calibration is performed on the camera's inherent intrinsic parameters to obtain the reprojection residuals of calibration points in the calibration board images; Specifically, firstly, the calibration board image is processed using a corner detection algorithm from image recognition technology to automatically identify all preset calibration points on the calibration board and record the two-dimensional coordinates of each calibration point in the image pixel coordinate system. Based on the calibration board's design drawings, the precise three-dimensional coordinates of each calibration point in the calibration board's own three-dimensional coordinate system are obtained. Using the Zhang Zhengyou calibration method, the three-dimensional coordinates of the calibration points and the corresponding image pixel coordinates are substituted into the mathematical model of camera calibration to construct an optimization objective function for the camera's intrinsic parameters. The objective function is iteratively optimized using the nonlinear least squares method to obtain the camera's intrinsic parameter matrix and distortion coefficients. Then, using the obtained intrinsic parameters and distortion coefficients, the three-dimensional coordinates of each calibration point on the calibration board are reprojected onto the image pixel coordinate system to obtain the reprojected coordinates of each calibration point. Finally, the reprojected coordinates of each calibration point are compared point-by-point with the pixel coordinates actually identified in the image, and the Euclidean distance between them is calculated. This distance is the reprojection residual of the calibration point in the calibration board image, and the reprojection residuals of all calibration points are summarized and recorded.

[0020] Set multiple different angles and directions; The projection value is obtained by performing a vector dot product operation on the reprojection residual and the unit vector of the angular direction. All projection values ​​are collected sequentially to obtain a one-dimensional residual sequence in the angular direction; Specifically, firstly, unit vectors corresponding to all angular directions within the range of 0 to 180 degrees are generated in a two-dimensional plane at fixed small angular intervals. Each unit vector is used to uniquely represent an angular direction. For each calibration point's reprojection residual vector and each generated angular direction unit vector, the vector dot product of the two is calculated to obtain the projection value of the reprojection residual of the calibration point in the corresponding angular direction. According to the order in which the angular directions are generated, the projection values ​​of all calibration points under each angular direction are collected sequentially, so that each angular direction corresponds to a set of projection values ​​arranged in the order of the calibration points, thereby obtaining a one-dimensional residual sequence corresponding to each angular direction.

[0021] In detail, a calibration board image is an image captured by a camera that contains a specific calibration pattern, such as a checkerboard or a dot array; calibration points are identifiable feature points in the calibration board image, usually the intersections of the calibration pattern or the positions of the dots. These calibration points are key points with known coordinates, used in the camera calibration process to calculate camera parameters and help determine the relationship between the image and physical space; reprojection residuals are the coordinates of the calibration points reprojected onto the image pixel coordinate system using camera intrinsic parameters, and the deviation between these coordinates and the actual identified calibration point pixel coordinates in the image, reflecting the error in the camera calibration or imaging process; projection values ​​represent the component of optical error caused by the protective window in that direction; and the one-dimensional residual sequence is a sequence formed by collecting the projection values ​​of the reprojection residuals of all calibration points onto the unit vector in the corresponding angular direction, in angular order, reflecting the overall distribution law of the residuals in that angular direction.

[0022] Perform a Fourier transform on the one-dimensional residual sequence to obtain the one-dimensional frequency domain spectrum in the angular direction; Calculate the spectral energy of a one-dimensional frequency domain spectrum; The maximum spectral energy is obtained by filtering all spectral energies. The unit vector in the angular direction corresponding to the maximum spectral energy is determined as the roll direction unit vector of the roll.

[0023] Specifically, firstly, a Fast Fourier Transform is performed on the one-dimensional residual sequence to transform it from the time domain to the frequency domain, thus obtaining the one-dimensional frequency domain spectrum corresponding to each angular direction. For each angular direction, the sum of the squares of the amplitudes of each frequency component in the spectrum is calculated to obtain the spectral energy of the one-dimensional frequency domain spectrum in that angular direction. The spectral energies corresponding to all angular directions are compared one by one, and the spectral energy with the largest value is selected from these spectral energies, which is the maximum spectral energy. The unit vector of the angular direction corresponding to the maximum spectral energy is found, and this unit vector is determined as the roll direction unit vector of the roll.

[0024] In detail, the unit vector of the angle direction corresponding to the maximum spectral energy is obtained by analyzing the spectral characteristics of periodic distortion in the image using Fourier transform. This unit vector represents the main direction of periodic optical error in the image, which is usually related to the optical effect caused by the protective window. In the spectrum, the direction corresponding to the energy peak of the frequency component is the main direction of distortion, reflecting the propagation or influence direction of the optical effect. The roll direction unit vector is the direction vector describing the influence of optical effect in the roll direction. In the tempering process, the pressing or processing of the roll surface will cause small undulations or ripples on the surface of the protective window. This ripple effect propagates along the roll direction. The roll direction unit vector points to this main ripple direction, indicating the propagation direction of the periodic error generated on the protective window surface.

[0025] In detail, the angular direction corresponding to the maximum spectral energy reflects the main direction of periodic errors in the image. Periodic errors are usually related to periodic optical effects caused by the protective window, such as roll ripples. Since the periodic optical effect manifests as periodic displacement in the image, its main direction is consistent with the angular direction corresponding to the maximum spectral energy. Therefore, the unit vector of the angular direction corresponding to the maximum spectral energy actually represents the main direction of periodic distortion caused by the protective window effect. By determining this unit vector as the roll-direction unit vector of the roll, the optical distortion caused by the protective window can be accurately corrected.

[0026] In detail, due to the tempering process, the protective window installed in front of the camera in the industrial production line introduces quasi-periodic undulating roller ripples along the roller axis. This causes the two surfaces of the protective window to have a slight, approximately sinusoidal thickness or surface undulation in this direction. During transmission imaging, this is equivalent to a sequence of microprisms arranged in this direction, which will produce a one-dimensional periodic angular deflection on the passing main light rays. The angular deflection projected onto the image plane will manifest as a periodic pixel displacement along a certain fixed direction. When the grasped part moves within the field of view, its pixel coordinates will oscillate cosine-like with its position, thus affecting the recognition stability and pose estimation accuracy. The roller direction is directly related to the periodic direction of the roller ripples of the protective window. Therefore, it is necessary to determine the roller direction unit vector based on the calibration board image to provide a key directional basis for subsequent correction of the periodic optical distortion caused by the protective window, improvement of image correction effect, and pose estimation accuracy.

[0027] In an embodiment of the present invention, a projection data pair is constructed based on the calibration plate image and the roll direction unit vector, including: Determine the axial direction of the rolls; Based on the pixel coordinates of the calibration point in the calibration plate image and the roller direction unit vector, calculate the first projection value of the pixel coordinates in the axial direction. The second projection value is obtained by performing a vector dot product operation on the reprojection residual and the roll direction unit vector; Construct projection data pairs based on the first projection value and the second projection value.

[0028] Specifically, the axial direction of the roll is the direction along which the roll's own axis extends; for each calibration point in the calibration plate image, the pixel coordinates of the calibration point in the calibration plate image are obtained, and the pixel coordinates are transposed and multiplied by the roll direction unit vector to obtain the first projection value of the calibration point in the axial direction; the reprojection residual of each calibration point is obtained, and the reprojection residual is multiplied by the roll direction unit vector to obtain the second projection value; the first projection value and the second projection value corresponding to each calibration point are combined to construct a projection data pair, ensuring that each calibration point corresponds to a projection data pair containing the first projection value and the second projection value.

[0029] In detail, a roll refers to a circular tool used in metal processing for extruding, rolling, or forming materials. In optics and image processing, the surface of the roll may cause minute ripples or irregularities on the protective window, which can affect the image quality passing through the protective window. The axial direction of the roll refers to the direction in which the central axis of the roll extends. The first projection value of the pixel coordinates in the axial direction is the result of projecting the pixel coordinate vector of the calibration point in the calibration plate image onto the unit vector in the axial direction of the roll, which is used to reflect the positional characteristics of the calibration point along the axial direction of the roll in the image. The second projection value is the result of projecting the reprojection residual vector of the calibration point onto the unit vector in the axial direction of the roll, which is used to reflect the distribution characteristics of the reprojection residual along the axial direction of the roll.

[0030] In general, due to the tempering process, the protective window has quasi-periodic roller ripples along the roll axis, which causes periodic pixel displacement along the roll axis in the imaging. The calibration plate image contains the pixel coordinate information of the calibration point, and the roll direction unit vector represents the axial direction of the roll. By calculating the first projection value of the calibration point pixel coordinates in the axial direction, and the second projection value obtained by the dot product of the reprojection residual and the roll direction unit vector, a projection data pair is constructed based on these two projection values. This provides accurate basic data support for the subsequent generation of a one-dimensional cosine curve to correct the pixel coordinates and eliminate the adverse effects of the periodic optical effect of the protective window on the imaging.

[0031] The data acquisition module is used to generate a one-dimensional cosine curve based on the projection data pairs and the candidate angular frequency set of the protective window; In an embodiment of the present invention, generating a one-dimensional cosine curve based on the projection data pair and the candidate angular frequency set of the protective window includes: A set of candidate angular frequencies for the protective window is generated based on the process error period range of the protective window. Specifically, firstly, through extensive detection and statistical analysis of the process errors in protective windows, the periodic range of process errors caused by tempering and other processes is obtained, and the minimum and maximum values ​​of the process error period are determined. Next, based on the mathematical relationship between angular frequency and period (angular frequency equals 2π divided by period), the minimum and maximum angular frequencies corresponding to the minimum and maximum process error periods are calculated, thus determining the range of angular frequency values. Then, within this range, multiple discrete angular frequency values ​​are generated sequentially at preset fixed angular frequency intervals of 0.1 rad / s, with the interval adjustable according to actual accuracy requirements. Finally, all generated discrete angular frequency values ​​are collected to form a candidate angular frequency set for the protective window, covering the possible range of angular frequencies corresponding to the process errors, providing comprehensive candidate data for subsequent selection of target angular frequencies.

[0032] For each candidate angular frequency in the candidate angular frequency set, the periodogram energy value of the candidate angular frequency and the projected data pair is calculated using the periodogram method. Specifically, firstly, all projection data pairs are preprocessed. The first projection value in each pair is used as the position parameter of the sampling point, and the second projection value is used as the sampling signal value under the corresponding position parameter. All projection data pairs are sorted in ascending order of the first projection value to ensure the continuity and order of the sampling signal in the position parameter dimension, avoiding subsequent calculation errors due to data disorder. Next, the process of traversing the candidate angular frequency set is started, selecting candidate angular frequencies in the set one by one as the current calculation object. For the current candidate angular frequency, the corresponding Fourier basis function is constructed based on its value. The Fourier basis function consists of cosine and sine terms, and the angular frequencies of both terms are consistent with the current candidate angular frequency. The cosine term is expressed as cos(ωx), and the sine term is expressed as sin(ωx), where x represents the first projection value in the sorted projection data pair. Then, for each sorted pair of projection data, calculate the product of the second projection value and the cosine term under the corresponding x value, and the product of the second projection value and the sine term under the corresponding x value for each pair of projection data. Summate the cosine terms of all pairs of projection data to obtain the sum of cosine coefficients, and sum the sine terms of all pairs of projection data to obtain the sum of sine coefficients. Then, square the sum of cosine coefficients and square the sum of sine coefficients, and add the two squared results to obtain the Fourier coefficient modulus square value corresponding to the current candidate angular frequency. Finally, count the total number of projection data pairs, divide the Fourier coefficient modulus square value by the total number, and obtain the periodogram energy value of the current candidate angular frequency and the projection data pair. Repeat the above calculation process for a single candidate angular frequency until the periodogram energy value calculation of all candidate angular frequencies in the candidate angular frequency set is completed.

[0033] From all the periodic chart energy values, select the maximum energy value; The candidate angular frequency corresponding to the maximum energy value is taken as the target angular frequency; All projected data pairs are compiled into a data pair sequence; Specifically, the process iterates through all the periodogram energy values ​​calculated by the periodogram method, compares the magnitude of each energy value, selects the energy value with the largest value and marks it as the maximum energy value; queries the candidate angular frequency corresponding to the maximum energy value and determines this candidate angular frequency as the target angular frequency; according to the generation or acquisition order of the projection data pairs, all projection data pairs are collected to construct an ordered data pair sequence, thus obtaining the data pair sequence.

[0034] Based on the target angular frequency, and by fitting the data pair sequence using the least squares method, a one-dimensional cosine curve is obtained.

[0035] Specifically, the form of the one-dimensional cosine curve to be fitted is first determined as follows: ,in Given the determined target angular frequency, Let the displacement amplitude be the value to be determined. For the phase to be determined, It is the first data pair in the sequence. The first projection value of a projection data pair It is the first The second projection value of each projection data pair It is an index; then, all projected data pairs in the data pair sequence are substituted into the curve form to construct information about the unknown parameters. and The error equation is then derived; based on the principle of least squares, the error equation is transformed into... and For the optimization problem of variables, by considering the sum of squared errors with respect to... and Taking the partial derivatives and setting them to zero yields the normal equations. Finally, solving these equations provides the displacement amplitude that minimizes the sum of squared errors for all data pairs. and phase The optimal solution is found by substituting it into the given one-dimensional cosine curve form, which yields the fitted one-dimensional cosine curve.

[0036] In detail, the periodogram energy value is a physical quantity calculated using the periodogram method. It reflects the intensity of the periodic component corresponding to a certain candidate angular frequency in the projected data pair. The higher the value, the more significant the periodicity of the projected data at that candidate angular frequency. It can be used to determine whether the candidate angular frequency is related to the periodic optical distortion caused by the protective window manufacturing error. The target angular frequency is the angular frequency corresponding to the maximum periodogram energy value selected from the candidate angular frequencies of the protective window. It represents the main frequency of the periodic optical distortion caused by the protective window manufacturing error in the projected data and is the core frequency parameter for subsequent fitting of the one-dimensional cosine curve. The one-dimensional cosine curve is a curve obtained by fitting the projected data pair sequence using the least squares method based on the target angular frequency. It uses the periodic variation law of the cosine function to characterize the periodic correlation between the first and second projected values ​​in the projected data pair. The correlation is essentially a manifestation of the periodic error of the protective window in the imaging data, providing key mathematical model support for subsequent periodic distortion correction of the pixel coordinates of the calibration board image.

[0037] The image correction module is used to perform coordinate correction on the pixel coordinates of the calibration board image based on a one-dimensional cosine curve to obtain a correction image. In an embodiment of the present invention, coordinate correction is performed on the pixel coordinates of the calibration board image based on a one-dimensional cosine curve to obtain a correction image, including: Parameters are extracted from the one-dimensional cosine curve to obtain the displacement amplitude and phase; The correction coordinates are calculated based on the displacement amplitude, phase, target angular frequency, and pixel coordinates of the calibration board image. The formula for calculating the correction coordinates is as follows: In the formula, Pixel coordinates Correction coordinates, It is the displacement amplitude. It is the target angular frequency. It is the unit vector in the roller direction. These are pixel coordinates. It is phase. It is the unit vector along the roller; Specifically, the pixel coordinates of the calibration board image include distortions caused by periodic process errors in the protective window, while The distortion displacement was quantified: the unit vector along the roller. This indicates the dominant direction of the error; the error of the protective window is periodically distributed along the roller direction. Calculate the projection of the pixel coordinates onto the roller direction to reflect its position along that direction. Combined with target angular frequency With phase This yields the phase state of the error at that location. This is the amplitude of the distortion displacement at this phase (because the error is periodic, the displacement amplitude changes with the phase according to a cosine law), multiplied by... This then becomes the distorted displacement vector along the roller direction. Since the distortion of the pixel coordinates originates from this periodic displacement, it is used... Subtracting this distorted displacement vector eliminates the influence of the protective window's periodic error on the pixel position, ultimately yielding corrected coordinates that reflect the real scene. This combination aligns with the physical process where the protective window's periodic process error causes the pixel to shift periodically along the roller direction in a cosine pattern. Coordinate correction is achieved by subtracting the offset in reverse.

[0038] A calibration map is generated based on all the calibration coordinates.

[0039] Specifically, firstly, the system iterates through the coordinates of each pixel in the calibration board image, and calculates the corresponding calibration coordinates for each pixel. Then, using image resampling technology, the pixel value at the location corresponding to the calibration coordinate is extracted from the calibration board image or obtained through interpolation. Finally, all pixels that have been mapped by the calibration coordinates and whose pixel values ​​have been obtained are rearranged and combined according to the spatial distribution of the calibration coordinates to generate a complete calibration image. This calibration image eliminates the pixel coordinate distortion caused by the periodic process error of the protective window and can more accurately reflect the true image information of the calibration board.

[0040] Specifically, due to periodic process errors in the protective window, the pixel coordinates of the calibration board image will be periodically distorted, failing to accurately reflect the true structure of the calibration board. However, the one-dimensional cosine curve obtained through the previous steps accurately depicts the pattern of pixel coordinate offset caused by this periodic error as the position changes. Therefore, based on this curve, the offset of each pixel caused by the error can be calculated, thereby correcting the distorted pixel coordinates and obtaining a corrected image that eliminates the influence of the periodic error of the protective window.

[0041] The key point extraction module is used to extract geometric features from the calibration image to obtain a set of key points; In an embodiment of the present invention, geometric feature extraction is performed on the correction image to obtain a set of key points, including: Edge gradient detection is performed on the correction image to obtain the edge region; Extract sub-pixel key points in the edge region; All sub-pixel keypoints are gathered into a keypoint set.

[0042] In detail, the edge region is the area in the calibration board calibration image where the pixel grayscale values ​​change drastically. Physically, it corresponds to the boundary of the actual structure of the calibration board, such as the dividing point of the marks and patterns on the calibration board. It is an important visual representation of the geometric structure of the calibration board. The sub-pixel key points in the edge region are key points with higher positional accuracy than pixels, obtained by sub-pixel level precision detection methods within the edge region. It more accurately locates the physical position of the edge of the calibration board structure and can provide more accurate geometric feature references for subsequent tasks such as determining the pose of the calibration board relative to the camera, thereby reducing pixel-level positioning errors and improving the accuracy of industrial robot grasping and positioning.

[0043] Specifically, the process begins with edge gradient detection on the calibration image. Using a pre-selected gradient operator, each pixel in the calibration image is traversed one by one, and the gradient values ​​in the horizontal and vertical directions are calculated for each pixel. Then, based on the distribution of gradient magnitude and gradient direction, a threshold for determining edge pixels is determined. Pixels with gradient magnitude greater than the threshold and gradient direction that meet the edge continuity requirement are marked as edge pixels, and all edge pixels form an edge region. For the obtained edge region, a sub-pixel interpolation algorithm is used to perform high-precision fitting on the gray values ​​of the pixels in the edge region and their neighboring pixels. By solving for the extreme points or intersections of the fitted curve, sub-pixel key points with sub-pixel-level accuracy in position coordinates are obtained. Finally, all sub-pixel key points extracted from the edge region are aggregated according to their spatial positions in the calibration image to form a key point set, providing basic data for subsequent steps.

[0044] It should be noted that edge gradient detection captures edge regions with drastic grayscale changes in the calibration board image, and then sub-pixel interpolation algorithms are used to obtain sub-pixel key points with higher accuracy than pixels. This allows for the precise extraction of geometric feature information of the calibration board from the visual image, providing core visual feature support for subsequent construction of objective functions based on camera imaging models and calculation of the pose relationship between the calibration board and the camera. This is a typical application of image-to-feature technology in computer vision.

[0045] The pose optimization module is used to determine the optimal pose of the calibration board relative to the camera based on the camera's intrinsic parameter matrix, the set of 3D reference points of the calibration board, and the set of key points. In embodiments of the present invention, determining the optimal pose of the calibration board relative to the camera based on the camera's intrinsic parameter matrix, the three-dimensional reference point set of the calibration board, and the key point set includes: The objective function of the calibration board is constructed based on the camera's intrinsic parameter matrix, the set of 3D reference points and key points of the calibration board. The expression of the objective function is as follows: In the formula, It is the first in the set of key points Sub-pixel key points, It is a reprojection function. It is the camera's intrinsic parameter matrix. It is the first in the set of three-dimensional reference points The three-dimensional reference point coordinates of each feature point It is the rotation matrix of the calibration plate relative to the camera. It is the translation vector of the calibration board relative to the camera. It is the square of the Euclidean distance. It is the identifier of sub-pixel key points. This means that the coordinates of the three-dimensional reference point are projected onto the image plane of the calibration map through the camera's intrinsic parameter matrix, the rotation matrix of the calibration plate relative to the camera, and the translation vector of the calibration plate relative to the camera; In detail, the camera's intrinsic parameter matrix is ​​a matrix describing the camera's optical characteristics and imaging geometry. It includes the camera's focal length, principal point coordinates, and distortion coefficients. The intrinsic parameter matrix converts three-dimensional world coordinates into two-dimensional image coordinates and is the fundamental data in the camera calibration process, ensuring a correct mapping from the physical world to the image plane. The calibration board's three-dimensional reference point set refers to the set of all known three-dimensional coordinate points on the calibration board. These points are used to correct the camera's intrinsic and extrinsic parameters by corresponding to pixel coordinates in the image. The calibration board's rotation matrix relative to the camera is a matrix representing the rotational relationship between the camera coordinate system and the calibration board coordinate system. It describes the orientation change of the calibration board relative to the camera. Through this rotation matrix, the three-dimensional coordinates of the calibration board can be transformed from the calibration board coordinate system to the camera coordinate system. The calibration board's translation vector relative to the camera describes the position of the calibration board in the camera coordinate system, representing the amount of translation of the calibration board relative to the camera.

[0046] Specifically, when the camera images the calibration board, the three-dimensional reference points of the calibration board need to be transformed into the camera coordinate system through the rotation matrix and translation vector of the calibration board relative to the camera. Then, combined with the camera intrinsic parameter matrix, and based on the physical laws of pinhole imaging, the points are projected onto the image plane of the calibration image through a reprojection function to obtain the theoretical image pixels. The sub-pixel keypoints in the keypoint set are the image pixels corresponding to the three-dimensional reference points actually extracted from the calibration image. Affected by imaging noise, lens distortion, and feature extraction errors, there is a deviation between the actual sub-pixel keypoints and the theoretical projected pixels. In order to make the pose of the calibration board relative to the camera and the camera intrinsic parameter matrix most accurately reflect the real imaging relationship, an objective function needs to be constructed. By minimizing the sum of the squared Euclidean distances between all sub-pixel keypoints and the theoretical pixels obtained by reprojection of the corresponding three-dimensional reference points, the least squares optimization idea is used to find the optimal rotation matrix and translation vector, so that the theoretical projection result coincides with the actual sub-pixel keypoints as much as possible. This provides a reliable mathematical model support for the subsequent determination of the calibration board pose and the precise grasping and positioning of the industrial robot.

[0047] Construct the constraints for the objective function; Under constraints, the parameter set of the objective function is optimized to obtain the minimum value of the objective function; The optimal pose of the calibration board relative to the camera is determined by minimizing the objective function. The optimal pose includes the optimal rotation matrix and the optimal translation vector.

[0048] Specifically, firstly, constraints on the objective function are constructed. These constraints are determined by combining the physical laws of camera imaging with the mathematical characteristics of the calibration board's pose parameters. For example, the rotation matrix must satisfy the constraint of orthogonality and a determinant of 1 to ensure the rationality of spatial rotation transformation, and the value of the translation vector must conform to the physical constraints of the actual relative position between the calibration board and the camera. Under these constraints, the selected optimization algorithm is used to iteratively optimize the parameter set of the objective function. The parameter set includes the rotation matrix and translation vector of the calibration board relative to the camera. By continuously adjusting the values ​​of each parameter in the parameter set, the objective function gradually approaches its minimum value. Finally, when the objective function reaches its minimum value, the corresponding rotation matrix and translation vector are the optimal pose of the calibration board relative to the camera. The optimal rotation matrix accurately describes the spatial rotation relationship between the calibration board and the camera, and the optimal translation vector accurately describes the spatial translation relationship between the calibration board and the camera.

[0049] The grasping pose module is used to determine the grasping pose of the industrial robot in the base coordinate system based on the optimal pose and the rigid body transformation matrix of the industrial robot.

[0050] In embodiments of the present invention, determining the grasping pose of the industrial robot in the base coordinate system based on the optimal pose and the rigid body transformation matrix of the industrial robot includes: Obtain the rigid body transformation matrix from the camera to the industrial machine; By performing matrix multiplication on the rotation matrix of the rigid body transformation matrix and the optimal rotation matrix of the optimal pose, the rotation components of the industrial robot in the base coordinate system are obtained. In detail, the rigid body transformation matrix of an industrial machine is a mathematical representation used to describe the position and orientation changes of the end effector of an industrial robot in three-dimensional space. It consists of a rotation matrix and a translation vector, used to describe the transformation of the robot's end effector from one coordinate system to another. The rotation matrix, a part of the rigid body transformation matrix, describes the rotation of the object about a certain axis in three-dimensional space, reflecting the directional change between the object's coordinate system and the target coordinate system, and is usually defined by rotation angle and rotation axis. The optimal rotation matrix for the optimal pose is determined by an optimization algorithm, representing the best position and orientation of the industrial robot under given task or constraint conditions. This rotation matrix enables the robot's end effector to interact with the environment in an optimal way and achieve precise grasping and manipulation tasks.

[0051] Specifically, the rigid body transformation matrix from the camera to the industrial robot is first obtained. This rigid body transformation matrix is ​​determined through a hand-eye calibration process and carries the rotation and translation correspondence between the camera coordinate system and the industrial robot base coordinate system. The rotation matrix in the rigid body transformation matrix is ​​extracted, and the optimal rotation matrix in the optimal pose of the calibration plate relative to the camera is also extracted. The rotation matrix of the rigid body transformation matrix and the optimal rotation matrix are then multiplied. By utilizing the linear fusion property of matrix multiplication for spatial rotation relationships, the rotation component of the industrial robot in the base coordinate system is obtained. This rotation component can accurately characterize the rotation state of the calibration plate relative to the industrial robot base coordinate system after being associated with the camera.

[0052] Specifically, the purpose of matrix multiplication between the rotation matrix of the rigid body transformation matrix and the optimal rotation matrix of the optimal pose is to obtain the rotation components of the industrial robot in the base coordinate system. The rotation matrix of the rigid body transformation matrix describes the rotation state of the end effector of the industrial robot at its current position, while the optimal rotation matrix of the optimal pose represents the best rotation state of the robot's end effector under task or constraint conditions. Combining these two rotation matrices yields a new rotation matrix, which reflects the comprehensive rotation effect of the robot's end effector relative to the base coordinate system. This calculation method ensures that the robot's rotation state can be precisely adjusted according to task requirements and can be seamlessly integrated with the base coordinate system, thereby ensuring that the industrial robot can perform precise operations according to the predetermined trajectory and posture.

[0053] Multiply the rotation matrix of the rigid body transformation matrix and the optimal translation vector of the optimal pose to obtain the translation vector after the rigid body transformation and rotation. Adding the translation vector of the rigid body transformation matrix to the translation vector after rigid body transformation and rotation yields the translation components of the industrial robot in the base coordinate system. Specifically, the translation vector of the rigid body transformation matrix is ​​a mathematical representation of the translational displacement of the end effector of an industrial robot relative to the base coordinate system. It indicates the translational part from one coordinate system to another, that is, the change in position of the object in three-dimensional space. The rigid body transformation matrix consists of a rotation matrix and a translation vector, where the translation vector defines the linear displacement between coordinate systems. It is usually represented in the form of a three-dimensional vector, which includes the displacement in the X, Y and Z directions. Through the translation vector, the position of the robot end effector can be accurately transformed from the local coordinate system to the global base coordinate system, ensuring that the robot can accurately locate and perform tasks in three-dimensional space.

[0054] Specifically, in the process of determining the pose association between the camera and the industrial robot and calculating the translation components of the calibration board relative to the robot's base coordinate system, in order to adapt the translation vector of the calibration board relative to the camera to the rotation characteristics of the camera to the industrial robot, it is necessary to first obtain the translation vector after rigid body transformation by multiplying the rotation matrix of the rigid body transformation matrix with the optimal translation vector of the optimal pose. The translation vector of the rigid body transformation matrix represents the translation relationship from the camera coordinate system to the industrial robot's base coordinate system. By adding the translation vector after rigid body transformation to this translation vector, the translation of the calibration board relative to the camera, the rotation of the camera relative to the robot, and the translation relationship can be integrated according to the superposition principle of coordinate system pose transformation, thereby accurately obtaining the translation components of the industrial robot in the base coordinate system, providing accurate position parameter support for the industrial robot to carry out precise grasping and other operations based on the calibration board.

[0055] Determine the complete pose of the industrial robot in the base coordinate system based on the rotation and translation components; A pose mapping transformation is performed on the complete pose of the object and the grasping reference of the calibration plate to obtain the grasping pose of the industrial robot in the base coordinate system.

[0056] In detail, the gripping reference of the calibration board refers to the reference coordinate system defined on the calibration board for the robot's gripping operation. This reference is a specific set of calibration points or geometric features on the calibration board, serving as the target for the robot's end effector to perform gripping operations. The gripping reference provides a fixed reference frame, enabling the robot to accurately identify the position and orientation of the calibration board and perform precise gripping tasks accordingly. The gripping pose refers to the combination of spatial position and orientation required by the robot's end effector when performing gripping operations. It consists of two components: position and orientation. The gripping pose ensures that the robot can contact and grip the target object in the correct spatial position and orientation, thereby achieving precise gripping tasks.

[0057] Specifically, based on the obtained rotation and translation components of the industrial robot in the base coordinate system, and following the mathematical expression standard of rigid body pose, the rotation matrix corresponding to the rotation component and the translation vector corresponding to the translation component are integrated to construct a homogeneous transformation matrix that can completely describe the pose of the industrial robot relative to the base coordinate system. This determines the complete pose of the industrial robot in the base coordinate system. Subsequently, since the grasping reference on the calibration board has a predefined pose relationship relative to the calibration board, the homogeneous transformation matrix corresponding to the complete pose of the object is used to perform a coordinate transformation operation on the pose of the grasping reference relative to the calibration board. That is, the pose of the grasping reference is mapped from the calibration board coordinate system to the base coordinate system of the industrial robot through the transformation relationship of the complete pose of the object. This yields the grasping pose of the industrial robot in the base coordinate system, providing accurate pose commands for the industrial robot to perform grasping operations subsequently.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A computer vision-based industrial robot grasping and positioning system, characterized in that, include: The projection calculation module is used to determine the roll direction unit vector of the roll based on the calibration plate image, and to construct projection data pairs based on the calibration plate image and the roll direction unit vector. The data acquisition module is used to generate a one-dimensional cosine curve based on the projection data pairs and the candidate angular frequency set of the protective window; The image correction module is used to perform coordinate correction on the pixel coordinates of the calibration board image based on a one-dimensional cosine curve to obtain a correction image. The key point extraction module is used to extract geometric features from the calibration image to obtain a set of key points; The pose optimization module is used to determine the optimal pose of the calibration board relative to the camera based on the camera's intrinsic parameter matrix, the set of 3D reference points of the calibration board, and the set of key points. The grasping pose module is used to determine the grasping pose of the industrial robot in the base coordinate system based on the optimal pose and the rigid body transformation matrix of the industrial robot.

2. The industrial robot grasping and positioning system based on computer vision according to claim 1, characterized in that, Determining the roll direction unit vector based on the calibration plate image includes: The calibration board was photographed through the protective window using a camera to obtain an image of the calibration board; Based on calibration board images and image recognition technology, conventional camera calibration is performed on the camera's inherent intrinsic parameters to obtain the reprojection residuals of calibration points in the calibration board images; Set multiple different angles and directions; The projection value is obtained by performing a vector dot product operation on the reprojection residual and the unit vector of the angular direction. All projection values ​​are collected sequentially to obtain a one-dimensional residual sequence in the angular direction; Perform a Fourier transform on the one-dimensional residual sequence to obtain the one-dimensional frequency domain spectrum in the angular direction; Calculate the spectral energy of a one-dimensional frequency domain spectrum; The maximum spectral energy is obtained by filtering all spectral energies. The unit vector in the angular direction corresponding to the maximum spectral energy is determined as the roll direction unit vector of the roll.

3. The computer vision-based industrial robot grasping and positioning system according to claim 2, characterized in that, Projected data pairs are constructed based on the calibration plate image and the roll direction unit vector, including: Determine the axial direction of the rolls; Based on the pixel coordinates of the calibration point in the calibration plate image and the roller direction unit vector, calculate the first projection value of the pixel coordinates in the axial direction. The second projection value is obtained by performing a vector dot product operation on the reprojection residual and the roll direction unit vector; Construct projection data pairs based on the first projection value and the second projection value.

4. The industrial robot grasping and positioning system based on computer vision according to claim 1, characterized in that, A one-dimensional cosine curve is generated based on the candidate angular frequency set of the projection data pairs and the protective window, including: A set of candidate angular frequencies for the protective window is generated based on the process error period range of the protective window. For each candidate angular frequency in the candidate angular frequency set, the periodogram energy value of the candidate angular frequency and the projected data pair is calculated using the periodogram method. From all the periodic chart energy values, select the maximum energy value; The candidate angular frequency corresponding to the maximum energy value is taken as the target angular frequency; All projected data pairs are compiled into a data pair sequence; Based on the target angular frequency, and by fitting the data pair sequence using the least squares method, a one-dimensional cosine curve is obtained.

5. The industrial robot grasping and positioning system based on computer vision according to claim 4, characterized in that, The pixel coordinates of the calibration board image are corrected based on a one-dimensional cosine curve to obtain a correction image, including: Parameters are extracted from the one-dimensional cosine curve to obtain the displacement amplitude and phase; The correction coordinates are calculated based on the displacement amplitude, phase, target angular frequency, and pixel coordinates of the calibration plate image; A calibration map is generated based on all the calibration coordinates.

6. The industrial robot grasping and positioning system based on computer vision according to claim 1, characterized in that, Geometric feature extraction is performed on the calibration image to obtain a set of key points, including: Edge gradient detection is performed on the correction image to obtain the edge region; Extract sub-pixel key points in the edge region; All sub-pixel keypoints are gathered into a keypoint set.

7. The industrial robot grasping and positioning system based on computer vision according to claim 1, characterized in that, The optimal pose of the calibration board relative to the camera is determined based on the camera's intrinsic parameter matrix, the 3D reference point set of the calibration board, and the key point set, including: The objective function of the calibration board is constructed based on the camera's intrinsic parameter matrix, the set of 3D reference points and key points of the calibration board; Construct the constraints for the objective function; Under constraints, the parameter set of the objective function is optimized to obtain the minimum value of the objective function; The optimal pose of the calibration board relative to the camera is determined by minimizing the objective function. The optimal pose includes the optimal rotation matrix and the optimal translation vector.

8. The computer vision-based industrial robot grasping and positioning system according to claim 1, characterized in that, The grasping pose of the industrial robot in the base coordinate system is determined based on the optimal pose and the rigid body transformation matrix of the industrial robot, including: Obtain the rigid body transformation matrix from the camera to the industrial machine; By performing matrix multiplication on the rotation matrix of the rigid body transformation matrix and the optimal rotation matrix of the optimal pose, the rotation components of the industrial robot in the base coordinate system are obtained. Multiply the rotation matrix of the rigid body transformation matrix and the optimal translation vector of the optimal pose to obtain the translation vector after the rigid body transformation and rotation. Adding the translation vector of the rigid body transformation matrix to the translation vector after rigid body transformation and rotation yields the translation components of the industrial robot in the base coordinate system. Determine the complete pose of the industrial robot in the base coordinate system based on the rotation and translation components; A pose mapping transformation is performed on the complete pose of the object and the grasping reference of the calibration plate to obtain the grasping pose of the industrial robot in the base coordinate system.

Images