Industrial robot pose calibration method, device and equipment and storage medium

By identifying and filtering corner locations with high confidence using a corner recognition model, the problem of unstable accuracy in visual measurement systems in complex industrial scenarios is solved, achieving higher accuracy and more stable pose calibration.

CN119048606APending Publication Date: 2024-11-29HEBEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411267071.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing positioning instruments have unstable measurement accuracy in complex industrial scenarios, and are prone to missed detections or false detections, especially camera systems based on vision measurement principles.

Method used

A corner recognition model is used to process the end effector image of an industrial robot, identify corner locations, and select target corners with high confidence for pose calibration.

Benefits of technology

It improves positioning accuracy and detection stability, reduces missed or false detections, and ensures the accuracy of industrial robot pose calibration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048606A_ABST
    Figure CN119048606A_ABST
Patent Text Reader

Abstract

The invention discloses a pose calibration method, device and equipment of an industrial robot and a storage medium, and relates to the technical field of computers.The method comprises the steps that a first lattice point image of the tail end of a mechanical arm of the industrial robot is obtained, and the first lattice point image is input into an angular point recognition model; the angular point position comprises an estimated position and an offset position of the angular point, and the angular point identification model is obtained by training based on a sample image, a sample position of the angular point in the sample image and a sample confidence coefficient corresponding to the sample position; the sample position is determined by the sample estimation position and the sample deviation position, screening out a first target corner point of which the confidence coefficient is greater than a preset confidence coefficient threshold value from the first corner points, and calibrating the pose of the industrial robot according to the corner point position of the first target corner point. According to the method, the precision can be improved, the stability of the detection method is improved, and missing detection or false detection is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for calibrating the posture of an industrial robot. Background Art

[0002] In the field of computer technology, the accuracy of positioning instruments plays a vital role in ensuring the accuracy of the production process. At present, the positioning instruments on the market can be divided into two categories based on their working principles and technical characteristics: laser triangulation instruments represented by laser trackers, and camera systems based on visual measurement principles.

[0003] Laser tracker, as a model of high-precision positioning technology, is based on the use of laser triangulation. This method emits a laser beam to the target object and receives the light reflected from the surface of the object. It uses the information such as the emission angle and reception angle of the laser beam and the propagation time or phase difference of the laser beam in the air to construct a triangular geometric relationship, thereby accurately calculating the three-dimensional coordinates of the robot end effector or any point in space. However, at least two devices are required to achieve this goal, and the equipment is expensive. For this reason, visual measuring instruments can be used for measurement.

[0004] Visual measuring instruments have emerged in recent years with the rapid development of computer vision technology. They rely on the precise calibration of cameras and robot systems, capture images within the robot's working area, and use image processing algorithms to extract key feature points, thereby deriving the spatial position of the robot's end.

[0005] Although the cost is low, the measurement accuracy is easily affected by the external environment. For example, in complex industrial scenes such as blurred images taken at industrial sites and uneven lighting, the stability is not high, which may lead to missed detections or false detections. Summary of the invention

[0006] The present application provides a method, device, equipment and storage medium for calibrating the posture of an industrial robot, which can improve the accuracy, improve the stability of the detection method, and reduce the occurrence of missed detection or false detection.

[0007] In order to achieve the above purpose, this application adopts the following technical solutions:

[0008] In a first aspect, the present application provides a method for calibrating a posture of an industrial robot, the method comprising:

[0009] Acquire a first grid point image of the end of the robotic arm of the industrial robot;

[0010] Inputting the first grid point image into a corner point recognition model, obtaining a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point; wherein the corner point recognition model is trained based on a sample image, sample positions of corner points in the sample image, and sample confidence levels corresponding to the sample positions, and the sample positions are determined by the sample estimated position and the sample offset position;

[0011] Filter out a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point;

[0012] The position and posture of the industrial robot are calibrated according to the corner point position of the first target corner point.

[0013] In some possible implementations, inputting the first grid point image to a corner point recognition model to obtain a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position includes:

[0014]

[0015] Wherein, Image represents the first grid point image, Function(Image) represents the corner point position of the first corner point output by the corner point recognition model for the first grid point image and the confidence of the corner point position, σ is the scaling factor, x map Indicates the X-direction coordinate of the estimated position of the corner point, y map represents the Y-direction coordinate of the estimated position in the corner point position, represents the X-direction offset of the offset position in the corner point position, Indicates the Y-direction offset of the offset position in the corner point position, Indicates the confidence of the corner point position.

[0016] In some possible implementations, acquiring a first grid point image of a robot's mechanical arm end includes:

[0017] Acquire a first grid point image of the end of the robot's mechanical arm when the robot is in the first position;

[0018] The method further comprises:

[0019] Acquire a second grid point image of the end of the robot's mechanical arm when the robot is at the second position;

[0020] Determining a corner point position of a second target corner point according to the second grid point image;

[0021] The moving trajectory of the end of the robot arm is drawn according to the corner point position of the first target corner point and the corner point position of the second target corner point.

[0022] In some possible implementations, the loss function used by the corner point extraction model is:

[0023]

[0024] Among them, S is the predicted corner point position set for the sample image, L is the sample position set corresponding to the sample image, K(S,L) is the loss value of the loss function, and x i is the x coordinate of the sample position corresponding to the i-th sample image, x j is the x coordinate of the sample position corresponding to the jth sample image, y i is the y coordinate of the sample position corresponding to the i-th sample image, y j is the y coordinate of the sample position corresponding to the jth sample image, is the x-coordinate of the predicted corner point position corresponding to the i-th sample image, is the x-coordinate of the predicted corner point position corresponding to the j-th sample image, is the y coordinate of the predicted corner point position corresponding to the i-th sample image, is the y coordinate of the predicted corner point position corresponding to the jth sample image, is the prediction confidence of the predicted corner point position corresponding to the i-th sample image, is the prediction confidence of the predicted corner point position corresponding to the jth sample image, and λ is the scale adjustment coefficient.

[0025] In some possible implementations, the sample image includes an initial sample image and a processed sample image, and the processed sample image is obtained by adding noise to the initial sample image.

[0026] In some possible implementations, the method of adding noise includes: performing translation, rotation, perspective and distortion processing on the image.

[0027] In some possible implementations, the corner point position is represented by corner point coordinates, the estimated position is represented by an integer part of the corner point coordinates, and the offset position is represented by a decimal part of the corner point coordinates.

[0028] In a second aspect, the present application provides a posture calibration device for an industrial robot, the device comprising:

[0029] An acquisition module, used for acquiring a first grid point image of the end of a mechanical arm of an industrial robot;

[0030] A prediction module, used for inputting the first grid point image into a corner point recognition model, obtaining a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point; wherein the corner point recognition model is trained based on a sample image, sample positions of corner points in the sample image, and sample confidence levels corresponding to the sample positions, and the sample positions are determined by the sample estimated position and the sample offset position;

[0031] A screening module, used to screen out a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point;

[0032] A calibration module is used to calibrate the position and posture of the industrial robot according to the corner point position of the first target corner point.

[0033] In a third aspect, the present application provides a computing device, including a memory and a processor;

[0034] One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the computing device executes the method as described in any one of the first aspects.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method as described in any one of the first aspects.

[0036] It can be seen from the above technical solution that the present application has at least the following beneficial effects:

[0037] In the present application, since the corner point recognition model is trained based on the sample image, the sample position of the corner point in the sample image and the sample confidence corresponding to the sample position, and the sample position is determined by the sample estimated position and the sample offset position, the first grid point image is input into the corner point recognition model to obtain the corner point position of at least one first corner point in the first grid point image and the confidence of the corner point position, and the corner point position includes the estimated position and the offset position of the corner point. The traditional solution is to use the pixel position of the corner point as a label to determine whether the pixel point is a corner point, or to make a Gaussian circle with the corner point position as the center of the circle, and the neural network calculates the approximate position of the Gaussian circle and then approximates the coordinates of the maximum Gaussian response in the circle by mathematical methods to obtain the corner point position. There are large errors in the position prediction of the corner point in these two methods, and the more accurate position of the corner point cannot be determined. It can be seen that in the present application, the output of the neural network is cleverly matched to the corner point position by adding an integer coordinate to an offset, and non-corner points are eliminated by the probability corresponding to each point. It can be seen that the solution of the present application can improve the accuracy, improve the stability of the detection method, and reduce the occurrence of missed detection or false detection. After the position and posture of the industrial robot are calibrated based on more accurate corner point positions, the calibration effect is better.

[0038] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a corner point position provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0041] Figure 3 A flowchart of a method for calibrating the posture of an industrial robot provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of an industrial robot positioning system provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a corner point recognition model structure provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of obtaining a corner point position through a corner point recognition model provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of a posture calibration device for an industrial robot provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The terms "first", "second", "third", etc. in the specification of this application and the accompanying drawings are used to distinguish different objects rather than to limit a specific order.

[0048] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0049] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0050] Neural Networks (NN) is a complex network system formed by a large number of simple processing units (called neurons) that are widely interconnected. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for dealing with imprecise and fuzzy information processing problems that require considering many factors and conditions at the same time. The development of neural networks is related to neuroscience, mathematical science, cognitive science, computer science, artificial intelligence, information science, cybernetics, robotics, microelectronics, etc., and is an emerging frontier interdisciplinary subject.

[0051] Corner points are very important features of images and play an important role in understanding and analyzing graphics in images. Figure 1As shown in FIG. 1 , the figure is a schematic diagram of a corner point position provided by an embodiment of the present application. While retaining the important features of the graphics in the image, the corner point can effectively reduce the amount of information data, making its information content very high, effectively improving the speed of calculation, facilitating reliable matching of images, and making real-time processing possible.

[0052] At present, the traditional solution is to use the pixel position of the corner point as a label to determine whether the pixel point is a corner point, or to make a Gaussian circle with the corner point position as the center. The neural network calculates the approximate position of the Gaussian circle and then uses mathematical methods to approximate the coordinates of the maximum Gaussian response in the circle to obtain the corner point position. This will result in the position of the corner point being inaccurate, and the stability is not high in complex environments, and there is a possibility of missed detection or false detection.

[0053] In view of this, the embodiment of the present application provides a method for calibrating the posture of an industrial robot. In this method, since the corner point recognition model is trained based on the sample image, the sample position of the corner point in the sample image and the sample confidence corresponding to the sample position, and the sample position is determined by the sample estimated position and the sample offset position, the first grid point image is input into the corner point recognition model to obtain the corner point position of at least one first corner point in the first grid point image and the confidence of the corner point position, and the corner point position includes the estimated position and the offset position of the corner point. The traditional scheme is to use the integer pixel position of the corner point as a label to determine whether the pixel point is a corner point, or to make a Gaussian circle with the corner point position as the center of the circle, and the neural network calculates the approximate position of the Gaussian circle and approximates the coordinates of the maximum Gaussian response in the circle by mathematical methods to obtain the corner point position. There are large errors in the prediction of the position of the corner point in these two methods, and the more accurate position of the corner point cannot be determined. It can be seen that in the present application, the output of the neural network is cleverly matched to the corner point position by adding the integer coordinates to the offset, and the non-corner points are eliminated by the probability corresponding to each point. It can be seen that the solution of the present application can improve the accuracy, improve the stability of the detection method, and reduce the occurrence of missed detection or false detection. After the position and posture of the industrial robot are calibrated based on more accurate corner point positions, the calibration effect is better.

[0054] In order to make the technical solution of the present application clearer and easier to understand, the following introduces an application scenario provided by the embodiment of the present application by introducing the posture calibration process of the above-mentioned industrial robot. Figure 2As shown, the figure is a schematic diagram of an application scenario provided by an embodiment of the present application. In this scenario, the binocular measurement device 201 obtains a first grid point image at the end of the mechanical arm 202 of the industrial robot. Subsequently, the first grid point image is input into a pre-trained corner point recognition model. The task of the corner point recognition model is to identify the corner point position of at least one first corner point in the first grid point image, and at the same time give the confidence of these corner point positions, wherein the corner point position specifically includes the estimated position and offset position of the corner point. The corner point recognition model is trained based on a large number of sample images, which contain known corner point sample positions and their corresponding confidences. During the training process, the sample position is determined by the sample estimated position and the sample offset position. After the corner point recognition model processes the first grid point image, it outputs a series of first corner point coordinate positions and their confidences. From these corner points, the first target corner point with a confidence greater than a preset confidence threshold is selected. According to the selected corner point positions of the first target corner point, the posture of the industrial robot is calibrated.

[0055] In order to make the technical solution of the present application clearer and easier to understand, the following describes a method for calibrating the posture of an industrial robot provided in an embodiment of the present application in conjunction with the accompanying drawings. Figure 3 As shown in FIG. 1 , the figure is a flow chart of a posture calibration method of an industrial robot provided in an embodiment of the present application. The posture calibration method of the industrial robot can be executed by a processing device, and the method includes:

[0056] S301: A processing device obtains a first grid point image of a robotic arm end of an industrial robot.

[0057] like Figure 4 As shown, the figure is a schematic diagram of an industrial robot positioning system provided by an embodiment of the present application. The positioning system includes an industrial robot and a binocular measurement device, the industrial robot includes a mechanical arm 401 and a chessboard 406 at the end 402 of the mechanical arm, and the binocular measurement device includes a first camera 403, a second camera 407, a light source 404 and a camera bracket 405. After the industrial robot reaches the specified position, the first camera 403 and the second camera 407 of the binocular measurement device can capture images of the chessboard 406 on the end 402 of the mechanical arm of the industrial robot, and then obtain a first grid point image, and the binocular measurement device can transmit the first grid point image to the processing device, and then the processing device can obtain the first grid point image.

[0058] It should be noted that the present application does not specifically limit the manner in which the processing device acquires the first grid point image. Those skilled in the art may select a manner in which the first grid point image is acquired based on actual needs. For example, when the processing device has an image acquisition function, the processing device may also directly acquire an image of the chessboard 406 to obtain the first grid point image.

[0059] S302: The processing device inputs the first grid point image into a corner point recognition model to obtain a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position.

[0060] The corner point position is composed of the corner point estimated position and the corner point offset position.

[0061] The corner point recognition model is used to identify corner points in an image. Corner points are points in an image where the brightness changes dramatically or points where the edge curve of an image has a maximum value. Corner points retain important features of the image and play a key role in the understanding, analysis, and subsequent processing of the image.

[0062] like Figure 5 As shown, this figure is a schematic diagram of a corner point recognition model structure provided in an embodiment of the present application. The corner point recognition model provided in the embodiment of the present application recognizes the corner points in the input image by identifying the estimated position and offset position of the corner points in the input image, as well as the confidence of the corner point position composed of the estimated position and the offset position. Even when the input image is distorted or deformed, the corner points in the input image can be accurately recognized.

[0063] Exemplarily, the processing device may input the acquired first grid point image Image into the corner point recognition model, and after being processed by the corner point recognition model, the corner point recognition model outputs the first estimated feature map X map , the first offset feature map BX map , the second estimated feature map Y map , the second offset feature map BY map And the probability feature map S map The processing device then converts the first offset feature map BX map After multiplying by σ, it is then combined with the first estimated feature map X map Add them together to get the X coordinate of the corner point position, and use the second offset feature map BY map After multiplying by σ, it is then combined with the second estimated feature map Y map The processing device can combine the X coordinate, Y coordinate and confidence level of each corner point to obtain the corner point position and the confidence level of each corner point.

[0064] In some embodiments, the processing device may obtain the corner point position of the first grid point image and the confidence level of the corner point position by using the following formula:

[0065]

[0066] Wherein, Image represents the first grid point image, Function(Image) represents the corner point position of the first corner point output by the corner point recognition model for the first grid point image and the confidence of the corner point position, σ is the scaling factor, x map Indicates the X-direction coordinate of the estimated position of the corner point, y map represents the Y-direction coordinate of the estimated position in the corner point position, represents the X-direction offset of the offset position in the corner point position, Indicates the Y-direction offset of the offset position in the corner point position, Indicates the confidence of the corner point position.

[0067] In the above formula, x map It can be further expressed by the x coordinate of the p×σth row and q×σth column in the pixel coordinate system with the upper left corner of the first grid image as the origin, that is, x map(p×σ,q×σ) ,y map It can be further expressed by the y coordinate of the p×σth row and q×σth column in the pixel coordinate system with the upper left corner of the first grid image as the origin, that is, y map(p×σ,q×σ) , It can be further obtained by the first offset feature map BX map It is represented by the x-coordinate offset value of the p-th row and q-th column in It can be further passed through the second offset feature map BY map It is represented by the x-coordinate offset value of the p-th row and q-th column in It can be further analyzed through the probability feature map S map To express, that is

[0068] The above formula can be further rewritten as:

[0069]

[0070] Wherein, Image represents the first grid point image, F(Image) represents the result output by the corner point recognition model for the first grid point image, σ is the scaling factor, and x map(p×σ,q×σ) represents the x-coordinate of the p×σth row and q×σth column in the pixel coordinate system with the upper left corner of the first grid image as the origin, and y map(p×σ,q×σ) represents the y coordinate of the p×σth row and q×σth column in the pixel coordinate system with the upper left corner of the image in the first grid image as the origin, Indicates the x-direction offset feature map BX map The x-coordinate offset value of the p-th row and q-th column in Indicates the y-direction offset feature map BY mapThe y-coordinate offset value of the p-th row and q-th column in , Probability feature map S map The probability value of the p-th row and q-th column in , H is the length of Image, and W is the width of Image.

[0071] like Figure 6 As shown in the figure, this figure is a schematic diagram of obtaining the corner point position through the corner point recognition model provided by an embodiment of the present application. In the embodiment of the present application, the first grid image is a 4×4 checkerboard image, and the 4×4 checkerboard is input to the corner point recognition model, and the corner point recognition model outputs an x-direction offset feature map BX scaled by σ times map1 , y-direction offset feature map and probability map BY map1 ,σ=2,x-direction offset feature map BX map1 , y-direction offset feature map BY map1 And the probability feature map S map1 The output width and height are both 2×2. Indicates the x-direction offset feature map BX map1 The x-coordinate offset value of row 0 and column 0 is Indicates the x-direction offset feature map BX map1 The x-coordinate offset value of row 0 and column 1 is Indicates the x-direction offset feature map BX map1 The x-coordinate offset value of row 1 and column 0 is Indicates the x-direction offset feature map BX map1 The x-coordinate offset value of row 1 and column 1 is and Same reason.

[0072] Secondly, if Figure 6 As shown, x map1 and map1 The output width and height are both 2×2, x map(0,0) The output is 0, x map(2,0) The output is 2, x map(0,2) The output is 0, x map(2,2) The output is 2, y map(0,0) The output is 0, y map(2,0) The output is 0, y map(0,2) The output is 2, y map(2,2) The output is 2.

[0073] After the offset feature map is multiplied by the scaling factor, it is added to the estimated feature map to obtain a one-dimensional expansion diagram as shown in Figure 6 As shown, the corner point position is obtained, wherein the corner point position is represented by the corner point coordinates, the estimated position is represented by the integer part of the corner point coordinates, and the offset position is represented by the decimal part of the corner point coordinates.

[0074] The coordinates of the corner points are expressed as follows:

[0075]

[0076]

[0077] in, is the x-coordinate of the sample position corresponding to the first sample image, is the x coordinate of the sample position corresponding to the second sample image, is the x-coordinate of the sample position corresponding to the third sample image, is the x-coordinate of the sample position corresponding to the 4th sample image; is the y coordinate of the sample position corresponding to the first sample image, is the y coordinate of the sample position corresponding to the second sample image, is the y coordinate of the sample position corresponding to the third sample image, is the y coordinate of the sample position corresponding to the 4th sample image, is the prediction confidence of the predicted corner point position corresponding to the first sample image, is the prediction confidence of the predicted corner point position corresponding to the second sample image, is the prediction confidence of the predicted corner point position corresponding to the third sample image, is the prediction confidence of the predicted corner point position corresponding to the 4th sample image.

[0078] The corner point recognition model is trained based on the sample image, the sample position of the corner point in the sample image and the sample confidence corresponding to the sample position. The sample position is determined by the sample estimated position and the sample offset position.

[0079] Among them, the loss function used by the corner recognition model is:

[0080]

[0081] Among them, S is the predicted corner point position set for the sample image, L is the sample position set corresponding to the sample image, K(S,L) is the loss value of the loss function, and x i is the x coordinate of the sample position corresponding to the i-th sample image, x j is the x coordinate of the sample position corresponding to the jth sample image, y i is the y coordinate of the sample position corresponding to the i-th sample image, y j is the y coordinate of the sample position corresponding to the jth sample image, is the x-coordinate of the predicted corner point position corresponding to the i-th sample image, is the x-coordinate of the predicted corner point position corresponding to the j-th sample image, is the y coordinate of the predicted corner point position corresponding to the i-th sample image, is the y coordinate of the predicted corner point position corresponding to the jth sample image, is the prediction confidence of the predicted corner point position corresponding to the i-th sample image, is the prediction confidence of the predicted corner point position corresponding to the jth sample image, and λ is the scale adjustment coefficient. The sample image includes the initial sample image and the processed sample image, and the processed sample image is obtained by adding noise to the initial sample image. The way of adding noise includes translation, rotation, perspective and distortion processing of the image.

[0082] The processing device generates an input image, which is a checkerboard calibration plate image. 1 to 4 operations are randomly selected from translation, rotation, perspective and distortion to perform geometric transformation on the checkerboard calibration plate image. The geometric transformation is performed by a mathematical model to obtain accurate corner point positions and generate a grid image. 0 to 3 operations are randomly selected from blur, uneven illumination and noise to transform the grid image. The uneven illumination includes different degrees of exposure, highlighting and dimming at a local position, and the noise includes scratches, dirt, etc. to generate a sample image. The algorithm is used to detect the checkerboard pattern area in the sample image, and the non-checkerboard pattern area is deleted and replaced with an image randomly taken at an industrial site. The checkerboard calibration plate is placed at the industrial site for simulation and shooting. A large number of images with different degrees of blur, exposure and noise can be obtained through the above steps, and the corner point positions remain unchanged.

[0083] The sample images and corner points are input into the corner point recognition model for training. When the loss is minimal, the optimal model is obtained for subsequent operations.

[0084] The above steps can improve the accuracy and find the specific position of the corner point more accurately.

[0085] S303: The processing device selects a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point.

[0086] The confidence threshold is preset, and at least one first target corner point can be screened out according to the confidence threshold.

[0087] Through the operations of S301 and S302, the processing device can obtain the corner point position of at least one first corner point and the confidence level of the corner point position. For example, in S302, the positions of four corner points and the confidence levels of the corner point positions are obtained. The confidence threshold is set to 80%, and the samples above the confidence threshold are retained. All less than 80%, confidence level is greater than 80%, so the corner point is Leave.

[0088] S304: The processing device calibrates the position and posture of the industrial robot according to the corner point position of the first target corner point.

[0089] After the processing device obtains the corner point position of the first target corner point, the position and posture of the industrial robot can be calibrated.

[0090] In some embodiments, the processing device may calibrate the binocular system parameters, including the intrinsic calibration of the monocular camera and the extrinsic calibration of the binocular camera.

[0091] Monocular camera intrinsic parameter calibration: The checkerboard calibration plate is a flat plate with a black and white checkerboard pattern array at a fixed spacing, which is used to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and the corresponding point of the point in the image. Place the checkerboard calibration plate at a position that occupies 1 / 3 to 1 / 2 of the imaging field of the monocular camera, change the position of the monocular camera to take multiple images, so that the positions of the corner points where the black and white checkerboards intersect are distributed at all positions in the image. Use the checkerboard corner point detection algorithm to obtain the corner point positions of the corner points in each image. The parameters of the checkerboard calibration plate (black and white checkerboard size, number of rows and columns of corner points) are known. The coordinate system is established with the first corner point in the upper left corner of the checkerboard calibration plate as the origin of the three-dimensional space to obtain the spatial three-dimensional positions of all corner points. The spatial three-dimensional position of the corner point, the corner point position of the corner point, and the parameters of the checkerboard calibration plate are input into the monocular camera calibration algorithm to calculate the intrinsic parameters of the monocular camera (focal length f, image center coordinates (c x , c y )) and distortion parameters. Similar operations are performed on both cameras to obtain the intrinsic parameters and distortion parameters of the left and right cameras. Based on the intrinsic parameters and distortion parameters of the two cameras, the image taken by the camera can be corrected to an undistorted image.

[0092] The processing device can determine the intrinsic parameters of the monocular camera through the following formula (1):

[0093]

[0094] u and v are the horizontal and vertical coordinates of the corner point, f is the focal length of the camera, (c x , c y ) represents the corner coordinates of the center of the first grid image, R s and t s is the rotation matrix and translation vector of the image coordinate system relative to the three-dimensional coordinate system of the corner point, X W , Y W and Z W Represents the three-dimensional coordinates of the corner point.

[0095] The processing device can determine the distortion parameter by the following formula (2):

[0096]

[0097] Among them, (u d , v d ) is the coordinate of the corner point in the ideal case, (u, v) is the coordinate of the corner point in the actual distortion case, k 1 is the first radial distortion parameter, k 2 is the second radial distortion parameter, k 3 is the third radial distortion parameter, p 1 is the first tangential distortion parameter, p 2 is the second tangential distortion parameter, and e is the distance between the corner point and the origin. Distortion parameter D = [k 1 , k 2 , k 3 , p 1 , p 2 , e].

[0098] The following is an introduction to the external parameter calibration of the binocular camera. First, the left and right cameras in the system need to be fixed to ensure that the distance between the optical centers of the two cameras does not change. The above-mentioned checkerboard calibration board is placed at the corner point position where both the left and right cameras can capture all the corner points, and the corner points try to occupy the entire longitudinal space of the image. In the distortion-corrected image, the checkerboard corner point detection algorithm (such as the method described above) is used to obtain the corner point positions of the corner points in the checkerboard images captured by the left and right cameras. Using the PnP (Perspective-n-Point) algorithm, the postures of the left and right cameras in the three-dimensional coordinate system of the calibration board can be obtained from the spatial three-dimensional positions of the corner points and the corner point positions of the corner points. By adjusting the left transformation matrix T1 and the right transformation matrix T2 related to the postures of the left and right images, the positions of the corner points on the images are changed, and finally the corner point coordinates in the left image multiplied by the left transformation matrix are equal to the corner point coordinates in the right image multiplied by the right transformation matrix, and the baseline b of the two cameras is obtained, as shown in formula (3):

[0099]

[0100] Where f is the focal length of the camera, c xl and c yl is the horizontal and vertical coordinates of the center corner of the left image, c xr and c yr is the horizontal and vertical coordinates of the center corner of the right image, u l and v l is the horizontal and vertical coordinates of the corner point in the left figure, u r and v r are the horizontal and vertical coordinates of the corner points in the right figure.

[0101] After completing the above calibration, spatial point measurement can be performed.

[0102] In some examples, a checkerboard corner points are attached to the end of the industrial robot to be measured (if the orientation posture needs to be measured, a≥3, because 3 points determine a surface, and the relative posture of two spatial surfaces can be calculated), or the checkerboard calibration plate can be fixed to the end of the industrial robot through a rigid connection. If checkerboard corner point stickers are used, they do not need to be as high-precision as the calibration plate, and can be printed on any paper. They are low-cost and easy to operate, and will not affect the final accuracy (because the corner point position in this step is used as a landmark point and does not affect the system error).

[0103] The binocular measurement system can clearly see all the checkerboard corner points at the end of the robot in the left and right cameras, and then start taking images and measuring. All measurements are performed on the distortion-corrected images. The checkerboard corner point detection algorithm detects a corner points in the images captured by the left and right cameras, and obtains the corresponding relationship (p li , p ri ), where p li Indicates the position of the i-th corner point in the left image taken by the left camera (x li ,y li ), p ri represents the position of the i-th corner point in the right image taken by the right camera (x ri ,y ri ), i = 1, 2, ···, a. The three-dimensional space coordinates Q of the i-th corner point in the image i (X i ,Y i ,Z i ) is expressed as follows:

[0104]

[0105] From formulas (4)-(6), we can find Q 1 To Q a There are a total of a three-dimensional space points.

[0106] When a≥3, the relative posture transformation of the two measurements, that is, the orientation change of the robot end, can also be calculated by iteratively using the following formula.

[0107]

[0108] Where Q i represents the 3D space point corresponding to the i-th chessboard corner point before the robot terminal moves, Q i ' represents the three-dimensional space point corresponding to the i-th chessboard corner point after the robot end moves, J is the error, and the relative change posture R, t is obtained by mathematical iterative optimization of the error J.

[0109] Based on the above description, the embodiment of the present application provides a method for calibrating the posture of an industrial robot. In this method, since the corner point recognition model is trained based on the sample image, the sample position of the corner point in the sample image and the sample confidence corresponding to the sample position, and the sample position is determined by the sample estimated position and the sample offset position, the first grid point image is input into the corner point recognition model to obtain the corner point position of at least one first corner point in the first grid point image and the confidence of the corner point position, and the corner point position includes the estimated position and the offset position of the corner point. The traditional solution is to use the integer pixel position of the corner point as a label to determine whether the pixel point is a corner point, or to make a Gaussian circle with the corner point position as the center of the circle, and the neural network calculates the approximate position of the Gaussian circle and approximates the coordinates of the maximum Gaussian response in the circle by mathematical methods, so as to obtain the corner point position. It can be seen that in the present application, the output of the neural network is cleverly matched to the corner point position by adding the integer coordinates to the offset, and the non-corner points are eliminated by the probability corresponding to each point. It can be seen that the scheme of the present application can improve the accuracy of the corner point value, improve the stability of the detection method, and reduce the occurrence of missed detection or false detection. After the position and posture of the industrial robot are calibrated based on more accurate corner point positions, the calibration effect is better.

[0110] The present application also provides a schematic diagram of a posture calibration device for an industrial robot, wherein the posture calibration device for the industrial robot includes: Figure 7 The shown acquisition module 701, prediction module 702, screening module 703, and calibration module 704.

[0111] The present application provides a posture calibration device for an industrial robot, the device comprising:

[0112] An acquisition module 701 is used to acquire a first grid point image of the end of a mechanical arm of an industrial robot;

[0113] A prediction module 702 is used to input the first grid point image into a corner point recognition model to obtain a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point; wherein the corner point recognition model is trained based on a sample image, sample positions of corner points in the sample image, and sample confidence levels corresponding to the sample positions, and the sample positions are determined by the sample estimated position and the sample offset position;

[0114] A screening module 703 is used to screen out a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point;

[0115] The calibration module 704 is used to calibrate the position and posture of the industrial robot according to the corner point position of the first target corner point.

[0116] Optionally, the acquisition module 701 is used to acquire a first grid point image at the end of a mechanical arm of the industrial robot;

[0117] Acquire a first grid point image of the end of the robot's mechanical arm when the robot is in the first position;

[0118] The method further comprises:

[0119] Acquire a second grid point image of the end of the robot's mechanical arm when the robot is at the second position;

[0120] Determining a corner point position of a second target corner point according to the second grid point image;

[0121] The moving trajectory of the end of the robot arm is drawn according to the corner point position of the first target corner point and the corner point position of the second target corner point.

[0122] Optionally, the recognition module 702 is used to input the first grid point image into a corner point recognition model to obtain a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point, including:

[0123]

[0124] Wherein, Image represents the first grid point image, Function(Image) represents the result output by the corner point recognition model for the first grid point image, σ is the scaling factor, and x map(p×σ,q×σ) represents the estimated position of the p×σth row and q×σth column in the x direction in the pixel coordinate system with the upper left corner of the first grid image as the origin, y map(p×σ,q×σ) represents the estimated position of the p×σth row and q×σth column in the y direction in the pixel coordinate system with the upper left corner of the first grid image as the origin, Indicates the offset position in the x direction, Indicates the offset position in the y direction, Indicates the confidence of the corner point location.

[0125] The corner point position is represented by the corner point coordinates, the estimated position is represented by the integer part of the corner point coordinates, and the offset position is represented by the decimal part of the corner point coordinates.

[0126] Optionally, the corner point recognition model is trained based on a sample image, a sample position of a corner point in the sample image, and a sample confidence corresponding to the sample position, the sample position is determined by a sample estimated position and a sample offset position, and the loss function adopted by the corner point recognition model is:

[0127]

[0128] Among them, S is the predicted corner point position set for the sample image, L is the sample position set corresponding to the sample image, K(S,L) is the loss value of the loss function, and x i is the x coordinate of the sample position corresponding to the i-th sample image, x j is the x coordinate of the sample position corresponding to the jth sample image, y i is the y coordinate of the sample position corresponding to the i-th sample image, y j is the y coordinate of the sample position corresponding to the jth sample image, is the x-coordinate of the predicted corner point position corresponding to the i-th sample image, is the x-coordinate of the predicted corner point position corresponding to the j-th sample image, is the y coordinate of the predicted corner point position corresponding to the i-th sample image, is the y coordinate of the predicted corner point position corresponding to the jth sample image, is the prediction confidence of the predicted corner point position corresponding to the i-th sample image, is the prediction confidence of the predicted corner point position corresponding to the jth sample image, and λ is the scale adjustment coefficient.

[0129] The sample image includes an initial sample image and a processed sample image, wherein the processed sample image is obtained by adding noise to the initial sample image;

[0130] Ways to add noise include: translation, rotation, perspective and distortion of the image.

[0131] Optionally, the calibration module 704 is used to calibrate the posture of the industrial robot according to the corner point position of the first target corner point.

[0132] The data processing device according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above-mentioned other operations and / or functions of each module / unit of the data processing device are respectively to implement Figure 3 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0133] The present application also provides a computing device, such as Figure 8 As shown, this figure is a schematic diagram of a computing device provided in an embodiment of the present application, such as Figure 8 As shown, computing device 800 includes bus 801, processor 802, communication interface 803 and memory 804. Processor 802, memory 804 and communication interface 803 communicate with each other via bus 801.

[0134] The bus 801 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] The processor 802 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0136] The communication interface 803 is used for communicating with the outside.

[0137] The memory 804 may include a volatile memory, such as a random access memory (RAM). The memory 804 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0138] The memory 804 stores executable codes, and the processor 802 executes the executable codes to perform the aforementioned virtual object allocation method.

[0139] Specifically, in implementing Figure 7 In the case of the embodiment shown, and Figure 7 When each module or unit of the posture calibration device for an industrial robot described in the embodiment is implemented by software, the execution Figure 7 The software or program code required for the functions of each module / unit in the embodiment may be partially or completely stored in the memory 804. The processor 802 executes the program code corresponding to each unit stored in the memory 804 to execute the above-mentioned method for calibrating the position and posture of the industrial robot.

[0140] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above method.

[0141] The embodiment of the present application further provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0142] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0143] When the computer program product is executed by a computer, the computer executes any of the aforementioned virtual object allocation methods. The computer program product may be a software installation package, and when any of the aforementioned virtual object allocation methods is needed, the computer program product may be downloaded and executed on a computer.

[0144] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0145] The above description is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A method for calibrating the posture of an industrial robot, characterized in that: The method comprises: Acquire a first grid point image of the end of the robotic arm of the industrial robot; Inputting the first grid point image into a corner point recognition model, obtaining a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point; wherein the corner point recognition model is trained based on a sample image, sample positions of corner points in the sample image, and sample confidence levels corresponding to the sample positions, and the sample positions are determined by the sample estimated position and the sample offset position; Filter out a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point; The position and posture of the industrial robot are calibrated according to the corner point position of the first target corner point.

2. The method according to claim 1, characterized in that The step of inputting the first grid point image to a corner point recognition model to obtain a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position includes: Wherein, Image represents the first grid point image, Function(Image) represents the corner point position of the first corner point output by the corner point recognition model for the first grid point image and the confidence of the corner point position, σ is the scaling factor, x map Indicates the X-direction coordinate of the estimated position of the corner point, y map represents the Y-direction coordinate of the estimated position in the corner point position, represents the X-direction offset of the offset position in the corner point position, Indicates the Y-direction offset of the offset position in the corner point position, Indicates the confidence of the corner point position.

3. The method according to claim 1, characterized in that The step of obtaining a first grid point image of the end of the robot's mechanical arm comprises: Acquire a first grid point image of the end of the robot's mechanical arm when the robot is in the first position; The method further comprises: Acquire a second grid point image of the end of the robot's mechanical arm when the robot is at the second position; Determining a corner point position of a second target corner point according to the second grid point image; The moving trajectory of the end of the robot arm is drawn according to the corner point position of the first target corner point and the corner point position of the second target corner point.

4. The method according to claim 1, characterized in that: The loss function used by the corner point recognition model is: Among them, S is the predicted corner point position set for the sample image, L is the sample position set corresponding to the sample image, K(S,L) is the loss value of the loss function, and x i is the x coordinate of the sample position corresponding to the i-th sample image, x j is the x coordinate of the sample position corresponding to the jth sample image, y i is the y coordinate of the sample position corresponding to the i-th sample image, y j is the y coordinate of the sample position corresponding to the jth sample image, is the x-coordinate of the predicted corner point position corresponding to the i-th sample image, is the x-coordinate of the predicted corner point position corresponding to the j-th sample image, is the y coordinate of the predicted corner point position corresponding to the i-th sample image, is the y coordinate of the predicted corner point position corresponding to the jth sample image, is the prediction confidence of the predicted corner point position corresponding to the i-th sample image, is the prediction confidence of the predicted corner point position corresponding to the jth sample image, and λ is the scale adjustment coefficient.

5. The method according to claim 1, characterized in that The sample image includes an initial sample image and a processed sample image, and the processed sample image is obtained by adding noise to the initial sample image.

6. The method according to claim 5, characterized in that Ways to add noise include: translation, rotation, perspective and distortion of the image.

7. The method according to any one of claims 1 to 6, characterized in that: The corner point position is represented by the corner point coordinates, the estimated position is represented by the integer part of the corner point coordinates, and the offset position is represented by the decimal part of the corner point coordinates.

8. A posture calibration device for an industrial robot, characterized in that: The device comprises: An acquisition module, used for acquiring a first grid point image of the end of a mechanical arm of an industrial robot; A prediction module, used for inputting the first grid point image into a corner point recognition model, obtaining a corner point position of at least one first corner point in the first grid point image and a confidence level of the corner point position, wherein the corner point position includes an estimated position and an offset position of the corner point; wherein the corner point recognition model is trained based on a sample image, sample positions of corner points in the sample image, and sample confidence levels corresponding to the sample positions, and the sample positions are determined by the sample estimated position and the sample offset position; A screening module, used to screen out a first target corner point whose confidence is greater than a preset confidence threshold from at least one first corner point; A calibration module is used to calibrate the position and posture of the industrial robot according to the corner point position of the first target corner point.

9. A computing device, characterized in that including memory and processor; One or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the processor, the computing device executes the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.