Joint calibration method, pore location detection method, electronic equipment and storage medium

Abstract

This application provides a joint calibration method, a hole position detection method, an electronic device, and a computer-readable storage medium. The joint calibration method includes: moving the end effector of a motion mechanism to several positions; obtaining the end position of the end effector at each position; obtaining the pose relationship between the checkerboard coordinate system and the binocular coordinate system; and obtaining the standard laser coordinates of a standard sphere in the linear laser coordinate system. Based on the end positions of the end effector at the several positions and the standard laser coordinates, the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system is obtained. Based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at the several positions and the standard laser coordinates, the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system is obtained. Through the above method, the device establishes a mathematical model for the pose relationship between the linear laser coordinate system and the world coordinate system, and between the binocular coordinate system and the world coordinate system, using a joint calibration method involving two hands and eyes.

Description

Technical Field

[0001] This application relates to the field of visual inspection and positioning technology, and in particular to a joint calibration method, a hole position detection method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the rapid development of high-end manufacturing sectors such as automobiles, shipbuilding, and aerospace, the structure and manufacturing processes of workpieces are becoming increasingly complex, and the requirements for processing and assembly precision and automation levels are becoming increasingly stringent. This presents greater challenges to industrial automation inspection technology. Typically, workpieces such as skins, frames, engines, and blades have circular hole features such as mounting holes, riveting holes, and positioning holes. Identifying and locating these circular holes is an essential step in the assembly process. However, during assembly, deviations in hole diameter or changes in hole position relationships on the workpiece often lead to assembly failure or poor assembly quality. Therefore, the quality of hole position and diameter detection on the workpiece directly affects the assembly precision and automation level of the workpiece.

[0003] Traditional contact measurement methods offer high accuracy, but suffer from low efficiency and automation, and are prone to damaging workpieces, failing to meet the demands of modern intelligent manufacturing. Currently, many advanced non-contact hole location detection methods and equipment, such as vision inspection, have emerged. Their advantages, including high flexibility, high efficiency, no workpiece damage, and ease of computer-integrated manufacturing, have led to their widespread application in high-end manufacturing. However, some shortcomings and areas for improvement remain, such as the need to enhance accuracy and automation. Furthermore, non-contact hole location detection methods involve the coordination of multiple instruments, requiring operators to calibrate each instrument separately and then establish the relationships between them, resulting in a significant workload and low automation levels. Summary of the Invention

[0004] This application provides a joint calibration method, a pore location detection method, an electronic device, and a computer-readable storage medium.

[0005] This application provides a joint calibration method, the joint calibration method comprising:

[0006] Move the end of the motion mechanism to several positions, obtain the end position of the end of the motion mechanism corresponding to each position, obtain the pose relationship between the chessboard coordinate system and the binocular coordinate system, and obtain the standard laser coordinates of the standard ball in the online laser coordinate system;

[0007] Based on the end position of the motion mechanism at the aforementioned positions and the standard laser coordinates, the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system is obtained.

[0008] Based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at the aforementioned positions, and the standard laser coordinates, the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system is obtained.

[0009] The acquisition of standard laser coordinates in the online laser coordinate system of a standard sphere includes:

[0010] Move the end of the motion mechanism to the current position and acquire the standard sphere point cloud obtained by line laser scanning;

[0011] Based on the point cloud of the standard sphere, obtain the tangential circle data of the standard sphere;

[0012] Based on the tangential circle data and the diameter of the standard sphere, obtain the standard laser coordinates of the center of the standard sphere in the online laser coordinate system at the current position.

[0013] The step of obtaining the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system based on the end position of the motion mechanism at the aforementioned positions and the standard laser coordinates includes:

[0014] The laser pose transformation equation for each position is established using the standard world coordinates of the standard sphere, the end position of the end of the motion mechanism at each position, the laser pose transformation relationship, and the standard laser coordinates.

[0015] Using the laser pose transformation equations at the aforementioned positions, a laser pose solution equation is established;

[0016] Solve the laser pose equation to obtain the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system.

[0017] The step of obtaining the pose relationship between the chessboard coordinate system and the binocular coordinate system includes:

[0018] Move the end of the motion mechanism to the current position and obtain a chessboard image at the current position;

[0019] Extract the corner points of the chessboard image and establish the chessboard coordinate system of the chessboard image according to the corner points;

[0020] Using a pre-calibrated binocular coordinate system, the pose relationship between the chessboard coordinate system and the binocular coordinate system is obtained.

[0021] The step of obtaining the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system based on the pose relationship between the chessboard coordinate system and the binocular coordinate system at the aforementioned positions and the standard laser coordinates includes:

[0022] The binocular pose transformation equation for each position is established using the binocular world coordinates of the standard sphere, the end position of the motion mechanism at each position, the binocular pose transformation relationship, and the standard laser coordinates.

[0023] Using the binocular pose transformation equations at the aforementioned positions, a binocular pose solution equation is established;

[0024] Solve the stereo pose equations to obtain the stereo pose transformation relationship between the stereo coordinate system and the world coordinate system.

[0025] This application also provides a hole position detection method, the hole position detection method comprising:

[0026] Acquire binocular images of the workpiece under test;

[0027] Using the left or right image from the binocular images, fit the region of interest for several aperture positions;

[0028] The binocular image is used to obtain a binocular 3D point cloud of the region of interest, and the 3D coordinates and numbers of the several holes in the binocular coordinate system are determined based on the height information of the binocular 3D point cloud.

[0029] According to the binocular pose transformation relationship, the three-dimensional coordinates in the binocular coordinate system are converted into binocular three-dimensional coordinates in the world coordinate system;

[0030] The binocular three-dimensional coordinates are used to determine the line laser scanning sequence;

[0031] According to the described line laser scanning sequence, obtain the three-dimensional point cloud of line laser at several aperture positions;

[0032] Clustering and segmenting the hole edge points in the line laser 3D point cloud to obtain the hole edge point set corresponding to each hole location;

[0033] According to the set of edge points of each hole, obtain the three-dimensional coordinates of each hole in the online laser coordinate system, and convert the three-dimensional coordinates in the laser coordinate system into the online laser three-dimensional coordinates in the world coordinate system according to the online laser pose transformation relationship.

[0034] Match the binocular 3D coordinates and the line laser 3D coordinates, and associate the line laser 3D coordinates with the number corresponding to the successfully matched binocular 3D coordinates;

[0035] The binocular pose conversion relationship and the line laser pose conversion relationship are calibrated using the aforementioned joint calibration method.

[0036] The step of fitting regions of interest for several aperture positions using the left or right image from the binocular images includes:

[0037] Perform edge detection on the left or right image in the binocular image to obtain a binarized edge contour map;

[0038] The edge pixel coordinates of several hole positions are identified and extracted from the binarized edge contour map;

[0039] The elliptical regions of the plurality of holes are fitted according to the edge pixel coordinates;

[0040] The region of interest for the plurality of holes is determined based on the elliptical regions of the plurality of holes.

[0041] The step of determining the three-dimensional coordinates and numbers of the plurality of apertures in the binocular coordinate system based on the height information of the binocular three-dimensional point cloud includes:

[0042] Define a circular region and its edge within the region of interest for each aperture location;

[0043] The first average height of the points inside the hole in the binocular 3D point cloud is obtained according to the circular region;

[0044] The second average height of the points outside the aperture in the binocular 3D point cloud is obtained according to the edge of the region.

[0045] When the difference between the second average height and the first average height is greater than or equal to a preset height threshold, it is determined that there is a hole-like structure in the region of interest of the hole location. The three-dimensional coordinates of the hole-like structure are obtained according to the edge of the region, and the three-dimensional coordinates of all hole-like structures are numbered.

[0046] The step of determining the line laser scanning sequence using the binocular three-dimensional coordinates includes:

[0047] The laser scanning sequence is determined by ranking the Y-values ​​of the binocular three-dimensional coordinates of the aforementioned apertures from largest to smallest.

[0048] The step of determining the laser scanning order according to the Y-values ​​of the binocular three-dimensional coordinates of the plurality of aperture positions from largest to smallest includes:

[0049] The scanning order of the holes is determined by ranking the Y values ​​of the binocular three-dimensional coordinates from largest to smallest.

[0050] Obtain the scanning reserved spacing, and determine the first scanning reserved value and the second scanning reserved value for each hole position, wherein the first scanning reserved value is less than the second scanning reserved value;

[0051] When the X-axis of the hole in the current hole scanning sequence is less than the X-axis of the hole in the next hole scanning sequence, the reserved scanning sequence of the next hole is determined according to the order from the first reserved value to the second reserved value.

[0052] The laser scanning sequence is determined according to the scanning order of the holes and the reserved scanning order for each hole.

[0053] Before clustering and segmenting the hole edge points in the line laser 3D point cloud to obtain the hole edge point set corresponding to each hole location, the hole location detection method further includes:

[0054] Traverse the adjacent data points of the line laser 3D point cloud and obtain the absolute value of the gradient between every two adjacent data points;

[0055] When the absolute value of the gradient is greater than the first gradient threshold, the two adjacent data points are listed as candidate hole edge points.

[0056] Obtain several adjacent data points of the candidate hole edge point;

[0057] When the average difference between the several adjacent data points and the candidate hole edge points is less than the second gradient threshold, the candidate hole edge points are determined as hole edge points.

[0058] The step of determining the candidate hole edge point as the hole edge point includes:

[0059] Obtain the mean Z-value of all data points in the line laser 3D point cloud;

[0060] Candidate hole edge points with Z values ​​greater than the average Z value are determined as hole edge points.

[0061] The matching of the binocular 3D coordinates and the line laser 3D coordinates includes:

[0062] The least squares optimization algorithm is used to optimize the binocular 3D coordinates and the line laser 3D coordinates after matching, so as to minimize the distance difference between each successfully matched binocular 3D coordinate and the line laser 3D coordinate.

[0063] This application also provides an electronic device, which includes a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to implement the joint calibration method and / or the hole position detection method as described above.

[0064] This application also provides a computer-readable storage medium for storing program data, which, when executed by a processor, is used to implement the above-described joint calibration method and / or hole position detection method.

[0065] The beneficial effects of this application are as follows: The electronic device moves the end effector of its motion mechanism to several positions, obtains the end effector position of the motion mechanism at each position, obtains the pose relationship between the checkerboard coordinate system and the binocular coordinate system, and obtains the standard laser coordinates of the standard sphere in the online laser coordinate system; based on the end effector positions of the motion mechanism at several positions and the standard laser coordinates, it obtains the laser pose transformation relationship between the online laser coordinate system and the world coordinate system; based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at several positions and the standard laser coordinates, it obtains the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system. Through the above methods, the electronic device establishes a mathematical model for the pose relationship between the online laser coordinate system and the world coordinate system, and between the binocular coordinate system and the world coordinate system, using a joint hand-eye calibration method. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0067] Figure 1 This is a schematic diagram of the framework of an embodiment of the multi-sensor-based pore position and aperture detection device provided in this application;

[0068] Figure 2 This is a schematic diagram of the overall process of the hole position and diameter detection method provided in this application;

[0069] Figure 3 This is a schematic flowchart of an embodiment of the joint calibration method provided in this application;

[0070] Figure 4 This is a schematic diagram of the framework of the joint calibration system provided in this application;

[0071] Figure 5 This is a flowchart illustrating an embodiment of the borehole detection method provided in this application;

[0072] Figure 6 This is a schematic diagram of the multi-point path planning algorithm provided in this application;

[0073] Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application;

[0074] Figure 8 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0076] The problem this application aims to solve is to address the high requirements for the detection accuracy and automation of workpiece hole diameter and hole position in the high-end manufacturing field. It designs an automated detection device and related detection methods for hole position and hole diameter, which can flexibly realize automatic planning of hole scanning path and high-precision detection of hole diameter and hole position.

[0077] Please refer to the details. Figure 1 , Figure 1 This is a schematic diagram of the framework of an embodiment of the multi-sensor-based pore position and aperture detection device provided in this application.

[0078] like Figure 1 As shown, the hole position and diameter detection device can be summarized into the following four modules according to its functions: global positioning module, motion control module, data acquisition module, and information processing module.

[0079] The functions of each module are described below:

[0080] 1. Global positioning module: Its main function is to pre-inspect the holes on the workpiece placed on the inspection table and roughly estimate the number and position of the holes on the workpiece. In this application, this module is implemented using a binocular stereo vision system: the binocular camera is installed directly above the inspection table to ensure that the entire area to be tested on the inspection table is within the field of view of the two cameras, and the orientation of the two cameras is kept as vertical as possible with the optical axis pointing downwards.

[0081] 2. Motion control module: Its main function is to drive the data acquisition module (line laser scanner) installed at its end to move to a suitable position so as to accurately acquire the three-dimensional point cloud data of the hole. In this application, the module is implemented by a multi-axis CNC motion mechanism, which is installed on the side of the detection table, so that the relative position of its base and the detection table is fixed, and its end can move to any area to be measured on the detection table. Its base coordinates are used as the world coordinate system of the whole system.

[0082] 3. Data Acquisition Module: Its main function is to acquire high-precision three-dimensional point cloud data of holes on the workpiece. In this application, this module is implemented using a line laser scanner. The line laser scanner is fixed on the motion control module, and its posture is kept as vertically downward as possible in the direction of laser emission. By using a fixed frequency, three-dimensional data of the linear position of the line laser can be acquired quickly.

[0083] 4. Information Processing Module: This module establishes an information transmission hub between the three modules by connecting and communicating with the devices in the above three modules. It is responsible for processing the signals of each module and calculating the hole position and diameter. In this application, an industrial control computer is used to implement this module. It is connected to a binocular stereo vision system to receive binocular images and process them to roughly obtain the number and position of holes. It is also connected to a multi-axis CNC motion mechanism to automatically plan the motion path of the multi-axis CNC motion mechanism based on the roughly obtained number and position of holes, so that the multi-axis CNC motion mechanism drives the line laser scanner to a suitable position to scan the hole to be measured. Finally, it is connected to a line laser scanner to control the line laser scanner to collect data and process and calculate the line laser three-dimensional point cloud data to obtain accurate data on the hole position and diameter.

[0084] This application is aimed at Figure 1 Please refer to the flowchart of the detection method proposed by the hole position and diameter detection device shown. Figure 2 Specifically, it can be divided into offline calibration and online detection. The offline calibration part is the joint calibration method, and the online detection part is the pore position detection method.

[0085] The following is combined with Figure 1 and Figure 2 The specific technical solution of this application will be further described through the introduction of the joint calibration method and the pore position detection method:

[0086] Please refer to the details. Figure 3 and Figure 4 , Figure 3 This is a schematic flowchart of an embodiment of the joint calibration method provided in this application. Figure 4 This is a schematic diagram of the framework of the joint calibration system provided in this application.

[0087] The joint calibration method of this application is applied to an electronic device, which can be a server or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts of the electronic device, such as various units, sub-units, modules, and sub-modules, can all be set in the server, or they can be set in the server and the terminal device respectively.

[0088] Furthermore, the aforementioned server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules, such as software or software modules used to provide distributed servers, or as a single software program or software module; no specific limitation is made here. In some possible implementations, the joint calibration method of this application embodiment can be implemented by a processor calling computer-readable instructions stored in memory.

[0089] Before implementing the joint calibration method of this application, the staff needs to set up the corresponding hardware environment in advance, as follows:

[0090] In this embodiment, a standard sphere of known diameter is fixed, and a standard chessboard of known dimensions is fixed to a line laser scanner via a connecting device. To facilitate measurement by a binocular camera and ensure the normal vector of the chessboard surface remains upward, a multi-axis CNC motion mechanism is controlled to drive the line laser scanner, making the laser plane of the line laser scanner tangent to the standard sphere. In this static state, the line laser scanner acquires the point cloud at the tangent circle of the standard sphere, and the binocular camera acquires the chessboard image.

[0091] The offline calibration of this application consists of three parts:

[0092] 1. Calibration and Stereo Correction of Binocular Cameras: Binocular camera calibration and stereo correction techniques are very mature. This application uses the most classic Zhang's calibration method to obtain the intrinsic parameter matrices and distortion coefficients of the left and right cameras, and corrects the distortion of the left and right cameras. Then, based on the calibration parameters of the two cameras, the rotation and displacement matrices from the left camera to the right camera are calculated. Finally, the Bouguet algorithm is used for stereo correction, including calculating the alignment rotation matrix of the left and right cameras to align the epipolar lines of the binocular cameras, calculating the focal length and principal point coordinates of the left and right cameras after stereo correction, and the correction mapping matrix or correction mapping table.

[0093] 2. Hand-eye calibration between line laser scanner and multi-axis CNC motion mechanism.

[0094] 3. Hand-eye calibration between binocular camera and multi-axis CNC motion mechanism.

[0095] For details on hand-eye calibration between a line laser scanner and a multi-axis CNC motion mechanism, and hand-eye calibration between a binocular camera and a multi-axis CNC motion mechanism, please refer to steps S13 to S15.

[0096] Specifically, such as Figure 3 As shown, the joint calibration method of this application embodiment specifically includes the following steps:

[0097] Step S11: Move the end effector of the motion mechanism to several positions, obtain the end position of the end effector of the motion mechanism corresponding to each position, obtain the pose relationship between the checkerboard coordinate system and the binocular coordinate system, and obtain the standard laser coordinates of the standard ball in the online laser coordinate system.

[0098] In this embodiment of the application, when the end of the moving mechanism of the electronic device moves to several positions and reaches each position and is in a stationary state, the control line laser scanner and binocular camera collect data.

[0099] Specifically, the electronic device fits the point cloud at the tangent circle obtained by the line laser scanning to obtain the coordinates of the center and radius of the tangent circle. Based on the diameter of the standard sphere and the Pythagorean theorem, the coordinates P of the center of the standard sphere in the line laser coordinate system are calculated. s , i.e., standard laser coordinates. Where, coordinate P... s The sign of the X value depends on the position of the sphere's center relative to the light plane during the actual measurement. For example, if the sphere's center is on the right side of the light plane, the X value is positive; if the sphere's center is on the left side of the light plane, the X value is negative.

[0100] Among them, the pose of the end effector of the multi-axis CNC motion mechanism in the world coordinate system [R] e2w t e2w The coordinates of the center of the standard sphere in the world coordinate system can be read through the teach pendant or controller of the multi-axis CNC motion mechanism. w That is, standard world coordinates, and the coordinates in the stereo coordinate system are P. c That is, standard binocular coordinates; let the hand-eye matrix between the line laser coordinate system and the end effector coordinate system of the multi-axis CNC motion mechanism be X. s2e That is, the laser pose transformation relationship; then the following transformation relationship exists:

[0101] P w = [R e2w t e2w X s2e P s (1)

[0102] It should be noted that since the positions of the standard sphere and the stereo camera remain unchanged after being fixed in actual space, i.e., P w and P c It is a fixed value during the joint calibration process.

[0103] The electronic device processes the checkerboard image acquired by the binocular camera and extracts the corner points. Based on the binocular calibration results, the coordinates of each corner point in the binocular coordinate system can be calculated, such as... Figure 4 As shown, let the origin of the chessboard coordinate system be the top left corner, the X-axis be the direction from the origin to the top right corner, the Y-axis be the direction from the origin to the bottom left corner, and the Z-axis be perpendicular to the chessboard and upwards. Then, the pose relationship between the chessboard coordinate system and the binocular coordinate system can be easily obtained. [R] t2c t t2c Or the pose relationship between the binocular coordinate system and the checkerboard coordinate system [R] c2t t c2t Let the pose relationship between the linear laser coordinate system and the checkerboard coordinate system be X. s2t Then the following transformation relationship exists:

[0104] P c = [Rt2c t t2c X s2t P s (2)

[0105] By performing the above operation multiple times at different locations, multiple sets of P were obtained. s n (x s n ,y s n ,z s n ), [R e2w n t e2w n ] and [R t2c n t t2c n ].

[0106] Step S12: Based on the end positions of the motion mechanism ends at several locations and the standard laser coordinates, obtain the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system.

[0107] In this embodiment of the application, because P is located at different positions at the end of the multi-axis CNC motion mechanism, w and P c All are fixed values. For line laser scanners and multi-axis CNC motion mechanisms, we have:

[0108] [R e2w 1 t e2w 1 ]X s2e P s 1 =[R e2w 2 t e2w 2 ]X s2e P s 2 =……=[R e2w n t e2w n ]X s2e P s n (3)

[0109] Where n is the location point.

[0110] Therefore, the key to hand-eye calibration between a line laser scanner and a multi-axis CNC motion mechanism lies in the matrix X in equation (3). s2e Solve for it.

[0111] Specifically, matrix X s2e The following steps are used to solve the problem: First, the equation in equation (3) is transformed into a homogeneous form:

[0112]

[0113] Let R x = (r1, r2, r3), substituting into equation (4) above and expanding, we get:

[0114]

[0115] Again

[0116] x=[r1 r2 r3 t x ] T

[0117]

[0118] The hand-eye matrix X can be obtained by solving for x using the least squares method. s2e :

[0119] x=(A T A) -1 A T b (6)

[0120] Finally, the parameter matrix X is obtained by solving equation (6). s2e The laser pose transformation relationship from the linear laser coordinate system to the world coordinate system can be calculated:

[0121] X s2w =X s2e [R e2w t e2w (7)

[0122] Step S13: Based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at several locations, as well as the standard laser coordinates, obtain the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system.

[0123] In this embodiment of the application, because P is located at different positions at the end of the multi-axis CNC motion mechanism, w and P c All are fixed values. For line laser scanners and binocular cameras, we have:

[0124] [R t2c 1 t t2c 1 ]X s2t P s 1 =[Rt2c 2 t t2c 2 ]X s2t P s 2 =……=[R t2c n t t2c n ]X s2t P s n (8)

[0125] Therefore, the key to hand-eye calibration between a binocular camera and a multi-axis CNC motion mechanism lies in the matrix X in equation (8). s2t Solving for X. Where, matrix X... s2t The solution calculation method is the same as that for the matrix X mentioned above. s2e The solution and calculation steps are the same, and will not be repeated here.

[0126] The parameter matrix X obtained by the above steps s2t The stereo pose transformation relationship from the stereo coordinate system to the world coordinate system can be calculated:

[0127] X c2w =[R c2t t c2t ][X s2t ] -1 X s2w (9)

[0128] At this point, the hand-eye joint calibration is complete, and the transformation relationships between the three coordinate systems—the line laser coordinate system, the binocular coordinate system, and the end-effector coordinate system of the multi-axis CNC motion mechanism—and the world coordinate system all become known quantities in the subsequent hole position detection process.

[0129] In this embodiment, the electronic device moves the end effector of its motion mechanism to several positions, obtains the end position of the end effector at each position, acquires the pose relationship between the checkerboard coordinate system and the binocular coordinate system, and acquires the standard laser coordinates of the standard ball in the linear laser coordinate system. Based on the end positions of the end effector of the motion mechanism at several positions and the standard laser coordinates, the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system is acquired. Based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at several positions and the standard laser coordinates, the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system is acquired. Through the above method, the electronic device establishes a mathematical model for the pose relationship between the linear laser coordinate system and the world coordinate system, and between the binocular coordinate system and the world coordinate system, using a joint hand-eye calibration method.

[0130] The joint calibration method of this application provides a joint calibration method for two hands and eyes based on a standard ball and a checkerboard grid: "line laser sensor and multi-axis CNC motion mechanism" and "binocular camera and multi-axis CNC motion mechanism". It establishes a mathematical model for the pose relationship between the line laser coordinate system and the world coordinate system, and between the binocular coordinate system and the world coordinate system.

[0131] Based on the above joint calibration method, please refer to [further details]. Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of the borehole detection method provided in this application.

[0132] Specifically, such as Figure 5 As shown, the hole position detection method of this application embodiment specifically includes the following steps:

[0133] Step S21: Acquire a binocular image of the workpiece being tested.

[0134] In the embodiments of this application, such as Figure 2 The online inspection process shown involves the electronic device using a binocular camera to acquire left and right images of the workpiece being inspected on the inspection table.

[0135] Step S22: Using the left or right image from the binocular image, fit the region of interest for several aperture positions.

[0136] In this embodiment, the electronic device first preprocesses the left image using median filtering, then performs edge detection on the image using the Canny operator to obtain a binarized edge contour map. Next, the electronic device uses an ellipse detection algorithm based on Hough transform to identify and extract the edge pixel coordinates of the holes to be tested, and fits them to preliminarily determine the number and corresponding positions of the holes in the left image.

[0137] Furthermore, the electronic device adds a region of interest to the fitted elliptical region, marked with a rectangle. Specifically, the center of the rectangle coincides with the center of the ellipse, and the length and width of the rectangle are parallel to the major and minor axes of the ellipse, respectively. The length of the rectangle is 1.2 times the length of the major axis of the ellipse, and the width is 1.2 times the length of the minor axis.

[0138] In other embodiments, the electronic device may also fit the region of interest (ROI) of the aperture location using the right image in the binocular images. For example, when the disparity map in the binocular images is the left image, the ROI can be fitted according to the left image.

[0139] Step S23: Use the stereo image to obtain the stereo 3D point cloud of the region of interest, and determine the 3D coordinates and numbers of several holes in the stereo coordinate system based on the height information of the stereo 3D point cloud.

[0140] In this embodiment of the application, the electronic device performs stereo matching and three-dimensional reconstruction based on the left image, the right image, and the stereo calibration results, i.e., the calibration and stereo correction of the stereo camera, to obtain stereo point cloud data, and thus obtains the stereo three-dimensional point cloud within the rectangular area mentioned above.

[0141] Electronic devices use a height threshold method to determine whether a rectangular region contains a hole-like structure. The specific description of the height threshold method is as follows:

[0142] Using the center point of the rectangle as the center and half the width of the rectangle as the radius, the weighted average of the Z values ​​of the spatial points corresponding to each pixel within this circular area is used to obtain the average height Z of the points inside the hole. in The Z-value of the spatial points corresponding to each pixel on the edge of the rectangle is weighted and averaged to obtain the Z-value of the point outside the hole. out Mean height, X-axis mean X out and Y-axis mean Y out .

[0143] Among them, the points inside the aperture are all the data points projected from the circular area on the 2D image onto the surface of the binocular 3D point cloud, and the points outside the aperture are all the data points projected from the edge of the rectangular frame on the 2D image onto the surface of the binocular 3D point cloud.

[0144] Set height threshold H th ,like:

[0145] Z out - Z in ≥ H th (10)

[0146] If a hole-like structure is found within the rectangular frame, the corresponding rectangular frame in the left image is retained, and the three-dimensional coordinates O of the hole are calculated. rou The coordinate values ​​are represented as: (X out Y out Z out Otherwise, determine that there is no hole-like structure within the rectangle and delete the corresponding rectangle in the left image.

[0147] Finally, the electronic device numbers the remaining rectangles in the left image, with each number corresponding to the three-dimensional coordinates of a hole-like structure.

[0148] Step S24: Convert the 3D coordinates in the binocular coordinate system to the binocular 3D coordinates in the world coordinate system according to the binocular pose transformation relationship.

[0149] In the embodiments of this application, the electronic device uses the binocular pose transformation relationship calibrated in the joint calibration method to uniformly convert the three-dimensional coordinates in the binocular coordinate system into the binocular three-dimensional coordinates in the world coordinate system.

[0150] Step S25: Determine the line laser scanning sequence using binocular three-dimensional coordinates.

[0151] In the embodiments of this application, Figure 1 The data acquisition module shown will use the three-dimensional coordinates O determined in step S24. rou The data is sent to the information processing module, which then uses the three-dimensional coordinates O of each borehole to be measured. rou Using the target point as the origin of the line laser coordinate system as the controlled point, path planning for the line laser scanner is achieved.

[0152] Specifically, please refer to Figure 6 , Figure 6 This is a schematic diagram of the multi-point path planning algorithm provided in this application. For example... Figure 6 As shown, the electronic device, in the world coordinate system, is based on the three-dimensional coordinates O of each hole to be measured. rou If the Y values ​​are sorted in descending order, then Figure 6 The wiring results are: hole A, hole B, hole C, hole D, and hole E.

[0153] Furthermore, in this embodiment of the application, a scanning reserved spacing L can be set. If the world coordinates of hole A are (X... A ,Y A Z A If ), then the world coordinates of its accompanying point A1 are (X... A -L,Y A Z A ), with the world coordinates of point A2 being (X A +L,Y A Z A For hole B, the world coordinates of the accompanying point B1 are (X... B -L,Y B Z B The world coordinates of point B2 are (XB+L, YB, ZB), and so on. The scan begins from point A1, the accompanying point of hole A, and its scan path is from A1 to A2. If the X value of the next hole coordinate is greater than the X value of the previous hole coordinate, that is, X... next >X before If X is the starting point, then the scan begins at point 1 of the next hole; otherwise, it begins at point 2. For example, when scanning from hole A to hole B, because X... A <X B Therefore, we first travel from A2 to B1; because X B >X C Therefore, when scanning from hole B to hole C, C2 is reached first, and so on.

[0154] like Figure 6 The motion path of the controlled point shown is A1A2B1B2C2C1D2D1E1E2, which is used to obtain the line laser three-dimensional point cloud data of each hole.

[0155] Step S26: Obtain three-dimensional point clouds of several holes using line laser scanning, following the line laser scanning sequence.

[0156] In this embodiment of the application, the electronic device performs line laser scanning on each hole according to the line laser scanning sequence determined in step S25, and obtains the line laser three-dimensional point cloud of each hole.

[0157] Step S27: Cluster and segment the hole edge points in the line laser 3D point cloud to obtain the hole edge point set corresponding to each hole location.

[0158] In this embodiment, the electronic device uses the gradient threshold method to extract the edge points of the obtained three-dimensional point cloud of the line laser of each hole, performs statistical filtering on the extracted edge points to filter out discrete noise points, and then performs clustering segmentation on the edge points of the holes. Each point cloud obtained by segmentation is a set of edge points of a single hole.

[0159] The specific steps for extracting hole edge points using the gradient threshold method are as follows:

[0160] For each frame of linear point cloud data containing m points obtained from line laser scanning {p i |i=1,2,…,m}, set the gradient threshold T min traverse two adjacent points p j and p j+1 The absolute value of the gradient between {|t j ||j=1,2,…,m-1}, if:

[0161]

[0162] Indicate point p j to p j+1 The depth has changed significantly at point p. j or p j+1 For points on the edge of the suspected hole, to further eliminate interference from points on the inner wall of the hole in the above equation, a second constraint is added: adjacent gradient values ​​cannot simultaneously satisfy the above equation, which can be expressed as follows:

[0163]

[0164] In the formula, k is the point p j The number of adjacent points, T max As an auxiliary gradient threshold, it is only when point p j or p j+1 When both of the above constraints are satisfied, then the decision point p is determined. j or p j+1 For point p, which is the edge of the hole, j and p j+1Points whose Z-value is greater than the average Z-value of m points are identified as hole edge points.

[0165] Step S28: Based on the set of edge points of each hole, obtain the three-dimensional coordinates of each hole in the online laser coordinate system, and convert the three-dimensional coordinates in the laser coordinate system into the online laser three-dimensional coordinates in the world coordinate system according to the online laser pose transformation relationship.

[0166] In this embodiment, the electronic device uses an optimal spatial circle fitting algorithm to fit the edge point set of each hole, based on the center coordinates O of the fitted circle. acc (X acc ,Y acc Z acc The precise hole position and diameter data of each hole are obtained by using the radius and normal vector, and then converted to the world coordinate system according to the line laser pose transformation relationship calibrated by the joint calibration method.

[0167] Step S29: Match the stereo 3D coordinates and the line laser 3D coordinates, and associate the line laser 3D coordinates with the corresponding numbers of the successfully matched stereo 3D coordinates.

[0168] In this embodiment, the electronic device uses a least squares optimization algorithm to determine the aperture coordinates O of the binocular detection. rou (X out ,Y out Z out Hole coordinates O detected by line laser acc (X acc ,Y acc Z acc Perform matching to minimize the difference T in the absolute distance between each pair of matched points:

[0169]

[0170] Finally, the point cloud data of the hole edge obtained by the matching line laser scanning and the corresponding calculated hole position and diameter data are jointly stored with the numbered holes in the image captured by the left camera. This allows the numbered holes in the two-dimensional image to be linked to their corresponding line laser hole edge point cloud data and hole diameter and position information, thus achieving data integration of "two-dimensional image + three-dimensional point cloud + hole position and diameter information".

[0171] The hole position detection method of this application uses a binocular camera to provide preliminary hole identification and positioning functions for the line laser scanning system, and utilizes the two-dimensional and three-dimensional detection functions of the binocular camera to provide data support for the path planning of the line laser scanning. It provides a path planning method for line laser scanning holes based on multi-point path planning, which improves the efficiency of hole position and diameter detection based on line laser scanning. It jointly stores the three-dimensional point cloud of the line laser scanning with the two-dimensional image, so that each numbered hole in the image is connected with its high-precision three-dimensional point cloud and hole position and diameter information data, realizing the diversification of detection results.

[0172] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0173] To implement the joint calibration method and / or hole position detection method described in the above embodiments, this application also proposes an electronic device, which can be found in detail below. Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application.

[0174] The electronic device 300 of this application embodiment includes a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled together.

[0175] The memory 31 is used to store program data, and the processor 32 is used to execute the program data to implement the joint calibration method and / or hole position detection method described in the above embodiments.

[0176] In this embodiment, processor 32 can also be referred to as a CPU (Central Processing Unit). Processor 32 may be an integrated circuit chip with signal processing capabilities. Processor 32 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 32 can be any conventional processor.

[0177] To implement the joint calibration method of the above embodiments, this application also provides a computer-readable storage medium, such as... Figure 8As shown, the computer-readable storage medium 400 is used to store program data 41, which, when executed by a processor, is used to implement the joint calibration method and / or hole position detection method as described in the above embodiments.

[0178] This application also provides a computer program product, wherein the computer program product includes a computer program operable to cause a computer to perform the joint calibration method as described in the embodiments of this application. The computer program product may be a software installation package.

[0179] The joint calibration method described in the above embodiments of this application, when implemented as a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting hole positions, characterized in that, The well location detection method includes: Acquire binocular images of the workpiece under test; Using the left or right image from the binocular images, fit the region of interest for several aperture positions; The binocular image is used to obtain a binocular 3D point cloud of the region of interest, and the 3D coordinates and numbers of the plurality of holes in the binocular coordinate system are determined based on the height information of the binocular 3D point cloud. According to the binocular pose transformation relationship, the three-dimensional coordinates in the binocular coordinate system are converted into binocular three-dimensional coordinates in the world coordinate system; The binocular three-dimensional coordinates are used to determine the line laser scanning sequence; According to the described line laser scanning sequence, obtain the three-dimensional point cloud of line laser at several aperture positions; Clustering and segmenting the hole edge points in the line laser 3D point cloud to obtain the hole edge point set corresponding to each hole location; According to the set of edge points of each hole, obtain the three-dimensional coordinates of each hole in the online laser coordinate system, and convert the three-dimensional coordinates in the laser coordinate system into the online laser three-dimensional coordinates in the world coordinate system according to the online laser pose transformation relationship. Match the binocular 3D coordinates and the line laser 3D coordinates, and associate the line laser 3D coordinates with the number corresponding to the successfully matched binocular 3D coordinates; The binocular pose transformation relationship and the line laser pose transformation relationship are calibrated using the following joint calibration method: Move the end of the motion mechanism to several positions, obtain the end position of the end of the motion mechanism corresponding to each position, obtain the pose relationship between the chessboard coordinate system and the binocular coordinate system, and obtain the standard laser coordinates of the standard ball in the online laser coordinate system; Based on the end position of the motion mechanism at the aforementioned positions and the standard laser coordinates, the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system is obtained. Based on the pose relationship between the checkerboard coordinate system and the binocular coordinate system at the aforementioned positions, and the standard laser coordinates, the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system is obtained.

2. The hole position detection method according to claim 1, characterized in that, The method of fitting regions of interest for several aperture positions using the left or right image from the binocular images includes: Perform edge detection on the left or right image in the binocular image to obtain a binarized edge contour map; The edge pixel coordinates of several hole positions are identified and extracted from the binarized edge contour map; The elliptical regions of the plurality of holes are fitted according to the edge pixel coordinates; The region of interest for the plurality of holes is determined according to the elliptical regions of the plurality of holes.

3. The hole position detection method according to claim 1 or 2, characterized in that, The step of determining the three-dimensional coordinates and numbers of the plurality of apertures in the binocular coordinate system based on the height information of the binocular three-dimensional point cloud includes: Define a circular region and its edge within the region of interest for each aperture location; The first average height of the points inside the hole in the binocular 3D point cloud is obtained according to the circular region; The second average height of the points outside the aperture in the binocular 3D point cloud is obtained according to the edge of the region. When the difference between the second average height and the first average height is greater than or equal to a preset height threshold, it is determined that there is a hole-like structure in the region of interest of the hole location. The three-dimensional coordinates of the hole-like structure are obtained according to the edge of the region, and the three-dimensional coordinates of all hole-like structures are numbered.

4. The hole position detection method according to claim 1, characterized in that, The process of determining the line laser scanning sequence using the binocular three-dimensional coordinates includes: The laser scanning sequence is determined by ranking the Y-values ​​of the binocular three-dimensional coordinates of the aforementioned apertures from largest to smallest.

5. The hole position detection method according to claim 4, characterized in that, Determining the laser scanning order according to the Y-values ​​of the binocular three-dimensional coordinates of the plurality of aperture positions from largest to smallest includes: The scanning order of the holes is determined by ranking the Y values ​​of the binocular three-dimensional coordinates from largest to smallest. Obtain the scanning reserved spacing, and determine the first scanning reserved value and the second scanning reserved value for each hole position, wherein the first scanning reserved value is less than the second scanning reserved value; When the X-axis of the hole in the current hole scanning sequence is less than the X-axis of the hole in the next hole scanning sequence, the reserved scanning sequence of the next hole is determined according to the order from the first reserved value to the second reserved value. The laser scanning sequence is determined according to the scanning sequence of the holes and the reserved scanning sequence for each hole.

6. The hole position detection method according to claim 1, characterized in that, Before clustering and segmenting the hole edge points in the line laser 3D point cloud to obtain the hole edge point set corresponding to each hole location, the hole location detection method further includes: Traverse the adjacent data points of the line laser 3D point cloud and obtain the absolute value of the gradient between every two adjacent data points; When the absolute value of the gradient is greater than the first gradient threshold, the two adjacent data points are listed as candidate hole edge points. Obtain several adjacent data points of the candidate hole edge point; When the average difference between the several adjacent data points and the candidate hole edge points is less than the second gradient threshold, the candidate hole edge points are determined as hole edge points.

7. The hole position detection method according to claim 6, characterized in that, The step of determining the candidate hole edge point as the hole edge point includes: Obtain the mean Z-value of all data points in the line laser 3D point cloud; Candidate hole edge points with Z values ​​greater than the average Z value are determined as hole edge points.

8. The hole position detection method according to claim 1, characterized in that, The matching of the binocular 3D coordinates and the line laser 3D coordinates includes: The least squares optimization algorithm is used to optimize the binocular 3D coordinates and the line laser 3D coordinates after matching, so as to minimize the distance difference between each successfully matched binocular 3D coordinate and the line laser 3D coordinate.

9. The hole position detection method according to claim 1, characterized in that, The process of obtaining the standard laser coordinates in the standard sphere online laser coordinate system includes: Move the end of the motion mechanism to the current position and acquire the standard sphere point cloud obtained by line laser scanning; Based on the point cloud of the standard sphere, obtain the tangential circle data of the standard sphere; Based on the tangential circle data and the diameter of the standard sphere, obtain the standard laser coordinates of the center of the standard sphere in the online laser coordinate system at the current position.

10. The hole position detection method according to claim 1, characterized in that, The process of obtaining the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system based on the end position of the motion mechanism at the aforementioned positions and the standard laser coordinates includes: The laser pose transformation equation for each position is established using the standard world coordinates of the standard sphere, the end position of the end of the motion mechanism at each position, the laser pose transformation relationship, and the standard laser coordinates. Using the laser pose transformation equations at the aforementioned positions, a laser pose solution equation is established; Solve the laser pose equation to obtain the laser pose transformation relationship between the linear laser coordinate system and the world coordinate system.

11. The hole position detection method according to claim 1, characterized in that, The process of obtaining the pose relationship between the chessboard coordinate system and the binocular coordinate system includes: Move the end of the motion mechanism to the current position and obtain a chessboard image at the current position; Extract the corner points of the chessboard image and establish the chessboard coordinate system of the chessboard image according to the corner points; Using a pre-calibrated binocular coordinate system, the pose relationship between the chessboard coordinate system and the binocular coordinate system is obtained.

12. The hole position detection method according to claim 1, characterized in that, The process of obtaining the binocular pose transformation relationship between the binocular coordinate system and the world coordinate system based on the pose relationship between the chessboard coordinate system and the binocular coordinate system at the aforementioned positions, and the standard laser coordinates, includes: The binocular pose transformation equation for each position is established using the binocular world coordinates of the standard sphere, the end position of the motion mechanism at each position, the binocular pose transformation relationship, and the standard laser coordinates. Using the binocular pose transformation equations at the aforementioned positions, a binocular pose solution equation is established; Solve the stereo pose equations to obtain the stereo pose transformation relationship between the stereo coordinate system and the world coordinate system.

13. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing program data, and the processor executing the program data to implement the hole position detection method as described in any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data, which, when executed by a processor, is used to implement the hole position detection method according to any one of claims 1-12.

Images