Visual reference hole positioning method and device for portable drilling equipment

Abstract

The invention relates to the technical field of aircraft digital assembling and manufacturing, in particular to a reference hole visual positioning method and device for portable drilling device.The method comprises the steps that hand-eye calibration is conducted on a visual unit of target portable drilling equipment, a target reference hole is shot through the calibrated visual unit, and the target reference hole is obtained; obtaining a reference hole coarse image; performing coarse positioning on the target reference hole according to the reference hole coarse image to obtain a reference hole coarse coordinate; moving a visual unit to the reference hole coarse coordinate to shoot the target reference hole to obtain a reference hole fine image; and carrying out local adaptive threshold binarization, morphological processing and circular hole feature fitting on the reference hole fine image to generate a reference hole fine coordinate. Therefore, the problems that an existing vision measurement method depends on accurate edge detection parameter setting and cautious light source arrangement, missing detection, error detection and positioning deviation are prone to occurring, and the datum hole positioning requirement of portable hole making equipment cannot be met are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital assembly and manufacturing of aircraft, and in particular to a method and device for visually positioning reference holes of a portable hole-making device suitable for machining internal plane lamination holes of aircraft in the aviation field. Background Art

[0002] There is a large demand for drilling of planar riveting holes inside aircraft, but due to the narrow internal space, it is difficult to drill holes through conventional robotic arms, machine tools and other automated equipment. Therefore, the drilling is currently mainly done manually by relying on drilling templates, which is time-consuming and labor-intensive. The three-axis portable hole-making equipment is designed and developed to meet this demand. Due to its small size and light weight, it can be carried into the cabin by hand. After positioning and clamping, it can be automatically drilled, which can improve efficiency, save manpower, and ensure the consistency of hole-making quality. However, due to the characteristics of the portable hole-making equipment itself, after the equipment is installed inside the cabin, it is necessary to locate the center of the hole with the laminated pre-connection nail as the reference hole to obtain the relative position of the hole to be processed in the equipment coordinate system. The positioning of the reference hole determines the absolute position of all holes to be processed. Therefore, accurate reference hole identification and positioning are the basis for high-quality hole-making of the equipment.

[0003] Visual measurement methods have been widely adopted by the industry due to their advantages of low cost, non-contact, and high efficiency. However, existing technologies generally rely on precise edge detection parameter settings and careful light source arrangement. In factories with changing lighting conditions and detection conditions where the scratch state of the workpiece surface is uncertain, missed detection, wrong detection, and positioning deviation often occur, which cannot meet the reference hole positioning requirements of portable hole making equipment.

[0004] Therefore, there is an urgent need for a reference hole visual positioning method for portable hole-making equipment to meet the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of the reference holes. Summary of the invention

[0005] The present invention provides a reference hole visual positioning method and device for portable hole-making equipment, so as to solve the problems that the existing visual measurement method relies on accurate edge detection parameter setting and careful light source arrangement, is prone to missed detection, wrong detection and positioning deviation, and cannot meet the reference hole positioning requirements of portable hole-making equipment.

[0006] A first aspect of the present invention provides a method for visual positioning of a reference hole of a portable hole-making device, comprising the following steps: performing hand-eye calibration on a visual unit of a target portable hole-making device to obtain a hand-eye relationship; photographing a target reference hole using the calibrated visual unit to obtain a rough image of the reference hole; coarsely positioning the target reference hole based on the rough image of the reference hole to obtain a rough coordinate of the reference hole; moving the visual unit to the rough coordinate of the reference hole based on the hand-eye relationship to photograph the target reference hole to obtain a fine image of the reference hole; performing local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole; performing circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate fine coordinates of the reference hole.

[0007] Optionally, the performing hand-eye calibration on the visual unit of the target portable hole-making device to obtain the hand-eye relationship includes:

[0008] Connecting a target spindle measuring tool to a device spindle unit of the target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0009] Placing a target calibration plate measuring tool flat on a processing plane, and performing plane positioning of the target calibration plate measuring tool by a positioning block, wherein the target calibration plate measuring tool is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls;

[0010] Connecting the spindle unit of the device to the visual unit in the horizontal direction and the vertical direction, and ensuring that the target calibration plate measurement fixture appears within the field of view of the visual unit;

[0011] Using a laser tracker to measure the positions of the first laser tracker measuring target ball and the second laser tracker measuring target ball, so as to fit the first posture of the target spindle measuring fixture relative to the laser tracker and the second posture of the target calibration plate measuring fixture relative to the laser tracker;

[0012] Using the visual unit to photograph the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0013] Performing sub-pixel corner point extraction on the checkerboard calibration plate image to obtain a third pose of the checkerboard calibration plate relative to the visual unit;

[0014] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose and the third pose, so as to use the fourth pose as the hand-eye relationship.

[0015] Optionally, the coarse positioning of the target reference hole according to the coarse image of the reference hole to obtain coarse coordinates of the reference hole includes:

[0016] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0017] Constructing a pixel set containing reference hole semantics according to the mask image;

[0018] The pixel set containing the reference hole semantics is used to solve the hole center coordinates of the reference hole, and the hole center coordinates are used as the reference hole rough coordinates.

[0019] Optionally, the step of moving the visual unit to the rough coordinates of the reference hole to photograph the target reference hole to obtain a precise image of the reference hole includes:

[0020] The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the main axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates;

[0021] The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates by using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

[0022] Optionally, the performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image retaining the edge of the target circular hole includes:

[0023] Performing Gaussian blur on the reference hole fine image to obtain a reference hole blurred image;

[0024] Performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image;

[0025] Performing morphological corrosion and morphological expansion on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0026] The binary image, the first mask sub-image and the second mask sub-image are subjected to scratch interference masking processing to generate the binary image retaining the edge of the target circular hole.

[0027] Optionally, performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate actual coordinates of the reference hole includes:

[0028] Actively drawing edges on the binary image retaining the edge of the target circular hole to obtain a binary image with edges;

[0029] Performing arc segment fitting on the binary image with edges, and merging arcs with similar centers and radii to obtain an arc image with complete edge points;

[0030] A reliable circle with the lowest false alarm value in the circular arc image with complete edge points is screened by utilizing gradient changes, and coordinate transformation is performed on the reliable circle to generate the precise coordinates of the reference hole.

[0031] A second aspect of the present invention provides a visual positioning device for a reference hole of a portable hole-making device, comprising: a calibration module, used to perform hand-eye calibration on a visual unit of a target portable hole-making device to obtain a hand-eye relationship; a first shooting module, used to use the calibrated visual unit to shoot the target reference hole to obtain a rough image of the reference hole; a coarse positioning module, used to coarsely position the target reference hole according to the rough image of the reference hole to obtain a rough coordinate of the reference hole; a second shooting module, used to move the visual unit to the rough coordinate of the reference hole based on the hand-eye relationship to shoot the target reference hole to obtain a fine image of the reference hole; a processing module, used to perform local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole; and a fine positioning module, used to perform circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate fine coordinates of the reference hole.

[0032] A third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reference hole visual positioning method for a portable hole making device as described in the above embodiment.

[0033] A fourth aspect of the present invention provides a computer program product, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0034] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0035] The reference hole visual positioning method and device for portable hole-making equipment proposed in the embodiment of the present invention combine advanced deep learning algorithms with traditional image processing methods, and perform module design for measurement conditions such as drastic changes in illumination and uncertain surface scratch states, thereby achieving the hole-making requirements of robust illumination, accurate positioning, efficient and rapid identification and positioning of reference holes; not only can high-precision hand-eye calibration be performed, but also the relative pose of the camera to the plane to be measured and the main axis to the plane to be measured can be solved, which is used to evaluate the verticality of the structure installation; compared with the previous method of finely setting thresholds for edge detection, a robust recognition method of active shielding of scratches and active edge drawing is adopted, which can achieve high-precision positioning effects when lighting conditions change and parameters are not finely adjusted; it can directly improve the position accuracy of automatic hole making of rivet holes on the frame beams inside the cabin, improve the reliability of the operation of automation equipment, and has a very positive significance.

[0036] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 A flowchart of a reference hole visual positioning method for a portable hole making device provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the layout of the main units and the relationship between the coordinate systems of a portable hole-making device provided by an embodiment of the present invention;

[0040] Figure 3 A structural diagram of a spindle measuring tool provided by an embodiment of the present invention;

[0041] Figure 4 A structural diagram of a calibration plate measurement tool provided by an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a specific operation execution of a reference hole visual positioning method for a portable hole making device provided by an embodiment of the present invention;

[0043] Figure 6 A block diagram of a reference hole visual positioning device for a portable hole making device provided by an embodiment of the present invention;

[0044] Figure 7 The present invention is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0045] Explanation of the reference numerals: 10 - equipment spindle unit, 11 - visual unit, 20 - spindle measuring fixture, 21 - first laser tracker measuring target ball, 22 - positioning pin, 30 - calibration plate measuring fixture, 31 - chessboard calibration plate, 32 - second laser tracker measuring target ball, 33 - clamping block, 34 - positioning block, 40 - laser tracker, 60 - reference hole visual positioning device for portable hole making equipment, 601 - calibration module, 602 - first shooting module, 603 - coarse positioning module, 604 - second shooting module, 605 - processing module and 606 - fine positioning module. DETAILED DESCRIPTION

[0046] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0047] The following describes a reference hole visual positioning method and device for a portable hole-making device according to an embodiment of the present invention with reference to the accompanying drawings.

[0048] Figure 1 A schematic flow chart of a reference hole visual positioning method for a portable hole making device provided in an embodiment of the present invention.

[0049] like Figure 1 As shown, the reference hole visual positioning method for a portable hole making device comprises the following steps:

[0050] In step S101, hand-eye calibration is performed on the visual unit of the target portable hole-making device to obtain the hand-eye relationship.

[0051] In some embodiments, hand-eye calibration is performed on a visual unit of a target portable hole-making device to obtain a hand-eye relationship, including:

[0052] Connecting a target spindle measuring tool to a device spindle unit of a target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0053] The target calibration plate measuring fixture is placed flat on the processing plane, and the target calibration plate measuring fixture is plane-positioned by the positioning block, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls;

[0054] Connect the equipment spindle unit and the vision unit in the horizontal and vertical directions, and ensure that the target calibration plate measurement fixture appears in the field of view of the vision unit;

[0055] Using a laser tracker to measure the positions of the target ball measured by the first laser tracker and the target ball measured by the second laser tracker, so as to fit the first posture of the target spindle measurement fixture relative to the laser tracker and the second posture of the target calibration plate measurement fixture relative to the laser tracker;

[0056] Using the visual unit to shoot the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0057] Perform sub-pixel corner extraction on the checkerboard calibration plate image to obtain the third pose of the checkerboard calibration plate relative to the visual unit;

[0058] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose, and the third pose, so as to use the fourth pose as a hand-eye relationship.

[0059] It should be noted that if Figure 2-4 As shown, the target portable hole-making equipment includes a device spindle unit 10, a visual unit 11, a spindle measuring fixture 20, a calibration plate measuring fixture 30 and a laser tracker 40, wherein the spindle measuring fixture 20 is equipped with a plurality of first laser tracker measuring target balls 21, which are fixed to the reserved spindle holes by positioning pins 22. The calibration plate measuring fixture 30 is placed flat on the processing plane, and is equipped with a checkerboard calibration plate 31 and a plurality of second laser tracker measuring target balls 32, wherein the checkerboard calibration plate 31 is plane-positioned by a positioning block 34 and clamped by a clamping block 33. The device spindle unit 10 and the visual unit 11 are fixedly connected in the X and Y directions by mechanical connection, and can be moved in the X and Y directions to ensure that the calibration plate appears in the camera field of view.

[0060] In the actual implementation process, the hand-eye calibration is completed by hardware such as the spindle measurement fixture 20, the calibration plate measurement fixture 30, the high-precision checkerboard calibration plate 31, the laser tracker 40, and the measurement target balls 21 and 32, combined with the extraction and conversion algorithms of each measurement coordinate system. Specifically, during the hand-eye calibration, the laser tracker 40 fits the first position of the spindle measurement fixture 20 relative to the laser tracker 40 by measuring the positions of the target balls 21 and 32. and the calibration plate measuring device 30 relative to the second position of the laser tracker 40 The visual unit 11 captures the high-precision checkerboard calibration plate 31 and extracts sub-pixel corner points. After calculation, the third position of the high-precision checkerboard calibration plate 31 relative to the visual unit 11 can be obtained. The fourth posture of the visual unit 11 relative to the device spindle unit 10 can be calculated based on the first posture, the second posture and the third posture. Complete the hand-eye calibration and use the fourth posture as the hand-eye relationship.

[0061] In step S102, the target reference hole is photographed using the calibrated visual unit to obtain a rough image of the reference hole.

[0062] In step S103, the target reference hole is roughly positioned according to the rough image of the reference hole to obtain the rough coordinates of the reference hole.

[0063] In some embodiments, coarse positioning of the target reference hole according to the coarse image of the reference hole to obtain coarse coordinates of the reference hole includes:

[0064] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0065] Construct a pixel set containing reference hole semantics according to the mask image;

[0066] The pixel set containing the semantics of the reference hole is used to solve the coordinates of the center of the reference hole, and the coordinates of the center of the hole are used as the rough coordinates of the reference hole.

[0067] In the actual execution process, the X and Y translation axes are controlled to move the calibrated visual unit to the vicinity of the reference hole position, and the calibrated visual unit 11 is used to capture the reference hole image to be processed, that is, the reference hole rough image.

[0068] Furthermore, the reference hole image can be processed by the deep learning segmentation network YoloV8-seg to extract the mask image. The mask-covered pixels are the pixel set containing the reference hole semantics obtained by network reasoning. Suppose the mask-covered pixel set of the reference hole is P = {(u1,v1),(u2,v2),...,(u n ,v n )}, where (u i ,v i ) represents the coordinates of the i-th pixel, i = 1, 2, ..., n, n is the total number of pixels covered by the mask, then the center coordinates of the reference hole (U c ,V c ) can be calculated by the following formula This coordinate is the rough positioning result of the hole center.

[0069] In step S104, based on the hand-eye relationship, the visual unit is moved to the rough coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole.

[0070] In some embodiments, the visual unit is moved to the rough coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole, including:

[0071] The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the main axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates;

[0072] The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates by using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

[0073] In the actual implementation process, the rough positioning hole center result and the camera pixel center result are transformed into the main axis coordinate system through the hand-eye relationship, and (X c ,Y c ) and (X o ,Y o ), using the incremental relationship Δx = x o -x c , Δy=y o -y c The mobile visual unit 11 moves the camera pixel center to the center of the rough positioning reference hole, and takes pictures again to obtain a precise image of the reference hole for precise positioning.

[0074] In step S105, local adaptive threshold binarization and morphological processing are performed on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole.

[0075] In some embodiments, the reference hole fine image is subjected to local adaptive threshold binarization and morphological processing to generate a binarized image that retains the edge of the target circular hole, including:

[0076] Gaussian blur is performed on the reference hole fine image to obtain a reference hole blurred image;

[0077] Performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image;

[0078] Performing morphological corrosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0079] The scratch interference masking process is performed on the binary image, the first mask sub-image and the second mask sub-image to generate a binary image that retains the edge of the target circular hole.

[0080] In the actual implementation process, the precisely positioned shooting results are Gaussian blurred using a 7×7 kernel function, and then the Sauvola method is used for local adaptive threshold binarization. According to the image size (2448×2048), 7×7 is selected as the binarization kernel function to extract the binary image of the surface of the measured object.

[0081] Furthermore, if Figure 5As shown in the figure, the mask image returned by the deep learning segmentation network is morphologically eroded and expanded twice with a 7×7 kernel function to obtain mask images that are slightly smaller and slightly larger than the edge of the target reference hole, respectively. The image-based intersection and union operation is used to obtain a binary image that only retains the edge of the target circular hole, thereby realizing the scratch interference masking of the reference hole binary image.

[0082] In step S106, circular hole feature fitting is performed on the binary image retaining the edge of the target circular hole to generate precise coordinates of the reference hole.

[0083] In some embodiments, performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate the actual coordinates of the reference hole includes:

[0084] Actively draw the edge of the binary image retaining the edge of the target circular hole to obtain a binary image with an edge; perform arc segment fitting on the binary image with an edge, and merge arcs with similar centers and radii to obtain an arc image with complete edge points;

[0085] The gradient change is used to select the reliable circle with the lowest false alarm value in the arc image with complete edge points, and the coordinate transformation of the reliable circle is performed to generate the precise coordinates of the reference hole.

[0086] In the actual implementation process, the edge is first drawn actively using the edge drawing method, and then the edge is used for arc segment fitting. In the arc segment fitting process, the edge is first approximated by the line segment, and then the rotation angle θ between adjacent line segments is calculated. 12 ,θ 23 ,θ 34 ,…, if three or more adjacent line segments have the same turning angle, they are considered to be the same arc segment and are fitted. After the arc segment fitting is completed, the arcs with similar centers and radii are merged to retain the more complete arcs with edge points.

[0087] Furthermore, through the false alarm verification step, the gradient change angle of the gray value is used to determine whether the edge point is within the angle change range limited by the roundness, so as to select reliable circles with higher roundness. Among them, the gray gradient value is verified using a 2*2 block, and the angle calculation formula is as follows When the number of sampling points of the fitted circle with α>22.5° accounts for the total number of sampling points When , the circle is considered to be a false alarm circle and is discarded. The circle with the lowest false alarm value among the remaining circles is extracted as the reference hole position, and the precise positioning is completed after coordinate conversion to obtain the precise coordinates of the reference hole.

[0088] In summary, the reference hole visual positioning method and device for portable hole making equipment proposed in the embodiments of the present invention have the following effects:

[0089] (1) Combining advanced deep learning algorithms with traditional image processing methods, the module is designed for measurement conditions such as drastic changes in illumination and uncertain surface scratches, achieving the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of reference holes;

[0090] (2) It can not only perform high-precision hand-eye calibration, but also solve the relative poses of the camera to the plane to be measured and the spindle to the plane to be measured, which is used to evaluate the verticality of the structure installation;

[0091] (3) Compared with the previous robust recognition method that uses active scratch shielding and active edge drawing, it can achieve high-precision positioning effect under changing lighting conditions and without fine parameter adjustment;

[0092] (4) It can directly improve the position accuracy of automatic drilling of rivet holes on the internal frame beams of the cabin and improve the reliability of the operation of automated equipment, which has a very positive significance.

[0093] Next, a reference hole visual positioning device for a portable hole making device according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0094] Figure 6 A block diagram of a reference hole visual positioning device for a portable hole making device provided in an embodiment of the present invention.

[0095] like Figure 6 As shown, the reference hole visual positioning device 60 for portable hole making equipment includes: a calibration module 601, a first shooting module 602, a coarse positioning module 603, a second shooting module 604, a processing module 605 and a fine positioning module 606.

[0096] Among them, the calibration module 601 is used to perform hand-eye calibration on the visual unit of the target portable hole-making equipment to obtain the hand-eye relationship. The first shooting module 602 is used to use the calibrated visual unit to shoot the target reference hole to obtain a rough image of the reference hole. The coarse positioning module 603 is used to coarsely position the target reference hole according to the rough image of the reference hole to obtain the rough coordinates of the reference hole. The second shooting module 604 is used to move the visual unit to the rough coordinates of the reference hole based on the hand-eye relationship to shoot the target reference hole to obtain a fine image of the reference hole. The processing module 605 is used to perform local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole. The fine positioning module 606 is used to perform circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate the fine coordinates of the reference hole.

[0097] In some embodiments, the calibration module 601 includes:

[0098] Connecting a target spindle measuring tool to a device spindle unit of a target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0099] The target calibration plate measuring fixture is placed flat on the processing plane, and the target calibration plate measuring fixture is plane-positioned by the positioning block, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls;

[0100] Connect the equipment spindle unit and the vision unit in the horizontal and vertical directions, and ensure that the target calibration plate measurement fixture appears in the field of view of the vision unit;

[0101] Using a laser tracker to measure the positions of the target ball measured by the first laser tracker and the target ball measured by the second laser tracker, so as to fit the first posture of the target spindle measurement fixture relative to the laser tracker and the second posture of the target calibration plate measurement fixture relative to the laser tracker;

[0102] Using the visual unit to shoot the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0103] Perform sub-pixel corner extraction on the checkerboard calibration plate image to obtain the third pose of the checkerboard calibration plate relative to the visual unit;

[0104] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose, and the third pose, so as to use the fourth pose as a hand-eye relationship.

[0105] In some embodiments, the coarse positioning module 603 includes:

[0106] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0107] Construct a pixel set containing reference hole semantics according to the mask image;

[0108] The pixel set containing the semantics of the reference hole is used to solve the coordinates of the center of the reference hole, and the coordinates of the center of the hole are used as the rough coordinates of the reference hole.

[0109] In some embodiments, the second shooting module 604 includes:

[0110] The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the main axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates;

[0111] The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates by using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

[0112] In some embodiments, the processing module 605 includes:

[0113] Gaussian blur is performed on the reference hole fine image to obtain a reference hole blurred image;

[0114] Performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image;

[0115] Performing morphological corrosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0116] The scratch interference masking process is performed on the binary image, the first mask sub-image and the second mask sub-image to generate a binary image that retains the edge of the target circular hole.

[0117] In some embodiments, the fine positioning module 606 includes:

[0118] Actively draw the edge of the binary image retaining the edge of the target circular hole to obtain a binary image with an edge;

[0119] Perform arc segment fitting on the binary image with edges, and merge arcs with similar centers and radii to obtain an arc image with complete edge points;

[0120] The gradient change is used to select the reliable circle with the lowest false alarm value in the arc image with complete edge points, and the coordinate transformation of the reliable circle is performed to generate the precise coordinates of the reference hole.

[0121] It should be noted that the aforementioned explanation of the embodiment of the reference hole visual positioning method for a portable hole-making device is also applicable to the reference hole visual positioning device for a portable hole-making device of this embodiment, and will not be repeated here.

[0122] The reference hole visual positioning device for portable hole making equipment proposed in an embodiment of the present invention has the following effects:

[0123] (1) Combining advanced deep learning algorithms with traditional image processing methods, the module is designed for measurement conditions such as drastic changes in illumination and uncertain surface scratches, achieving the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of reference holes;

[0124] (2) It can not only perform high-precision hand-eye calibration, but also solve the relative poses of the camera to the plane to be measured and the spindle to the plane to be measured, which is used to evaluate the verticality of the structure installation;

[0125] (3) Compared with the previous robust recognition method that uses active scratch shielding and active edge drawing, it can achieve high-precision positioning effect under changing lighting conditions and without fine parameter adjustment;

[0126] (4) It can directly improve the position accuracy of automatic drilling of rivet holes on the internal frame beams of the cabin and improve the reliability of the operation of automated equipment, which has a very positive significance.

[0127] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0128] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0129] When the processor 702 executes the program, the reference hole visual positioning method for the portable hole making device provided in the above embodiment is implemented.

[0130] Furthermore, the electronic device further comprises:

[0131] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0132] The memory 701 is used to store computer programs that can be executed on the processor 702 .

[0133] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0134] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0135] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0136] The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0137] An embodiment of the present invention further provides a computer program product, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0138] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0140] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0141] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0143] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0145] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0146] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A reference hole visual positioning method for a portable hole making device, characterized in that: The following steps are involved: Perform hand-eye calibration on the visual unit of the target portable hole-making equipment to obtain the hand-eye relationship; The target reference hole is photographed using the calibrated visual unit to obtain a rough image of the reference hole; Coarsely positioning the target reference hole according to the coarse image of the reference hole to obtain coarse coordinates of the reference hole; Based on the hand-eye relationship, the visual unit is moved to the rough coordinates of the reference hole to photograph the target reference hole to obtain a precise image of the reference hole; Performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole; The binary image retaining the edge of the target circular hole is subjected to circular hole feature fitting to generate precise coordinates of the reference hole.

2. The reference hole visual positioning method for portable hole making equipment according to claim 1 is characterized in that: The hand-eye calibration of the visual unit of the target portable hole-making device to obtain the hand-eye relationship includes: Connecting a target spindle measuring tool to a device spindle unit of the target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls; Placing a target calibration plate measuring tool flat on a processing plane, and performing plane positioning of the target calibration plate measuring tool by a positioning block, wherein the target calibration plate measuring tool is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls; Connecting the spindle unit of the device to the visual unit in the horizontal direction and the vertical direction, and ensuring that the target calibration plate measurement fixture appears within the field of view of the visual unit; Using a laser tracker to measure the positions of the first laser tracker measuring target ball and the second laser tracker measuring target ball, so as to fit the first posture of the target spindle measuring fixture relative to the laser tracker and the second posture of the target calibration plate measuring fixture relative to the laser tracker; Using the visual unit to photograph the checkerboard calibration plate to obtain a checkerboard calibration plate image; Performing sub-pixel corner point extraction on the checkerboard calibration plate image to obtain a third pose of the checkerboard calibration plate relative to the visual unit; A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose and the third pose, so as to use the fourth pose as the hand-eye relationship.

3. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The step of roughly positioning the target reference hole according to the rough image of the reference hole to obtain rough coordinates of the reference hole includes: Performing mask extraction on the reference hole rough image to obtain a mask image; Constructing a pixel set containing reference hole semantics according to the mask image; The pixel set containing the reference hole semantics is used to solve the hole center coordinates of the reference hole, and the hole center coordinates are used as the reference hole rough coordinates.

4. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The step of moving the visual unit to the rough coordinates of the reference hole to photograph the target reference hole to obtain a precise image of the reference hole includes: The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the main axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates; The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates by using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

5. The reference hole visual positioning method for portable hole making equipment according to claim 3, characterized in that: The performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image retaining the edge of the target circular hole includes: Performing Gaussian blur on the reference hole fine image to obtain a reference hole blurred image; Performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image; Performing morphological corrosion and morphological expansion on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole; The binary image, the first mask sub-image and the second mask sub-image are subjected to scratch interference masking processing to generate the binary image retaining the edge of the target circular hole.

6. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate the actual coordinates of the reference hole includes: Actively drawing edges on the binary image retaining the edge of the target circular hole to obtain a binary image with edges; Performing arc segment fitting on the binary image with edges, and merging arcs with similar centers and radii to obtain an arc image with complete edge points; A reliable circle with the lowest false alarm value in the circular arc image with complete edge points is screened by utilizing gradient changes, and coordinate transformation is performed on the reliable circle to generate the precise coordinates of the reference hole.

7. A reference hole visual positioning device for a portable hole making device, characterized in that: include: A calibration module, used for performing hand-eye calibration on a visual unit of a target portable hole-making device to obtain a hand-eye relationship; A first shooting module is used to shoot the target reference hole using the calibrated visual unit to obtain a rough image of the reference hole; A coarse positioning module, used for coarsely positioning the target reference hole according to the coarse image of the reference hole to obtain the coarse coordinates of the reference hole; A second shooting module is used to move the visual unit to the rough coordinates of the reference hole based on the hand-eye relationship to shoot the target reference hole to obtain a fine image of the reference hole; A processing module, used for performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole; The precise positioning module is used to perform circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate precise coordinates of the reference hole.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reference hole visual positioning method for a portable hole making device as described in any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program / instruction is executed by a processor, the reference hole visual positioning method for a portable hole making device described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the reference hole visual positioning method for a portable hole making device as described in any one of claims 1 to 6.

Images