Reference hole visual positioning method and device for portable hole making equipment

Abstract

The present invention relates to the field of aircraft digital assembly and manufacturing technology, and more particularly to a method and apparatus for visually locating reference holes in portable hole-making equipment. The method comprises: performing hand-eye calibration on a visual unit of a target portable hole-making equipment, photographing a target reference hole using the calibrated visual unit to obtain a coarse image of the reference hole; coarsely locating the target reference hole based on the coarse image of the reference hole to obtain coarse coordinates of the reference hole; moving the visual unit to the coarse coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole; and performing local adaptive threshold binarization, morphological processing, and circular hole feature fitting on the fine image of the reference hole to generate the fine coordinates of the reference hole. This method solves the problem that existing visual measurement methods rely on precise edge detection parameter settings and careful light source arrangement, are prone to missed detections, false detections, and positioning offsets, and cannot meet the requirements for reference hole positioning in portable hole-making equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of digital assembly and manufacturing of aircraft, and in particular to a method and device for visually positioning reference holes of a portable hole-making device suitable for machining internal plane laminated holes of aircraft in the aviation field. Background Art

[0002] Aircraft require a large number of flat riveted holes to be drilled inside them. However, due to the confined interior, conventional robotic arms, machine tools, and other automated equipment are difficult to perform. Therefore, drilling holes is currently performed manually using drilling templates, which is time-consuming and labor-intensive. A three-axis portable drilling device was designed and developed to meet this need. Due to its small size and light weight, it can be manually carried into the cabin. Once positioned and clamped, it can then be automated for drilling. This improves efficiency, saves manpower, and ensures consistent hole quality. However, due to the inherent characteristics of the portable drilling device, after installation inside the cabin, the center of the hole must be located using the laminated pre-connector pins as the reference hole to determine the relative position of the hole to be machined within the device's coordinate system. The positioning of the reference hole determines the absolute position of all holes to be machined. Therefore, accurate reference hole identification and positioning are essential for high-quality drilling.

[0003] Visual measurement methods have been widely adopted across the industry due to their low-cost, non-contact, and efficient nature. However, existing technologies generally rely on precise edge detection parameter settings and careful light source placement. In factory environments with fluctuating lighting conditions and workpiece surface scratches with varying states, missed detections, incorrect detections, and positioning offsets are common, making them inadequate for the reference hole positioning requirements of portable hole-making equipment.

[0004] Therefore, there is an urgent need for a reference hole visual positioning method for portable hole-making equipment to meet the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of reference holes. Summary of the Invention

[0005] The present invention provides a method and device for visual positioning of reference holes in portable hole-making equipment, so as to solve the problems that existing visual measurement methods rely on precise edge detection parameter settings and careful light source arrangement, are prone to missed detection, wrong detection and positioning offset, and cannot meet the reference hole positioning requirements of portable hole-making equipment.

[0006] A first aspect of the present invention provides a method for visual positioning of a reference hole of a portable hole-making device, comprising the following steps: performing hand-eye calibration on a visual unit of a target portable hole-making device to obtain a hand-eye relationship; photographing a target reference hole using the calibrated visual unit to obtain a coarse image of the reference hole; coarsely positioning the target reference hole based on the coarse image of the reference hole to obtain a coarse coordinate of the reference hole; moving the visual unit to the coarse coordinate of the reference hole based on the hand-eye relationship to photograph the target reference hole to obtain a fine image of the reference hole; performing local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole; and performing circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate fine coordinates of the reference hole.

[0007] Optionally, performing hand-eye calibration on a visual unit of the target portable hole-making device to obtain a hand-eye relationship includes:

[0008] Connecting a target spindle measuring tool to a device spindle unit of the target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0009] Placing a target calibration plate measuring fixture flat on a processing plane and performing plane positioning of the target calibration plate measuring fixture by a positioning block, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls;

[0010] Connecting the spindle unit of the device to the vision unit in the horizontal and vertical directions, and ensuring that the target calibration plate measurement fixture appears within the field of view of the vision unit;

[0011] Using a laser tracker to measure the positions of the first laser tracker measurement target sphere and the second laser tracker measurement target sphere, so as to fit a first pose of the target spindle measurement fixture relative to the laser tracker and a second pose of the target calibration plate measurement fixture relative to the laser tracker;

[0012] Using the visual unit to photograph the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0013] Performing sub-pixel corner extraction on the checkerboard calibration plate image to obtain a third pose of the checkerboard calibration plate relative to the visual unit;

[0014] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose, and the third pose, so as to use the fourth pose as the hand-eye relationship.

[0015] Optionally, the coarse positioning of the target reference hole according to the reference hole coarse image to obtain the reference hole coarse coordinates includes:

[0016] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0017] Constructing a pixel set containing reference hole semantics according to the mask image;

[0018] The pixel set containing the reference hole semantics is used to solve the hole center coordinates of the reference hole, and the hole center coordinates are used as the reference hole rough coordinates.

[0019] Optionally, the moving the visual unit to the rough coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole includes:

[0020] The reference hole coarse coordinates and the pixel center coordinates of the visual unit are transformed into a principal axis coordinate system through the hand-eye relationship to obtain transformed reference hole coarse coordinates and transformed pixel center coordinates;

[0021] The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

[0022] Optionally, performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole includes:

[0023] Performing Gaussian blurring on the reference hole fine image to obtain a reference hole blurred image;

[0024] performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image;

[0025] Performing morphological erosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0026] The binary image, the first mask sub-image, and the second mask sub-image are subjected to scratch interference masking processing to generate the binary image retaining the edge of the target circular hole.

[0027] Optionally, performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate actual coordinates of the reference hole includes:

[0028] Actively drawing edges on the binary image retaining the edge of the target circular hole to obtain a binary image with edges;

[0029] Performing arc segment fitting on the binary image with edges, and merging arcs with similar centers and radii to obtain an arc image with complete edge points;

[0030] Gradient changes are used to screen a reliable circle with the lowest false alarm value in the arc image with complete edge points, and coordinate transformation is performed on the reliable circle to generate the precise coordinates of the reference hole.

[0031] A second aspect of the present invention provides a visual positioning device for a reference hole of a portable hole-making device, comprising: a calibration module for performing hand-eye calibration on a visual unit of a target portable hole-making device to obtain a hand-eye relationship; a first shooting module for using the calibrated visual unit to shoot the target reference hole to obtain a rough image of the reference hole; a coarse positioning module for coarsely positioning the target reference hole according to the rough image of the reference hole to obtain a rough coordinate of the reference hole; a second shooting module for moving the visual unit to the rough coordinate of the reference hole based on the hand-eye relationship to shoot the target reference hole to obtain a fine image of the reference hole; a processing module for performing local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole; and a fine positioning module for performing circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate fine coordinates of the reference hole.

[0032] An embodiment of the third aspect of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the reference hole visual positioning method for a portable hole-making device as described in the above embodiment.

[0033] A fourth aspect of the present invention provides a computer program product, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0034] A fifth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0035] The visual positioning method and device for reference holes in portable hole-making equipment proposed in the embodiments of the present invention combine advanced deep learning algorithms with traditional image processing methods, and perform module design for measurement conditions such as drastic changes in lighting and uncertain surface scratch states, thereby achieving the hole-making requirements of robust lighting, accurate positioning, efficient and rapid identification and positioning of reference holes; it can not only perform high-precision hand-eye calibration, but also solve the relative pose of the camera to the plane to be measured and the main axis to the plane to be measured, which is used to evaluate the verticality of the structural installation; compared with the previous method of finely setting thresholds for edge detection, a robust recognition method of active shielding of scratches and active edge drawing is adopted, which can achieve high-precision positioning effects when lighting conditions change and parameters are not fine-tuned; it can directly improve the position accuracy of automatic hole making of rivet holes on the frame beams inside the cabin, improve the reliability of the operation of automated equipment, and has very positive significance.

[0036] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 A flowchart of a reference hole visual positioning method for a portable hole-making device provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the layout of the main units and the coordinate system relationship of a portable hole-making device provided by an embodiment of the present invention;

[0040] Figure 3 A structural diagram of a spindle measurement tool provided by an embodiment of the present invention;

[0041] Figure 4 A structural diagram of a calibration plate measurement tool provided by an embodiment of the present invention;

[0042] Figure 5 A schematic diagram of a specific operation execution of a reference hole visual positioning method for a portable hole-making device provided by an embodiment of the present invention;

[0043] Figure 6 A block diagram of a reference hole visual positioning device for a portable hole-making device provided by an embodiment of the present invention;

[0044] Figure 7 The present invention provides a schematic structural diagram of an electronic device.

[0045] Explanation of the accompanying drawings: 10-equipment spindle unit, 11-vision unit, 20-spindle measuring fixture, 21-first laser tracker measuring target ball, 22-positioning pin, 30-calibration plate measuring fixture, 31-chessboard calibration plate, 32-second laser tracker measuring target ball, 33-clamping block, 34-positioning block, 40-laser tracker, 60-reference hole visual positioning device for portable hole-making equipment, 601-calibration module, 602-first shooting module, 603-coarse positioning module, 604-second shooting module, 605-processing module and 606-fine positioning module. DETAILED DESCRIPTION

[0046] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0047] The following describes a method and apparatus for visually positioning a reference hole in a portable hole-making device according to an embodiment of the present invention with reference to the accompanying drawings.

[0048] Figure 1 The present invention provides a flowchart of a method for visually locating a reference hole in a portable hole-making device.

[0049] like Figure 1 As shown, the reference hole visual positioning method for a portable hole making device includes the following steps:

[0050] In step S101 , hand-eye calibration is performed on the visual unit of the target portable hole-making device to obtain a hand-eye relationship.

[0051] In some embodiments, performing hand-eye calibration on a vision unit of a target portable hole-making device to obtain a hand-eye relationship includes:

[0052] Connecting a target spindle measuring tool to a device spindle unit of a target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0053] Placing the target calibration plate measuring fixture flat on the processing plane and performing plane positioning by using the positioning block target calibration plate measuring fixture, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measurement target balls;

[0054] Connect the equipment spindle unit and the vision unit in the horizontal and vertical directions, and ensure that the target calibration plate measurement fixture appears in the field of view of the vision unit;

[0055] Using a laser tracker to measure the positions of the target sphere measured by the first laser tracker and the target sphere measured by the second laser tracker, so as to fit a first pose of the target spindle measurement fixture relative to the laser tracker and a second pose of the target calibration plate measurement fixture relative to the laser tracker;

[0056] Using the visual unit to shoot the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0057] Perform sub-pixel corner extraction on the checkerboard calibration plate image to obtain the third pose of the checkerboard calibration plate relative to the visual unit;

[0058] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose, and the third pose, so as to use the fourth pose as a hand-eye relationship.

[0059] It should be noted that if Figure 2-4 As shown, the target portable hole-making equipment includes a spindle unit 10, a vision unit 11, a spindle measurement fixture 20, a calibration plate measurement fixture 30, and a laser tracker 40. The spindle measurement fixture 20 is equipped with a plurality of first laser tracker measurement target balls 21, which are fixed to the spindle reserved holes by positioning pins 22. The calibration plate measurement fixture 30 is placed flat on the processing surface and is equipped with a checkerboard calibration plate 31 and a plurality of second laser tracker measurement target balls 32. The checkerboard calibration plate 31 is positioned in the plane by positioning blocks 34 and clamped by clamping blocks 33. The spindle unit 10 and the vision unit 11 are fixedly connected in the X and Y directions by a mechanical connection and can be moved in the X and Y directions to ensure that the calibration plate appears in the camera's field of view.

[0060] In the actual implementation process, the hand-eye calibration is completed by the spindle measurement fixture 20, the calibration plate measurement fixture 30, the high-precision checkerboard calibration plate 31, the laser tracker 40 and the measurement target balls 21 and 32, etc., combined with the extraction and conversion algorithms of each measurement coordinate system. Specifically, during the hand-eye calibration, the laser tracker 40 fits the position of the spindle measurement fixture 20 relative to the laser tracker 40 by measuring the target balls 21 and 32. and the calibration plate measuring fixture 30 relative to the second posture of the laser tracker 40 The visual unit 11 shoots the high-precision checkerboard calibration plate 31 and performs sub-pixel corner extraction. After solving, the third pose of the high-precision checkerboard calibration plate 31 relative to the visual unit 11 can be obtained. , the fourth posture of the visual unit 11 relative to the device spindle unit 10 can be calculated based on the first posture, the second posture and the third posture , complete the hand-eye calibration, and use the fourth posture as the hand-eye relationship.

[0061] In step S102, the target reference hole is photographed using the calibrated visual unit to obtain a rough image of the reference hole.

[0062] In step S103 , the target reference hole is roughly positioned according to the reference hole rough image to obtain the reference hole rough coordinates.

[0063] In some embodiments, coarse positioning of the target reference hole according to the coarse image of the reference hole to obtain coarse coordinates of the reference hole includes:

[0064] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0065] Construct a pixel set containing reference hole semantics based on the mask image;

[0066] The pixel set containing the semantics of the reference hole is used to solve the coordinates of the center of the reference hole, and the coordinates of the center of the hole are used as the rough coordinates of the reference hole.

[0067] In the actual execution process, the X and Y translation axes are controlled to move the calibrated visual unit to the vicinity of the reference hole position, and the calibrated visual unit 11 is used to capture the reference hole image to be processed, that is, the reference hole rough image.

[0068] Furthermore, the reference hole image can be processed by the deep learning segmentation network YoloV8-seg to extract the mask image. The mask-covered pixels are the pixel set containing the reference hole semantics obtained by network inference. Let the mask-covered pixel set of the reference hole be ,in, Indicates the The coordinates of the pixels, , is the total number of pixels covered by the mask, then the center coordinates of the reference hole It can be calculated by the following formula , , which is the rough positioning result of the hole center.

[0069] In step S104 , based on the hand-eye relationship, the visual unit is moved to the rough coordinates of the reference hole to photograph the target reference hole and obtain a fine image of the reference hole.

[0070] In some embodiments, moving the vision unit to the rough coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole includes:

[0071] The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the principal axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates;

[0072] The pixel center coordinates of the visual unit are moved to the coarse coordinates of the reference hole using the converted coarse coordinates of the reference hole and the converted pixel center coordinates, and the target reference hole is photographed to obtain a fine image of the reference hole.

[0073] In the actual execution process, the rough positioning hole center result and the camera pixel center result are transformed into the main axis coordinate system through the hand-eye relationship to obtain and , using the incremental relationship , The mobile vision unit 11 moves the camera pixel center to the center of the rough positioning reference hole, and takes pictures again to obtain a fine image of the reference hole for fine positioning.

[0074] In step S105 , the reference hole fine image is subjected to local adaptive threshold binarization and morphological processing to generate a binarized image that retains the edge of the target circular hole.

[0075] In some embodiments, performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole includes:

[0076] Performing Gaussian blur on the reference hole fine image to obtain a reference hole blurred image;

[0077] Performing local adaptive threshold binarization on the reference hole blurred image to extract a binary image;

[0078] Performing morphological corrosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0079] The binary image, the first mask sub-image and the second mask sub-image are subjected to scratch interference masking processing to generate a binary image that retains the edge of the target circular hole.

[0080] In the actual implementation process, the precise positioning shooting results are Gaussian blurred using a 7×7 kernel function, and then the Sauvola method is used for local adaptive threshold binarization. According to the image size (2448×2048), 7×7 is selected as the binarization kernel function to extract the binary image of the surface of the measured object.

[0081] Further, if Figure 5 As shown in the figure, the mask image returned by the deep learning segmentation network is subjected to two generations of morphological corrosion and two generations of morphological dilation with a 7×7 kernel function to obtain mask images that are slightly smaller and slightly larger than the edge of the target reference hole, respectively. The image-based intersection and union operation is used to obtain a binary image that only retains the edge of the target circular hole, thereby realizing the scratch interference masking of the reference hole binary image.

[0082] In step S106 , circular hole feature fitting is performed on the binary image retaining the edge of the target circular hole to generate precise coordinates of the reference hole.

[0083] In some embodiments, performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate the actual coordinates of the reference hole includes:

[0084] Actively draw edges on the binary image that retains the edge of the target circular hole to obtain a binary image with edges;

[0085] Perform arc segment fitting on the binary image with edges and merge arcs with similar centers and radii to obtain an arc image with complete edge points.

[0086] Gradient changes are used to screen the reliable circle with the lowest false alarm value in the arc image with complete edge points, and the coordinate transformation of the reliable circle is performed to generate the precise coordinates of the reference hole.

[0087] In the actual implementation process, the edge is first drawn actively using the edge drawing method, and then the edge is used to perform arc fitting. In the arc fitting process, the edge is first approximated by the line segment, and then the angle between adjacent line segments is calculated. , , If three or more adjacent line segments have the same turning angle, they are considered to be the same arc segment and are fitted. After the arc segment fitting is completed, the arcs with similar centers and radii are merged to retain the arcs with more complete edge points.

[0088] Furthermore, through the false alarm verification step, the gradient change angle of the grayscale value is used to determine whether the edge point is within the angle change range limited by the roundness, so as to screen reliable circles with higher roundness. Among them, the grayscale gradient value is verified using a 2*2 square. The angle calculation formula is as follows , , When the sampling points of the fitted circle are The number of sampling points accounts for When , the circle is considered a false alarm circle and discarded. The circle with the lowest false alarm value among the remaining circles is extracted as the reference hole position, and the coordinate conversion is performed to complete the precise positioning and obtain the precise coordinates of the reference hole.

[0089] In summary, the reference hole visual positioning method and device for portable hole-making equipment proposed in the embodiments of the present invention have the following specific effects:

[0090] (1) Combining advanced deep learning algorithms with traditional image processing methods, a module is designed for measurement conditions such as drastic changes in illumination and uncertain surface scratches, achieving the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of reference holes;

[0091] (2) It can not only perform high-precision hand-eye calibration, but also solve the relative poses of the camera to the plane to be measured and the spindle to the plane to be measured, which is used to evaluate the verticality of the structure installation;

[0092] (3) Compared with the previous robust recognition method that uses active scratch shielding and active edge drawing, it can achieve high-precision positioning under changing lighting conditions and without fine-tuning parameters;

[0093] (4) It can directly improve the position accuracy of the automatic drilling of rivet holes on the internal frame beams of the cabin and improve the reliability of the operation of the automation equipment, which has a very positive significance.

[0094] Next, a reference hole visual positioning device for a portable hole-making device according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0095] Figure 6 A block diagram of a reference hole visual positioning device for a portable hole-making device provided by an embodiment of the present invention.

[0096] like Figure 6 As shown, the reference hole visual positioning device 60 for portable hole-making equipment includes: a calibration module 601 , a first shooting module 602 , a coarse positioning module 603 , a second shooting module 604 , a processing module 605 and a fine positioning module 606 .

[0097] Among them, the calibration module 601 is used to perform hand-eye calibration on the visual unit of the target portable hole-making device to obtain the hand-eye relationship. The first shooting module 602 is used to use the calibrated visual unit to shoot the target reference hole to obtain a coarse image of the reference hole. The coarse positioning module 603 is used to coarsely locate the target reference hole based on the coarse image of the reference hole to obtain the coarse coordinates of the reference hole. The second shooting module 604 is used to move the visual unit to the coarse coordinates of the reference hole based on the hand-eye relationship to shoot the target reference hole to obtain a fine image of the reference hole. The processing module 605 is used to perform local adaptive threshold binarization and morphological processing on the fine image of the reference hole to generate a binarized image that retains the edge of the target circular hole. The fine positioning module 606 is used to perform circular hole feature fitting on the binarized image that retains the edge of the target circular hole to generate the fine coordinates of the reference hole.

[0098] In some embodiments, the calibration module 601 includes:

[0099] Connecting a target spindle measuring tool to a device spindle unit of a target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls;

[0100] Placing the target calibration plate measuring fixture flat on the processing plane and performing plane positioning by using the positioning block target calibration plate measuring fixture, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measurement target balls;

[0101] Connect the equipment spindle unit and the vision unit in the horizontal and vertical directions, and ensure that the target calibration plate measurement fixture appears in the field of view of the vision unit;

[0102] Using a laser tracker to measure the positions of the target sphere measured by the first laser tracker and the target sphere measured by the second laser tracker, so as to fit a first pose of the target spindle measurement fixture relative to the laser tracker and a second pose of the target calibration plate measurement fixture relative to the laser tracker;

[0103] Using the visual unit to shoot the checkerboard calibration plate to obtain a checkerboard calibration plate image;

[0104] Perform sub-pixel corner extraction on the checkerboard calibration plate image to obtain the third pose of the checkerboard calibration plate relative to the visual unit;

[0105] A fourth pose of the vision unit relative to the device spindle unit is calculated according to the first pose, the second pose, and the third pose, so as to use the fourth pose as the hand-eye relationship.

[0106] In some embodiments, the coarse positioning module 603 includes:

[0107] Performing mask extraction on the reference hole rough image to obtain a mask image;

[0108] Construct a pixel set containing reference hole semantics based on the mask image;

[0109] The pixel set containing the semantics of the reference hole is used to solve the coordinates of the center of the reference hole, and the coordinates of the center of the hole are used as the rough coordinates of the reference hole.

[0110] In some embodiments, the second camera module 604 includes:

[0111] The rough coordinates of the reference hole and the pixel center coordinates of the visual unit are transformed into the principal axis coordinate system through the hand-eye relationship to obtain the transformed rough coordinates of the reference hole and the transformed pixel center coordinates;

[0112] The pixel center coordinates of the visual unit are moved to the coarse coordinates of the reference hole using the converted coarse coordinates of the reference hole and the converted pixel center coordinates, and the target reference hole is photographed to obtain a fine image of the reference hole.

[0113] In some embodiments, the processing module 605 includes:

[0114] Performing Gaussian blur on the reference hole fine image to obtain a reference hole blurred image;

[0115] Performing local adaptive threshold binarization on the reference hole blurred image to extract a binary image;

[0116] Performing morphological corrosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole;

[0117] The binary image, the first mask sub-image and the second mask sub-image are subjected to scratch interference masking processing to generate a binary image that retains the edge of the target circular hole.

[0118] In some embodiments, the fine positioning module 606 includes:

[0119] Actively draw edges on the binary image that retains the edge of the target circular hole to obtain a binary image with edges;

[0120] Perform arc segment fitting on the binary image with edges and merge arcs with similar centers and radii to obtain an arc image with complete edge points.

[0121] Gradient changes are used to screen the reliable circle with the lowest false alarm value in the arc image with complete edge points, and the coordinate transformation of the reliable circle is performed to generate the precise coordinates of the reference hole.

[0122] It should be noted that the above explanation of the embodiment of the reference hole visual positioning method for portable hole-making equipment is also applicable to the reference hole visual positioning device for portable hole-making equipment of this embodiment, and will not be repeated here.

[0123] The reference hole visual positioning device for portable hole making equipment proposed in an embodiment of the present invention has the following specific effects:

[0124] (1) Combining advanced deep learning algorithms with traditional image processing methods, a module is designed for measurement conditions such as drastic changes in illumination and uncertain surface scratches, achieving the requirements of robust illumination, accurate positioning, and efficient and fast identification and positioning of reference holes;

[0125] (2) It can not only perform high-precision hand-eye calibration, but also solve the relative poses of the camera to the plane to be measured and the spindle to the plane to be measured, which is used to evaluate the verticality of the structure installation;

[0126] (3) Compared with the previous robust recognition method that uses active scratch shielding and active edge drawing, it can achieve high-precision positioning under changing lighting conditions and without fine-tuning parameters;

[0127] (4) It can directly improve the position accuracy of the automatic drilling of rivet holes on the internal frame beams of the cabin and improve the reliability of the operation of the automation equipment, which has a very positive significance.

[0128] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device may include:

[0129] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0130] When the processor 702 executes the program, the reference hole visual positioning method for the portable hole-making device provided in the above embodiment is implemented.

[0131] Furthermore, the electronic device further includes:

[0132] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0133] The memory 701 is used to store computer programs that can be run on the processor 702 .

[0134] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0135] If the memory 701, processor 702, and communication interface 703 are implemented independently, the communication interface 703, memory 701, and processor 702 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0136] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0137] The processor 702 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0138] An embodiment of the present invention further provides a computer program product, which implements the above-mentioned reference hole visual positioning method for a portable hole-making device when the computer program / instructions are executed by a processor.

[0139] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned reference hole visual positioning method for a portable hole-making device.

[0140] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0141] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0142] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0143] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0144] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any of the following technologies known in the art or a combination thereof can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0145] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0146] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0147] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A reference hole visual positioning method for portable hole making equipment, characterized in that: The following steps are involved: Perform hand-eye calibration on the visual unit of the target portable hole-making device to obtain the hand-eye relationship, which specifically includes: Connecting a target spindle measuring tool to a device spindle unit of the target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls; Placing a target calibration plate measuring fixture flat on a processing plane and performing plane positioning of the target calibration plate measuring fixture by a positioning block, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls; Connecting the spindle unit of the device to the vision unit in the horizontal and vertical directions, and ensuring that the target calibration plate measurement fixture appears within the field of view of the vision unit; Using a laser tracker to measure the positions of the first laser tracker measurement target sphere and the second laser tracker measurement target sphere, so as to fit a first pose of the target spindle measurement fixture relative to the laser tracker and a second pose of the target calibration plate measurement fixture relative to the laser tracker; Using the visual unit to photograph the checkerboard calibration plate to obtain a checkerboard calibration plate image; Performing sub-pixel corner extraction on the checkerboard calibration plate image to obtain a third pose of the checkerboard calibration plate relative to the visual unit; calculating a fourth pose of the vision unit relative to the device spindle unit according to the first pose, the second pose, and the third pose, so as to use the fourth pose as the hand-eye relationship; The target reference hole is photographed using the calibrated visual unit to obtain a rough image of the reference hole; Coarsely positioning the target reference hole according to the reference hole coarse image to obtain the reference hole coarse coordinates; Based on the hand-eye relationship, the visual unit is moved to the rough coordinates of the reference hole to photograph the target reference hole to obtain a fine image of the reference hole; Performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole; The circular hole feature fitting is performed on the binary image retaining the edge of the target circular hole to generate the precise coordinates of the reference hole.

2. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The coarse positioning of the target reference hole according to the reference hole coarse image to obtain the reference hole coarse coordinates includes: Performing mask extraction on the reference hole rough image to obtain a mask image; Constructing a pixel set containing reference hole semantics according to the mask image; The pixel set containing the reference hole semantics is used to solve the hole center coordinates of the reference hole, and the hole center coordinates are used as the reference hole rough coordinates.

3. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The step of moving the visual unit to the rough coordinates of the reference hole to photograph the target reference hole and obtain a fine image of the reference hole includes: The reference hole coarse coordinates and the pixel center coordinates of the visual unit are transformed into a principal axis coordinate system through the hand-eye relationship to obtain transformed reference hole coarse coordinates and transformed pixel center coordinates; The pixel center coordinates of the visual unit are moved to the reference hole coarse coordinates using the converted reference hole coarse coordinates and the converted pixel center coordinates, and the target reference hole is photographed to obtain the reference hole fine image.

4. The reference hole visual positioning method for portable hole making equipment according to claim 2, characterized in that: The performing local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole includes: Performing Gaussian blurring on the reference hole fine image to obtain a reference hole blurred image; performing local adaptive threshold binarization on the reference hole blurred image to extract a binarized image; Performing morphological erosion and morphological dilation on the mask image respectively to obtain a first mask sub-image smaller than the edge of the target reference hole and a second mask sub-image larger than the edge of the target reference hole; The binary image, the first mask sub-image, and the second mask sub-image are subjected to scratch interference masking processing to generate the binary image retaining the edge of the target circular hole.

5. The reference hole visual positioning method for portable hole making equipment according to claim 1, characterized in that: The performing circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate the actual coordinates of the reference hole includes: Actively drawing edges on the binary image retaining the edge of the target circular hole to obtain a binary image with edges; Performing arc segment fitting on the binary image with edges, and merging arcs with similar centers and radii to obtain an arc image with complete edge points; Gradient changes are used to screen a reliable circle with the lowest false alarm value in the arc image with complete edge points, and coordinate transformation is performed on the reliable circle to generate the precise coordinates of the reference hole.

6. A reference hole visual positioning device for portable hole making equipment, characterized in that: include: The calibration module is used to perform hand-eye calibration on the visual unit of the target portable hole-making device to obtain the hand-eye relationship, which specifically includes: Connecting a target spindle measuring tool to a device spindle unit of the target portable hole-making device, wherein the target spindle measuring tool is equipped with a plurality of first laser tracker measuring target balls; Placing a target calibration plate measuring fixture flat on a processing plane and performing plane positioning of the target calibration plate measuring fixture by a positioning block, wherein the target calibration plate measuring fixture is equipped with a checkerboard calibration plate and a plurality of second laser tracker measuring target balls; Connecting the spindle unit of the device to the vision unit in the horizontal and vertical directions, and ensuring that the target calibration plate measurement fixture appears within the field of view of the vision unit; Using a laser tracker to measure the positions of the first laser tracker measurement target sphere and the second laser tracker measurement target sphere, so as to fit a first pose of the target spindle measurement fixture relative to the laser tracker and a second pose of the target calibration plate measurement fixture relative to the laser tracker; Using the visual unit to photograph the checkerboard calibration plate to obtain a checkerboard calibration plate image; Performing sub-pixel corner extraction on the checkerboard calibration plate image to obtain a third pose of the checkerboard calibration plate relative to the visual unit; calculating a fourth pose of the vision unit relative to the device spindle unit according to the first pose, the second pose, and the third pose, so as to use the fourth pose as the hand-eye relationship; A first shooting module is used to shoot the target reference hole using the calibrated visual unit to obtain a rough image of the reference hole; A coarse positioning module, configured to coarsely position the target reference hole according to the coarse image of the reference hole to obtain coarse coordinates of the reference hole; A second shooting module is used to move the visual unit to the rough coordinates of the reference hole based on the hand-eye relationship to shoot the target reference hole and obtain a fine image of the reference hole; a processing module, configured to perform local adaptive threshold binarization and morphological processing on the reference hole fine image to generate a binarized image that retains the edge of the target circular hole; The precise positioning module is used to perform circular hole feature fitting on the binary image retaining the edge of the target circular hole to generate precise coordinates of the reference hole.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reference hole visual positioning method for a portable hole-making device according to any one of claims 1 to 5.

8. A computer program product, characterized in that When the computer program / instruction is executed by a processor, the reference hole visual positioning method for a portable hole-making device according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the reference hole visual positioning method for a portable hole-making device as described in any one of claims 1 to 5.

Images