Machine vision-based laser radar scanner initialization positioning method and system

Abstract

The application provides a laser radar scanner initialization positioning method and system based on machine vision, which mainly comprises the following steps: starting a laser radar scanner and a high-definition network camera; the high-definition network camera shoots an image of a product surface; a target rivet in the image is identified based on a machine vision algorithm; a laser point in the image is identified based on a machine vision algorithm; and it is judged whether the laser point and the target rivet overlap. The application can complete the laser point initialization positioning process in real time and efficiently without any manual intervention. In addition, the specifications and positions of the high-definition network camera and the laser radar scanner do not need to be limited, so the method is suitable for various application scenarios.

Description

Technical Field

[0001] This invention relates to the field of machine vision, and more particularly to a machine vision-based LiDAR scanner initialization and positioning method and a LiDAR scanner system implementing the method. Background Technology

[0002] In the assembly of large sections of industrial products, such as aircraft, lidar scanners are typically used to acquire point cloud data of these sections for quality analysis and to guide assembly through reverse engineering. Before scanning begins, the lidar scanner needs to be initialized and positioned at a scanning reference point. This reference point is located at the center of a rivet within a specific area enclosed by three arcs on the large section. The lidar scanner then scans the specific area along a planned path. Because the lidar scanner needs to be disassembled and moved after use, there is a certain degree of error during the initial positioning process when pointing the lidar scanner to the reference point according to a predetermined procedure.

[0003] Currently, the primary method for eliminating errors is using the camera integrated into the radar system. By displaying the camera's image on the control interface of the lidar scanner, operators can observe the positions of the rivet and the laser point, determine the positional difference between the target rivet and the laser point, and then manually control the lidar pan-tilt unit to achieve initial positioning. However, due to the limited zoom range, low resolution, and complex operating environment of the lidar system's built-in camera, operators need to repeatedly observe and adjust using artificial light, resulting in high labor and time costs.

[0004] The information included in the background section of this invention is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission of prior art known to those skilled in the art or any form of advice. Summary of the Invention

[0005] This invention addresses at least some of the aforementioned problems through a machine vision-based LiDAR scanner initialization and positioning method. This method uses an external high-definition, high-magnification, variable-focus network camera to capture images of the target section of a product and analyzes these images using machine vision algorithms. Specifically, the method first identifies four marked arc regions on the surface of the target section and locates the center of the target rivet within these regions. Then, it locates the center of a laser point on the surface of the target section. Furthermore, the machine vision-based LiDAR scanner initialization and positioning method calculates the pixel coordinate difference between the laser point center and the target rivet center in real time. Combined with pixel distance analysis, it controls the LiDAR scanner to move efficiently in different directions and step sizes until the laser point center and the target rivet center overlap, thus completing the initialization and positioning process.

[0006] According to some embodiments, a machine vision-based LiDAR scanner initialization positioning method includes: activating the LiDAR scanner and a high-definition network camera; capturing an image of the product surface using the high-definition network camera; identifying a target rivet in the image based on a machine vision algorithm; identifying a laser point in the image based on a machine vision algorithm; and determining whether the laser point and the target rivet overlap.

[0007] The method described in this invention is based on images from high-definition network cameras. It uses machine vision algorithms to analyze the images and controls the movement of a LiDAR scanner through control signals, thereby achieving fully autonomous initialization and positioning of the LiDAR scanner. This effectively improves the initialization and positioning efficiency and accuracy of the LiDAR scanner.

[0008] According to some embodiments, if the laser dot and the target rivet overlap, the process ends; if the laser dot and the target rivet do not overlap, the control device generates a control signal and transmits the control signal to the lidar scanner; and the lidar scanner receives the control signal, causing the laser dot to move toward the target rivet.

[0009] According to some embodiments, the machine vision algorithm includes the YOLOv5 algorithm, which includes an anchor layer and a head layer that can improve the recognition of small targets.

[0010] According to some embodiments, determining whether a laser point and a target rivet overlap includes: calculating the coordinate difference between the laser point and the target rivet, and if the coordinate difference is less than a predetermined threshold, then the laser point and the target rivet overlap.

[0011] According to some embodiments, the maximum resolution of a high-definition network camera is at least greater than 720P.

[0012] According to some embodiments, identifying a target rivet in an image based on a machine vision algorithm includes: splitting the image into multiple low-resolution images, and identifying the target rivet for each low-resolution image.

[0013] According to some embodiments, for each low-resolution image, four arc regions are first identified, and then the pixel coordinates of the center of the smallest bounding rectangle of the target rivet are identified and calculated within the four arc regions as the pixel coordinates of the target rivet.

[0014] According to some embodiments, identifying laser points in an image based on machine vision algorithms includes: selecting a low-resolution image that includes a target rivet, and stitching together multiple split low-resolution images around the low-resolution image that includes the target rivet, until the stitched low-resolution image includes both the laser point and the target rivet.

[0015] According to some embodiments, overlapping edges between multiple low-resolution images after stitching are removed using a non-maximum suppression method.

[0016] According to some embodiments, for the stitched low-resolution image, machine vision algorithms are used to identify and calculate the center pixel coordinates of the smallest bounding rectangle where the laser point is located as the pixel coordinates of the laser point.

[0017] According to some embodiments, the control signal indicates the direction of movement and the step distance of the LiDAR scanner.

[0018] According to some embodiments, the step distance decreases as the distance between the laser point and the target rivet decreases.

[0019] Another aspect of the present invention provides a machine vision-based lidar scanning system that implements the lidar scanner initialization and positioning method of the present invention, including a high-definition network camera, a lidar scanner, and a control device.

[0020] The machine vision-based LiDAR scanner initialization and positioning method and system of this invention can accurately identify the target to be tracked, especially small targets such as rivets and laser dots, across all segments of a product through machine vision algorithms. Furthermore, it can guide the LiDAR scanner to move in real-time and efficiently according to different areas, directions, and time lengths of the product to complete the laser dot initialization and positioning process without any human intervention. Therefore, the laser dot initialization and positioning process is time-efficient, highly accurate, and highly effective. In addition, there are no limitations on the specifications and locations of the high-definition network camera and LiDAR scanner; therefore, this invention is applicable to a variety of application scenarios. Attached Figure Description

[0021] The above-described objects, as well as other objects, features, and advantages of this disclosure, will be more fully understood by referring to the following illustrative and non-limiting detailed description of exemplary embodiments of this disclosure when taken in conjunction with the accompanying drawings.

[0022] Figure 1 A schematic diagram of a machine vision-based lidar system according to some embodiments is shown.

[0023] Figure 2 A flowchart is shown for an initialization localization method for a machine vision-based LiDAR system according to some embodiments.

[0024] Figure 3 An example of laser point initialization positioning according to some embodiments is shown.

[0025] Figure 4 Examples of identifying laser dots and target rivets according to some embodiments are shown. Detailed Implementation

[0026] This disclosure will become apparent from the detailed description given below. The detailed description and specific embodiments disclose preferred embodiments of this disclosure by way of example only. Those skilled in the art will understand, based on the guidance in the detailed description, that changes and modifications can be made within the scope of this disclosure.

[0027] Therefore, it should be understood that the content of this disclosure is not limited to the specific components of the described apparatus or the steps of the described method, as such apparatus and method can be modified. It should also be understood that the terminology used herein is for descriptive purposes only and is not intended to be limiting. It should be noted that when used in the specification and appended claims, the words “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more elements unless the context clearly indicates otherwise. Thus, for example, a reference to “a unit” or “the unit” can include several devices, etc. Furthermore, the words “comprising,” “including,” “containing,” and similar terms do not exclude other elements or steps.

[0028] This disclosure will now be described with reference to the accompanying drawings, in which preferred exemplary embodiments of the disclosure are illustrated. However, this disclosure may be implemented in other forms and should not be construed as limited to the embodiments disclosed herein. The disclosed embodiments are provided to fully convey the scope of this disclosure to those skilled in the art.

[0029] Figure 1 A schematic diagram of a machine vision-based LiDAR system according to some embodiments is shown. The machine vision-based LiDAR system includes a high-definition network camera, a LiDAR scanner, and a control device. A high-definition network camera refers to a network camera capable of continuously acquiring images at a maximum resolution greater than 720P (1280×720), or even reaching or exceeding 1080P (1920×1080 resolution), at a rate greater than 12 FPS. The 2K high-definition network camera (11) used in this invention, such as... Figure 1 As shown, this is to capture images of the target section in real time. However, the invention is not limited to this; other types of high-definition network cameras can also be used.

[0030] A lidar scanner consists of a laser emitter, receiver, time counter, motor-controlled rotatable filter, control circuit board, and software. It provides three-dimensional point cloud data of the scanned product surface, thus enabling the acquisition of high-precision, high-resolution digital models of products. The lidar scanner (12) used in this invention is as follows: Figure 1 As shown, it has an industrial-grade measurement accuracy of 0.01 mm / m, such as the Nikon MV331. However, it is understood that other models of LiDAR scanners can also be used in this invention.

[0031] The control device can be any electronic device capable of controlling the LiDAR scanner, including memory, processor, display, etc. According to some embodiments, the control device is an electronic device independent of and communicatively connected to both the HD network camera and the LiDAR scanner. Preferably, the control device can be integrated with or part of the LiDAR scanner.

[0032] Figure 2 A flowchart is shown for an initialization localization method for a machine vision-based LiDAR system according to some embodiments.

[0033] The method includes step 21: activating the lidar system. Activating the lidar system includes activating the lidar scanner to illuminate the target rivet area on the surface of the product section with a laser point, and activating the high-definition network camera so that the high-definition network camera lens is pointed at the surface of the product section.

[0034] According to some embodiments, the LiDAR scanner is activated, and the laser point is aligned with the initialization point according to a program programmed for different parts of the product. Typically, because the LiDAR scanner is moved after each use, there is a distance between the laser point and the initialization point. After the LiDAR scanner is activated, the control device starts the high-definition network camera.

[0035] Step 22 follows step 21: A high-definition network camera captures an image of a section of the product surface. The high-definition network camera captures an image of a section of the product surface, with the image resolution being at least 720P. The image shows the section of the product surface, the target rivet area on the surface, and the laser dots near the target rivet area.

[0036] According to some embodiments, the high-definition network camera captures a high-definition image of the product surface in real time after startup. The image has a pixel resolution of at least 720P, preferably greater than 1080P, and the image shows the surface of a certain section of the product, the target rivet area on the surface, and the laser points near it.

[0037] Step 23 follows step 22: Identify the target rivet. The image captured by the high-definition network camera is transmitted to the control device, which uses machine vision algorithms to analyze the laser point and the position of the target rivet.

[0038] According to some implementation examples, taking the YOLOv5 machine vision deep learning algorithm in PyTorch as an example, a large number of actual on-site initialization positioning images are captured for different parts of the product under different shooting angles and ambient lighting conditions, such as... Figure 3As shown, the weights of the training model are adjusted to analyze the positions of the laser point and the target rivet in the image. During training, a large number of images need to be captured around the four arcs where the laser point is located, around the rivet, and overlapping with the rivet, to serve as the training dataset. Since targets such as the laser point and the target rivet occupy relatively few pixels in the image, the targets to be tracked are defined as small targets, and the existing YOLOv5 machine vision algorithm model is improved to enhance the recognition ability for small targets. For example, the input anchors layer can be modified as follows:

[0039] --[5,6,8,14,15,11]

[0040] --[10,13,16,30,33,23]

[0041] --[30,61,62,45,59,119]

[0042] --[116,90,156,198,373,326]

[0043] Alternatively, the layer depth at the head output can be increased:

[0044] --[-1,1,Conv,[512,3,2]]

[0045] --[[-1,10],1,Concat,[1]]

[0046] --[-1,3,C3,[1024,False]]

[0047] The improved YOLOv5 machine vision algorithm model, by increasing the number of iterations, can quickly and accurately locate small targets such as laser dots and target rivets, significantly improving the accuracy of small target recognition.

[0048] According to some embodiments, the control device splits the high-resolution image captured by the high-definition network camera into images at intervals of 100-200 pixels. For example, a 2096×1080 resolution image is split into multiple images of 640×480 resolution. Then, the control device searches for the target rivet in each of the split images. The target rivet is a central rivet surrounded by four arc segments, such as... Figure 4 As shown.

[0049] Because other rivets may exist around the target rivet, interfering with the control device's recognition capability, the control device uses machine vision algorithms to identify and lock onto the four arc-shaped regions surrounding the target rivet, such as... Figure 4The circumscribed rectangle shown represents the four arc-shaped regions. After identifying these regions, the control device uses machine vision algorithms to identify the target rivet and calculates its pixel coordinates, which are the pixel coordinates of the center of the smallest circumscribed rectangle. When multiple rivets are identified, the control device determines the target rivet by judging whether it falls within the four arc-shaped regions.

[0050] Step 24 follows step 23: identify laser points, including the spliced ​​and split images, remove overlapping image edges, output an image including laser points and target rivets, and identify the laser points in the image.

[0051] According to some embodiments, the split images are stitched together based on an image including the target rivet, for example, a split image with a resolution of 640×480. Then, overlapping image edges are removed using a non-maximum suppression method until the stitched image includes the laser point and the target rivet. Preferably, the stitched image has a resolution of approximately 2000 pixels, more preferably, a resolution greater than 2000 pixels. The stitched image is as follows: Figure 4 As shown. For the stitched image, the control device uses machine vision algorithms to identify the laser points and calculate the pixel coordinates of the laser points, that is, the pixel coordinates of the center of the smallest bounding rectangle.

[0052] After step 24, proceed to step 25: calculate the coordinate difference between the laser point and the target rivet, and determine whether the laser point and the target rivet overlap.

[0053] According to some embodiments, the coordinate difference between the laser point and the target rivet is calculated based on the pixel coordinates of the laser point and the target rivet calculated in steps 23-24.

[0054] According to some embodiments, a threshold is preset, for example, 4 pixels. When the coordinate difference between the laser point and the target rivet is greater than 4, it can be determined that the laser point and the target rivet do not overlap; when the coordinate difference between the laser point and the target rivet is less than or equal to 4, it can be determined that the laser point and the target rivet overlap. It is understood that this threshold can be other values, such as 3 or 5.

[0055] If the laser point and the target rivet are determined to overlap, the initialization positioning method ends. If the laser point and the target rivet are determined not to overlap, the initialization positioning method proceeds to step 26.

[0056] Step 26 includes the control device generating a control signal and transmitting the control signal to the lidar scanner.

[0057] According to some embodiments, the control device generates a control signal based on the coordinate difference between the laser point and the target rivet. This control signal indicates the moving direction and step size of the LiDAR scanner filter. For example, when the coordinate difference between the laser point and the target rivet is positive, the control signal indicates that the LiDAR scanner filter moves clockwise; when the coordinate difference between the laser point and the target rivet is negative, the control signal indicates that the LiDAR scanner filter moves counterclockwise. For example, when the coordinate difference between the laser point and the target rivet is greater than or equal to 6 times the side length of the minimum bounding rectangle of the target rivet, the rotation angle of the LiDAR scanner filter is 1°; when the coordinate difference between the laser point and the target rivet is greater than or equal to 3 times the side length of the minimum bounding rectangle but less than 6 times the side length of the minimum bounding rectangle, the rotation angle of the LiDAR scanner filter is 0.3°; when the coordinate difference between the laser point and the target rivet is greater than 0.5 times the side length of the minimum bounding rectangle of the target rivet but less than 3 times the side length of the minimum bounding rectangle, the rotation angle of the LiDAR scanner filter is 0.1°; and when the coordinate difference between the laser point and the target rivet is less than 0.5 times the side length of the minimum bounding rectangle of the target rivet, the rotation angle of the LiDAR scanner filter is 0.05°. However, the present invention is not limited to these, and the rotation direction and angle of the LiDAR scanner filter can vary based on the coordinate difference between the laser point and the target rivet and its ratio to the side length of the minimum bounding rectangle of the target rivet.

[0058] Step 27 follows step 26: The lidar scanner receives a control signal and moves the laser point toward the target rivet.

[0059] According to some embodiments, the lidar scanner receives control signals transmitted by the control device and controls the filter to rotate according to the calculated rotation direction and rotation step, so that the laser point approaches the target rivet.

[0060] After step 27, the initialization positioning method proceeds to step 22 to recalculate the pixel coordinates of the target rivet and the laser point, and to determine whether they overlap.

[0061] Existing LiDAR scanner initialization and positioning methods involve using a built-in camera to capture images of a laser point and a target rivet. The operator then visually observes a crosshair indicating the laser point's position and the target rivet on a display screen, determining if the crosshair overlaps with the rivet. Based on this assessment, the operator manually controls the LiDAR scanner to adjust the laser point's position to complete the initial positioning. However, existing LiDAR scanners have low camera zoom capabilities, and the positioning operation is complex, requiring multiple observations and adjustments by the operator. Therefore, initialization and positioning are time-consuming, inefficient, and costly.

[0062] The lidar scanning system of this invention utilizes a high-definition network camera to capture images of the laser point and the target rivet. It then analyzes the captured images using machine vision algorithms to generate control signals to control the lidar scanner and calibrate the laser point position. This process effectively improves the initial positioning accuracy of the laser point, reduces the workload of operators, and increases the efficiency of laser point initial positioning.

[0063] Those skilled in the art will recognize that this disclosure is not limited to the preferred embodiments described above. They will also recognize that modifications and variations are possible within the scope of the appended claims. Furthermore, through a study of the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments.

Claims

1. A machine vision-based LiDAR scanner initialization and localization method, comprising: Activate the LiDAR scanner and the high-definition network camera, wherein the high-definition network camera has a maximum resolution of at least 720P; The high-definition network camera captures high-resolution images of the product surface; The high-resolution image is split into multiple low-resolution images; For each of the low-resolution images, four arc regions in the image are identified based on the improved YOLOv5 machine vision algorithm, and a target rivet is identified within the four arc regions. The target rivet is the central rivet surrounded by the four arc regions. When multiple rivets are identified, the target rivet is identified by determining whether the rivet is within the four arc regions. Select a low-resolution image including the target rivet, and stitch together multiple low-resolution images that were split around the low-resolution image including the target rivet until the stitched low-resolution image includes the laser point and the target rivet, and delete the overlapping edges between the stitched multiple low-resolution images using a non-maximum suppression method; The machine vision algorithm is used to identify laser points in the stitched low-resolution image; and to determine whether the laser points and the target rivet overlap. If so, then the process ends; If not, the control device generates a control signal based on the coordinate difference between the laser point and the target rivet. The control signal indicates the moving direction and moving step of the lidar scanner. The moving step decreases as the distance between the laser point and the target rivet decreases. The lidar scanner receives the control signal, causing the laser point to move toward the target rivet.

2. The method according to claim 1, wherein, The machine vision algorithm includes the YOLOv5 algorithm, which includes an anchor layer and a head layer that can improve the recognition ability of small targets.

3. The method according to claim 1, wherein, Determining whether the laser point and the target rivet overlap includes: calculating the coordinate difference between the laser point and the target rivet, and if the coordinate difference is less than a predetermined threshold, then the laser point and the target rivet overlap.

4. The method according to claim 1, further comprising: For each of the low-resolution images, four arc regions are first identified, and then the pixel coordinates of the center of the smallest bounding rectangle of the target rivet are identified and calculated within the four arc regions as the pixel coordinates of the target rivet.

5. The method according to claim 4, wherein, Identifying laser points in the image based on machine vision algorithms includes: Select a low-resolution image that includes the target rivet, and stitch together multiple split low-resolution images around the low-resolution image that includes the target rivet, until the stitched low-resolution image includes the laser point and the target rivet.

6. The method of claim 5, further comprising: Overlapping edges between the stitched low-resolution images are removed using a non-maximum suppression method.

7. The method of claim 6, further comprising: For the stitched low-resolution image, a machine vision algorithm is used to identify and calculate the center pixel coordinates of the smallest bounding rectangle where the laser point is located, which are then used as the pixel coordinates of the laser point.

8. The method of claim 1, further comprising: The control signal indicates the direction of movement and the step distance of the lidar scanner.

9. The method according to claim 8, wherein, The movement step distance decreases as the distance between the laser point and the target rivet decreases.

10. A machine vision-based lidar scanning system, implemented according to claim 1, comprising a high-definition network camera, a lidar scanner, and a control device.

Images