Lane inspection method and inspection equipment using multi-camera fusion single-beam laser radar

Through multi-camera fusion single-wire harness lidar technology, the problems of insufficient scenario adaptability and high cost in lane inspection are solved, and accurate window positioning and detection of different models and parking positions are achieved, vehicle inspection efficiency is improved and equipment is reduced cost.

CN118731966BActive Publication Date: 2024-11-29SHENZHEN MAXVISION TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411207983.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-29
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The existing technology has problems of insufficient scenario adaptability and high cost in lane inspection. Especially when dealing with differences in different models and parking positions, the depth camera has a small viewing angle range and insufficient maximum detection distance, resulting in inaccurate positioning and expensive cost. .

Method used

Using the multi-camera fusion single-wire beam lidar method, the camera parameter transformation matrix and spatial conversion relationship are obtained through calibration, and the vehicle side point cloud detected by the single-wire beam lidar is fitted into a spatial straight line, and the physics of the window target is calculated by combining the multi-positioning camera. Coordinates and the slide mechanism drives the capture camera to accurately capture and detect them.

Benefits of technology

It improves the window detection range and accuracy, enhances the adaptability of lane scenes, reduces equipment costs, can effectively handle differences in different models and parking positions, and improves vehicle inspection efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118731966B_ABST
    Figure CN118731966B_ABST
Patent Text Reader

Abstract

The present application provides a lane inspection method and inspection equipment for multi-camera fusion single-beam laser radar, the method comprising the steps of: transforming the single-beam laser radar, each positioning camera and the spatial straight line into a reference space coordinate system according to the spatial transformation relationship, and calculating the body distance from each positioning camera to the side of the vehicle respectively; each positioning camera obtains the side image of the vehicle, and obtains the window target through target detection; according to the parameter transformation matrix and the body distance, the pixel coordinates of the window target are converted into the window space physical coordinates based on the camera coordinate system; the window space physical coordinates obtained by all positioning cameras are mapped to the reference space coordinate system, and the capture camera is controlled to move to the window space physical coordinates for capture detection. The method of the present application can improve the window detection range and accuracy, improve the adaptability of lane scenes, and reduce equipment costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of lane inspection technology, and more specifically, to a lane inspection method and inspection equipment using a multi-camera fused single-beam laser radar. Background Art

[0002] In the vehicle transport security inspection channel, in order to prevent people from using vehicles to evade security inspection and illegal transportation, it is necessary to conduct simultaneous security inspections on people and vehicles passing through customs. According to relevant regulations, when a vehicle enters the inspection lane, only the driver is allowed to drive in and open all the windows in advance. After arriving at the inspection parking area, the window position is located by the depth camera, and then the robotic arm is moved to the window position to capture the interior image of the vehicle.

[0003] However, in actual application, it was found that this method is only suitable for cars. For cars, trucks, RVs, trailers and other vehicles passing through the lane, they are affected by the differences in window position, height and number of different models, as well as the differences in parking position and body inclination. When using the depth camera to locate the window position, the depth camera has its own parameter limitations in three aspects: small viewing angle range, insufficient maximum detection distance and minimum detection distance limit. As a result, the depth camera cannot detect all windows or the positioning is inaccurate. Therefore, the depth camera has the disadvantage of insufficient scene adaptability.

[0004] In addition, if multiple depth cameras are used to improve adaptability, the price of depth cameras is dozens of times that of conventional cameras, which has the disadvantage of high cost. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a lane inspection method and inspection equipment using a multi-camera fusion single-beam laser radar to solve the technical problems of insufficient scene adaptability and high cost in the prior art during vehicle window positioning.

[0006] To achieve the above purpose, the technical solution adopted in this application is: to provide a lane inspection method of multi-camera fusion single-beam laser radar, and the lane inspection method of multi-camera fusion single-beam laser radar comprises the following steps:

[0007] Through calibration, the parameter transformation matrix of each positioning camera based on the camera coordinate system and the spatial transformation relationship between each positioning camera and the single-beam laser radar based on the reference space coordinate system are obtained;

[0008] Fit the side point cloud of the vehicle detected by the single-beam laser radar into a spatial straight line;

[0009] According to the spatial transformation relationship, the single-beam laser radar, each positioning camera and the spatial straight line are transformed into the reference spatial coordinate system, and the distance from each positioning camera to the vehicle side is calculated respectively;

[0010] Each positioning camera obtains the side image of the vehicle and obtains the window target through target detection;

[0011] According to the parameter transformation matrix and the distance between the vehicle body and the vehicle window, the pixel coordinates of the vehicle window target are converted into the physical coordinates of the vehicle window space based on the camera coordinate system;

[0012] The physical coordinates of the vehicle window space obtained by all positioning cameras are mapped to the reference space coordinate system, and the capture camera is controlled to move to the physical coordinates of the vehicle window space for capture detection.

[0013] In another embodiment of the present application, a method for obtaining a parameter transformation matrix of each positioning camera based on a camera coordinate system by calibration includes the steps of:

[0014] Set any two positioning cameras to be calibrated at positions with a common field of view;

[0015] Place the checkerboard calibration plate at different positions and angles in the common field of view;

[0016] Use two positioning cameras to collect images and obtain calibration images respectively;

[0017] Corner points are extracted from the calibration images respectively, and then the two calibration images after corner point extraction are stereo matched to obtain the intrinsic parameter transformation matrix and distortion coefficient of the two positioning cameras and the extrinsic parameter transformation matrix between the two positioning cameras.

[0018] In another embodiment of the present application, a method for obtaining a spatial transformation relationship between each positioning camera and a single-beam laser radar based on a reference space coordinate system through calibration includes the steps of:

[0019] The single-beam laser radar and any one of the positioning cameras are arranged at the same position, and the midline of the field of view of the single-beam laser radar coincides with the optical axis of the positioning camera;

[0020] A reference space coordinate system is established with the positioning camera as the origin.

[0021] In another embodiment of the present application, a method for converting the pixel coordinates of a vehicle window target into the physical coordinates of the vehicle window space based on a camera coordinate system according to a parameter transformation matrix and a vehicle body distance comprises the steps of:

[0022] Get the center point P of the detection box of the window target in the pixel coordinates of the side image. x , pixel y ) relative to the vertical pixel difference δ of the center point of the side image h and the horizontal pixel difference δ v , the calculation formula is:

[0023]

[0024] Where, h is the pixel height of the side image, and w is the pixel width of the side image;

[0025] Based on the parameter transformation matrix, the vertical pixel difference δ h and the horizontal pixel difference δ v The corresponding transformation is the vertical angle Degree between the center point P of the detection frame and the center point C of the side image v And the horizontal angle Degree h , the calculation formula is:

[0026]

[0027] Among them, fov v To locate the vertical viewing angle of the camera, fov h To locate the horizontal viewing angle of the camera;

[0028] Through the vertical angle Degree v And the horizontal angle Degree h Converted into physical coordinates of the window space based on the camera coordinate system.

[0029] In another embodiment of the present application, by the vertical angle Degree v And the horizontal angle Degree h The method of converting into the physical coordinates of the vehicle window space based on the camera coordinate system includes the following steps:

[0030] The vertical angle Degree v And the horizontal angle Degree h Combined with the vehicle body distance D i Calculate the vertical physical distance D between the center point P of the detection frame and the center point C of the side image v and the horizontal physical distance D h , i represents the order of positioning cameras, and the calculation formula is:

[0031]

[0032] Based on trigonometric functions, the straight-line physical distance D between the center point P of the detection frame and the center point C of the side image is calculated. win , the calculation formula is:

[0033]

[0034] According to the straight line physical distance D win Get the physical coordinates of the vehicle window space relative to the positioning camera.

[0035] In another embodiment of the present application, according to the straight line physical distance D win The method for obtaining the physical coordinates of the vehicle window space of the vehicle window target relative to the positioning camera comprises the steps of:

[0036] Assume that the physical coordinates of the window space in the camera coordinate system (X, Y, Z) and the optical center coordinates of the positioning camera are (ppx, ppy), f x and f y Respectively represent the focal length of the positioning camera in the horizontal direction and the focal length in the vertical direction, and the calculation formula is:

[0037] .

[0038] In another embodiment of the present application, the positioning camera is a wide-angle camera.

[0039] In another embodiment of the present application, a method for fitting a point cloud of a vehicle side surface detected by a single-beam laser radar into a spatial straight line comprises the steps of:

[0040] Clustering is performed on the side point cloud of the vehicle to obtain several point cloud clusters;

[0041] Perform straight line fitting on each point cloud cluster to obtain several fitting line segments;

[0042] Calculate the distance between any two fitted line segments, and group the fitted line segments whose distance is less than the distance threshold into line segment clusters;

[0043] The largest line segment cluster is extracted, and all the fitting line segments in the largest line segment cluster are fitted into spatial straight lines.

[0044] In another embodiment of the present application, after mapping the physical coordinates of the vehicle window space acquired by all positioning cameras to the reference space coordinate system, the steps include:

[0045] Assume that the number of fitted segments in the largest segment cluster is N;

[0046] Clustering the spatial coordinates of the vehicle window targets based on the spatial distance in the reference spatial coordinate system to obtain several vehicle window target clusters;

[0047] Extract the window target clusters ranked as top N-1 in size;

[0048] Select one window target with the highest prediction probability from each of the extracted window target clusters to obtain N-1 window targets, and filter all remaining window targets.

[0049] The present application also provides an inspection device for executing the lane inspection method of multi-camera fusion single-beam laser radar as described above, the inspection device includes a lane and a gantry arranged above the lane, a number of positioning cameras and at least one single-beam laser radar are respectively arranged on both sides of the lane, a sliding mechanism and a number of snapshot cameras arranged on the sliding mechanism are arranged on the gantry, the windows on both sides of the vehicle are positioned by the positioning camera fusion single-beam laser radar, and the snapshot camera is driven by the sliding mechanism to the corresponding window position for snapshot detection.

[0050] Compared with the prior art, the lane inspection method of multi-camera fusion single-beam laser radar provided in the present application fits the vehicle side point cloud detected by the single-beam laser radar into a spatial straight line, and obtains the body distance from each positioning camera to the vehicle side in the reference space coordinate system according to the spatial transformation relationship, and then converts the pixel coordinates of the window target into physical coordinates of the window space based on the camera coordinate system according to the parameter transformation matrix and the body distance, and integrates the multi-positioning camera and the single-beam laser radar to improve the window detection range and accuracy, improve the adaptability of lane scenes, and reduce equipment costs.

[0051] Compared with the prior art, the inspection equipment provided by the present application uses the positioning camera to fuse the single-beam laser radar to locate the windows on both sides of the vehicle, and the sliding mechanism drives the capture camera to the corresponding window position for capture detection, thereby improving the vehicle inspection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A schematic diagram of the three-dimensional structure of a lane inspection method using a multi-camera fusion single-beam laser radar provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of the detection range of a positioning camera and a single-beam laser radar provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of fitting a point cloud of a vehicle side surface into a spatial straight line provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of converting the pixel coordinates of a vehicle window target into the physical coordinates of the vehicle window space provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram comparing the effects of window deduplication before and after provided in an embodiment of the present application;

[0058] Figure 6 A schematic diagram of the inspection device provided in an embodiment of the present application under the main viewing angle;

[0059] Figure 7 A schematic diagram of the inspection device provided in an embodiment of the present application from a top view. DETAILED DESCRIPTION

[0060] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0062] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0063] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] Please also read Figures 1 to 4 , the lane inspection method of multi-camera fusion single-beam laser radar provided in the embodiment of the present application is now described. The lane inspection method of multi-camera fusion single-beam laser radar is applied to a lane, on which a fixed single-beam laser radar, multiple positioning cameras and multiple movable snapshot cameras are installed. When a vehicle enters the inspection area in the lane, the vehicle can be inspected.

[0065] The lane inspection method of multi-camera fusion single-beam laser radar includes the following steps:

[0066] Step S1, obtaining, by calibration, a parameter transformation matrix of each positioning camera based on a camera coordinate system and a spatial transformation relationship between each positioning camera and a single-beam laser radar based on a reference space coordinate system;

[0067] Step S2, fitting the side point cloud of the vehicle detected by the single-beam laser radar into a spatial straight line;

[0068] Step S3, transforming the single-beam laser radar, each positioning camera and the spatial straight line into a reference space coordinate system according to the spatial transformation relationship, and calculating the distance from each positioning camera to the vehicle side;

[0069] Step S4, each positioning camera acquires a side image of the vehicle and acquires a vehicle window target through target detection;

[0070] Step S5, converting the pixel coordinates of the vehicle window target into the physical coordinates of the vehicle window space based on the camera coordinate system according to the parameter transformation matrix and the vehicle body distance;

[0071] Step S6, mapping the physical coordinates of the vehicle window space acquired by all positioning cameras to the reference space coordinate system, and controlling the capture camera to move to the physical coordinates of the vehicle window space for capture detection.

[0072] It can be understood that in step S1, after the single-beam laser radar, multiple positioning cameras and multiple snapshot cameras are installed, the parameter transformation matrix is ​​calibrated for the positioning camera, and the parameter transformation matrix feeds back the relative position conversion relationship between the camera coordinate system and the pixel coordinate system and between different coordinate systems. The camera coordinate system should be understood as a coordinate system established at the position of the corresponding positioning camera. Therefore, the camera coordinate system under each positioning camera is independent. Similarly, the single-beam laser radar also has a radar coordinate system established based on its own position. This embodiment also obtains the spatial conversion relationship between each positioning camera and the single-beam laser radar based on the reference space coordinate system through calibration, that is, obtains the mutual conversion relationship between multiple camera coordinate systems and radar coordinate systems.

[0073] In step S2, unlike the depth camera, the single-beam laser mine can only obtain point cloud data of a plane at a certain height, but due to the wide field of view of the single-beam laser mine, a maximum of 360-degree detection can be achieved. For example, in this application, the detection range is set to 220 degrees to ensure that the detection range of any position of the vehicle in the lane can cover the entire side of the vehicle. In this embodiment, the point cloud data of the entire side of the vehicle is fitted into a spatial straight line, and the position of the spatial straight line represents the position of the side of the vehicle.

[0074] In step S3, based on the spatial transformation relationship calibrated in step S1 and the results of step S2 are all transformed into the reference space coordinate system, the body distance from each positioning camera to the side of the vehicle can be calculated in the reference space coordinate system. It must be noted that since the body direction and parking position are random, the body distance calculated based on each positioning camera may be different.

[0075] In step S4, each positioning camera obtains a side image of the vehicle, which includes windows and doors. The window target can be obtained through a deep learning target detection algorithm. When there are multiple windows in the side image, multiple window targets can also be obtained simultaneously through the target detection algorithm, and the pixel coordinates of the window target in the side image are also random.

[0076] In step S5, the parameter transformation matrix calibrated in step S1 is fused with the vehicle body distance to give actual spatial physical meaning to each pixel coordinate in the side image, and the pixel coordinates of the vehicle window target are converted into physical coordinates of the vehicle window space based on the camera coordinate system.

[0077] In step S6, based on the spatial conversion relationship obtained by calibration in step S1, the physical coordinates of the vehicle window space acquired by all positioning cameras are mapped to the reference space coordinate system, and finally a reference space coordinate system integrating the physical coordinates of all vehicle window spaces is obtained, thereby improving the accuracy of vehicle window positioning. Therefore, controlling the capture camera to move to the physical coordinates of the vehicle window space for capture detection in the reference space coordinate system can improve the detection effect.

[0078] In this way, when encountering any type of vehicle (including sedans, trucks, RVs, trailers, etc.) and any parking position in the lane, the physical coordinates of all vehicle windows can be accurately obtained by integrating the multi-positioning camera with the single-beam laser radar. In addition, the cost of single-beam laser radar over multi-beam laser radar and ordinary RGB positioning camera over depth camera is absolutely advantageous.

[0079] Compared with the prior art, the lane inspection method of multi-camera fusion single-beam laser radar provided in the present application fits the vehicle side point cloud detected by the single-beam laser radar into a spatial straight line, and obtains the body distance from each positioning camera to the vehicle side in the reference space coordinate system according to the spatial transformation relationship, and then converts the pixel coordinates of the window target into physical coordinates of the window space based on the camera coordinate system according to the parameter transformation matrix and the body distance, and integrates the multi-positioning camera and the single-beam laser radar to improve the window detection range and accuracy, improve the adaptability of lane scenes, and reduce equipment costs.

[0080] In another embodiment of the present application, in step S1, a method for obtaining a parameter transformation matrix of each positioning camera based on a camera coordinate system by calibration includes the steps of:

[0081] Set any two positioning cameras to be calibrated at positions with a common field of view;

[0082] Place the checkerboard calibration plate at different positions and angles in the common field of view;

[0083] Use two positioning cameras to collect images and obtain calibration images respectively;

[0084] Corner points are extracted from the calibration images respectively, and then the two calibration images after corner point extraction are stereo matched to obtain the intrinsic parameter transformation matrix and distortion coefficient of the two positioning cameras and the extrinsic parameter transformation matrix between the two positioning cameras.

[0085] It can be understood that the positioning camera distortion is mainly due to the positioning camera being affected by the manufacturing and installation accuracy of the lens, resulting in nonlinear distortion of the acquired image. This distortion phenomenon is collectively referred to as lens distortion. Common distortions are mainly divided into radial distortion and tangential distortion. By placing the checkerboard calibration plate at different positions and angles in the common field of view, the radial distortion and tangential distortion can be corrected after corner point extraction and the intrinsic reference transformation matrix can be obtained. On this basis, the two calibration images after corner point extraction are stereo matched to obtain the extrinsic reference transformation matrix between the two positioning cameras. In this way, through the embodiments of the present application, the intrinsic reference transformation matrix and distortion coefficients of the two positioning cameras and the extrinsic reference transformation matrix between the two positioning cameras are obtained through a one-time calibration, thereby improving the window positioning accuracy.

[0086] For further information, please also refer to Figure 2 In step S1, a method for obtaining a spatial transformation relationship between each positioning camera and a single-beam laser radar based on a reference space coordinate system by calibration includes the following steps:

[0087] The single-beam laser radar and any one of the positioning cameras are arranged at the same position, and the midline of the field of view of the single-beam laser radar coincides with the optical axis of the positioning camera;

[0088] A reference space coordinate system is established with the positioning camera as the origin.

[0089] It is understandable that the positioning camera can be directly set on the single-beam laser radar, and the center line of the field of view of the single-beam laser radar and the optical axis of the positioning camera are both facing the lane direction, so that the spatial conversion between the positioning camera and the single-beam laser radar is achieved with zero error. Combined with the external parameter transformation matrix between any two positioning cameras obtained in the previous embodiment, the spatial conversion relationship between each positioning camera and the single-beam laser radar based on the reference space coordinate system is obtained.

[0090] In another embodiment of the present application, in step S5, a method for converting the pixel coordinates of a vehicle window target into physical coordinates of a vehicle window space based on a camera coordinate system according to a parameter transformation matrix and a vehicle body distance comprises the following steps:

[0091] Get the center point P of the detection box of the window target in the pixel coordinates of the side image. x , pixel y ) relative to the vertical pixel difference δ of the center point of the side image h and the horizontal pixel difference δ v , the calculation formula is:

[0092]

[0093] Where, h is the pixel height of the side image, and w is the pixel width of the side image;

[0094] Based on the parameter transformation matrix, the vertical pixel difference δ h and the horizontal pixel difference δ v The corresponding transformation is the vertical angle Degree between the center point P of the detection frame and the center point C of the side image v And the horizontal angle Degree h , the calculation formula is:

[0095]

[0096] Among them, fov v To locate the vertical viewing angle of the camera, fov h To locate the horizontal viewing angle of the camera;

[0097] Through the vertical angle Degree v And the horizontal angle Degree h Converted into physical coordinates of the window space based on the camera coordinate system.

[0098] It can be understood that in this embodiment, by calculating the vertical pixel difference δ h and the horizontal pixel difference δ v Realize dimensionality reduction to adapt the vertical viewing angle fov of the positioning camera v With horizontal viewing angle fovh . The vertical viewing angle fov of the positioning camera v With horizontal viewing angle fov h These are fixed parameters and can be obtained by combining the hardware parameters provided by the manufacturer.

[0099] For further information, please also refer to Figure 4 In step S5, the vertical angle Degree v And the horizontal angle Degree h The method of converting into the physical coordinates of the vehicle window space based on the camera coordinate system includes the following steps:

[0100] The vertical angle Degree v And the horizontal angle Degree h Combined with the vehicle body distance D i Calculate the vertical physical distance D between the center point P of the detection frame and the center point C of the side image v and the horizontal physical distance D h , i represents the order of positioning cameras, and the calculation formula is:

[0101]

[0102] Based on trigonometric functions, the straight-line physical distance D between the center point P of the detection frame and the center point C of the side image is calculated. win , the calculation formula is:

[0103]

[0104] According to the straight line physical distance D win Get the physical coordinates of the vehicle window space relative to the positioning camera.

[0105] It can be understood that in this embodiment, the distance D between the positioning camera and the vehicle side is i Blend to vertical angle Degree v And the horizontal angle Degree h , the conversion from angle to physical distance is realized by increasing the dimension, and then the straight line physical distance D is calculated by trigonometric function win , and ultimately achieve the purpose of giving the positioning camera a depth value through a single-beam laser radar.

[0106] Further, in step S5, according to the straight line physical distance D win The method for obtaining the physical coordinates of the vehicle window space of the vehicle window target relative to the positioning camera comprises the steps of:

[0107] Assume that the physical coordinates of the window space in the camera coordinate system (X, Y, Z) and the optical center coordinates of the positioning camera are (ppx, ppy), f x and fy Respectively represent the focal length of the positioning camera in the horizontal direction and the focal length in the vertical direction, and the calculation formula is:

[0108] .

[0109] It can be understood that in this embodiment, the straight line physical distance D is directly win As the depth value of the window target. In this way, no matter where the window target is located in the side image acquired by the positioning camera, its depth value can be accurately obtained by the above method, and the physical coordinates (X, Y, Z) of the window space in the camera coordinate system can be calculated.

[0110] Further, in step S5, the positioning camera is a wide-angle camera. It is understandable that the wide-angle characteristic can be used to further expand the field of view. Generally, the viewing angle of a wide-angle camera is 120 to 180 degrees. When the viewing angle of a wide-angle camera is 120 degrees, after being fused with a single-beam laser radar through the method of this embodiment, the effective space size of the window that can be detected is equivalent to the effective space of 6 depth cameras with a conventional viewing angle of 60 degrees, because the depth camera is not only limited by the viewing angle, but also by the maximum detection distance and the minimum detection distance, while the wide-angle camera not only has a larger viewing angle than the depth camera, but also has no minimum detection distance limit, and its maximum detection distance is also greater than the maximum detection distance of the depth camera.

[0111] In another embodiment of the present application, please refer to Figure 3 In step S2, the method for fitting the side point cloud of the vehicle detected by the single-beam laser radar into a spatial straight line comprises the steps of:

[0112] Clustering is performed on the side point cloud of the vehicle to obtain several point cloud clusters;

[0113] Perform straight line fitting on each point cloud cluster to obtain several fitting line segments;

[0114] Calculate the distance between any two fitted line segments, and group the fitted line segments whose distance is less than the distance threshold into line segment clusters;

[0115] The largest line segment cluster is extracted, and all the fitting line segments in the largest line segment cluster are fitted into spatial straight lines.

[0116] It is understandable that after the vehicle is parked, its body is tilted, and according to relevant regulations, the window needs to be opened, resulting in the side point cloud of the vehicle not being a straight line, but having a large amount of interfering point cloud data. In this embodiment, the irregular point cloud data is first clustered to form point cloud clusters, and the point cloud clusters can be fitted by the existing least squares method.

[0117] It is conceivable that when all windows are open, the side of the window that can be detected by the laser is similar to a dotted line, and each line segment that constitutes the dotted line is on a straight line with a distance of zero. Thus, in this embodiment, the fitted line segments whose distance is less than the distance threshold are grouped into line segment clusters, and the largest line segment cluster represents the side of the window when all windows are open. The largest line segment cluster can be understood as containing the largest number of fitted line segments or having the largest sum of lengths. The method of fitting all the fitted line segments in the largest line segment cluster into a spatial straight line can adopt the existing linear regression method.

[0118] For further information, please also refer to Figure 5 In step S6, after mapping the physical coordinates of the vehicle window space acquired by all positioning cameras to the reference space coordinate system, the following steps are included:

[0119] Assume that the number of fitted segments in the largest segment cluster is N;

[0120] Clustering the spatial coordinates of the vehicle window targets based on the spatial distance in the reference spatial coordinate system to obtain several vehicle window target clusters;

[0121] Extract the window target clusters ranked as top N-1 in size;

[0122] Select one window target with the highest prediction probability from each of the extracted window target clusters to obtain N-1 window targets, and filter all remaining window targets.

[0123] It is understandable that, since each side image acquired by the positioning camera may detect multiple window targets, after mapping the physical coordinates of the window space acquired by all positioning cameras to the reference space coordinate system, there will inevitably be overlapping of window targets. In this embodiment, the number of fitting line segments N involved in the spatial straight line fitting process is combined to determine the actual number of windows. Since there are support columns between two adjacent windows, when the number of window targets is 2, N=3, when the number of window targets is 3, N=4, when the number of window targets is 4, N=5, and so on. After determining the actual number of windows, small window target clusters with interference can be excluded, and only the maximum N-1 window target clusters are retained. In this way, each window target cluster corresponds to a real window target, and the one with the highest prediction probability is used as the standard. In this way, the purpose of deduplication of windows is achieved and the accuracy of window positioning is improved.

[0124] See also Figure 6 and Figure 7The present application also provides an inspection device for executing the lane inspection method of multi-camera fusion single-beam laser radar as described above, the inspection device includes a lane and a gantry arranged above the lane, a plurality of positioning cameras and at least one single-beam laser radar are respectively arranged on both sides of the lane, a sliding mechanism and a plurality of snapshot cameras arranged on the sliding mechanism are arranged on the gantry, the windows on both sides of the vehicle are positioned by the positioning camera fused with the single-beam laser radar, and the snapshot camera is driven by the sliding mechanism to the corresponding window position for snapshot detection.

[0125] Compared with the prior art, the inspection equipment provided by the present application uses the positioning camera to fuse the single-beam laser radar to locate the windows on both sides of the vehicle, and the sliding mechanism drives the capture camera to the corresponding window position for capture detection, thereby improving the vehicle inspection efficiency.

[0126] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A lane inspection method using multi-camera fusion single-beam laser radar, characterized in that: Includes steps: Through calibration, the parameter transformation matrix of each positioning camera based on the camera coordinate system and the spatial transformation relationship between each positioning camera and the single-beam laser radar based on the reference space coordinate system are obtained; Fit the side point cloud of the vehicle detected by the single-beam laser radar into a spatial straight line; According to the spatial conversion relationship, the single-beam laser radar, each positioning camera and the spatial straight line are transformed into the reference spatial coordinate system, and the distance from each positioning camera to the vehicle side is calculated respectively; Each positioning camera obtains the side image of the vehicle and obtains the window target through target detection; According to the parameter transformation matrix and the distance between the vehicle body and the vehicle window, the pixel coordinates of the vehicle window target are converted into the physical coordinates of the vehicle window space based on the camera coordinate system; The physical coordinates of the vehicle window space obtained by all positioning cameras are mapped to the reference space coordinate system, and the capture camera is controlled to move to the physical coordinates of the vehicle window space for capture detection.

2. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 1, characterized in that: The method of obtaining the parameter transformation matrix of each positioning camera based on the camera coordinate system through calibration includes the following steps: Set any two positioning cameras to be calibrated at positions with a common field of view; Place the checkerboard calibration plate at different positions and angles in the common field of view; Use two positioning cameras to collect images and obtain calibration images respectively; Corner points are extracted from the calibration images respectively, and then the two calibration images after corner point extraction are stereo matched to obtain the intrinsic parameter transformation matrix and distortion coefficient of the two positioning cameras and the extrinsic parameter transformation matrix between the two positioning cameras.

3. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 2, characterized in that: The method for obtaining the spatial transformation relationship between each positioning camera and the single-beam laser radar based on a reference space coordinate system through calibration includes the following steps: The single-beam laser radar and any one of the positioning cameras are arranged at the same position, and the midline of the field of view of the single-beam laser radar coincides with the optical axis of the positioning camera; A reference space coordinate system is established with the positioning camera as the origin.

4. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 1, characterized in that: The method for converting the pixel coordinates of a vehicle window target into the physical coordinates of the vehicle window space based on the camera coordinate system according to the parameter transformation matrix and the vehicle body distance comprises the following steps: Get the center point P (pixel) of the detection box of the window target in the pixel coordinates of the side image. x ,pixel y ) relative to the vertical pixel difference δ of the center point of the side image h and the horizontal pixel difference δ v , the calculation formula is: ; Where, h is the pixel height of the side image, and w is the pixel width of the side image; Based on the parameter transformation matrix, the vertical pixel difference δ h and the horizontal pixel difference δ v The corresponding transformation is the vertical angle Degree between the center point P of the detection frame and the center point C of the side image v And the horizontal angle Degree h , the calculation formula is: ; Among them, fov v To locate the vertical viewing angle of the camera, fov h To locate the horizontal viewing angle of the camera; Through the vertical angle Degree v And the horizontal angle Degree h Converted into physical coordinates of the window space based on the camera coordinate system.

5. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 4, characterized in that: Through the vertical angle Degree v And the horizontal angle Degree h The method of converting into the physical coordinates of the vehicle window space based on the camera coordinate system includes the following steps: The vertical angle Degree v And the horizontal angle Degree h Combined with the vehicle body distance D i Calculate the vertical physical distance D between the center point P of the detection frame and the center point C of the side image v and the horizontal physical distance D h , i represents the order of positioning cameras, and the calculation formula is: ; Based on trigonometric functions, the straight-line physical distance D between the center point P of the detection frame and the center point C of the side image is calculated. win , the calculation formula is: ; According to the straight line physical distance D win Get the physical coordinates of the vehicle window space relative to the positioning camera.

6. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 5, characterized in that: According to the straight line physical distance D win The method for obtaining the physical coordinates of the vehicle window space of the vehicle window target relative to the positioning camera comprises the steps of: Assume that the physical coordinates of the window space in the camera coordinate system (X, Y, Z) and the optical center coordinates of the positioning camera are (ppx, ppy), f x and f y Respectively represent the focal length of the positioning camera in the horizontal direction and the focal length in the vertical direction, and the calculation formula is: ; ; 。 7. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 6, characterized in that: The positioning camera is a wide-angle camera.

8. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 1, characterized in that: The method for fitting a vehicle side point cloud detected by a single-beam laser radar into a spatial straight line comprises the steps of: Clustering is performed on the side point cloud of the vehicle to obtain several point cloud clusters; Perform straight line fitting on each point cloud cluster to obtain several fitting line segments; Calculate the distance between any two fitted line segments, and group the fitted line segments whose distance is less than the distance threshold into line segment clusters; The largest line segment cluster is extracted, and all the fitting line segments in the largest line segment cluster are fitted into spatial straight lines.

9. The lane inspection method of multi-camera fusion single-beam laser radar as claimed in claim 8, characterized in that: After mapping the physical coordinates of the window space obtained by all positioning cameras to the reference space coordinate system, the following steps are included: Assume that the number of fitted segments in the largest segment cluster is N; Clustering the spatial coordinates of the vehicle window targets based on the spatial distance in the reference spatial coordinate system to obtain several vehicle window target clusters; Extract the window target clusters ranked as top N-1 in size; Select one window target with the highest prediction probability from each of the extracted window target clusters to obtain N-1 window targets, and filter all remaining window targets.

10. An inspection device for executing the lane inspection method of multi-camera fusion single-beam laser radar according to any one of claims 1 to 9, characterized in that: It includes a lane and a gantry arranged above the lane, a plurality of positioning cameras and at least one single-beam laser radar are respectively arranged on both sides of the lane, a sliding mechanism and a plurality of snapshot cameras arranged on the sliding mechanism are arranged on the gantry, the windows on both sides of the vehicle are positioned by the positioning camera and the single-beam laser radar, and the snapshot camera is driven by the sliding mechanism to the corresponding window position for snapshot detection.

Citation Information

Patent Citations

  • Vehicle detection method and device based on sensor multi-information fusion

    CN114118252A

  • In-vehicle inspection method and system based on intelligent vision

    CN118038423A