A target capturing method based on vision and 3D radar

Abstract

The application discloses a target capturing method based on vision and 3D radar, and belongs to the field of data processing, aiming to solve the problem of high precision requirement of 3D laser radar in identifying target objects. In the method, the target object is detected in real time through visual recognition, and when the target object is detected, the central angle thereof is output and transmitted to the 3D laser radar algorithm part through ROS2 topic communication. After the 3D laser radar receives the output angle value a0, data processing is performed to obtain a fifth distance; and then the obtained M fifth distances are sequentially subjected to average filtering, low-pass filtering, Kalman filtering and difference filtering, so that the final distance is obtained. The application can realize real-time capturing and positioning of the target object at a frequency of more than 10 Hz at a low cost, and complete the target object tracking task with long distance, high precision, high reliability and high frame rate. The final distance error obtained by the method is less than 5 cm.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically a target acquisition method based on vision and 3D radar. Background Technology

[0002] A certain 3D LiDAR is a hybrid solid-state LiDAR. Its actual point cloud output is 20,000 points / second, and the point cloud frame rate is 10Hz (typical value). Its near blind zone is 0.1m; its FOV is 360° horizontally and -7°~52° vertically; its ranging random error (1σ) is ≤2cm (@10m) and ≤3cm (@0.2m); its angle random error (1σ) is ≤0.15°.

[0003] When stationary, this 3D LiDAR exhibits drift in its measured point cloud data. When used to measure a specific object, which contains dozens or even more point cloud data points, the distance from the extracted center point coordinates to the 3D LiDAR data drifts. Currently, this 3D LiDAR is widely used in robotics, lawnmowers, and autonomous vehicles because its data accuracy requirements are not high, and it can meet navigation needs.

[0004] In some cases, it is necessary to directly process the point cloud to extract the corresponding point cloud of the target object and obtain stable distance data from the center point of the target object to the 3D LiDAR. In such cases, high accuracy is required for the processed data. When extracting the center point of a specific target object, the aforementioned 3D LiDAR cannot directly meet the accuracy requirements.

[0005] Therefore, this application provides a target acquisition method based on vision and 3D radar to achieve low-cost, high-frame-rate, high-accuracy, and high-stability distance detection of target objects. Summary of the Invention

[0006] The purpose of this invention is to solve the problem of high accuracy requirements for identifying target objects using 3D LiDAR, and to provide a target capture method based on vision and 3D radar.

[0007] To achieve the above objectives, this application adopts the following technical solution:

[0008] A target acquisition method based on vision and 3D radar includes the following steps:

[0009] S1. Pre-determine the height H0 of the lower end of the target object from the plane where the center of the 3D LiDAR is located, and the height H1 of the upper end of the target object from the plane where the center of the 3D LiDAR is located; use the 3D LiDAR to scan the target object, retain the point cloud data between height H0 and H1 in Cartesian coordinates, and filter out the remaining point cloud data.

[0010] The point cloud data between H0 and H1 is transformed using polar coordinates to obtain the point cloud dataset in polar coordinates, which is denoted as the first point cloud dataset.

[0011] S2. Taking the center point of the 3D LiDAR as the origin, set the front of the 3D LiDAR as the polar coordinate zero point, determine the angle between the target object and the 3D LiDAR through visual recognition, and output the angle value α0.

[0012] The point cloud data in the first point cloud dataset with an angle range of [α0-A, α0+A] is retained, the remaining point cloud data is filtered out, and the retained point cloud data is converted into point cloud data in the Cartesian coordinate system. The point cloud data converted into the Cartesian coordinate system is denoted as the second point cloud dataset; A is a constant with a range of 2-15.

[0013] S3. Use a point cloud clustering algorithm to cluster the point cloud data in the second point cloud dataset to obtain the point cloud cluster of the target object and the point cloud clusters of irrelevant objects to the left, right and behind the target object, which are denoted as the third point cloud cluster set.

[0014] Based on the target object, the approximate number of point clouds of the target object is determined in advance. Based on the approximate number of point clouds of the target object, a first threshold is set. Based on the first threshold, the point cloud clusters of irrelevant objects to the left and right of the target object in the third point cloud cluster set are filtered out to obtain the fourth point cloud cluster set.

[0015] S4. Using 3D convex hull, solve the point cloud clusters in the fourth point cloud cluster set one by one to obtain the center point of each point cloud cluster. Then calculate the distance from the center point of the point cloud cluster to the center point of the 3D LiDAR. Only retain the point cloud cluster with the shortest distance from the center point of the point cloud cluster to the center point of the 3D LiDAR. This point cloud cluster is the point cloud cluster of the target object and is denoted as the fifth point cloud cluster. Take the distance from the center point of the fifth point cloud cluster to the center point of the 3D LiDAR and denot it as the fifth distance.

[0016] S5: A 3D LiDAR scans once to obtain a fifth distance; within a certain time period, the 3D LiDAR scans N times to obtain N fifth distances; N is a natural number, and N≥2;

[0017] S6. As time progresses, new fifth distances will be continuously generated. Select the latest M fifth distances and process them sequentially using average filtering, low-pass filtering, Kalman filtering, and difference filtering to obtain the final distance and output it. M is a natural number and M≥3.

[0018] S7. Repeat step S6 to obtain the final distance at different times.

[0019] Preferably, M is 6.

[0020] The difference filtering operation is as follows: establish a container to store distance data, set the upper limit threshold and the maximum difference threshold of the container; calculate the difference between the first value entering the container and the second value entering the container, calculate the difference between the second value entering the container and the third value entering the container, and so on.

[0021] When the data stored in the container reaches the upper limit threshold, if the absolute value of all differences is less than the maximum difference threshold, then all the data in the container is the final distance; if the absolute value of any of the differences is greater than or equal to the maximum difference threshold, then all the data in the container is cleared, and the container waits for the next set of data to enter.

[0022] The upper limit threshold for storage is 3.

[0023] The maximum difference threshold is set to 0.13m.

[0024] In step S3, the operation of clustering the point cloud data in the second point cloud dataset using a point cloud clustering algorithm is as follows:

[0025] S301. Traverse each point cloud in the second point cloud dataset and assign an initial label to each point cloud. The initial label indicates whether the corresponding point cloud has been visited and the clustering label.

[0026] S302. Draw a circle with the point cloud as the center, set the radius of the circle as the neighborhood radius, and read the number of point clouds containing the point cloud within the circle. If the minimum number of point clouds within the circle is greater than or equal to the minimum number of points, then the point cloud is a core point cloud, and the point cloud is marked as visited, and the clustering label of the point cloud is modified. If the minimum number of point clouds within the circle is less than the minimum number of points, then the point cloud is an impurity point cloud, and the point cloud can be filtered out.

[0027] S303. All points within the neighborhood radius of the core point cloud are assigned to the same cluster. For each point within the neighborhood radius, if it is also a core point cloud, continue to add the point clouds within its neighborhood radius to the cluster until no new point clouds can be added.

[0028] S304. Repeat steps S302 and S303 to obtain several point cloud clusters.

[0029] In step S2, the angle between the target object and the 3D LiDAR is determined by visual recognition, and the angle value α0 is output. The specific operation is as follows:

[0030] S201. Two monocular cameras are symmetrically positioned on both sides of the 3D LiDAR. The center lines of the two monocular cameras converge at a point, which is recorded as the reverse center point. The angle formed by the center lines of the two monocular cameras is 72°. The 3D LiDAR is positioned at the reverse center point.

[0031] S202, The field of view for visual recognition is (α+72°), where α is the horizontal field of view angle of the monocular camera;

[0032] The monocular camera on the left side of the 3D LiDAR is designated as the left camera, and the monocular camera on the right side of the 3D LiDAR is designated as the right camera.

[0033] S203. Two monocular cameras are used to identify the angle of the target object;

[0034] The angle of the target object identified by the left camera is denoted as... When the target object is within the field of view of the left camera, the output angle value is... The angle of the target object identified by the right camera is recorded as... When the target object is within the field of view of the right camera, the output angle value is... ;

[0035] If the target object is simultaneously within the field of view of both the left and right cameras, then the recognition angle of one of the monocular cameras can be selected.

[0036] In step S2, if the installation error angle of the monocular camera is δ, assuming the actual installation angle of the left camera is (36°+δ) and the actual installation angle of the right camera is (-36°-δ); when the target object is within the field of view of the left camera, the output angle value is... When the target object is within the field of view of the right camera, the output angle value is... .

[0037] When δ is controlled within 0.5°, the angle conversion error is within an acceptable range.

[0038] The formula for calculating the horizontal field of view α of a monocular camera is as follows:

[0039] ;

[0040] in, The horizontal size of the image sensor for a monocular camera. The effective focal length of a monocular camera lens.

[0041] When using the aforementioned 3D LiDAR for navigation, it is necessary to establish an entire spatial point cloud system using the 3D LiDAR, and then use all the point cloud data to achieve the 3D LiDAR's own positioning and the planning of the corresponding target path. This method requires processing a large amount of point cloud data and incurs high hardware costs.

[0042] Unlike existing technologies, this invention combines vision with 3D LiDAR for detecting specific static objects. By visually identifying the angle of the static object, the scanning direction of the 3D LiDAR is determined for corresponding scanning. This invention only requires processing the point cloud within the target object's area, resulting in low cost and effectively ensuring high frame rates and high data stability and reliability.

[0043] In practical applications, the device equipped with a 3D LiDAR needs to move continuously, necessitating the use of a 3D LiDAR with a higher frame rate. Increasing the frame rate reduces the amount of point cloud data acquired per unit time, or results in incomplete shapes; conversely, reducing the frame rate fails to meet the needs of continuous device movement. Since the device equipped with the 3D LiDAR needs to move constantly, and the aforementioned 3D LiDAR is a medium-to-high frame rate 3D LiDAR, the amount of point cloud data acquired within the target object's range is relatively small, leading to target detection distance deviations and failing to meet accuracy requirements. Therefore, this invention first extracts the point cloud of the target object's center point in a specific order, then combines it with four types of filtering to ensure data stability and frame rate.

[0044] The test results show that by using this invention, the accuracy of distance data (it is indeed the distance from the 3D lidar to the target object; the error of the 3D lidar within a 10m range) and the stability of distance data can be effectively guaranteed, and the point cloud frame rate of the 3D lidar is not less than 10Hz, which meets the needs of the body movement equipped with the 3D lidar.

[0045] In summary, this invention discloses a target acquisition method based on vision and 3D radar. It employs a dual-camera visual recognition system in conjunction with a 3D LiDAR. Through data processing, it reliably detects the angle and distance of the target object relative to the 3D radar, which is then transmitted to the host device equipped with the 3D LiDAR, enabling the host device to accurately reach the set position. (Using this method, the visually determined angle and stable 3D radar distance data can be transmitted to the host device's lower-level chassis, thus completing the target object capture and positioning.) This invention achieves accurate and reliable detection of target objects and obtains high-precision (within a 10m range, error <5cm) and high frame rate (>10Hz) distance and angle data. This invention uses a specific sequence of point cloud processing algorithms to extract the target object's center point (the order cannot be reversed to ensure distance reliability); then, it employs a combination of four filtering methods to ensure stable distance data and a high frame rate (especially the differential filtering process proposed by the inventors, which directly guarantees the stability of the distance data; and by adjusting the self-defined parameters, a certain response rate is also guaranteed). Attached Figure Description

[0046] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0047] Figure 1 A schematic diagram of the test scenario in Example 1 is provided.

[0048] Figure 2 This is a schematic diagram of the target basketball hoop and net obtained by visual recognition in Example 1.

[0049] Figure 3 The above is the overall point cloud map of the scene obtained by scanning the target object with a 3D LiDAR in step S1 of Example 1.

[0050] Figure 4 In step S1 of Example 1, point cloud data with heights between H0 and H1 is retained in Cartesian coordinates.

[0051] Figure 5 This is a schematic diagram of the fifth cloud cluster in S4 of Example 1.

[0052] Figure 6 A schematic diagram showing the relative positions of the two monocular cameras and the 3D LiDAR in Example 1 is provided.

[0053] The markings in the diagram are: 1. Monocular camera, 2. 3D LiDAR, 3. Mounting bracket. Detailed Implementation

[0054] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0055] Any feature disclosed in this specification, unless otherwise stated, may be replaced by other equivalent or similar features. That is, unless otherwise stated, each feature is merely one example of a series of equivalent or similar features.

[0056] Example 1

[0057] In one example, the aiming frame, basket, and net are considered as a single unit, which is then taken as the target object. During testing, the target object is stationary. Two monocular cameras and a 3D LiDAR are mounted on a small cart, which is in motion at a speed of 2-3 m / s. The 3D LiDAR scans at 10-15 Hz. The net is set on the basket, and the aiming frame is a small rectangular frame set on the backboard near the basket. The backboard is made of transparent glass. Figure 1 A schematic diagram of the test scenario is provided. In actual scenarios, there may be several interferences.

[0058] This embodiment provides a target acquisition method based on vision and 3D radar, which includes the following steps.

[0059] S1. Pre-determine the height H0 of the lower end of the target object from the plane containing the center of the 3D LiDAR, and the height H1 of the upper end of the target object from the plane containing the center of the 3D LiDAR. In this embodiment, the actual height H0 of the lower end of the target object from the plane containing the center of the 3D LiDAR is 0.45m; the actual height H1 of the upper end of the target object from the plane containing the center of the 3D LiDAR is 1.70m.

[0060] 3D LiDAR is used to scan the target object. Figure 3 The overall point cloud map of the scene obtained by scanning the target object using a 3D LiDAR is presented. In Cartesian coordinates, point cloud data with heights between H0 and H1 are retained, while other point cloud data are filtered out, such as... Figure 4 As shown.

[0061] The point cloud data between H0 and H1 is transformed using polar coordinates to obtain the point cloud dataset in polar coordinates, which is denoted as the first point cloud dataset.

[0062] S2. Using the center point of the 3D LiDAR as the origin, and setting the area directly in front of the 3D LiDAR as the polar coordinate zero point, determine the angle between the target object and the 3D LiDAR through visual recognition, and output the angle value α0. In one test, the output angle value α0 was -18.12°. Figure 2 In the image, the combination of the target basket and the net is called object a, and the confidence level of a is 0.90; id: 1 indicates the sequence number of the first target detected in the current image.

[0063] Point cloud data within the angle range [α0-A, α0+A] of the first point cloud dataset is retained, where A is set to 3.1. Point cloud data within the angle range [-21.22°, -15.02°] of the first point cloud dataset is also retained. The remaining point cloud data is filtered out. The retained point cloud data is then converted to Cartesian coordinates and denoted as the second point cloud dataset. The number of points in the first point cloud dataset is between 1500 and 2500, while the number of points in the corresponding second point cloud dataset is between 20 and 150.

[0064] S3. Use a point cloud clustering algorithm to cluster the point cloud data in the second point cloud dataset to obtain the point cloud cluster of the target object and the point cloud clusters of irrelevant objects to the left, right and behind the target object, which are denoted as the third point cloud cluster set.

[0065] In a specific example, the operation of clustering the point cloud data in the second point cloud dataset using a point cloud clustering algorithm is as follows.

[0066] S301. Traverse each point cloud in the second point cloud dataset and assign an initial label to each point cloud. The initial label indicates whether the corresponding point cloud has been visited and the clustering label.

[0067] S302. Draw a circle with the point cloud as the center, set the radius of the circle to the neighborhood radius, and read the number of point clouds within the circle that include the current point cloud. If the minimum number of point clouds within the circle is greater than or equal to the minimum number of points, then the point cloud is a core point cloud, and it is marked as visited, and its clustering label is modified. If the minimum number of point clouds within the circle is less than the minimum number of points, then the point cloud is an impurity point cloud, and it is filtered out.

[0068] S303. All points within the neighborhood radius of the core point cloud are assigned to the same cluster. For each point within the neighborhood radius, if it is also a core point cloud, continue to add the point clouds within its neighborhood radius to the cluster until no new point clouds can be added.

[0069] S304. Repeat steps S302 and S303 to obtain several point cloud clusters.

[0070] Based on the target object, the approximate number of point clouds around the target object is pre-determined. A first threshold is then set based on this approximate number of point clouds around the target object. Based on this first threshold, point cloud clusters of irrelevant objects to the left and right of the target object in the third point cloud cluster set are filtered out, resulting in a fourth point cloud cluster set.

[0071] In a specific example, the neighborhood radius was set to 0.8m, and the minimum number of points was 3. During a test, two point cloud clusters were obtained; the approximate number of point clouds for the target object was 25-50 (i.e., the first threshold was 25-50). After filtering, one point cloud cluster remained, which is the fourth point cloud cluster set. Considering the penetrating effect of 3D LiDAR, for objects that 3D LiDAR can penetrate (such as glass), objects behind the target object along the laser emission direction of the 3D LiDAR may also form several point cloud clusters, thus making the fourth point cloud cluster set contain one or more point cloud clusters.

[0072] S4. Using 3D convex hull, solve for the point cloud clusters in the fourth point cloud cluster set sequentially to obtain the center point of each point cloud cluster. Then calculate the distance from the center point of each point cloud cluster to the center point of the 3D LiDAR. Only retain the point cloud cluster whose center point is closest to the center point of the 3D LiDAR; this point cloud cluster is the point cloud cluster of the target object, denoted as the fifth point cloud cluster. Figure 5 A schematic diagram of the fifth cloud cluster is given. The distance from the center point of the fifth cloud cluster to the center point of the 3D LiDAR is denoted as the fifth distance.

[0073] An S5 3D LiDAR scans once to obtain one fifth distance. Within a certain time period, the 3D LiDAR scans N times to obtain N fifth distances. N is a natural number, and N≥2. A certain 3D LiDAR scans 10 times per second at a frequency of 10Hz, obtaining 10 fifth distances.

[0074] S6. As time progresses, new fifth distances will be continuously generated. The latest M fifth distances are selected and processed sequentially using average filtering, low-pass filtering, Kalman filtering, and difference filtering to obtain the final distance and output it. M is a natural number, and M is determined by the average filtering window; in a certain instance, M is chosen as 6. This is equivalent to forming an array of fifth distances, which will be continuously filled with new fifth distances as time progresses. At a certain moment, the latest M fifth distances are selected for subsequent processing; at the next moment, a new fifth distance is added, and this new fifth distance becomes the last one in the array. Then, (M-1) fifth distances are taken sequentially from the previous position for subsequent processing.

[0075] The difference filtering operation is as follows: A container is created to store distance data, and an upper limit threshold and a maximum difference threshold are set for the container. The difference between the first and second values ​​entering the container is calculated, the difference between the second and third values ​​entering the container is calculated, and so on. When the container reaches its upper limit, if the absolute value of all differences is less than the maximum difference threshold, then all data in the container represents the final distance. If the absolute value of any difference is greater than or equal to the maximum difference threshold, then all data in the container is cleared, and the container awaits the next set of data. In a specific example, the upper limit threshold is 3, and the maximum difference threshold is set to 0.13m.

[0076] S7. Repeat step S6 to obtain the final distance at different times.

[0077] Table 1 presents the distance data measured using the method of this embodiment. Test results show that, under normal conditions, the distance data is stable when refreshed at a frequency of 10 Hz, with a distance error of less than 5 cm.

[0078] Table 1. Test data from Example 1

[0079] time distance Chapter 9.8s 6.86m 9.9s 6.85m 10.0s 6.84m Section 10.1s 6.84m 10.2s 6.84m 10.3s 6.83m 10.4s 6.83m 10.5s 6.84m 10.6s 6.83m 10.7s 6.83m Chapter 10.8s 6.84m Chapter 10.9s 6.85m 11.0s 6.85m Section 11.1s 6.85m 11.2s 6.87m 11.3s 6.85m 11.4s 6.86m 11.5s 6.87m 11.6s 6.87m 11.7s 6.85m 11.8s 6.86m Chapter 11.9s 6.87m 12.0s 6.85m Section 12.1s 6.86m 12.2s 6.88m 12.3s 6.87m 12.4s 6.85m 12.5s 6.86m

[0080] Furthermore, the inventive process of this application is described below.

[0081] 3D LiDAR uses a Cartesian coordinate system by default. Therefore, this application first performs height threshold filtering on the z-axis of the subscribed point cloud in the Cartesian coordinate system (i.e., step S1). This height threshold should allow the 3D LiDAR to scan the point cloud of the target object and objects at the same height as the target object, while discarding all point clouds at other heights. This step can be implemented using the filtering algorithm in the PCL point cloud library. Afterwards, this part of the point cloud is stored and converted to a spherical coordinate system for easier visual processing (i.e., step S2). Specifically, in the spherical coordinate system, based on the output angle value α0, 2-15° is added or subtracted from the angle value α0 (i.e., the range of A is 2-15°), retaining only the point cloud data within the angle range [α0-A, α0+A], discarding all other point cloud data. After this processing, based on the visually recognized output angle value α0, a region is formed in the spherical coordinate system. Combined with the height filtering of the point cloud, the approximate point cloud of the target object can be obtained, i.e., the second point cloud dataset. The second point cloud dataset is then sent to the distance processing algorithm via ROS2 (i.e., in step S3).

[0082] If there are point clouds of other objects near the target object, in order to eliminate the point cloud, a clustering algorithm needs to be applied to the point cloud within that small range (including the point cloud of the target object and the point clouds of other interfering objects, i.e., the second point cloud set) so that the point cloud of the target object forms its own class and is separated from the point clouds of other objects.

[0083] To further improve the frame rate of real-time monitoring and ensure the stability of distance data, traditional Euclidean clustering cannot be used. This application proposes a novel point cloud clustering method. Using this method, it is possible to determine which point clouds are close to each other and which are far apart. Closely spaced point clouds are grouped into a single type, forming a point cloud cluster. By adjusting the neighborhood radius and the minimum number of points, point cloud clusters that distinguish the target object from its surrounding objects can be created.

[0084] In practical testing, this point cloud clustering method was adopted, enabling each point cloud to be categorized based on its own position. From the perspective of the object, the point cloud of the target object is divided into one point cloud cluster, while other objects, due to less or no contact with the target object, are divided into other point cloud clusters. Within a distance range of 0-10m, the number of point clouds collected by the aforementioned 3D LiDAR does not change significantly with distance. Therefore, a threshold range can be roughly set based on the number of point clouds contained in the target object's point cloud cluster, allowing the target object's point cloud cluster to enter the next step of the center extraction algorithm processing, while excluding and discarding the point cloud clusters of other objects.

[0085] After obtaining the fourth point cloud cluster, the 3D convex hull is used to solve for the point cloud clusters in the fourth point cloud cluster sequentially to obtain the center point of each point cloud cluster. Then, the distance from the center point of the point cloud cluster to the center point of the 3D LiDAR is calculated. Since the shapes of target objects are different, the 3D convex hull centering algorithm in the PCL point cloud library is used. That is, a point is selected in the point cloud cluster of the target object, and a line is drawn connecting it to the adjacent points. The shape of the connecting line must be outward. If the connecting line has a tendency to be concave inward, the next point is selected and connected to it, and so on, so as to extract the outer contour features of the target point cloud cluster. Then, the center point of this outer contour is extracted to obtain the geometric center coordinates of the target object point cloud cluster (in the target object point cloud cluster). Then, through the transformation of the world coordinate system, the center point is transformed to the distance relative to the radar, thus completing the preliminary distance data extraction, i.e., the fifth distance.

[0086] Within a certain time period, a 3D LiDAR scans N times, obtaining N fifth distances. As time progresses, new fifth distances are continuously generated. The latest M fifth distances are selected and sequentially processed using averaging filtering, low-pass filtering, Kalman filtering, and difference filtering to obtain the final distance, which is then output.

[0087] To prevent large jumps in the final distance data due to minor fluctuations in the original radar point cloud, averaging filtering, low-pass filtering, Kalman filtering, and finally difference filtering are applied sequentially to ensure data stability. First, the total amount of data for averaging filtering must not be too large. To ensure real-time performance, the optimal averaging window is set to 6, meaning averaging filtering is performed every 6 distance data points; this step aims to roughly guarantee data stability. Next, the data obtained from averaging filtering is processed by low-pass filtering. The low-pass filter parameter should not be too small; otherwise, it will lead to sluggish and inaccurate data, failing to respond quickly to the latest situation and constantly being influenced by previous data. After nearly a hundred experiments, a value of 0.7 was finally determined. Next, Kalman filtering is applied to the low-pass filtered data, with process noise set to 0.3 and measurement noise set to 1.8 as optimal values. Finally, the Kalman-filtered data is fed into the difference filter container.

[0088] The so-called difference filtering means creating a container for storing the fifth distance data, setting the upper storage threshold and the maximum difference threshold, calculating the difference between the first value entering the container and the second value entering the container next, then calculating the difference between the second and the third entering the container, and so on; when the container reaches the maximum data storage capacity, that is, the upper storage threshold, if all the differences are less than the absolute value of the maximum difference threshold, all the data in the container is qualified and sent to the lower microcontroller through the serial port. If one difference is greater than the absolute value of the maximum threshold, the container is emptied and all the data is unqualified. Therefore, the maximum data storage capacity threshold should not be too large. After repeated tests, when set to 3, the frame rate can be guaranteed and the data is very stable, and the difference threshold can be set according to requirements. In this embodiment, it is set to 0.13 m to meet the requirement that the distance error is less than 0.05 m.

[0089] In step S2, the angle between the target object and the 3D lidar is determined through visual recognition, and the angle value α0 is output. The specific operation is as follows.

[0090] S201, as Figure 6 shown, two monocular cameras and a 3D lidar are set on the mounting bracket; among them, the two monocular cameras are symmetrically arranged on both sides of the 3D lidar, the midlines of the two monocular cameras converge at a point, which is denoted as the reverse center point, and the included angle formed by the midlines of the two monocular cameras is 72°; the 3D lidar is set at the reverse center point.

[0091] S202, the visual recognition field of view is (α + 72°), where α is the horizontal field of view angle of the monocular camera. The monocular camera on the left side of the 3D lidar is denoted as the left camera, and the monocular camera on the right side of the 3D lidar is denoted as the right camera.

[0092] S203, use the two monocular cameras to recognize the angle of the target object.

[0093] In conventional situations, depth cameras are typically used to identify the angle of a target object. These cameras are expensive, costing between 2,000 and 10,000 yuan per unit. Furthermore, when using mid-to-low-end depth cameras that combine point cloud and camera functions for distance measurement, their inherent ranging principles limit the reliability of data at distances of 4 meters and above. Compared to the high reliability of the 3D LiDAR used in this application at greater distances, low-to-mid-cost depth cameras clearly cannot meet the requirements. Therefore, this application employs a symmetrical layout of two monocular cameras to expand the overall visual perception range. Specifically, two identical monocular cameras are mounted on either side of the 3D LiDAR, with the left camera's optical axis pointing at +36° and the right camera's optical axis pointing at -36°. The core principle of this layout design is to significantly expand the total field of view by stitching together the fields of view of the two monocular cameras, while simultaneously ensuring continuous coverage of the central area.

[0094] According to the pinhole imaging model, the horizontal field of view of a single camera... Due to its image sensor horizontal size and lens effective focal length The calculation formula is as follows, jointly determined:

[0095] ;

[0096] in, The horizontal size of the image sensor for a monocular camera. The effective focal length of a monocular camera lens.

[0097] Taking a typical 1 / 2.7-inch sensor (5.37 mm horizontal dimension) paired with a 3.6 mm focal length lens as an example, its horizontal field of view... The calculation formula is as follows:

[0098] ;

[0099] Its arctangent is 36.73°, therefore the horizontal field of view of a single camera is... = 2 × 36.73° = 73.46°.

[0100] The total field of view (i.e., field of view range) after stitching together two monocular cameras is geometrically derived. In the established unified visual coordinate system, the angle directly in front of the 3D LiDAR is 0°, the optical axis of the left camera points to +36°, the optical axis of the right camera points to -36°, the angle directly to the left is +90°, and the angle directly to the right is -90°. The field of view range of the left camera in its own coordinate system is... After being converted to a unified visual coordinate system, it becomes Similarly, the field of view of the right camera becomes after conversion. The total field of view (i.e., the range of the field of view) is obtained by subtracting the leftmost boundary from the rightmost boundary, and the calculation formula is as follows:

[0101] .

[0102] Substituting α = 73.46°, we get... = 73.46° + 72° = 145.46°. The specific boundary positions are: leftmost field of view boundary -72.73°, rightmost field of view boundary 72.73°, which means that the established unified visual coordinate system can cover a wide area of ​​145.46° in front of it.

[0103] Furthermore, coordinate system transformation is crucial for the system to achieve unified angle output. The target angle detected by each monocular camera needs to be transformed from its own camera coordinate system to the established unified visual coordinate system. The transformation formula is based on the installation angle.

[0104] The angle of the target object identified by the left camera is denoted as... When the target object is within the field of view of the left camera, the output angle value is... The angle of the target object identified by the right camera is recorded as... When the target object is within the field of view of the right camera, the output angle value is... If the target object is simultaneously within the field of view of both the left and right cameras, then the recognition angle of one of the monocular cameras can be selected. The accuracy of this conversion directly depends on the calibration precision of the monocular camera's installation angle, therefore calibration is required. Assuming the calibration measurement shows the actual installation angle of the left camera to be 36° + δ, and the right camera's to be (-36° - δ) (where δ is the installation error), then the conversion formula needs to be corrected accordingly. Specifically: when the target object is within the field of view of the left camera, the output angle value... When the target object is within the field of view of the right camera, the output angle value is... When δ is controlled within 0.5°, the angle conversion error is within an acceptable range. In actual use of a monocular camera for recognition, YOLOv8 is used for processing to obtain the corresponding angle. In one example, the output angle value α0 is passed to the subsequent algorithm part through ROS2 topic communication.

[0105] Analyzing the overlapping region of the fields of view is crucial for system performance evaluation. In the established unified visual coordinate system, the overlapping region of the fields of view of the two monocular cameras appears... The interval has an overlap angle of ( ). Substituting α = 73.46°, we obtain an overlap angle of 1.46°; a positive value indicates the existence of a 1.46° field-of-view overlap area. This overlap area lies within an extremely narrow range from +0.73° to -0.73° directly in front of the 3D LiDAR, meaning there is double coverage in the central area directly in front of the system. This improves the reliability of target detection in the central area, and the small overlap area prevents parallax conflicts. Compared to the 16.54° central blind zone produced by the 45° installation scheme, the 36° scheme almost completely eliminates the central blind zone problem.

[0106] This application employs a visual recognition system consisting of two monocular cameras, in conjunction with a 3D LiDAR. The design of the two monocular cameras, while maintaining a wide visual perception range, significantly improves the coverage continuity of the central area, providing more accurate and stable target angle information, thus laying a more solid foundation for subsequent 3D LiDAR ranging.

[0107] In summary, this application employs a visual recognition system comprised of two monocular cameras, working in conjunction with a 3D LiDAR. First, the target object is detected in real-time using visual recognition. Upon detection, its center angle is output and transmitted to the 3D LiDAR algorithm via ROS2 topic communication. After receiving the output angle value α0, the 3D LiDAR processes the data to obtain the fifth distance. Then, the M fifth distances are sequentially subjected to averaging, low-pass filtering, Kalman filtering, and difference filtering to obtain the final distance. Experimental results show that the final distance to the target object obtained using this application has an error of less than 5cm, exhibiting extremely high measurement accuracy, data stability, and reliability. Based on the obtained output angle value and final distance, this information is transmitted to the chassis equipped with the two monocular cameras and the 3D LiDAR. The chassis calculates the angle, rotates to face the target object, and, knowing the distance, moves forward, effectively capturing the target object.

[0108] Compared with existing technologies, this application has the advantage of low cost. Currently, existing detection methods of the same type and price range that achieve the same purpose, if using the same hardware as this application, will suffer from problems such as insufficient point cloud data due to radar quality or insufficient processor computing power, ultimately leading to data drift and insufficient frame rate.

[0109] This application enables real-time capture and positioning of target objects at frequencies above 10Hz at low cost, achieving long-distance, high-precision, high-reliability, and high-frame-rate target object tracking tasks.

[0110] Comparative Example 1

[0111] In the target detection process of Example 1, the processing order in step S6 cannot be changed. In one test, the processing order of averaging filtering and low-pass filtering in step S6 was swapped, while everything else remained the same as in Example 1. The test revealed that performing low-pass filtering first, followed by averaging filtering, caused irreversible damage to the original data and reduced the data refresh rate. The inventors analyzed this and concluded that the impact of averaging filtering on the original data can be adjusted according to the size of the averaging window. A small number of data points can be averaged together to more accurately reproduce the original radar data and achieve a faster frame rate; alternatively, multiple data points can be grouped together, reducing the reliability of the radar data and lowering the frame rate. However, placing low-pass filtering before averaging filtering only reduces the reliability of the radar data and affects the overall frame rate, which is unacceptable in real-time target acquisition.

[0112] Comparative Example 2

[0113] In one test, in step S6, the difference filtering was placed before the averaging filter, low-pass filter, or Kalman filter, otherwise the process was the same as in Example 1. Experimental results showed that changing the relative position of the difference filtering process significantly reduced the frame rate, failing to meet the requirements of real-time measurement and causing severe lag in target object capture.

[0114] After analysis, the inventor concluded that in step S6, the difference filtering must be placed as the last step, as he controls the final output of the data. The algorithm is designed to stabilize the data as much as possible. If it is placed earlier, it will significantly affect the frame rate and the core function of the algorithm.

[0115] Comparative Example 3

[0116] In one test, the difference filtering process was removed in step S6, otherwise it was the same as in Example 1. The test results are shown in Table 2. The test results show that the distance data began to show serious jumps, with the maximum even reaching 16cm, and the data was no longer stable.

[0117] Table 2 Comparative Example 3 Test Data

[0118] time distance 5.5s 6.96m 5.6s 7.03m 5.7s 6.98m Chapter 5.8s 7.05m Chapter 5.9s 7.18m 6.0s 7.15m Section 6.1s 7.07m 6.2s 6.94m 6.3s 7.06m 6.4s 7.04m 6.5s 7.05m 6.6s 7.06m 6.7s 6.90m Chapter 6.8s 6.86m 6.9s 6.90m 7.0s 7.05m Section 7.1s 6.92m 7.2s 6.87m 7.3s 6.94m 7.4s 6.98m 7.5s 7.05m 7.6s 7.02m 7.7s 6.95m Chapter 7.8s 6.98m Chapter 7.9s 7.08m 8.0s 7.02m Section 8.1s 6.92m 8.2s 6.86m

[0119] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A target acquisition method based on vision and 3D radar, characterized in that, Includes the following steps: S1. Pre-determine the height H0 of the lower end of the target object from the plane where the center of the 3D LiDAR is located, and the height H1 of the upper end of the target object from the plane where the center of the 3D LiDAR is located; use the 3D LiDAR to scan the target object, retain the point cloud data between height H0 and H1 in Cartesian coordinates, and filter out the remaining point cloud data. The point cloud data between H0 and H1 is transformed using polar coordinates to obtain the point cloud dataset in polar coordinates, which is denoted as the first point cloud dataset. S2. Taking the center point of the 3D LiDAR as the origin, set the front of the 3D LiDAR as the polar coordinate zero point, determine the angle between the target object and the 3D LiDAR through visual recognition, and output the angle value α0. The point cloud data in the first point cloud dataset with an angle range of [α0-A, α0+A] is retained, the remaining point cloud data is filtered out, and the retained point cloud data is converted into point cloud data in the Cartesian coordinate system. The point cloud data converted into the Cartesian coordinate system is denoted as the second point cloud dataset; A is a constant with a range of 2-15. S3. Use a point cloud clustering algorithm to cluster the point cloud data in the second point cloud dataset to obtain the point cloud cluster of the target object and the point cloud clusters of irrelevant objects to the left, right and behind the target object, which are denoted as the third point cloud cluster set. Based on the target object, the approximate number of point clouds of the target object is determined in advance, and a first threshold is set based on the approximate number of point clouds of the target object. Based on the first threshold, the point cloud clusters of irrelevant objects to the left and right of the target object in the third point cloud cluster set are filtered out to obtain the fourth point cloud cluster set. S4. Use 3D convex hull to solve the point cloud clusters in the fourth point cloud cluster set in turn to obtain the center point of each point cloud cluster, and then calculate the distance from the center point of the point cloud cluster to the center point of the 3D lidar. Only the point cloud cluster whose center point is closest to the center point of the 3D LiDAR is retained. This point cloud cluster is the point cloud cluster of the target object and is denoted as the fifth point cloud cluster. The distance from the center point of the fifth point cloud cluster to the center point of the 3D LiDAR is denoted as the fifth distance. S5: A 3D LiDAR scans once to obtain a fifth distance; within a certain time period, the 3D LiDAR scans N times to obtain N fifth distances; N is a natural number, and N≥2; S6. As time progresses, new fifth distances will be continuously generated. Select the latest M fifth distances and process them sequentially using average filtering, low-pass filtering, Kalman filtering, and difference filtering to obtain the final distance and output it. M is a natural number and M≥3. S7. Repeat step S6 to obtain the final distance at different times; The difference filtering operation is as follows: establish a container to store distance data, set the upper limit threshold and the maximum difference threshold of the container; calculate the difference between the first value entering the container and the second value entering the container, calculate the difference between the second value entering the container and the third value entering the container, and so on. When the data stored in the container reaches the upper limit threshold, if the absolute value of all differences is less than the maximum difference threshold, then all the data in the container is the final distance; if the absolute value of any of the differences is greater than or equal to the maximum difference threshold, then all the data in the container is cleared, and the container waits for the next set of data to enter.

2. The target acquisition method based on vision and 3D radar according to claim 1, characterized in that, M is 6.

3. The target acquisition method based on vision and 3D radar according to claim 1, characterized in that, The upper limit threshold for storage is 3, and the maximum difference threshold is set to 0.13m.

4. The target acquisition method based on vision and 3D radar according to claim 1, characterized in that, In step S3, the operation of clustering the point cloud data in the second point cloud dataset using a point cloud clustering algorithm is as follows: S301. Traverse each point cloud in the second point cloud dataset and assign an initial label to each point cloud. The initial label indicates whether the corresponding point cloud has been visited and the clustering label. S302. Draw a circle with the point cloud as the center, set the radius of the circle as the neighborhood radius, and read the number of point clouds inside the circle that include the point cloud. If the minimum number of point clouds inside the circle is greater than or equal to the minimum number of points, then the point cloud is the core point cloud. Then mark the point cloud as visited and modify the clustering label of the point cloud. If the minimum number of point clouds within the circle is less than the minimum number of points, then the point cloud is an impurity point cloud, and it can be filtered out. S303. All points within the neighborhood radius of the core point cloud are assigned to the same cluster. For each point within the neighborhood radius, if it is also a core point cloud, continue to add the point clouds within its neighborhood radius to the cluster until no new point clouds can be added. S304. Repeat steps S302 and S303 to obtain several point cloud clusters.

5. The target acquisition method based on vision and 3D radar according to any one of claims 1 to 4, characterized in that, In step S2, the angle between the target object and the 3D LiDAR is determined by visual recognition, and the angle value α0 is output. The specific operation is as follows: S201. Two monocular cameras are symmetrically positioned on both sides of the 3D LiDAR. The center lines of the two monocular cameras converge at a point, which is recorded as the reverse center point. The angle formed by the center lines of the two monocular cameras is 72°. The 3D LiDAR is positioned at the reverse center point. S202, The field of view for visual recognition is (α+72°), where α is the horizontal field of view angle of the monocular camera; The monocular camera on the left side of the 3D LiDAR is designated as the left camera, and the monocular camera on the right side of the 3D LiDAR is designated as the right camera. S203. Two monocular cameras are used to identify the angle of the target object; The angle of the target object identified by the left camera is denoted as... When the target object is within the field of view of the left camera, the output angle value is... ; The angle of the target object identified by the right camera is recorded as... When the target object is within the field of view of the right camera, the output angle value is... ; If the target object is simultaneously within the field of view of both the left and right cameras, then the recognition angle of one of the monocular cameras can be selected.

6. The target acquisition method based on vision and 3D radar according to claim 5, characterized in that, In step S2, if the installation error angle of the monocular camera is δ, assuming the actual installation angle of the left camera is (36° + δ) and the actual installation angle of the right camera is (-36° - δ); when the target object is within the field of view of the left camera, the output angle value is... ; When the target object is within the field of view of the right camera, the output angle value is... .

Images