A clustering-based semantic radar generation and mapping method

Abstract

The present invention discloses a clustering-based semantic radar generation and mapping method, comprising the following steps: obtaining input data; clustering and target detection; matching radar clusters with detection frames; optimizing matching results; semantic radar generation; and dual-path radar generation. The present invention uses distance clustering to effectively classify radar points into different categories based on distance, with high real-time performance, laying a solid foundation for subsequent object and background distinction. Furthermore, semantic information can be effectively assigned to radar points based on the ratio of each cluster of radar points to the target frame. The ratio of each cluster of radar points falling within the detection frame to all radar points within the frame is then calculated to distinguish whether the radar points in that cluster are objects or background. Furthermore, radar points with semantic information are only updated when they are observed within the visual range, preventing newly constructed semantic objects from disappearing as the robot moves.

Description

Technical Field

[0001] The present invention relates to the field of visual processing, and in particular to a semantic radar generation and mapping method based on clustering. Background Art

[0002] LiDAR-based SLAM mapping is widely used in sweeping robots. The robots avoid obstacles based on a grid map to complete their cleaning tasks. However, due to cost constraints, sweeping robots often use single-line radar for SLAM mapping. Since single-line radar only scans a plane at a fixed height, the scanning frequency is low and the information it acquires about the surrounding environment is limited. This results in the sweeping robot being unaware of its surroundings and its responses to changes in the surrounding environment being simplistic, hindering its intelligent development.

[0003] In recent years, solutions using multi-sensor fusion mapping have emerged, enabling sweeping robots to obtain richer information and adapt to different environments. A common approach is the semantic mapping solution based on lidar and cameras. Compared to traditional lidar-based mapping solutions, the semantic maps generated by this solution not only depict the location and shape of obstacles but also identify the objects they represent, further enhancing the "intelligence" of sweeping robots.

[0004] In existing technologies, semantic mapping often uses a mapping strategy based on semantic radar, which relies on processing the images captured by the camera through the output of the target detection model. However:

[0005] (1) Due to factors such as occlusion and unstable target detection models, the semantic radar generated is not accurate. For example, if the target detection box is too large, the background radar points will be assigned semantic information. If the two object detection boxes intersect, the radar points in the intersecting part will lack spatial information, which will lead to confusion of radar semantics, that is, incorrect assignment or omission of radar semantic information.

[0006] (2) In semantic SLAM mapping, since the radar semantics are given by the camera, and there is a difference between the camera view and the radar view, objects with semantics will be gradually updated during the movement of the robot. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a semantic radar generation and mapping method based on clustering.

[0008] The object of the present invention is achieved through the following technical solutions:

[0009] A first aspect of the present invention provides a clustering-based semantic radar generation and mapping method, comprising the following steps:

[0010] Acquiring input data: acquiring radar data collected by the radar and image data collected by the camera, wherein the radar data includes radar information, angle information, and distance information;

[0011] Clustering and target detection: Radar information is clustered by distance to form several radar clusters, each of which includes several radar points. The image data is input into the target detection model to obtain the detection box and the corresponding object category.

[0012] Matching radar clusters with detection frames: Determine the first ratio of each radar cluster falling into the corresponding detection frame. If the first ratio is greater than a first preset ratio, match the radar cluster with the corresponding detection frame. If a radar cluster falls into multiple detection frames and all meet the first preset ratio, match the radar cluster with the smaller detection frame.

[0013] Optimize matching results: When a detection frame matches at least two radar clusters, calculate the second ratio of the number of radar points in each radar cluster that fall within the detection frame relative to the number of all radar points in the detection frame. If the second ratio is greater than a second preset ratio, the radar cluster is considered a background radar cluster and its matching with the detection frame is canceled.

[0014] Semantic radar generation: Assign a semantic label to each item category, assign the semantic label corresponding to the detection box to all radar points in the radar cluster that match the detection box, and generate semantic radar information. The semantic radar information, angle information, and distance information form semantic radar data.

[0015] Dual-path radar generation: The radar points whose angle information of the semantic radar data matches the camera's field of view are recorded as view radar points. All view radar points and their corresponding angle information and distance information are combined to obtain visual range data. The semantic radar data and visual range data are separately used for SLAM mapping and superimposed.

[0016] Furthermore, the radar information is clustered by distance to form a plurality of radar clusters, specifically including:

[0017] Clustering is performed on all radar points in a frame of radar data: if the difference between the distance information of the current radar point and the distance information of the previous radar point is less than a third preset ratio of the distance information of the previous radar point, the current radar point and the previous radar point are considered to belong to the same class and the same cluster; otherwise, the current radar point belongs to a new class.

[0018] Furthermore, the third preset ratio is 0.06.

[0019] Furthermore, the image data is input into the target detection model to obtain a detection frame and a corresponding item category, which is specifically determined by the YOLOv5 target detection model; the detection frame is a square frame in the same direction as the image data.

[0020] Furthermore, the first ratio of each radar cluster falling into the corresponding detection frame is determined respectively, specifically a length ratio or a point ratio.

[0021] Furthermore, the first preset ratio is 0.8, and the second preset ratio is 0.2.

[0022] Furthermore, the camera viewing angle range is 50-130 degrees.

[0023] The beneficial effects of the present invention are:

[0024] In an exemplary embodiment of the present invention, compared with the particle swarm clustering method adopted by some existing technologies, which has the problem of low real-time performance, distance clustering can effectively divide radar points into different categories according to distance, with high real-time performance, laying a good foundation for subsequent distinction between objects and backgrounds; at the same time, according to the ratio information of each cluster of radar points to the target frame, semantic information can be effectively assigned to the radar points, and then the ratio of each cluster of radar points falling into the detection frame relative to the number of all radar points in the frame is calculated to distinguish whether the cluster of radar points is an object or background, such as a hollow table.

[0025] Moreover, in this exemplary embodiment, radar points with semantic information are updated only when they are observed within the visual range (generated by dual-path radar), and are not updated outside the visual range. This can avoid the problem of newly built semantic objects disappearing as the robot moves. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a clustering-based semantic radar generation and mapping method provided by an exemplary embodiment of the present invention;

[0027] Figure 2 A schematic diagram illustrating the relative positions of a detection frame and a radar cluster provided by an exemplary embodiment of the present invention;

[0028] Figure 3 A schematic diagram of the relative positions of a detection frame and a radar cluster provided in accordance with another exemplary embodiment of the present invention;

[0029] Figure 4 A schematic diagram of two radar clusters located in a unified detection frame provided by an exemplary embodiment of the present invention;

[0030] Figure 5 A schematic diagram comparing the radar and camera fields of view provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described clearly and completely below with reference to the accompanying drawings. It is apparent that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "up," "down," "left," "right," "vertical," "horizontal," "inside," and "outside" are based on the directions or positional relationships described in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on the present invention. Furthermore, terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0033] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention.

[0034] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] See also Figure 1 , Figure 1 A flowchart of a clustering-based semantic radar generation and mapping method provided by an exemplary embodiment of the present invention is shown, comprising the following steps:

[0036] Acquiring input data: acquiring radar data collected by the radar and image data collected by the camera, wherein the radar data includes radar information, angle information, and distance information;

[0037] Clustering and target detection: Radar information is clustered by distance to form several radar clusters, each of which includes several radar points. The image data is input into the target detection model to obtain the detection box and the corresponding object category.

[0038] Matching radar clusters with detection frames: Determine the first ratio of each radar cluster falling into the corresponding detection frame. If the first ratio is greater than a first preset ratio, match the radar cluster with the corresponding detection frame. If a radar cluster falls into multiple detection frames and all meet the first preset ratio, match the radar cluster with the smaller detection frame.

[0039] Optimize matching results: When a detection frame matches at least two radar clusters, calculate the second ratio of the number of radar points in each radar cluster that fall within the detection frame relative to the number of all radar points in the detection frame. If the second ratio is greater than a second preset ratio, the radar cluster is considered a background radar cluster and its matching with the detection frame is canceled.

[0040] Semantic radar generation: Assign a semantic label to each item category, assign the semantic label corresponding to the detection box to all radar points in the radar cluster that match the detection box, and generate semantic radar information. The semantic radar information, angle information, and distance information form semantic radar data.

[0041] Dual-path radar generation: The radar points whose angle information of the semantic radar data matches the camera's field of view are recorded as view radar points. All view radar points and their corresponding angle information and distance information are combined to obtain visual range data. The semantic radar data and visual range data are separately used for SLAM mapping and superimposed.

[0042] Specifically, in this exemplary embodiment, compared to the particle swarm clustering method used in some existing technologies, which has low real-time performance, this exemplary embodiment uses distance clustering to effectively classify radar points into different categories based on distance, with high real-time performance, laying a solid foundation for subsequent distinction between objects and background. At the same time, this exemplary embodiment effectively assigns semantic information to radar points based on the ratio of each cluster of radar points to the target frame. The ratio of each cluster of radar points falling within the detection frame to the total number of radar points within the frame is then calculated to distinguish whether the radar points in that cluster are objects or background. For example, in the image of a hollow table within the detection frame, the background accounts for a large proportion, but there is only semantic information and probability for the table. If the method of assigning semantic information to radar points based solely on the semantic labels and probabilities given by target detection, as in some existing technologies, is used, it is not possible to effectively distinguish between objects and background, and it is impossible to distinguish which radar points belong to the table and which are background.

[0043] Moreover, in this exemplary embodiment, radar points with semantic information are updated only when they are observed within the visual range (generated by dual-path radar), and are not updated outside the visual range. This can avoid the problem of newly built semantic objects disappearing as the robot moves.

[0044] The following exemplary embodiment will explain each step in detail, where the required input information includes: detection box information (bboxs) in a frame of image after the target detection model is used, binocular ranging information (dwha) of objects in the detection box after the binocular ranging model is used, the angle and distance information of surrounding obstacles output by the radar sensor (laser), the three-dimensional coordinate information converted from the laser (pointcloud), and the radar cluster information after the laser is clustered (pointcloud_cluster).

[0045] The specific format and content of the input information are as follows:

[0046] bboxs: bboxs is in matrix form. Assuming that n object boxes are detected after a frame of image is detected by the target detection model, the size of bboxs is n*6. Each line is the information of an object detection box. For each line, the 1st to 4th columns are the coordinates of the detection box (x min ,y min ,x max ,y max ), the fifth column is the confidence, and the sixth column is the object category.

[0047] dwha: dwha is in matrix form. Assuming that n object boxes are detected in an image frame after the object detection model, the size of dwha is n*4. Each row contains the binocular ranging information of an object detection box. For each row, the first column is the distance from the camera to the object, the second column is the width of the object, and the third column is the height of the object. The fourth column is the angle of the object relative to the front of the camera.

[0048] Laser: Laser is the data collected by the radar. This includes the number of points in a radar frame (len), and the distance, angle, intensity, and semantic label of each radar point.

[0049] pointcloud: pointcloud is the three-dimensional coordinates of the radar. From the laser above, we can see that the radar stored in the laser is distance and angle. The three-dimensional coordinates of the radar point are required in the conversion process from the radar coordinate system to the camera coordinate system. Therefore, it is necessary to convert the radar point into three-dimensional coordinates through trigonometric calculations based on the distance and angle information of the radar point. Assuming that the number of points in a radar frame is len, the matrix size of pointcloud is len*4. The first three columns are the x, y, and z axis coordinates respectively, and the fourth column is the index of the radar point. The purpose of adding an index to each radar point is to obtain a mapping between the pointcloud and the intensities index value in FrameData, which facilitates the modification of the intensities value.

[0050] pointcloud_cluster: pointcloud_cluster is the result of radar clustering. If the difference between the distance of the current radar point and the distance of the previous radar point is less than 6% of the distance of the previous radar point, the current radar point and the previous radar point are considered to belong to the same category. Otherwise, the current radar point belongs to a new category. And so on, all radar points of a frame of radar are clustered. The data type of pointcloud_cluster is vector <vector <mat>>, a Mat of size 1*4, stores the 3D coordinate information and index value of a radar point. Assuming the number of radar points in a frame is len, and there are n categories after clustering, then the size of pointcloud_cluster is n, the sum of the elements of pointcloud_cluster[0]-pointcloud_cluster[n-1] is len, and pointcloud_cluster[i][j] is the jth radar point in the i-th category.

[0051] More preferably, in an exemplary embodiment, clustering the radar information by distance to form a plurality of radar clusters specifically includes:

[0052] Clustering is performed on all radar points in a frame of radar data: if the difference between the distance information of the current radar point and the distance information of the previous radar point is less than a third preset ratio of the distance information of the previous radar point, the current radar point and the previous radar point are considered to belong to the same class and the same cluster; otherwise, the current radar point belongs to a new class.

[0053] For target detection, the image data is input into the target detection model to obtain the detection frame and the corresponding object category, which is specifically determined by the YOLOv5 target detection model; the detection frame is a square frame in the same direction as the image data.

[0054] Matching radar clusters with detection frames includes:

[0055] (1) Calculation of the proportion of each radar cluster in a given detection frame

[0056] Assume (x0, y0) is the first radar point of a cluster of radars (laser b ) corresponds to the pixel coordinates, (x1, y1) is the last radar point of a cluster of radars (laser e ) corresponding pixel coordinates, bbox is the information of a detection box, and the coordinates of the upper left corner of the detection box are (x min ,y min ), the coordinate of the lower right corner is (x max ,y max ). Based on the above information, determine the proportion of the radar points of this cluster that fall within the detection frame.

[0057] In one exemplary embodiment, the first ratio is a length ratio, and the specific steps are as follows:

[0058] ① Determine the laser b and laser e Whether the y-axis coordinates y0 and y1 fall into the y-axis interval [y min ,y max ], if at least one of them does not fall into this interval, the radar cluster is assigned a label of -1.

[0059] ②Calculate the proportion of the cluster radar falling into a detection frame

[0060] Figure 2 and Figure 3 The relative positions of the detection box bbox and a cluster of radar clusters are listed respectively. The large rectangular box is a frame of image, and its coordinate system is the pixel coordinate system; the small rectangular box is the bbox, and its upper left corner coordinate is (x min ,y min ), the coordinate of the lower right corner is (x max ,y max ); the dotted line represents a radar cluster, and each point represents a radar point in the radar cluster. The first radar point (laser b ) is at (x0,y0), and the last radar point (laser e )’s coordinates are (x1,y1).

[0061] Figure 2 The straight line on the left side of the detection frame divides the radar cluster into two parts, inside and outside the frame. The total length of the radar cluster is x1-x0, of which the length of the part falling into the detection frame is x1-x m i , so the proportion of the cluster radar falling into the detection frame is

[0062] Figure 3 The length of the radar cluster falling into the detection frame is 0, so the proportion of the radar cluster falling into the detection frame is 0.

[0063] In another exemplary embodiment, the first ratio is a point ratio, and the specific details are not repeated here.

[0064] (2) Match each cluster of radar points with all detection boxes

[0065] This process matches an optimal detection frame for each cluster of radars and assigns a detection frame index label. If no detection frame matches, it is assigned -1.

[0066] ① Loop through the above (1) to calculate the proportion of a cluster of radar points in each frame. If the proportion is greater than a threshold (0.8), the radar cluster is considered to be in the detection frame and assigned the frame index. If there is no detection frame, -1 is assigned. When a radar cluster falls into multiple detection frames, a smaller frame is matched for the radar cluster. The result is that the radar cluster corresponds to the index cluster_bboxindex of a detection frame. That is, the semantic probability calculated by the detection frame ratio of this exemplary embodiment ensures that the correct semantic information can be assigned to each radar cluster. Because many radar clusters fall into the same target frame, it is necessary to find the radar point that is most likely to correspond to the detection frame based on the detection frame ratio and assign semantic information.

[0067] ② Loop through the above ① process and traverse each radar cluster. The result is the index clusters_bboxindex of a detection box corresponding to each radar cluster.

[0068] For optimizing radar cluster and detection box matching (i.e. optimizing matching results):

[0069] This process primarily removes radar clusters from background areas of hollow objects. For hollow objects like tables and chairs, a large number of background radar clusters will fall within the detection frame. For each radar cluster falling within the detection frame, its ratio relative to the total number of radar points within the frame is calculated. If the ratio is greater than a threshold (0.2), the cluster is considered background and the detection frame index corresponding to the cluster is changed to -1.

[0070] like Figure 4 As shown in the figure, there are two radar clusters in the detection box. The upper radar cluster (two points) represents the table legs, while the lower radar cluster (eight points) is the background. According to the ratio restriction, the semantic information of the table can be accurately assigned to the two radar points on the upper side.

[0071] For semantic radar generation:

[0072] This step is to assign semantic information labels to each radar point. The specific steps are as follows:

[0073] ① Assign a semantic label to each object category, and pass in clusters_bboxindex and detection box information bboxs.

[0074] ② Traverse each cluster of radars. Assume that the detection box index corresponding to the i-th cluster radar cluster_bboxindex[i] is j. If j is -1, it means that there is no detection box corresponding to the cluster radar, and no semantic assignment is performed. In other cases, the semantic c corresponding to the object category of the box is found based on the detection box index j.

[0075] ③ Traverse all radar points of the cluster radar. Assuming that the index value stored in a certain radar point is i, assign c to the semantic part of laser{i}.

[0076] ④ Repeat the ② and ③ processes until all radar clusters are traversed.

[0077] For dual radar generation:

[0078] Conventional radars have a 360-degree field of view, while cameras usually have a smaller field of view (about 80 degrees). During the SLAM mapping process, due to the difference in field of view, the semantic map that has been built will be updated after the robot moves. Figure 5 As shown in the figure, the rectangle in the circle represents the robot's position, the circle represents the radar scanning angle range, the shaded area in the circle represents the camera's field of view, and the triangle represents the scanned object. According to the SLAM mapping strategy, when an object is within the field of view of both the radar and the camera, the object appears on the grid map and is assigned correct semantic information. When the robot rotates a certain angle and the object is no longer within the camera's field of view, SLAM gradually updates the already constructed semantically correct grid map. To address these issues, a dual-path radar SLAM mapping scheme based on different viewpoints is proposed.

[0079] The original laser angle range is 0-360 degrees. Assuming that the camera angle range is 50-130 degrees in the radar coordinate system, the steps to generate a dual-path radar are:

[0080] ① Take a radar point l in the laser i

[0081] ②If l i If the angle is within the range of 50-130 degrees, then l i Place in laser1

[0082] ③ Loop ① and ② processes until all radar points in the laser are traversed

[0083] After completing the above steps, we obtain a laser with an angular range of 0-360 degrees and laser1 with an angular range of 50-130 degrees. Both laser and laser1 are fed into the SLAM mapping system simultaneously. Laser and laser1 construct maps separately and then overlay them, ensuring both construction speed and preventing updates to the semantic grid map. Radar points with semantic information are updated only when they are within the visual range; updates are not performed outside of the visual range. This prevents newly constructed semantic objects from disappearing as the robot moves.

[0084] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications can be made based on the above descriptions. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.< / mat>

Claims

1. A clustering-based semantic radar generation and mapping method, characterized by: The following steps are involved: Acquiring input data: acquiring radar data collected by the radar and image data collected by the camera, wherein the radar data includes radar information, angle information, and distance information; Clustering and target detection: Radar information is clustered by distance to form several radar clusters, each of which includes several radar points. The image data is input into the target detection model to obtain the detection box and the corresponding object category. Matching radar clusters with detection frames: Determine the first ratio of each radar cluster falling into the corresponding detection frame. If the first ratio is greater than a first preset ratio, match the radar cluster with the corresponding detection frame. If a radar cluster falls into multiple detection frames and all meet the first preset ratio, match the radar cluster with the smaller detection frame. Optimize matching results: When a detection frame matches at least two radar clusters, calculate the second ratio of the number of radar points in each radar cluster that fall within the detection frame relative to the number of all radar points in the detection frame. If the second ratio is greater than a second preset ratio, the radar cluster is considered a background radar cluster and its matching with the detection frame is canceled. Semantic radar generation: Assign a semantic label to each item category, assign the semantic label corresponding to the detection box to all radar points in the radar cluster that match the detection box, and generate semantic radar information. The semantic radar information, angle information, and distance information form semantic radar data. Dual-path radar generation: The radar points whose angle information of the semantic radar data matches the camera's field of view are recorded as viewpoint radar points. All viewpoint radar points, their corresponding angle information, and distance information are combined to obtain the visual range data. Semantic radar data and visual range data are separately used for SLAM mapping and superimposed.

2. The clustering-based semantic radar generation and mapping method according to claim 1, characterized in that: The radar information is clustered by distance to form several radar clusters, specifically including: Clustering is performed on all radar points in a frame of radar data: if the difference between the distance information of the current radar point and the distance information of the previous radar point is less than a third preset ratio of the distance information of the previous radar point, the current radar point and the previous radar point are considered to belong to the same class and the same cluster; otherwise, the current radar point belongs to a new class.

3. The clustering-based semantic radar generation and mapping method according to claim 2, characterized in that: The third preset ratio is 0.

06.

4. The clustering-based semantic radar generation and mapping method according to claim 1, characterized in that: The image data is input into the target detection model to obtain a detection frame and the corresponding object category, which is specifically determined using the YOLOv5 target detection model; the detection frame is a square frame in the same direction as the image data.

5. The clustering-based semantic radar generation and mapping method according to claim 1, characterized in that: The first ratio of each radar cluster falling into the corresponding detection frame is determined respectively, specifically a length ratio or a point ratio.

6. The clustering-based semantic radar generation and mapping method according to claim 1 or 5, characterized in that: The first preset ratio is 0.8, and the second preset ratio is 0.

2.

7. The clustering-based semantic radar generation and mapping method according to claim 1, characterized in that: The camera viewing angle range is 50-130 degrees.

Images