Long-tail data acquisition method, device and equipment and readable storage medium

Abstract

The embodiment of the invention provides a long tail data acquisition method, device and equipment and a readable storage medium, and the method comprises the steps: recognizing whether a noisy point region exists in a first BEV segmentation image in the normal operation process of a robot or not, thereby collecting reasons caused by illumination change, ground reflection, unknown obstacle types and the like, and carrying out the recognition of the noisy point region. According to the method, the BEV segmentation image which is not good in segmentation effect in the normal operation process is acquired, so that the coverage range of long-tail data is improved, and the aim of improving generalization and robustness of the perception model is achieved. Besides, whether the first BEV segmented image is long-tail data or not is identified by taking reliable laser radar data as a reference value, so that the BEV segmented image with a poor segmentation effect caused by inaccurate prediction of the spatial distance of the obstacle by the sensing model is acquired, the coverage range of the long-tail data is expanded, and the accuracy of the detection result is improved. Therefore, the generalization and robustness of the perception model are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a long-tail data acquisition method, apparatus, device, and readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, various self-moving devices are increasingly entering people's lives, such as shopping mall service robots, logistics robots, sweeping robots, and self-driving cars.

[0003] Self-moving devices achieve autonomous navigation and obstacle avoidance by perceiving their environment. Bird's-eye view (BEV) perception models are widely used due to their relatively low cost. However, while BEV perception models have good accuracy in ordinary scenarios, they exhibit poor generalization and robustness in challenging scenarios. Challenging scenarios refer to diverse and rare situations. To train a BEV perception model with strong generalization and high robustness, it is necessary to collect sample data for challenging scenarios, also known as long-tail data. A common approach is to record data n seconds before and after a preset event occurs during the self-moving device's movement, thus obtaining long-tail data. Preset events include collisions, manual intervention in autonomous driving, and other similar events.

[0004] However, the aforementioned long-tail data collection methods only collect long-tail data based on whether anomalies occur during the operation of the self-moving device. The long-tail data is singular, has a small coverage, and has limited improvement on the BEV perception model. Summary of the Invention

[0005] This application provides a long-tail data acquisition method, apparatus, device, and readable storage medium. By determining whether the first BEV segmentation image during the normal operation of the robot is long-tail data and collecting it, the purpose of collecting long-tail data during the normal operation of the robot is achieved, thereby improving the generalization and robustness of the perception model.

[0006] In a first aspect, embodiments of this application provide a long-tail data acquisition device, comprising:

[0007] During normal driving, the perception model is used to process the environmental image of the current location to generate the first BEV segmentation image;

[0008] The first BEV segmentation image is binarized to obtain a binarized image. The first BEV segmentation image contains binarized values ​​of pixels with semantics of ground and fuselage, which are different from the binarized values ​​of pixels with other semantics, including obstacles and unknown areas.

[0009] Based on the regions occupied by pixels representing other semantic meanings in the binarized image, multiple connected regions are determined;

[0010] Determine whether noisy regions exist within the plurality of connected regions;

[0011] When the noise region exists in the multiple connected regions, the first BEV segmentation image is determined to be long-tailed data and is collected.

[0012] Secondly, embodiments of this application provide a long-tail data acquisition device, comprising:

[0013] The generation module is used to process the environmental image of the current location using a perception model during normal driving to generate the first BEV segmentation image.

[0014] The processing module is used to binarize the first BEV segmentation image to obtain a binarized image. The first BEV segmentation image contains binarized values ​​of pixels whose semantics are ground and fuselage, which are different from the binarized values ​​of pixels with other semantics, including obstacles and unknown areas.

[0015] The determination module is used to determine multiple connected regions based on the regions occupied by pixels of other semantic meanings in the binarized image, and to determine whether there are noise regions in the multiple connected regions.

[0016] The acquisition module is used to determine that the first BEV segmentation image is long-tailed data and acquire it when the noise region exists in the plurality of connected regions.

[0017] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it causes the electronic device to implement the method described in the first aspect or various possible implementations of the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the method described in the first aspect or various possible implementations of the first aspect.

[0019] Fifthly, embodiments of this application provide a computer program product comprising a computing program, wherein when the computer program is executed by a processor, it implements the method described in the first aspect or various possible implementations of the first aspect.

[0020] The long-tail data acquisition method, apparatus, device, and readable storage medium provided in this application involve processing the environmental image of the current position using a perception model during normal robot operation to generate a first BEV segmentation image. Then, the robot binarizes the first BEV segmentation image to obtain a binarized image. Based on the binarized image, multiple connected regions are determined, and it is determined whether noisy regions exist within these regions, i.e., whether obstacle regions are surrounded by passable areas. When noisy regions are present in multiple connected regions, the first BEV segmentation image is collected as long-tail data. This approach, by identifying whether noisy regions exist in the first BEV segmentation image during normal robot operation, collects BEV segmentation images where the perception model mistakenly identifies ground areas as obstacles due to factors such as changes in lighting, ground reflection, and unknown obstacle types. In other words, it collects BEV segmentation images with poor segmentation results during normal operation, improving the coverage of long-tail data and thereby enhancing the generalization and robustness of the perception model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the long-tail data acquisition method provided in the embodiments of this application;

[0023] Figure 2 This is a schematic diagram illustrating the generation process of the first BEV segmentation image in the long-tail data acquisition method provided in this application embodiment;

[0024] Figure 3 This is a schematic diagram of long-tail data in the long-tail data acquisition method provided in the embodiments of this application;

[0025] Figure 4 This is another flowchart of the long-tail data acquisition method provided in the embodiments of this application;

[0026] Figure 5 This is another flowchart of the long-tail data acquisition method provided in the embodiments of this application;

[0027] Figure 6 This is a schematic diagram of a first BEV segmentation image in the long-tail data acquisition method provided in the embodiments of this application;

[0028] Figure 7 A schematic diagram of a long-tail data acquisition device provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] With the rapid development of artificial intelligence, more and more AI-related applications are emerging in the fields of mobile robotics and autonomous driving. In the perception domain, bird's-eye view (BEV) images are receiving increasing application and attention; BEV images are also known as BEV segmentation images. A BEV segmentation image is an image that displays information about the robot's surrounding environment from a top-down perspective. It is generated by using a camera to capture environmental images, inputting these images into a perception model, and the cost is relatively low. Currently, many downstream tasks require BEV segmentation images. For example, BEV segmentation images can be used as ground truth data to supervise the model training process to obtain object detection models. Additionally, BEV segmentation images can be used for environmental perception.

[0031] However, perceptual models do not achieve good accuracy in all scenarios, exhibiting poor generalization and robustness in complex scenarios. Complex scenarios, also known as challenging scenarios, refer to unusual and infrequently occurring situations. To improve the generalization and robustness of perceptual models in challenging scenarios, it is necessary to collect sample data specifically for these scenarios; this type of sample data is also known as long-tail data.

[0032] Currently, the industry uses two main approaches to collect long-tail data. The first approach involves recording data from n seconds before and after a pre-defined event occurs during the robot's movement, thus collecting a large amount of long-tail data for this scenario. The second approach primarily uses artificial intelligence technology to train an evaluation model, which then evaluates the output of the perception model. Data with low evaluation results is then collected and stored as long-tail data.

[0033] However, the first type of approach mentioned above mainly judges whether the robot has encountered abnormalities during its movement from a macroscopic perspective, such as collisions or human intervention, and can only collect long-tail data during abnormal movement. However, during normal robot operation, due to various factors such as changes in lighting, ground reflections, and unknown obstacle types, the BEV segmentation images generated by the perception model may be of poor quality. For example, the BEV segmentation images may show small, discrete noise areas around the robot or in the drivable area of ​​the road. Such BEV segmentation images have a significant impact on downstream tasks such as navigation.

[0034] In addition, during actual operation, the perception model may not accurately predict the spatial distance to obstacles. For example, it may fail to accurately predict the location of pedestrians or misidentify the passable area between adjacent vehicles as an obstacle area.

[0035] Clearly, the first type of solution described above cannot cover the long-tail data that occurs during normal driving. The second type of solution introduces other artificial intelligence models, which increases the system's power consumption to some extent. Furthermore, using two unstable models to collect long-tail data increases the overall instability to some extent.

[0036] Based on this, embodiments of this application provide a long-tail data acquisition method, apparatus, device, and readable storage medium. By determining and collecting whether a first BEV segmentation image during the robot's normal operation is long-tail data, the purpose of collecting long-tail data during the robot's normal operation is achieved, thereby improving the generalization and robustness of the perception model. Furthermore, reliable LiDAR data can be used as a reference value to identify whether the first BEV segmentation image is long-tail data, thus more accurately filtering out long-tail data, thereby improving the robustness and generalization of the perception model to unknown scenarios.

[0037] In the early, unstable stage of the perception model, perceptual error examples are prone to occur frequently. To address the need to collect these difficult examples, this application embodiment deploys a lightweight data backhaul method on the robot, namely the long-tail data acquisition method described in this application embodiment. This method aims to efficiently collect difficult examples with poor recognition performance in difficult scenarios, which is beneficial for promoting rapid iteration of the perception model.

[0038] The long-tail data acquisition method provided in this application is applied to a robot, on which a perception model is deployed. During the robot's movement, environmental images are acquired using surround view images as input to the perception model, enabling the perception model to output a first BEV segmentation image. Subsequently, the robot uses two schemes to determine whether the first BEV segmentation image is long-tail data. If at least one scheme determines that the first BEV segmentation image is long-tail data, then the first BEV segmentation image is collected as long-tail data. The two schemes are BEV cavity state detection and laser BEV radar comparison. Therefore, the long-tail data in this application embodiment is obtained by identifying data with poor inference results from the perception model; data with poor inference results is the first BEV segmentation image of poor quality, i.e., the first BEV segmentation image with poor segmentation effect.

[0039] In this application, long-tail data, also known as difficult examples or corner cases, refers to first BEV segmentation images acquired for difficult scenarios that exhibit poor segmentation results. Poor segmentation results mean that the regions contained in the first BEV segmentation image contain areas that do not conform to the actual physical environment. For example, the first BEV segmentation image may contain an obstacle area, but in the actual physical environment, this area is a passable ground area. Such first BEV segmentation images with poor segmentation results can be acquired through the BEV hole state detection scheme.

[0040] For example, the first BEV segmentation image shows a pedestrian 3 meters directly in front of the vehicle, but in the actual physical environment, the pedestrian is 2 meters directly in front of the vehicle. Another example is that the first BEV segmentation image shows an obstacle zone between the two vehicles, but in the actual physical environment, passage between the two vehicles is possible. In these two examples, the poor segmentation results may be due to inaccurate prediction of the spatial distance to obstacles by the perception model. By using a laser BEV radar comparison scheme, we can acquire BEV segmentation images with poor results due to inaccurate prediction of the spatial distance to obstacles by the perception model.

[0041] After the robot collects long-tail data, it uploads it to servers. During subsequent training of the perception model, this long-tail data is used as ground truth data. Because long-tail data is collected during normal operation, it expands the data coverage to some extent, thus accelerating the learning process of the perception model and effectively improving its performance in various complex scenarios. This achieves the goal of enhancing the generalization and robustness of the BEV perception model in challenging situations.

[0042] The robots described in this application refer to devices capable of performing various tasks unattended, including but not limited to cruise robots, sweeping robots, lawnmower robots, air purifying robots, automatic spraying equipment, care robots, shopping guide robots, shopping mall service robots, and automated guided vehicles (AVGs).

[0043] Figure 1 This is a flowchart of the long-tail data acquisition method provided in this embodiment. The executing entity in this embodiment is a robot, and this embodiment includes:

[0044] 101. During normal driving, the perception model is used to process the environmental image of the current location to generate the first BEV segmentation image.

[0045] In this embodiment, a surround-view camera and a perception model are deployed on the robot. During movement, the surround-view camera acquires multi-view surround-view images. The surround-view camera is, for example, a fisheye camera, and the number of surround-view cameras can be 4, 3, 6, etc., and this embodiment is not limited. The robot acquires surround-view images at its current position using the surround-view camera and uses these images as input to the perception model, enabling the perception model to infer from the surround-view images and output a first BEV segmentation image. Figure 2 This is a schematic diagram illustrating the generation process of the first BEV segmentation image in the long-tail data acquisition method provided in this application embodiment.

[0046] Please refer to Figure 2 The perception model takes a multi-view surround view image as input and outputs a first BEV segmentation image. The regions in the first segmentation image are categorized as unknown, ground, general obstacles, pedestrians, vehicles, pillars, fuselages, etc. The region categories are also known as the region semantics. The perception model is also called the BEV perception model.

[0047] In this embodiment, the first BEV segmentation image is a top-down view centered on the robot. Pixels with the same semantic meaning form a region in this image, and different regions are represented by different colors, each color representing a different semantic meaning. This divides an image into multiple regions, hence the name "first BEV segmentation image." Because the color for each semantic meaning is predefined, the environment around the robot can be intuitively understood through the first BEV segmentation image.

[0048] 102. Determine whether there is a noise region in the first BEV segmentation image. If there is a noise region in the first segmentation image, proceed to step 103; if there is no noise region in the first segmentation image, proceed to step 104.

[0049] After obtaining the first BEV segmentation image using the perception model, the robot determines whether there are noisy regions, i.e., whether there are obstacle regions that meet the conditions, within the multiple regions contained in the first BEV segmentation image. If there are obstacle regions that meet the conditions in multiple regions, step 103 is executed; if there are no obstacle regions that meet the conditions in multiple regions, it is determined that the first BEV segmentation image is not long-tailed data, and there is no need to collect the first BEV segmentation image, but the robot continues to move. After moving to a new position, the robot continues to acquire a new environmental image of the current position and uses it as input to the perception model, and continues to execute the method described in the embodiments of this application.

[0050] A qualifying obstacle region is defined as an area that is an obstacle region and is surrounded by a passable region. If an area is not an obstacle region, it cannot be considered a noise region even if it is surrounded by a passable region.

[0051] In this embodiment, to identify whether there are noisy regions in the first BEV image, the robot binarizes the first BEV segmentation image to obtain a binarized image. Binarization refers to representing the grayscale value of all pixels in the first BEV segmentation image with 0 or 255, thereby making the first BEV segmentation image present a clear visual effect of only black and white. For example, the robot binarizes pixels with the semantics of ground and fuselage to 0, and binarizes pixels with other semantics to 255, such as obstacles and unknown areas.

[0052] Next, the robot determines multiple connected regions based on the areas occupied by pixels representing other semantic elements in the binarized image, and then determines whether noisy regions exist within these connected regions. If noisy regions exist within the multiple connected regions, it indicates that noisy regions exist in the first BEV image, and the robot proceeds to step 103; if no noisy regions do not exist within the multiple connected regions, the robot proceeds to step 104. In the first BEV segmentation image, the binarized values ​​of pixels representing the ground and fuselage are different from the binarized values ​​of pixels representing other semantic elements, including obstacles and unknown regions.

[0053] For example, the robot determines multiple connected regions based on the areas occupied by pixels of other semantic meanings in the binarized image, traverses all connected regions, and filters out noisy regions from the multiple connected regions.

[0054] Figure 3 This is a schematic diagram of long-tail data in the long-tail data acquisition method provided in this application embodiment. Please refer to... Figure 3 The first BEV segmentation image contains multiple regions, such as region 31 (unknown semantics), region 32 (car semantics), region 33 (pillar semantics), region 34 (A semantics), region 35 (ground semantics), and region 38 (fuselage semantics). There is at least one region for each semantic meaning. Region A represents a type of obstacle, such as a wall.

[0055] Please refer to Figure 3 In rectangle 36, a region 32 with the semantic meaning of a vehicle is surrounded by a ground region, and in rectangle 37, a region 31 with an unknown semantic meaning is surrounded by a ground region. Since region 32 in rectangle 36 meets the conditions that region 32 is identified as a vehicle and is surrounded by a passable area, the robot determines that there is a noisy region in the first BEV segmentation image.

[0056] The obstacles in this application embodiment include, but are not limited to, vehicles, walls, pillars, and people. That is to say, regardless of whether an area is identified as a vehicle, wall, pillar, person, or other obstacle, as long as the area is surrounded by a passable area, the area is considered a noise area.

[0057] If the semantics of a region are unknown, such as the unknown region contained in rectangle 37 in the figure, since this unknown region has not been identified as an obstacle, it cannot be considered a noise region even though it is surrounded by a passable region. In other words, if there is a region in the first BEV segmentation image that is surrounded by a passable region, but the surrounded region is an unknown region rather than an obstacle region, then this region cannot be considered a noise region.

[0058] 103. When the noise region exists in multiple connected regions, the first BEV segmentation image is determined to be long-tailed data and collected.

[0059] When the robot determines that there is an obstacle region surrounded by a passable area in the first BE segmentation image, that is, there is a noisy region in the first BEV segmentation image, it will treat the first BEV segmentation image as long-tail data and send it back.

[0060] 104. Control the robot to continue moving to the new position, and then execute step 101.

[0061] The long-tail data acquisition method provided in this application involves using a perception model to process the environmental image of the current position during normal robot operation to generate a first BEV segmentation image. Then, the robot binarizes the first BEV segmentation image to obtain a binarized image. Based on the binarized image, multiple connected regions are determined, and it is determined whether noisy regions exist within these regions, i.e., whether obstacle regions are surrounded by passable areas. When noisy regions are present in multiple connected regions, the first BEV segmentation image is collected as long-tail data. This approach, by identifying the presence of noisy regions in the first BEV segmentation image during normal robot operation, collects BEV segmentation images where the perception model mistakenly identifies ground areas as obstacles due to factors such as changes in lighting, ground reflection, and unknown obstacle types. In other words, it collects BEV segmentation images with poor segmentation results during normal operation, increasing the coverage of long-tail data and thereby improving the generalization and robustness of the perception model.

[0062] In the above embodiments, the presence of noisy regions is used to determine whether the first BEV segmentation image is long-tailed data. This method is also known as BEV hole state detection. In addition to this method, embodiments of this application also utilize LiDAR to acquire high-precision LiDAR point clouds and generate a second BEV segmentation image using the LiDAR point clouds. The second BEV segmentation image is then used to determine whether the first BEV segmentation image is long-tailed data. This method is also known as the LiDAR-BEV comparison scheme.

[0063] Figure 4 This is another flowchart of the long-tail data acquisition method provided in this application embodiment. This embodiment includes:

[0064] 401. During normal driving, the perception model is used to process the environmental image of the current location to generate the first BEV segmentation image.

[0065] For details, please refer to the description in step 101, which will not be repeated here.

[0066] 402. Generate a second BEV segmentation image based on the laser point cloud at the current location.

[0067] In this embodiment, a lidar is installed on the robot. During movement, the robot uses the lidar to collect a lidar point cloud and generates a high-precision second BEV segmentation image based on the lidar point cloud.

[0068] 403. Determine whether the first BEV segmentation image is long-tailed data based on the second BEV segmentation image. If the first BEV segmentation image is long-tailed data, proceed to step 404; if the first BEV segmentation image is not long-tailed data, proceed to step 405.

[0069] During actual robot operation, the perception model may inaccurately predict the spatial distance to obstacles. For example, it may fail to accurately predict pedestrian positions or misidentify passable areas between adjacent vehicles as obstacle areas. The second BEV segmentation image, however, is a high-precision BEV segmentation image. Therefore, the robot compares the first and second BEV segmentation images to determine whether the first BEV segmentation image contains long-tailed data. For instance, the robot identifies semantically different pixels from the first and second BEV segmentation images; if the number of semantically different pixels exceeds a threshold, the first BEV segmentation image is determined to contain long-tailed data.

[0070] For example, the robot determines the similarity between the first BEV segmentation image and the second BEV segmentation image. When the similarity is less than the preset similarity, the first BEV segmentation image is considered to be long-tailed data, that is, the first BEV segmentation image output by the perception model is of poor quality; when the similarity is greater than or equal to the preset similarity, the first BEV segmentation image is considered not to be long-tailed data.

[0071] 404. Acquire the first BEV segmentation image.

[0072] When the first BEV segmentation image is long-tailed data, the robot collects the first BEV segmentation image.

[0073] 405. Continue moving forward to the new current position. Then, execute steps 401 and 402.

[0074] This approach uses reliable LiDAR data as a reference to identify whether the first BEV segmentation image is long-tailed data. This allows for the collection of BEV segmentation images with poor segmentation results due to inaccurate prediction of spatial distances to obstacles by the perception model. This increases the coverage of long-tailed data and ultimately improves the generalization and robustness of the perception model.

[0075] In this embodiment of the application, after the robot acquires the first BEV segmentation image at its current location, it can execute... Figure 1 and / or Figure 4 The scheme shown determines whether the first BEV segmented image is long-tailed data. Figure 1 or Figure 4 If any scheme identifies the first BEV segmentation image as long-tailed data, then the first BEV segmentation image is treated as long-tailed data and collected. For example, please refer to... Figure 5 .

[0076] Figure 5 This is another flowchart of the long-tail data acquisition method provided in this application embodiment. This embodiment includes:

[0077] 501. Obtain the first BEV segmentation image.

[0078] The first BEV segmentation image, also known as the predicted BEV segmentation image, is obtained as described above. Figure 1 The description of step 101 will not be repeated here. Afterwards, the robot executes a LiDAR comparison scheme and / or a BEV hole detection scheme to determine whether the first BEV segmentation image is long-tailed data. The BEV hole detection scheme mainly targets the specific scenario of noise by collecting a large amount of data, achieving the goal of effectively collecting long-tailed data during normal operation. The LiDAR comparison scheme mainly addresses the problem of poor spatial distance perception in the perception model.

[0079] Therefore, the embodiments of this application provide a lightweight long-tail data backhaul method suitable for deployment on robots. By using a lidar comparison scheme and a BEV hole detection scheme, it can specifically collect difficult samples with poor perception performance of the perception model, thereby accelerating the automatic collection of long-tail data in difficult scenarios and reducing the time cost of manual collection.

[0080] The lidar comparison scheme includes the following steps:

[0081] 502. Use lidar to collect laser point clouds.

[0082] In this step, the robot uses a lidar to collect a laser point cloud at its current location.

[0083] 503. Obtain the odometer information for the current location.

[0084] For example, the robot is equipped with an odometer, wheel speed meter, etc., so it can obtain odometer information.

[0085] 504. Laser point cloud distortion correction.

[0086] In this embodiment, since the LiDAR moves with the robot, there may be distorted 3D points in the LiDAR point cloud. Distortion removal refers to deleting distorted 3D points from the LiDAR point cloud. If distortion removal is not considered, the robot generates a second BEV segmentation image based on the LiDAR point cloud and uses the high-precision second BEV segmentation image to determine whether the first BEV segmentation image is long-tailed data.

[0087] If distortion correction is considered, the input to the LiDAR comparison scheme includes a first BEV segmentation image, a laser point cloud, and odometry information. The output is a poorly segmented first BEV image, i.e., long-tailed data. The odometry information includes the robot's pose, etc. The purpose of the odometry information is to correct the distortion of the laser point cloud. After the robot acquires the odometry information for its current position, it corrects the distortion of the laser point cloud based on this information, and then generates the second BEV segmentation image based on the corrected laser point cloud.

[0088] Using this approach, the robot corrects the distortion of the laser point cloud based on odometry information, and then generates a second BEV segmentation image based on the corrected laser point cloud, thereby improving the quality of the second BEV segmentation image and accurately determining whether the first BEV segmentation image is long-tailed data.

[0089] Optionally, during the distortion correction process of the laser point cloud based on the odometry information, the robot determines the projection position of the lidar in the world coordinate system and the transformation amount of the lidar projection onto the body coordinate system based on the odometry information. Then, for 3D points in the laser point cloud, the robot corrects the distortion of the 3D points based on the projection position and the transformation amount.

[0090] For example, during robot initialization, a fixed-size 2D BEV segmentation image is created in the body coordinate system. The length and width of this 2D BEV segmentation image are represented by size_x and size_y, respectively. The distance represented by each pixel in the real-world coordinate system is represented by resolution. The robot body is located at the exact center of the 2D BEV segmentation image. Each pixel in the 2D BEV segmentation image has a corresponding point in the real physical world. This physical world corresponding point is back-projected into camera space using camera intrinsic and extrinsic parameters to obtain a projection point. If this projection point can be correctly back-projected into image space, the corresponding point in the physical world is initialized as a ground element; otherwise, the corresponding point in both the physical and world worlds is initialized as the robot body. For the robot, the area where the robot body is located is an invisible region.

[0091] The 2D BEV segmentation map is an initial BEV segmentation map, mainly used to segment the fuselage region. When the robot processes multi-view environmental images using its perception model, it generates the first BEV segmentation image based on this initial BEV segmentation map and the inference results.

[0092] The robot subscribes to and parses topic nodes in the LiDAR ( / lidar) and wheel speed odometry ( / odom) modules of the operating system in real time. The LiDAR point cloud data format is x, y, z, t, where x, y, and z represent the coordinates of the point cloud, and t represents the timestamp of each 3D point in that frame. The odometry data format is x, y, z, and quaternions, where w_x, w_y, w_z, and w_w are the quaternions. Here, x, y, and z represent the robot's coordinates. Since the LiDAR point cloud moves with the robot, odometry information is needed to correct distortion in the LiDAR point cloud. Assume the maximum scan time (scan_time) for one LiDAR scan is 0.1 seconds, and the LiDAR's extrinsic matrix (lidar_extrinsics) is known. Assuming the timestamp of a point cloud frame is tm, the timestamp of the LiDAR scan completion is tm + scan_time. By indexing the odometry information according to the timestamps, the initial pose start_pose corresponding to tm and the ending pose end_pose corresponding to tm + scan_time can be obtained. Based on these parameters, the robot obtains the pose change of the LiDAR in the world coordinate system corresponding to the start frame, as well as the displacement v_t and rotation change v_r corresponding to each small time interval within scan_time. The start frame refers to a frame of laser point cloud acquired at the beginning of one scan cycle by the LiDAR. In this process, the robot determines the initial and ending poses of the LiDAR when acquiring the laser point cloud based on the odometry information. Based on the initial pose, the ending pose, and the extrinsic parameter matrix of the LiDAR, the robot obtains the displacement v_t and rotation change v_r corresponding to each small time interval within scan_time. The following formulas are involved:

[0093] diff = start_pose -1 ×end_pose formula (1)

[0094] diff_lidar = lidar_extrinsice -1 ×diff×lidar_extrinsics formula (2)

[0095] v_t = T diff_lidar / scan_time formula(3)

[0096] v_r = R diff_lidar / scan_time formula (4)

[0097] Where, diff represents the pose change of the aircraft body, diff_lidar represents the pose change of the lidar, and T diff_lidar R is the translation vector representing the transformation amount of the lidar projection onto the fuselage coordinate system. diff_lidarThe rotation vector represents the transformation amount of the lidar projection onto the fuselage coordinate system.

[0098] After acquiring the displacement v_t and rotational change v_r corresponding to each tiny time interval within scan_time, the robot determines the transformation amount of the lidar projection onto the robot's coordinate system based on the displacement v_t and rotational change v_r. Simultaneously, the robot determines the projected position of the lidar in the world coordinate system based on the initial pose and the lidar's extrinsic parameter matrix. Then, for each 3D point in the lidar point cloud, distortion is corrected based on the projection position and transformation amount. Assuming the 3D point in the lidar point cloud is pt, the distortion correction process involves the following formula:

[0099] dtm = tm - f(tm) Formula (5)

[0100] start_lidar2world = start_pose × lidar_extrinsics formula (6)

[0101] Pose r = exp(v r ×dt) Formula (7)

[0102] Pose t = v t ×dt formula (8)

[0103] pt =start_lidar2world× Pose ×pt Formula (9)

[0104] bev_pt = start_pose -1 ×pt formula(10)

[0105] Where f(tm) represents the rounding operation on the timestamp, exp(*) is the exponential function, and tm is the timestamp of the start frame. start_lidar2world represents the projected position of the lidar in the world coordinate system. Pose r Pose represents the amount of displacement per unit time. t It represents the amount of rotational transformation per unit time.

[0106] In formula (9), pt on the left side of the equal sign represents the 3D point after distortion removal, and pt on the right side of the equal sign represents the 3D point before distortion removal. Pose in formula (9) represents the 3D point based on Pose... r and Pose t The determined transformation amount. In formula (10), bev_pt represents the point cloud representation of the distorted 3D points projected into the BEV space, and pt in formula (10) represents the distorted 3D points.

[0107] Using this approach, the robot determines the projection position of the lidar in the world coordinate system based on the odometry information and determines the transformation amount. It then performs distortion correction on the 3D points based on the projection position and transformation amount, thereby improving the quality of the lidar point cloud. Finally, it generates a second BEV segmentation image based on the distortion-corrected lidar point cloud, thus improving the quality of the second BEV segmentation image.

[0108] 505. Generate the second BEV segmentation image.

[0109] In this step, the robot generates the second BEV segmentation image based on the distortion-free laser point cloud.

[0110] Optionally, in the above embodiments, during the process of generating the second BEV segmentation image from the distortion-corrected laser point cloud, the robot assigns semantic information to the 3D points in the distortion-corrected laser point cloud to generate a preliminary BEV segmentation image. Then, the robot performs occlusion removal on the preliminary BEV segmentation image to modify the semantics of pixels with semantic errors in the BEV segmentation image, and uses the occlusion-removed preliminary BEV segmentation image as the second BEV segmentation image.

[0111] For example, the robot projects each distorted 3D point onto a two-dimensional BEV image to obtain projection points, and assigns semantic meaning to the projection points based on their z-axis height. For instance, if the z-axis height of a 3D point is greater than a preset height min_d, the projection point is assigned the semantic meaning of "obstacle," which can further identify whether the obstacle is a pillar, pedestrian, etc., thus assigning a specific semantic meaning to the projection point, which becomes a pixel in the second BEV image. If the z-axis height of a 3D point is less than or equal to the preset height min_d, the projection point is assigned the semantic meaning of "ground." After projecting each 3D point in the distorted laser point cloud, a preliminary BEV segmentation image is obtained.

[0112] Because the lidar is installed higher than the surrounding vision sensor, it can observe objects that the vision sensor cannot, often objects that appear occluded from the vision sensor's perspective. However, the preliminary BEV segmentation image obtained from the distortion-corrected lidar point cloud contains these occluded objects, which is unreasonable. In other words, the preliminary BEV segmentation image contains some semantically incorrect pixels. Therefore, it is necessary to remove these supposedly occluded objects and correct the semantics of the semantically incorrect pixels in the BEV segmentation image. After occlusion removal, the preliminary BEV segmentation image no longer displays the objects that should be occluded, thus making the observation range of the lidar and the observation range of the vision sensor approximately the same. The vision sensor can be, for example, a surround-view camera, and this embodiment is not limited to this.

[0113] This approach improves the quality of the second BEV segmentation image by eliminating occlusion in the distortion-reduced laser point cloud, thereby modifying the semantics of pixels with semantic errors in the initial BEV segmentation image and ultimately enhancing the quality of long-tail data collection.

[0114] Optionally, in the above embodiments, during the occlusion removal process of the robot on the preliminary BEV segmentation image, for edge pixels of the preliminary BEV segmentation image, a ray passing through the edge pixel is determined, and the starting point of the ray is the center point of the preliminary BEV segmentation image. Then, starting from the center point, when the semantic priority of the current pixel on the ray is higher than the semantic priority of the next pixel, the obstacle pixel closest to the next pixel is determined from the pixels included in the ray, and the semantics of the pixels between the next pixel and the obstacle pixel are modified to unknown regions.

[0115] For example, the robot first traverses the edges of the initial BEV segmentation map in a clockwise or counterclockwise direction to obtain edge pixels. For each edge pixel, a ray extends from the center of the initial BEV segmentation map towards the edge pixel, resulting in multiple rays. For each ray, the robot traverses from the center point to the edge pixel and modifies the semantics of the next pixel according to preset rules. These preset rules include, but are not limited to:

[0116] A. Each semantic category is prioritized in the following order: unknown area, ground, obstacles. Among these, the unknown area has the lowest priority, and obstacles have the highest priority.

[0117] B. During the traversal, if the semantic priority of the current pixel is higher than that of the next pixel, the obstacle pixel closest to the next pixel is determined from the ray; the semantics of the pixels between the next pixel and the obstacle pixel are modified to unknown regions.

[0118] For example, the semantics of the current pixel might be an obstacle, while the semantics of the next pixel might be the ground. Clearly, the ground corresponding to the next pixel is occluded by the obstacle and shouldn't be visible. Therefore, the semantics of the next pixel are modified to "unknown region." Then, the robot determines the nearest obstacle pixel to the current pixel, following the direction from the center point to the edge pixels. An obstacle pixel is defined as a pixel whose semantics are "obstacle." Then, regardless of the semantics of the pixels between the current pixel and the obstacle pixel, they are all modified to "unknown region." For example, if a pixel's semantics are "ground," it is modified to "unknown region." Conversely, if a pixel's semantics are "unknown region," the pixel's semantics remain unchanged.

[0119] After traversing each ray, the occlusion removal process is equivalent to completing the semantic correction of semantically incorrect pixels in the initial BEV segmentation image. The robot then uses the occlusion-removed initial BEV segmentation image as the second BEV segmentation image.

[0120] Using this approach, the robot starts from the center point and modifies the semantics of the pixels along the direction from the center point to the edge pixels, thereby identifying and correcting semantically erroneous pixels as much as possible, resulting in high accuracy and speed.

[0121] 506. Truth value comparison.

[0122] In this step, the robot compares the first BEV segmentation image with the high-precision second BEV segmentation image to determine whether the first BEV segmentation image is long-tailed data.

[0123] 507. Determine whether the number of target pixels in the second BEV segmentation image is greater than a preset number. If the number of target pixels is greater than the preset number, proceed to step 508; if the number of target pixels is less than or equal to the preset number, proceed to step 509. Target pixels refer to pixels in the second BEV segmentation image whose semantics are ground regions, and the corresponding pixels in the first BEV segmentation image whose semantics are obstacle regions.

[0124] 508. Determine that the first BEV segmentation image is long-tailed data and collect it.

[0125] 509. It is determined that the first BEV segmentation image is not long-tailed data and does not need to be collected.

[0126] For example, the robot compares the first BEV segmentation image and the second BEV segmentation image, traversing the ground region in the second BEV segmentation image. That is, for each pixel in the second BEV segmentation image whose semantics are ground region, it determines whether the corresponding pixel in the first BEV segmentation image also has a ground region semantics. If both pixels have ground region semantics, the robot continues traversing to the next pixel. If a pixel in the second BEV segmentation image has a ground region semantics, but the corresponding pixel in the first BEV segmentation image has an obstacle semantics, such as a general obstacle, pedestrian, vehicle, or pillar, then this pixel is designated as a target pixel and its count is accumulated. The accumulated count of target pixels can be represented, for example, as diff_obstacle. In other words, for each target pixel identified by the robot, diff_obstacle = diff_obstacle + 1.

[0127] After traversal, the robot compares the number of target pixels with a preset number min_obstacl. If the number of target pixels is greater than the preset number min_obstacl, the first BEV segmented image is determined to be long-tailed data. If the number of target pixels is less than or equal to the preset number min_obstacl, the first BEV segmented image is determined to have good segmentation performance and is not long-tailed data.

[0128] Figure 6 This is a schematic diagram of a first BEV segmentation image in the long-tail data acquisition method provided in this application embodiment. Please refer to... Figure 6 The first BEV segmentation image contains multiple regions, such as the fuselage region 61, the ground region 62, the unknown region 63, the obstacle region 64, and the obstacle region 65. After obtaining the second BEV segmentation image, the robot identifies the continuous ground region. Then, for each pixel in the ground region, the semantics of the corresponding pixel in the first BEV segmentation image are determined. After traversing every pixel in the ground region, it is found that each pixel in the obstacle region 66 is an obstacle in the first BEV segmentation image, but semantically represents the ground in the second BEV segmentation image. Therefore, the robot counts the number of pixels in the obstacle region 65, i.e., the number of target pixels. Since the number of target pixels is greater than a preset number, this first BEV segmentation image is treated as long-tail data.

[0129] It is understandable that target pixels may be scattered or form multiple contiguous regions. When calculating the number of target pixels, one can either accumulate the number for each contiguous region separately, or count the total number of target pixels across all contiguous regions.

[0130] This approach determines whether a first BEV segmentation image is long-tailed data based on the number of target pixels, offering both speed and high accuracy.

[0131] The BEV cavity detection solution includes the following steps:

[0132] 510. Binarize the first BEV segmented image to obtain a binarized image.

[0133] In this step, the robot binarizes the first BEV segmentation image. The binarized values ​​of pixels with semantic meanings of ground and fuselage in the first BEV segmentation image are different from the binarized values ​​of pixels with other semantic meanings, including obstacles and unknown areas. For example, the robot binarizes pixels with semantic meanings of ground and fuselage as 0, and binarizes pixels with other semantic meanings as 255, including obstacles and unknown areas.

[0134] 511. Calculate connected components.

[0135] The robot determines multiple connected regions based on the areas occupied by pixels representing other semantic meanings in the binarized image. For example, the robot uses OpenCV functions to find connected components of pixels representing other semantic meanings in the BEV image space, such as connected components formed by pixels representing obstacles or pixels representing unknown regions. This function can also obtain the corner coordinates of connected components. An example function is the `cv::connected Components With Stats` function from the OpenCV library.

[0136] 512. Connected component filtering.

[0137] In this step, the robot traverses all connected components, filtering them using the four corner points of each component. For example, connected components whose corner points extend beyond the image boundaries are filtered out, ultimately obtaining connected components surrounded by the ground area, such as... Figure 3 Connected components in rectangles 36 and 37.

[0138] 513. Whether the semantic category judgment and area judgment are passed. If the judgment is passed, proceed to step 508; if the judgment is not passed, proceed to step 509.

[0139] For example, the robot performs semantic category determination on the remaining connected components and compares the area of ​​the connected components with a preset area. If the semantics of the connected component are not obstacles such as cars or pillars, but rather unknown regions, they are filtered out. Figure 3 The rectangle 37 in the image is then used. Next, for the remaining connected components, it is determined whether their areas exceed a preset area. If at least one connected component's semantics are obstacles and its area exceeds the preset area, then the first BEV segmentation image is considered long-tail data. If a connected component's semantics are an unknown region, and / or its area is less than or equal to a preset area, then the first BEV segmentation image is considered to have good segmentation performance and does not belong to long-tail data.

[0140] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0141] Figure 7 This is a schematic diagram of a long-tail data acquisition device provided in an embodiment of this application. The long-tail data acquisition device 700 includes: a generation module 71, a processing module 72, and an acquisition module 73.

[0142] The generation module 71 is used to process the environmental image of the current position using a perception model during normal driving to generate a first BEV segmentation image;

[0143] Processing module 72 is used to binarize the first BEV segmentation image to obtain a binarized image. The first BEV segmentation image contains binarized values ​​of pixels whose semantics are ground and fuselage, which are different from the binarized values ​​of pixels with other semantics, including obstacles and unknown areas.

[0144] The determination module 73 is used to determine multiple connected regions based on the regions occupied by pixels of other semantic meanings in the binarized image, and to determine whether there are noise regions in the multiple connected regions.

[0145] The acquisition module 74 is used to determine that the first BEV segmentation image is long-tailed data and acquire it when the noise region exists in the plurality of connected regions.

[0146] In one feasible implementation, the processing module 72 is further configured to generate a second BEV segmentation image based on the laser point cloud at the current location; determine whether the first BEV segmentation image is long-tailed data based on the second BEV segmentation image; and acquire the first BEV segmentation image when the first BEV segmentation image is long-tailed data.

[0147] In one feasible implementation, when the processing module 72 generates a second BEV segmentation image based on the laser point cloud at the current location, it is used to obtain odometer information at the current location; to distort the laser point cloud based on the odometer information; and to generate the second BEV segmentation image based on the distorted laser point cloud.

[0148] In one feasible implementation, when the processing module 72 performs distortion correction on the laser point cloud based on the odometer information, it is used to determine the projection position of the lidar in the world coordinate system based on the odometer information; determine the transformation amount of the lidar projection onto the fuselage coordinate system; and for 3D points in the laser point cloud, perform distortion correction on the 3D points based on the projection position and the transformation amount.

[0149] In one feasible implementation, when the processing module 72 generates the second BEV segmentation image based on the distortion-corrected laser point cloud, it assigns semantic information to the 3D points in the distortion-corrected laser point cloud to generate a preliminary BEV segmentation image; it performs occlusion removal on the preliminary BEV segmentation image to modify the semantics of pixels with semantic errors in the preliminary BEV segmentation image, and uses the occlusion-removed preliminary BEV segmentation image as the second BEV segmentation image.

[0150] In one feasible implementation, when the processing module 72 performs occlusion removal on the preliminary BEV segmentation image to modify the semantics of pixels with semantic errors in the preliminary BEV segmentation image, it is used to determine, for edge pixels of the preliminary BEV segmentation image, a ray passing through the edge pixel, the starting point of the ray being the center point of the preliminary BEV segmentation image; starting from the center point, when the semantic priority of the current pixel on the ray is higher than the semantic priority of the next pixel, the obstacle pixel closest to the current pixel is determined from the pixels included in the ray; the semantics of the pixels between the current pixel and the obstacle pixel are modified to an unknown region.

[0151] In one feasible implementation, when the processing module 72 determines whether the first BEV segmentation image is long-tailed data based on the second BEV segmentation image, it is used to determine the number of target pixels in the second BEV segmentation image. The target pixels refer to pixels in the second BEV segmentation image whose semantics are ground areas, and the corresponding pixels in the first BEV segmentation image whose semantics are obstacle areas. When the number of target pixels is greater than a preset number, the first BEV segmentation image is determined to be long-tailed data.

[0152] The long-tail data acquisition device provided in this application embodiment can perform the actions of the robot in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0153] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be, for example, the robot described above. Please refer to... Figure 8 The electronic device 800 described in this application embodiment includes: at least one processor 81, at least one communication bus 82, a user interface 83, at least one network interface 84, and a memory 85.

[0154] The communication bus 82 is used to enable communication between these components.

[0155] The user interface 83 may include a display screen and a camera. Optionally, the user interface 83 may also include a standard wired interface and a wireless interface.

[0156] Among them, the network interface 84 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0157] The processor 81 may include one or more processing cores. The processor 81 connects to various parts within the electronic device 800 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 85, and by calling data stored in the memory 85. Optionally, the processor 81 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 81 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the panoramic sphere required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 81 and may be implemented as a separate chip.

[0158] The memory 85 may include random access memory (RAM) or read-only memory. Optionally, the memory 85 may include a non-transitory computer-readable storage medium. The memory 85 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 85 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 85 may also be at least one storage device located remotely from the aforementioned processor 81. Figure 8 As shown, the memory 85, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and operating applications for electronic devices.

[0159] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the long-tail data acquisition method described above.

[0160] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the long-tail data acquisition method described above.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0166] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0167] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disks or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0168] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0169] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A long-tail data acquisition method, characterized in that, Applied to robots, the method includes: During normal driving, the perception model is used to process the environmental image of the current location to generate the first BEV segmentation image; The first BEV segmentation image is binarized to obtain a binarized image. The first BEV segmentation image contains binarized values ​​of pixels with semantics of ground and fuselage, which are different from the binarized values ​​of pixels with other semantics, including obstacles and unknown areas. Based on the regions occupied by pixels representing other semantic meanings in the binarized image, multiple connected regions are determined; Determine whether noisy regions exist within the plurality of connected regions; When the noise region exists in the multiple connected regions, the first BEV segmentation image is determined to be long-tailed data and is collected.

2. The method according to claim 1, characterized in that, Also includes: A second BEV segmentation image is generated based on the laser point cloud at the current location; Determine whether the first BEV segmentation image is long-tailed data based on the second BEV segmentation image; When the first BEV segmentation image is long-tailed data, the first BEV segmentation image is acquired.

3. The method according to claim 2, characterized in that, The step of generating a second BEV segmentation image based on the laser point cloud at the current location includes: Obtain the odometer information for the current location; The laser point cloud is distorted based on the odometer information; The second BEV segmentation image is generated based on the distortion-free laser point cloud.

4. The method according to claim 3, characterized in that, The distortion correction of the laser point cloud based on the odometer information includes: The projection position of the lidar in the world coordinate system is determined based on the odometer information. Determine the transformation amount of the lidar projection onto the fuselage coordinate system; For the 3D points in the laser point cloud, the distortion of the 3D points is removed according to the projection position and the transformation amount.

5. The method according to claim 3, characterized in that, The step of generating the second BEV segmentation image based on the distortion-corrected laser point cloud includes: Semantic information is assigned to the 3D points in the distortion-free laser point cloud to generate a preliminary BEV segmentation image; Occlusion removal is performed on the preliminary BEV segmentation image to modify the semantics of pixels with semantic errors in the preliminary BEV segmentation image, and the preliminary BEV segmentation image after occlusion removal is used as the second BEV segmentation image.

6. The method according to claim 5, characterized in that, The step of performing occlusion removal on the preliminary BEV segmentation image to modify the semantics of pixels with semantic errors in the preliminary BEV segmentation image includes: For the edge pixels of the preliminary BEV segmentation image, determine the ray passing through the edge pixels, and the starting point of the ray is the center point of the preliminary BEV segmentation image; Starting from the center point, when the semantic priority of the current pixel on the ray is higher than the semantic priority of the next pixel, the obstacle pixel closest to the current pixel is determined from the pixels contained in the ray; Modify the semantics of the pixels between the current pixel and the obstacle pixel to an unknown region.

7. The method according to claim 2, characterized in that, The step of determining whether the first BEV segmentation image is long-tailed data based on the second BEV segmentation image includes: The number of target pixels in the second BEV segmentation image is determined. The target pixels refer to pixels in the second BEV segmentation image whose semantics are ground regions, and the corresponding pixels in the first BEV segmentation image whose semantics are obstacle regions. When the number of target pixels is greater than a preset number, the first BEV segmentation image is determined to be long-tailed data.

8. A long-tail data acquisition device, characterized in that, include: The generation module is used to process the environmental image of the current location using a perception model during normal driving to generate the first BEV segmentation image. The processing module is used to binarize the first BEV segmentation image to obtain a binarized image. The first BEV segmentation image contains binarized values ​​of pixels whose semantics are ground and fuselage, which are different from the binarized values ​​of pixels with other semantics, including obstacles and unknown areas. The determination module is used to determine multiple connected regions based on the regions occupied by pixels of other semantic meanings in the binarized image, and to determine whether there are noise regions in the multiple connected regions. The acquisition module is used to determine that the first BEV segmentation image is long-tailed data and acquire it when the noise region exists in the plurality of connected regions.

9. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images