Bird-eye view generation method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of computer vision, and discloses a bird's-eye view generation method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining initial point cloud data in the movement process of a target robot, carrying out the preprocessing, obtaining target point cloud data, obtaining an initial bird's-eye view through projection, and obtaining a bird's-eye view; pixels in the initial aerial view are endowed with initial semantics. For a first pixel point in the initial aerial view, determining a priority comparison result based on a size relationship between the highest priority of the initial semantics of all second pixel points on the first connecting line and the priority of the initial semantics of the first pixel point, and determining an adjustment result of the initial semantics of a target pixel point based on the priority comparison result, and obtaining the target aerial view. According to the method, the high-precision aerial view can be generated by acquiring the data in the movement process of the target robot, and the accuracy of the generated aerial view is improved by setting the priorities to perform semantic adjustment on the pixel points in the initial aerial view.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to methods, apparatus, devices, storage media, and program products for generating bird's-eye views. Background Technology

[0002] The development of autonomous driving and mobile robots relies heavily on advanced perception and navigation technologies. In recent years, visual BEV (Bird's Eye View) has been widely researched and applied. Visual BEV technology is a widely used sensor data representation method in the field of autonomous driving. It utilizes camera images around the vehicle or robot and converts them into a bird's-eye view, providing richer perceptual information. With the rapid development of artificial intelligence, more and more visual BEV perception networks have emerged recently. However, few studies focus on how to quickly and efficiently generate ground truth BEV images in real-world environments for training and iterating visual BEV perception networks. Current visual BEV solutions may suffer from point cloud occlusion issues, and robot tilting during movement can lead to inaccurate point cloud semantics, affecting the accuracy of BEV generation and consequently impacting the normal navigation of the vehicle or robot. Summary of the Invention

[0003] In view of this, the present invention provides a method, apparatus, device, storage medium and program product for generating bird's-eye view, so as to solve the problem of insufficient accuracy in generating bird's-eye view.

[0004] In a first aspect, the present invention provides a method for generating a bird's-eye view, the method comprising:

[0005] Acquire initial point cloud data during the target robot's movement;

[0006] The initial point cloud data is preprocessed to obtain the target point cloud data;

[0007] The target point cloud data is projected to obtain an initial bird's-eye view, and the pixels in the initial bird's-eye view have initial semantics and the priority of the initial semantics;

[0008] For the first pixel in the initial bird's-eye view, the priority comparison result is determined based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel. The first line is used to characterize the connection between the center pixel in the initial bird's-eye view and the first pixel. The first pixel and the second pixels are on the first line.

[0009] Based on the priority comparison result, the initial semantic adjustment result of the target pixel is determined to obtain a target bird's-eye view, wherein the target pixel is any of the first pixel.

[0010] In one optional implementation, determining the initial semantic adjustment result of the target pixel based on the priority comparison result to obtain a target bird's-eye view includes:

[0011] If the initial semantic priority of the target pixel is lower than the highest priority, the initial semantics of the target pixel is adjusted to a preset target semantics to obtain the target bird's-eye view. In an optional embodiment, the preprocessing of the initial point cloud data to obtain target point cloud data includes:

[0012] Acquire pose data of the target robot during its motion process, wherein the pose data includes the correspondence between pose and time.

[0013] Based on the pose data, the initial point cloud data at the corresponding time is subjected to distortion correction processing to obtain the target point cloud data.

[0014] In one optional implementation, the step of performing distortion correction processing on the initial point cloud data at the corresponding time based on the pose data to obtain the target point cloud data includes:

[0015] Obtain the start scanning time, scan completion time, and external parameter matrix of the scanning device on the target robot;

[0016] Index the pose data to determine the start pose corresponding to the start scanning time and the end pose corresponding to the scan completion time.

[0017] Based on the start pose, end pose, and extrinsic parameter matrix, determine the point cloud displacement and angle change within each preset time period;

[0018] Based on the point cloud displacement, the angle change, and the extrinsic parameter matrix, calculate the rotation and translation variables of the initial point cloud data;

[0019] The target point cloud data is determined based on the rotation and translation variables.

[0020] In one optional implementation, projecting the target point cloud data to obtain an initial bird's-eye view includes:

[0021] The pose data is filtered to identify abnormal poses, and the abnormal poses are corrected to determine the target pose.

[0022] Projecting the target point cloud data based on the target pose yields a first initial bird's-eye view.

[0023] The initial semantic values ​​of the pixels in the first initial bird's-eye view are assigned based on the target point cloud data and a preset height threshold to obtain the initial bird's-eye view. The preset height threshold corresponds to the obstacle type.

[0024] In one optional implementation, the step of assigning initial semantic values ​​to pixels in the first initial bird's-eye view based on target point cloud data and a preset height threshold to obtain the initial bird's-eye view includes:

[0025] Based on the preset height threshold and the height information of the pixels in the first initial bird's-eye view, the pixels in the first initial bird's-eye view are classified to obtain the classification result;

[0026] Based on the classification results, semantic values ​​are assigned to the pixels in the first initial bird's-eye view to determine the initial bird's-eye view.

[0027] Secondly, the present invention provides an apparatus for generating a bird's-eye view, the apparatus comprising:

[0028] The initial point cloud acquisition module is used to acquire initial point cloud data during the target robot's motion process;

[0029] The point cloud preprocessing module is used to preprocess the initial point cloud data to obtain the target point cloud data;

[0030] The data projection module is used to project the target point cloud data to obtain an initial bird's-eye view, wherein the pixels in the initial bird's-eye view have initial semantics.

[0031] The priority comparison module is used to determine the priority comparison result for the first pixel in the initial bird's-eye view based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel. The first line is used to characterize the connection between the center pixel in the initial bird's-eye view and the first pixel. The first pixel and the second pixels are on the first line.

[0032] An image generation module is used to determine the initial semantic adjustment result of the target pixel based on the priority comparison result, so as to obtain a target bird's-eye view, wherein the target pixel is any of the first pixel.

[0033] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0034] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0035] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0036] The method for generating a bird's-eye view provided in this embodiment acquires initial point cloud data during the movement of a target robot, preprocesses the initial point cloud data to obtain target point cloud data, projects it to obtain an initial bird's-eye view, and assigns initial semantics to the pixels in the initial bird's-eye view, with the initial semantics having priorities. For the first pixel in the initial bird's-eye view, the priority comparison result is determined based on the relationship between the highest priority of the initial semantics of all second pixels on the first connecting line and the priority of the initial semantics of the first pixel. The first connecting line is used to represent the connection between the center pixel and the first pixel in the initial bird's-eye view. Based on the priority comparison result, the adjustment result of the initial semantics of the target pixel is determined to obtain the target bird's-eye view. This method can generate a high-precision bird's-eye view by acquiring data during the movement of the target robot, and improve the accuracy of the generated bird's-eye view by setting priorities to adjust the semantics of the pixels in the initial bird's-eye view. This solution can be implemented using only appropriate data acquisition equipment and low-beam scanning equipment, and the cost is low. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a method for generating a bird's-eye view according to an embodiment of the present invention;

[0039] Figure 2 This is a structural block diagram of a bird's-eye view generation apparatus according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Visual BEV solutions utilize camera images around the vehicle or robot, converting them into a bird's-eye view, providing richer perceptual information. Simultaneously, the bird's-eye view has a consistent scale, allowing for more accurate estimation and measurement of obstacle size and distance. In contrast, traditional perspective projection transformation images are affected by perspective effects, and obstacle perception is influenced by the perspective of objects appearing larger when closer and smaller when farther away, often resulting in inaccurate distance estimation for distant obstacles. Furthermore, pure visual BEV solutions are less expensive than LiDAR solutions, and visual solutions provide richer perceptual information, exhibiting better perception of irregular obstacles. Existing technologies are mostly based on high-cost, high-beam LiDAR, which is costly and unsuitable for large-scale data acquisition scenarios. Therefore, this invention provides a method for generating a bird's-eye view, applicable to desktop or mobile terminals, where the mobile terminal can be at least one of a mobile phone, tablet, or laptop.

[0043] According to an embodiment of the present invention, a method for generating a bird's-eye view is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] This embodiment provides a method for generating a bird's-eye view. Figure 1 This is a flowchart of a method for generating a bird's-eye view according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0045] Step S101: Obtain initial point cloud data during the target robot's movement.

[0046] The target robot is a mobile robot with self-movement capabilities, applicable to fields such as autonomous driving and smart homes. This method can be applied to the mobile robot itself or to a cloud platform for remotely controlling the mobile robot; there are no specific limitations. The target robot is equipped with scanning equipment, such as a camera or LiDAR. During the robot's movement, the scanning equipment scans and photographs its surrounding environment, obtaining initial point cloud data collected by the robot over a period of time. The initial point cloud data includes x, y, z, and t information, where x, y, and z represent the coordinates of the point cloud, and t represents the timestamp of each 3D point in the point cloud.

[0047] When the target robot is equipped with the ROS2 operating system, taking LiDAR as the scanning device as an example, the initial point cloud data can be obtained by calling the LiDAR topic node / lidar in the ROS2 operating system.

[0048] On the other hand, wheel speed odometers can also be installed on the target robot to record the robot's position information during movement.

[0049] Step S102: Preprocess the initial point cloud data to obtain the target point cloud data.

[0050] The initial point cloud data may contain noise, redundant points, and incomplete regions. Data collected during the target robot's movement may be distorted due to motion, affecting the accuracy and efficiency of subsequent processing. Therefore, this step requires preprocessing the initial point cloud data. Preprocessing may include filtering and smoothing, denoising, and distortion correction. This process yields the target point cloud data, reducing the surface roughness of the initial point cloud data and improving its overall quality.

[0051] During the preprocessing process, the pose of the acquired mobile robot can also be corrected to obtain the corrected pose.

[0052] Step S103: Project the target point cloud data to obtain an initial bird's-eye view. The point cloud in the initial bird's-eye view has initial semantics and the priority of the initial semantics.

[0053] Create a fixed-size 2D bird's-eye view in the target robot's body coordinate system, representing the projection of the target robot's surrounding environment onto the ground. The length and width of this 2D bird's-eye view image can be set to size_x and size_y, respectively, as needed. The distance represented by each pixel in the real-world coordinate system is represented by resolution, used to convert physical space coordinates to image coordinates. The target robot's body is located in the center of the image, meaning each pixel in the image has a corresponding point in the real physical world. These physical world points are back-projected into the camera space using the intrinsic and extrinsic parameters of the scanning device. It is then determined whether the back-projected points exceed the image boundary; points exceeding the image boundary cannot be projected into the image space.

[0054] In this step, a fixed-size BEV image is first created and initialized. Each pixel in the BEV image is treated as a point in the physical world, and these points are back-projected into camera space using the intrinsic and extrinsic parameters of the scanning device (including focal length, optical center position, rotation matrix, and translation vector). All points in the image are then binary-classified: points that can be projected into space are initialized as ground elements, while others are initialized as unknown region elements. Unknown regions refer to areas not visible to the scanning device, while ground refers to areas accessible to the target robot.

[0055] The preprocessed target point cloud data is projected onto the fuselage coordinate system to generate the fuselage point cloud, as follows:

[0056] pt 机身 =pose 矫正 -1 ×pt

[0057] Among them, pt 机身 This represents the point cloud on the fuselage, pose. 矫正 This represents the corrected pose, where pt represents the point cloud in the world coordinate system. The pose data of the target robot is required for projection.

[0058] After acquiring the point cloud in the fuselage coordinate system, the point cloud is projected onto the created BEV image, and the point cloud is filtered out based on the perception distance of the BEV image. This projection process transforms points in three-dimensional space into pixels on a two-dimensional plane, while each pixel retains its original three-dimensional information of the point cloud (e.g., x, y, z coordinate data). Each point cloud is assigned an initial semantic based on its x, y, and z coordinate data. Semantics can include unknown areas, ground, low obstacles, and high obstacles. The height of each semantic is related to the z-axis height of the point cloud. After initial assignment, each pixel in the initial bird's-eye view has an initial semantic. Priorities are set for each semantic according to requirements, and the priority order is as follows: unknown areas, ground, low obstacles, and high obstacles, with unknown areas having the lowest priority and high obstacles having the highest priority.

[0059] Step S104: For the first pixel in the initial bird's-eye view, determine the priority comparison result based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel.

[0060] The first line is used to represent the connection between the center pixel and the first pixel in the initial bird's-eye view, and the first pixel and the second pixel are on the first line.

[0061] After obtaining the initial bird's-eye view, the scanning device, which may be installed higher than the target robot's vision sensor, can observe objects that the vision sensor cannot. For example, if there is a cuboid obstacle in front of the target robot, the target robot cannot observe the object behind the cuboid obstacle (i.e., the occluded object). However, the scanning device, positioned on top of the target robot, may be able to collect the occluded object behind the cuboid obstacle due to its height. The initial point cloud data collected by the scanning device contains point cloud data of the occluded object, so these occluded objects need to be removed.

[0062] Edge pixels are the pixels at the edges of the initial bird's-eye view image, and center pixels are the pixels at the very center of the initial bird's-eye view. A target ray can be obtained by extending from the center pixel to any edge pixel. By traversing along the edges of the initial bird's-eye view, multiple target rays can be determined for each edge pixel. Let any pixel on a target ray be the first pixel. Each target ray has multiple first pixels, and each first pixel has an initial semantic meaning.

[0063] Let the target ray extend from the center pixel to the edge pixels. For each first pixel, the line connecting the center pixel and the first pixel is called the first line. Any pixel on the first line is called the second pixel. By reading the initial semantics of all second pixels, the highest priority of the initial semantics on the first line can be determined. The priority of the initial semantics of the first pixel is compared with the highest priority of the initial semantics of all second pixels on the first line to obtain the comparison result.

[0064] The comparison results include that the initial semantic priority of the first pixel is less than the highest initial semantic priority of all second pixels on the first line, and that the initial semantic priority of the first pixel is not greater than the highest initial semantic priority of all second pixels on the first line.

[0065] Taking the initial semantics of the first pixel A as A1 as an example, the highest priority of the initial semantics of all second pixels on the first line depends on whether there is a second pixel with an initial semantic priority higher than A1 between the center pixel and the first pixel (including the first pixel). If not, the highest priority of the initial semantics on the first line is determined to be A1. Next, extending from the center pixel towards the edge pixels, the next first pixel B is determined, with an initial semantics of B1. The priority of B1 is compared with the priority of A1. If the priority of B1 is higher than the priority of A1, the highest priority of the initial semantics of the first line changes to B1; if the priority of B1 is lower than the priority of A1, the highest priority of the initial semantics of the first line remains A1.

[0066] Step S105: Based on the priority comparison result, determine the adjustment result of the initial semantics of the target pixel to obtain the target bird's-eye view.

[0067] If the initial semantic priority of the first pixel is lower than the highest priority of the initial semantics of all second pixels on the first line, the initial semantics of the first pixel are adjusted. Specifically, in this embodiment, the initial semantics of the first pixel are adjusted to the lowest priority semantics. After step S104, the initial semantics of all pixels in the initial bird's-eye view are specifically adjusted according to the comparison results to obtain the semantically optimized target bird's-eye view. The target pixel is any specific first pixel. This method helps to identify high obstacles that pose a major obstacle to the passage of mobile robots more quickly in complex scenes and reduces attention to other lower priority objects.

[0068] The method for generating a bird's-eye view provided in this embodiment acquires initial point cloud data during the movement of a target robot, preprocesses the initial point cloud data to obtain target point cloud data, projects it to obtain an initial bird's-eye view, and assigns initial semantics to the pixels in the initial bird's-eye view, with the initial semantics having priorities. For the first pixel in the initial bird's-eye view, the priority comparison result is determined based on the relationship between the highest priority of the initial semantics of all second pixels on the first connecting line and the priority of the initial semantics of the first pixel. The first connecting line is used to represent the connection between the center pixel and the first pixel in the initial bird's-eye view. Based on the priority comparison result, the adjustment result of the initial semantics of the target pixel is determined to obtain the target bird's-eye view. This method can generate a high-precision bird's-eye view by acquiring data during the movement of the target robot, and improve the accuracy of the generated bird's-eye view by setting priorities to adjust the semantics of the pixels in the initial bird's-eye view. This solution can be implemented using only appropriate data acquisition equipment and low-beam scanning equipment, and the cost is low.

[0069] This embodiment provides a method for generating a bird's-eye view, which includes the following steps:

[0070] Step S201: Obtain initial point cloud data during the target robot's movement.

[0071] Step S202: Preprocess the initial point cloud data to obtain the target point cloud data.

[0072] Step S203: Project the target point cloud data to obtain an initial bird's-eye view, wherein the pixels in the initial bird's-eye view have initial semantics.

[0073] Please see details Figure 1 Steps S101-S103 of the illustrated embodiment will not be repeated here.

[0074] Step S204: For the first pixel in the initial bird's-eye view, determine the priority comparison result based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel.

[0075] Specifically, in step S204 above, the priority comparison result is determined based on the relationship between the highest priority of the initial semantics of all second pixels on the first connection line and the priority of the initial semantics of the first pixel, including:

[0076] Step S2041: Determine the first connection between the center pixel and the first pixel.

[0077] The center pixel is the pixel at the very center of the initial bird's-eye view. A target ray can be obtained by extending from the center pixel to any edge pixel. By traversing along the edges of the initial bird's-eye view, multiple target rays can be determined for each edge pixel. Let any pixel on a target ray be the first pixel. Each target ray has multiple first pixels, and each first pixel has an initial semantic meaning. For each target ray, starting from the center pixel and traversing towards the edge pixels, each first pixel on the target ray forms a first line with the center pixel.

[0078] Step S2042: Obtain the initial semantic priority of all second pixels on the first connection line, and determine the highest priority on the first connection line.

[0079] The initial semantics of all second pixels along the first connection are read. The highest-priority initial semantics and corresponding highest priority are obtained by comparing the priorities of the initial semantics of each second pixel along the first connection. After adjusting the semantics of the current first pixel, a new first connection is determined based on its next pixel, at which point the highest priority of the first connection is redefined.

[0080] Step S2043: Compare the highest priority with the initial semantic priority of the first pixel to obtain the priority comparison result.

[0081] The initial semantic priority of the first pixel is compared with the highest priority of the first connection to obtain the comparison result.

[0082] Step S205: Based on the priority comparison result, determine the adjustment result of the initial semantics of the target pixel to obtain the target bird's-eye view.

[0083] Specifically, step S205 includes: if the priority of the initial semantic of the target pixel is less than the highest priority, then the initial semantic of the target pixel is adjusted to the preset target semantic to obtain the target bird's-eye view.

[0084] In this embodiment, the preset target semantics can be set to the semantics with the lowest priority. When the initial semantics of a target pixel has a lower priority than the highest priority, the initial semantics of the target pixel are adjusted. For each target ray, the target pixel is traversed from the center pixel to the edge pixel and reassigned. Each target ray is determined along the edge of the initial bird's-eye view in a clockwise or counterclockwise direction. The initial semantics of each target pixel are compared and adjusted in the same way to obtain the target bird's-eye view after semantic adjustment.

[0085] Taking the semantic categories of unknown region, ground, low obstacle, and high obstacle as a specific example, the semantic order, from lowest to highest priority, is: unknown region, ground, low obstacle, high obstacle. If the initial semantic priority of a target pixel is lower than the highest priority, the initial semantics of the target pixel are adjusted to unknown region.

[0086] Taking the initial semantics of the first target pixel A as a low obstacle as an example, if the highest priority initial semantics of the current first connection is low obstacle, then there is no need to reassign the current target pixel. Next, the second target pixel B is read on the target ray. Target pixel B and the center pixel form a new first connection. If the initial semantics of the second target pixel B is ground, the ground has a lower priority than low obstacle, so the second target pixel B is reassigned as unknown region. Next, the initial semantics of the third target pixel C is read. If the initial semantics of the third target pixel C is high obstacle, then the highest priority initial semantics of the first connection is high obstacle, and the priority of the third target pixel C's initial semantics is not lower than the priority of high obstacle, so there is no need to reassign the third target pixel. Next, the initial semantics of the fourth target pixel D is read. If the initial semantics of the fourth target pixel D is ground, the ground has a lower priority than the highest priority of the current first connection (high obstacle), so the fourth target pixel D is adjusted to unknown region.

[0087] The semantics in this method are just examples. In practical applications, they can be extended. If there are 3D bounding boxes for multiple object categories (e.g., pedestrians, cars, etc.), these 3D bounding boxes can be projected onto the bird's-eye view to form denser bird's-eye view observation information.

[0088] The method for generating a bird's-eye view provided in this embodiment uses a method of connecting and traversing pixels and setting semantic priorities to adjust the semantics of pixels, thereby improving the accuracy of the generated bird's-eye view.

[0089] In some optional embodiments, step S102 of the above-described method for generating a bird's-eye view includes the following steps:

[0090] Step S301: Obtain pose data of the target robot during its movement.

[0091] The pose data includes the correspondence between pose and time. Pose data is collected during the target robot's movement, specifically through wheel speed odometers. The pose data includes the target robot's position information (x, y, z) and quaternion information (w_x, w_y, w_z, w_w).

[0092] Step S302: Based on the pose data, perform distortion removal processing on the initial point cloud data at the corresponding time to obtain the target point cloud data.

[0093] Since the initial point cloud data of the target robot is collected during its movement, it may be inaccurate. In this embodiment, distortion correction processing is performed on the initial point cloud data. Distortion correction is a process that eliminates image or point cloud data distortion or deformation caused by the physical characteristics of devices such as camera lenses or sensors, thus avoiding affecting the accuracy and reliability of the point cloud data. The distortion correction process requires pose data at the corresponding time points and specifically includes the following steps:

[0094] Step S3021: Obtain the start time of scanning, the completion time of scanning, and the external parameter matrix of the scanning device on the target robot.

[0095] Let the time when the point cloud receives the data from the scanning device be the start time of scanning, let the maximum time for the scanning device to complete one scan be scan_time, and let the extrinsic matrix of the scanning device be lidar_extrinsics. Therefore, we can determine that the time when the scanning device completes scanning is tm + scan_time.

[0096] The extrinsic parameter matrix of the scanning device describes the relative position and orientation between the scanning device coordinate system and the world coordinate system. This matrix typically includes two parts: rotation and translation, used to transform points in the scanning device coordinate system to the world coordinate system.

[0097] Step S3022: Index the pose data to determine the start pose corresponding to the start scanning time and the end pose corresponding to the scan completion time.

[0098] The pose data stores the correspondence between pose and time. Read the pose data and retrieve the start pose (start_pose) corresponding to the start scan time (tm) and the end pose (end_pose) corresponding to the scan completion time (tm+scan_time) in chronological order.

[0099] Step S3023: Determine the point cloud displacement and angle change within each preset time period based on the start pose, end pose, and extrinsic parameter matrix.

[0100] Obtain the pose change of the scanning device in the world coordinate system at the start of the scan, and the point cloud displacement v_t and angle change v_r for each preset time interval within the maximum scan time (scan_time) of one scan cycle. The specific calculation method is as follows:

[0101] diff = start_pose -1 ×end_pose

[0102] diff_lidar = lidar_extrinsice -1 ×diff×lidar_extrinsics

[0103] v_t=T diff_lidar / scan_time

[0104] v_r=R diff_lidar / scan_time

[0105] Where, `diff` represents the pose change between the scan start time `tm` and the scan completion time `tm+scan_time`, `start_pose` represents the start pose, `end_pose` represents the end pose, `diff_lidar` represents the pose change of the point cloud acquired by the scanning device in the body coordinate system between the scan start time `tm` and the scan completion time `tm+scan_time`, `lidar_extrinsics` represents the extrinsic matrix of the scanning device, and T... diff_lidar R represents the translation vector representing the transformation amount of the scanning device projected onto the machine coordinate system. diff_lidar The rotation vector represents the transformation amount of the scanning device projected onto the machine coordinate system.

[0106] Step S3024: Calculate the rotation and translation variables of the initial point cloud data based on the point cloud displacement, angle change, and extrinsic parameter matrix.

[0107] Step S3025: Determine the target point cloud data based on rotation and translation variables.

[0108] Rotation and translation variables are variables relative to a specific moment. First, the rotation and translation variables are calculated based on the point cloud displacement, angle change, and extrinsic parameter matrix. Then, distortion correction is performed on each 3D point in the initial point cloud data acquired by the scanning device. The specific calculation method is as follows:

[0109] dt = tf(t)

[0110] start_lidar2world=start_pose×lidar_extrinsics

[0111] Pose r =exp(v_r×dt)

[0112] Pose t =v_t×dt

[0113] pt = start_lidar2world × [Pose] r Pose t ]×pt

[0114] Where dt represents the relative time of each point in the scanning device, t represents the absolute time of each point in the scanning device, f(t) represents the rounding operation of the time of each point in the scanning device, start_lidar2world represents projecting the initial point cloud data onto the world coordinate system corresponding to the scanning start time tm, and Pose r Pose represents the rotation variable of the initial point cloud data at time dt relative to time f(t). t dt represents the translation variable of the initial point cloud data relative to f(t), exp(v_r×dt) represents the rotation variable of the point cloud at time dt relative to the point cloud at time f(t), and pt represents the target point cloud data.

[0115] In this embodiment, distortion correction processing is performed on the initial point cloud data to reduce point cloud data distortion caused by the movement of the target robot and reduce the impact on the subsequent construction of the bird's-eye view.

[0116] In some optional embodiments, step S103 of the above-described method for generating a bird's-eye view includes the following steps:

[0117] Step S401: Filter the pose data, identify abnormal poses, correct the abnormal poses, and determine the target pose.

[0118] If the target robot passes over objects of height during its movement, its pitch angle may change significantly, potentially leading to errors in the final bird's-eye view and misidentifying ground point clouds as obstacles. Therefore, in this embodiment, abnormal poses are identified and corrected to obtain the target pose. The target pose refers to all acquired poses, including corrected abnormal poses and non-abnormal poses that do not require correction. Based on the distortion removal operation of the initial point cloud, target point cloud data is obtained. The target point cloud data and the corresponding pose data are traversed in chronological order to determine the pose of the nth time step before any target time step. 前 pose at the nth time step 后 and the current pose at the target time. 当前 The quaternions in the pose data are converted into pitch, roll, and yaw angles. Abnormal poses also need to be identified, as detailed below:

[0119]

[0120] in, The pitch angle represents the target's pose at any given moment. The pitch angle represents the pose at the nth time step before the target time. The pitch angle represents the pose at the nth time after the target time, diff 前 The diff represents the pose change from the nth time before the target time to the target time. 后 K represents the pose change from the target time to the nth time after the target time. titlt This indicates the criteria for judging abnormal poses, and bump represents the tilt angle threshold.

[0121] When K titlt If true, it means that the target's pose at that moment is abnormal and needs to be corrected.

[0122] The rotations in the abnormal pose are corrected, and the target pose is obtained after correction, as follows:

[0123]

[0124] Among them, R 矫正 This indicates correction in the target pose. The pitch angle represents the pose at the nth time step before the target time. The yaw angle represents the target's pose at any given time. The roll angle represents the pose at the nth time after the target time. fun(*) represents converting the pitch, roll, and yaw angles into quaternions.

[0125] When K titlt A value of false indicates that no correction is needed.

[0126] Step S402: Project the target point cloud data based on the target pose to obtain the first initial bird's-eye view.

[0127] Create a fixed-size 2D bird's-eye view in the target robot's body coordinate system, representing the projection of the target robot's surrounding environment onto the ground. The length and width of this 2D bird's-eye view image can be set as size_x and size_y, respectively, as needed. The distance represented by each pixel in the real-world coordinate system is represented by resolution, used to convert physical space coordinates to image coordinates. The target robot's body is located in the center of the image, meaning that each pixel in the image has a corresponding point in the real physical world. The corresponding points in the physical world are back-projected into the camera space using the intrinsic and extrinsic parameters of the scanning device. It is then determined whether the back-projected points exceed the image boundary; points exceeding the image boundary indicate that they cannot be projected into the image space.

[0128] In this step, a fixed-size BEV image is first created and initialized. Each pixel in the BEV image is treated as a point in the physical world, and these points are back-projected into camera space using the intrinsic and extrinsic parameters of the scanning device (including focal length, optical center position, rotation matrix, and translation vector). All points in the image are then binary-classified: points that can be projected into space are initialized as ground elements, while others are initialized as unknown region elements. Unknown regions refer to areas not visible to the scanning device, while ground refers to areas accessible to the target robot.

[0129] The preprocessed target point cloud data is projected onto the fuselage coordinate system by correcting the pose, generating the fuselage point cloud, as shown in the following formula:

[0130] pt 机身 =pose 矫正 -1 ×pt

[0131] Among them, pt 机身 This represents the point cloud on the fuselage, pose. 矫正 pt represents the corrected pose (target pose), and pt represents the point cloud in the world coordinate system.

[0132] After acquiring the point cloud in the fuselage coordinate system, the point cloud is projected onto the created BEV image, and the point cloud is filtered out according to the perceptual distance of the BEV image. This projection process transforms points in three-dimensional space into pixels on a two-dimensional plane. Each pixel still retains its original three-dimensional information of the point cloud (such as x, y, and z coordinate data), thus obtaining the first initial bird's-eye view.

[0133] Step S403: Based on the target point cloud data and the preset height threshold, the pixels in the first initial bird's-eye view are assigned initial semantic values ​​to obtain the initial bird's-eye view. The preset height threshold corresponds to the obstacle type.

[0134] Specifically, it includes the following steps:

[0135] Step S4031: Classify the pixels in the first initial bird's-eye view based on the preset height threshold and the height information of the pixels in the first initial bird's-eye view to obtain the classification result.

[0136] In addition to the ground, the semantics of a pixel also include obstacles. In this embodiment, obstacles are classified according to their height, and a preset height threshold is determined based on the height of the obstacle. When there are obstacles of various heights, multiple preset height thresholds need to be set.

[0137] Step S4032: Based on the classification results, semantic values ​​are assigned to the pixels in the first initial bird's-eye view to determine the initial bird's-eye view.

[0138] For example, let the semantics include: unknown area, ground, low obstacle, and high obstacle, wherein the obstacle includes low obstacle and high obstacle. Let the preset height thresholds include a first height threshold min_d and a second height threshold min_low, where the first height threshold is less than the second height threshold. Based on the comparison of the z-axis height of the pixel in the image with the first height threshold and the second height threshold respectively, the pixel is divided into three categories: z-axis height less than the first height threshold min_d, z-axis height greater than the first height threshold min_d and less than the second height threshold min_low, and z-axis height greater than the second height threshold min_low.

[0139] If a pixel's z-axis height is less than the first height threshold min_d, the pixel is assigned ground semantics; if a pixel's z-axis height is greater than the first height threshold min_d but less than the second height threshold min_low, the pixel is assigned low obstacle semantics; if a pixel's z-axis height is greater than the second height threshold min_low, the pixel is assigned high obstacle semantics. After semantic assignment, the initial bird's-eye view is determined.

[0140] The bird's-eye view generation method provided in this embodiment can detect and correct abnormal poses of the target robot, thereby solving the problem of incorrect identification of point cloud generation due to height issues when the target robot moves over speed bumps or ground protrusions, and improving the accuracy of the generated bird's-eye view.

[0141] This embodiment also provides a bird's-eye view generation apparatus, which is used to implement the above embodiments and implementation methods, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0142] This embodiment provides a device for generating a bird's-eye view, such as... Figure 2 As shown, it includes:

[0143] The initial point cloud acquisition module 501 is used to acquire the initial point cloud data during the movement of the target robot;

[0144] Point cloud preprocessing module 502 is used to preprocess the initial point cloud data to obtain target point cloud data;

[0145] The data projection module 503 is used to project the target point cloud data to obtain an initial bird's-eye view, wherein the pixels in the initial bird's-eye view have initial semantics.

[0146] The priority comparison module 504 is used to determine the priority comparison result for the first pixel in the initial bird's-eye view based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel. The first line is used to characterize the connection between the center pixel in the initial bird's-eye view and the first pixel. The first pixel and the second pixels are on the first line.

[0147] Image generation module 505 is used to determine the initial semantic adjustment result of the target pixel based on the priority comparison result, so as to obtain a target bird's-eye view, wherein the target pixel is any of the first pixel.

[0148] In some alternative implementations, the image generation module 505 includes:

[0149] An image determination unit is configured to adjust the initial semantics of the target pixel to a preset target semantics if the priority of the initial semantics of the target pixel is less than the highest priority, so as to obtain the target bird's-eye view.

[0150] In some alternative implementations, the point cloud preprocessing module 502 includes:

[0151] The pose data acquisition unit is used to acquire pose data during the movement of the target robot. The pose data includes the correspondence between pose and time.

[0152] The distortion correction unit is used to perform distortion correction processing on the initial point cloud data at the corresponding time based on the pose data to obtain the target point cloud data.

[0153] In some alternative implementations, the distortion correction unit includes:

[0154] The first acquisition subunit is used to acquire the start scanning time, the completion scanning time, and the external parameter matrix of the scanning device on the target robot.

[0155] The data indexing subunit is used to index the pose data to determine the start pose corresponding to the start scanning time and the end pose corresponding to the scan completion time.

[0156] The first calculation subunit is used to determine the point cloud displacement and angle change within each preset time period based on the start pose, end pose and the extrinsic parameter matrix.

[0157] The target point cloud computing subunit is used to calculate the rotation and translation variables of the initial point cloud data based on the point cloud displacement, the angle change, and the extrinsic parameter matrix.

[0158] The target point cloud data is determined based on the rotation and translation variables.

[0159] In some alternative implementations, the data projection module 503 includes:

[0160] The pose correction unit is used to filter the pose data, identify abnormal poses, correct the abnormal poses, and determine the target pose.

[0161] The first view generation unit is used to project the target point cloud data based on the target pose to obtain a first initial bird's-eye view;

[0162] The semantic assignment unit is used to perform initial semantic assignment on the pixels in the first initial bird's-eye view based on the target point cloud data and a preset height threshold to obtain the initial bird's-eye view. The preset height threshold corresponds to the obstacle type.

[0163] In some optional implementations, the semantic assignment unit includes:

[0164] The classification subunit is used to classify the pixels in the first initial bird's-eye view based on the preset height threshold and the height information of the pixels in the first initial bird's-eye view, and obtain the classification result;

[0165] An initial view generation unit is used to assign semantic values ​​to the pixels in the first initial bird's-eye view based on the classification results, and to determine the initial bird's-eye view.

[0166] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0167] In this embodiment, the bird's-eye view generation device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0168] This invention also provides a computer device having the above-described features. Figure 2 The apparatus for generating the bird's-eye view shown.

[0169] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0170] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0171] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0172] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0174] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0175] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0176] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0177] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the invention.

Claims

1. A method for generating a bird's-eye view, characterized in that, The method includes: Acquire initial point cloud data during the target robot's movement; The initial point cloud data is preprocessed to obtain the target point cloud data; The target point cloud data is projected to obtain an initial bird's-eye view, and the pixels in the initial bird's-eye view have initial semantics and the priority of the initial semantics; For the first pixel in the initial bird's-eye view, the priority comparison result is determined based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel. The first line is used to characterize the connection between the center pixel in the initial bird's-eye view and the first pixel. The first pixel and the second pixels are on the first line. Based on the priority comparison result, the initial semantic adjustment result of the target pixel is determined to obtain a target bird's-eye view, wherein the target pixel is any of the first pixel.

2. The method according to claim 1, characterized in that, The adjustment result of determining the initial semantics of the target pixel based on the priority comparison result to obtain the target bird's-eye view includes: If the initial semantic priority of the target pixel is lower than the highest priority, then the initial semantic priority of the target pixel is adjusted to the preset target semantic to obtain the target bird's-eye view.

3. The method according to claim 1, characterized in that, The preprocessing of the initial point cloud data to obtain the target point cloud data includes: Acquire pose data of the target robot during its motion process, wherein the pose data includes the correspondence between pose and time. Based on the pose data, the initial point cloud data at the corresponding time is subjected to distortion correction processing to obtain the target point cloud data.

4. The method according to claim 3, characterized in that, The step of performing distortion correction processing on the initial point cloud data at the corresponding time based on the pose data to obtain the target point cloud data includes: Obtain the start scanning time, scan completion time, and external parameter matrix of the scanning device on the target robot; Index the pose data to determine the start pose corresponding to the start scanning time and the end pose corresponding to the scan completion time. Based on the start pose, end pose, and extrinsic parameter matrix, determine the point cloud displacement and angle change within each preset time period; Based on the point cloud displacement, the angle change, and the extrinsic parameter matrix, calculate the rotation and translation variables of the initial point cloud data; The target point cloud data is determined based on the rotation and translation variables.

5. The method according to claim 3, characterized in that, The process of projecting the target point cloud data to obtain an initial bird's-eye view includes: The pose data is filtered to identify abnormal poses, and the abnormal poses are corrected to determine the target pose. Projecting the target point cloud data based on the target pose yields a first initial bird's-eye view. The initial semantic values ​​of the pixels in the first initial bird's-eye view are assigned based on the target point cloud data and a preset height threshold to obtain the initial bird's-eye view. The preset height threshold corresponds to the obstacle type.

6. The method according to claim 5, characterized in that, The step of assigning initial semantic values ​​to pixels in the first initial bird's-eye view based on target point cloud data and a preset height threshold to obtain the initial bird's-eye view includes: Based on the preset height threshold and the height information of the pixels in the first initial bird's-eye view, the pixels in the first initial bird's-eye view are classified to obtain the classification result; Based on the classification results, semantic values ​​are assigned to the pixels in the first initial bird's-eye view to determine the initial bird's-eye view.

7. A device for generating a bird's-eye view, characterized in that, The device includes: The initial point cloud acquisition module is used to acquire initial point cloud data during the target robot's motion process; The point cloud preprocessing module is used to preprocess the initial point cloud data to obtain the target point cloud data; The data projection module is used to project the target point cloud data to obtain an initial bird's-eye view, wherein the pixels in the initial bird's-eye view have initial semantics. The priority comparison module is used to determine the priority comparison result for the first pixel in the initial bird's-eye view based on the relationship between the highest priority of the initial semantics of all second pixels on the first line and the priority of the initial semantics of the first pixel. The first line is used to characterize the connection between the center pixel in the initial bird's-eye view and the first pixel. The first pixel and the second pixels are on the first line. An image generation module is used to determine the initial semantic adjustment result of the target pixel based on the priority comparison result, so as to obtain a target bird's-eye view, wherein the target pixel is any of the first pixel.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the method for generating a bird's-eye view according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method for generating a bird's-eye view according to any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the method for generating a bird's-eye view according to any one of claims 1 to 6.

Images