An aerial floating object detection method, device, equipment and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for detecting airborne floating objects. It integrates image data and point cloud data collected by autonomous vehicles to obtain obstacle attribute information, including category, trajectory, three-dimensional contour, and motion state information. Obstacles are filtered based on this attribute information. The method determines whether a target obstacle is a rigid body based on feature point information of the target obstacle in the filtered set. If not, the target obstacle is determined to be an airborne floating object. If it is, the vertical axis acceleration of the target obstacle is obtained based on its trajectory. If the vertical axis acceleration is less than the reference gravitational acceleration and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be an airborne floating object. This improves upon existing technologies that treat airborne floating objects as obstacles, which can lead to unreasonable behavior by autonomous vehicles and consequently safety accidents.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, device, and storage medium for detecting airborne objects. Background Technology

[0002] With the development of Level 4 autonomous driving technology, autonomous vehicles will encounter more special scenarios, one of which is the handling of various floating or flying objects, such as plastic bags, balloons, and birds. Due to the vast number of categories, it is difficult to classify them precisely. Existing technologies typically treat these objects as obstacles, causing autonomous vehicles to make a series of unnecessary reactions, such as sharp steering or emergency braking, affecting passenger experience and, in extreme cases, easily leading to loss of control, rear-end collisions, and other safety accidents. Therefore, to prevent autonomous vehicles from behaving irrationally due to floating objects and causing safety accidents, providing a method for detecting floating objects is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for detecting airborne objects, which improves the technical problem that the prior art treats airborne objects as obstacles, making autonomous vehicles prone to unreasonable behavior and thus leading to safety accidents.

[0004] In view of this, the first aspect of this application provides a method for detecting airborne objects, comprising:

[0005] By integrating image data and point cloud data collected by autonomous vehicles, the attribute information of obstacles is obtained, including category, motion trajectory, three-dimensional contour and motion state information;

[0006] The obstacles are filtered based on the attribute information to obtain a set of filtered obstacles to be judged;

[0007] Based on the feature point information of the target obstacle in the set of obstacles to be judged, it is determined whether the target obstacle is a rigid body. If not, the target obstacle is determined to be a floating object in the air. If so, the vertical axis acceleration of the target obstacle is obtained based on the motion trajectory of the target obstacle. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be a floating object in the air.

[0008] Optionally, the process of fusing image data and point cloud data collected by the autonomous vehicle to obtain obstacle attribute information includes:

[0009] Semantic segmentation and classification are performed sequentially on the image data and point cloud data currently collected by the autonomous vehicle to obtain the classification results of the image data and the point cloud data, and the classification results include the categories of obstacles;

[0010] Based on the classification results of the image data and the point cloud data, the image data is projected into a three-dimensional space to obtain pseudo point cloud data. The pseudo point cloud data and the point cloud data are then matched, aligned, and clustered sequentially to obtain the point group information of the obstacle. The point group information includes the number of points, three-dimensional center coordinates, and three-dimensional contour.

[0011] The obstacle is tracked based on the point group information of the obstacle to obtain the motion trajectory and motion state information of the obstacle, wherein the motion state information includes velocity and acceleration;

[0012] By integrating the classification results, the point group information of the obstacles, the motion trajectory, and the motion state information, the attribute information of the obstacles is obtained.

[0013] Optionally, the step of filtering the obstacles based on the attribute information to obtain a filtered set of obstacles to be judged includes:

[0014] Based on the category, running status information, three-dimensional contour, and motion trajectory in the attribute information, obstacles of known category, static obstacles, ground obstacles, and obstacles whose trajectory time length is less than a preset time length threshold are filtered out from the obstacles to obtain a preliminary set of obstacles;

[0015] From the initially screened set of obstacles, obstacles whose trajectories conform to projectile motion are removed, resulting in a screened set of obstacles to be judged.

[0016] Alternatively, based on the motion trajectory, obstacles that do not interact with the path planned by the autonomous vehicle within a future preset time period and obstacles whose motion trajectories conform to projectile motion trajectories are filtered out from the initially screened obstacle set to obtain a screened obstacle set to be judged.

[0017] Optionally, the step of filtering out known category obstacles, static obstacles, ground obstacles, and obstacles with trajectory durations less than a preset time length threshold from the obstacles based on the category, running status information, three-dimensional contour, and motion trajectory in the attribute information, to obtain a pre-filtered obstacle set, including:

[0018] Based on the category in the attribute information, obstacles of known category are filtered out from the obstacles to obtain obstacles of unknown category;

[0019] Based on the operating status information, static obstacles are filtered out from the unknown category obstacles to obtain the unknown category dynamic obstacles;

[0020] Based on the height of the lower surface in the three-dimensional contour, ground obstacles are filtered out from the unknown category of dynamic obstacles to obtain the unknown category of aerial obstacles;

[0021] Based on the motion trajectory, obstacles with a trajectory time length less than a preset time length threshold are filtered out from the unknown category of aerial obstacles to obtain a preliminary set of obstacles.

[0022] Optionally, the step of filtering out obstacles from the initially screened obstacle set based on the motion trajectory that do not interact with the path planned by the autonomous vehicle within a future preset time period includes:

[0023] Predict the movement trajectory of the obstacle in a future preset time period based on the movement trajectory of the obstacles in the preliminary set of obstacles;

[0024] The movement trajectories of obstacles in the pre-selected obstacle set within a future preset time period are projected onto the ground to obtain the ground projection trajectory of the obstacles within the future preset time period.

[0025] From the initially screened set of obstacles, obstacles whose ground projection trajectories in a future preset time period do not interact with the planned path of the autonomous vehicle in that future preset time period are filtered out.

[0026] Optionally, obtaining the vertical acceleration of the target obstacle based on its motion trajectory includes:

[0027] The centroid trajectory of the target obstacle is obtained based on its motion trajectory.

[0028] The vertical acceleration of the target obstacle is obtained based on the trajectory of its center of mass.

[0029] Optionally, obtaining the vertical acceleration of the target obstacle based on its center-of-mass trajectory includes:

[0030] The trajectory of the centroid of the target obstacle is fitted to a curve to obtain a trajectory fitting curve.

[0031] The vertical acceleration of the target obstacle is obtained by performing a second derivative operation on the trajectory fitting curve in the vertical direction.

[0032] Optionally, the reference gravitational acceleration is determined based on the latitude and altitude of the current location of the autonomous vehicle.

[0033] A second aspect of this application provides an airborne object detection device, comprising:

[0034] The fusion unit is used to fuse image data and point cloud data collected by the autonomous vehicle to obtain the attribute information of the obstacle, including category, motion trajectory, three-dimensional contour and motion state information;

[0035] The filtering unit is used to filter the obstacles according to the attribute information to obtain a set of filtered obstacles to be judged.

[0036] The judgment unit is used to determine whether the target obstacle is a rigid body based on the feature point information of the target obstacle in the set of obstacles to be judged. If not, the target obstacle is determined to be a floating object in the air. If so, the vertical axis acceleration of the target obstacle is obtained based on the motion trajectory of the target obstacle. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be a floating object in the air.

[0037] A third aspect of this application provides an airborne object detection device, the device including a processor and a memory;

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is used to execute any one of the methods for detecting airborne objects according to the instructions in the program code.

[0040] A fourth aspect of this application provides a computer-readable storage medium for storing program code that, when executed by a processor, implements the airborne object detection method according to any one of the first aspects.

[0041] As can be seen from the above technical solutions, this application has the following advantages:

[0042] This application provides a method for detecting floating objects in the air, comprising: fusing image data and point cloud data collected by an autonomous vehicle to obtain attribute information of obstacles, including category, motion trajectory, three-dimensional contour and motion state information; filtering obstacles according to the attribute information to obtain a set of obstacles to be judged; determining whether the target obstacle is a rigid body based on the feature point information of the target obstacle in the set of obstacles to be judged; if not, determining the target obstacle to be a floating object in the air; if so, obtaining the vertical axis acceleration of the target obstacle based on its motion trajectory; if the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, then determining the target obstacle to be a floating object in the air.

[0043] In this application, image data and point cloud data collected by autonomous vehicles are fused to obtain attribute information of obstacles around the vehicle. Obstacles that are clearly not floating objects are filtered out based on the attribute information, resulting in a set of obstacles to be judged. Then, based on the feature point information of the target obstacles in the set of obstacles to be judged, it is determined whether the target obstacle is a rigid body. For non-rigid target obstacles, the detection result of floating object is given. For rigid target obstacles, the vertical axis acceleration determined by its motion trajectory is used to determine whether it is a floating object. This realizes the detection of floating objects, which helps to avoid unnecessary sudden braking and steering by autonomous vehicles treating floating objects as obstacles. It improves the technical problem of existing technology treating floating objects as obstacles, which makes autonomous vehicles prone to unreasonable behavior and thus safety accidents. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a method for detecting airborne objects provided in this application embodiment;

[0046] Figure 2 This is a schematic diagram of a device for detecting airborne objects provided in an embodiment of this application. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0048] For easier understanding, please refer to Figure 1 This application provides a method for detecting airborne objects, including:

[0049] Step 101: Integrate the image data and point cloud data collected by the autonomous vehicle to obtain the attribute information of the obstacle. The attribute information includes category, motion trajectory, three-dimensional contour and motion state information.

[0050] Semantic segmentation and classification are performed sequentially on the image data and point cloud data currently collected by the autonomous vehicle to obtain classification results for the image data and point cloud data, including the category of obstacles. For general small obstacle detection scenarios, semantic segmentation can be performed on the point cloud data collected by the LiDAR on the autonomous vehicle and the image data collected by the camera, and then a classification model can be used to identify the type of each pixel or point cloud, such as pedestrians, bicycles, motorcycles, trucks, roadbeds, etc. For some unknown categories of obstacles, such as balloons, birds, stones, etc., they can be uniformly classified as unknown category obstacles. Therefore, the final classification results of the image data and point cloud data include known category obstacles such as pedestrians and bicycles, as well as unknown category obstacles.

[0051] Based on the classification results of image data and point cloud data, the image data is projected into three-dimensional space to obtain pseudo point cloud data. Then, the pseudo point cloud data and point cloud data are matched, aligned, and clustered in three-dimensional space to obtain the point group information of the obstacle. The point group information includes, but is not limited to, the number of points, the three-dimensional center coordinates, and the three-dimensional contour. The three-dimensional contour includes the height of the upper surface and the height of the lower surface.

[0052] Obstacles are tracked based on their point cluster information to obtain their trajectory and motion state information, which may include velocity and acceleration. The tracking method involves using point cluster information corresponding to multiple frames of image data and point cloud data to obtain the obstacle's trajectory and motion state information. Since floating objects may be small in size, and the vertical resolution of LiDAR is relatively low, obstacles may lie precisely between two longitudinal beams of LiDAR without point cloud segmentation results. In such cases, the vertical resolution advantage of the camera can be utilized for tracking. Historical obstacle point clusters can be stitched together, and then optical flow algorithms can be used for obstacle tracking.

[0053] Finally, by integrating the classification results, obstacle point group information, motion trajectory, and motion state information, the attribute information of the obstacle is obtained.

[0054] Step 102: Filter obstacles based on attribute information to obtain a set of filtered obstacles to be judged.

[0055] After obtaining the attribute information of the obstacles, the obstacles can be filtered based on the attribute information, and obstacles that are obviously not floating objects in the air can be filtered out. This results in a set of obstacles to be judged after filtering, which can reduce the number of floating objects to be detected in the future and help reduce the system load.

[0056] Specifically, obstacles of known categories, static obstacles, ground obstacles, and obstacles with trajectory durations shorter than a preset threshold can be filtered out from the obstacle set based on their category, operational status, 3D contour, and motion trajectory information. This results in a preliminary set of obstacles. In most driving scenarios, obstacles are categorized into static and dynamic obstacles. Common static obstacles include roadblocks, traffic cones, curbs, and guardrails, while dynamic obstacles are generally dynamic traffic participants such as vehicles and pedestrians. Current autonomous driving systems can detect, classify, and track these traffic objects with a high degree of accuracy and have mature systems to handle various scenarios. Therefore, if an obstacle is a known obstacle that the autonomous driving system can classify with high confidence, it can be filtered out without further determination of whether it is a floating object. In other words, known category obstacles can be filtered out based on their category information to obtain unknown category obstacles.

[0057] Among the unknown category obstacles, based on the properties of floating objects, they should be dynamic. Therefore, static obstacles can be filtered out from the unknown category obstacles based on the speed in the running status information, resulting in dynamic obstacles of unknown category. Static obstacles can be directly handled by the autonomous driving system. The autonomous driving system will consider whether the static obstacle is within the drivable area and whether it is on the current trajectory of the autonomous vehicle during path planning, and then take corresponding actions such as deceleration or detour.

[0058] Based on the properties of airborne objects, they should be floating in the air. Therefore, the height of the lower surface in the 3D contour can be used to distinguish between airborne and ground obstacles. This allows us to filter out ground obstacles from unknown categories of dynamic obstacles and obtain unknown categories of airborne obstacles. In real-world roads, some special cases need to be considered. An obstacle might still be floating even if its lower surface is relatively close to the ground. For example, because the dynamic patterns of floating objects are difficult to grasp, and actual 3D velocity changes rapidly, if the velocity is upward, or if a bouncing object experiences a sudden upward velocity, it could be a floating object, such as a plastic bag blown by the wind. On the other hand, sufficiently elastic wheels, balls, jumping cats or dogs, or even oddly dressed children, will also be considered airborne objects and will be handled in the subsequent airborne object identification process. Furthermore, when performing aerial object detection, it is possible to perform aerial object detection on every frame of data collected by the autonomous vehicle in real time. In the current frame of data, the lower surface height of an aerial object is very close to the ground, but in the next frame of data, due to the properties of the aerial object, its lower surface height may change, thus enabling the detection of whether it is an aerial object.

[0059] When identifying floating objects in the air, their motion trajectory can also be used for judgment. With only one frame of data for detection, and without knowing the obstacle's category, it's difficult to distinguish between a floating object and a ground object. If an obstacle is continuously detected for at least 0.3 seconds, and its speed, acceleration, distance, size, and other basic information are stably tracked, at a detection frequency of 10Hz, three frames of historical data can be obtained, leading to a more reliable detection result for floating objects. Therefore, obstacles with trajectory durations shorter than a preset threshold can be filtered from unknown categories of floating obstacles to ensure the accuracy of subsequent floating object detection results, thus obtaining a preliminary set of filtered obstacles.

[0060] In one embodiment, obstacles whose trajectories conform to projectile motion can be filtered out from the initially screened obstacle set to obtain a filtered obstacle set to be judged. If the trajectories of obstacles in the initially screened obstacle set conform to projectile motion, they can be determined to be non-floating objects and can be filtered out to obtain a filtered obstacle set to be judged.

[0061] In another embodiment, obstacles that do not interact with the path planned by the autonomous vehicle within a future preset time period and obstacles whose motion trajectories conform to the trajectory of a projectile can be filtered out from the initially screened set of obstacles to obtain a screened set of obstacles to be judged.

[0062] Most objects cannot change their direction of motion in the air. Only a few objects can actively or passively change their direction of motion significantly, such as birds in flight or plastic bags in turbulent waters. Therefore, the trajectory of an obstacle in a future preset time period can be predicted based on the trajectory of the obstacles in the initially screened obstacle set. Curve fitting can be used to fit the trajectory of the obstacles in the initially screened obstacle set, and the curve obtained from the fitting can be used to predict the trajectory of the obstacle in the future preset time period (such as within the next 8 seconds). Of course, other methods can also be used to predict the trajectory, and no specific limitation is made here. The movement trajectories of obstacles in the initially filtered obstacle set during the future preset time period are then projected onto the ground to obtain the ground projection trajectory of the obstacles during the future preset time period. By comparing the ground projection trajectory of the obstacles with the planned trajectory of the autonomous vehicle during the future preset time period, it can be determined whether these two trajectories interact. This allows obstacles that do not interact with the planned path of the autonomous vehicle during the future preset time period from the initially filtered obstacle set. For obstacles that do not interact with the planned path of the autonomous vehicle during the future preset time period, even if the obstacle is a floating object, since the trajectory does not interact, the obstacle will not affect the normal driving of the autonomous vehicle during the future preset time period. The autonomous vehicle can ignore such obstacles, that is, it will not make sudden braking or steering actions. In this case, it is not necessary to determine whether such obstacles are floating objects. By filtering out obstacles whose ground projection trajectory does not interact with the planned path of the autonomous vehicle during the future preset time period, the amount of computation in subsequent processing can be reduced, which helps to alleviate the system burden.

[0063] Step 103: Determine whether the target obstacle is a rigid body based on the feature point information of the target obstacle in the set of obstacles to be judged. If not, determine that the target obstacle is a floating object in the air. If so, obtain the vertical axis acceleration of the target obstacle based on its motion trajectory. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than the preset difference threshold, then determine that the target obstacle is a floating object in the air.

[0064] If the trajectory of the target obstacle in the set of obstacles to be judged is chaotic and cannot be fitted to the trajectory of a projectile motion, and the feature points of the target obstacle cannot maintain relative distance in the optical flow algorithm, even if the target obstacle has a certain volume, its mass is likely not very large, and it is a lightweight deformable object such as a plastic bag, bed sheet, or clothing. Therefore, this type of target obstacle can be determined as a floating object. In other words, whether a target obstacle is a floating object can be determined by whether it is a rigid body. According to the properties of rigid bodies, the size and shape of a rigid body remain basically unchanged during motion, and the relative positions between its internal feature points also remain basically unchanged. Therefore, it is possible to determine whether a target obstacle is a rigid body based on its feature point information. Feature point matching can be performed on the target obstacle in two adjacent frames of data. If most of the feature points of the target obstacle in two adjacent frames of data cannot be matched, or if the feature points can be matched but the relative distance between the feature points changes significantly (i.e., exceeding a certain threshold), then the target obstacle can be determined to be a non-rigid body. If the feature points of the target obstacle in two adjacent frames of data can be basically matched, and the relative distance between the feature points of the target obstacle in the two frames of data remains basically unchanged or changes very little (within a certain threshold range), then the target obstacle can be determined to be a rigid body.

[0065] When a target obstacle has a certain angular velocity, its shape may be unusual, and its texture may not be uniform, making it a potentially dense object whose trajectory is difficult to fit. This scenario makes it challenging to determine if the object is floating. However, according to the principle of conservation of angular momentum, any rigid body rotating without external force revolves around its center of mass, and the scenario where the detected target obstacle is located falls within this range. Therefore, when tracking the feature points of the target obstacle and calculating its angular velocity, the position of the object's center of mass at that moment can also be obtained. If the target obstacle is a rigid body, its center of mass trajectory can be obtained from its motion trajectory; and its vertical acceleration can be obtained from the center of mass trajectory.

[0066] Curve fitting can be performed on the trajectory of the centroid of the target obstacle to obtain a trajectory fitting curve. The second derivative of the trajectory fitting curve along the vertical axis is then performed to obtain the vertical acceleration of the target obstacle. Curve fitting can be performed on the centroid positions at multiple moments. The least squares method can be used to obtain a trajectory fitting curve that minimizes the second-order distance in Cartesian coordinates. The second derivative of the trajectory fitting curve along the vertical axis (i.e., the Z-axis direction) is then performed to obtain the vertical acceleration of the target obstacle.

[0067] The baseline gravitational acceleration at the current location of the autonomous vehicle can be calculated based on its latitude and altitude. The specific calculation process is existing technology and will not be elaborated here. Considering the effects of air resistance or airflow during movement, a certain degree of fluctuation in the baseline gravitational acceleration is allowed. The specific fluctuation threshold can be determined by analyzing the vertical axis acceleration of multiple airborne objects. If a target obstacle has a vertical axis acceleration greater than the baseline gravitational acceleration, and the difference between the vertical axis acceleration and the baseline gravitational acceleration is greater than a preset difference threshold, then the target obstacle is determined to be a non-airborne object. For non-airborne objects, their landing point can be calculated. In downstream planning, the vehicle should try to avoid the non-airborne object as much as possible, provided it is safe. If the autonomous vehicle interacts before the non-airborne object lands, the non-airborne object will not be considered in the path planning. If the non-airborne object bounces to a certain height after landing, it can be handed over to the next airborne object detection process. If the vertical acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical acceleration and the reference gravitational acceleration is greater than a preset difference threshold, then the target obstacle is determined to be a floating object in the air. In downstream processing, it will pass through by slightly decelerating longitudinally rather than by lateral movement.

[0068] In this embodiment, image data and point cloud data collected by the autonomous vehicle are fused to obtain the attribute information of obstacles around the vehicle. Obstacles that are clearly not floating objects are filtered out based on the attribute information, resulting in a set of obstacles to be judged. Then, based on the feature point information of the target obstacles in the set of obstacles to be judged, it is determined whether the target obstacle is a rigid body. For non-rigid target obstacles, the detection result of floating object is given. For rigid target obstacles, the vertical axis acceleration determined by its motion trajectory is used to determine whether it is a floating object. This realizes the detection of floating objects, which helps to avoid unnecessary sudden braking and sharp steering by the autonomous vehicle treating floating objects as obstacles. It improves the technical problem of the prior art treating floating objects as obstacles, which makes the autonomous vehicle prone to unreasonable behavior and thus leading to safety accidents.

[0069] The above is an embodiment of a method for detecting airborne objects provided in this application. The following is an embodiment of a device for detecting airborne objects provided in this application.

[0070] Please refer to Figure 2 This application provides an airborne object detection device, comprising:

[0071] The fusion unit is used to fuse image data and point cloud data collected by autonomous vehicles to obtain the attribute information of obstacles, including category, motion trajectory, three-dimensional contour and motion state information;

[0072] The filtering unit is used to filter obstacles based on attribute information to obtain a set of filtered obstacles to be judged.

[0073] The judgment unit is used to determine whether a target obstacle is a rigid body based on the feature point information of the target obstacle in the set of obstacles to be judged. If not, the target obstacle is determined to be a floating object in the air. If it is, the vertical axis acceleration of the target obstacle is obtained based on the motion trajectory of the target obstacle. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be a floating object in the air.

[0074] As a further improvement, the fusion unit is specifically used for:

[0075] Semantic segmentation and classification are performed on the image data and point cloud data currently collected by the autonomous vehicle in sequence to obtain the classification results of the image data and point cloud data, including the category of obstacles;

[0076] Based on the classification results of image data and point cloud data, the image data is projected into three-dimensional space to obtain pseudo point cloud data. The pseudo point cloud data and point cloud data are then matched, aligned, and clustered sequentially to obtain the point group information of the obstacle. The point group information includes the number of points, the three-dimensional center coordinates, and the three-dimensional contour.

[0077] The obstacle is tracked based on the point group information of the obstacle to obtain the obstacle's trajectory and motion state information, including velocity and acceleration.

[0078] By integrating classification results, obstacle cluster information, motion trajectory, and motion state information, the attribute information of the obstacles is obtained.

[0079] As a further improvement, the filtering unit is specifically used for:

[0080] Based on the category, running status information, 3D contour, and motion trajectory in the attribute information, obstacles of known category, static obstacles, ground obstacles, and obstacles whose trajectory time length is less than a preset time length threshold are filtered out from the obstacles to obtain a preliminary set of obstacles;

[0081] From the initial set of obstacles, obstacles whose trajectories conform to projectile motion are removed, resulting in the set of obstacles to be judged after filtering.

[0082] Alternatively, based on the motion trajectory, obstacles that do not interact with the path planned by the autonomous vehicle within the future preset time period and obstacles whose motion trajectories conform to the projectile motion trajectory can be filtered out from the initially screened obstacle set to obtain the filtered obstacle set to be judged.

[0083] In this embodiment, image data and point cloud data collected by the autonomous vehicle are fused to obtain the attribute information of obstacles around the vehicle. Obstacles that are clearly not floating objects are filtered out based on the attribute information, resulting in a set of obstacles to be judged. Then, based on the feature point information of the target obstacles in the set of obstacles to be judged, it is determined whether the target obstacle is a rigid body. For non-rigid target obstacles, the detection result of floating object is given. For rigid target obstacles, the vertical axis acceleration determined by its motion trajectory is used to determine whether it is a floating object. This realizes the detection of floating objects, which helps to avoid unnecessary sudden braking and sharp steering by the autonomous vehicle treating floating objects as obstacles. It improves the technical problem of the prior art treating floating objects as obstacles, which makes the autonomous vehicle prone to unreasonable behavior and thus leading to safety accidents.

[0084] This application embodiment also provides an airborne object detection device, the device including a processor and a memory;

[0085] The memory is used to store program code and transfer the program code to the processor;

[0086] The processor is used to execute the airborne object detection method in the aforementioned method embodiments according to the instructions in the program code.

[0087] This application also provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the airborne object detection method in the aforementioned method embodiments.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0089] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0090] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0092] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0095] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method of detecting airborne objects, characterized by, include: By integrating image data and point cloud data collected by autonomous vehicles, the attribute information of obstacles is obtained, including category, motion trajectory, three-dimensional contour and motion state information; The obstacles are filtered based on the attribute information to obtain a set of filtered obstacles to be judged, including: Based on the category, motion state information, three-dimensional contour, and motion trajectory in the attribute information, obstacles of known category, static obstacles, ground obstacles, and obstacles whose trajectory time length is less than a preset time length threshold are filtered out from the obstacles to obtain a preliminary set of obstacles; From the initially screened set of obstacles, obstacles whose trajectories conform to projectile motion are removed, resulting in a screened set of obstacles to be judged. Alternatively, based on the motion trajectory, obstacles that do not interact with the path planned by the autonomous vehicle within a future preset time period and obstacles whose motion trajectories conform to projectile motion trajectories are filtered out from the initially screened obstacle set to obtain a screened obstacle set to be judged. Based on the feature point information of the target obstacle in the set of obstacles to be judged, it is determined whether the target obstacle is a rigid body. If not, the target obstacle is determined to be a floating object in the air. If so, the vertical axis acceleration of the target obstacle is obtained based on the motion trajectory of the target obstacle. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be a floating object in the air.

2. The aerial object detection method of claim 1, wherein, The method of fusing image data and point cloud data collected by autonomous vehicles to obtain obstacle attribute information includes: Semantic segmentation and classification are performed sequentially on the image data and point cloud data currently collected by the autonomous vehicle to obtain the classification results of the image data and the point cloud data, and the classification results include the categories of obstacles; Based on the classification results of the image data and the point cloud data, the image data is projected into a three-dimensional space to obtain pseudo point cloud data. The pseudo point cloud data and the point cloud data are then matched, aligned, and clustered sequentially to obtain the point group information of the obstacle. The point group information includes the number of points, three-dimensional center coordinates, and three-dimensional contour. The obstacle is tracked based on the point group information of the obstacle to obtain the motion trajectory and motion state information of the obstacle, wherein the motion state information includes velocity and acceleration; By integrating the classification results, the point group information of the obstacles, the motion trajectory, and the motion state information, the attribute information of the obstacles is obtained.

3. The method for detecting airborne objects according to claim 1, characterized in that, The process involves filtering out obstacles of known categories, static obstacles, ground obstacles, and obstacles with trajectory durations shorter than a preset threshold from the obstacles based on the category, motion state information, three-dimensional contour, and motion trajectory information, resulting in a pre-filtered obstacle set, including: Based on the category in the attribute information, obstacles of known categories are filtered out from the obstacles to obtain obstacles of unknown categories; Based on the motion state information, static obstacles are filtered out from the unknown category obstacles to obtain the unknown category dynamic obstacles; Based on the height of the lower surface in the three-dimensional contour, ground obstacles are filtered out from the unknown category of dynamic obstacles to obtain the unknown category of aerial obstacles; Based on the motion trajectory, obstacles with a trajectory time length less than a preset time length threshold are filtered out from the unknown category of aerial obstacles to obtain a preliminary set of obstacles.

4. The method for detecting airborne objects according to claim 1, characterized in that, The step of filtering out obstacles from the initially selected obstacle set based on the motion trajectory that do not interact with the path planned by the autonomous vehicle within a future preset time period includes: Predict the movement trajectory of the obstacle in a future preset time period based on the movement trajectory of the obstacles in the preliminary set of obstacles; The movement trajectories of obstacles in the pre-selected obstacle set within a future preset time period are projected onto the ground to obtain the ground projection trajectory of the obstacles within the future preset time period. From the initially screened set of obstacles, obstacles whose ground projection trajectories in a future preset time period do not interact with the planned path of the autonomous vehicle in that future preset time period are filtered out.

5. The method for detecting airborne objects according to claim 1, characterized in that, The step of obtaining the vertical acceleration of the target obstacle based on its motion trajectory includes: The centroid trajectory of the target obstacle is obtained based on its motion trajectory. The vertical acceleration of the target obstacle is obtained based on the trajectory of its center of mass.

6. The method for detecting airborne objects according to claim 5, characterized in that, The step of obtaining the vertical axis acceleration of the target obstacle based on its center-of-mass trajectory includes: The trajectory of the centroid of the target obstacle is fitted to a curve to obtain a trajectory fitting curve. The vertical acceleration of the target obstacle is obtained by performing a second derivative operation on the trajectory fitting curve in the vertical direction.

7. The method for detecting airborne objects according to claim 1, characterized in that, The reference gravitational acceleration is determined based on the latitude and altitude of the current location of the autonomous vehicle.

8. A device for detecting airborne objects, characterized in that, include: The fusion unit is used to fuse image data and point cloud data collected by autonomous vehicles to obtain the attribute information of obstacles, including category, motion trajectory, three-dimensional contour and motion state information; The filtering unit is used to filter the obstacles according to the attribute information to obtain a set of filtered obstacles to be judged. The filtering unit is specifically used for: Based on the category, motion state information, three-dimensional contour, and motion trajectory in the attribute information, obstacles of known category, static obstacles, ground obstacles, and obstacles whose trajectory time length is less than a preset time length threshold are filtered out from the obstacles to obtain a preliminary set of obstacles; From the initially screened set of obstacles, obstacles whose trajectories conform to projectile motion are removed, resulting in a screened set of obstacles to be judged. Alternatively, based on the motion trajectory, obstacles that do not interact with the path planned by the autonomous vehicle within a future preset time period and obstacles whose motion trajectories conform to projectile motion trajectories are filtered out from the initially screened obstacle set to obtain a screened obstacle set to be judged. The judgment unit is used to determine whether the target obstacle is a rigid body based on the feature point information of the target obstacle in the set of obstacles to be judged. If not, the target obstacle is determined to be a floating object in the air. If so, the vertical axis acceleration of the target obstacle is obtained based on the motion trajectory of the target obstacle. If the vertical axis acceleration of the target obstacle is less than the reference gravitational acceleration, and the difference between the vertical axis acceleration and the reference gravitational acceleration is greater than a preset difference threshold, the target obstacle is determined to be a floating object in the air.

9. A device for detecting airborne objects, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the airborne object detection method according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the airborne object detection method according to any one of claims 1-7.

Images