Point cloud filtering method and device, electronic equipment and storage medium

Abstract

The application provides a point cloud filtering method and device, electronic equipment and storage medium. The method comprises: acquiring horizontal images collected by visual sensors with multiple horizontal angles; converting each horizontal image into a top-view image with a top-view angle; splicing multiple top-view images into a surround-view image according to the angles of the visual sensors; calculating a concentration probability map of a current frame of the surround-view image according to at least one frame of surround-view image before the current frame, wherein the feature concentration corresponding to the current frame of the surround-view image in the concentration probability map is used to indicate whether the corresponding feature is a dynamic obstacle; and filtering laser point clouds at features with a feature concentration within a dynamic obstacle concentration range based on the concentration probability map. The application realizes filtering of dynamic obstacles in point cloud data without a large amount of laser point cloud data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision and image processing, and in particular to a point cloud filtering method and device, an electronic device and a storage medium. BACKGROUND

[0002] At present, in the field of automatic driving, it is often necessary to rely on laser point clouds collected by laser sensors to establish a map and position a vehicle. In the map establishment and vehicle positioning based on laser point clouds, the processing of dynamic point clouds is particularly important. The existing dynamic processing can only identify whether the laser point cloud corresponds to a dynamic obstacle according to the morphological structure of the laser point cloud. However, such a method requires a large amount of laser point cloud data to distinguish different object categories.

[0003] Therefore, how to filter out dynamic obstacles in point cloud data without a large amount of laser point cloud data is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0004] In order to overcome the defects of the prior art, the present application provides a point cloud filtering method, device, electronic device and storage medium to filter out dynamic obstacles in point cloud data without a large amount of laser point cloud data.

[0005] According to one aspect of the present application, a point cloud filtering method is provided, comprising:

[0006] obtaining a plurality of horizontal images collected by visual sensors with a horizontal direction as a viewing angle;

[0007] converting each of the horizontal images into a top-view image with a top-view angle;

[0008] splicing a plurality of the top-view images into a surround-view image according to the viewing angle of each of the visual sensors;

[0009] calculating a density probability map of the current frame surround-view image according to at least one frame of surround-view image before the current frame, wherein the feature density corresponding to the current frame surround-view image in the density probability map is used to indicate whether the corresponding feature is a dynamic obstacle;

[0010] filtering the laser point cloud at the feature whose feature density meets the dynamic obstacle density range based on the density probability map.

[0011] In some embodiments of the present application, the conversion of each of the horizontal images into a top-view image with a top-view angle comprises:

[0012] segmenting the top-view image to obtain a segmentation feature and a feature label.

[0013] In some embodiments of the present application, the feature density corresponding to the current frame surround view image in the density probability map is used to indicate the probability that the corresponding segmented feature is a dynamic obstacle, and the laser point cloud at the feature where the feature density meets the dynamic obstacle density range is filtered based on the density probability map, and the filtered laser point cloud is retained.

[0014] The feature label of the segmented feature is assigned to the filtered and retained laser point cloud.

[0015] In some embodiments of the present application, the feature density corresponding to the current frame surround view image in the density probability map is calculated according to the relative motion distance difference between the corresponding features of adjacent frames, the distance error, and the feature density of the corresponding features of the previous frame.

[0016] In some embodiments of the present application, the density probability map of the current frame surround view image is calculated according to at least one frame surround view image before the current frame, which includes:

[0017] According to the acquisition time, align each feature in the previous frame surround view image to the current time;

[0018] Calculate the relative motion distance difference and the distance error between each feature in the aligned previous frame surround view image and the corresponding feature in the current frame surround view image;

[0019] Calculate the density probability map of the current frame surround view image according to the relative motion distance difference between the corresponding features of the previous frame and the current frame surround view image, the distance error, and the feature density of the corresponding features of the previous frame surround view image.

[0020] In some embodiments of the present application, the feature density corresponding to the current frame surround view image in the density probability map is calculated according to the relative motion distance difference between the corresponding features of the previous frame and the current frame surround view image, the distance error, and the feature density of the corresponding features of the previous frame surround view image. According to the following formula:

[0021]

[0022] Wherein, Q dis is the relative motion distance difference between the corresponding features label of the previous frame and the current frame surround view image, Q miss is the distance error, is the feature density of the corresponding features label of the previous frame surround view image.

[0023] In some embodiments of the present application, the laser point cloud at the feature where the feature density meets the dynamic obstacle density range is filtered based on the density probability map, which includes:

[0024] Convert the current frame surround view image to the laser coordinate system to filter the laser point cloud at the feature where the feature density meets the dynamic obstacle density range based on the density probability map.

[0025] According to another aspect of the present application, a point cloud filtering device is also provided, comprising:

[0026] a horizontal image acquisition module, configured to acquire horizontal images collected by visual sensors with horizontal view angles;

[0027] a top view angle conversion module, configured to convert the horizontal images into top view images with top view angles;

[0028] a surround view splicing module, configured to splice the top view images into a surround view image according to the view angles of the visual sensors;

[0029] a concentration probability map calculation module, configured to calculate a concentration probability map of a current frame of the surround view image according to at least one frame of the surround view image before the current frame, wherein a feature concentration corresponding to the current frame of the surround view image in the concentration probability map is used to indicate whether the corresponding feature is a dynamic obstacle;

[0030] a laser point cloud filtering module, configured to filter laser point clouds at features with a feature concentration within a dynamic obstacle concentration range based on the concentration probability map.

[0031] According to another aspect of the present application, an electronic device is also provided, comprising a processor, and a storage medium having a computer program stored thereon, wherein the computer program is executed by the processor to perform the steps as described above.

[0032] According to another aspect of the present application, a storage medium is also provided, having a computer program stored thereon, wherein the computer program is executed by a processor to perform the steps as described above.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] By converting the horizontal images collected by the visual sensors into top view images and then splicing, a concentration probability map corresponding to the spliced surround view image is calculated, so as to obtain the probability of each feature being a dynamic obstacle. Based on the calculated concentration probability map, laser point clouds at features with a feature concentration within a dynamic obstacle concentration range can be filtered, so as to filter the dynamic obstacles of the laser point clouds. Thus, the present application can realize the filtering of the laser point clouds without a large amount of laser point clouds for object classification, and the concentration probability map filters the laser point clouds based on probability, rather than the result of the frame difference of the current frame with complete confidence, which has certain anti-interference effect on the subsequent semantic segmentation of the top view, so that the filtered laser point clouds can obtain higher accuracy when performing vehicle positioning and map construction. The present application is particularly suitable for scenes with many dynamic obstacles, so as to avoid the interference of the dynamic obstacles on the positioning of the vehicle. BRIEF DESCRIPTION OF DRAWINGS

[0035] The above and other features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

[0036] Figure 1 A flow chart of a point cloud filtering method according to an embodiment of the present application is shown;

[0037] Figure 2 A schematic diagram of a top view image of a top view perspective obtained by transformation using a transformer model according to an embodiment of the present application is shown;

[0038] Figure 3 A schematic diagram of a top view image of a top view perspective obtained by transformation using a homography matrix according to an embodiment of the present application is shown;

[0039] Figure 4 A schematic diagram of surround view images aligned according to time of acquisition according to an embodiment of the present application is shown;

[0040] Figure 5 A flow chart of a point cloud filtering method according to a specific embodiment of the present application is shown;

[0041] Figure 6 A block diagram of a point cloud filtering apparatus according to an embodiment of the present application is shown;

[0042] Figure 7 A schematic diagram of a computer readable storage medium in an exemplary embodiment of the present disclosure is shown schematically;

[0043] Figure 8 A schematic diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0044] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the inventive aspects to those skilled in the art. Features described in the description, examples, or claims that are unified under a single header are not necessarily mutually exclusive.

[0045] In addition, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure. The same reference numbers in different drawings represent the same or similar elements.

[0046] To solve the defects of the prior art, the present application provides a point cloud filtering method. See below Figure 1 , Figure 1 A flowchart of the point cloud filtering method according to an embodiment of the present application is shown. Figure 1 The following steps are as follows:

[0047] Step S110: Obtain horizontal images collected by visual sensors with multiple perspectives in the horizontal direction.

[0048] Step S120: Convert each of the horizontal images into a top-view image with a top-view perspective.

[0049] Step S130: Stitch multiple top-view images into a surround-view image according to the perspectives of the visual sensors.

[0050] Step S140: Calculate a density probability map of the surround-view image of the current frame based on at least one surround-view image of a previous frame, wherein the feature density corresponding to the surround-view image of the current frame in the density probability map is used to indicate whether the corresponding feature is a dynamic obstacle.

[0051] Step S150: Filter laser point clouds at features with feature densities conforming to a dynamic obstacle density range based on the density probability map.

[0052] In the point cloud filtering method provided by the present application, the horizontal images collected by the visual sensors are converted into top-view images and then stitched to calculate a density probability map corresponding to the stitched surround-view image, so as to obtain the probability that each feature is a dynamic obstacle. Based on the calculated density probability map, laser point clouds at features with feature densities conforming to a dynamic obstacle density range can be filtered, thereby filtering dynamic obstacles in the laser point clouds. Thus, the present application can realize filtering of laser point clouds without a large amount of laser point clouds for object classification, and the density probability map filters laser point clouds based on probability, rather than the result of frame difference of the current frame with complete confidence, which has certain anti-interference effect on subsequent semantic segmentation of the top-view image, so that the filtered laser point clouds can obtain higher accuracy when performing vehicle positioning and map construction. The present application is particularly suitable for scenes with many dynamic obstacles, thereby avoiding interference of dynamic obstacles on vehicle positioning.

[0053] Specifically, the visual sensor can be a camera module, for example, and the horizontal field of view can have a certain amount of deviation, that is, there can be a deviation angle in the vertical direction based on the horizontal direction. Further, the images captured by the multiple visual sensors with horizontal field of view can cover a 360-degree range of the vehicle body, thereby facilitating the subsequent surround view stitching step. In some embodiments, one visual sensor can be provided in each of the four directions of the vehicle body. For example, one can be provided at each corner of the vehicle body, at the midpoint of each of the four sides of the vehicle, and the like. The present application is not limited in this regard, and the position of each visual sensor is not limited in this regard. At the same time, the position and parameters of each visual sensor are saved at the vehicle end and / or the server end so as to determine the field of view of each visual sensor in the subsequent surround view stitching step.

[0054] Specifically, step S120 can convert the horizontal image with horizontal field of view to the overhead image with overhead field of view by using a Transformer model. The Transformer model does not produce distortion compared to the conversion method using homography matrix, as shown in Figure 2 and Figure 3 Further, the overhead image can also be segmented in step S120 to obtain segmentation features and feature labels. Specifically, image segmentation can be implemented based on a deep learning model. The deep learning model can distinguish the pixel regions of different objects in the image. The deep learning model can include, but is not limited to, VGG16, FCN (Fully Convolutional Network), SegNet (Segmentation Network), DeepLab, and the like, and the present application is not limited in this regard. After segmenting the overhead image to obtain segmentation features, each segmentation feature can also be assigned a feature label. Various image samples of different objects can be used for learning, so that the feature labels of each feature region (segmentation feature) can be identified during or after segmentation. Further, since the number of image samples of different objects is large, it is easier to train than laser point cloud to identify the object categories of each feature region (segmentation feature), thereby assigning the corresponding feature labels.

[0055] In some embodiments, after the segmentation of the overhead image in step S120, the segmentation features and feature labels are obtained, a preliminary screening of the dynamic obstacles can be performed based on the feature labels. For example, for the segmentation features with feature labels of pedestrians, vehicles, etc., they are necessarily dynamic obstacles, and thus can be marked in advance without subsequent calculation of feature concentration, and can be filtered in the laser point cloud. Specifically, in this embodiment, the preliminary screening of the feature labels can be performed to filter the segmentation features with feature labels conforming to the preset dynamic obstacle feature labels directly at the corresponding segmentation feature positions of the laser point cloud, and to retain the segmentation features with feature labels not conforming to the preset dynamic obstacle feature labels for calculation of the concentration probability map and feature concentration in the subsequent steps to determine whether the segmentation features are dynamic obstacles by the calculation of the feature concentration. Further, in some variations, static feature labels such as plants, buildings, etc. can also be maintained, since these static features will not move, and thus the feature labels of the segmentation features are determined after the preliminary screening to avoid the calculation of the concentration probability map and feature concentration in the subsequent steps for the segmentation features with feature labels conforming to the static feature labels, and the laser point cloud corresponding to the segmentation features conforming to the static feature labels is retained without filtering. In this way, the calculation amount of the feature concentration is reduced, the filtering efficiency of the laser point cloud is improved, and the system load is reduced.

[0056] In some embodiments, after the segmentation of the overhead image in step S120, the feature labels of the segmentation features can be assigned to the filtered and retained laser point cloud after step S150. In this way, the object recognition and feature labeling of the laser point cloud are not required, the calculation amount of the feature labels of the laser point cloud is reduced, the object recognition efficiency of the laser point cloud is improved, and the system load is reduced.

[0057] Specifically, the concentration probability map includes each feature (segmentation feature) in the current frame surround view image and the feature concentration thereof. The feature concentration of the concentration probability map corresponding to the current frame surround view image can be calculated according to the relative motion distance difference between the corresponding features of adjacent frames, the distance error, and the feature concentration of the corresponding features of the previous frame. The relative motion distance difference can be used to indicate whether the corresponding features of adjacent frames are in a motion state. The distance error can be set to select a permissible distance error and can reduce the error caused by deep learning processing. The feature concentration of the corresponding features of the previous frame is used to provide a reference for whether the corresponding features are in a motion state in the previous frame. In this way, the feature concentration corresponding to the current frame surround view image in the concentration probability map is calculated according to the relative motion distance difference between the corresponding features of adjacent frames, the distance error, and the feature concentration of the corresponding features of the previous frame, so that the calculated feature concentration can be used to indicate whether the corresponding feature is a dynamic obstacle.

[0058] Specifically, calculating the density probability map of the current frame's surrounding image based on at least one previous frame's surrounding image may include the following steps: aligning each feature in the previous frame's surrounding image to the current time based on the acquisition time; calculating the relative motion distance difference and distance error between each feature in the aligned previous frame's surrounding image and the corresponding feature in the current frame's surrounding image; and calculating the density probability map of the current frame's surrounding image based on the relative motion distance difference, distance error, and feature concentration of the corresponding feature in the previous frame's surrounding image.

[0059] Specifically, the concentration probability map corresponds to the feature concentration of the current frame's circular image. It can be calculated using the following formula:

[0060]

[0061] Among them, Q dis Q represents the difference in relative motion distance between the corresponding feature labels of the panoramic images in the previous and current frames. miss For distance error, This represents the feature concentration of the corresponding feature label from the previous frame of the panoramic image. Therefore, this application can obtain multiple previous frames of panoramic images at the current time, and use the above formula to sequentially calculate the motion state of the feature over continuous time, starting from the first frame image.

[0062] Furthermore, an initial value can be set for the feature concentration of the first frame image. This initial value can be set as needed, and this application is not limited to it. For example, in setting... After the initial value, it can be based on The relative motion distance difference and distance error of the feature labels in the first and second frames are calculated to obtain... Then, based on The relative motion distance difference and distance error of the feature labels in the second and third frames of the image were calculated. This process continues until the current frame is calculated.

[0063] Furthermore, when calculating the feature concentration of the current frame surrounding image in the concentration probability map, the number of surrounding images prior to the current time can be determined to balance the calculation speed of the feature concentration of the current frame surrounding image in the concentration probability map and whether the continuous time for determining whether the corresponding feature is in motion is long enough. In some embodiments, the number of surrounding images prior to the current time can be manually set. In other embodiments, different numbers of surrounding images prior to the current time can be used for testing to determine whether the feature concentration of the current frame surrounding image in the concentration probability map calculated with different numbers can accurately indicate whether the feature is a moving obstacle, thereby determining the number of surrounding images prior to the current time based on the calculation accuracy. In still other embodiments, the number of surrounding images prior to the current time can be predicted based on an artificial intelligence model. Many more variations are possible in this application, which will not be elaborated here.

[0064] Furthermore, the distance error can be set as needed. In some variations, the distance error can also be calculated based on the mode of the error pixels. This application can implement even more variations, which will not be elaborated here.

[0065] Furthermore, the dynamic obstacle concentration range can be set as needed. For example, the dynamic obstacle concentration range can be less than 1. This application is not limited to this; the dynamic obstacle concentration range can be less than 2, 3, etc. In some specific implementations, the dynamic obstacle concentration range can be set synchronously with the number of multi-frame surround view images before the current time to be acquired. Since the characteristic concentrations of the corresponding current frame surround view image are different in the output concentration probability map of different numbers of multi-frame surround view images before the current time, the dynamic obstacle concentration range can be determined based on the different characteristic concentrations. This application can implement more variations, which will not be elaborated here.

[0066] Specifically, the relative motion distance (X1, X2, ... X1) of the images captured by the visual sensors at each moment can be determined based on the vehicle's movement. t ), where X1, X2, ... X t These represent the distances of relative motion between time points 1, 2, ..., t and time point t, respectively, thus enabling the calculation based on... Figure 3The method shown allows each frame of the surround view image to be aligned to the current frame in time. Therefore, the relative motion distance difference between each feature in the previous frame and its corresponding feature in the current frame can be calculated based on the aligned surround view image. For example, if a feature is stationary, its position remains unchanged in each aligned frame, resulting in a relative motion distance difference of 0. If a feature is stationary, its position changes, allowing the distance between the same feature in adjacent frames to be used as the relative motion distance difference. Without alignment, if a vehicle is moving, features that are not stationary will experience distance differences between frames due to their relative movement to the vehicle. These distance differences are not caused by the feature's own movement, making it difficult to directly calculate the distance difference caused by the feature's movement based on the distance between corresponding features in the image. The surround view image alignment at each time step removes the influence of vehicle movement on the features, thus obtaining an accurate relative motion distance difference. In some variations, this application may omit alignment and remove the relative movement distance caused by vehicle travel when calculating the relative motion distance difference of corresponding features in adjacent frames. This application can implement many more variations, which will not be elaborated here.

[0067] Specifically, step S150 may further include the following step: transforming the current frame panoramic image to a laser coordinate system, so as to filter the laser point cloud at features whose feature concentrations conform to the dynamic obstacle concentration range based on the concentration probability map. Thus, the panoramic image and the laser point cloud can be placed in the same coordinate dimension to facilitate the determination of corresponding features.

[0068] See below. Figure 5 , Figure 5 A flowchart of a point cloud filtering method according to a specific embodiment of the present invention is shown. Figure 5 The following steps are shown:

[0069] Step S310: Acquire horizontal images from multiple visual sensors with a horizontal perspective.

[0070] Step S320: Convert each of the horizontal images into a top-down view image, and segment the top-down view image to obtain segmentation features and feature labels.

[0071] Step S330: Based on the viewing angles of each of the vision sensors, stitch together the multiple top-view images into a surround-view image.

[0072] Step S340: Calculate the density probability map of the current frame's surrounding view image based on at least one previous frame's surrounding view image. The feature density corresponding to the current frame's surrounding view image in the density probability map is used to indicate whether the corresponding feature is a dynamic obstacle.

[0073] Step S350: Transform the current frame's panoramic image to the laser coordinate system.

[0074] Step S360: Acquire the laser point cloud collected by the laser sensor. The laser sensor and the vision sensor are mounted on the same vehicle.

[0075] Step S370: Cluster and segment the laser point cloud to obtain multiple point cloud features so as to correspond them with the features of the panoramic image.

[0076] Step S380: Based on the concentration probability map, filter the laser point cloud at the feature concentration that matches the dynamic obstacle concentration range.

[0077] Step S390: Assign the feature labels of the segmentation features to the filtered and retained laser point cloud.

[0078] Specifically, after clustering in step S360, the laser point cloud yields multiple point cloud features. After filtering in step S380, some point cloud features are retained, and these features correspond to the segmentation features from a top-down perspective. In some embodiments, since both point cloud features and segmentation features have regional ranges, the correspondence between them can be determined based on the overlap of these ranges. For example, an overlap threshold can be set; when the overlap exceeds this threshold, the correspondence is determined, and the feature labels of the segmentation features can be assigned to the filtered and retained laser point cloud. In other embodiments, since both point cloud features and segmentation features have regional ranges, the correspondence can also be determined based on the distance between the centers of these regions. For example, a distance threshold can be set; when the distance between the centers of the point cloud features and the segmentation features is less than this threshold, the correspondence is determined, and the feature labels of the segmentation features can be assigned to the filtered and retained laser point cloud. This application can implement many more variations in determining the correspondence between point cloud features and segmentation features, which will not be elaborated upon here.

[0079] The above are merely several specific implementations of the point cloud filtering method of the present invention. Each implementation can be implemented independently or in combination, and the present invention is not intended to limit it. Furthermore, the flowchart of the present invention is merely illustrative, and the execution order between the steps is not limited thereto. The splitting, merging, sequential exchange, and other synchronous or asynchronous execution methods of the steps are all within the protection scope of the present invention.

[0080] The present invention also provides a point cloud filtering device. Figure 6A block diagram of a point cloud filtering device according to an embodiment of the present invention is shown. The point cloud filtering device 400 includes a horizontal image acquisition module 410, a top-view conversion module 420, a panoramic stitching module 430, a density probability map calculation module 440, and a laser point cloud filtering module 450.

[0081] The horizontal image acquisition module 410 is used to acquire horizontal images collected by multiple visual sensors with a horizontal viewing angle;

[0082] The top-view conversion module 420 is used to convert each of the horizontal images into a top-view image from a top-view perspective;

[0083] The surround view stitching module 430 is used to stitch together multiple top view images into a surround view image according to the viewing angle of each of the vision sensors;

[0084] The concentration probability map calculation module 440 is used to calculate the concentration probability map of the current frame's surrounding view image based on at least one previous frame's surrounding view image. The feature concentration corresponding to the current frame's surrounding view image in the concentration probability map is used to indicate whether the corresponding feature is a dynamic obstacle.

[0085] The laser point cloud filtering module 450 is used to filter laser point clouds where the characteristic concentration conforms to the dynamic obstacle concentration range based on the concentration probability map.

[0086] In the point cloud filtering device provided by this invention, the horizontal image acquired by the visual sensor is converted into a top-down view image and then stitched together. A density probability map corresponding to the stitched panoramic view image is calculated to obtain the probability that each feature is a dynamic obstacle. Based on the calculated density probability map, laser point clouds at features whose concentrations match the dynamic obstacle concentration range can be filtered, thus filtering the laser point cloud for dynamic obstacles. Therefore, this application achieves laser point cloud filtering without requiring a large amount of laser point cloud data for object classification. Furthermore, the density probability map is used for laser point cloud filtering based on probability, rather than completely relying on the frame difference of the current frame, which provides some resistance to interference in subsequent semantic segmentation effects such as top-down views. Thus, the filtered laser point cloud achieves higher accuracy in vehicle localization and map building. This application is particularly suitable for scenarios with many dynamic obstacles, thereby avoiding interference from dynamic obstacles in vehicle localization.

[0087] Figure 6 The point cloud filtering device 400 provided by this invention is merely illustrated schematically. Without departing from the inventive concept, the splitting, merging, and addition of modules are all within the scope of protection of this invention. The point cloud filtering device 400 provided by this invention can be implemented by software, hardware, firmware, plugins, and any combination thereof; this invention is not limited thereto.

[0088] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon a computer program that, when executed by, for example, a processor, can implement the steps of the point cloud filtering method described in any of the above embodiments. In some possible implementations, various aspects of the invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the point cloud filtering method section of this specification.

[0089] refer to Figure 7 As shown, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0090] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0091] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the tenant's computing device, partially on the tenant's device, as a standalone software package, partially on the tenant's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the tenant's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0093] In exemplary embodiments of this disclosure, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the steps of the point cloud filtering method described in any of the above embodiments by executing the executable instructions.

[0094] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0095] The following reference Figure 8 To describe an electronic device 600 according to this embodiment of the present invention. Figure 8 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0096] like Figure 8 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0097] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the point cloud filtering method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform, as follows: Figure 1The steps are shown in the figure.

[0098] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0099] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0100] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0101] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable tenants to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the above-described point cloud filtering method according to the embodiments of this disclosure.

[0103] Compared with the prior art, the advantages of this invention are:

[0104] By converting horizontal images acquired by a visual sensor into top-down images and then stitching them together, a density probability map corresponding to the stitched panoramic image is calculated. This yields the probability that each feature is a dynamic obstacle. Based on the calculated density probability map, laser point clouds containing features whose concentrations fall within the dynamic obstacle concentration range can be filtered out, thus filtering dynamic obstacles from the laser point cloud. Therefore, this application achieves laser point cloud filtering without requiring a large amount of laser point cloud data for object classification. Furthermore, the density probability map is used for laser point cloud filtering based on probability, rather than completely relying on the frame difference of the current frame. This provides some resistance to interference in subsequent semantic segmentation, such as top-down views, resulting in higher accuracy for vehicle localization and map building using the filtered laser point cloud. This application is particularly suitable for scenarios with many dynamic obstacles, thereby avoiding interference from dynamic obstacles in vehicle localization.

[0105] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A point cloud filtering method, characterized in that, include: Acquire horizontal images from multiple horizontal-oriented visual sensors; The horizontal images are converted into top-view images from a top-down perspective; Based on the perspective of each of the aforementioned visual sensors, the multiple top-view images are stitched together to form a circumferential view image; The density probability map of the current frame's surrounding image is calculated based on at least one previous frame's surrounding image. The feature concentration in the density probability map corresponding to the current frame's surrounding image is used to indicate whether the corresponding feature is a dynamic obstacle. The feature concentration in the density probability map corresponding to the current frame's surrounding image is calculated based on the relative motion distance difference and distance error between corresponding features in adjacent frames, as well as the feature concentration of the corresponding feature in the previous frame. The calculation of the density probability map of the current frame's surrounding image based on at least one previous frame's surrounding image includes: aligning each feature in the previous frame's surrounding image to the current time based on the acquisition time; calculating the relative motion distance difference and distance error between each feature in the aligned previous frame's surrounding image and the corresponding feature in the current frame's surrounding image; and calculating the density probability map of the current frame's surrounding image based on the relative motion distance difference and distance error between corresponding features in the previous and current frames' surrounding images, as well as the feature concentration of the corresponding feature in the previous frame's surrounding image. The feature concentration in the density probability map corresponding to the current frame's surrounding image... Calculate using the following formula: Among them, Q dis Q represents the difference in relative motion distance between the corresponding feature labels of the panoramic images in the previous and current frames. miss For distance error, The feature concentration of the corresponding feature label of the previous frame's panoramic image; Based on the concentration probability map, the laser point cloud is filtered to find features whose concentrations match the dynamic obstacle concentration range.

2. The point cloud filtering method as described in claim 1, characterized in that, The process of converting each of the horizontal images into a top-down view includes: The top-view image is segmented to obtain segmentation features and feature labels.

3. The point cloud filtering method as described in claim 2, characterized in that, The feature concentration corresponding to the current frame panoramic image in the concentration probability map is used to indicate the probability that the corresponding segmentation feature is a dynamic obstacle. The step of filtering the laser point cloud at features whose feature concentrations match the dynamic obstacle concentration range based on the concentration probability map includes: The feature labels of the segmentation features are assigned to the filtered and retained laser point cloud.

4. The point cloud filtering method as described in claim 1, characterized in that, The step of filtering the laser point cloud at features whose characteristic concentrations match the dynamic obstacle concentration range based on the concentration probability map includes: The current frame panoramic image is transformed to the laser coordinate system so that, based on the concentration probability map, the laser point cloud at the feature concentration that conforms to the dynamic obstacle concentration range is filtered.

5. A point cloud filtering device, characterized in that, include: The horizontal image acquisition module is used to acquire horizontal images collected by multiple visual sensors with a horizontal perspective. The top-view conversion module is used to convert each of the horizontal images into a top-view image from a top-view perspective. The surround view stitching module is used to stitch together multiple top view images into a surround view image according to the viewing angles of each of the vision sensors; The concentration probability map calculation module is used to calculate the concentration probability map of the current frame's surrounding image based on at least one previous frame's surrounding image. The feature concentration in the concentration probability map corresponding to the current frame's surrounding image is used to indicate whether the corresponding feature is a dynamic obstacle. The feature concentration in the concentration probability map corresponding to the current frame's surrounding image is calculated based on the relative motion distance difference and distance error between corresponding features in adjacent frames, as well as the feature concentration of the corresponding feature in the previous frame. The calculation of the concentration probability map of the current frame's surrounding image based on at least one previous frame's surrounding image includes: aligning each feature in the previous frame's surrounding image to the current time based on the acquisition time; calculating the relative motion distance difference and distance error between each feature in the aligned previous frame's surrounding image and the corresponding feature in the current frame's surrounding image; and calculating the concentration probability map of the current frame's surrounding image based on the relative motion distance difference and distance error between corresponding features in the previous and current frames' surrounding images, as well as the feature concentration of the corresponding feature in the previous frame's surrounding image. Calculate using the following formula: Among them, Q dis Q represents the difference in relative motion distance between the corresponding feature labels of the panoramic images in the previous and current frames. miss For distance error, The feature concentration of the corresponding feature label of the previous frame's panoramic image; The laser point cloud filtering module is used to filter laser point clouds where the characteristic concentration conforms to the dynamic obstacle concentration range based on the concentration probability map.

6. An electronic device, characterized in that, The electronic device includes: processor; A storage medium having a computer program stored thereon, the computer program being executed by the processor to perform the point cloud filtering method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the point cloud filtering method as described in any one of claims 1 to 4.

Images