Dynamic point cloud culling method and device based on multi-modal data perception, equipment and storage medium

Abstract

The application relates to the technical field of image processing, in particular to a dynamic point cloud removal method and device based on multi-modal data perception, equipment and a storage medium. The method comprises the following steps: acquiring a single-frame point cloud and determining a corresponding image to be processed; inputting the image to be processed into a model after camera distortion correction to perform semantic segmentation and obtaining a mask image; acquiring a previous frame point cloud and a next frame point cloud closest to the image acquisition time of the image to be processed; performing linear transition on the single-frame point cloud based on the previous frame point cloud, the next frame point cloud and the image acquisition time; projecting the single-frame point cloud after linear transition to the mask image, and extracting dynamic object point cloud and static point cloud; performing clustering analysis on the dynamic object point cloud to extract ground point cloud; and combining the static point cloud and the ground point cloud to obtain three-dimensional point cloud after dynamic point cloud removal. The application can confirm the specific details of the object in the fusion point cloud data, and can meet the beneficial effect of the increasingly improved scene understanding requirement.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a method, apparatus, device and storage medium for dynamic point cloud removal based on multimodal data perception. Background Art

[0002] Point cloud data refers to a collection of vectors in a three-dimensional coordinate system. Each point contains three-dimensional coordinates and can carry other information about its attributes, such as color and reflectivity. This highly accurate, high-resolution, and high-dimensional geometric information can intuitively represent the shape, surface, and texture of objects in space. With the continuous development of technology, the application scope of point cloud data is becoming increasingly broad, with broad application prospects and market demand in areas such as autonomous driving, intelligent robotics, and smart city construction.

[0003] In practical applications, eliminating dynamic point clouds is one of the indispensable technical means in the fields of autonomous driving, mobile robots, etc. It not only helps to improve the system's positioning accuracy and map construction quality, but also reduces computing costs, enhances environmental perception and the ability to adapt to complex scenarios, thereby providing users with a safer, more comfortable and intelligent travel experience.

[0004] However, currently, point cloud data is mainly fused with the results of target detection, that is, a rectangular frame is used in the detection results to roughly frame the position and range of the target object. The expression of information such as the internal structure, shape details and precise relationship with the surrounding environment of the object is relatively limited, and its complex shape cannot be accurately represented. It is difficult to meet the increasingly accurate scene understanding needs, such as in the technical fields of robot navigation, high-precision map drawing and building information modeling, and it is difficult to play an ideal role.

[0005] Based on this, how to confirm the specific details of objects in the fused point cloud data to meet the increasingly accurate scene understanding needs is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the present application provides a dynamic point cloud elimination method, device, equipment and storage medium based on multimodal data perception, which can realize the beneficial effect of confirming the specific details of objects in the fused point cloud data to meet the increasingly accurate scene understanding needs.

[0007] The technical solution adopted by this application to solve its technical problems is:

[0008] In a first aspect, the present application provides a dynamic point cloud removal method based on multimodal data perception, the method comprising:

[0009] Acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud;

[0010] Performing camera distortion correction on the image to be processed, and inputting the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed by the image semantic segmentation model to obtain a mask image;

[0011] Acquire a previous frame point cloud and a subsequent frame point cloud closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time;

[0012] Projecting the linearly transitioned single-frame point cloud onto the mask image, extracting a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud;

[0013] performing cluster analysis on the dynamic object point cloud to extract the ground point cloud;

[0014] The static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0015] Optionally, the step of performing linear transition on the single-frame point cloud based on the previous frame point cloud, the next frame point cloud and the image acquisition time includes:

[0016] Obtaining a front frame pose and posture corresponding to the front frame point cloud and a rear frame pose and posture corresponding to the rear frame point cloud, and obtaining a front frame image acquisition time corresponding to the front frame point cloud and a rear frame image acquisition time corresponding to the rear frame point cloud;

[0017] Determining a time ratio based on the previous frame acquisition time, the next frame acquisition time, and the image acquisition time;

[0018] According to the previous frame pose, the subsequent frame pose and the time ratio, a linear interpolation calculation is performed on the pose corresponding to the single frame point cloud to obtain a transition pose, where the transition pose is the pose corresponding to the image acquisition time.

[0019] Optionally, the step of projecting the linearly transitioned single-frame point cloud onto the mask image and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud includes:

[0020] Obtaining an extrinsic parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition pose matrix corresponding to the transition pose and posture;

[0021] Determine a transformation matrix based on the extrinsic parameter matrix, the transition pose matrix, and the posture pose corresponding to the single-frame point cloud, and transform each point in the single-frame point cloud from the radar coordinate system to the camera coordinate system using the transformation matrix;

[0022] Based on a preset camera intrinsic parameter matrix, project each point corresponding to the single-frame point cloud in the camera coordinate system onto the mask image to obtain multiple pixel coordinates;

[0023] A pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

[0024] Optionally, the step of performing camera distortion correction on the image to be processed includes:

[0025] Performing radial distortion correction and tangential distortion correction on each image point of the image to be processed;

[0026] The input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

[0027] Optionally, the step of obtaining a pixel value corresponding to each pixel coordinate in the mask image to extract the dynamic object point cloud and the static point cloud according to each pixel value includes:

[0028] Determining a correspondence between each pixel coordinate and the mask image according to each pixel coordinate;

[0029] By means of the correspondence between each pixel coordinate and the mask image, the pixel coordinates of all pixels whose corresponding pixel values ​​in the mask image are non-zero are obtained by screening;

[0030] The dynamic object point cloud is extracted based on all the filtered pixel coordinates, and the static point cloud is determined.

[0031] Optionally, the step of determining, according to each pixel coordinate, a correspondence between each pixel coordinate and the mask image includes:

[0032] Determining whether each of the pixel coordinates is an integer pixel coordinate;

[0033] If the pixel coordinates are integer pixel coordinates, determining a correspondence between the pixel coordinates and the mask image according to the pixel coordinates;

[0034] Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates;

[0035] An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

[0036] Optionally, the step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud includes:

[0037] Determine any point in the dynamic object point cloud as a starting point, and add the starting point to a preset cluster set;

[0038] Calculating the Euclidean distance between each unvisited point in the dynamic object point cloud and all points in the cluster set one by one;

[0039] Comparing the obtained Euclidean distance of each point with a preset distance threshold, so as to add points whose Euclidean distance exceeds the distance threshold into the cluster set, until all points in the dynamic object point cloud are visited;

[0040] The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

[0041] In a second aspect, the present application provides a dynamic point cloud removal device based on multimodal data perception, comprising:

[0042] A data acquisition module, configured to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud;

[0043] A semantic segmentation module is used to perform camera distortion correction on the image to be processed, and input the corrected image to be processed into a pre-trained image semantic segmentation model to perform semantic segmentation on the corrected image to be processed by the image semantic segmentation model to obtain a mask image;

[0044] a linear transition module, configured to obtain a previous frame point cloud and a subsequent frame point cloud closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time;

[0045] a point cloud extraction module, configured to project the linearly transitioned single-frame point cloud onto the mask image, and extract a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud;

[0046] a ground point cloud determination module, configured to perform cluster analysis on the dynamic object point cloud to extract the ground point cloud;

[0047] The point cloud data fusion module is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0048] In a third aspect, the present application provides an electronic device, comprising:

[0049] one or more processors;

[0050] one or more memories;

[0051] and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the above method.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program or instruction is stored. When the program or instruction is executed, the above method is implemented.

[0053] The technical solution provided by this application is as follows: radar and camera are used to collect data of the same scene respectively, wherein the radar is used to obtain a three-dimensional point cloud and the camera is used to collect a two-dimensional image. The data of the same scene are collected simultaneously by the camera and radar, one frame of the point cloud is determined to be the point cloud to be processed, and the image to be processed that is closest to the point cloud to be processed in time and space is found; further, the image to be processed is subjected to camera distortion correction, and the corrected image to be processed is input into the model for semantic segmentation to obtain a mask image; further, based on the trajectory point pose of the previous frame point cloud and the trajectory point pose of the subsequent frame point cloud, A linear transition is performed between the acquisition time of the image to be processed to transition the trajectory point pose corresponding to the single-frame point cloud to the interpolated pose; further, based on the correspondence between the single-frame point cloud and the mask image, the single-frame point cloud is projected onto the mask image to separate the dynamic object point cloud corresponding to the mask image, and the rest of the single-frame point cloud is determined to be a static point cloud; further, a cluster analysis is performed on the dynamic object point cloud to screen out the ground point cloud in the dynamic object point cloud; finally, the ground point cloud and the static point cloud are combined to obtain the point cloud with the dynamic object point cloud removed after the fusion point cloud data.

[0054] By adopting the above technical solution, this application has at least the following beneficial effects:

[0055] 1. This application uses semantic segmentation to detect objects. Compared to traditional target detection methods, semantic segmentation provides a more refined representation of spatial information, assigning semantic labels to each point in the scene, providing more detailed spatial information, and processing and analyzing all point cloud data, fully mining the information in the data and avoiding information omissions. It can be seen that the dynamic object point cloud separated by semantic segmentation has a richer representation of information such as the object's internal structure, shape details, and precise relationship with the surrounding environment. This can achieve the beneficial effect of confirming the specific details of the object in the fused point cloud data, thereby meeting the increasingly precise needs of scene understanding.

[0056] 2. This application uses multi-sensor fusion calibration technology that combines cameras and radars to make up for the shortcomings of a single sensor, thereby improving the overall perception capability and further improving the accuracy and completeness of point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the process of the dynamic point cloud elimination method based on multimodal data perception provided by the embodiment of the present application

[0058] Figure 2 This is a virtual structural diagram of a dynamic point cloud removal device based on multimodal data perception provided by this application;

[0059] Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application is further described below with reference to the accompanying drawings and examples.

[0061] The following will clearly and completely describe the concept, specific structure and technical effects of this application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0062] Reference Figure 1 , Figure 1 This is a flow chart of a dynamic point cloud removal method based on multimodal data perception provided by an embodiment of the present application, which specifically includes the following steps, which are described in detail below:

[0063] In step S1, a single-frame point cloud is acquired, and an image to be processed corresponding to the single-frame point cloud is determined.

[0064] Specifically, a single-frame point cloud acquires point cloud data using a three-dimensional point cloud acquisition method such as a lidar. The image to be processed is a two-dimensional image of the target scene, acquired by an image acquisition device such as a camera, and containing dynamic objects, such as vehicles and pedestrians. In an embodiment of the present application, multiple sensors (camera, radar) are used to simultaneously capture the same scene multiple times at the same frequency, obtaining multiple images captured by the camera and multiple-frame point clouds captured by the radar, wherein each point cloud corresponds to a SLAM trajectory point pose. A single-frame point cloud containing a dynamic object is selected from the acquired multi-frame point cloud, and then a to-be-processed image that is aligned in time and space is found based on the acquisition time of the single-frame point cloud.

[0065] In step S2, camera distortion correction is performed on the image to be processed, and the corrected image to be processed is input into a pre-trained image semantic segmentation model, so that semantic segmentation is performed on the corrected image to be processed by the image semantic segmentation model to obtain a mask image.

[0066] Specifically, an image semantic segmentation model is a model with semantic segmentation capabilities. It can segment an image into regions with specific semantic meanings and identify the semantic category of each region, such as pedestrians and vehicles. A mask image is a binary image used to represent different categories of regions in an image. In this application, in the mask image obtained by segmentation after semantic segmentation, dynamic objects can be marked as black, and static backgrounds can be marked as white.

[0067] More specifically, considering the camera distortion, before obtaining the mask image (that is, before the image to be processed is input into the image semantic segmentation model), the distortion parameters need to be used to correct the camera distortion of the image to be processed, including:

[0068] Perform radial distortion correction and tangential distortion correction on each image point of the image to be processed.

[0069] Specifically, the camera distortion usually includes radial distortion and tangential distortion. Let the uncorrected point in the camera coordinate system be , the corrected point is , in which case first get the normalized coordinates , , distortion correction is performed using the normalized coordinates using the radial distortion calculation formula and the tangential distortion calculation formula. The radial distortion calculation formula is:

[0070] ;

[0071] ;

[0072] ;

[0073] Wherein, p1 and p2 in the above radial distortion calculation formula are radial distortion parameters.

[0074] The calculation formula for tangential distortion is:

[0075] ;

[0076] ;

[0077] Wherein, p1 and p2 in the above tangential distortion calculation formula are tangential distortion parameters.

[0078] Furthermore, the input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

[0079] Specifically, the distortion-corrected point can be expressed as , this algorithm can complete the camera distortion correction of the image to be processed.

[0080] In step S3, the previous frame point cloud and the next frame point cloud closest to the image acquisition time of the image to be processed are obtained, and the single frame point cloud is linearly transitioned based on the previous frame point cloud, the next frame point cloud and the image acquisition time.

[0081] Specifically, in the embodiment of the present application, the laser radar frequency used is 10 Hz, and the image data acquisition frequency is also 10 Hz. However, due to various errors, the image to be processed acquired at the current time (any one of the multiple acquired images contains an image of a dynamic object) and the frame of point cloud closest to the acquisition time are not exactly the same time. Based on this, before the subsequent point cloud data fusion, it is necessary to find the two frames of point clouds closest to the current image acquisition time, and perform spatiotemporal unification and kinematic linear transition operations based on this to determine the single frame of point cloud corresponding to the acquisition time.

[0082] More specifically, the step of performing a linear transition on the single-frame point cloud based on the previous frame point cloud, the next frame point cloud, and the image acquisition time includes:

[0083] Obtain a front frame pose and posture corresponding to the front frame point cloud and a rear frame pose and posture corresponding to the rear frame point cloud, and obtain a front frame image acquisition time corresponding to the front frame point cloud and a rear frame image acquisition time corresponding to the rear frame point cloud.

[0084] Specifically, let the acquisition time of the image to be processed be , the corresponding acquisition time of the previous frame point cloud is , and the SLAM trajectory point posture corresponding to the previous frame point cloud is ,in Indicates the position coordinates, roll, pitch, and yaw identify the attitude angles of dynamic objects (such as vehicles); the corresponding acquisition time of the subsequent frame point cloud is , the corresponding SLAM trajectory point posture is .

[0085] Among them, the SLAM trajectory point posture refers to the three-dimensional position and orientation information of the robot or camera in space obtained by the Simultaneous Localization and Mapping (SLAM) technology. The posture is the information describing the position and orientation of the robot or camera in space, usually including three-dimensional coordinates. and rotation angles (pitch, yaw, roll).

[0086] Furthermore, a time ratio is determined based on the previous frame acquisition time, the next frame acquisition time and the image acquisition time.

[0087] Specifically, a first time difference with the previous frame is determined based on the previous frame acquisition time and the image acquisition time, and a second time difference with the next frame is determined based on the next frame acquisition time and the image acquisition time. and , and then calculate the total time difference based on the first time difference and the second time difference Then, based on the total time difference and the first time difference, the time ratio of the previous frame time difference to the total time difference is determined. .

[0088] Furthermore, according to the previous frame posture and position, the subsequent frame posture and position and the time ratio, a linear interpolation calculation is performed on the posture and position corresponding to the single frame point cloud to obtain a transition posture and position, and the transition posture and position is the posture and position corresponding to the image acquisition time.

[0089] Specifically, determine the position coordinates of any point in the point cloud , interpolate the position coordinates based on the time ratio obtained in the above steps, and the interpolated position coordinates are , similarly we can get , ;

[0090] Furthermore, for the posture and position corresponding to the interpolated position coordinates, including the interpolated roll angle , the interpolated pitch angle and the interpolated yaw angle , where the attitude angles roll, pitch, and yaw represent the rotation around the x-axis, y-axis, and z-axis respectively, and are used to describe the spatial attitude of the object.

[0091] Furthermore, each position coordinate in the single frame point cloud is interpolated to obtain all transition postures corresponding to the image acquisition time. , this pose can be converted into matrix M2, based on which the transformation matrix for converting from the radar coordinate system to the camera coordinate system can be calculated.

[0092] In step S4, the single-frame point cloud after linear transition is projected onto the mask image, and dynamic object point cloud and static point cloud are extracted from the single-frame point cloud, where the dynamic object point cloud includes a dynamic point cloud and a ground point cloud.

[0093] Specifically, the linearly transitioned single-frame point cloud is projected onto the pixel coordinate system of the mask image, and the point clouds of the dynamic objects and static parts in the mask image are classified and saved, thereby extracting the dynamic object point cloud and the static point cloud. Specifically, the pose corresponding to the single-frame point cloud is first determined as M0, and the pose transformation matrix obtained by linear interpolation of the pose corresponding to the single-frame point cloud is M2. Then, the extrinsic parameter matrix of the camera relative to the radar is defined as M1, and the transformation matrix is ​​defined as M3. The extrinsic parameter matrix of the camera relative to the radar can be obtained by calibrating the camera and radar. This calibration process can be implemented using MATLAB software.

[0094] More specifically, projecting the linearly transitioned single-frame point cloud onto the mask image, and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud specifically includes the following steps:

[0095] Obtain an extrinsic parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition pose matrix corresponding to the transition pose and posture.

[0096] Specifically, obtain the extrinsic parameter matrix M1 of the camera relative to the radar, and the matrix M2 converted from the transition posture obtained in step S3 above. Wherein, M2 is obtained by linear interpolation of the posture M0 corresponding to the single-frame point cloud.

[0097] Furthermore, a transformation matrix is ​​determined based on the extrinsic parameter matrix, the transition pose matrix and the posture pose corresponding to the single-frame point cloud, and each point in the single-frame point cloud is transformed from the radar coordinate system to the camera coordinate system through the transformation matrix.

[0098] Specifically, let the extrinsic parameter matrix of the camera coordinate system relative to the radar coordinate system be M1, and the attitude and pose matrix corresponding to the transition attitude and pose be M2 (that is, the matrix M2 obtained by linear interpolation of the attitude corresponding to the above point cloud). In this case, the transformation matrix M3 used to transform the coordinate system is:

[0099] At rest: ; In motion: ;

[0100] Among them, M3 is a matrix, 3 3's rotation matrix, 3 Specifically, the difference between the radar coordinate system and the camera coordinate system of the point cloud is an M3 matrix, that is, the point cloud can be converted to the camera coordinate system by multiplying M3 on the right, and associated with the camera. Conversion from radar coordinate system to camera coordinate system:

[0101] For example: for the point cloud coordinates in the radar coordinate system , converted to the coordinates of the camera coordinate system The formula is:

[0102]

[0103] At this point, the radar coordinate system can be obtained The corresponding point in the camera coordinate system .

[0104] Furthermore, based on a preset camera intrinsic parameter matrix, each point corresponding to the single-frame point cloud in the camera coordinate system is projected onto the mask image to obtain a plurality of pixel coordinates.

[0105] Specifically, the camera intrinsic parameter matrix is ​​a known preset value. Specifically, the known camera intrinsic parameter matrix is ,in and are the focal lengths of the camera in the x-axis and y-axis directions, and are the principal point coordinates of the image. Then, based on the point P in the camera coordinate system The calculation formula of the pixel coordinates (u, v) projected onto the image plane is:

[0106] ;

[0107] In summary, in this step, the points in the single frame point cloud are (radar coordinate system) converted to (camera coordinate system) and then converted to (u,v) pixel coordinates (pixel coordinate system).

[0108] Furthermore, a pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

[0109] Specifically, the pixel coordinates are indexed on the mask image to determine which pixel positions correspond to the valid region (generally, the pixel value is non-zero as a valid value), and if a pixel coordinate is valid in the mask image, the corresponding radar point cloud can be determined by indexing. Specifically, the calculated pixel coordinates (u, v) have corresponding pixel values in the single-frame point cloud, and each pixel coordinate corresponds to a point cloud part of a mask image in the single-frame point cloud. Based on this correspondence, the point cloud corresponding to the mask image part can be extracted from the single-frame point cloud using the pixel coordinates, thereby obtaining the dynamic object point cloud, and the remaining part is determined as the static point cloud.

[0110] The following describes the steps of obtaining the pixel value corresponding to each pixel coordinate in the mask image, and extracting the dynamic object point cloud and static point cloud according to each pixel value:

[0111] According to each pixel coordinate, the correspondence between each pixel coordinate and the mask image is determined.

[0112] According to the correspondence between each pixel coordinate and the mask image, all pixel coordinates corresponding to non-zero pixel values in the mask image are screened.

[0113] According to all the screened pixel coordinates, the dynamic object point cloud is extracted, and the static point cloud is determined.

[0114] Specifically, the position corresponding to the calculated pixel coordinates (u, v) can be indexed in the mask image, and based on whether the pixel value at this position is a valid value, generally a non-zero value, the point cloud corresponding to the pixel coordinate with a valid value can be determined as a dynamic object point cloud, thereby screening all dynamic object point clouds one by one, and the point cloud in other parts of the single-frame point cloud can be determined as a static point cloud. A data structure (including but not limited to a dictionary or a list) can be used to store the correspondence between the pixel coordinates and the single-frame point cloud index.

[0115] More specifically, the embodiments of the present application consider that the pixel coordinates can be integers or non-integers. Based on integer pixel coordinates, the index can be directly obtained, while based on non-integer pixel coordinates, the index can be obtained after processing. Specifically:

[0116] Determine whether each pixel coordinate is an integer pixel coordinate.

[0117] If the pixel coordinate is an integer pixel coordinate, the correspondence between the pixel coordinate and the mask image is determined according to the pixel coordinate.

[0118] Specifically, if the calculated pixel coordinate is an integer, i.e., an integer pixel coordinate, it can be directly used as an index to check whether the pixel value corresponding to the pixel coordinate in the mask image is a valid value. All valid values ​​are then determined and extracted one by one to obtain the entire point cloud corresponding to the integer pixel coordinate, that is, the point cloud of all dynamic objects corresponding to the integer pixel coordinate in a single-frame point cloud. The index of the radar point cloud corresponding to the integer pixel coordinate can be recorded using a data structure (such as a dictionary or list) to store the correspondence between the integer pixel coordinate and the single-frame point cloud index.

[0119] Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates;

[0120] An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

[0121] Specifically, if the pixel coordinates are non-integer, i.e., non-integer pixel coordinates, a method such as bilinear interpolation can be used to determine whether the pixel value in the mask image corresponding to the pixel position is valid. If the pixel value is valid, the corresponding point cloud is determined in the single-frame point cloud. This application takes the bilinear interpolation method as an example, specifically:

[0122] Obtain the four integer pixel coordinates closest to the pixel coordinate. That is, find the four integer pixel coordinates closest to the pixel coordinate;

[0123] Get the pixel values ​​corresponding to these four integer pixel coordinates from the mask image 、 、 、 , respectively 、 、 、 ;

[0124] Then, interpolation calculations are performed based on these four pixel values, including calculating the interpolation weight of each pixel value, and then obtaining the corresponding value of each non-integer pixel coordinate in the mask image based on the interpolation weight. The formula for calculating the interpolation weight is:

[0125] ;

[0126] ;

[0127] ;

[0128] .

[0129] The formula used to obtain the corresponding value of each non-integer pixel coordinate in the mask image based on the interpolation weight is:

[0130] ;

[0131] Similar to the point cloud index corresponding to the integer pixel coordinates mentioned above, it is possible to ensure that different non-integer pixel coordinates correspond to different index records by some means (for example, fine-tuning according to the decimal part of the pixel coordinates).

[0132] In summary, combining the above steps can filter out the corresponding position of the mask image part (that is, the part corresponding to the dynamic object in the image) in the single-frame point cloud, and based on this, obtain the dynamic object point cloud (the point cloud corresponding to the mask image) and the static point cloud (the point cloud remaining after removing the point cloud of the mask image).

[0133] In step S5, cluster analysis is performed on the dynamic object point cloud to extract the ground point cloud.

[0134] Specifically, the extracted dynamic object point cloud is a mixed point cloud of dynamic objects (such as vehicles) and the ground. To remove the point cloud containing the dynamic object, cluster analysis is required to separate the point clouds of each cluster within the mixed point cloud. In this embodiment, the Euclidean distance between points is calculated. If the Euclidean distance between two points is less than a preset threshold, the two points are considered to belong to the same point cloud.

[0135] More specifically, the step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud includes:

[0136] Determine any point in the dynamic object point cloud as a starting point, and add the starting point to a preset cluster set.

[0137] Specifically, let the input point cloud set be , where each point is The point cloud set is the point cloud set formed by the above-mentioned dynamic object point cloud, including mixed point clouds of multiple categories of point clouds. Then, select any point in the point cloud set as the starting point, or select the point with the smallest value as the starting point. Assume that the starting point is .

[0138] The Euclidean distance between each unvisited point in the dynamic object point cloud and all points in the cluster set is calculated one by one.

[0139] Specifically, we can create an empty cluster set C = {} and a visited point set V = {}, and determine the starting point Add them to the cluster set C and the visited set V respectively. Then, for each unvisited point in the point cloud set that is not in the visited set , calculate whether its Euclidean distance to all points in the cluster set meets the preset clustering conditions (that is, whether there is at least one point in the cluster set below whose Euclidean distance to the currently visited point is less than the preset distance threshold).

[0140] The calculation formula of Euclidean distance is:

[0141] .

[0142] Among them, Pj=(xj,yj,zj) in this formula is the point with the smallest Euclidean distance to this point in the cluster set C, that is, the minimum Euclidean distance.

[0143] The obtained Euclidean distance of each point is compared with a preset distance threshold, so as to add points whose Euclidean distance exceeds the distance threshold into the cluster set until all points in the dynamic object point cloud are visited.

[0144] Specifically, assuming the Euclidean distance threshold is , if the minimum Euclidean distance is less than the preset distance threshold , then the point Add it to the cluster set C and the visited point set V, and repeat the process until all points have been visited.

[0145] The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

[0146] Specifically, after the above clustering process, multiple clusters may be obtained. The characteristics of each cluster need to be analyzed to obtain the cluster corresponding to the ground point cloud. Typically, the ground point cloud is the largest cluster or the cluster with specific characteristics. In addition, the present application proposes that the cluster can be determined based on ground point cloud characteristics such as flatness and height range to determine which cluster is the ground point cloud. Then, the cluster determined to be the ground point cloud is segmented from the dynamic object point cloud to obtain the ground point cloud.

[0147] In step S6, the static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0148] Specifically, the ground point cloud is combined with the static point cloud to obtain a point cloud after dynamic objects are removed and data cleaning is performed. This results in a 3D point cloud after dynamic point cloud removal. If the resulting 3D point cloud needs to be converted from the camera coordinate system to the radar coordinate system, this can be done by multiplying it by the inverse of the transformation matrix M3 mentioned in the previous step.

[0149] By adopting the above technical solutions, this application can be widely used in application fields such as autonomous driving, intelligent robots, and smart city construction. Specifically:

[0150] 1. In the field of autonomous driving, this technology can improve the quality of point cloud data, thereby enhancing the perception and decision-making accuracy of autonomous driving systems. For example, by accurately isolating the point cloud of dynamic objects, collisions with other dynamic objects can be avoided during autonomous driving. Furthermore, multi-sensor fusion calibration technology can improve the accuracy and completeness of point cloud data, thereby enhancing the environmental perception capabilities of autonomous driving systems.

[0151] 2. In the field of intelligent robotics, this technology can provide higher-precision three-dimensional environmental perception, thereby improving the robot's navigation and obstacle avoidance capabilities. For example, by accurately separating the point cloud of dynamic objects, collisions between the robot and these objects can be avoided. Furthermore, multi-sensor fusion calibration technology can improve the accuracy and completeness of point cloud data, thereby enhancing the robot's environmental perception capabilities.

[0152] 3. In the field of smart city construction, it can provide higher-precision 3D urban environment modeling capabilities, thereby improving the daily lives of city managers and residents. For example, by accurately isolating dynamic object point clouds, dynamic objects in the urban environment, such as vehicles and pedestrians, can be more accurately modeled. At the same time, multi-sensor fusion calibration technology can improve the accuracy and completeness of point cloud data, thereby enhancing the accuracy and sophistication of urban environment modeling.

[0153] In the second aspect, the present application provides a dynamic point cloud removal device based on multimodal data perception, referring to Figure 2 , Figure 2 This is a virtual structural diagram of a dynamic point cloud removal device based on multimodal data perception provided by this application, including:

[0154] The data acquisition module 100 is used to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud;

[0155] A semantic segmentation module 200 is configured to perform camera distortion correction on the image to be processed, and input the corrected image to be processed into a pre-trained image semantic segmentation model to perform semantic segmentation on the corrected image to be processed using the image semantic segmentation model to obtain a mask image;

[0156] A linear transition module 300 is configured to obtain a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud, and the image acquisition time;

[0157] A point cloud extraction module 400 is configured to project the linearly transitioned single-frame point cloud onto the mask image, and extract a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud;

[0158] A ground point cloud determination module 500 is configured to perform cluster analysis on the dynamic object point cloud to extract the ground point cloud;

[0159] The point cloud data fusion module 600 is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0160] The dynamic point cloud removal device based on multimodal data perception described in the embodiment of the present application can execute the dynamic point cloud removal method based on multimodal data perception provided in the above embodiment. The dynamic point cloud removal device based on multimodal data perception has the corresponding functional steps and beneficial effects of the dynamic point cloud removal method based on multimodal data perception described in the above embodiment. Please refer to the above embodiment of the dynamic point cloud removal method based on multimodal data perception for details. The embodiment of the present application will not be repeated here.

[0161] The present application also provides an electronic device. Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of these chips. The memory, as a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the dynamic point cloud culling method based on multimodal data perception in the embodiment of the present application. The processor executes the non-transitory software programs, instructions, and modules stored in the memory to perform various functional applications and data processing of the processor, thereby implementing the dynamic point cloud culling method based on multimodal data perception in the above method embodiment.

[0162] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the dynamic point cloud removal method based on multimodal data perception as in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.

[0163] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0164] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present application.

[0165] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A dynamic point cloud elimination method based on multimodal data perception, characterized in that: The method comprises: Acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud; Performing camera distortion correction on the image to be processed, and inputting the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed by the image semantic segmentation model to obtain a mask image; Acquire a previous frame point cloud and a subsequent frame point cloud closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud, and the image acquisition time to obtain a single frame point cloud corresponding to the acquisition time; Projecting the linearly transitioned single-frame point cloud onto the mask image, extracting a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; performing cluster analysis on the dynamic object point cloud to extract the ground point cloud; The static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

2. The dynamic point cloud elimination method based on multimodal data perception according to claim 1 is characterized in that: The step of performing camera distortion correction on the image to be processed comprises: Performing radial distortion correction and tangential distortion correction on each image point of the image to be processed; The input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

3. The dynamic point cloud elimination method based on multimodal data perception according to claim 1, wherein the step of performing a linear transition on the single-frame point cloud based on the previous frame point cloud, the next frame point cloud, and the image acquisition time comprises: Obtaining a front frame pose and posture corresponding to the front frame point cloud and a rear frame pose and posture corresponding to the rear frame point cloud, and obtaining a front frame image acquisition time corresponding to the front frame point cloud and a rear frame image acquisition time corresponding to the rear frame point cloud; Determining a time ratio based on the previous frame acquisition time, the next frame acquisition time, and the image acquisition time; According to the previous frame pose, the subsequent frame pose and the time ratio, a linear interpolation calculation is performed on the pose corresponding to the single frame point cloud to obtain a transition pose, where the transition pose is the pose corresponding to the image acquisition time.

4. The dynamic point cloud elimination method based on multimodal data perception according to claim 3 is characterized in that: The step of projecting the linearly transitioned single-frame point cloud onto the mask image and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud comprises: Obtaining an extrinsic parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition pose matrix corresponding to the transition pose and posture; Determine a transformation matrix based on the extrinsic parameter matrix, the transition pose matrix, and the posture pose corresponding to the single-frame point cloud, and transform each point in the single-frame point cloud from the radar coordinate system to the camera coordinate system using the transformation matrix; Based on a preset camera intrinsic parameter matrix, project each point corresponding to the single-frame point cloud in the camera coordinate system onto the mask image to obtain multiple pixel coordinates; A pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

5. The dynamic point cloud elimination method based on multimodal data perception according to claim 4 is characterized in that: The step of obtaining a pixel value corresponding to each pixel coordinate in the mask image to extract the dynamic object point cloud and the static point cloud according to each pixel value includes: Determining a correspondence between each pixel coordinate and the mask image according to each pixel coordinate; By means of the correspondence between each pixel coordinate and the mask image, the pixel coordinates of all pixels whose corresponding pixel values ​​in the mask image are non-zero are obtained by screening; The dynamic object point cloud is extracted based on all the filtered pixel coordinates, and the static point cloud is determined.

6. The dynamic point cloud elimination method based on multimodal data perception according to claim 5 is characterized in that: The step of determining the corresponding relationship between each pixel coordinate and the mask image according to each pixel coordinate includes: Determining whether each of the pixel coordinates is an integer pixel coordinate; If the pixel coordinates are integer pixel coordinates, determining a correspondence between the pixel coordinates and the mask image according to the pixel coordinates; Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates; An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

7. The dynamic point cloud elimination method based on multimodal data perception according to claim 1 is characterized in that: The step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud includes: Determine any point in the dynamic object point cloud as a starting point, and add the starting point to a preset cluster set; Calculating the Euclidean distance between each unvisited point in the dynamic object point cloud and all points in the cluster set one by one; Comparing the obtained Euclidean distance of each point with a preset distance threshold, so as to add points whose Euclidean distance exceeds the distance threshold into the cluster set, until all points in the dynamic object point cloud are visited; The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

8. A dynamic point cloud removal device based on multimodal data perception, characterized in that: include: A data acquisition module, configured to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud; A semantic segmentation module is used to perform camera distortion correction on the image to be processed, and input the corrected image to be processed into a pre-trained image semantic segmentation model to perform semantic segmentation on the corrected image to be processed by the image semantic segmentation model to obtain a mask image; a linear transition module, configured to obtain a previous frame point cloud and a subsequent frame point cloud closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud, and the image acquisition time to obtain a single frame point cloud corresponding to the acquisition time; a point cloud extraction module, configured to project the linearly transitioned single-frame point cloud onto the mask image, and extract a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; a ground point cloud determination module, configured to perform cluster analysis on the dynamic object point cloud to extract the ground point cloud; The point cloud data fusion module is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program or instruction, and when the program or instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images