Dynamic point cloud elimination method and device based on multi-modal data perception, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to a dynamic point cloud elimination method and device based on multi-modal data perception, equipment and a storage medium. The method comprises the following steps: acquiring a single-frame point cloud, and determining a corresponding to-be-processed image; performing camera distortion correction on a to-be-processed image, and inputting the to-be-processed image into the model for semantic segmentation to obtain a mask image; acquiring a front-frame point cloud and a rear-frame point cloud which are closest to the image acquisition time of the to-be-processed image, and performing linear transition on the single-frame point cloud based on the front-frame point cloud, the rear-frame point cloud and the image acquisition time; projecting the single-frame point cloud after linear transition to a mask image, and extracting a dynamic object point cloud and a static point cloud; performing clustering analysis on the dynamic object point cloud to extract ground point cloud; and combining the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after the dynamic point cloud is removed. According to the invention, the specific details of the object can be confirmed in the fused point cloud data, so that the scene understanding requirement with increasingly improved precision can be met.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to a dynamic point cloud removal method, device, equipment and storage medium based on multimodal data perception. Background Art

[0002] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Each point contains three-dimensional coordinates and can carry other information about the point's attributes, such as color, reflectivity, and other geometric information with high precision, high resolution, and high dimensions. It can intuitively represent the shape, surface, and texture of objects in space. With the continuous development of technology, the application scope of point cloud data is becoming more and more extensive, and it has broad application prospects and market demand in the fields of autonomous driving, intelligent robots, and smart city construction.

[0003] In practical applications, removing dynamic point clouds is one of the indispensable technical means in the fields of autonomous driving and mobile robots. It not only helps to improve the system's positioning accuracy and map construction quality, but also reduces computing costs, enhances environmental perception and the ability to adapt to complex scenarios, thereby providing users with a safer, more comfortable and intelligent travel experience.

[0004] However, currently, point cloud data is mainly fused with the results of target detection, that is, a rectangular frame is used in the detection results to roughly frame the position and range of the target object. The expression of information such as the internal structure of the object, shape details, and the precise relationship with the surrounding environment is relatively limited, and its complex shape cannot be accurately represented. It is difficult to meet the increasingly accurate scene understanding needs, such as robot navigation, high-precision map drawing, and building information modeling. The technical field is difficult to play an ideal role.

[0005] Based on this, how to confirm the specific details of objects in the fused point cloud data to meet the increasingly accurate scene understanding needs is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present application provides a dynamic point cloud elimination method, device, equipment and storage medium based on multimodal data perception, which can achieve the beneficial effect of confirming the specific details of objects in fused point cloud data to meet the increasingly accurate scene understanding needs.

[0007] The technical solution adopted by this application to solve its technical problem is: In a first aspect, the present application provides a dynamic point cloud removal method based on multimodal data perception, the method comprising: Acquire a single-frame point cloud, and determine an image to be processed corresponding to the single-frame point cloud; Performing camera distortion correction on the image to be processed, and inputting the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image; Acquire a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time; Projecting the single-frame point cloud after linear transition onto the mask image, extracting a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; Performing cluster analysis on the dynamic object point cloud to extract the ground point cloud; The static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0008] Optionally, the step of performing linear transition on the single frame point cloud based on the previous frame point cloud, the next frame point cloud and the image acquisition time includes: Obtain a front frame posture and position corresponding to the front frame point cloud and a rear frame posture and position corresponding to the rear frame point cloud, and obtain a front frame image acquisition time corresponding to the front frame point cloud and a rear frame image acquisition time corresponding to the rear frame point cloud; Determine a time ratio based on the previous frame acquisition time, the next frame acquisition time, and the image acquisition time; According to the previous frame posture and position, the subsequent frame posture and position and the time ratio, a linear interpolation calculation is performed on the posture and position corresponding to the single frame point cloud to obtain a transition posture and position, and the transition posture and position is the posture and position corresponding to the image acquisition time.

[0009] Optionally, the step of projecting the linearly transitioned single-frame point cloud onto the mask image and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud comprises: Obtaining an external parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition posture matrix corresponding to the transition posture and position; Determine a transformation matrix based on the external parameter matrix, the transition pose matrix and the posture pose corresponding to the single-frame point cloud, and transform each point in the single-frame point cloud from the radar coordinate system to the camera coordinate system through the transformation matrix; Based on a preset camera intrinsic parameter matrix, project each point corresponding to the single-frame point cloud in the camera coordinate system onto the mask image to obtain a plurality of pixel coordinates; The pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

[0010] Optionally, the step of performing camera distortion correction on the image to be processed includes: Performing radial distortion correction and tangential distortion correction on each image point of the image to be processed; The input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

[0011] Optionally, the step of obtaining a pixel value corresponding to each pixel coordinate in the mask image to extract the dynamic object point cloud and the static point cloud according to each pixel value includes: According to each of the pixel coordinates, determining a corresponding relationship between each of the pixel coordinates and the mask image; Through the correspondence between each pixel coordinate and the mask image, all pixel coordinates whose corresponding pixel values ​​in the mask image are non-zero are obtained by screening; According to all the filtered pixel coordinates, the dynamic object point cloud is extracted and the static point cloud is determined.

[0012] Optionally, the step of determining, according to each pixel coordinate, a corresponding relationship between each pixel coordinate and the mask image includes: Determining whether each of the pixel coordinates is an integer pixel coordinate; If the pixel coordinates are integer pixel coordinates, determining a corresponding relationship between the pixel coordinates and the mask image according to the pixel coordinates; Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates; An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

[0013] Optionally, the step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud includes: Determine any point in the point cloud of the dynamic object as a starting point, and add the starting point to a preset cluster set; Calculating the Euclidean distance between each unvisited point in the dynamic object point cloud and all points in the cluster set one by one; Comparing the obtained Euclidean distance of each point with a preset distance threshold, so as to add points whose Euclidean threshold exceeds the distance threshold into the cluster set, until all points in the dynamic object point cloud are visited; The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

[0014] In a second aspect, the present application provides a dynamic point cloud removal device based on multimodal data perception, comprising: A data acquisition module, used to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud; A semantic segmentation module, used to perform camera distortion correction on the image to be processed, and input the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image; A linear transition module, used to obtain a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time; A point cloud extraction module, used to project the single-frame point cloud after linear transition onto the mask image, and extract dynamic object point cloud and static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; A ground point cloud determination module, used for performing cluster analysis on the dynamic object point cloud to extract the ground point cloud; The point cloud data fusion module is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0015] In a third aspect, the present application provides an electronic device, including: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the above method.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, in which a program or instruction is stored. When the program or instruction is executed, the above method is implemented.

[0017] The technical solution provided by the present application is as follows: radar and camera are used to collect data of the same scene respectively, wherein the radar is used to obtain a three-dimensional point cloud and the camera is used to collect a two-dimensional image. The data of the same scene are collected by the camera and radar at the same time, one frame of the point cloud is determined to be the point cloud to be processed, and the image to be processed that is closest to the point cloud to be processed in time and space is found; further, camera distortion correction is performed on the image to be processed, and the corrected image to be processed is input into the model for semantic segmentation to obtain a mask image; further, based on the trajectory point pose of the previous frame point cloud and the trajectory point pose of the subsequent frame point cloud A linear transition is performed between the acquisition time of the single-frame point cloud and the acquisition time of the image to be processed, so as to transition the trajectory point pose corresponding to the single-frame point cloud to the interpolated pose; further, based on the correspondence between the single-frame point cloud and the mask image, the single-frame point cloud is projected onto the mask image to separate the dynamic object point cloud corresponding to the mask image, and the rest of the single-frame point cloud is determined to be a static point cloud; further, a cluster analysis is performed on the dynamic object point cloud to screen out the ground point cloud in the dynamic object point cloud; finally, the ground point cloud and the static point cloud are combined to obtain the point cloud with the dynamic object point cloud removed after the fused point cloud data.

[0018] By adopting the above technical solution, the present application has at least the following beneficial effects: 1. This application uses semantic segmentation to detect objects. Compared with the traditional target detection method, semantic segmentation has a more refined expression of spatial information, assigns semantic labels to each point in the scene, provides more detailed spatial information, processes and analyzes all point cloud data, and can fully mine the information in the data to avoid information omission. It can be seen that the dynamic object point cloud separated based on semantic segmentation has a richer expression of information such as the internal structure, shape details, and precise relationship with the surrounding environment of the object, which can realize the confirmation of the specific details of the object in the fused point cloud data to meet the increasingly accurate scene understanding needs.

[0019] 2. This application adopts multi-sensor fusion calibration technology that combines camera and radar to make up for the shortcomings of a single sensor, thereby improving the overall perception capability and further improving the accuracy and completeness of point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of the process of the dynamic point cloud removal method based on multimodal data perception provided by the embodiment of the present application Figure 2 It is a virtual structural schematic diagram of a dynamic point cloud rejection device based on multimodal data perception provided by the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0022] The following will clearly and completely describe the concept, specific structure and technical effects of the present application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present application. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0023] Reference Figure 1 , Figure 1 : is a flow chart of a dynamic point cloud removal method based on multimodal data perception provided by an embodiment of the present application, which specifically includes the following steps, which are described in detail below: In step S1, a single-frame point cloud is acquired, and an image to be processed corresponding to the single-frame point cloud is determined.

[0024] Specifically, a single-frame point cloud uses a three-dimensional point cloud acquisition method such as a laser radar to obtain point cloud data. The image to be processed is a two-dimensional image of a target scene obtained by an image acquisition device such as a camera, containing dynamic objects. Dynamic objects include vehicles, pedestrians, etc. In an embodiment of the present application, multiple sensors (cameras, radars) are used to simultaneously collect multiple images of the same scene at the same frequency to obtain multiple images collected by the camera and multiple frames of point clouds collected by the radar, wherein each point cloud corresponds to a SLAM trajectory point pose. A single-frame point cloud containing a dynamic object is selected from the collected multi-frame point clouds, and then an image to be processed that is aligned in time and space is found based on the acquisition time of the single-frame point cloud.

[0025] In step S2, camera distortion correction is performed on the image to be processed, and the corrected image to be processed is input into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image.

[0026] Specifically, the image semantic segmentation model is a model with semantic segmentation capabilities, which can segment an image into regional blocks with specific semantic meanings and identify the semantic category of each regional block, such as pedestrians, vehicles, etc. A mask image is a binary image used to represent different categories of regions in an image. In this application, in the mask image obtained by segmentation after semantic segmentation, dynamic objects can be marked as black and static backgrounds can be marked as white.

[0027] More specifically, considering the camera distortion, before obtaining the mask image (that is, before the image to be processed is input into the image semantic segmentation model), the distortion parameters need to be used to correct the camera distortion of the image to be processed, including: Radial distortion correction and tangential distortion correction are performed on each image point of the image to be processed.

[0028] Specifically, the camera distortion usually includes radial distortion and tangential distortion. Let the uncorrected point in the camera coordinate system be , the corrected point is , in which case first obtain the normalized coordinates , , and the distortion correction is performed by normalizing the coordinates using the radial distortion calculation formula and the tangential distortion calculation formula. The radial distortion calculation formula is: ; ; ; Among them, p1 and p2 in the above radial distortion calculation formula are radial distortion parameters.

[0029] The calculation formula for tangential distortion is: ; ; Among them, p1 and p2 in the above tangential distortion calculation formula are tangential distortion parameters.

[0030] Furthermore, the input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

[0031] Specifically, the distortion-corrected point can be expressed as , this algorithm can be used to correct the camera distortion of the image to be processed.

[0032] In step S3, the previous frame point cloud and the next frame point cloud closest to the image acquisition time of the image to be processed are obtained, and the single frame point cloud is linearly transitioned based on the previous frame point cloud, the next frame point cloud and the image acquisition time.

[0033] Specifically, in the embodiment of the present application, the laser radar frequency used is 10 Hz, and the image data acquisition frequency is also 10 Hz. However, due to various errors, the image to be processed acquired at the current time (any one of the multiple images acquired contains an image of a dynamic object) and the frame of point cloud closest to the acquisition time are not exactly the same time. Based on this, before the subsequent point cloud data fusion, it is necessary to find the two frames of point clouds closest to the current image acquisition time, and perform time-space unification and kinematic linear transition operations based on this to determine the single frame of point cloud corresponding to the acquisition time.

[0034] More specifically, based on the previous frame point cloud, the next frame point cloud and the image acquisition time, the step of performing linear transition on the single frame point cloud includes: The front frame posture and position corresponding to the front frame point cloud and the rear frame posture and position corresponding to the rear frame point cloud are obtained, and the front frame image acquisition time corresponding to the front frame point cloud and the rear frame image acquisition time corresponding to the rear frame point cloud are obtained.

[0035] Specifically, let the acquisition time of the image to be processed be , the corresponding acquisition time of the previous frame point cloud is , and the SLAM trajectory point posture corresponding to the previous frame point cloud is ,in represents the position coordinates, roll, pitch, and yaw represent the attitude angles of dynamic objects (such as vehicles); the corresponding acquisition time of the subsequent frame point cloud is , the corresponding SLAM trajectory point posture is .

[0036] Among them, SLAM trajectory point posture refers to the three-dimensional position and orientation information of the robot or camera in space obtained by Simultaneous Localization and Mapping (SLAM) technology. The posture is the information describing the position and orientation of the robot or camera in space, usually including three-dimensional coordinates. and rotation angles (pitch, yaw, roll).

[0037] Furthermore, a time ratio is determined based on the previous frame acquisition time, the next frame acquisition time and the image acquisition time.

[0038] Specifically, a first time difference with the previous frame is determined based on the previous frame acquisition time and the image acquisition time, and a second time difference with the next frame is determined based on the next frame acquisition time and the image acquisition time. and , and then calculate the total time difference based on the first time difference and the second time difference , then determine the time ratio of the previous frame time difference to the total time difference based on the total time difference and the first time difference .

[0039] Furthermore, according to the previous frame posture and position, the subsequent frame posture and position and the time ratio, a linear interpolation calculation is performed on the posture and position corresponding to the single frame point cloud to obtain a transition posture and position, and the transition posture and position is the posture and position corresponding to the image acquisition time.

[0040] Specifically, determine the position coordinates of any point in the point cloud , interpolate the position coordinates based on the time ratio obtained in the above steps, and get the interpolated position coordinates as , similarly, , ; Furthermore, for the posture and position corresponding to the interpolated position coordinates, including the interpolated roll angle , the interpolated pitch angle and the interpolated yaw angle , where the attitude angles roll, pitch, and yaw represent the rotation around the x-axis, y-axis, and z-axis respectively, and are used to describe the spatial attitude of the object.

[0041] Furthermore, each position coordinate in the single frame point cloud is interpolated to obtain all transition postures corresponding to the image acquisition time. , this pose can be converted into a matrix M2, based on which the transformation matrix for converting from the radar coordinate system to the camera coordinate system can be calculated later.

[0042] In step S4, the single-frame point cloud after linear transition is projected onto the mask image, and dynamic object point cloud and static point cloud are extracted from the single-frame point cloud, wherein the dynamic object point cloud includes dynamic point cloud and ground point cloud.

[0043] Specifically, the single-frame point cloud after linear transition is projected to the pixel coordinate system of the mask image, and the point clouds of the dynamic objects and static parts in the mask image obtained above are classified and saved, so as to extract the dynamic object point cloud and the static point cloud. Specifically, first determine the pose corresponding to the single-frame point cloud as M0, the pose corresponding to the single-frame point cloud is obtained by linear interpolation. The pose conversion matrix is ​​M2, and then define the external parameter matrix of the camera relative to the radar as M1, and the conversion matrix as M3. Among them, the external parameter matrix of the camera relative to the radar can be obtained by calibrating the camera and the radar, and this calibration process can be implemented using MATLAB software.

[0044] More specifically, projecting the single-frame point cloud after linear transition onto the mask image, and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud specifically includes the following steps: Obtain an external parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition posture matrix corresponding to the transition posture and position.

[0045] Specifically, obtain the external parameter matrix M1 of the camera relative to the radar, and the matrix M2 converted from the transition posture obtained in the above step S3, wherein M2 is obtained by linear interpolation of the posture M0 corresponding to the single-frame point cloud.

[0046] Furthermore, a transformation matrix is ​​determined based on the extrinsic parameter matrix, the transition posture matrix and the posture and pose corresponding to the single-frame point cloud, and each point in the single-frame point cloud is transformed from the radar coordinate system to the camera coordinate system through the transformation matrix.

[0047] Specifically, let the external parameter matrix of the camera coordinate system relative to the radar coordinate system be M1, and the attitude and posture matrix corresponding to the transition attitude and posture be M2 (that is, the matrix M2 obtained by linear interpolation of the attitude and posture corresponding to the above point cloud). In this case, the transformation matrix M3 used to transform the coordinate system is: At rest: ; In motion: ; Among them, M3 is a matrix, Serves 3 3's rotation matrix, Serves 3 1. Specifically, the difference between the radar coordinate system and the camera coordinate system of the point cloud is an M3 matrix, that is, the point cloud can be converted to the camera coordinate system by multiplying M3 on the right, and associated with the camera. Conversion from radar coordinate system to camera coordinate system: For example: for the point cloud coordinates in the radar coordinate system , converted to the coordinates in the camera coordinate system The formula is:

[0048] At this point, the radar coordinate system can be obtained The corresponding point in the camera coordinate system .

[0049] Furthermore, based on a preset camera intrinsic parameter matrix, each point corresponding to the single-frame point cloud in the camera coordinate system is projected onto the mask image to obtain a plurality of pixel coordinates.

[0050] Specifically, the camera intrinsic parameter matrix is ​​a known preset value. Specifically, the known camera intrinsic parameter matrix is ,in and are the focal lengths of the camera in the x-axis and y-axis directions, respectively, and are the principal point coordinates of the image. Then, based on the point P in the camera coordinate system The calculation formula of the pixel coordinates (u, v) projected onto the image plane is: ; In summary, in this step, the points in the single-frame point cloud (radar coordinate system) converted to (Camera coordinate system) and then converted to (u,v) pixel coordinates (pixel coordinate system).

[0051] Furthermore, a pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

[0052] Specifically, an index is established on the mask image according to the pixel coordinates to determine which pixel positions correspond to the valid area (generally speaking, non-zero pixel values ​​are valid values). If a pixel coordinate is valid in the mask image, the corresponding radar point cloud can be determined by the index. Specifically, the calculated pixel coordinates (u, v) have corresponding pixel values ​​in the single-frame point cloud. Each pixel coordinate corresponds to a point cloud part of a mask image in the single-frame point cloud. Based on this correspondence, the pixel coordinates can be used to extract the point cloud corresponding to the mask image part from the single-frame point cloud, thereby obtaining a dynamic object point cloud. The remaining extracted part is determined as a static point cloud.

[0053] The following specifically describes the steps of obtaining the pixel value corresponding to each pixel coordinate in the mask image to extract the dynamic object point cloud and the static point cloud according to each pixel value: According to each of the pixel coordinates, determining a corresponding relationship between each of the pixel coordinates and the mask image; Through the correspondence between each pixel coordinate and the mask image, all pixel coordinates whose corresponding pixel values ​​in the mask image are non-zero are obtained by screening; According to all the filtered pixel coordinates, the dynamic object point cloud is extracted and the static point cloud is determined.

[0054] Specifically, the position corresponding to the calculated pixel coordinate (u, v) can be checked in the mask image to establish an index, and based on whether the pixel value at the position is a valid value, a non-zero value is usually a valid value, and the point cloud corresponding to the pixel coordinate with a valid value can be determined as a dynamic object point cloud, thereby screening all dynamic object point clouds one by one, and the point clouds of other parts in the single-frame point cloud can be determined as static point clouds. Among them, a data structure (including but not limited to a dictionary or a list) can be used to store the correspondence between pixel coordinates and single-frame point cloud indexes.

[0055] More specifically, in the embodiments of the present application, it is considered that there are two cases of pixel coordinates, integer and non-integer. Integer pixel coordinates can be directly used as indexes, while non-integer pixel coordinates need to be processed before obtaining the corresponding index. Specifically: Determining whether each of the pixel coordinates is an integer pixel coordinate; If the pixel coordinates are integer pixel coordinates, the corresponding relationship between the pixel coordinates and the mask image is determined according to the pixel coordinates.

[0056] Specifically, if the calculated pixel coordinates are integers, that is, integer pixel coordinates, they can be directly used as indexes to check whether the pixel values ​​corresponding to the pixel coordinates in the mask image are valid values, and all valid values ​​are judged and extracted one by one to obtain all point clouds corresponding to the integer pixel coordinates, that is, to obtain all dynamic object point clouds corresponding to the integer pixel coordinates in the single-frame point cloud. Among them, the index of the radar point cloud corresponding to the integer pixel coordinates can be recorded using a data structure (such as a dictionary or a list) to store the correspondence between the integer pixel coordinates and the single-frame point cloud index.

[0057] Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates; An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

[0058] Specifically, if the pixel coordinates are non-integer, i.e., non-integer pixel coordinates, a method such as bilinear interpolation can be used to determine whether the pixel value in the mask image corresponding to the pixel position is valid. If the pixel value is valid, the corresponding point cloud is determined in the single-frame point cloud. This application takes the bilinear interpolation method as an example, specifically: Obtain the four integer pixel coordinates closest to the pixel coordinates. That is, find the four integer pixel coordinates closest to the pixel coordinates; Get the pixel values ​​corresponding to these four integer pixel coordinates from the mask image , , , , respectively , , , ; Then interpolation calculations are performed based on these four pixel values, including calculating the interpolation weight of each pixel value, and then the corresponding value of each non-integer pixel coordinate in the mask image is obtained based on the interpolation weight. The formula for calculating the interpolation weight is: ; ; ; .

[0059] The formula used to obtain the corresponding value of each non-integer pixel coordinate in the mask image based on the interpolation weight is: ; Similar to the point cloud index corresponding to the integer pixel coordinates mentioned above, it can be ensured that different non-integer pixel coordinates correspond to different index records in some way (for example, fine-tuning according to the decimal part of the pixel coordinates).

[0060] In summary, combining the above steps, we can filter out the corresponding position of the mask image part (that is, the part corresponding to the dynamic object in the image) in the single-frame point cloud, and based on this, we can obtain the dynamic object point cloud (the point cloud corresponding to the mask image) and the static point cloud (the remaining point cloud after removing the point cloud of the mask image).

[0061] In step S5, cluster analysis is performed on the dynamic object point cloud to extract the ground point cloud.

[0062] Specifically, the extracted dynamic object point cloud is a mixed point cloud of a dynamic object (such as a vehicle) and the ground. In order to remove the point cloud of the dynamic object part, cluster analysis is also required to separate the point clouds of each cluster in the mixed point cloud. In the embodiment of the present application, by calculating the Euclidean distance between points, if the Euclidean distance between two points is less than a preset threshold, it is considered that the two points belong to the same type of point cloud.

[0063] More specifically, the step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud includes: Determine any point in the dynamic object point cloud as a starting point, and add the starting point to a preset cluster set.

[0064] Specifically, let the input point cloud set be , where each point is . The point cloud set is the point cloud set formed by the above-mentioned dynamic object point cloud, including mixed point clouds of multiple categories of point clouds. Then, select any point in the point cloud set as the starting point, or select the point with the smallest value as the starting point. Assume that the starting point is .

[0065] The Euclidean distance between each unvisited point in the dynamic object point cloud and all the points in the cluster set is calculated one by one.

[0066] Specifically, we can create an empty cluster set C = {} and a visited point set V = {}, and determine the starting point Add them to the cluster set C and the visited set V respectively. Then, for each unvisited point in the point cloud set that is not in the visited set , calculate whether the Euclidean distance between it and all points in the cluster set meets the preset clustering condition (that is, whether there is at least one point in the cluster set below whose Euclidean distance with the currently visited point is less than the preset distance threshold).

[0067] The calculation formula of Euclidean distance is: .

[0068] Among them, Pj=(xj,yj,zj) in this formula is the point with the smallest Euclidean distance to this point in the cluster set C, that is, the minimum Euclidean distance.

[0069] The obtained Euclidean distance of each point is compared with a preset distance threshold, so as to add points whose Euclidean distance exceeds the distance threshold into the cluster set until all points in the dynamic object point cloud are visited.

[0070] Specifically, assuming the Euclidean distance threshold is , if the minimum Euclidean distance is less than the preset distance threshold , then the point Add it to the cluster set C and the visited point set V, and repeat the process until all points have been visited.

[0071] The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

[0072] Specifically, after the above clustering process, multiple clusters may be obtained, and the characteristics of each cluster need to be analyzed to obtain the cluster corresponding to the ground point cloud. Usually, the ground point cloud is the largest cluster or a cluster with specific characteristics. In addition, the present application proposes that which cluster is the ground point cloud can be determined based on the ground point cloud features such as flatness, height range, etc. Then, the cluster determined to be the ground point cloud is segmented from the dynamic object point cloud to obtain the ground point cloud.

[0073] In step S6, the static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0074] Specifically, the obtained ground point cloud is combined with the static point cloud to obtain a point cloud after removing the dynamic object part and data cleaning, that is, a three-dimensional point cloud after removing the dynamic point cloud. If the obtained three-dimensional point cloud needs to be converted from the camera coordinate system to the radar coordinate system, the coordinate system conversion can be performed by multiplying the inverse matrix of the conversion matrix M3 mentioned in the above step.

[0075] By adopting the above technical solution, this application can be widely used in application fields such as autonomous driving, intelligent robots, and smart city construction. Specifically: 1. In the field of autonomous driving, the quality of point cloud data can be improved, thereby improving the perception ability and decision-making accuracy of the autonomous driving system. For example, by accurately separating the point cloud of dynamic objects, it is possible to avoid collisions between autonomous driving vehicles and other dynamic objects during driving. At the same time, through multi-sensor fusion calibration technology, the accuracy and completeness of point cloud data can be improved, thereby improving the environmental perception ability of the autonomous driving system.

[0076] 2. In the field of intelligent robots, it can provide higher-precision three-dimensional environmental perception capabilities, thereby improving the navigation and obstacle avoidance capabilities of intelligent robots. For example, by accurately separating the point cloud of dynamic objects, it is possible to avoid collisions between intelligent robots and dynamic objects. At the same time, through multi-sensor fusion calibration technology, the accuracy and completeness of point cloud data can be improved, thereby improving the environmental perception capabilities of intelligent robots.

[0077] 3. In the field of smart city construction, it can provide higher-precision 3D urban environment modeling capabilities, thereby improving the daily life quality of city managers and residents. For example, by accurately separating the point cloud of dynamic objects, dynamic objects in the urban environment, such as vehicles and pedestrians, can be more accurately modeled. At the same time, through multi-sensor fusion calibration technology, the accuracy and completeness of point cloud data can be improved, thereby improving the accuracy and precision of urban environment modeling.

[0078] In the second aspect, the present application provides a dynamic point cloud removal device based on multimodal data perception, referring to Figure 2 , Figure 2 It is a virtual structural diagram of a dynamic point cloud rejection device based on multimodal data perception provided by the present application, including: The data acquisition module 100 is used to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud; A semantic segmentation module 200, used for performing camera distortion correction on the image to be processed, and inputting the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image; A linear transition module 300 is used to obtain a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time; A point cloud extraction module 400 is used to project the single-frame point cloud after linear transition to the mask image, and extract dynamic object point cloud and static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes dynamic point cloud and ground point cloud; A ground point cloud determination module 500, configured to perform cluster analysis on the dynamic object point cloud to extract the ground point cloud; The point cloud data fusion module 600 is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

[0079] The dynamic point cloud removal device based on multimodal data perception described in the embodiment of the present application can execute the dynamic point cloud removal method based on multimodal data perception provided in the above embodiment. The dynamic point cloud removal device based on multimodal data perception has the corresponding functional steps and beneficial effects of the dynamic point cloud removal method based on multimodal data perception described in the above embodiment. Please refer to the above embodiment of the dynamic point cloud removal method based on multimodal data perception for details. The embodiment of the present application will not be repeated here.

[0080] The present application also provides an electronic device. Figure 3 , Figure 3: is a structural diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the dynamic point cloud removal method based on multimodal data perception in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the dynamic point cloud removal method based on multimodal data perception in the above method embodiment is realized.

[0081] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the dynamic point cloud removal method based on multimodal data perception in the above method embodiment is executed. The specific details of the above electronic device can be understood by corresponding to the corresponding descriptions and effects in the above method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0082] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0083] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0084] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A dynamic point cloud elimination method based on multimodal data perception, characterized in that: The method comprises: Acquire a single-frame point cloud, and determine an image to be processed corresponding to the single-frame point cloud; Performing camera distortion correction on the image to be processed, and inputting the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image; Acquire a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time; Projecting the single-frame point cloud after linear transition onto the mask image, extracting a dynamic object point cloud and a static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; Performing cluster analysis on the dynamic object point cloud to extract the ground point cloud; The static point cloud and the ground point cloud are combined to obtain a three-dimensional point cloud after removing the dynamic point cloud.

2. The dynamic point cloud elimination method based on multimodal data perception according to claim 1 is characterized in that: The step of performing camera distortion correction on the image to be processed comprises: Performing radial distortion correction and tangential distortion correction on each image point of the image to be processed; The input of the image semantic segmentation model is determined according to the radial distortion correction results and the tangential distortion correction results of all the image points.

3. According to the dynamic point cloud removal method based on multimodal data perception according to claim 1, the step of performing linear transition on the single frame point cloud based on the previous frame point cloud, the next frame point cloud and the image acquisition time comprises: Obtain a front frame posture and position corresponding to the front frame point cloud and a rear frame posture and position corresponding to the rear frame point cloud, and obtain a front frame image acquisition time corresponding to the front frame point cloud and a rear frame image acquisition time corresponding to the rear frame point cloud; Determine a time ratio based on the previous frame acquisition time, the next frame acquisition time, and the image acquisition time; According to the previous frame posture and position, the subsequent frame posture and position and the time ratio, a linear interpolation calculation is performed on the posture and position corresponding to the single frame point cloud to obtain a transition posture and position, and the transition posture and position is the posture and position corresponding to the image acquisition time.

4. The dynamic point cloud elimination method based on multimodal data perception according to claim 3 is characterized in that: The step of projecting the linearly transitioned single-frame point cloud onto the mask image and extracting the dynamic object point cloud and the static point cloud from the single-frame point cloud comprises: Obtaining an external parameter matrix of the camera coordinate system relative to the radar coordinate system and a transition posture matrix corresponding to the transition posture and position; Determine a transformation matrix based on the external parameter matrix, the transition pose matrix and the posture pose corresponding to the single-frame point cloud, and transform each point in the single-frame point cloud from the radar coordinate system to the camera coordinate system through the transformation matrix; Based on a preset camera intrinsic parameter matrix, project each point corresponding to the single-frame point cloud in the camera coordinate system onto the mask image to obtain a plurality of pixel coordinates; A pixel value corresponding to each pixel coordinate in the mask image is obtained to extract the dynamic object point cloud and the static point cloud according to each pixel value.

5. The dynamic point cloud elimination method based on multimodal data perception according to claim 4 is characterized in that: The step of obtaining a pixel value corresponding to each pixel coordinate in the mask image to extract the dynamic object point cloud and the static point cloud according to each pixel value comprises: According to each of the pixel coordinates, determining a corresponding relationship between each of the pixel coordinates and the mask image; Through the correspondence between each pixel coordinate and the mask image, all pixel coordinates whose corresponding pixel values ​​in the mask image are non-zero are obtained by screening; According to all the filtered pixel coordinates, the dynamic object point cloud is extracted and the static point cloud is determined.

6. The dynamic point cloud elimination method based on multimodal data perception according to claim 5 is characterized in that: The step of determining the corresponding relationship between each pixel coordinate and the mask image according to each pixel coordinate comprises: Determining whether each of the pixel coordinates is an integer pixel coordinate; If the pixel coordinates are integer pixel coordinates, determining a corresponding relationship between the pixel coordinates and the mask image according to the pixel coordinates; Otherwise, obtain four integer pixel coordinates closest to the pixel coordinate, and obtain four pixel values ​​corresponding to the four integer pixel coordinates; An interpolation calculation is performed based on the obtained four pixel values, and a corresponding relationship between the pixel coordinates and the mask image is determined according to the interpolation calculation result.

7. The dynamic point cloud elimination method based on multimodal data perception according to claim 1 is characterized in that: The step of performing cluster analysis on the dynamic object point cloud to extract the ground point cloud comprises: Determine any point in the point cloud of the dynamic object as a starting point, and add the starting point to a preset cluster set; Calculating the Euclidean distance between each unvisited point in the dynamic object point cloud and all points in the cluster set one by one; Comparing the obtained Euclidean distance of each point with a preset distance threshold, so as to add points whose Euclidean threshold exceeds the distance threshold into the cluster set, until all points in the dynamic object point cloud are visited; The ground point cloud is extracted from the cluster set according to preset ground point cloud features.

8. A dynamic point cloud removal device based on multimodal data perception, characterized in that: include: A data acquisition module, used to acquire a single-frame point cloud and determine an image to be processed corresponding to the single-frame point cloud; A semantic segmentation module, used to perform camera distortion correction on the image to be processed, and input the corrected image to be processed into a pre-trained image semantic segmentation model, so as to perform semantic segmentation on the corrected image to be processed through the image semantic segmentation model to obtain a mask image; A linear transition module, used to obtain a previous frame point cloud and a subsequent frame point cloud that are closest to the image acquisition time of the image to be processed, and perform a linear transition on the single frame point cloud based on the previous frame point cloud, the subsequent frame point cloud and the image acquisition time; A point cloud extraction module, used to project the single-frame point cloud after linear transition onto the mask image, and extract dynamic object point cloud and static point cloud from the single-frame point cloud, wherein the dynamic object point cloud includes a dynamic point cloud and a ground point cloud; A ground point cloud determination module, used for performing cluster analysis on the dynamic object point cloud to extract the ground point cloud; The point cloud data fusion module is used to combine the static point cloud and the ground point cloud to obtain a three-dimensional point cloud after removing the dynamic point cloud.

9. An electronic device, characterized in that: include: one or more processors; one or more memories; and one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, and the one or more computer programs include instructions, which, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 7 is implemented.

Images