Data processing method, device and equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment and a computer readable storage medium. The method comprises the following steps: performing semantic feature extraction on a plurality of data points contained in point cloud data of specified equipment to obtain semantic tags corresponding to the plurality of data points respectively; performing voxel division on the point cloud data to obtain a plurality of first voxels, and selecting a target data point from the plurality of data points contained in each first voxel based on semantic tags corresponding to the plurality of data points; based on the semantic tag of each target data point, determining an associated feature point matched with each target data point from a preset local map point cloud; and calculating the current pose information of the specified equipment based on the plurality of associated feature points. Through the method, the accuracy of pose estimation can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, apparatus, device and computer-readable storage medium. Background Art

[0002] Pose estimation refers to determining the pose information of a robot in three-dimensional space, including position and orientation, in the field of robot perception and navigation.

[0003] When performing pose estimation, sensors such as LiDAR are usually used to collect point cloud data of the current robot's surroundings and match this data with a known map. The matching process involves finding the associated points in the map for each data point in the point cloud data to determine the current robot's pose, that is, where it is located and which direction it is facing. However, point cloud data usually has a large number of data points, and similar points may be very close to each other, which increases the complexity of determining associated points. Incorrect selection of associated points will lead to inaccurate pose estimation.

[0004] Therefore, how to improve the accuracy of pose estimation is an urgent problem to be solved. Summary of the invention

[0005] To solve the above technical problems, the embodiments of the present application provide a data processing method, apparatus, device and computer-readable storage medium, which can improve the accuracy of posture estimation.

[0006] Among them, the technical solution adopted in this application is:

[0007] A data processing method, comprising:

[0008] Extracting semantic features of a plurality of data points contained in the point cloud data of the specified device to obtain semantic labels corresponding to the plurality of data points;

[0009] Performing voxel division on the point cloud data to obtain a plurality of first voxels, and selecting a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points;

[0010] Based on the semantic label of each target data point, determining the associated feature points matching each target data point from the preset local map point cloud;

[0011] The current position information of the designated device is calculated based on the multiple associated feature points.

[0012] A data processing device, comprising:

[0013] An extraction unit, used to extract semantic features from a plurality of data points contained in the point cloud data of a specified device, and obtain semantic labels corresponding to the plurality of data points respectively;

[0014] A processing unit, configured to perform voxel division on the point cloud data to obtain a plurality of first voxels, and select a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points;

[0015] The processing unit is further used to determine, based on the semantic label of each target data point, an associated feature point matching each target data point from a preset local map point cloud;

[0016] A calculation unit is used to calculate the current posture information of the designated device based on multiple associated feature points.

[0017] In one embodiment of the present application, based on the aforementioned scheme, the computing unit is further used to predict the current posture information of the designated device based on the point cloud data of a previous preset number of frames of the point cloud data; the processing unit is further used to transform the point cloud data to a world coordinate system based on the predicted posture information to obtain the transformed point cloud data; based on the transformed point cloud data and the semantic label of each target data point, determine the associated feature points matching each target data point from the preset local map point cloud.

[0018] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is further used to perform voxel division on the transformed point cloud data to obtain multiple second voxels; for each target data point, the acquisition unit is used to obtain the position information of the second voxel to which the target data point belongs, and the processing unit is further used to obtain the data points contained in the voxels that match the position information in the preset local map point cloud; based on the semantic label of the target data point and the semantic label corresponding to the acquired data point, calculate the distance value between the target data point and the acquired data point; and use the data point corresponding to the smallest distance value as the associated feature point of the target data point.

[0019] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is further used to perform voxel division on the point cloud data to obtain multiple third voxels; select designated data points from the multiple data points contained in each third voxel, and perform semantic feature extraction on each designated data point to obtain the semantic label of each designated data point; and use the semantic label of each designated data point as the semantic label of the data point contained in the third voxel to which the designated data point belongs.

[0020] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to classify multiple data points based on semantic labels corresponding to multiple data points contained in the point cloud data to obtain multiple instance point clouds; wherein the same instance point cloud includes data points corresponding to the same type of semantic labels; based on the type information corresponding to the multiple instance point clouds, the point cloud data is voxel-divided to obtain multiple first voxels.

[0021] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to determine the voxel volume for voxel division of each instance point cloud based on the type information corresponding to the multiple instance point clouds respectively; based on the voxel volume corresponding to each instance point cloud, voxel division is performed on each instance point cloud to obtain the multiple first voxels.

[0022] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to determine a dynamic instance point cloud from the point cloud data based on type information corresponding to the multiple instance point clouds; remove the dynamic instance point cloud to obtain updated point cloud data; and perform voxel division on the updated point cloud data to obtain multiple first voxels.

[0023] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to detect static instance point clouds among the multiple instance point clouds based on type information corresponding to the multiple instance point clouds; for each instance point cloud, detect the static instance point cloud existing within a preset range around the instance point cloud, and identify whether the instance point cloud is a dynamic instance point cloud based on the static instance point cloud.

[0024] In one embodiment of the present application, based on the aforementioned scheme, the processing unit is also used to determine that the instance point cloud is a dynamic instance point cloud if, within a preset range of the instance point cloud, the proportion of data points of the static instance point cloud to all data points within the preset range is less than a preset proportion threshold.

[0025] In one embodiment of the present application, based on the aforementioned scheme, the acquisition unit is further used to obtain the posture information of the designated device when collecting the point cloud data of a preset number of frames before the point cloud data is collected; the calculation unit is further used to calculate the current predicted moving speed of the designated device based on the acquired posture information; the processing unit is further used to perform error correction of the collection time difference of the point cloud data based on the predicted moving speed and the type of the designated device, so that all data points in the point cloud data have the same collection time to obtain corrected point cloud data; the extracted point cloud is further used to perform semantic feature extraction on multiple data points contained in the corrected point cloud data to obtain semantic labels corresponding to the multiple data points respectively.

[0026] A data processing device includes a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above data processing method is implemented.

[0027] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor of a computer, enable the computer to execute the above data processing method.

[0028] A computer program product includes computer-readable instructions, and when the computer-readable instructions are executed by a processor, the above data processing method is implemented.

[0029] In the above technical solution:

[0030] By performing semantic feature extraction on multiple data points in the point cloud data of the specified device, the semantic label of each data point can be obtained. Next, the point cloud data is voxel-divided to obtain multiple first voxels, and a target data point is selected from each first voxel. The target data point is a representative point of the first voxel, which can be used to reduce the data density of the point cloud data, reduce the computational complexity, and maintain the key features of the first voxel. Next, for the semantic label of each target data point, the associated feature points that match each target data point can be determined from the preset local map point cloud. In this way, selecting the target data point can reduce the point calculation complexity, and combined with the semantic label of each target data point, each target data point can be more accurately matched to the associated feature point in the preset local map point cloud, so that the current posture information of the specified device calculated at the end is more accurate, thereby improving the accuracy of posture estimation.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0033] Figure 1 It is a schematic diagram of an implementation environment involved in this application;

[0034] Figure 2 It is a schematic diagram of the overall process of a data processing method;

[0035] Figure 3 is a flow chart of a data processing method according to an exemplary embodiment;

[0036] Figure 4 is a flow chart of a data processing method according to another exemplary embodiment;

[0037] Figure 5 is a flow chart of a data processing method according to another exemplary embodiment;

[0038] Figure 6 is a flow chart of a data processing method according to another exemplary embodiment;

[0039] Figure 7 is a flow chart of a data processing method according to another exemplary embodiment;

[0040] Figure 8 is a flow chart of a data processing method according to another exemplary embodiment;

[0041] Fig. 9 is a flow chart of a data processing method according to another exemplary embodiment;

[0042] Fig.10 is a block diagram of a data processing device according to an exemplary embodiment;

[0043] Fig.11 The figure is a schematic diagram of the hardware structure of a data processing device according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments that are the same as the present application. Instead, they are only examples of devices and methods that are the same as some aspects of the present application as detailed in the attached claims.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be integrated or partially integrated, so the actual execution order may change according to actual conditions.

[0047] It should be noted that the "multiple" mentioned in this application refers to two or more than two. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship.

[0048] It should be noted that in the specific implementation of the present application, when the user-related data is involved, when the embodiment of the present application is applied to a specific product or technology, the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. At the same time, the formulas involved in the embodiments of the present application can be flexibly adjusted, such as increasing or decreasing the corresponding parameters.

[0049] Before introducing the technical solutions of the embodiments of the present application, the technical terms involved in the embodiments of the present application are first introduced here.

[0050] LiDAR (Light Detection and Ranging) is a sensor technology used to measure distance and create high-resolution maps or point clouds. It works on the principle of laser beams, determining the distance of objects or terrain by emitting laser pulses and measuring their return time. LiDAR usually includes a laser transmitter, a receiver, a scanning mechanism, a time measurement unit, and a computing unit. The working principle of LiDAR is to emit a laser beam and then measure the time from the laser beam to the target object and back again. By knowing the speed of light, the distance can be calculated. By rotating the laser beam or scanning multiple laser beams, multiple distance points can be obtained, thereby creating high-resolution three-dimensional maps or point cloud data to help devices or vehicles perceive their surroundings.

[0051] The Iterative Closest Point (ICP) algorithm is an iterative optimization method for point cloud registration (alignment). It matches point clouds by finding the closest point pairs in two sets of point clouds and calculating the transformation between them. This process can be iterated repeatedly to optimize the match and minimize the distance between point clouds.

[0052] Simultaneous Localization and Mapping (SLAM) can be used to simultaneously determine the position of a robot or vehicle and create a map of the environment without knowing the map information of the environment in advance. SLAM usually uses sensor data, such as lidar or cameras, to estimate the position of the robot and build a map to achieve autonomous navigation.

[0053] Semantic segmentation in the field of point clouds refers to associating each point in the point cloud with a specific category or part of an object. This can help machine vision systems better understand point cloud data, thereby supporting various applications such as autonomous driving, robot navigation, building information modeling, etc. Semantic point cloud segmentation can be used to identify different objects or ground object categories in point clouds, such as buildings, trees, roads, pedestrians, vehicles, etc. This information is particularly important for autonomous vehicles because they need to identify roads, obstacles, pedestrians, etc., and make corresponding decisions. Typically, semantic point cloud segmentation is performed using deep learning methods such as convolutional neural networks (CNNs) or point cloud-specific neural network structures. These methods can extract features from point cloud data and assign points to different semantic categories. In the embodiment of the present application, the semantic segmentation network structure can use Cylinder3D, and some structures are removed to speed up the reasoning time, and the real-time performance of the module is verified by porting it to ROS2. In order to increase the reasoning speed, the feature dimension of each layer of the convolution layer is one-fourth of the original. If necessary, the segmentation accuracy can be improved by knowledge distillation.

[0054] A voxel in point cloud processing is a cube or cube-shaped area in three-dimensional space. Point cloud data is usually divided into many voxels for more efficient processing and analysis. Each voxel can be regarded as a small three-dimensional data block that contains point cloud data inside.

[0055] Downsampling refers to the process of reducing the number or density of data points in a point cloud. This is usually achieved by deleting some data points or merging multiple adjacent data points into one. Downsampling helps reduce computational load, improve processing speed, and make point cloud data easier to process. Downsampling methods can include downsampling methods that retain the original coordinates of the point cloud and voxel center downsampling methods. In an embodiment of the present application, the point cloud data can be downsampled using a downsampling method that retains the original coordinates of the point cloud.

[0056] In the related art, when performing pose estimation, a common practice is to use sensors such as lidar to collect point cloud data of the surrounding environment, and match this data with a known map to estimate the position and orientation of the robot. However, point cloud data is very large, especially in the case of high resolution. Processing a large number of data points requires a lot of computing resources, which may cause performance problems. In addition, point cloud data may be affected by noise and occlusion, and some data points may be inaccurate or missing. Therefore, the large amount of data and the influence of noise will result in low accuracy when finding the associated points of each data point in the map point cloud, resulting in low accuracy of the final pose estimation.

[0057] Based on this, the embodiments of the present application respectively propose a data processing method, a data processing device, a data processing equipment, a computer-readable storage medium, and a computer program product. In these embodiments, semantic features can be extracted from the point cloud data of a specified device to obtain a semantic label for each data point. Then, the data is voxel-divided and representative points are selected to reduce data density and computational complexity. Next, matching points are found in the map based on the semantic labels to reduce computational complexity and improve the accuracy of pose estimation.

[0058] See also Figure 1 , Figure 1 This is a schematic diagram of an implementation environment involved in this application

[0059] Figure 1 The illustrated implementation environment includes a designated device 110 and a server 130 .

[0060] exist Figure 1 The implementation environment shown includes a designated device with mobile capabilities (such as a robot, a car, a sweeping robot, etc.). The designated device (marked as 110) is equipped with a point cloud acquisition module (marked as 120), which is used to collect point cloud data in the environment. The point cloud acquisition module 120 can use different types of sensors, such as a lidar, an image sensor, or a time-of-flight (TOF) camera. It is worth noting that the point cloud acquisition module 120 of the designated device 110 can be a single sensor or a combination of multiple sensors. For example, it can be a lidar or a combination of a lidar and an image sensor. This makes the point cloud data collected by the point cloud acquisition module 120 more diverse and accurate, which helps to more accurately reflect the characteristics of the current environment.

[0061] It should be noted that Figure 1In the example, the point cloud acquisition module 120 is located on the top of the designated device 110. However, in actual applications, the installation position of the point cloud acquisition module 120 may be different depending on factors such as the type, structure and function of the designated device 110. This is not limited in this example, and the position of the point cloud acquisition module 120 can be freely selected according to specific actual needs.

[0062] This implementation environment may be used for various tasks, including but not limited to environmental perception, autonomous navigation, map construction, etc. In this process, the point cloud data collected by the point cloud acquisition module 120 will be transmitted to the server (marked as 130), and the server 130 plays the role of data processing and decision support to assist the designated device 110 to perform specific tasks.

[0063] The server 130 includes a semantic segmentation module 131 and a point cloud matching module 132. The semantic segmentation module 131 is responsible for semantic analysis of the point cloud data to distinguish different objects or different parts of the environment. The point cloud matching module 132 is used to match the collected point cloud data with the map to determine the current position of the designated device 110, that is, its position and orientation.

[0064] Optionally, the implementation environment may not include the server 130. In addition, the semantic segmentation module 131 and the point cloud matching module 132 may be integrated in the designated device 110. In this case, the designated device 110 will be responsible for the point cloud semantic segmentation and point cloud matching operations of the point cloud data, thereby more autonomously performing tasks such as environmental perception, autonomous navigation, and map construction without relying on an external server. This integrated approach makes the designated device 110 more independent and flexible, can adapt to various application scenarios, and reduces data transmission delays.

[0065] Therefore, according to specific needs, it can be selected whether to integrate the semantic segmentation module and the point cloud matching module in the designated device 110 according to actual conditions. In this way, flexible configuration helps to meet the needs of different application fields.

[0066] Combine the following Figure 2 The overall process of the data processing method of this application is described as follows:

[0067] The overall process can be executed by a computer, specifically involving a semantic segmentation module and a point cloud matching module. The semantic segmentation module includes process 1, process 2, and process 3. The point cloud matching module includes motion prediction compensation, dynamic instance point cloud removal, point cloud data downsampling, pose estimation, and pose information output. That is, semantic segmentation is a step in the point cloud matching process.

[0068] Specifically, after the computer receives the point cloud data, it can first perform motion prediction compensation on it. This is because the point cloud acquisition module takes a certain amount of time, such as 0.1 seconds, to collect the point cloud data. During this 0.1 second, the environmental information around the specified device may change. Therefore, the acquisition time of each point in the point cloud data needs to be compensated to the zero moment when the point cloud acquisition module starts collecting, so as to keep the acquisition time of all data points the same, thereby obtaining the corrected point cloud data and replacing the original point cloud data with the corrected point cloud data.

[0069] Next, the computer can input the point cloud data into process 1 in the semantic segmentation module. Process 1 mainly divides the point cloud data into voxels in the cylindrical coordinate system to obtain multiple voxels. At the same time, a point is randomly selected from the data points contained in each voxel as the designated data point, and a three-layer multi-layer perceptron (MLP) is performed on it to obtain the feature tensor of the designated data point, which can be used as the feature of the corresponding voxel. In this way, after processing by process 1, the feature tensors of multiple designated data points can be obtained. Then, in order to achieve real-time operation, the computer can process two frames of point cloud and package them and send them to the sampling queue of process 2.

[0070] In process 2, the computer can perform sparse 3D convolution on two frames of point clouds at the same time, including downsampling (AD) and upsampling (AU). Finally, through the dimension-decomposition based context modeling method (DDCM), the semantic label of each voxel can be obtained, and the semantic label can be assigned to all data points in the voxel, that is, the semantic label of a voxel is the same as the semantic label of all data points in the voxel. In this way, each data point in the point cloud data has a semantic label, and the point cloud data at this time can also be called a semantic point cloud. It can be understood that the semantic point cloud contains point cloud data and the semantic label of each data point of the point cloud data.

[0071] In process 3, the computer may publish the semantic point cloud generated in process 2, that is, the semantic point cloud may be sent to the point cloud matching module for subsequent processing. Optionally, in process 3, the length of time for publishing the semantic point cloud has a fixed delay with the original point cloud data, for example, 300ms.

[0072] After the point cloud is published, the computer can combine the semantic labels of each data point to remove the dynamic instance point clouds in the point cloud data. In this way, it can be ensured that the point cloud contains static instance point clouds, so that the current location information can be judged more accurately.

[0073] Next, the computer can downsample the point cloud data. Specifically, the point cloud with different labels in the point cloud data is divided into voxels of different volumes, and a target data point is extracted from each voxel as the representative point of the voxel to reduce the amount of calculation in the point cloud matching process. Based on the label information of multiple target data points, the computer can match the multiple target data points with the preset local map point cloud, and calculate the current posture information through robust optimization.

[0074] It should be noted that in the embodiment of the present application, when matching point clouds, line-to-line and surface-to-surface solutions such as the generalized iterative closest point algorithm (GICP) or the variational generalized iterative closest point algorithm (VGICP) can also be used to replace the point-to-point matching solution, and the output of the odometer can still be achieved while keeping other modules unchanged.

[0075] It should be noted that the designated device end in the embodiment of the present application may be a terminal device, which may be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile Internet device (MID, mobile internet device), a vehicle-mounted device, an aircraft, a wearable device (smart devices such as smart watches, smart bracelets, pedometers, etc.), a virtual reality device (for example, a VR (Virtual Reality) device, an AR (Augmented Reality) device), and the like.

[0076] In the embodiments of the present application, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0077] See also Figure 3 , Figure 3 is a flow chart of a data processing method according to an exemplary embodiment. The method can be applied to Figure 1 The implementation environment shown in the figure is specifically executed by a computer. Of course, the method can also be applied to other implementation environments, and the execution subject of the method is not limited here.

[0078] The data processing method will be described in detail below using a computer as an exemplary execution subject. Figure 3 As shown, in an exemplary embodiment, the method comprises at least the following steps:

[0079] S310, extracting semantic features from a plurality of data points contained in the point cloud data of the designated device to obtain semantic labels corresponding to the plurality of data points.

[0080] For the point cloud data collected by the specified device, the computer can analyze each data point through a specific algorithm or model to obtain semantic information related to the content of the data point. This process can identify the type of object, structure or environment represented by the data point, such as vehicle, wall, ground, etc., and assign a corresponding semantic label to each data point. Each semantic label reflects the semantic characteristics of the corresponding data point.

[0081] Optionally, the computer can obtain the position and posture information of the point cloud data of the preset number of frames before the designated device collects the point cloud data, and based on the obtained position and posture information, calculate the current predicted moving speed of the designated device, and based on the predicted moving speed and the type of the designated device, perform error correction of the acquisition time difference on the point cloud data, so that all data points in the point cloud data have the same acquisition time, and obtain corrected point cloud data. Thus, semantic features can be extracted for multiple data points contained in the corrected point cloud data to obtain semantic labels corresponding to the multiple data points.

[0082] Specifically, the computer can perform voxel division on the point cloud data to obtain multiple third voxels, select designated data points from the multiple data points contained in each third voxel, and perform semantic feature extraction on each designated data point to obtain a semantic label for each designated data point, thereby using the semantic label of each designated data point as the semantic label of the data point contained in the third voxel to which the designated data point belongs.

[0083] S320, performing voxel division on the point cloud data to obtain a plurality of first voxels, and selecting a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points.

[0084] The computer can divide the point cloud data into many first voxels, which can be cubic volume units, and each first voxel contains multiple point cloud data points. The division of these first voxels can simplify processing and reduce computational complexity while retaining key features of the point cloud. Within each first voxel, based on the semantic label of each data point, the computer can select a target data point from the first voxel. This target data point is considered to be a representative point of the voxel, which is used to reduce the data density of the point cloud data, thereby processing it more efficiently.

[0085] Specifically, the computer can classify multiple data points based on the semantic labels corresponding to the multiple data points contained in the point cloud data to obtain multiple instance point clouds; wherein the same instance point cloud includes data points corresponding to the same type of semantic labels, and then, based on the type information corresponding to the multiple instance point clouds, the point cloud data is voxel-divided to obtain multiple first voxels.

[0086] In one embodiment of the present application, when point cloud data is voxel-divided to obtain multiple first voxels based on type information respectively corresponding to multiple instance point clouds, the computer can determine the voxel volume for voxel division of each instance point cloud based on the type information respectively corresponding to the multiple instance point clouds, and perform voxel division on each instance point cloud based on the voxel volume corresponding to each instance point cloud to obtain multiple first voxels.

[0087] Optionally, when multiple first voxels are obtained by division, the computer can determine a dynamic instance point cloud from the point cloud data based on the type information corresponding to the multiple instance point clouds, remove the dynamic instance point cloud, obtain updated point cloud data, and finally perform voxel division on the updated point cloud data to obtain multiple first voxels. In this way, after the dynamic example point cloud is removed, the noise data in the point cloud data can be reduced, and the accuracy of subsequent point cloud matching can be improved.

[0088] In one embodiment of the present application, when identifying a dynamic example point cloud, the computer can first detect static instance point clouds among multiple instance point clouds based on type information corresponding to the multiple instance point clouds, and then, for each instance point cloud, detect static instance point clouds within a preset range around the instance point cloud, and identify whether the instance point cloud is a dynamic instance point cloud based on the static instance point cloud.

[0089] If, within the preset range of the instance point cloud, the proportion of data points of the static instance point cloud to all data points within the preset range is less than a preset proportion threshold, the instance point cloud is determined to be a dynamic instance point cloud.

[0090] S330, based on the semantic label of each target data point, determining the associated feature points matching each target data point from the preset local map point cloud.

[0091] Using the semantic label information of the target data point, the computer searches for associated feature points that match each target data point in the preset local map point cloud. These associated feature points represent the locations in the map that correspond to the target data points in the point cloud data. In this way, the collected point cloud data can be matched with the map to understand the current environment and location of the specified device.

[0092] Specifically, the computer can predict the current posture information of the designated device based on the point cloud data of the previous preset number of frames of point cloud data, and transform the point cloud data to the world coordinate system based on the predicted posture information to obtain the transformed point cloud data, and finally, based on the transformed point cloud data and the semantic label of each target data point, determine the associated feature points matching each target data point from the preset local map point cloud. Among them, the point cloud data is initially in the radar coordinate system. When the predicted posture information is known, the point cloud data of the radar coordinate system can be transformed to the world coordinate system, so that the position coordinates of each data point in the world coordinate system can be known.

[0093] Wherein, when the associated feature points matching each target data point are determined in the preset local map point cloud, the computer can perform voxel division on the transformed point cloud data to obtain a plurality of second voxels, obtain the position information of the second voxel to which the target data point belongs for each target data point, and obtain the data points contained in the voxels matching the position information in the preset local map point cloud. Furthermore, the computer can calculate the distance value between the target data point and the acquired data point based on the semantic label of the target data point and the semantic label corresponding to the acquired data point, and use the data point corresponding to the minimum distance value as the associated feature point of the target data point.

[0094] S340, calculating the current posture information of the designated device based on the multiple associated feature points.

[0095] After obtaining the associated feature points of each target data point, the computer performs the following steps to calculate the current posture information of the specified device. First, the computer adds the distance values ​​between each target data point and its corresponding associated feature points to obtain the total distance value under the current predicted posture information. Next, through robust optimization and iterative closest point algorithm (ICP), the computer gradually replaces the current predicted posture information with other possible posture information, and after each change of the predicted posture information, it re-searches for each target data point and its corresponding associated feature points, and calculates the total distance value at this time. This process is repeated until a certain predicted posture information has a corresponding total distance value less than a preset critical value, and the predicted posture information at this time can be used as the current posture information of the specified device. Among them, the preset critical value can be set by a technician in this field.

[0096] Among them, except for the first predicted pose information which is predicted based on the point cloud data of a preset number of frames before the point cloud data, the second and subsequent predicted pose information are calculated based on the previous predicted pose information. For example, the predicted pose information k (k≠1) is calculated based on the predicted pose information k-1.

[0097] It should be noted that if the change between the current predicted posture information k and the predicted posture information k-1 is less than the preset change threshold, it means that the total distance value corresponding to the predicted posture information k is already small enough and cannot be significantly reduced further. There is little point in further iteration. Therefore, the predicted posture information k can be directly used as the current posture information of the specified device.

[0098] It should be noted that in the process of pose estimation, as the predicted pose information changes, the computer needs to re-divide the point cloud data transformed to the world coordinate system corresponding to the current predicted pose information into multiple second voxels. This process is actually an iterative optimization process, in which each change of the predicted pose information will affect the distribution of the target data points, that is, the same target data point may be divided into different voxels under different predicted pose information.

[0099] In one embodiment of the present application, the i-th target data point in the point cloud data in the radar coordinate system can be represented as p i Point, the target data point in the point cloud data (i.e. the transformed point cloud data) in the world coordinate system is represented as Tp i Point, the preset local map point cloud, Tp i The associated feature point of the point is represented by q i Point, in this way, the total distance value ΔT under the k-th predicted pose information can be calculated k , and its calculation formula is:

[0100]

[0101] Among them, ‖Tp i -q i ‖2 represents Tp i Dot and q i The distance between points can be Euclidean distance, where n is Tp in the point cloud data. i Point (or p i points). m is the robust kernel function, σ t It is a parameter that can be adjusted online adaptively.

[0102] In this way, the computer can iteratively calculate the total distance value based on the predicted posture information until the total distance value of a certain predicted posture information is less than the preset critical value. The predicted posture information at this time can be used as the current posture information of the specified device. Based on the current posture information, the computer can obtain the odometer output accordingly.

[0103] In one embodiment of the present application, in order to achieve efficient storage and fast neighboring search, the present application embodiment proposes a semantic voxel map based on hash table storage to process key semantic label information with higher storage density. In the preset local map update process, for each data point that needs to be added to the map, its voxel coordinates are first calculated. Each voxel includes two storage units: one for regular data point storage and the other for key data point storage. When the regular data point storage unit does not reach the upper limit of capacity, the new point cloud data is stored in sequence at the end of the unit. Once the regular data point storage unit reaches the upper limit of capacity, the data points are divided into two categories according to the semantic labels of the new point cloud: regular point cloud and key point cloud (such as road signs, tree trunks, etc.). Regular point cloud is used to replace the unlabeled point cloud (usually the distant point cloud during semantic segmentation) in the regular data point storage unit, while the key point cloud is stored in the key data point storage unit. In addition, according to the real-time pose information of the specified device, voxels that are too far away from the center point of the specified device are removed to optimize memory usage.

[0104] Through this method, the computer first extracts semantic features from each data point in the point cloud data of the specified device to obtain the semantic label of each data point. Next, it divides the point cloud data into voxels to obtain multiple first voxels. In each first voxel, the computer selects a specific data point as the target data point, which represents the entire first voxel and is also used to reduce the density of the point cloud data, thereby reducing the computational complexity but maintaining the key features.

[0105] Next, the computer determines the associated feature points that best match each target data point from the preset local map point cloud based on the semantic label of each target data point. This process helps improve the accuracy of the match because it combines semantic information, allowing each target data point to find a matching point in the map more accurately, thereby improving the accuracy of the pose estimation. Finally, by integrating the information of multiple associated feature points, the computer calculates the current pose information of the specified device, which improves the accuracy of the pose estimation.

[0106] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Figure 4 As shown, the data processing method may include S310 to S320, S410 to S430 and S340. That is, S410 to S430 are Figure 3 The specific implementation method of S330 is shown.

[0107] S410 to S430 are described below:

[0108] S410, predicting the current position and posture information of the designated device based on point cloud data of a previous preset number of frames of point cloud data.

[0109] The computer uses a preset number of point cloud data, which are historical data collected from the specified device, such as the two frames of point cloud data before the current frame of point cloud data, to predict the current pose information of the specified device. This prediction process can be understood as the computer trying to estimate where the specified device is currently located and where it is facing.

[0110] S420, transforming the point cloud data into a world coordinate system based on the predicted posture information to obtain transformed point cloud data.

[0111] The computer transforms the current point cloud data from the radar coordinate system to the world coordinate system based on the predicted pose information (also called predicted pose information). Because the point cloud data is collected in the radar coordinate system of the specified device, in order to match the preset local map point cloud (in the world coordinate system), the point cloud data needs to be converted to a world coordinate system.

[0112] It should be noted that the computer can calculate multiple predicted posture information, and the current predicted posture information can be expressed as the kth predicted posture information.

[0113] S430, based on the transformed point cloud data and the semantic label of each target data point, determining the associated feature points matching each target data point from the preset local map point cloud.

[0114] Specifically, S430 may include S431 to S434.

[0115] S431 to S434 are described below:

[0116] S431, performing voxel division on the transformed point cloud data to obtain a plurality of second voxels.

[0117] The preset local map point cloud is also divided into voxels. At this time, the computer can divide the transformed point cloud data into voxels based on the voxel division method of the preset local map point cloud to obtain multiple second voxels. In this way, each second voxel can find a matching voxel in the preset local map point cloud.

[0118] S432, for each target data point, obtain the position information of the second voxel to which the target data point belongs, and obtain the data point contained in the voxel matching the position information in the preset local map point cloud.

[0119] For each target data point, the position information of the second voxel to which the target data point belongs can be obtained. Then, the computer searches for voxels with matching position information in the preset local map point cloud, and extracts all data points contained in the voxel. Among them, the voxels with matching position information in the preset local map point cloud can be the voxels corresponding to the position information and their neighboring voxels. The voxel corresponding to the position information can be regarded as the center of a 3*3*3 cube. The cube has a total of 27 blocks. Corresponding to the world coordinate system, it is equivalent to the cube having 27 voxels. These 27 voxels can all be used as voxels matching the position information.

[0120] For example, the coordinates of a target data point 1 in the transformed point cloud data are (35, 42, 50), and the coordinates of the second voxel to which it belongs are (3, 4, 5). Then the computer can search for the voxel with coordinates (3, 4, 5) and its adjacent voxels in the preset local map point cloud, and extract all the data points in these voxels.

[0121] S433, calculating the distance value between the target data point and the acquired data point based on the semantic label of the target data point and the semantic label corresponding to the acquired data point.

[0122] Each data point in the preset local map point cloud also has a semantic label. For each target data point, the computer can combine its semantic label with the semantic labels of the acquired multiple data points to calculate the distance values ​​between the target data point and the multiple data points. It can be assumed that the number of the acquired multiple data points is m.

[0123] For example, as in the above example, the target data point 1 belongs to the second voxel with coordinates (3, 4, 5), and the corresponding voxel in the preset local map point cloud contains m = 40 data points. Then the computer needs to calculate the distance value between the target data point 1 and these 40 data points respectively.

[0124] Specifically, the computer can obtain the target data point (ie, Tp i point), and the semantic label of each data point (i.e., q j point, where j represents the jth data point among the multiple data points), and detect the type of each semantic label. Based on the type of the semantic label, calculate the distance value between the target data point and the multiple data points. The distance value can be a Euclidean distance value with parameters, which can be calculated by the following formula:

[0125] d sem =γd

[0126] Where, d represents Tp i Dot and qj The Euclidean distance between points, γ is based on Tp i The semantic label of the point (also known as p i The semantic labels of q are the same j The parameters determined by the relationship between the semantic labels of the points are calculated as follows:

[0127]

[0128] Wherein, γ0 is a constant between 0 and 1, which can be set by a person skilled in the art, and l=0 represents an unlabeled point cloud. Indicates p i The semantic tag type, Indicates q j The semantic label type of the point. express and At least one of them is of unlabeled type.

[0129] The calculated d sem , we can make the Euclidean distance values ​​of semantic labels of the same type as small as possible, and the Euclidean distance values ​​of semantic labels of different types as large as possible. This can reduce the interference of data points of different types but with small Euclidean distance values ​​and improve the accuracy of point cloud matching.

[0130] S434, taking the data point corresponding to the minimum distance value as the associated feature point of the target data point.

[0131] In this way, each target data point can find a matching associated feature point in the preset local map point cloud, forming multiple associated feature points. That is, multiple target data points can find multiple associated feature points that match each other in the preset local map point cloud. With the help of these multiple associated feature points, the computer can perform more accurate pose estimation.

[0132] Since the target data point is represented by p i , then we can put p i The corresponding associated feature point is represented as q i ,q i are multiple associated feature points (i.e. m q j ), and p i D sem The smallest data point.

[0133] It should be noted that since the multiple associated feature points at this time are determined based on the kth predicted posture information, if the total distance value corresponding to the kth predicted posture information is greater than the preset critical value, the kth predicted posture information can be replaced with the k+1th predicted posture information, so as to calculate the total distance value corresponding to the k+1th predicted posture information, until a total distance value corresponding to a predicted posture information is obtained that is less than the preset critical value, and the predicted posture information can be used as the current posture information of the specified device.

[0134] Through this method, the computer transforms the point cloud data into the world coordinate system based on the predicted posture information, and combines the semantic label of each target data point when determining the associated feature points of each target data point in the preset local map point cloud. Through the semantic label, the Euclidean distance between each target data point and the data point in the preset local map point cloud can be parameterized, which can remove the interference of noise and more accurately determine the associated feature points of each target data point, thereby improving the accuracy of point cloud matching and thus improving the accuracy of posture estimation.

[0135] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Figure 5 As shown, the data processing method may include S510 to S530 and S320 to S340. That is, S510 to S530 are Figure 3 The specific implementation method of S310 is shown.

[0136] S510 to S530 are described below:

[0137] S510, performing voxel division on the point cloud data to obtain a plurality of third voxels.

[0138] This allows the computer to break down the point cloud data into smaller chunks for further analysis and processing.

[0139] S520 , selecting a designated data point from the multiple data points contained in each third voxel, and performing semantic feature extraction on each designated data point to obtain a semantic label for each designated data point.

[0140] Specifically, the computer can input multiple specified data points into a three-layer multi-layer perceptron (MLP) to obtain the feature tensor of each specified data point, which can be used as the feature of the corresponding voxel. In this way, the feature tensor characteristics of multiple specified data points can be obtained. Then, in order to achieve real-time operation, the computer can process two frames of point cloud and package them and send them to the sampling queue of process 2.

[0141] The computer can perform sparse 3D convolution on two frames of point clouds simultaneously, including downsampling (AD) and upsampling (AU). Finally, through DDCM, the semantic label of each specified data point can be obtained, and the semantic label of each specified data point can be used as the semantic label of its corresponding voxel.

[0142] S530 , using the semantic label of each designated data point as the semantic label of the data point contained in the third voxel to which the designated data point belongs.

[0143] The computer uses the semantic label of each specified data point as the semantic label of all data points in the third voxel to which the specified data point belongs. This means that all data points in the same third voxel have the same semantic label, which helps the system better understand the semantic meaning of this voxel.

[0144] Through this method, the computer's voxel division and semantic information extraction of point cloud data helps to more comprehensively understand the semantic characteristics and distribution of point cloud data, thereby providing more information support for subsequent pose estimation.

[0145] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Figure 6 As shown, the data processing method may include S310, S610 to S630 and S330 to S340. That is, S610 to S630 are Figure 3 The specific implementation method of S320 is shown.

[0146] S610 to S630 are described below:

[0147] S610, classifying multiple data points based on semantic labels corresponding to multiple data points contained in the point cloud data to obtain multiple instance point clouds; wherein the same instance point cloud includes data points corresponding to the same type of semantic labels.

[0148] Computers can classify point cloud data based on the semantic labels of each data point. This classification process divides the point cloud data into multiple instance point clouds, where each instance point cloud contains data points corresponding to the same type of semantic labels. That is, in the same instance point cloud, all data points belong to the same semantic category or type. The method for generating instance point clouds can adopt the Euclidean clustering method.

[0149] Specifically, the computer can detect the semantic label of each data point and detect the instance type to which the semantic label of the data point belongs. If the instance type is an instance type that is necessarily stationary, such as a traffic light pole, a tree, a building, etc., the data points of the instance type may not be clustered to form an instance point cloud. On the contrary, if the instance type is an instance type that cannot be determined whether it is stationary, such as a vehicle, a pedestrian, etc., the time period of the instance type may be clustered to form an instance point cloud.

[0150] For example, in the point cloud data, the data points corresponding to the semantic label representing car A can be Euclidean clustering to obtain the instance point cloud of car A; the data points corresponding to the semantic label representing pedestrian B can be Euclidean clustering to obtain the instance point cloud of pedestrian B, and so on.

[0151] S620, based on type information respectively corresponding to the plurality of instance point clouds, perform voxel division on the point cloud data to obtain a plurality of first voxels.

[0152] Specifically, S620 may include S621 to S622.

[0153] S621 to S622 are described below:

[0154] S621, based on type information respectively corresponding to a plurality of instance point clouds, determine a voxel volume for voxel division of each instance point cloud.

[0155] The computer uses the type information corresponding to each of the multiple instance point clouds to determine the volume of each instance point cloud divided by voxels. That is, different instance point clouds may require voxels of different sizes to better represent their features. Based on the type information, the computer defines a voxel volume of appropriate size for each instance point cloud.

[0156] For example, for an example point cloud of a vehicle type, the computer may divide it into voxels using voxels of 0.4m*0.4m*0.4m; for an example point cloud of a building type, the computer may divide it into voxels using voxels of 1m*1m*1m.

[0157] S622 , based on the voxel volume corresponding to each instance point cloud, perform voxel division on each instance point cloud to obtain a plurality of first voxels.

[0158] It can be understood that, among the multiple first voxels, different instance point clouds may have first voxels with different volumes.

[0159] S630 , selecting a target data point from the multiple data points contained in each first voxel based on the semantic labels corresponding to the multiple data points.

[0160] This process can be called a downsampling process. The downsampling method that retains the original coordinates of the point cloud can be used to perform downsampling, and the target data point is selected from the multiple data points contained in each first voxel to retain more accurate point cloud spatial information.

[0161] Through this method, the computer performs semantic classification and voxel division of point cloud data, providing more semantic and structural information about the point cloud data, which is helpful for subsequent pose estimation and environmental perception tasks.

[0162] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Figure 7 As shown, the data processing method may include S310, S610, S710 to S730, S630 and S330 to S340. That is, S710 to S730 are Figure 6 The specific implementation method of S620 is shown.

[0163] S710 to S730 are described below:

[0164] S710, determining a dynamic instance point cloud from the point cloud data based on type information corresponding to the plurality of instance point clouds.

[0165] Specifically, S710 may include S711 to S712.

[0166] S711 to S712 are described below:

[0167] S711, based on the type information corresponding to the multiple instance point clouds, detect the static instance point clouds among the multiple instance point clouds.

[0168] The computer can detect static instance point clouds based on the type information corresponding to each of the multiple instance point clouds. In this way, it can be determined which instance point clouds are static, that is, they do not change or move over time, and the static instance point clouds can represent unchanging objects or landmarks in the environment.

[0169] S712: for each instance point cloud, detect static instance point clouds existing within a preset range around the instance point cloud, and identify whether the instance point cloud is a dynamic instance point cloud based on the static instance point clouds.

[0170] It should be noted that the static instance point cloud here may refer to a static instance point cloud of a preset type, such as a parking lot, a zebra crossing, a sidewalk, etc., which hardly moves with the change of time. As for a car parked on the roadside, although it is also a static instance point cloud, it is more likely to move, so the dynamic instance point cloud detection will not be performed based on these static instance point clouds.

[0171] Therefore, in fact, the preset type of static instance point cloud here is a kind of prior knowledge, which is helpful for the detection and distinction of dynamic point clouds.

[0172] In one embodiment of the present application, if within a preset range of a certain instance point cloud, the proportion of data points of a static instance point cloud to all data points within the preset range is less than a preset ratio threshold, the instance point cloud is determined to be a dynamic instance point cloud. The preset range can be a sphere range of a certain diameter expanded with the instance point cloud as the center, or a cube range, which is not limited here.

[0173] For example, for a moving vehicle A, the computer detects whether there is a static instance point cloud such as a parking lot, a sidewalk or a zebra crossing within the preset range of the instance point cloud corresponding to the vehicle A. If so, it is necessary to detect the ratio of the total number of data points of the instance point cloud of the parking lot, the sidewalk or the zebra crossing to all data points within the preset range, for example, 50%. Assuming that the preset ratio threshold is 60%, it can be determined that the instance point cloud corresponding to the vehicle A is a dynamic instance point cloud.

[0174] S720: Remove the dynamic instance point cloud to obtain updated point cloud data.

[0175] That is, the computer can directly delete the dynamic instance point cloud in the point cloud data to obtain updated point cloud data.

[0176] It should be noted that, since the dynamic instance point cloud has been deleted, the semantic labels corresponding to the deleted data points have lost their meaning of existence, and the computer can delete them together to save storage resources.

[0177] S730, performing voxel division on the updated point cloud data to obtain a plurality of first voxels.

[0178] The voxel division method here is the same as the voxel division method described in S620, and will not be described in detail here.

[0179] In this method, dynamic instance point clouds usually represent moving objects or temporary changes in the environment. Deleting these dynamic instance point clouds can reduce noise, improve the quality of point cloud data, and help obtain more accurate and stable results in subsequent point cloud processing and pose estimation. In addition, dynamic point clouds are usually large in number, and retaining them will increase computational complexity and storage requirements. By deleting dynamic point clouds, the scale of point cloud data can be reduced and computational efficiency can be improved.

[0180] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Figure 8As shown, the data processing method may include S810 to S840 and S320 to S340. That is, S810 to S840 are Figure 3 The specific implementation method of S310 is shown.

[0181] S810 to S840 are described below:

[0182] S810, obtaining the position and posture information of the point cloud data of the preset number of frames before the specified device collects the point cloud data.

[0183] The preset number may be two, that is, the computer may obtain the posture information corresponding to the two frames of point cloud data before the current point cloud data.

[0184] S820, calculating the current predicted moving speed of the designated device based on the acquired posture information.

[0185] The predicted moving speed may include an angular speed and a linear speed.

[0186] Optionally, the computer may also directly obtain the angular velocity and linear velocity from a wheel odometer or an inertial measurement unit (IMU).

[0187] S830, based on the predicted moving speed and the type of the specified device, error correction of the acquisition time difference is performed on the point cloud data so that all data points in the point cloud data have the same acquisition time, thereby obtaining corrected point cloud data.

[0188] Different types of designated equipment may use different mechanical point cloud acquisition modules. Δt represents the scanning interval between two adjacent frames of point cloud data collected by the point cloud acquisition module. The scanning time difference of each data point relative to the first batch of data points in the current frame is s. i ∈[0,Δt], therefore, for each point p in the point cloud data i ∈P, the corrected point cloud data can be expressed as:

[0189]

[0190] Among them, Exp:R 3 →SO(3) represents the rotation matrix, w t and v t represent angular velocity and linear velocity respectively.

[0191] S840, extracting semantic features from the multiple data points contained in the corrected point cloud data to obtain semantic labels corresponding to the multiple data points.

[0192] The semantic feature extraction process here is the same as the extraction process in S310, and will not be described in detail here.

[0193] Through this method, the computer can unify the collection time of each data point in the point cloud data to make the point cloud data more reliable, thereby improving the accuracy of subsequent point cloud matching and the accuracy of pose estimation.

[0194] In one embodiment of the present application, another data processing method is provided, which can be executed by a computer. Fig. 9 As shown, the data processing method may include S901 to S923.

[0195] S901 to S923 are described below:

[0196] S901, receiving point cloud data from a specified device.

[0197] S902, performing error correction of acquisition time difference on the point cloud data so that all data points in the point cloud data have the same acquisition time, thereby obtaining corrected point cloud data.

[0198] S903, using the corrected point cloud data as the point cloud data of the designated device.

[0199] S904, extracting semantic features from a plurality of data points contained in the point cloud data of the designated device to obtain semantic labels corresponding to the plurality of data points.

[0200] S905 , performing voxel division on the point cloud data to obtain n first voxels.

[0201] It should be noted that, before executing S905, the computer may also remove the dynamic instance point cloud in the point cloud data, which is not described in detail in the above embodiment and will not be described here.

[0202] S906 , based on the semantic labels corresponding to the multiple data points, select a target data point from the multiple data points contained in each first voxel to obtain n target data points.

[0203] S907, set k=1.

[0204] S908, predicting the current posture information of the designated device based on the point cloud data of a preset number of frames before the point cloud data, and obtaining predicted posture information k.

[0205] Here, k is a constant greater than or equal to 1.

[0206] S909, transforming the point cloud data into a world coordinate system based on the predicted posture information k to obtain transformed point cloud data.

[0207] S910, performing voxel division on the transformed point cloud data to obtain a plurality of second voxels.

[0208] S911, let i=1.

[0209] S912, obtain the i-th target data point (Tp i The position information of the second voxel to which the point belongs.

[0210] S913, obtaining m data points contained in the voxel matching the position information in the preset local map point cloud.

[0211] S914, let j=1.

[0212] S915, calculate Tp i Dot and q j The Euclidean distance value d of the point with parameters sem .

[0213] S916, whether j is equal to m.

[0214] If yes, execute S916; if no, set j=j+1 and execute S914.

[0215] S917, the calculated multiple d sem The data point corresponding to the minimum distance value in is taken as Tp i The associated feature point q o .

[0216] S918, whether i is equal to n.

[0217] If yes, execute S919; if no, set i=i+1 and execute S912.

[0218] S919, calculating the total distance value ΔT corresponding to the predicted posture information k k .

[0219] S920: Whether the total distance value is less than a preset critical value.

[0220] If yes, execute S923; if no, and k=1, execute S922; if no, and k≠1, execute S921.

[0221] S921, ΔT k With ΔT k-1 Whether the change between them is less than the preset change threshold.

[0222] That is, the computer needs to calculate that the change between the total distance values ​​of the predicted posture information k and the predicted posture information k-1 is less than the preset change threshold.

[0223] If yes, execute S923; if no, execute S922.

[0224] In this case, k≠1, and the calculation formula of the change is: Change = |ΔT k -ΔT k-1 |.

[0225] S922, calculate the predicted posture information k+1, and set k=k+1.

[0226] After executing S922, execute S909.

[0227] Among them, the predicted posture information k+1 is calculated based on the total distance value corresponding to the predicted posture information k.

[0228] S923, using the predicted posture information k at this time as the current posture information of the designated device.

[0229] Through this method, the computer determines the associated feature points that best match each target data point from the preset local map point cloud based on the semantic label of each target data point. This process helps improve the accuracy of matching because it combines semantic information, allowing each target data point to find a matching point in the map more accurately, thereby improving the accuracy of pose estimation. Finally, by integrating the information of multiple associated feature points, the computer calculates the current pose information of the specified device, which improves the accuracy of pose estimation.

[0230] The effect of the data processing method of this application is analyzed in combination with experimental data.

[0231] The data processing of this application is rich and has low demand for hardware resources. It does not require additional work such as complex multi-device time synchronization and calibration. At the same time, it is more accurate in filtering dynamic instance point clouds and has better real-time performance.

[0232] This application combines semantic segmentation and point cloud matching to achieve real-time odometer output. By directly integrating semantic information into the classic ICP matching algorithm, it avoids the complex algorithm design of back-end optimization, improves accuracy and reduces the computational load, which is almost the same as the computational load of the classic algorithm. Therefore, it can be run on most portable or vehicle-mounted devices, which is conducive to the implementation of the algorithm.

[0233] Based on this data processing method, the experiment tested the first five sequences in the KITTI Road dataset (2011_09_26_drive_0015-2011_09_26_drive_0032). These sequences were collected on rural roads or high-speed highways, with more high-speed dynamic vehicles and fewer geometric feature points, which are challenging for SLAM. This application is compared with the current point cloud matching method-KISS-ICP. The test parameters are relative translation error (RelativeTranslation Error, RTE) and relative rotation error (Relative Rotational Error, RRE), which are defined by KITTI as the mean of the trajectory error from 100m to 800m, as shown in Table 1 below:

[0234]

[0235] Table 1

[0236] As can be seen from Table 1, in terms of the performance of RTE and RRE, the schemes of this application are almost all better than KISS-ICP.

[0237] At the same time, the experiment also verified the real-time performance of the system. The average time required for each module is as follows:

[0238] As shown in Table 2:

[0239]

[0240]

[0241] Table 2

[0242] The registration frame rate of this application is higher than the LiDAR frequency, about 11.60Hz, and each process of the semantic segmentation part takes less than 200ms, indicating that this application can be applied to real-world scenarios. The test environment of this application is a laptop equipped with a 4.70GHz 14-core Intel i7-12700H CPU, 16GB RAM, 8GB Nvidia GeForce RTX 3070Ti, and the system is Ubuntu 20.04, ROS2 Humble.

[0243] Fig.10 FIG. 1 is a block diagram of a data processing device according to an embodiment of the present application. Fig.10 As shown, the data processing device can be applied to a computer, and the device includes:

[0244] An extraction unit 1010 is used to extract semantic features from a plurality of data points contained in the point cloud data of a specified device to obtain semantic labels corresponding to the plurality of data points;

[0245] The processing unit 1020 is used to perform voxel division on the point cloud data to obtain a plurality of first voxels, and select a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points;

[0246] The processing unit 1020 is further configured to determine, based on the semantic label of each target data point, associated feature points matching each target data point from a preset local map point cloud;

[0247] The calculation unit 1030 is used to calculate the current posture information of the specified device based on multiple associated feature points.

[0248] In one embodiment of the present application, based on the aforementioned scheme, the computing unit 1030 is further used to predict the current posture information of the specified device based on the point cloud data of a previous preset number of frames of the point cloud data; the processing unit 1020 is further used to transform the point cloud data to the world coordinate system based on the predicted posture information to obtain the transformed point cloud data; based on the transformed point cloud data and the semantic label of each target data point, determine the associated feature points matching each target data point from the preset local map point cloud.

[0249] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is further used to perform voxel division on the transformed point cloud data to obtain multiple second voxels; for each target data point, the acquisition unit 1040 is used to obtain the position information of the second voxel to which the target data point belongs, and the processing unit 1020 is further used to obtain the data points contained in the voxels that match the position information in the preset local map point cloud; based on the semantic label of the target data point and the semantic label corresponding to the acquired data point, the distance value between the target data point and the acquired data point is calculated; and the data point corresponding to the smallest distance value is used as the associated feature point of the target data point.

[0250] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is further used to perform voxel division on the point cloud data to obtain multiple third voxels; select a specified data point from the multiple data points contained in each third voxel, and perform semantic feature extraction on each specified data point to obtain a semantic label for each specified data point; and use the semantic label of each specified data point as the semantic label of the data point contained in the third voxel to which the specified data point belongs.

[0251] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is also used to classify multiple data points based on the semantic labels corresponding to the multiple data points contained in the point cloud data to obtain multiple instance point clouds; wherein the same instance point cloud includes data points corresponding to the same type of semantic labels; based on the type information corresponding to the multiple instance point clouds, the point cloud data is voxel-divided to obtain multiple first voxels.

[0252] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is also used to determine the voxel volume for voxel division of each instance point cloud based on the type information corresponding to the multiple instance point clouds; based on the voxel volume corresponding to each instance point cloud, voxel division is performed on each instance point cloud to obtain multiple first voxels.

[0253] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is also used to determine a dynamic instance point cloud from the point cloud data based on type information corresponding to multiple instance point clouds; remove the dynamic instance point cloud to obtain updated point cloud data; and perform voxel division on the updated point cloud data to obtain multiple first voxels.

[0254] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is also used to detect static instance point clouds among multiple instance point clouds based on type information corresponding to the multiple instance point clouds; for each instance point cloud, detect static instance point clouds within a preset range around the instance point cloud, and identify whether the instance point cloud is a dynamic instance point cloud based on the static instance point cloud.

[0255] In one embodiment of the present application, based on the aforementioned scheme, the processing unit 1020 is also used to determine that the instance point cloud is a dynamic instance point cloud if, within a preset range of the instance point cloud, the proportion of data points of the static instance point cloud to all data points within the preset range is less than a preset proportion threshold.

[0256] In one embodiment of the present application, based on the aforementioned scheme, the acquisition unit 1040 is further used to obtain the posture information of the point cloud data of the preset number of frames before the designated device collects the point cloud data; the calculation unit 1030 is further used to calculate the current predicted moving speed of the designated device based on the acquired posture information; the processing unit 1020 is further used to perform error correction of the acquisition time difference of the point cloud data based on the predicted moving speed and the type of the designated device, so that all data points in the point cloud data have the same acquisition time to obtain the corrected point cloud data; the extraction unit 1010 is further used to perform semantic feature extraction on the multiple data points contained in the corrected point cloud data to obtain semantic labels corresponding to the multiple data points.

[0257] It should be noted that the apparatus provided in the aforementioned embodiment and the method provided in the aforementioned embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment.

[0258] An embodiment of the present application also provides a data processing device, including: one or more processors; a memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the above data processing method.

[0259] Fig.11 It is a structural diagram of a computer system suitable for implementing a data processing device of an embodiment of the present application.

[0260] It should be noted that Fig.11 The computer system 1100 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0261] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1102 or the program loaded from the storage part 1108 to the random access memory (RAM) 1103, such as executing the method in the above embodiment. In the RAM 1103, various programs and data required for system operation are also stored. The CPU 1101, the ROM 1102 and the RAM 1103 are connected to each other through the bus 1104. The input / output (I / O) interface 1105 is also connected to the bus 1104.

[0262] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0263] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1109, and / or installed from a removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

[0264] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable media may be, for example, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0265] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0266] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0267] Another aspect of the present application also provides a computer readable medium having a computer program stored thereon, which implements the above data processing method when the computer program is executed by a processor. The computer readable medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0268] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable medium. A processor of a computer device reads the computer instruction from the computer-readable medium, and the processor executes the computer instruction, so that the computer device executes the data processing method provided in each of the above embodiments.

[0269] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A data processing method, characterized in that: include: Extracting semantic features of a plurality of data points contained in the point cloud data of the designated device to obtain semantic labels corresponding to the plurality of data points; Performing voxel division on the point cloud data to obtain a plurality of first voxels, and selecting a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points; Based on the semantic label of each target data point, determining the associated feature points matching each target data point from the preset local map point cloud; The current position information of the designated device is calculated based on the multiple associated feature points.

2. The method according to claim 1, characterized in that The determining, based on the semantic label of each target data point, from a preset local map point cloud, associated feature points matching each target data point includes: Predicting the current position and posture information of the designated device based on point cloud data of a previous preset number of frames of point cloud data; Based on the predicted posture information, the point cloud data is transformed into a world coordinate system to obtain transformed point cloud data; Based on the transformed point cloud data and the semantic label of each target data point, an associated feature point matching each target data point is determined from a preset local map point cloud.

3. The method according to claim 2, characterized in that The determining, based on the transformed point cloud data and the semantic label of each target data point, from a preset local map point cloud, an associated feature point matching each target data point comprises: Performing voxel division on the transformed point cloud data to obtain a plurality of second voxels; For each target data point, obtain the position information of the second voxel to which the target data point belongs, and obtain the data points contained in the voxels matching the position information in the preset local map point cloud; Calculating a distance value between the target data point and the acquired data point based on the semantic label of the target data point and the semantic label corresponding to the acquired data point; The data point corresponding to the minimum distance value is used as the associated feature point of the target data point.

4. The method according to claim 1, characterized in that: The extracting semantic features of the plurality of data points contained in the point cloud data of the designated device to obtain semantic labels corresponding to the plurality of data points respectively includes: Performing voxel division on the point cloud data to obtain a plurality of third voxels; Selecting a designated data point from a plurality of data points contained in each third voxel, and performing semantic feature extraction on each designated data point to obtain a semantic label of each designated data point; The semantic label of each designated data point is used as the semantic label of the data point contained in the third voxel to which the designated data point belongs.

5. The method according to claim 1, characterized in that The step of performing voxel division on the point cloud data to obtain a plurality of first voxels includes: Classifying the multiple data points based on the semantic labels corresponding to the multiple data points contained in the point cloud data to obtain multiple instance point clouds; wherein the same instance point cloud includes data points corresponding to the same type of semantic labels; Based on the type information respectively corresponding to the multiple instance point clouds, the point cloud data is voxel-divided to obtain multiple first voxels.

6. The method according to claim 5, characterized in that The step of performing voxel division on the point cloud data to obtain a plurality of first voxels based on the type information respectively corresponding to the plurality of instance point clouds comprises: Determine, based on the type information respectively corresponding to the plurality of instance point clouds, a voxel volume for voxel division of each instance point cloud; Based on the voxel volume corresponding to each instance point cloud, each instance point cloud is respectively voxel-divided to obtain the plurality of first voxels.

7. The method according to claim 5, characterized in that The step of performing voxel division on the point cloud data to obtain a plurality of first voxels based on the type information respectively corresponding to the plurality of instance point clouds comprises: Determining a dynamic instance point cloud from the point cloud data based on type information respectively corresponding to the plurality of instance point clouds; Removing the dynamic instance point cloud to obtain updated point cloud data; The updated point cloud data is voxel-divided to obtain a plurality of first voxels.

8. The method according to claim 7, characterized in that The determining of a dynamic instance point cloud from the point cloud data based on the type information respectively corresponding to the plurality of instance point clouds comprises: Based on the type information corresponding to the multiple instance point clouds, detecting static instance point clouds among the multiple instance point clouds; For each instance point cloud, the static instance point cloud existing within a preset range around the instance point cloud is detected, and based on the static instance point cloud, it is identified whether the instance point cloud is a dynamic instance point cloud.

9. The method according to claim 8, characterized in that The step of identifying whether the instance point cloud is a dynamic instance point cloud based on the static instance point cloud includes: If within the preset range of the instance point cloud, the proportion of the data points of the static instance point cloud to all the data points within the preset range is less than a preset proportion threshold, the instance point cloud is determined to be a dynamic instance point cloud.

10. The method according to claim 1, characterized in that The extracting semantic features of the plurality of data points contained in the point cloud data of the designated device to obtain semantic labels corresponding to the plurality of data points respectively includes: Obtaining the position and posture information of the designated device when collecting point cloud data of a preset number of frames before the point cloud data; Based on the acquired position information, calculate the current predicted moving speed of the designated device; Based on the predicted moving speed and the type of the designated device, error correction of the acquisition time difference is performed on the point cloud data so that all data points in the point cloud data have the same acquisition time, thereby obtaining corrected point cloud data; Semantic features are extracted from the plurality of data points contained in the corrected point cloud data to obtain semantic labels corresponding to the plurality of data points.

11. A data processing device, characterized in that: include: An extraction unit, used to extract semantic features from a plurality of data points contained in the point cloud data of a specified device, and obtain semantic labels corresponding to the plurality of data points respectively; A processing unit, configured to perform voxel division on the point cloud data to obtain a plurality of first voxels, and select a target data point from a plurality of data points contained in each first voxel based on semantic labels corresponding to the plurality of data points; The processing unit is further used to determine, based on the semantic label of each target data point, an associated feature point matching each target data point from a preset local map point cloud; A calculation unit is used to calculate the current posture information of the designated device based on multiple associated feature points.

12. A data processing device, characterized in that: include: a memory storing computer-readable instructions; A processor reads the computer-readable instructions stored in the memory to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 10.

Images