Unmanned platform collaborative perception method and device based on voxelization and medium

Abstract

The invention discloses an unmanned platform collaborative perception method and device based on voxelization and a medium, and belongs to the field of unmanned platform collaborative perception. Relative position positioning is completed in combination with sensing information between unmanned platforms, a blind area high in value for the platform is calculated according to the sensing information of the platform in a voxelization mode, and the cooperative sensing range is limited in a blind area of each agent in the system. And huge communication traffic caused by sending complete original data and the calculation amount of feature extraction based on a neural network are avoided. And finally, running a local multi-view joint feature optimization method based on blind area information of neighbor nodes to achieve a multi-unmanned-platform collaborative sensing effect with minimum redundancy and maximum complementarity.

Description

Technical Field

[0001] The present invention belongs to the field of collaborative perception of unmanned platforms, and in particular relates to methods such as collaborative positioning, data sharing and optimization, and multimodal fusion, and specifically relates to a voxel-based collaborative perception method, device, and medium for unmanned platforms. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] In the collaborative perception field for small unmanned vehicles and micro-UAVs, the weak perception capabilities of micro-UAVs and extreme environmental conditions often lead to problems such as insufficient perception range, blind spots, and insufficient precision due to sparse perception. Under these conditions, comprehensive, accurate, and robust environmental perception is difficult. Collaborative perception, which relies on communication networks to share and integrate perception information with neighboring nodes, can address these issues and meet the needs of unmanned systems for beyond-line-of-sight perception and blind-spot perception.

[0004] Collaborative perception among unmanned platforms relies on communication technology to enable real-time, dynamic, and on-demand sharing of multimodal sensor data across groups. Based on highly precise relative positional relationships, it fuses multi-perspective data to expand the breadth and depth of perception, addressing issues such as single-node perception failure in sparse and obscured areas. First, existing positioning methods cannot provide centimeter-level positioning accuracy in scenarios like cities and jungles, making it difficult to align multi-perspective sensor data. Second, existing network technologies struggle to support the sharing of raw perception data across large-scale unmanned platforms. Due to the high bandwidth and high frequency of interaction associated with perception data, shared communication resources among all nodes cannot support the exchange of all raw perception data across multiple platforms.

[0005] Therefore, in order to expand the application scenarios of collaborative perception, the present invention first introduces a real-time relative positioning algorithm based on graph matching to complete relative positioning between nodes that meets the accuracy requirements without high-precision maps; then introduces a voxelized blind spot calculation method to efficiently calculate high-collaborative value areas and transmit data corresponding to blind spots between multiple nodes, thereby ensuring collaborative perception performance while reducing communication volume. At the same time, through a joint optimization method for multi-perspective blind spot data, the collaborative objects are further optimized through the interaction and adaptive measurement of perception feature key frames, realizing efficient and high-quality intelligent unmanned platform collaborative perception. Supporting the application of unmanned platforms in military and civilian fields. Summary of the Invention

[0006] The purpose of the present invention is to address the following problems existing in the prior art:

[0007] 1. When considering collaboration between multiple platforms, existing technologies usually do not consider the selection and control of collaborative objects and spaces. This will result in the low-information cross-sensing space between multiple platforms being used for collaborative perception at the same time, leading to communication redundancy within the system.

[0008] 2. Each intelligent platform can only determine the relative position relationship between platforms with accurate coordinates in a synchronized coordinate system built on the basis of a high-precision local map. This means that before running collaborative perception in the corresponding scenario, it is necessary to collect and build a high-precision map in advance, which greatly limits the application scope of collaborative perception.

[0009] 3. Before selecting an effective collaborator, existing technologies need to conduct complete perception information interaction with all platforms in the system and run spatiotemporal feature extraction algorithms on a large amount of perception information. This requires large amounts of computation, large communication requirements, and highly redundant information interaction.

[0010] A voxel-based collaborative perception method, device, and medium for unmanned platforms are presented. These methods combine perception information between unmanned platforms to achieve relative positioning. Using voxelization, they calculate high-value blind spots for each platform based on its perception information. This limits the scope of collaborative perception to the blind spots of each agent in the system, avoiding the massive communication overhead associated with sending complete raw data and the computational complexity of neural network-based feature extraction. Finally, a local multi-perspective joint feature optimization method is implemented based on blind spot information from neighboring nodes, achieving multi-platform collaborative perception with minimal redundancy and maximum complementarity.

[0011] The technical solutions of the present invention are as follows:

[0012] A voxel-based unmanned platform collaborative perception method, comprising:

[0013] Step S1: The unmanned platform integrates its own lidar point cloud data and depth image data to generate a BEV feature map with semantic information, compresses the BEV feature map to obtain a compressed feature map, and exchanges the compressed feature map between the unmanned platform and its surrounding neighboring nodes. Based on the characteristics of multi-platform interaction, the unmanned platform collaborative positioning algorithm based on graph matching is used to obtain the relative position of the platform and its surrounding neighboring nodes by constructing a rotation matrix and a translation matrix between the platform coordinate system and the neighboring node coordinate system.

[0014] Step S2: The unmanned platform voxelizes the lidar point cloud data and uses a voxelized blind spot mining method to discover blind spots based on the density of the point cloud. After discovering the blind spot, the unmanned platform sends the spatial area of ​​the blind spot to other nodes, and multiple platforms interact with each other in the blind spot.

[0015] Step S3: Based on the discovered blind spots, a local multi-view joint information optimization algorithm is used to discover collaborative objects.

[0016] Furthermore, the BEV feature map generation method is as follows:

[0017] First, the unmanned platform integrates the lidar point cloud data and depth image data of the platform, uses the voxelization method to extract the BEV features of the radar point cloud, and projects the image pixel features with color semantic information onto the feature plane of the point cloud through voxel grid projection to construct a BEV feature map with semantic information.

[0018] Furthermore, the unmanned platform collaborative positioning algorithm based on graph matching includes:

[0019] First, a Gaussian kernel function G(x,y,σ) in a multi-scale space is generated from the image scale space, and the feature L(x,y,σ)=G(x,y,σ)*I(x,y) in the multi-scale space is calculated and extracted from the original BEV feature map grid I(x,y). Then, spatial extreme points are detected in the scale space as feature points in the BEV feature map, and the accumulated vector in the image gradient direction is extracted based on the feature points. If the domain grid of the feature point contains color semantic information, the RGB color histogram features in the gradient direction are extracted and spliced ​​with the accumulated vector to form the representation vector of the feature point grid in BEV.

[0020] Then, the Euclidean distance between the representation vectors of the feature points held by the platform and the neighboring nodes is calculated and the nearest neighbor search algorithm is run. At the same time, the color histogram of the matching results is double-checked, and only feature point pairs with consistent color features are retained to obtain a set of feature point pairs.

[0021] Finally, the optimal rotation matrix is ​​searched based on the RANSAC algorithm. Starting from a randomly initialized set of feature point pairs, point pairs are randomly sampled from feature point pairs with consistent color features each time, and the rotation matrix H that satisfies the spatial mapping relationship between each point pair is calculated. Then, the reverse reprojection error rate of all global point pairs under this rotation matrix is ​​calculated based on H.

[0022] Furthermore, the error rate reflects the distance and position correlation between the two platforms. The error rate threshold is used to control whether a reliable relative position conversion relationship is established with the corresponding neighboring nodes, and it is dynamically maintained and updated during operation.

[0023] Furthermore, the required area in the collaborative perception process of the unmanned platform is limited to the blind spot range, and the perception data sharing is completed based on the coordinate system conversion relationship between the blind spot and the collaborative positioning.

[0024] Furthermore, the voxelized blind area mining method includes:

[0025] First, the perception information demand space is delineated with this platform as the center. Then, the space is voxelized and evenly divided N times on each axis in the space to obtain the demand space composed of uniform grids. Then, the point cloud density in each grid is calculated and saved. Finally, the Bresenham algorithm in the field of computer vision is used to calculate the point density of a given point (x a ,y a ) and (x b ,y b ) covered grid;

[0026] Based on the output of the Bresenham algorithm, the platform's required area in the perception space is further limited to uncovered spatial areas as its blind spots, and the area is broadcast to neighboring nodes to share and request collaborative perception data. The platform also listens to requests from neighboring nodes and trims its own perception data and transmits it back to the neighboring nodes, completing the collaborative perception process based on the original data.

[0027] Furthermore, the Bresenham algorithm in the field of computer vision is used to calculate the given point (x a ,y a ) and (x b ,y b ) covered grids, including:

[0028] Whenever the line segment connecting two points passes through a grid (x i ,y i ) edge, calculate:

[0029] d1-d2=(yy i )-(y i +Δy-y)=2y-2y i -c

[0030] Where c is the grid size defined in the algorithm;

[0031] If d1-d2<0, then (x a ,y a ) is covered, otherwise it is considered that (x a ,y a +c) is covered.

[0032] Furthermore, the local multi-view joint information optimization algorithm includes:

[0033] For each blind spot, the neighboring node first sends the key frame of the feature data corresponding to the blind spot to this platform. This platform considers the data corresponding to the blind spot and the blind spot key frame transmitted by the neighboring node, learns the information volume measurement value of the key frame through the network, and then selects the most suitable neighbor for feature interaction based on the information volume measurement value; the key frame is equivalent to control signaling. Through the interaction of control signaling, it can determine which neighboring node to interact with, and finally transmit the BEV feature map corresponding to this neighboring node.

[0034] The present invention also proposes a voxel-based unmanned platform collaborative perception device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned voxel-based unmanned platform collaborative perception method.

[0035] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned voxel-based unmanned platform collaborative perception method.

[0036] Compared with the existing technology, the beneficial effects of the present invention are:

[0037] In existing collaborative perception technologies that are similar to the present invention, the problem of coordinate conversion in the absence of high-precision maps is generally not considered. Through a graph matching method based on perception information, the present invention gets rid of the problem of strong dependence of collaborative perception on high-precision maps and expands the use scenarios of collaborative perception. In addition, through efficient blind spot calculation and area clipping strategies, the collaborative perception strategy of the present invention can effectively support the direct sharing of the most credible original perception data between intelligent unmanned platforms in scenarios with limited communication resources, which is different from the method of sharing mid-term features extracted by neural networks in the prior art. On this basis, the present invention optimizes the collaborative objects of the unmanned platform by extracting and exchanging blind spot feature key frames and adaptive measurement of feature information, and further controls the problem of increasing redundant communication volume in the collaborative perception process as the number of unmanned platforms increases. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a voxel-based collaborative perception method for unmanned platforms;

[0039] Figure 2 This is a schematic diagram of the collaborative sensing data communication flow from the perspective of this platform;

[0040] Figure 3 Optimizing network structure graph for multi-view joint information. DETAILED DESCRIPTION

[0041] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0042] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0043] Example 1

[0044] This embodiment proposes a voxel-based unmanned platform collaborative perception method, which specifically includes the following three core algorithms: an unmanned platform collaborative positioning algorithm based on graph matching, a voxel-based blind spot collaborative perception method (i.e., the method adopted in step S2), and a local multi-view joint information optimization algorithm.

[0045] 1) Unmanned Platform Collaborative Localization Algorithm Based on Graph Matching

[0046] Because training and validation are typically performed using public datasets, existing collaborative perception systems or algorithms typically assume that the precise position of the unmanned platform in the global coordinate system is known. Collaborative perception between platforms then performs coordinate system conversion and matching based on this global position information. In real-world applications, inter-platform collaboration typically requires centimeter-level accuracy for perception information, but the errors in conventional GPS-based positioning are unacceptable for unmanned systems. Under these conditions, global positioning is typically accomplished using a high-precision map constructed using a SLAM algorithm. After matching the real-time point cloud to the map coordinate frame, the unmanned platform is converted to the global coordinate system. This approach has two drawbacks: First, a high-precision map of the operating area must be constructed before the unmanned platform can operate, which introduces complex alignment and computing power challenges as the scenario expands. Second, the platform must not leave the mapped area during operation; otherwise, the accumulated errors due to the lack of map information will lead to positioning errors.

[0047] In order to solve this problem, this embodiment considers an unmanned platform collaborative positioning algorithm based on graph matching. In a collaborative perception scenario, in order to align the coordinate systems between unmanned systems, it is actually only necessary to base it on the relative posture relationship between the two platforms, and it is unnecessary to transfer the two to the global coordinate system to complete the alignment in this scenario. Therefore, the platforms can construct a feature map with semantic information from the BEV perspective based on the perception information held by each party, and complete the feature map sharing through an efficient feature map compression method, and run a graph matching algorithm based on the result to determine the matching degree and posture transformation relationship between the semantic feature maps of both parties, and further remap it to the coordinate system transformation relationship from the BEV perspective, thereby completing efficient relative positioning.

[0048] 2) Voxel-based blind spot collaborative perception method

[0049] Existing collaborative algorithms are usually discussed at the theoretical level under the premise that "the amount of perceived information is evenly distributed", that is, after determining the collaborative object, the complete perceived information is usually shared, or the features or algorithm results extracted based on the complete perceived information. However, from the perspective of a single platform, this assumption does not hold. This is because the field of view of the platform and the field of view of the neighboring nodes usually overlap, and from the perspective of the platform, the amount of information contained in the perceived information in the overlapping area is much smaller than the amount of information contained in the perceived information in the blind area. Therefore, this embodiment proposes a voxel-based blind spot collaborative perception method. Under this method, when collaborating between platforms, the blind spots of the platforms will first be efficiently calculated and broadcast based on voxelization, and then the data of the platform will be shared based on the blind spots of the neighboring nodes.

[0050] 3) Local multi-view joint information optimization algorithm

[0051] Under the aforementioned collaborative perception method, scenarios may arise where multiple neighboring nodes possess the blind spot information required by the platform, and the frequency of such scenarios increases with the number of collaborating unmanned platforms. Given the limited communication and computing resources of this platform, requesting perception information from more nodes within the same blind spot region has diminishing returns. Therefore, adaptively selecting information through a local multi-perspective approach, and further optimizing and scheduling collaborative objects, can further improve communication resource utilization.

[0052] See also Figure 1 In this embodiment, a voxel-based unmanned platform collaborative perception method specifically includes the following steps:

[0053] Step S1: The unmanned platform integrates its own lidar point cloud data and depth image data to generate a BEV feature map with semantic information, compresses the BEV feature map to obtain a compressed feature map, and exchanges the compressed feature map between the unmanned platform and its surrounding neighboring nodes. Based on the characteristics of multi-platform interaction, the unmanned platform collaborative positioning algorithm based on graph matching is used to obtain the relative position of the platform and its surrounding neighboring nodes by constructing a rotation matrix and a translation matrix between the platform coordinate system and the neighboring node coordinate system.

[0054] Step S2: The unmanned platform voxelizes the lidar point cloud data and uses a voxelized blind spot mining method to discover blind spots based on the density of the point cloud. After discovering the blind spot, the unmanned platform sends the spatial area of ​​the blind spot rather than the perception data to other nodes, allowing multiple platforms to interact with the blind spot. In this embodiment, the blind spot is defined as a continuous voxel area with a density less than a certain threshold.

[0055] Step S3: Based on the discovered blind spots, a local multi-view joint information optimization algorithm is used to discover collaborative objects.

[0056] In this embodiment, specifically, the BEV characteristic map generation method is as follows:

[0057] First, the unmanned platform integrates the lidar point cloud data and depth image data of the platform, uses the voxelization method to extract the BEV features of the radar point cloud, and projects the image pixel features with color semantic information onto the feature plane of the point cloud through voxel grid projection to construct a BEV feature map with semantic information.

[0058] In this embodiment, specifically, a BEV feature map compressed and fused using a deep neural network is used to obtain a compressed feature map, thereby minimizing redundant information while retaining complementary information.

[0059] In this embodiment, inter-platform collaborative localization is performed based on a semantic feature map projected onto the BEV plane from the unmanned platform's semantic visual information. During operation, a platform constructs a feature map centered on its own position. Specifically, based on real-time depth image and point cloud information, after obtaining an extrinsic conversion matrix using radar-camera pre-calibration, the color semantic information contained in the image pixels is projected into the radar's 3D coordinate system, combined with the depth information contained in the image and the reflection intensity of the point cloud from the lidar information. The 3D point cloud information with color semantics is then projected into a 2D feature map from the BEV's perspective. The number and installation positions of the depth cameras determine the range of color semantic information covered. Based on this, a voxelization method is introduced to grid the BEV space, resulting in a BEV grid feature map containing the color semantic information of the depth camera's field of view from the platform's perspective. Because the color information under these conditions is clearly sparse and continuous, the resulting grid feature map can be considered a sparse matrix. Therefore, this sparse feature map is further compressed before collaborative localization to improve communication efficiency.

[0060] Under the condition of having neighboring nodes and the BEV feature map of this platform, we further run the scale-invariant feature conversion matching algorithm to extract feature points. Specifically, the unmanned platform collaborative positioning algorithm based on graph matching includes:

[0061] First, generate the Gaussian kernel function G(x,y,σ) of the multi-scale space from the image scale space;

[0062]

[0063] The feature L(x,y,σ)=G(x,y,σ)*I(x,y) in the multi-scale space is calculated and extracted together with the original BEV feature map grid I(x,y); the spatial extreme points are detected in the scale space as feature points in the BEV feature map, and the accumulated vector in the image gradient direction is extracted based on the feature points. If the domain grid of the feature point contains color semantic information, the RGB color histogram feature in the gradient direction is extracted and concatenated with the accumulated vector to form the representation vector of the feature point grid in BEV;

[0064] Then, the Euclidean distance between the representation vectors of the feature points held by the platform and the neighboring nodes is calculated and the nearest neighbor search algorithm is run. At the same time, the color histogram of the matching results is double-checked, and only feature point pairs with consistent color features are retained to obtain a set of feature point pairs.

[0065] Finally, the RANSAC algorithm is used to search for the optimal rotation matrix. Starting from a randomly initialized set of feature point pairs, point pairs are randomly sampled from the feature point pairs with consistent color features each time, and the rotation matrix H that satisfies the spatial mapping relationship between each point pair is calculated. Then, the reverse reprojection error rate of all global point pairs under this rotation matrix is ​​calculated based on H:

[0066]

[0067] In this embodiment, it is important to note that the iterative optimization search minimizes the error rate in the reverse reprojection between the two feature map planes. In practical applications, the error rate reflects the distance and position correlation between the two platforms. An error rate threshold is used to control whether a reliable relative position transformation relationship is established with the corresponding neighboring nodes, and this relationship is dynamically maintained and updated during operation.

[0068] In this embodiment, it should be noted that in order to achieve practical and efficient collaborative perception, it is necessary to optimize the algorithm's communication and computing capacity while ensuring the maximum amount of shared information, so that the algorithm's real-time operating efficiency can match the scanning frequency of the perception equipment within the system (based on LiDAR, conventional equipment is 10Hz). Existing collaborative perception algorithms are limited by communication capacity and typically adopt a collaborative solution in which a neural network extracts mid-term features from the original complete point cloud and then performs scheduling and communication. However, existing mid-term feature fusion solutions typically extract features from the complete point cloud before performing collaboration. This process itself implicitly assumes that the amount of information contained in the point cloud is uniformly distributed across spatial scales, which does not meet the actual requirements of collaborative perception. For unmanned platforms, their preferred areas of high information value tend to be their blind spots and beyond-line-of-sight areas. This means that local perception information from neighboring nodes that is already within the platform's field of view is highly redundant and has low utilization value.

[0069] To avoid this problem, this embodiment limits the required area in the unmanned platform's collaborative perception process to the blind spot range, which means that the exchange of blind spot information is increased during the communication interaction between the platform and neighboring nodes, and the perception data sharing is further completed based on the coordinate system conversion relationship between the blind spot and the collaborative positioning.

[0070] Before the collaboration begins, the platform's blind spots need to be calculated. Limited by the sheer volume of data in the point cloud, the intuitive approach of searching for blind spots point by point in the three-dimensional perception space is completely unfeasible in practice. Therefore, the blind spot calculation is performed in voxelized space.

[0071] In this embodiment, specifically, the voxelized blind area mining method includes:

[0072] First, the perception information demand space is delineated with this platform as the center. Since this method has no restrictions on the spatial dimension, the commonly used BEV space or a three-dimensional space with higher spatial precision can be delineated. Then, the space is voxelized and evenly divided N times on each axis in the space to obtain a demand space composed of uniform grids. The point cloud density in each grid is then calculated and saved. Finally, the Bresenham algorithm in the field of computer vision is used to calculate the given point (x a ,y a ) and (x b ,y b ) covered by the grid, specifically,

[0073] Whenever the line segment connecting two points passes through a grid (x i ,y i ) edge, calculate:

[0074] d1-d2=(yy i )-(y i +Δy-y)=2y-2y i -c

[0075] Where c is the grid size defined in the algorithm;

[0076] If d1-d2<0, then (x a ,y a ) is covered, otherwise it is considered that (x a ,y a +c) is covered;

[0077] Based on the output of the Bresenham algorithm, the platform's required area in the perception space is further limited to uncovered spatial areas as its blind spots, and the area is broadcast to neighboring nodes to share and request collaborative perception data. The platform also listens to requests from neighboring nodes and trims its own perception data and transmits it back to the neighboring nodes, completing the collaborative perception process based on the original data.

[0078] In this embodiment, it should be noted that through the above steps, the platform can calculate blind spots and sparsely sensed areas. For each blind spot, multiple surrounding nodes can perceive it. If all neighboring nodes transmit data corresponding to this blind spot, it will result in significant communication overhead. To address this issue, for each blind spot, the multi-perspective data of the surrounding neighboring nodes is optimized to select the neighbor with the highest information content and complementarity for blind spot feature exchange. The core concept is: for each blind spot, the neighboring node first sends a keyframe of feature data corresponding to the blind spot to the platform. The platform considers the blind spot data and the blind spot keyframes transmitted by the neighboring nodes, learns the information content metric of the keyframe through the network, and then selects the most appropriate neighbor for feature exchange based on the information content metric. In this case, the keyframe is equivalent to control signaling and has a small data size. Through the exchange of control signaling, it can determine which neighboring node to interact with, and finally transmits the BEV feature map corresponding to this neighbor, thereby reducing communication overhead.

[0079] In collaborative perception among multiple unmanned platforms, adjacent platforms can perceive information about the same area. Therefore, the multi-perspective data exchanged between these platforms is redundant. Given a discovered blind spot, multiple neighboring nodes can also perceive this information. This necessitates the use of a local multi-perspective joint information optimization algorithm to discover collaborative objects. The core idea is to exchange feature keyframes with neighboring nodes and use a learnable feature information metric to measure the redundancy and complementarity of data from different perspectives. This minimizes redundancy and maximizes complementarity in blind spot information transmission between neighboring nodes, preventing all neighbors from transmitting blind spot perception information and addressing the resource consumption associated with redundant information transmission. Unmanned platforms and collaborative objects exchange blind spot perception data to achieve collaborative perception.

[0080] From the perspective of this platform, the communication process and data flow described above are as follows: Figure 2 shown.

[0081] Before the final perception data sharing of the collaborative process begins, the information exchanged between this platform and neighboring nodes is a highly compressed BEV color semantic feature map (i.e., BEV feature map), which is used for collaborative positioning and sends the positioning results to surrounding nodes. Secondly, this platform calculates the blind spot based on the laser point cloud data and exchanges the blind spot information with the neighboring nodes. After receiving the blind spot area from this platform, if the neighboring node has data corresponding to this blind spot, it extracts the corresponding BEV color semantic feature map based on the blind spot, extracts the key frame of the BEV color semantic feature map corresponding to the blind spot, and transmits this key frame to this platform. Finally, this platform combines its own feature map with the blind spot feature key frames extracted by each neighboring node to perform adaptive joint information measurement, and selects the optimal collaborative perception object based on the information measurement value, enters the perception information sharing process, sends a request to the selected collaborative object, and obtains high-information point cloud information.

[0082] This embodiment also proposes a voxel-based unmanned platform collaborative perception device, including: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor implements the steps of the above-mentioned voxel-based unmanned platform collaborative perception method when executing the computer program; preferably, the computer program can be run on a terminal device, such as a personal computer.

[0083] This embodiment also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of a voxel-based unmanned platform collaborative perception method as described above; however, the readable storage medium of the present invention is not limited to this. In this document, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in combination with an instruction execution system, apparatus or device.

[0084] The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0085] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0086] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0087] Example 2

[0088] The second embodiment is a further explanation of the first embodiment, and the same components are not described again here.

[0089] See also Figure 3 In order to effectively extract the key frames in the BEV color semantic feature map, this embodiment uses a transformer structure based on the local spatial cross attention mechanism to extract the blind area feature key frames from the aggregated BEV feature sequence.

[0090] In the sequence of spliced ​​feature maps from multiple parties, for each feature map, firstly, internal blind area feature extraction is performed based on deformable attention, that is, only focusing on the features and coordinates of a set of key sampling points around the blind area feature points in the BEV map, so as to concentrate the attention weight on the blind area key points:

[0091]

[0092] Among them, m represents the mth attention head in the multi-head attention, p q Represents the qth feature point z in all K blind area grid points and their neighboring gridsq The corresponding coordinates in the BEV graph, x is the entire BEV feature graph of the input, A mqk With Δp mqk Corresponding to the learnable attention weight and offset of the kth blind area grid under the mth attention head, W m With W ′ m They are the linear transformation parameters from input features to the value of the attention mechanism and from the value to the output of the attention head.

[0093] Through deformable attention, it is possible to ensure that the key areas in the BEV feature map are focused on through the attention mechanism, that is, the blind area grid p as the reference point q Based on the assumption that the blind spot information of neighboring nodes is consistent with that of their neighboring grids, feature extraction is performed from any spliced ​​BEV blind spot feature maps. Subsequently, to fuse the blind spot information from neighboring nodes with temporal information features and improve the accuracy of feature keyframe extraction, a cross-feature map attention layer is introduced. The blind spot features of consecutive frames extracted from the neighboring node feature maps are used as the query of the attention layer, and the blind spot features of the consecutive frames of this platform are used as the key and value of the attention layer. This implicitly weights and combines the blind spot feature sequences of both parties into a communication-friendly low-dimensional feature.

[0094] The output features of the deformable attention and cross-feature map attention layers aggregate the key blind area information of the time domain and both the platform and the neighboring nodes, which are regarded as the feature key frames of the platform and the neighboring nodes. The feature key frames held by each unmanned platform are exchanged during the communication process, and eventually the platform will hold the feature key frames of itself and all potential collaborative objects. Thereafter, in order to maximize the amount of effective information brought by the collaborative object selection, the key frame extraction network and the collaborative object selection layer are trained by supervising the accuracy of the downstream classification and regression tasks. That is, it can be considered that the supervision of the accuracy of the downstream tasks itself implicitly includes an adaptive measurement of the amount of information in the feature extraction and collaborative object selection processes, and finally the network is optimized to minimize the loss L=L cls +L loc , where L cls With L loc are the losses for downstream classification and anchor box regression tasks, respectively.

[0095] The beneficial effects of the voxel-based unmanned platform collaborative perception method proposed in this embodiment are as follows:

[0096] 1) Unmanned Platform Collaborative Localization Algorithm Based on Graph Matching

[0097] Through the registration and spatial projection of the lidar-depth camera, the perception information containing color semantics is projected to the BEV perspective to form a feature map and further compressed to improve communication utilization. The feature point matching between the two BEV feature maps is achieved by the scale-invariant feature transformation matching algorithm. The optimal coordinate system rotation matrix between the two feature maps is further searched by iteratively optimizing the back reprojection error rate between the two BEV feature maps, realizing a collaborative positioning method between unmanned platforms without the need for external positioning device data.

[0098] 2) Voxel-based blind spot collaborative perception method

[0099] The spatial voxelization method can effectively reduce the computational complexity brought by the amount of point cloud data by orders of magnitude, and complete the calculation of the unmanned platform's perception blind spots and the high information gain area cropping and sharing method of the original perception data in the voxelized space, thereby achieving efficient and fast collaborative perception.

[0100] 3) Local multi-view joint information optimization algorithm

[0101] Through the deformable attention mechanism, we focus on the key points of the unmanned platform blind spot in the BEV feature map, and aggregate the key information of the feature map in the time domain through the self-attention mechanism to realize the BEV blind spot feature key frame extraction and exchange process. Finally, through the adaptive information measurement method, we realize the optimization process of the collaborative object, further improving the utilization rate of the system communication resources.

[0102] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

[0103] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of the description in this section that did not constitute prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A voxel-based unmanned platform collaborative perception method, characterized in that: include: Step S1: The unmanned platform integrates its own lidar point cloud data and depth image data to generate a BEV feature map with semantic information, compresses the BEV feature map to obtain a compressed feature map, and exchanges the compressed feature map between the unmanned platform and its surrounding neighboring nodes. Based on the characteristics of multi-platform interaction, the unmanned platform collaborative positioning algorithm based on graph matching is used to obtain the relative position of the platform and its surrounding neighboring nodes by constructing a rotation matrix and a translation matrix between the platform coordinate system and the neighboring node coordinate system. Step S2: The unmanned platform voxelizes the lidar point cloud data and uses a voxelized blind spot mining method to discover blind spots based on the density of the point cloud. After discovering the blind spot, the unmanned platform sends the spatial area of ​​the blind spot to other nodes, and multiple platforms interact with each other in the blind spot. Step S3: Based on the discovered blind spots, a local multi-view joint information optimization algorithm is used to discover collaborative objects.

2. The voxel-based unmanned platform collaborative perception method according to claim 1, characterized in that: The BEV feature map generation method is as follows: First, the unmanned platform integrates the lidar point cloud data and depth image data of the platform, uses the voxelization method to extract the BEV features of the radar point cloud, and projects the image pixel features with color semantic information onto the feature plane of the point cloud through voxel grid projection to construct a BEV feature map with semantic information.

3. The voxel-based unmanned platform collaborative perception method according to claim 1, characterized in that: Unmanned platform collaborative positioning algorithm based on graph matching, including: First, a Gaussian kernel function G(x,y,σ) in a multi-scale space is generated from the image scale space, and the feature L(x,y,σ)=G(x,y,σ)*I(x,y) in the multi-scale space is calculated and extracted from the original BEV feature map grid I(x,y). Then, spatial extreme points are detected in the scale space as feature points in the BEV feature map, and the accumulated vector in the image gradient direction is extracted based on the feature points. If the domain grid of the feature point contains color semantic information, the RGB color histogram feature in the gradient direction is extracted and concatenated with the accumulated vector to form the representation vector of the feature point grid in BEV. Then, the Euclidean distance between the representation vectors of the feature points held by the platform and the neighboring nodes is calculated and the nearest neighbor search algorithm is run. At the same time, the color histogram of the matching results is double-checked, and only feature point pairs with consistent color features are retained to obtain a set of feature point pairs. Finally, the optimal rotation matrix is ​​searched based on the RANSAC algorithm. Starting from a randomly initialized set of feature point pairs, point pairs are randomly sampled from feature point pairs with consistent color features each time, and the rotation matrix H that satisfies the spatial mapping relationship between each point pair is calculated. Then, the reverse reprojection error rate of all global point pairs under this rotation matrix is ​​calculated based on H.

4. The voxel-based unmanned platform collaborative perception method according to claim 3, characterized in that: The error rate reflects the distance and position correlation between the two platforms. The error rate threshold is used to control whether a reliable relative position conversion relationship is established with the corresponding neighboring nodes, and it is dynamically maintained and updated during operation.

5. The voxel-based unmanned platform collaborative perception method according to claim 1, characterized in that: The required area in the collaborative perception process of the unmanned platform is limited to the blind spot range, and the perception data sharing is completed based on the coordinate system conversion relationship between the blind spot and the collaborative positioning.

6. The voxel-based unmanned platform collaborative perception method according to claim 5, characterized in that: Voxelized blind area mining methods include: First, the perception information demand space is delineated with this platform as the center. Then, the space is voxelized and evenly divided N times on each axis in the space to obtain the demand space composed of uniform grids. Then, the point cloud density in each grid is calculated and saved. Finally, the Bresenham algorithm in the field of computer vision is used to calculate the point density of a given point (x a ,y a ) and (x b ,y b ) covered grid; Based on the output of the Bresenham algorithm, the platform's required area in the perception space is further limited to uncovered spatial areas as its blind spots, and the area is broadcast to neighboring nodes to share and request collaborative perception data. The platform also listens to requests from neighboring nodes and trims its own perception data and transmits it back to the neighboring nodes, completing the collaborative perception process based on the original data.

7. The voxel-based unmanned platform collaborative perception method according to claim 6, characterized in that: Based on the Bresenham algorithm in the field of computer vision, the given point (x a ,y a ) and (x b ,y b ) covered grid, include: Whenever the line segment connecting two points passes through a grid (x i ,y i ) edge, calculate: d1-d2=(y-y i )-(y i +Δy-y)=2y-2y i -c Where c is the grid size defined in the algorithm; If d1-d2<0, then (x a ,y a ) is covered, otherwise it is considered that (x a ,y a +c) is covered.

8. The voxel-based unmanned platform collaborative perception method according to claim 1, characterized in that: Local multi-view joint information optimization algorithm, including: For each blind spot, the neighboring node first sends the key frame of the feature data corresponding to the blind spot to this platform. This platform considers the data corresponding to the blind spot and the blind spot key frame transmitted by the neighboring node, learns the information volume measurement value of the key frame through the network, and then selects the most suitable neighbor for feature interaction based on the information volume measurement value; the key frame is equivalent to control signaling. Through the interaction of control signaling, it can determine which neighboring node to interact with, and finally transmit the BEV feature map corresponding to this neighboring node.

9. A voxel-based unmanned platform collaborative perception device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of a voxel-based unmanned platform collaborative perception method as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a voxel-based unmanned platform collaborative perception method as described in any one of claims 1 to 8 are implemented.

Images