Multi-modal collaborative detection method and system based on heterogeneous graph network
By fusing multimodal features of autonomous vehicles and infrastructure through heterogeneous graph networks, the shortcomings of vehicle-to-vehicle and vehicle-to-infrastructure cooperation are addressed, enabling real-time and accurate target detection and improving the perception capabilities of autonomous vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JIAOTONG UNIV
- Filing Date
- 2022-09-15
- Publication Date
- 2026-05-08
AI Technical Summary
Existing collaborative target detection methods mainly suffer from problems such as difficulties in real-time application of vehicle-to-vehicle collaboration due to unpredictable vehicle dynamic changes, insufficient infrastructure visibility due to the uncertainty of vehicle-road collaboration, and sparsity and semantic lack due to the use of only single point cloud data by most methods.
A multimodal collaborative detection method based on heterogeneous graph networks is adopted. The agent extracts BEV features based on point cloud and image respectively. The heterogeneous graph is used to fuse the multimodal BEV features of multiple agents at the node layer and semantic layer to reduce the transmission bandwidth. The feature fusion is performed through a self-attention mechanism, and finally the target is detected at the central vehicle.
It achieves an effective combination of vehicle-to-vehicle and vehicle-to-infrastructure cooperation, improves the perception performance of autonomous vehicles, overcomes the problems of large parameters and slow inference speed of the Transformer model, and realizes real-time and accurate target detection.
Smart Images
Figure CN115512319B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, and specifically to a multimodal collaborative detection method and system based on heterogeneous graph networks. Background Technology
[0002] Object detection is a crucial research area in the field of autonomous driving vision, where vehicles are referred to as agents in autonomous driving scenarios. Traditional object detection methods rely on single-agent detection using onboard sensors. However, due to target occlusion and the limitations of onboard sensors, single-vehicle detection suffers from blind spots, often resulting in suboptimal detection results. To address the challenges of single-vehicle object detection, cooperative object detection has emerged. Cooperative object detection is a detection method based on multi-agent information fusion, achieved by incorporating a multi-agent collaborative module into the traditional object detection framework. In autonomous driving scenarios, multiple autonomous vehicles and road infrastructure are present, all equipped with sensors such as LiDAR and RGB cameras. A vehicle's blind spot may fall within the detection area of other agents. By transmitting target information observed by other vehicles and infrastructure to the central vehicle, the central vehicle gains a more comprehensive field of view, leading to more accurate object detection. Cooperative object detection can be categorized into vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) cooperation based on the type of participating agents.
[0003] The solutions for vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) collaboration are largely similar. The difference lies in that V2I requires consideration of heterogeneous data sources due to the different sensor types of the intelligent agents. Furthermore, the unpredictable dynamic changes of vehicles in V2V can prevent real-time application, while the fixed location of road infrastructure in V2I means that the field of vision provided by the infrastructure may not be usable by the autonomous vehicle. To compensate for the shortcomings of these two individual tasks in real-world scenarios, combining them is more beneficial for providing autonomous vehicles with a complete field of vision.
[0004] Collaborative target detection methods can be discussed from two perspectives: the collaboration stage and the fusion strategy. The collaboration stage refers to at which stage of target detection the collaborative module is inserted. Based on the different collaboration stages, collaborative target detection methods can be divided into three categories: data-level collaboration, feature-level collaboration, and decision-level collaboration. Data-level collaboration refers to fusing the raw observation data of the agents; feature-level collaboration refers to fusing the data features of the agents; and decision-level collaboration refers to fusing the final detection data of the agents. The fusion strategy refers to the specific fusion calculation process of the collaborative module, which can be divided into simple fusion, feature-based fusion, and graph-based fusion. Simple fusion employs strategies such as averaging, maximizing, and concatenation; feature-based fusion selects the vehicles with the highest correlation; and graph-based fusion constructs the multi-vehicle collaboration process into a graph and fuses the information of multiple agents through graph learning.
[0005] Data-level collaborative transmission of raw data puts excessive bandwidth pressure on the detection results, while decision-level collaborative detection results have lost some target information. In order to maintain a balance between accuracy and bandwidth, this invention selects a target detection method based on feature-level collaborative transmission.
[0006] Depending on the sensor type, perception tasks can be categorized into image-based perception, point cloud-based perception, and multimodal perception. RGB images are characterized by clear semantics and dense pixels, but suffer from limited field of view and target occlusion. LiDAR offers wide coverage but suffers from sparse point clouds. Single-vehicle 3D target detection typically uses multimodal data as model input to compensate for the shortcomings of single-modal perception. However, existing collaborative perception methods are all based on point clouds and have not yet considered multimodal fusion methods. Most existing collaborative perception methods are designed for vehicle-to-vehicle collaborative tasks and employ feature-level fusion to achieve a balance between bandwidth and accuracy. Among these, graph learning and attention mechanisms have become the mainstream methods. Existing graph-based collaborative target detection methods mainly include V2VNet and DiscoNet. V2VNet uses a spatially aware graph neural network (GNN) to fuse multi-vehicle information. V2VNet first compensates for the transmission latency of different vehicles, then uses the GNN to gather the features of surrounding vehicles to the central vehicle, and determines the neighborhood of vehicles based on global location. This method effectively expands the vehicle's field of view, thereby detecting occluded objects. DiscoNet also employs a Graph Attention Network (GAT) to achieve multi-vehicle collaboration. Unlike V2VNet, the edges of DiscoNet's fusion graph are not scalars, but rather a matrix that reflects the contribution of each pixel's features. Furthermore, DiscoNet introduces a teacher-student network: the teacher network performs data-level collaborative object detection, while the student network performs feature-level collaborative object detection. Features from the teacher network are used as supervision for the student network, thus improving the performance of feature-level collaborative object detection. In addition, inspired by the Transformer, Attentive fusion fuses elements at the same location in multiple feature maps through self-attention operations, thereby obtaining new features, such as... Figure 1 As shown, this method dynamically learns the feature relationships between different agents, thereby greatly improving the accuracy of fusion.
[0007] Vehicle-to-infrastructure (V2I) cooperation is another important task in cooperative perception, which can be divided into cooperation between a single vehicle and infrastructure, and cooperation between multiple vehicles and road infrastructure. The simplest approach is to use pre- or post-fusion; currently, only one work, V2X-ViT, has achieved feature-level V2I cooperation. The V2X-ViT framework (such as...) Figure 2The model (shown) includes both vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) tasks. It uses a Transformer module to fuse features, which considers the heterogeneity of agents. However, the Transformer module has many parameters and consumes significant resources. Furthermore, this model only considers point cloud data and does not employ multimodal fusion methods.
[0008] In summary, current mainstream cooperative object detection methods based on fusion graphs suffer from two main problems: First, most methods focus on single vehicle-to-vehicle (V2V) or vehicle-to-infrastructure (V2I) tasks. However, single V2V tasks cannot guarantee real-time application due to the unpredictable dynamic changes of vehicles, while V2I tasks suffer from the fixed location of road infrastructure, meaning the field of view provided by the infrastructure may not be usable by autonomous vehicles. Second, mainstream methods all use a single point cloud as the model input, but point clouds suffer from sparsity and a lack of semantic meaning. Summary of the Invention
[0009] The purpose of this invention is to provide a multimodal cooperative detection method and system based on heterogeneous graph networks to solve at least one of the technical problems existing in the background art.
[0010] To achieve the above objectives, the present invention adopts the following technical solution:
[0011] On one hand, the present invention provides a multimodal cooperative detection method based on heterogeneous graph networks, comprising:
[0012] The agent extracts BEV features based on point clouds and images respectively;
[0013] Multiple intelligent agents transmit the generated multimodal BEV features to the central vehicle;
[0014] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features;
[0015] The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
[0016] Preferably, the agent extracts BEV features based on point clouds and images respectively, including:
[0017] A bird's-eye view is used as the transition feature between the two modalities; assuming there are C categories of objects, n autonomous vehicles, and m roadside infrastructure, for each agent A... i Point cloud data X i (i = 1, 2, 3, ..., n + m), use a point cloud feature extractor to extract 3D point cloud data and convert it into 2D bird's-eye view features. Using an image feature extractor, features of a two-dimensional bird's-eye view are generated by projecting multiple images. Where h, k, and c represent the height, width, and number of channels of the BEV feature, respectively; based on the point cloud range, a grid is divided to generate anchor boxes for single-stage target detection for final region extraction.
[0018] Preferably, multiple agents transmit the generated multimodal BEV features to the central vehicle, including:
[0019] Reduce transmission bandwidth by using feature compression;
[0020] For each agent, a feature h is given. i ∈R k×k×c Compress it to in The compressed features are decoded at the central vehicle end to obtain a feature h with the same size as the original feature. i ∈R k ×k×c .
[0021] Preferably, the heterogeneous graph-based method fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features, including:
[0022] Meta-paths are set for heterogeneous collaborative graphs for different agents; attention mechanism feature fusion is performed at the node level within each meta-path based on the meta-paths; semantic-level attention mechanism feature fusion is performed between different meta-paths; the features output by the heterogeneous collaborative graph are used as the fusion features of the central vehicle.
[0023] Preferably, based on the meta-path, attention mechanism feature fusion is performed at the node level within each meta-path, including:
[0024] The multimodal features of multi-source heterogeneous agents are input into the heterogeneous graph, and attention mechanisms are first fused at the node level within each meta-path.
[0025] For a certain metapath φ i First, design a specific transformation matrix. Transform the features, h' i Features of projection:
[0026]
[0027] After feature transformation, self-attention is used to calculate the weights between node features within the metapath;
[0028] Given a node pair (i,j) within the metapath Φ, node-level attention. It can learn the importance of node j to node i. The formula for calculating self-attention is:
[0029]
[0030] Among them, att node For self-attention operations:
[0031]
[0032] The structural information of the graph is injected into the model to ensure that the weights are applied only to the current edge, and the regularized features are calculated using softmax, where || is the concatenation operation and σ is the activation function.
[0033]
[0034] After obtaining the regularized weights, the nearest neighbor features are aggregated using the following formula:
[0035]
[0036] in, For the current metapath φ i The features obtained through node-level fusion learn information within the current metapath.
[0037] Preferably, semantic-level attention mechanism feature fusion is performed between different meta-paths, including:
[0038] Since different nodes in a heterogeneous graph contain multiple semantics, feature fusion within a meta-path can only consider the semantics within the current path. In order to more comprehensively consider the multiple semantics of a node, multiple meta-paths need to be fused.
[0039] Given the features output by the node layer fusion, the weights between meta-paths can be calculated using the following formula.
[0040]
[0041] Among them, att sem with att node It is also a self-attention operation.
[0042] Secondly, the present invention provides a multimodal cooperative detection system based on heterogeneous graph networks, comprising:
[0043] The extraction module is used by the agent to extract BEV features based on point clouds and images, respectively.
[0044] The transmission module is used by multiple agents to transmit the generated multimodal BEV features to the central vehicle;
[0045] The fusion module is used to fuse multi-agent multimodal BEV features at the node and semantic layers using heterogeneous graph-based methods to obtain new collaborative features;
[0046] The detection module is used by the central vehicle to perform target detection based on new collaborative features and obtain the final detection results.
[0047] Thirdly, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the multimodal cooperative detection method based on heterogeneous graph networks as described above.
[0048] Fourthly, the present invention provides a computer program product, including a computer program that, when run on one or more processors, is used to implement the multimodal cooperative detection method based on heterogeneous graph networks as described above.
[0049] Fifthly, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the multimodal cooperative detection method based on heterogeneous graph networks as described above.
[0050] Terminology Explanation:
[0051] Object detection: Object detection is a fundamental task in computer vision, aiming to identify the location and category of objects in space. Based on the dimension of the predicted object, object detection methods can be divided into 2D object detection and 3D object detection.
[0052] Collaborative Object Detection: Collaborative object detection is a key vision technology in the field of autonomous driving. It refers to assisting a single intelligent agent to complete a more accurate object detection task through information interaction and data fusion among multiple intelligent agents in a scene, thereby alleviating problems such as object occlusion and abnormal sensor capture in autonomous driving scenarios.
[0053] Heterogeneous graph networks: Heterogeneous graphs refer to graphs containing nodes and edges of different types (at least one node or edge has multiple types), commonly seen in knowledge graph scenarios. The simplest way to handle heterogeneous information is to use one-hot encoding of type information and concatenate it to the original node representation. In cooperative detection tasks for autonomous driving, when data comes from different types of intelligent agents, such as autonomous vehicles and road infrastructure (i.e., vehicle-to-infrastructure communication), the graph composed of multiple intelligent agents is a heterogeneous graph.
[0054] Multimodal sensing: Multimodal sensing refers to fusing features obtained from data from different types of sensors. Since different sensors have different characteristics and defects, fusing features obtained from multimodal data helps to overcome the shortcomings of single-modal sensing.
[0055] The beneficial effects of this invention are as follows: applying a multimodal fusion method to cooperative detection combines the two to provide a complete field of vision for autonomous vehicles, making up for the shortcomings of vehicle-to-vehicle and vehicle-to-infrastructure cooperation as separate tasks; addressing the problem of multi-source heterogeneity, heterogeneous graph networks are used to fuse the multimodal features of agents, thereby effectively improving cooperative perception performance; the simple and efficient heterogeneous modules adopted overcome the shortcomings of Transformer model, such as large parameters and slow inference speed, effectively improving the model inference speed and realizing real-time cooperative detection.
[0056] The advantages of additional aspects of the invention will be set forth more clearly in the following description or will be learned by practice of the invention. Attached Figure Description
[0057] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0058] Figure 1 This is a structural diagram of the existing Attentive Fusion framework.
[0059] Figure 2 This is a diagram of the existing V2X-ViT framework architecture.
[0060] Figure 3 This is a flowchart of the multimodal collaborative detection method based on heterogeneous graph networks according to an embodiment of the present invention.
[0061] Figure 4 This is a diagram of the multimodal collaborative detection framework based on heterogeneous graph networks as described in an embodiment of the present invention.
[0062] Figure 5 This is a flowchart illustrating the multimodal feature fusion process of heterogeneous intelligent agents according to an embodiment of the present invention. Detailed Implementation
[0063] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0065] It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as here.
[0066] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.
[0067] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0068] To facilitate understanding of the present invention, the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.
[0069] Those skilled in the art should understand that the accompanying drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily essential for implementing the present invention.
[0070] Example 1
[0071] First, in this embodiment 1, a multimodal cooperative detection system based on heterogeneous graph networks is provided, including:
[0072] The extraction module is used by the agent to extract BEV features based on point clouds and images, respectively.
[0073] The transmission module is used by multiple agents to transmit the generated multimodal BEV features to the central vehicle;
[0074] The fusion module is used to fuse multi-agent multimodal BEV features at the node and semantic layers using heterogeneous graph-based methods to obtain new collaborative features;
[0075] The detection module is used by the central vehicle to perform target detection based on new collaborative features and obtain the final detection results.
[0076] In this embodiment 1, based on the above system, a multimodal cooperative detection method based on heterogeneous graph networks is implemented, including:
[0077] The agent extracts BEV features based on point clouds and images respectively;
[0078] Multiple intelligent agents transmit the generated multimodal BEV features to the central vehicle;
[0079] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features;
[0080] The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
[0081] The agent extracts BEV features based on point clouds and images respectively, including:
[0082] A bird's-eye view is used as the transition feature between the two modalities; assuming there are C categories of objects, n autonomous vehicles, and m roadside infrastructure, for each agent A... i Point cloud data X i (i = 1, 2, 3, ..., n + m), use a point cloud feature extractor to extract 3D point cloud data and convert it into 2D bird's-eye view features. Using an image feature extractor, features of a two-dimensional bird's-eye view are generated by projecting multiple images. Where h, k, and c represent the height, width, and number of channels of the BEV feature, respectively; based on the point cloud range, a grid is divided to generate anchor boxes for single-stage target detection for final region extraction.
[0083] Multiple agents transmit the generated multimodal BEV features to the central vehicle, including:
[0084] Reduce transmission bandwidth by using feature compression;
[0085] For each agent, a feature h is given. i ∈R k×k×c Compress it to in The compressed features are decoded at the central vehicle end to obtain a feature h with the same size as the original feature. i ∈R k ×k×c .
[0086] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features, including:
[0087] Meta-paths are set for heterogeneous collaborative graphs for different agents; attention mechanism feature fusion is performed at the node level within each meta-path based on the meta-paths; semantic-level attention mechanism feature fusion is performed between different meta-paths; the features output by the heterogeneous collaborative graph are used as the fusion features of the central vehicle.
[0088] Based on the meta-path, attention mechanism feature fusion is performed at the node level within each meta-path, including:
[0089] The multimodal features of multi-source heterogeneous agents are input into the heterogeneous graph, and attention mechanisms are first fused at the node level within each meta-path.
[0090] For a certain metapath φ i First, design a specific transformation matrix. Transform the features, h' i Features of projection:
[0091]
[0092] After feature transformation, self-attention is used to calculate the weights between node features within the metapath;
[0093] Given a node pair (i,j) within the metapath Φ, node-level attention. It can learn the importance of node j to node i. The formula for calculating self-attention is:
[0094]
[0095] Among them, att node For self-attention operations:
[0096]
[0097] The structural information of the graph is injected into the model to ensure that the weights are applied only to the current edge, and the regularized features are calculated using softmax, where || is the concatenation operation and σ is the activation function.
[0098]
[0099] After obtaining the regularized weights, the nearest neighbor features are aggregated using the following formula:
[0100]
[0101] in, For the current metapath φ i The features obtained through node-level fusion learn information within the current metapath.
[0102] Semantic attention mechanism feature fusion is performed between different meta-paths, including:
[0103] Since different nodes in a heterogeneous graph contain multiple semantics, feature fusion within a meta-path can only consider the semantics within the current path. In order to more comprehensively consider the multiple semantics of a node, multiple meta-paths need to be fused.
[0104] Given the features output by the node layer fusion, the weights between meta-paths can be calculated using the following formula.
[0105]
[0106] Among them, att sem with att node It is also a self-attention operation.
[0107] Meta-paths for setting up heterogeneous collaboration graphs for different agents include:
[0108] Due to the unique characteristics of sensors in autonomous vehicles and road infrastructure, and the different features generated by different modalities, it is necessary to define the types of nodes and edges in the heterogeneous graph to represent these heterogeneous features. Using V to represent vehicles, I to represent infrastructure, P to represent point clouds, and C to represent images, there are four types of nodes in the heterogeneous graph:
[0109] c∈{VP,VC,IP,IC}
[0110] Accordingly, the edge types of heterogeneous graphs include:
[0111] e∈{VP-VP,VP-VC,VP-IP,VP-IC,
[0112] VC-VP, VC-VC, VC-IP, VC-IC
[0113] IP-VP, IP-VC, IP-IP, IP-IC
[0114] IC-VP, IC-VC, IC-IP, IC-IC
[0115] The directed edges defined above are meta-paths of heterogeneous graphs. Each meta-path describes a feature pair, such as VC-IP, which represents the fusion of image features of vehicles and point cloud features of infrastructure.
[0116] The features output by the heterogeneous collaboration graph, as the fused features of the central vehicle, include:
[0117] Through node fusion within metapaths and semantic fusion between metapaths, the features of heterogeneous agents and multimodal interactions are fully learned. Finally, an averaging method is used to... The final collaborative feature of the central vehicle is integrated and output.
[0118] Example 2
[0119] In this embodiment 2, a multimodal cooperative detection method based on heterogeneous graph networks is proposed. Based on features extracted from 3D point clouds and RGB images, a heterogeneous cooperative graph is constructed to adaptively fuse the perception features of autonomous vehicles and infrastructure, thereby assisting the central vehicle in achieving accurate target detection. This embodiment 2 combines vehicle-to-vehicle and vehicle-to-infrastructure (V2I) tasks, proposing a multimodal cooperative perception algorithm based on heterogeneous graph networks. Furthermore, existing cooperative target detection models are all based on single-modal perception using point clouds. To compensate for the semantic gaps and data sparsity issues of point clouds, both point cloud and image features of the agent are extracted simultaneously, thereby achieving multimodal cooperative perception.
[0120] This embodiment 2 mainly describes a multimodal cooperative detection method based on heterogeneous graph networks. It includes four steps: First, agents extract BEV features based on point clouds and images respectively; second, multiple agents transmit the generated multimodal BEV features to the central vehicle; third, based on a heterogeneous graph method, the features of multiple agents are fused at the node and semantic layers, giving the central vehicle new cooperative features; fourth, target detection is performed based on the cooperative features to obtain the final detection result. This invention effectively combines vehicle-to-vehicle and vehicle-to-infrastructure cooperative tasks by constructing a heterogeneous cooperative graph, alleviating the problems associated with single tasks. Figure 3 As shown, the specific steps include the following;
[0121] S10, the agent extracts BEV features based on point cloud and image respectively.
[0122] This invention addresses the task of multimodal single-stage 3D object detection. Due to the different dimensions of point clouds and images, mutual projection transformation faces the problem of information loss. Autonomous driving 3D object detection generally only involves the same plane, ignoring height targets. A bird's-eye view (BEV) can clearly represent the position of objects in an autonomous driving scene, and objects do not obscure each other. Therefore, this invention adopts a bird's-eye view (BEV) as the transformation feature between the two modalities.
[0123] In an autonomous driving scenario, assuming there are C categories of objects, n autonomous vehicles, and m roadside infrastructure, for each agent A... iPoint cloud data X i (i = 1, 2, 3, ..., n + m), using a point cloud feature extractor, the 3D point cloud data is converted into 2D bird's-eye view features. Similarly, using an image feature extractor, the features of a two-dimensional bird's-eye view generated from multiple images are obtained through projection. Where h, k, and c represent the height, width, and number of channels of the BEV feature, respectively.
[0124] In this embodiment, PointPillar and ImVoxelNet are used as feature extractors for point clouds and images, respectively. Next, based on the point cloud extent, a grid is divided to generate anchor boxes for single-stage object detection, which are then used for final region extraction.
[0125] S20, multiple agents transmit the generated multimodal BEV features to the central vehicle.
[0126] Feature-level cooperative object detection requires the transmission of agent features. Because there are many agents in a cooperative perception scenario, the number of features that need to be transmitted is also large, resulting in significant bandwidth consumption. For example... Figure 4 As shown, in this embodiment 2, feature compression is used to reduce the transmission bandwidth.
[0127] For each agent, a feature h is given. i ∈R k×k×c In this embodiment, an encoder is used to compress the features to... in The compressed features are decoded at the central vehicle end to obtain a feature h with the same size as the original feature. i ∈R k×k×c The feature compression module uses a series of two-dimensional convolutions as the encoder and a series of two-dimensional deconvolutions as the decoder.
[0128] S30, a heterogeneous graph-based method, fuses multi-modal features of multiple agents at the node and semantic layers.
[0129] This invention uses heterogeneous graph networks to fuse the multimodal features of multi-source heterogeneous agents, and the process is as follows: Figure 5 As shown below. The steps for this module will be described in detail below.
[0130] S31 sets up meta-paths for heterogeneous collaboration graphs for different agents.
[0131] Due to the unique characteristics of sensors in autonomous vehicles and road infrastructure, and the different features generated by different modalities, it is necessary to define the types of nodes and edges in the heterogeneous graph to represent these heterogeneous features. Using V to represent vehicles, I to represent infrastructure, P to represent point clouds, and C to represent images, there are four types of nodes in the heterogeneous graph:
[0132] c∈{VP,VC,IP,IC}
[0133] Accordingly, the edge types of heterogeneous graphs include:
[0134] e∈{VP-VP,VP-VC,VP-IP,VP-IC,VC-VP,VC-VC,VC-IP,VC-IC,IP-VP,IP-VC,IP-IP,IP-IC,IC-VP,IC-VC,IC-IP,IC-IC}
[0135] The directed edges defined above are meta-paths of heterogeneous graphs. Each meta-path describes a feature pair, such as VC-IP, which represents the fusion of image features of vehicles and point cloud features of infrastructure.
[0136] S32, based on the meta-path, performs node-level attention mechanism feature fusion within each meta-path.
[0137] The multimodal features of multi-source heterogeneous agents are input into the heterogeneous graph, and attention mechanisms are first fused at the node level within each meta-path.
[0138] For a certain metapath φ i First, design a specific transformation matrix. The features are transformed as shown in the following formula, h' i Features of projection.
[0139]
[0140] After feature transformation, self-attention is used to calculate the weights between node features within the metapath. Given a node pair (i,j) within the metapath Φ, node-level attention is applied. The importance of node j to node i can be learned. The calculation process of self-attention is shown in the following formula.
[0141]
[0142] Among them att node The self-attention operation is shown in the following equation.
[0143]
[0144] Next, the structural information of the graph is injected into the model to ensure that the weights are only applied to the current edge, and the regularized features are calculated using softmax, as shown in the following formula, where || is the concatenation operation and σ is the activation function.
[0145]
[0146] Once the regularized weights are obtained, the nearest neighbor features can be aggregated using the following formula.
[0147]
[0148] For the current metapath φ i The features obtained through node-level fusion learn information within the current metapath.
[0149] S33 performs semantic-level attention mechanism feature fusion between different meta-paths.
[0150] Since different nodes in a heterogeneous graph contain multiple semantics, feature fusion within a metapath can only consider the semantics within the current path. In order to more comprehensively consider the multiple semantics of nodes, we also need to fuse multiple metapaths.
[0151] Given the features output by the node layer fusion, the weights between meta-paths can be calculated using the following formula, where att sem with att node It is also a self-attention operation.
[0152]
[0153] S34, the features output by the heterogeneous collaboration graph are used as the fusion features of the central vehicle.
[0154] Through node fusion within metapaths and semantic fusion between metapaths, the features of heterogeneous agents and multimodal interactions are fully learned. Finally, an averaging method is used to... The final collaborative feature of the central vehicle is integrated and output.
[0155] S40, the central vehicle performs target detection based on collaborative features.
[0156] Thus, the central vehicle obtains multi-modal fusion features of multiple agents. In this embodiment 2, the PointPillar framework is continued, and features of multiple sizes are fused using feature pyramids. These features are then input into the 3D detection head to classify and regress each candidate box. After threshold filtering, more accurate detection results can be obtained.
[0157] It is important to note that in collaborative target detection scenarios, each vehicle can act as a central vehicle or as a collaborative vehicle with other vehicles. Therefore, the entire collaborative detection process is parallel. Employing a multimodal single-stage fusion detection model significantly outperforms the single-modal single-stage detection model in terms of detection accuracy. Through feature fusion of heterogeneous collaborative graphs, the perception field of a single vehicle is greatly expanded, enriching the perception information and thus improving collaborative perception performance.
[0158] In summary, the method described in Example 2 is the first attempt to apply multimodal fusion to cooperative detection. To compensate for the shortcomings of vehicle-to-vehicle and vehicle-to-infrastructure (V2I) cooperation as separate tasks, it combines the two to provide autonomous vehicles with a complete field of vision. To address the issue of multi-source heterogeneity, a heterogeneous graph network is designed to fuse the multimodal features of the agent, thereby effectively improving cooperative perception performance. Furthermore, the simple and efficient heterogeneous modules used overcome the drawbacks of the Transformer model, such as its large number of parameters and slow inference speed, thus effectively improving the model's inference speed and enabling real-time cooperative detection.
[0159] Example 3
[0160] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium for storing computer instructions. When executed by a processor, the computer instructions implement a multimodal cooperative detection method based on heterogeneous graph networks. The method includes:
[0161] The agent extracts BEV features based on point clouds and images respectively;
[0162] Multiple intelligent agents transmit the generated multimodal BEV features to the central vehicle;
[0163] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features;
[0164] The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
[0165] Example 4
[0166] Embodiment 4 of the present invention provides a computer program (product), including a computer program that, when run on one or more processors, is used to implement a multimodal cooperative detection method based on heterogeneous graph networks. The method includes:
[0167] The agent extracts BEV features based on point clouds and images respectively;
[0168] Multiple intelligent agents transmit the generated multimodal BEV features to the central vehicle;
[0169] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features;
[0170] The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
[0171] Example 5
[0172] Embodiment 5 of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein, the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing a multimodal cooperative detection method based on heterogeneous graph networks, the method including:
[0173] The agent extracts BEV features based on point clouds and images respectively;
[0174] Multiple intelligent agents transmit the generated multimodal BEV features to the central vehicle;
[0175] The heterogeneous graph-based approach fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features;
[0176] The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
[0177] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0179] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solutions disclosed in the present invention, various modifications or variations that can be made by those skilled in the art without creative effort should be included within the scope of protection of the present invention.
Claims
1. A multimodal cooperative detection method based on heterogeneous graph networks, characterized in that, include: The agent extracts BEV features based on point clouds and images respectively; Multiple intelligent agents transmit BEV features extracted from point clouds and images to the central vehicle; The heterogeneous graph-based method fuses multi-agent multimodal BEV features at the node and semantic layers to obtain new collaborative features. This includes: setting meta-paths for different agents in the heterogeneous collaborative graph; performing node-level attention mechanism feature fusion within each meta-path; performing semantic-level attention mechanism feature fusion between different meta-paths; and using the features output by the heterogeneous collaborative graph as the fused features of the central vehicle. Specifically, the node-level attention mechanism feature fusion within each meta-path includes: inputting multimodal features from multiple heterogeneous agents into the heterogeneous graph; firstly, performing node-level attention mechanism fusion within each meta-path; and then, for a given meta-path... First, design a specific transformation matrix. Transform the features Features of projection: After feature transformation, self-attention is used to calculate the weights between features of nodes within the metapath; given a metapath Inner node pairs Attention at the node level Learning Nodes For nodes The importance of self-attention is explained by the following formula: ; in, For self-attention operations: ； The structural information of the graph is injected into the model to ensure that the weights are applied only to the current edge, and the regularized features are calculated using softmax, where || is the concatenation operation. It is an activation function: ； After obtaining the regularized weights, the nearest neighbor features are aggregated using the following formula: ； in, For the current metapath The features obtained through node-level fusion learn information within the current meta-path; The semantic-level attention mechanism for feature fusion between different meta-paths includes: since different nodes in a heterogeneous graph contain multiple semantics, feature fusion within a meta-path can only consider the semantics within the current path. To more comprehensively consider the multiple semantics of nodes, multiple meta-paths need to be fused; given the features output by the node-level fusion, the weights between meta-paths are calculated using the following formula: ； in, and It's the same self-attention operation; The central vehicle performs target detection based on the new collaborative features to obtain the final detection result.
2. The multimodal cooperative detection method based on heterogeneous graph networks according to claim 1, characterized in that, The agent extracts BEV features based on point clouds and images respectively, including: A bird's-eye view is used as the transition feature between the two modes; assuming there are a total of Given n categories of objects, n autonomous vehicles, and m roadside infrastructure, for each intelligent agent... Point cloud data (i=1,2,3,…,n+m), use a point cloud feature extractor to extract 3D point cloud data and convert it into 2D bird's-eye view features. Using an image feature extractor, features of a two-dimensional bird's-eye view are generated by projecting multiple images. Where h, k, and c represent the height, width, and number of channels of the BEV feature, respectively; based on the point cloud range, a grid is divided to generate anchor boxes for single-stage target detection for final region extraction.
3. The multimodal cooperative detection method based on heterogeneous graph networks according to claim 2, characterized in that, Multiple agents transmit the generated multimodal BEV features to the central vehicle, including: Reduce transmission bandwidth by using feature compression; For each agent, a feature is given. Compress it to ,in The compressed features are decoded at the central vehicle end to obtain features with the same size as the original features. .
4. A multimodal cooperative detection system based on heterogeneous graph networks, based on the method described in any one of claims 1-3, characterized in that, include: The extraction module is used by the agent to extract BEV features based on point clouds and images, respectively. The transmission module is used by multiple intelligent agents to transmit BEV features extracted from point clouds and images to the central vehicle. The fusion module is used to fuse multi-agent multimodal BEV features at the node and semantic layers using heterogeneous graph-based methods to obtain new collaborative features; The detection module is used by the central vehicle to perform target detection based on new collaborative features and obtain the final detection results.
5. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the multimodal cooperative detection method based on heterogeneous graph networks as described in any one of claims 1-3.
6. A computer program product, characterized in that, The method includes a computer program that, when run on one or more processors, is used to implement the multimodal cooperative detection method based on heterogeneous graph networks as described in any one of claims 1-3.
7. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions that implement the multimodal cooperative detection method based on heterogeneous graph networks as described in any one of claims 1-3.