Multi-vehicle cooperative sensing method based on context sensing

Abstract

The invention provides a multi-vehicle cooperative sensing method based on context sensing, and the method comprises the steps: obtaining the sensing data of a vehicle and a cooperative vehicle, including the position and speed of the vehicle, and the context information of historical frame data; performing feature extraction on the acquired data by using a backbone network to obtain own vehicle features and cooperative features, and sharing feature information through compression and sparsification; the motion state information and the sparse feature mapping graph of the vehicle are utilized, and a graph attention mechanism is combined to dynamically select a cooperative vehicle capable of providing significant perception performance improvement in the current scene; fusing and aligning the sensing features of the historical frame and the current frame from the local of the vehicle by adopting a feature fusion algorithm based on Transform to form a time-space fusion feature of the vehicle; and carrying out multi-scale attention fusion on the self-vehicle space-time fusion feature and the cooperative vehicle feature, and carrying out a target detection task based on the fused global perception feature.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent transportation systems, and particularly relates to a multi-vehicle cooperative perception method based on context perception. BACKGROUND

[0002] With the rapid development of intelligent transportation systems (ITS), autonomous driving technology has gradually become an important research direction in the future transportation field. In order to achieve efficient and safe autonomous driving, the perception ability of vehicles is crucial. However, the perception ability of a single autonomous vehicle has certain limitations, especially in complex urban environments or adverse weather conditions, the field of view and detection range of its sensors may be severely affected. This limitation may lead to incomplete or inaccurate perception information, which adversely affects the safety and reliability of driving decisions. In order to solve the deficiency of single vehicle perception ability, vehicle-to-everything (V2X) technology emerges as the times require, through cooperative perception between vehicles, vehicles can share each other's perception data, thereby significantly enhancing their perception ability. Cooperative perception plays a crucial role in intelligent transportation systems. By fusing and sharing the perception information of multiple vehicles, the deficiency of single vehicle perception range can be effectively compensated, and the globality and accuracy of perception can be enhanced. This not only helps to reduce the perception blind area, but also improves the response ability to dynamic environment, thereby improving the driving safety and traffic efficiency of the entire vehicle fleet or road network. Especially in dealing with complex road environments and unexpected situations, cooperative perception can provide more rich and real-time environmental information, support more intelligent and robust autonomous driving decisions.

[0003] However, traditional cooperative perception methods usually rely on the sharing of full-quantity feature information, which, although can improve the perception effect, also brings huge communication overhead and processing delay. Therefore, Who2com proposes a multi-stage handshake communication mechanism, which allows neural networks to learn to compress the required relevant information at each stage. When2com constructs a communication group and decides when to communicate, while Where2comm proposes a spatial key location selection strategy.

[0004] Although these methods solve the problem to some extent, current methods usually focus on the spatial relationship between agents, ignoring valuable semantic information in the temporal context. Insufficient representation of moving objects in the current frame can limit the perception ability of the target vehicle. In addition, it is also crucial to select appropriate collaborators, as inappropriate collaborators may introduce interference, thus negatively affecting the perception performance. Due to data redundancy, it is challenging to identify collaborators that can truly enhance the perception performance. Recent studies, such as CoCa3D and Scope, rely solely on confidence maps to determine with whom to collaborate, however, errors or interference in the confidence map can lead to incorrect collaboration decisions. The importance of information provided by different collaborators varies in enhancing the perception performance. SUMMARY

[0005] To solve the above problems of the prior art, the present application proposes a collaborative perception method that can integrate spatial and temporal context information, has efficient collaborator selection and feature fusion capabilities. By using the motion state information and sparse mapping feature map of the vehicle, the collaboration partner selection is performed through the graph attention network, and an innovative feature fusion algorithm is adopted to improve the perception accuracy, finally realizing better perception performance and lower communication cost, thereby promoting the development of intelligent transportation systems.

[0006] The present application aims to provide a multi-vehicle collaborative perception method based on context perception, in order to reduce communication overhead while improving perception performance; break through the limitations of single-vehicle perception in the Internet of Vehicles scenario, obtain a more comprehensive understanding of the environment, and thus improve the safety of autonomous driving. It relates to a collaborative perception method in V2X (Vehicle to Everything, Internet of Vehicles), proposes a collaborative perception framework based on context perception, for improving the perception ability of autonomous vehicles. The present application realizes efficient information sharing between collaborative vehicles by integrating context semantic information in spatial and temporal dimensions. First, a collaborator selection method based on vehicle motion state information and graph attention mechanism is proposed, which can more effectively select collaborative vehicles that significantly improve the perception performance. Second, a feature fusion algorithm based on Transformer is proposed for fusing and aligning the features of historical frames and current frames. The present application not only achieves significant improvement in perception accuracy, but also shows robustness to feature compression and positioning errors.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] A multi-vehicle collaborative perception method based on context perception:

[0009] Obtain perception data of the ego vehicle and collaborative vehicles, including the position, speed, and context information of historical frame data of the vehicles;

[0010] The obtained data is subjected to feature extraction by using a backbone network to obtain ego vehicle features and collaborative features, and shared feature information is compressed and sparsified;

[0011] By using the motion state information of the vehicle and the sparse feature map, in combination with a graph attention mechanism, a collaborative vehicle that can provide significant perception performance improvement in the current scene is dynamically selected;

[0012] A feature fusion algorithm based on a Transformer is adopted to fuse and align the perception features from the historical frames and the current frame of the vehicle locally, to form ego vehicle spatio-temporal fusion features;

[0013] The ego vehicle spatio-temporal fusion features are subjected to multi-scale attention fusion with the collaborative vehicle features, and based on the fused global perception features, a target detection task is performed.

[0014] Further, the motion state information of all vehicles around the ith vehicle includes:

[0015] p i ={x,y,z,ψ,v,d},

[0016] which includes: vehicle center position (x, y, z), distance d from the ego vehicle, yaw angle ψ, and relative speed v.

[0017] Further, when the obtained data is subjected to feature extraction by using the backbone network:

[0018] A bird's eye view perspective is adopted for feature representation, and all vehicles project the perception information into the same global coordinate system; for the ith vehicle, the given input is X i , the feature map is mapped to F i =Φ enc (X i ), wherein Φ enc () is an encoder; the feature extraction uses a PointPillars backbone network;

[0019] The extracted feature map is input into a spatial confidence generator to generate a spatial confidence map according to the feature map of each vehicle; after the given feature map F i , the corresponding spatial confidence map is generated as:

[0020] M i =Φ generator (F i )∈{0,1},

[0021] wherein Φ generator () is a detection decoder.

[0022] Further, the shared feature information is compressed and sparsified using an encoder-decoder structure; the encoder consists of a series of 2D convolution and max-pooling layers, and the shared message is broadcast to all communicating vehicles; at the ego vehicle end, the decoder with multiple de-convolution layers is used to recover the compressed information and pass it to other modules for further processing.

[0023] Further, the motion state information and sparse feature map of the vehicle are used to dynamically select the specific cooperative vehicle that can provide significant performance improvement in the current scene by combining the graph attention mechanism, which is:

[0024] Each vehicle acts as a node; the feature of each node is composed of the sparse feature map s i ∈R N×F and the motion state matrix p i ∈R N×F of the vehicle, where N represents the number of nodes, and F represents the number of features of each node; then a multi-layer perception is used to project the motion state matrix to the same feature dimension as the sparse feature map, and the mapping process is represented as:

[0025]

[0026] where W1 and W2 are weight matrices, and b1 and b2 are corresponding bias terms; the sparse feature map is subjected to a pooling operation to average the features; the projected motion state matrix and the pooled sparse feature map are connected along the channel dimension to generate a vectorized node feature,

[0027]

[0028] which is then input to the graph attention network together with the adjacency matrix; the input of the graph attention network is a set of node features Then the shared self-attention mechanism a is performed on the nodes; the attention coefficient is:

[0029]

[0030] where W is a learnable parameter matrix, and represents the importance of the connection between nodes;

[0031] Next, a nonlinear function is applied to map the attention coefficient to a nonlinear space; and using the hyperbolic tangent activation function, each weight value is mapped to the interval [-1, 1], thereby generating the attention weight of the graph attention network output which represents the importance of the neighbor nodes to the current node;

[0032] The communication range of the ith vehicle is defined as R i ={j}, j∈N-1; according to the attention weight Selecting beneficial cooperative vehicles, deleting vehicles that do not actively contribute to improving perception performance:

[0033]

[0034] The features shared by the cooperative vehicles are represented as:

[0035]

[0036] Further, the Transformer-based feature fusion algorithm is used to fuse and align the perception features from the local history frames and the current frame of the vehicle, forming the self-vehicle spatio-temporal fusion features, which are specifically:

[0037] Using the temporal context information, including data features from history frames and current frame features, using the max-pooling operation to retain the most significant features in the current and history frames, then mapping and concatenating the retained features, and applying attention operations through the convolutional network to generate semantic weights of the temporal context information; then assign the semantic weight values to the feature map of the history frame to obtain useful history features after screening:

[0038]

[0039] where σ represents the sigmoid activation function, represents the convolutional layer for fusion, and represents the concatenation operation;

[0040] Subsequently, pixel-by-pixel attention weighting is used to filter and select information to highlight important parts of the history features:

[0041]

[0042] Then the spatio-temporal features of the ego vehicle are fused through the Transformer algorithm, given the history frame features of the ego vehicle and the current frame features to obtain the hidden layer spatio-temporal representation of T frames and N feature pixels, F h ist ∈R T×N×D and F curr ∈R T×N×D , where D is the dimension number of the hidden layer; the query matrix, key matrix and value matrix are obtained through the temporal Transformer layer:

[0043] Q t ,K t ,V t =Temporal_Self_Attention(F h ist ,F curr ;Θ),

[0044] where Θ is a learnable parameter; then, the self-attention score is calculated as follows:

[0045]

[0046] such that A t captures the temporal relationship between different spatial nodes; the output of the temporal Transformer is:

[0047] O t = A t V t

[0048] The representation of the spatial Transformer layer is as follows:

[0049] O s = Spatial_Self_Attention(O t )

[0050] The output after the temporal and spatial Transformer layers is fed into a projection layer composed of layer normalization, residual connection and multi-head mechanism to obtain the self-vehicle spatio-temporal feature representation:

[0051]

[0052] where H represents the desired spatio-temporal feature, and d represents the output feature dimension number.

[0053] Further, the multi-scale attention fusion of the self-vehicle spatio-temporal fusion feature and the cooperative vehicle feature is based on the fused global perception feature, and is used for a target detection task, specifically: the attention mechanism weighted feature is spliced to ensure that the fused feature encapsulates comprehensive information; the fused feature is fed into a prediction head to generate a bounding box suggestion and a related confidence score to realize the target detection task.

[0054] And a multi-vehicle cooperative perception system based on context perception, comprising:

[0055] An acquisition module is configured to acquire perception data of a self-vehicle and cooperative vehicles, including position, speed, context information of historical frame data of the vehicles;

[0056] A feature extraction module is configured to extract features of the acquired data by using a backbone network to obtain self-vehicle features and cooperative features, and share feature information by compression and sparsification;

[0057] A dynamic selection module is configured to dynamically select cooperative vehicles that can provide significant perception performance improvement in the current scene by using motion state information of the vehicles and sparse feature mapping, in combination with a graph attention mechanism;

[0058] The vehicle spatiotemporal fusion feature module is used to fuse and align the perception features from historical frames and the current frame from the vehicle's local area using a Transformer-based feature fusion algorithm to form vehicle spatiotemporal fusion features.

[0059] The multi-scale attention fusion module is used to perform multi-scale attention fusion of the spatiotemporal fusion features of the autonomous vehicle and the features of the cooperative vehicles. Based on the fused global perception features, it is used for target detection tasks.

[0060] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a context-aware multi-vehicle cooperative perception method as described above.

[0061] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a context-aware multi-vehicle cooperative perception method as described above.

[0062] Compared to existing technologies, this invention and its preferred solutions, by introducing a context-aware mechanism, comprehensively consider semantic information in both spatial and temporal dimensions, thereby capturing dynamic changes of objects in a scene to a greater extent and enhancing perception accuracy and reliability. Furthermore, by introducing a collaborator selection mechanism, this invention achieves efficient perception with lower communication overhead. Through the fusion and alignment of temporal features, this invention can better address perception challenges in complex scenes and enhance the system's robustness, especially in the case of feature compression or localization errors. Attached Figure Description

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0064] Figure 1 This is a structural diagram of a context-aware multi-vehicle collaborative perception method according to an embodiment of the present invention.

[0065] Figure 2 This is a comparison chart of the perception performance of two open datasets in an embodiment of the present invention.

[0066] Figure 3 This is a comparison chart of the perceived performance of the present invention on the open dataset OPV2V for different compression ratios.

[0067] Figure 4 This is a comparison chart of the perceived performance of the present invention on the open dataset V2XSet for different compression ratios.

[0068] Figure 5 This is a comparison chart of the perception performance of the embodiments of the present invention under different datasets for different positioning errors. DETAILED DESCRIPTION

[0069] In order to make the features and advantages of the present application more apparent, specific examples are described in detail below, as follows:

[0070] It should be noted that the following detailed description is illustrative only and is intended to provide further description in addition to that of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0071] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of the features, steps, operations, devices, components and / or combinations thereof.

[0072] As shown in Figure 1 The present application provides a context-aware multi-vehicle cooperative perception method, comprising the following steps:

[0073] Step S1, acquiring perception data of the ego vehicle and the cooperative vehicle through the vehicle-mounted sensor and the communication device, wherein the context information such as the position, speed, historical frame data, etc. of the vehicle is acquired from the vehicle-mounted sensor;

[0074] Step S2, using the backbone network to extract features from the acquired data to obtain ego features and cooperative features, and reducing the communication overhead by compressing and sparsifying the shared feature information;

[0075] Step S3, using the motion state information of the vehicle and the sparse feature mapping graph, and combining the graph attention mechanism to dynamically select the cooperative vehicle that can provide significant perception performance improvement in the current scene;

[0076] Step S4, using a feature fusion algorithm based on Transformer to fuse and align the perception features from the historical frames and the current frames of the vehicle locally to form ego spatio-temporal fusion features;

[0077] Step S5, finally performing multi-scale attention fusion on the ego spatio-temporal fusion features and the cooperative vehicle features, and performing a target detection task based on the fused global perception features.

[0078] As a preferred scheme of the present embodiment, the specific implementation process of step S1 is as follows:

[0079] In the vehicle networking scene, each vehicle is equipped with sensors to generate raw data for the system, such as 3D point cloud and RGB image. At the same time, the vehicle locally generates historical data of different time frames. The i-th vehicle collects environmental perception information, i.e., the motion state information of all surrounding vehicles:

[0080] p i ={x,y,z,ψ,v,d}

[0081] which includes the vehicle center position (x, y, z), the distance d from the ego vehicle, the yaw angle ψ, and the relative speed v.

[0082] As a preferred scheme of the embodiment, the specific implementation process of step S2 is as follows:

[0083] A bird's eye view perspective is used for feature representation, and all vehicles project their perception information into the same global coordinate system. For the i-th vehicle, given its input as X i , the feature map is mapped to F i =Φ enc (X i ), where Φ enc () is an encoder. The feature extraction uses a PointPillars backbone network, which is selected because it can significantly reduce computational complexity and effectively handle large-scale data.

[0084] The extracted feature map is input into a spatial confidence generator module, which generates a spatial confidence map based on the feature map of each vehicle. Given the feature map F i , the corresponding spatial confidence map is generated as:

[0085] M i =Φ generator (F i )∈{0,1},

[0086] where Φ generator () is a detection decoder.

[0087] Transmitting raw high-dimensional feature maps typically requires a large amount of bandwidth, so feature compression is needed. The present application uses a typical encoder-decoder structure to compress the shared message. The encoder consists of a series of 2D convolution and max pooling layers, and the shared message is broadcast to all communicating vehicles. At the ego vehicle end, the decoder with multiple deconvolution layers restores the compressed information and passes it to other modules for further processing.

[0088] As a preferred scheme of the embodiment, the specific implementation process of step S3 is as follows:

[0089] Next, the application designs a graph attention network (GAT) to select relevant collaborators by effectively aggregating context information, where each vehicle serves as a node. The feature of each node is composed of the sparse feature map s i ∈R N×F and the motion state matrix p i ∈R N×F of the vehicle, where N represents the number of nodes, and F represents the number of features of each node. Then, the motion state matrix is projected to the same feature dimension as the sparse feature map using a custom multilayer perceptron, denoted as Meanwhile, the sparse feature map is subjected to a pooling operation to average its features. Both the projected motion state matrix and the pooled sparse feature map are concatenated along the channel dimension to generate a vectorized node feature,

[0090]

[0091] which is input to the GAT layer together with the adjacency matrix to understand the importance of the semantic information provided by the collaborators. That is, the input of the GAT layer is a set of node features To obtain sufficient expressive power, a shared self-attention mechanism a is performed on the nodes. Therefore, the attention coefficient is:

[0092]

[0093] where W is a learnable parameter matrix representing the importance of node connections. Next, a nonlinear function is applied to map the attention coefficient to a nonlinear space. Meanwhile, the hyperbolic tangent activation function is used to map each weight value to the interval [-1, 1], thereby generating the attention weight of the GAT layer output representing the importance of neighbor nodes to the current node. The communication range of the i-th vehicle is defined as R i ={j}, j∈N-1. According to the attention weight obtained previously, beneficial collaborative vehicles are selected, and vehicles that do not actively contribute to improving perception performance are deleted:

[0094]

[0095] Having determined the range of collaborative vehicles that can effectively improve perception performance, the key features that need to be transmitted within the range can be shared. Finally, the features shared by the collaborative vehicles are represented as:

[0096]

[0097] As a preferred scheme of the present embodiment, the specific implementation process of step S4 is as follows:

[0098] To further enhance the perception performance, the present invention utilizes temporal context information, including data features from historical frames as well as current frame features. To extract meaningful features from historical frames, a max-pooling operation is used to retain the most salient features in the current and historical frames, thus preserving key information while minimizing redundancy. These feature maps are then concatenated and attention operations are applied through a convolutional network to generate semantic weights for the temporal context information. Semantic information that is not fully captured in the current frame is supplemented by leveraging historical frames. These weight values are then assigned to the feature maps of the historical frames, resulting in useful historical features after screening. This process is illustrated as follows:

[0099]

[0100] where σ denotes the sigmoid activation function, denotes the convolutional layer for fusion, and denotes the concatenation operation. Subsequently, pixel-wise attention weighting is employed to filter and select information, highlighting important parts of historical features:

[0101]

[0102] To integrate historical information and enhance the perception ability of the current representation, the present invention designs a Transformer algorithm to fuse the spatiotemporal features of the ego vehicle to enhance the accuracy of downstream tasks. The Transformer layer utilizes attention mechanisms to learn the internal dependencies between frame-level features, capturing the relationship between each element and all other elements. By calculating attention weights, the Transformer focuses on historical features related to the current element and encodes them as a context-aware feature vector. Transformers are used in both temporal and spatial dimensions to capture complex temporal relationships. Given the historical frame features and the current frame features the hidden layer spatiotemporal representation of T frames N feature pixels, F h ist ∈R T ×N×D and F curr ∈R T×N×D where D is the number of hidden layer dimensions. The query matrix, key matrix, and value matrix are obtained through the temporal Transformer layer:

[0103] Q t ,K t ,V t =Temporal_Self_Attention(F h ist ,F curr ;Θ)

[0104] where Θ is a learnable parameter. Then, the self-attention score is calculated as follows:

[0105]

[0106] A t capture the temporal relationship between different spatial nodes. Finally, the output of the temporal Transformer is:

[0107] O t = A t V t

[0108] Similar to the temporal Transformer layer, the representation of the spatial Transformer layer is as follows:

[0109] O s = Spatial_Self_Attention(O t )

[0110] Finally, the output after the temporal and spatial Transformer layers is passed through a projection layer composed of layer normalization, residual connection and multi-head mechanism to obtain the self-vehicle spatio-temporal feature representation:

[0111]

[0112] where H represents the desired spatio-temporal feature, and d represents the output feature dimension number. The introduction of the Transformer enables the model to more effectively learn the long-term dependencies between sequence data and improve the modeling ability of sequence data, thereby providing more accurate feature representation for subsequent tasks.

[0113] As a preferred scheme of the present embodiment, the specific implementation process of step S5 is as follows:

[0114] After step S4, the obtained collaborative features, self-vehicle spatio-temporal features and self-vehicle features are integrated using a multi-scale attention mechanism. First, as shown in Figure 1 , the present application performs multi-scale feature extraction on each type of feature. Subsequently, an attention mechanism is introduced to distinguish the importance of the features at each scale. For collaborative features, spatio-temporal fusion features and self-vehicle features, their respective attention weights are calculated to emphasize the most relevant parts in the current task. Finally, the features weighted by the attention mechanism are spliced to ensure that the fused features encapsulate comprehensive information. The fused features will be fed into the prediction head to generate bounding box proposals and related confidence scores to achieve the target detection task.

[0115] The method of the present application significantly improves the accuracy of target detection, and compared with other collaborative perception methods, it is found that the present scheme can obtain the best perception accuracy compared with existing algorithms. Figure 2is an embodiment of the present application, which compares the performance of the present application at intersection-over-union ratio of 0.5 and 0.7 respectively. Figure 3 and Figure 4 is an embodiment of the present application, which compares the perceptual performance of the present application at different compression ratios. The results show that the present application is robust to feature compression, and the method is always superior to other co-perception methods at different compression ratios. And when the intersection-over-union ratio is 0.7, the advantage is particularly obvious, which shows that the present application can reduce the communication bandwidth requirement while maintaining high performance. Figure 5 is an embodiment of the present application, which compares the perceptual performance of the present application at different positioning errors. The results show that the present application has better perceptual performance than other methods at different positioning errors.

[0116] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0117] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0118] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0119] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable devices provide a process for implementing the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0120] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the present application should be understood as their common meanings to those skilled in the art. The terms "first", "second" and similar terms used in the present application do not indicate any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to indicate relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0121] The above is only the preferred embodiment of the present application, and is not intended to limit the other forms of the present application. Any skilled person in the art can use the disclosed technical content to make changes or modifications to equivalent embodiments. However, any simple modification, equivalent change and modification of the above embodiments made without departing from the technical solution of the present application, in accordance with the technical essence of the present application, still falls within the protection scope of the present application.

[0122] The present application is not limited to the above-mentioned best mode, and anyone can derive other various forms of a multi-vehicle cooperative perception method based on context awareness under the inspiration of the present application. Any equivalent change and modification made in accordance with the scope of the present application should be within the scope of the present application.

Claims

1. A context-aware multi-vehicle cooperative perception method, characterized in that: obtaining perception data of the ego vehicle and cooperating vehicles, including the position, speed, and context information of historical frame data of the vehicles; using a backbone network to extract features from the obtained data to obtain ego vehicle features and cooperating vehicle features, and sharing feature information through compression and sparsification; using the motion state information of the vehicles and the sparse feature map to dynamically select cooperating vehicles that can provide significant perception performance improvement in the current scene by combining the graph attention mechanism; using a feature fusion algorithm based on Transformer to fuse and align the perception features from the local historical frames and the current frame of the vehicle to form ego vehicle spatio-temporal fusion features; and performing multi-scale attention fusion on the ego vehicle spatio-temporal fusion features and the cooperating vehicle features, and using the fused global perception features for target detection tasks.

2. The context-aware multi-vehicle cooperative perception method according to claim 1, characterized in that: the motion state information of all vehicles around the ith vehicle includes: the vehicle center position (x, y, z), the distance d from the ego vehicle, the yaw angle ψ, and the relative speed v.

3. The context-aware multi-vehicle cooperative perception method according to claim 1, characterized in that: when using the backbone network to extract features from the obtained data: a bird's eye view perspective is used for feature representation, and all vehicles project the perception information into the same global coordinate system; a PointPillars backbone network is used for extraction; an encoder-decoder structure is used for compression and sparsification of shared feature information; the encoder consists of a series of 2D convolution and max pooling layers, and the shared message is broadcast to all communicating vehicles; at the ego vehicle end, the decoder with multiple deconvolution layers is used to restore the compressed information and pass it to other modules for further processing.

5. The context-aware multi-vehicle cooperative perception method according to claim 1, characterized in that: the use of the motion state information of the vehicles and the sparse feature map, combined with the graph attention mechanism, to dynamically select cooperating vehicles that can provide significant perception performance improvement in the current scene is specifically: wherein W is a learnable parameter matrix representing the importance of node-to-node connections; the feature representation shared by the cooperating vehicles is:

6. The context-aware multi-vehicle cooperative perception method according to claim 5, characterized in that: the use of the feature fusion algorithm based on Transformer to fuse and align the perception features from the local historical frames and the current frame of the vehicle to form ego vehicle spatio-temporal fusion features is specifically: using the temporal context information, including the data features from the historical frames and the current frame features, using the max pooling operation to retain the most significant features in the current and historical frames, then mapping and splicing the retained features, and applying attention operations through the convolution network to generate semantic weights of the temporal context information; then assigning the semantic weight values to the feature map of the historical frames to obtain useful historical features after screening: ​ ​ ​ p i = {x, y, z, ψ, v, d}, ​ ​ ​ ​ For the i-th vehicle, the given input is X i , the feature map is mapped as F i = Φ enc (X i ), where Φ enc () is the encoder; the feature ​ The extracted feature map is input to the spatial confidence generator to generate a spatial confidence map according to the feature map of each vehicle; given a feature map F i After that, the corresponding spatial confidence map is generated as: M i = Φ generator (F i ) ∈ {0,1}, where Φ generator () is a detection decoder.

4. The method of claim 1, wherein: ​ ​ ​ ​ Each vehicle acts as a node; the feature of each node is represented by the sparse feature map s of the corresponding vehicle i ∈ R N×F and the motion state matrix p of the vehicle i ∈ R N×F where N represents the number of nodes and F represents the number of features of each node; then a multi-layer perception is used to project the motion state matrix to the same feature dimension as the sparse feature map, and the mapping process of the multi-layer perception is represented as: wherein W1, W2 are weight matrices, b1, b2 are corresponding bias terms; the sparse feature map is subjected to a pooling operation to average features; the projected motion state matrix and the pooled sparse feature map are both concatenated along the channel dimension to generate vectorized node features, Then input to the graph attention network together with the adjacency matrix; the input of the graph attention network is a set of node features Then perform a shared self-attention mechanism a on the nodes; the attention coefficient is ​ Next, a nonlinear function is applied to map the attention coefficients to a nonlinear space; and a hyperbolic tangent activation function is used to map each weight value to the interval [-1, 1], thereby generating the attention weights of the graph attention network output representing the importance of the neighbor nodes to the current node The communication range of the i-th vehicle is defined as R i = {j}, j e N - 1 ; according to the attention weights Selecting beneficial cooperating vehicles, removing vehicles that do not positively contribute to the improved perception performance: ​ ​ ​ ​ where σ denotes a sigmoid activation function, denotes a convolutional layer for fusion, denotes a concatenation operation; Subsequently, the information is filtered and selected by pixel-wise attention weighting to highlight the important parts of the historical features: Then the spatio-temporal features of the ego vehicle are fused by the Transformer algorithm to give the historical frame features of the ego vehicle and the current frame features to obtain the hidden layer spatio-temporal representation F of T frames and N feature pixels hist ∈R T×N×D and F curr ∈R T×N×D , wherein D is the dimension number of the hidden layer; the query matrix, the key matrix and the value matrix are obtained through a time sequence Transformer layer: Q t ,K t ,V t = Temporal_Self_Attention(F hist ,F curr ; Θ), where Θ is the learnable parameter; then, the self-attention score is calculated as follows: So that A t Capture the temporal relationship between different spatial nodes; the output of the timing Transformer is: O t = A t V t The representation of the spatial Transformer layer is as follows: O s = Spatial_Self_Attention(O t ) The output of the temporal and spatial Transformer layers is projected by a projection layer composed of layer normalization, residual connection, and multi-head mechanism to obtain the fusion spatio-temporal feature representation of the ego vehicle: where H represents the desired spatio-temporal feature, and d represents the output feature dimension number.

7. The method of claim 1, wherein: The multi-scale attention fusion of the ego vehicle spatio-temporal fusion feature and the cooperative vehicle feature is based on the fused global perception feature, which is used for the target detection task, specifically: the attention mechanism weighted feature is spliced to ensure that the fused feature encapsulates comprehensive information; the fused feature is fed into the prediction head to generate bounding box proposals and related confidence scores to achieve the target detection task.

8. A context-aware based multi-vehicle cooperative perception system, characterized in that, It comprises: The acquisition module is configured to acquire perception data of the ego vehicle and the cooperative vehicle, including the position, speed, and context information of the historical frame data of the vehicle; The feature extraction module is configured to extract features of the acquired data by using a backbone network to obtain ego vehicle features and cooperative features, and to share feature information by compression and sparsification; The dynamic selection module is configured to dynamically select a cooperative vehicle that can provide significant perception performance improvement in the current scene by using motion state information of the vehicle and a sparse feature mapping graph in combination with a graph attention mechanism; The ego vehicle spatio-temporal fusion feature module is configured to use a Transformer-based feature fusion algorithm to fuse and align perception features from the local historical frames and the current frame of the vehicle to form ego vehicle spatio-temporal fusion features; The multi-scale attention fusion module is configured to perform multi-scale attention fusion of the ego vehicle spatio-temporal fusion features and the cooperative vehicle features, based on the fused global perception features, for the target detection task.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the context-aware multi-vehicle cooperative perception method according to any one of claims 1-7. 10.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the context-aware multi-vehicle cooperative perception method according to any one of claims 1-7.

Images