Vehicle-road cooperative joint sensing method and system under non-ideal condition
By using the V2X attention fusion module and delay compensation module in the vehicle-road collaborative perception system, the perception error and delay problems under non-ideal communication conditions are solved, and the robustness and accuracy of vehicle-road collaborative perception are improved, and the The safe and efficient sharing of perceived information reduces the occurrence of traffic accidents.
Patent Information
- Application Number
- CN202411155073.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing vehicle-road collaborative perception technology has not been effectively carried out under non-ideal communication conditions, and there are communication bandwidth, rate and distance limitations, resulting in perception error and delay problems.
The V2X attention fusion module and delay compensation module are used to extract the visual feature map of the lidar point cloud image through the PointPillar model, compress and share features, and use the self-attention mechanism to perform feature fusion between nodes, combining the vertical and crisscrossing attention module and Features - Attention symbiosis estimation component to reduce perceptual errors caused by communication delay.
It improves the robustness and accuracy of vehicle-road collaborative perception, realizes the safe, efficient and rapid sharing of perceived information in real scenarios, reduces traffic accidents, and improves traffic traffic efficiency.
Smart Images

Figure CN119091152A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of vehicle-road collaboration technology, and specifically relates to a vehicle-road collaboration joint perception method and system under non-ideal conditions. Background Art
[0002] With the widespread application of cutting-edge technologies such as artificial intelligence, autonomous driving has gradually become a new focus of social attention. Fast and accurate environmental perception is a necessary condition for autonomous vehicles to make scientific and reasonable decisions and precise real-time control. The results of environmental perception directly affect the autonomous vehicle's assessment of driving risks. Traditional single-vehicle perception technology is limited by the installation location and layout space of sensors, resulting in huge challenges in high-precision and blind-spot-free perception of dynamic and static traffic elements and road conditions. The development of vehicle-road cooperative perception systems provides a good opportunity for autonomous vehicles to obtain low-latency, highly reliable road traffic information.
[0003] Providing support for information interaction in the vehicle-road cooperative perception system is vehicle to everything (V2X) wireless communication technology. Currently, the mainstream technical standards of V2X mainly include dedicated short range communication technology (DSRC) and C-V2X (Cellular-V2X) technology based on cellular mobile communication systems.
[0004] Most of the existing collaborative sensing technologies do not take into account real-world scenarios and are based only on ideal communication conditions. However, in real-world scenarios, C-V2X technology is limited in communication bandwidth, communication rate, and communication distance, and there is also a certain communication delay. Summary of the invention
[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a vehicle-road collaborative joint perception method and system under non-ideal conditions, which solves the problems in the prior art.
[0006] The purpose of the present invention can be achieved through the following technical solutions:
[0007] A vehicle-road cooperative joint perception method under non-ideal conditions includes the following steps:
[0008] Select a node from smart cars and smart roadside infrastructure as the self-node, build a V2X graph around the self-node, and share metadata;
[0009] At each node, the PointPillar model is used to extract visual feature maps from the point cloud images obtained by the lidar sensing environment;
[0010] The extracted visual feature map is gradually compressed along the channel dimension using a series of 1×1 convolutions, and then each node shares the visual feature map using the Internet of Vehicles communication channel;
[0011] The feature map received by the self-node is corrected by setting the V2X attention fusion module and the delay compensation module. The self-node aggregates the corrected feature map and completes the feature fusion between nodes based on the self-attention mechanism to obtain the fused feature map.
[0012] The fused feature map is passed through two 1×1 convolutional layers for detection frame regression and classification, and the 3D detection frame results of the self-node perceiving other nodes are output.
[0013] Furthermore, the metadata includes: posture, timestamp, external parameters and node type.
[0014] Furthermore, in the feature map aggregation process, in order to address the data sparsity problem existing in a single-frame point cloud, the context information of the self-node from the previous frame is aggregated based on the context-aware information to capture the informative temporal clues and historical frame context semantic information.
[0015] Furthermore, the V2X attention fusion module adopts two consecutive criss-cross attention modules to implement the V2X attention mechanism in point cloud data. The criss-cross attention module aggregates context information in the horizontal and vertical directions, and collects context information from all pixels by serially stacking two criss-cross attention modules.
[0016] Furthermore, the V2X attention fusion module includes: a self-attention component and an inter-node attention component. The final fusion feature output A in the V2X attention module out for:
[0017] A out =F(A intra +A inter )
[0018] In the formula, F represents a set of maximum pooling layers, average pooling layers and convolutional layers, A inter A represents the sum of the output feature maps obtained by considering the collaboration between the self-agent and other agents. intra Represents the output feature map of the ego agent taking into account all the spatial information of the feature map.
[0019] Furthermore, the delay compensation module includes a feature-attention symbiosis estimation component and a time modulation component; the feature-attention symbiosis estimation component obtains the most informative collaborative features at present; the time modulation component fuses the original features and estimated features conditioned on the delay time to generate an estimation result.
[0020] A vehicle-road cooperative joint perception system under non-ideal conditions, comprising:
[0021] Metadata sharing module: selects a node from smart cars and smart roadside infrastructure as the self-node, builds a V2X graph around the self-node, and shares metadata;
[0022] Feature extraction module: At each node, the PointPillar model is used to extract visual feature maps from the point cloud image obtained by the lidar perception environment;
[0023] Feature compression sharing module: The extracted visual feature map is gradually compressed along the channel dimension using a series of 1×1 convolutions, and then each node uses the Internet of Vehicles communication channel to share the visual feature map;
[0024] Feature fusion module: The feature map received by the self-node is corrected by setting the V2X attention fusion module and the delay compensation module. The self-node aggregates the corrected feature map and completes the feature fusion between nodes based on the self-attention mechanism to obtain a fused feature map.
[0025] And, the detection module: the fused feature map is passed through two 1×1 convolutional layers for detection frame regression and classification, and the 3D detection frame results of the self-node perceiving other nodes are output.
[0026] A computer storage medium stores a readable program, which, when running, can execute the above-mentioned vehicle-road cooperative joint perception method under non-ideal conditions.
[0027] An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;
[0028] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the vehicle-road cooperative joint perception method under non-ideal conditions as described above.
[0029] A computer program product includes computer instructions, characterized in that the computer instructions instruct a computing device to perform operations corresponding to the above-mentioned vehicle-road collaborative joint perception method under non-ideal conditions.
[0030] Beneficial effects of the present invention:
[0031] 1. The method of the present invention aims at solving the problems of communication delay, sensor posture error and so on that may exist in real scenes. It uses V2X attention fusion method to enhance the interaction between self-node and other nodes, and uses delay compensation method to reduce the perception error caused by communication delay, thereby improving the robustness and accuracy of joint perception and making it closer to real scenes.
[0032] 2. The method of the present invention realizes the safe, efficient, accurate and rapid sharing of perception information between smart cars and smart roadside facilities, provides information guarantee for the development of vehicle-road cooperative applications and the scientific and reasonable decision-making of autonomous driving vehicles, and with the assistance of the vehicle-road cooperative perception system, reduces the occurrence of traffic accidents and improves traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0034] Figure 1 It is a schematic diagram of the framework of the vehicle-road cooperative perception method of the present invention;
[0035] Figure 2 It is a schematic diagram of the structure of the V2X attention fusion module of the present invention;
[0036] Figure 3 is a flow chart of the algorithm of the feature-attention co-occurrence estimation component of the present invention;
[0037] Figure 4 It is a schematic diagram of the principle of the time modulation component of the present invention;
[0038] Figure 5 is a schematic diagram of the vehicle-road cooperative application of the present invention;
[0039] Figure 6 is a performance comparison diagram of the method of the present invention and other methods as the communication delay increases;
[0040] Figure 7 This is a performance comparison diagram between the method of the present invention and other methods as the positioning error increases. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] Example 1
[0043] like Figure 1 As shown, a vehicle-road cooperative joint perception method under non-ideal conditions includes the following steps:
[0044] S1, select a node from smart cars and smart roadside infrastructure as the self-node, build a V2X graph around the self-node, and share metadata;
[0045] In this embodiment, the vehicle-road cooperative joint perception is regarded as a heterogeneous multi-node perception system, in which different types of nodes (including smart cars and smart roadside infrastructure) perceive the surrounding environment and communicate with each other;
[0046] The shared metadata includes: pose, timestamp, external parameters, and node type; and the possible delay caused by metadata sharing is ignored here, that is, it is assumed that the transmission of metadata is well synchronized, which means that each node can receive the pose and coordinate information of the self node at time t. After receiving the information of the self vehicle, other nodes within the communication range act as collaborators and project their own LiDAR point cloud to the coordinate system of the self vehicle before feature extraction.
[0047] S2, at each node, uses the PointPillar model to extract visual feature maps from the point cloud image obtained by the lidar perception environment;
[0048] The steps to extract visual feature maps from point cloud images using the PointPillar model include:
[0049] First, the 3D point cloud data is divided into different grid intervals according to the X-axis and Y-axis coordinates. The Z-axis coordinate is not considered. The points in the same grid interval together form a "cylinder". Each point is represented by a nine-dimensional vector, which is (x, y, z, r, x c ,y c ,z c ,x p ,y p ). Where x, y, z represent the real three-dimensional coordinate information of the Point, r represents the reflection intensity of the Point, c ,y c ,z c Indicates the geometric center of all point clouds in the Pillar where the Point is located, x p =xx c ,y p =yx c , indicating the relative position of this Point to the above geometric center.
[0050] Then assume that there are P non-empty pillars in each sample, and each pillar has N points. D = 9 means that each point is represented by a nine-dimensional vector, so each point cloud sample data can be represented by a (D, P, N) tensor. If the number of points in each pillar is greater than N, the points in the pillar will be randomly sampled to ensure that the number is N; if the number of points in each pillar is less than N, the missing parts will be filled with 0. The above steps are used to realize the tensorization of 3D point cloud sample data.
[0051] Next, use PointNet to perform dimensional processing on the tensorized point cloud data. The dimension of the original point cloud data is D=9, which is converted to C after dimensional processing, so that it can be represented by a tensor of (C, P, N). Then, the maximum pooling is performed according to the dimension of Pillar, and finally a feature map with a dimension of (C, P) is obtained. In order to reduce the amount of calculation, the original 3D point cloud data needs to be converted into a 2D "pseudo-image". For this purpose, P is converted into (H, W), that is, P→H×W, to obtain a "pseudo-image" of the shape of (C, H, W).
[0052] Finally, two networks are used to further extract features from the 2D "pseudo-image": one is to continuously reduce the resolution of the feature map and increase the dimension of the feature map at the same time, in order to obtain three feature maps with different resolutions; the other is to upsample the three feature maps to the same size and then connect them. Since feature maps with different resolutions have different receptive fields, the advantage of the above processing is that feature maps with different resolutions can be used for the detection of objects of different sizes. For example, high-resolution ones can be used for pedestrian detection in the distance, and low-resolution ones can be used for nearby vehicle detection.
[0053] S3, the visual feature map extracted by S2 is gradually compressed along the channel dimension using a series of 1×1 convolutions, and then each node uses the IoV communication channel to share the visual feature map extracted from S2;
[0054] In this step, due to the imperfect communication channel, communication delay and communication noise will occur, resulting in the feature graphs received by the self-node from other nodes being unsatisfactory.
[0055] S4, by setting the V2X attention fusion module and the delay compensation module to correct the feature map received by the self-node, the self-node aggregates the corrected feature map, and completes the feature fusion between nodes based on the self-attention mechanism to obtain the fused feature map, so as to achieve the purpose of joint perception;
[0056] In the feature map aggregation process, in order to solve the data sparsity problem of single-frame point cloud, the context information of the self-node from the previous frame is aggregated based on the context-aware information, and the rich temporal clues and historical frame context semantic information are captured to improve the perception ability.
[0057] like Figure 2 As shown, the V2X self-attention fusion module includes: a self-attention component EgoAV and an inter-node attention component AV / Infra;
[0058] For the self-node only, the self-attention component EgoAV can make the features from any position globally aware, thereby utilizing all contextual information to better capture representative features. Let H e ∈R C×H×W is the input feature map of the self node, which is the “perfect” data obtained by the self node without suffering any lossy communication. In the self-attention component, the feature map H e It will be calculated through three 1×1 convolutional layers to generate three feature vectors Q e , K e and V e , they have the same dimensions, {Q e ,K e ,V e}∈R C×H×W Calculate Q e and K e Then we separate them using the scaling factor (i.e. the dimension of the feature vector) and apply the softmax function to get V e The weight of is defined as follows:
[0059]
[0060] in It's K e The dimension of A is , and the standard softmax() function is used as the activation function. intra Represents the output feature map of the ego agent taking into account all the spatial information of the feature map.
[0061] The intermediate feature map H aggregated from other nodes s ∈R C×H×W There is "noise" in lossy communication. In order to solve this problem, an uncertainty-aware node-to-node attention fusion method is proposed. In this module, the shared feature map will be calculated through two 1×1 convolutional layers to generate two feature vectors K s and V s , another eigenvector Q e It is obtained directly from the self-node rather than from other nodes. The inter-node attention is defined as:
[0062]
[0063] in yes dimension, N is the number of other agents within the communication range, A inter It represents the sum of the output feature maps obtained by considering the collaboration between the self-agent and other agents.
[0064] Two consecutive criss-cross (CC) attention modules are used to implement the V2X attention mechanism in point cloud data, instead of scaling dot product attention. The latter generates a huge attention map to measure the relationship of each point pair, resulting in O(H×W) 2 ) is very complex, where H×W is the input feature H e and H s The CC attention module aggregates contextual information in the horizontal and vertical directions, and collects contextual information from all pixels by stacking two CC attention modules in series. After obtaining self-attention and inter-node attention, they are respectively input into the maximum pooling layer and the average pooling layer to obtain rich spatial information, and then they are connected as the input of the 2D convolutional layer with ReLU activation function; therefore, the final fusion feature output A in the V2X attention module is out for:
[0065] A out =F(A intra +A inter )
[0066] Where F represents a set of maximum pooling layers, average pooling layers, and convolutional layers.
[0067] The delay compensation module includes a feature-attention co-occurrence estimation component and a temporal modulation component;
[0068] The feature-attention co-occurrence estimation component is based on a dual-branch architecture (including a feature estimation branch and an attention estimation branch) to simultaneously estimate features and their corresponding collaborative attention. The two branches of the dual LSTM network share the same input, including the real-time features perceived by the ego node i and k frames of historical features perceived by its collaborative node j. Each branch is implemented by a pyramid LSTM, which models a series of historical collaborative information and estimates the current state. The pyramid LSTM can capture local-to-global features at multiple scales, and the same movement occurs in similar regions on the features when other nodes move relative to the ego node.
[0069] The feature estimation branch aims to obtain the most informative collaborative features at the moment, and the feature estimation branch should have attention perception capabilities; the attention estimation branch aims to find the most informative collaborative areas at the moment and suppress areas with large estimation errors, so the attention estimation branch should have feature perception capabilities. In order to allow feature estimation to correspond to the corresponding attention, the feature estimation map of the previous timestamp and the collaborative attention are used as the input of the next timestamp of any branch. The algorithm flow is as follows: Figure 3 As shown, where τ is the delay time, k is the historical frame, t0 is the current time, and is the collaborative attention and features from the jth agent to the ith agent at time t, and are the estimates of collaborative attention and features at timestamp t, respectively, (t) is the input of the pyramid LSTM at timestamp t, and are the hidden state and cell state of the pyramid LSTM in each branch, respectively.
[0070] The principle of the time modulation component is as follows Figure 4 As shown, based on the feature-attention co-occurrence estimation component, the temporal modulation component fuses the original features conditioned on the delay time (which works well at low delay) and the estimated features (which works well at high delay) to generate more comprehensive and reliable estimates.
[0071] make is the confidence matrix, reflecting the estimated uncertainty level of each spatial region, T F ∈R H×W×C and T W ∈R H×W×C is the delay tensor obtained by expanding the delay time τ∈R, and its dimensions are and Similarly, the expression for time modulation is as follows:
[0072]
[0073] Where m F (·) and m W (·) are all lightweight convolutional neural networks with sigmoid activation function, 1∈R H×W is a matrix with all elements set to 1. The confidence matrix of feature estimates for each spatial region is obtained by utilizing the collaborative features / attention estimated by feature-attention co-occurrence, the latest asynchronous features / attention, and the concatenation of the delay tensor. Then combine the estimated features / attention and the latest asynchronous features respectively / attention We expect that when the delay is higher, the confidence matrix will have higher weight and the estimated features / attention will contribute more to the final estimate.
[0074] After the feature fusion process is corrected by the V2X attention fusion module and the delay compensation module, the robustness and accuracy of the obtained fused feature map are greatly improved.
[0075] S5, the fused feature map obtained in S4 is passed through two 1×1 convolutional layers of the detection head to perform detection frame regression and classification, and output the 3D detection frame results of the self-node perceiving other nodes.
[0076] The detection head part belongs to the decoder, including the classification head, regression head and direction classification head. After the feature map is decoded, a predicted 3D detection frame is output, which is represented by a seven-dimensional vector (x, y, z, w, h, l, θ), where (x, y, z) represents the coordinate center, w, h, l is the size information of the width, height and length of the target, and θ is the heading angle. The parameters to be learned in the detection frame regression task are the offsets of these seven variables, as shown in the formula:
[0077]
[0078] Among them, the coordinate variable with superscript gt represents ground truth, that is, the coordinate variable of the real annotation label, and the superscript ɑ represents the coordinate variable predicted by the model.
[0079] The loss function of PointPillars consists of three parts: position coordinate loss L loc , direction loss L dir , classification loss L cls , where the position coordinate loss L loc The Smooth L1 loss function is used for training, as follows
[0080]
[0081] In the formula, Δb corresponds to the 7 variable offsets calculated in formula 4-5.
[0082] Direction loss L dir Softmax loss is used to avoid direction discrimination errors.
[0083] Classification loss L cls , calculated using Focal Loss, is defined as follows:
[0084] L cls =-α a (1-p a ) γ log p a(4-7)
[0085] In the formula, p a For the category probability of an anchor, set the hyperparameters α=0.25 and γ=2.
[0086] The overall function definition is as follows:
[0087]
[0088] Where N pos is the number of activated anchors. Set the weight parameter β loc =2,β cls =1,β dir =0.2.
[0089] By minimizing the loss function training and reducing the offset between the predicted box variable and the labeled box, the most accurate detection result has been obtained and the classification result is output according to the detection box position.
[0090] Example 2
[0091] like Figure 5 As shown, the vehicle-road cooperative joint perception method under non-ideal conditions in Example 1 is applied to actual traffic scenarios, and the nodes include cars and road test equipment. At the T-junction, the east-west self-vehicle EgoAV is blocked by the green vehicle in the same direction, and cannot see pedestrians crossing the road, which may cause traffic accidents. At this time, the southbound vehicle uses lidar to perceive pedestrians crossing the road, uses PointPillar for feature extraction, and uses LTE-V2X technology to share the extracted voxel features with the self-vehicle. Communication loss and delay are inevitable in this process. The self-vehicle receives this information to expand the field of view, and uses the developed vehicle-road cooperative joint perception method under non-ideal conditions for information fusion. The V2X attention fusion method is used to enhance the interactive information between the self-vehicle and other vehicles, and the delay compensation method is used to reduce the perception error caused by communication delay. In the end, a more accurate perception result is obtained, and pedestrians crossing the road are perceived when the single vehicle's field of view is blocked, providing a perception basis for subsequent self-vehicle decision-making and control.
[0092] Example 3
[0093] In this embodiment, a comparative experiment is used to analyze and verify the vehicle-road cooperative joint perception method under non-ideal conditions of the present invention; the verification is carried out from the following two stages:
[0094] (1) First, we conduct research on the joint perception method of vehicle-road collaboration under ideal conditions. This method can be applied to a variety of open datasets for autonomous driving, such as OPV2V, V2Xset, Dair-V2X, etc. On the OPV2V dataset, the perception accuracy of this method is more than 2.7% higher than that of other algorithms under the condition of AP@IoU=0.7.
[0095] (2) Research on vehicle-road cooperative joint perception algorithm under non-ideal conditions. Based on the research content (1), considering the problems of delay and posture error in real scenes, further research is conducted on the vehicle-road cooperative joint perception algorithm under non-ideal conditions. The V2X attention fusion method is used to enhance the interaction between the self-node and other nodes, and the delay compensation method is used to reduce the perception error caused by communication delay.
[0096] The experimental test process is:
[0097] 1. Data collection: Use the Carla simulator to collect collaborative perception data in traffic scenarios, and add two types of noise to simulate non-ideal environments in reality. The two types of noise are posture error. The position and heading angle errors of the sensor are taken from Gaussian distribution, and the default standard deviations are 0.1m and 0.2° respectively; the second is communication error, which mainly includes possible delay and noise problems. The relevant data has been collected using the Carla simulator.
[0098] 2. Model setting: The model optimization process uses the Adam optimizer, and the initial learning rate is 10 -3 The decay factor is 0.1 and decays steadily every 10 epochs. All models are iteratively trained on an NVIDIA GeForce RTX4090 GPU. The intersection over union (IoU) and average precision (AP) are selected as the evaluation indicators for the performance of the vehicle-road cooperative joint perception algorithm. The AP of target detection is calculated at the thresholds of IoU = 0.5 and IoU = 0.7, and the results are compared, evaluated, and analyzed.
[0099] 3. Training model: Select single-vehicle perception (No Fusion) as the detection benchmark, that is, only use the ego vehicle's LiDAR 3D point cloud data, without collaboration with other agents, and compare the performance with the existing early / mid / late joint perception models. Enter the python command to train the model.
[0100] 4. Test model: After training, a configuration file containing model information is generated. Load the configuration file and enter the python command for testing. The test results are shown in Table 1.
[0101] Table 1 shows the test results of the vehicle-road cooperative perception algorithm proposed in the present invention and other cooperative perception algorithms on a self-made non-ideal cooperative perception dataset:
[0102]
[0103] The experimental results show that the vehicle-road cooperative joint perception algorithm proposed in this invention still achieves the best detection results among all perception algorithms, which is 3.2% higher than other algorithms under the condition of AP@IoU=0.5, and the performance degradation is the smallest compared with the ideal condition, which is only 3.4%, which shows the effectiveness of the V2X attention fusion module and delay compensation module proposed in this invention. In addition, the experiment evaluates the impact of different degrees of communication delay and positioning error on algorithm performance, such as Figure 6 As shown in Figure 7, the results show that with the increase of communication delay and positioning error, except for the single-agent perception No fusion algorithm, the performance of other algorithms has declined to varying degrees. The performance of the algorithm of the present invention decreases significantly less than that of other algorithms with the increase of communication delay and positioning error, which proves that the algorithm of the present invention has high robustness.
[0104] The method of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.
[0105] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.
Claims
1. A joint perception method for vehicle-road collaboration under non-ideal conditions, It is characterized in that The following steps are involved: Select a node from smart cars and smart roadside infrastructure as the self-node, build a V2X graph around the self-node, and share metadata; At each node, the PointPillar model is used to extract visual feature maps from the point cloud images obtained by the lidar sensing environment; The extracted visual feature map is gradually compressed along the channel dimension using a series of 1×1 convolutions, and then each node shares the visual feature map using the Internet of Vehicles communication channel; The feature map received by the self-node is corrected by setting the V2X attention fusion module and the delay compensation module. The self-node aggregates the corrected feature map and completes the feature fusion between nodes based on the self-attention mechanism to obtain the fused feature map. The fused feature map is passed through two 1×1 convolutional layers for detection frame regression and classification, and the 3D detection frame results of the self-node perceiving other nodes are output.
2. According to the vehicle-road cooperative joint perception method under non-ideal conditions according to claim 1, It is characterized in that The metadata includes: posture, timestamp, external parameters and node type.
3. According to the vehicle-road cooperative joint perception method under non-ideal conditions as described in claim 1, It is characterized in that In the feature map aggregation process, in order to address the data sparsity problem existing in a single-frame point cloud, the context information of the self-node from the previous frame is aggregated based on context-aware information to capture informative temporal clues and historical frame context semantic information.
4. According to the vehicle-road cooperative joint perception method under non-ideal conditions according to claim 3, It is characterized in that The V2X attention fusion module adopts two consecutive criss-cross attention modules to implement the V2X attention mechanism in point cloud data. The criss-cross attention module aggregates context information in the horizontal and vertical directions, and collects context information from all pixels by serially stacking two criss-cross attention modules.
5. According to the vehicle-road cooperative joint perception method under non-ideal conditions according to claim 4, It is characterized in that The V2X attention fusion module includes: a self-attention component and an inter-node attention component. The final fusion feature output A in the V2X attention module out for: A out =F(A intra +A inter ) In the formula, F represents a set of maximum pooling layers, average pooling layers and convolutional layers, A inter A represents the sum of the output feature maps obtained by considering the collaboration between the self-agent and other agents. intra Represents the output feature map of the ego agent taking into account all the spatial information of the feature map.
6. The vehicle-road cooperative joint perception method under non-ideal conditions according to claim 5, It is characterized in that The delay compensation module includes a feature-attention co-occurrence estimation component and a time modulation component; The feature-attention co-occurrence estimation component obtains the most informative collaborative features at the moment; The time modulation component fuses the original features and the estimated features conditioned on the delay time to generate an estimation result.
7. A vehicle-road cooperative joint perception system under non-ideal conditions, It is characterized in that include: Metadata sharing module: selects a node from smart cars and smart roadside infrastructure as the self-node, builds a V2X graph around the self-node, and shares metadata; Feature extraction module: At each node, the PointPillar model is used to extract visual feature maps from the point cloud image obtained by the lidar perception environment; Feature compression sharing module: The extracted visual feature map is gradually compressed along the channel dimension using a series of 1×1 convolutions, and then each node uses the Internet of Vehicles communication channel to share the visual feature map; Feature fusion module: The feature map received by the self-node is corrected by setting the V2X attention fusion module and the delay compensation module. The self-node aggregates the corrected feature map and completes the feature fusion between nodes based on the self-attention mechanism to obtain a fused feature map. And, the detection module: the fused feature map is passed through two 1×1 convolutional layers for detection frame regression and classification, and the 3D detection frame results of the self-node perceiving other nodes are output.
8. A computer storage medium storing a readable program, It is characterized in that When the program is running, it can execute the vehicle-road collaborative joint perception method under non-ideal conditions as described in any one of claims 1-6.
9. An electronic device, include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to a vehicle-road cooperative joint perception method under non-ideal conditions as described in any one of claims 1-6.
10. A computer program product comprising computer instructions, It is characterized in that The computer instructions instruct the computing device to perform operations corresponding to a vehicle-road collaborative joint perception method under non-ideal conditions as described in any one of claims 1-6.
Citation Information
Patent Citations
Road vehicle sensing method based on multi-sensor fusion
CN116625383A
Multi-source data collaboration and fusion perception method for network automatic driving
CN117237772A
Vehicle-road collaboration-oriented sensing information fusion representation and target detection method
WO2022141912A1
Cited By
Intelligent collaborative awareness-oriented instance-level semantic simplification interaction method
CN120047914A
An instance-level semantic interaction method for intelligent collaborative perception
CN120047914B
Internet of vehicles cooperative sensing parameter high-efficiency domain self-adaption method and device and medium
CN121711658A
