BEV-based multi-source sensing data fusion and vehicle and road resource allocation method

Abstract

The invention relates to a BEV-based multi-source sensing data fusion and vehicle and road resource allocation method, and belongs to the technical field of intelligent traffic. In order to solve the problem of high sensing time delay caused by difficulty in data fusion of a multi-modal sensor and low scheduling efficiency of edge computing resources in an Internet of Vehicles environment, the following technical scheme is provided: a vehicle-road double-layer network architecture is constructed, and data acquisition is realized by cooperation of an intelligent networked vehicle and a road side unit; designing a collaborative parallel perception time delay optimization model, and reconstructing a shielding area through role dynamic switching of a demand vehicle and a response vehicle; establishing an aerial view visual angle multi-modal fusion framework, and unifying a heterogeneous sensor feature space; and optimizing a computing resource allocation and unloading decision by adopting a mixed action space depth deterministic strategy gradient algorithm. The method has the technical effects that the global sensing fusion time delay is remarkably reduced, the multi-source data alignment precision is improved, the dynamic traffic scene adaptability is enhanced, and the edge resource utilization rate is optimized.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology and relates to a multi-source perception data fusion and vehicle-road resource allocation method based on BEV. Background Art

[0002] In recent years, artificial intelligence (AI)-enabled vehicle-to-everything (V2X) communication networks (V2X) have become a growing research hotspot in academia and industry due to their potential to improve user quality of experience (QoE) in complex traffic environments. With the development of key V2X technologies, the number of computationally intensive and latency-sensitive tasks onboard vehicles has increased dramatically, such as real-time traffic flow prediction and collision warning. However, local processing faces the dual constraints of low energy efficiency and response latency. In this context, Mobile Edge Computing (MEC) has emerged as a distributed complementary architecture to cloud computing. By decentralizing computing power to edge nodes on the network, it leverages the advantages of near-user deployment and additional computing resources to significantly reduce service latency. This marginal deployment of computing and caching resources simultaneously reduces task processing latency, transmission energy consumption, and backhaul bandwidth usage. Notably, the high mobility of vehicles requires that offloading decisions dynamically adapt to real-time network conditions. This makes it difficult for traditional optimization methods to ensure robustness in complex and dynamic VEC scenarios. Therefore, a collaborative optimization mechanism combining secure offloading and resource allocation is urgently needed.

[0003] To achieve accurate road perception in complex traffic scenarios, autonomous driving systems are generally equipped with multi-source heterogeneous sensors, each providing complementary perception signals. For example, cameras capture high-density visual information, LiDAR generates precise three-dimensional point clouds, and millimeter-wave radar acquires the velocity vector of moving targets in real time. Therefore, multimodal sensor fusion is essential for achieving high-precision and reliable perception. However, the native data representations of heterogeneous sensors vary significantly, making cross-modal feature alignment difficult. To overcome this challenge, it is urgent to construct a unified representation space to achieve highly consistent feature fusion of multi-source perception data.

[0004] To address the above issues, the present invention designs a multi-source perception data fusion and vehicle-road resource allocation method based on a bird's-eye view (BEV). First, a vehicle-road dual-layer communication network architecture is constructed. Second, based on this network architecture, the perception fusion task is performed, and a collaborative parallel perception delay optimization model is established. Then, a multimodal data fusion model for BEV is constructed, and offloading decisions are made through perception quality assessment. Multimodal data fusion is completed with the assistance of RSUs. Finally, a resource allocation optimization algorithm based on HDDPG is proposed to minimize the global perception fusion delay. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a multi-source perception data fusion and vehicle-road resource allocation method based on BEV, which is used to solve the problem of heterogeneous data fusion and vehicle-road collaborative perception processing delay caused by multimodal sensors.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention proposes a method for multimodal data fusion and edge computing resource scheduling based on the complexity of vehicle networking scenarios to reduce processing latency. The method includes the following steps:

[0008] S1: Propose a vehicle-road dual-layer network architecture based on perception fusion;

[0009] S2: Build a collaborative parallel perception delay optimization model;

[0010] S3: Construct a multimodal data fusion model based on BEV fusion;

[0011] S4: Modeling the resource allocation problem of multi-source perception data fusion;

[0012] S5: Design of resource scheduling algorithm based on Hybrid Action Space Based Deep Deterministic Policy Gradient (HDDPG);

[0013] On the second aspect, in S1, an embodiment of the present invention constructs a vehicle-road dual-layer communication network architecture based on collaborative perception, which includes a mobile layer composed of intelligent connected vehicles (Connected and Autonomous Vehicles, CAV) and an edge layer composed of road side units (Road Side Unit, RSU). Among them, the CAV end integrates multimodal sensors and embedded edge computing units, deploys a lightweight BEV fusion model, performs local perception data collection, multi-source feature extraction and BEV feature generation, and realizes inter-vehicle / vehicle-road data sharing through V2V / V2I communication protocols. The RSU end is equipped with a high-performance edge server and a complete BEV fusion model, supports dynamic resource scheduling and collaborative computing offloading, implements feature enhancement processing, and adopts Orthogonal Frequency Division Multiplexing (OFDM) mechanism to achieve low-latency data transmission.

[0014] On the third aspect, in S2, an embodiment of the present invention establishes a Collaborative Parallel Perception Latency Optimization (CPPLO) model, and the CPPLO model deploys a dual-role dynamic switching mechanism on the CAV side. When the demanding vehicle (D-CAV) perceives an obstruction and there is a blind spot in the field of view, a collaborative perception demand instruction is first generated and broadcast to the adjacent CAV, and then the perception information submitted by the multiple responding R-CAVs is spatially fused with the assistance of the RSU to reconstruct the global region of interest (ROI). When the R-CAV receives the collaborative perception demand issued by the D-CAV, it starts the parallel perception data processing pipeline, performs perception compression and ROI extraction on the original perception information in parallel, and finally sends the ROI extraction information to the D-CAV that initiated the perception demand.

[0015] In the fourth aspect, in S3, an embodiment of the present invention establishes a multimodal data fusion model based on BEV fusion, and aligns the multimodal data across modalities in the BEV space to complete data fusion with the assistance of the RSU. Specifically, a lightweight BEV fusion model is deployed on the D-CAV side to perform operations such as timestamp synchronization, feature extraction, perspective conversion, and quality assessment of heterogeneous data, thereby executing unloading decisions based on quality assessment. At the same time, a complete BEV fusion model is deployed on the RSU side to perform feature enhancement on the data transmitted from the CAV, and serialize the results and transmit them back to the D-CAV. Finally, the D-CAV will select a description for each occluder from the perception fusion information for the final result fusion. The present invention defines the description of ROI perception fusion quality as a quantitative indicator based on multi-dimensional joint evaluation, including observation distance, vehicle computing resources, and perception confidence.

[0016] Fifthly, in S4, an embodiment of the present invention considers the multimodal data perception fusion process based on the CPPLO model, takes minimizing the vehicle-road dual-layer network collaborative global perception fusion delay as the goal, formally defines the objective function and constraints through mathematical modeling methods, and models the resource allocation optimization problem.

[0017] In a sixth aspect, in S5, the present invention designs a resource allocation algorithm based on HDDPG to address the non-convex characteristics of the ROI fusion resource allocation optimization problem and the local optimal convergence defects of traditional gradient descent methods. First, the resource allocation problem is modeled as a parameterized action space Markov decision process (PAMDPs), and then the hybrid action space Actor-Critic framework and the Deep Q-Network (DQN) algorithm mechanism are integrated to construct an HDDPG optimization model to achieve global optimal decision-making for ROI fusion resource allocation.

[0018] The beneficial effects of the present invention are:

[0019] (1) Through the collaborative parallel perception delay optimization model, the present invention realizes the dual-role dynamic switching mechanism of the demand vehicle and the response vehicle. When the vehicle senses the blind spot of vision, it immediately triggers the coordinated instruction broadcast and parallel data processing pipeline, effectively avoiding the cumulative delay caused by traditional serial processing. The response vehicle synchronously performs perception compression and region of interest extraction, and uses computing and communication overlap technology to significantly shorten the task processing chain. Combined with the edge computing assistance of the roadside unit, the global perception fusion process is seamlessly connected, thereby significantly reducing the overall service response time and improving the real-time decision-making ability in complex traffic scenarios.

[0020] (2) Based on the unified representation space of the bird's-eye view perspective, the present invention solves the problem of heterogeneous alignment of multimodal sensor data. Heterogeneous data such as cameras, lidars, and millimeter-wave radars realize cross-modal feature fusion in the bird's-eye view space to generate a highly consistent environmental model. By embedding a quality assessment module, the system intelligently selects the optimal perception description (such as comprehensive observation distance, computing resource availability, and perception confidence) to ensure the reconstruction accuracy of occluded areas. The final fusion result enhances the recognition reliability of dynamic targets, effectively copes with complex factors such as lighting changes and occlusion interference, and provides safer and more reliable autonomous driving environment perception.

[0021] (3) By adopting a hybrid action space deep deterministic policy gradient algorithm, the present invention breaks through the local convergence limitations of traditional optimization methods. The algorithm models resource allocation as a parameterized action space decision process, dynamically captures the real-time resource status of vehicles and roadside units through the state space, and accurately controls the computing resource allocation, offloading decisions, and data segmentation parameters in the action space. The reward function drives the policy network to learn the long-term benefits and ensure the efficient use of computing resources and communication bandwidth. This not only reduces the load pressure on edge nodes, but also avoids resource fragmentation, achieving lightweight operation of the whole domain collaboration.

[0022] (4) The innovative design of the vehicle-road dual-layer network architecture enables the system to naturally support dynamic expansion in high-mobility environments. Intelligent connected vehicles, as mobile layer nodes, achieve low-interference data transmission through an orthogonal frequency division multiplexing mechanism; roadside units constitute the edge layer and provide elastic computing support. This architecture not only simplifies the deployment complexity of inter-vehicle and vehicle-road collaboration, but also adapts to traffic flow changes through a dual-role switching mechanism (demand vehicle and response vehicle). Ultimately, the system maintains stable operation when vehicle density fluctuates and network topology evolves dynamically, providing solid support for large-scale Internet of Vehicles applications.

[0023] (5) Through intelligent perception compression and region of interest extraction technology, the present invention effectively suppresses data redundancy. Responding vehicles only transmit information in key areas with high semantic density, significantly reducing the load on the wireless channel. Combined with the dynamic regulation of the offloading decision module (such as the intelligent trade-off between local processing and edge offloading), the system avoids unnecessary remote data transmission. This not only reduces the energy consumption of on-board equipment and roadside units, but also alleviates network congestion, providing technical support for a sustainable green transportation system.

[0024] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0026] Figure 1 This is a diagram of the vehicle-road two-layer network architecture based on the CPPLO model;

[0027] Figure 2 This is a diagram of the multimodal data fusion model based on BEV fusion;

[0028] Figure 3 This is a flow chart for training the resource scheduling algorithm based on HDDPG;

[0029] Figure 4 Flowchart for the implementation of multi-source perception data fusion and vehicle-road resource allocation method based on BEV fusion framework. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0032] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0033] Figure 1 FIG. 1 shows a possible structural diagram of a communication system involved in an embodiment of the present invention. Figure 1 As shown in the figure, the network architecture considers two layers of network, mobile layer and edge layer, which are composed of N CAVs and R RSUs respectively. and Among them, each CAV in the mobile layer is equipped with computing resources There are two roles in CAV: R-CAV and D-CAV. In the CPPLO framework, CAV can play both D-CAV and R-CAV roles at the same time. Specifically, R-CAV performs perception compression and ROI extraction in a pipelined and parallel manner, and finally performs information transmission; D-CAV goes through multiple steps: demand broadcasting, information reception, BEV feature generation, quality assessment and result fusion to reconstruct a high-quality and low-latency global perception fusion scenario. and In addition, in order to reduce the computational load of CAV and achieve efficient perception fusion, the present invention adopts RSU to assist CAV to complete the ROI fusion process.

[0034] 1. Vehicle-road dual-layer network architecture based on perception fusion

[0035] This paper considers a vehicle-road dual-layer network architecture based on sensor fusion. Data transmission and sharing between vehicles and between vehicles and roads can be supported by communication technologies such as V2V and V2I. This paper focuses on uplink offload delay analysis, ignoring the downlink feedback delay components of V2V and V2I. The OFDM-based communication link scheme can effectively avoid interference in the inter-vehicle transmission channel. Therefore, in time slot t, when CAV i connects to RSU m via a V2I link, the uplink transmission rate is:

[0036]

[0037] Among them, W V2I is the transmission channel bandwidth, B V2I is the bandwidth of the V2I link. Therefore, when multiple CAVs simultaneously offload fused information to the edge RSU, the maximum number of vehicles that the RSU can connect to is: K = B V2I / W V2I . is the transmission power of CAV i in time slot t, is the V2I channel gain at time slot t. In wireless communication, the signal power will attenuate as the distance increases, so the path loss factor is introduced. represents the path loss from CAV i to RSU m at time slot t. σ 2 represents the power of Gaussian channel noise.

[0038] According to Shannon’s formula, at time slot t, when R-CAV i and D-CAV j share data through the V2V link, the transmission rate is:

[0039]

[0040] in, is the V2V channel gain at time slot t; considering the signal power attenuation, represents the path loss between R-CAV i and D-CAV j.

[0041] 2. Collaborative Parallel Perception Latency Optimization Model

[0042] In complex traffic scenarios, CAV can play a dual role according to the CPPLO model, initiating collaborative perception needs and receiving multi-vehicle perception information at any time, and processing data in parallel.

[0043] 1) Respond to vehicle roles.

[0044] CAV CAV as R-CAV will perform three steps: perceptual compression, ROI extraction and information transmission.

[0045] In the perception compression step, due to the high-dimensional nature of the original multimodal perception data stream, direct transmission will cause significant communication delays. To this end, R-CAV i uses feature extraction and semantic entropy compression technology to convert the original perception data into a low-dimensional, high-semantic representation, effectively suppressing information redundancy and communication overhead while retaining the key features of collaborative perception. Therefore, at time slot t, the execution delay of R-CAV i perception compression is:

[0046]

[0047] Among them, D i represents the original perceptual information size of R-CAV i; represents the computational intensity of perceptual compression performed by R-CAV i; represents the ratio of computing power allocated to perceptual compression by R-CAV i.

[0048] In the ROI extraction step, to avoid extra bandwidth consumption and meet low-latency transmission constraints, ROI screening is required for the compressed sensing data: based on a dynamic target detection algorithm (such as the YOLOv7 architecture), target areas with significant spatial correlation with the D-CAVj's driving trajectory are extracted. Based on this, the execution delay of R-CAVi performing ROI extraction on the data sent to D-CAVj in time slot t is:

[0049]

[0050] Among them, η i→j Indicates the ROI extraction ratio of R-CAV i relative to D-CAVj, and the amount of compressed data The computational intensity required to perform ROI extraction for R-CAV i, Indicates the computing power ratio allocated by R-CAV i to ROI extraction.

[0051] In the information transmission step, the extracted ROI information is encapsulated and transmitted to the target D-CAVj, where the transmitted data includes the perception information of the ROI occluder. Therefore, the delay of transmitting the perception information from R-CAV i to D-CAVj is:

[0052]

[0053] Among them, the amount of data after extracting ROI

[0054] Ultimately, when the onboard CPU is performing sensor compression or ROI extraction, the communication module is idle; while when transmitting data, the CPU is idle. Therefore, the computation-communication pipeline and data block parallel processing are used to optimize the R-CAV i collaborative sensing latency. The raw sensor data is divided into L data blocks, and sensor compression and ROI extraction are performed in an overlapping manner, that is, the nth block is compressed while the n-1th block is extracted. Therefore, the total latency for the sensor data in time slot t to be transmitted after being processed by R-CAV i to D-CAVj is:

[0055]

[0056] Among them, T overhead =k×L+C0 is the system's additional scheduling time overhead, k is the average processing time overhead of each block of data, C0 is the basic fixed time overhead of system startup, L is the number of data blocks, and the size of the data block is:

[0057]

[0058] in, B0 is the base block size. The purpose of is to make the time consumption of the perceptual compression and ROI extraction stages close. According to the relative computing power of the compression module and the ROI extraction module, the size of the data block is dynamically adjusted to optimize the overall system throughput. Specifically, when the compression computing power is dominant, the block size is increased to fully utilize the ROI extraction computing power; conversely, when the ROI extraction computing power is dominant, the block size is reduced to prevent the compression process from becoming a system performance bottleneck. The number of data blocks is

[0059] 2) Requires vehicle role.

[0060] CAV The CAV as a D-CAV will perform the following steps: demand broadcasting, information reception, BEV feature generation, quality assessment, and result fusion.

[0061] In the demand broadcast step, when a D-CAV i senses an occluder in the ROI, it generates a collaborative sensing demand instruction, encapsulates the sensing demand of the occluder, and broadcasts it to nearby CAVs and RSUs. The sensing demand includes the demand instruction, ROI range, and location.

[0062] In the information receiving step, D-CAV refuses to receive perception information that exceeds the time limit to ensure the reconstruction of high-quality full-area BEV. The time delay for D-CAV i to receive all perception information is:

[0063]

[0064] Wherein, τ is the tolerable data reception delay of D-CAV, and the timed-out sensing data packets will be discarded.

[0065] During the BEV feature generation phase, after receiving all the sensor data from R-CAVj, D-CAVi first performs multi-source data decapsulation and spatiotemporal alignment. This step aims to compensate for the temporal drift and spatial deviation introduced by heterogeneous sensor sources and establish a consistent spatiotemporal reference for subsequent multi-source sensor fusion. Subsequently, D-CAVi extracts features based on the received ROI information and projects the features into the BEV space using perspective conversion technology. The feature extraction latency and BEV feature generation latency are:

[0066]

[0067] Among them, D j is the amount of original perception data received after decompression, M feat Compute intensity for feature extraction; Compression ratio generated for BEV features, number of features after extraction M view Indicates the computational intensity of the perspective transformation.

[0068] During the quality assessment step, D-CAV i selects an optimal description for each blind spot based on its own BEV perception information and that of the blind spots. This is then used for subsequent ROI fusion to reconstruct a high-quality global BEV perception scenario. Furthermore, D-CAV i performs quality assessment based on the received perception information and dynamically makes computation offloading decisions based on the assessment results.

[0069] In the result fusion stage, given that the high-quality fusion process of ROI places strict demands on on-board computing resources, its real-time performance and computational accuracy are difficult to guarantee simultaneously. To this end, the present invention proposes a multimodal data fusion architecture based on RSU assistance. D-CAV i unloads the intermediate fusion data of the selected ROI to RSU m according to the dynamic unloading strategy to perform feature enhancement calculations. After RSU m completes the feature enhancement processing, it transmits the serialized data back to D-CAV i, and finally D-CAV i integrates the outputs of each module to achieve full-domain perception fusion. Therefore, ROI fusion includes local computing and edge computing, and its total latency can be expressed as:

[0070]

[0071] Among them, T i local The local delay for D-CAV i in time slot t is calculated as: To calculate the delay at the edge of RSU m in time slot t, it can be expressed as:

[0072]

[0073] in, is the unloading ratio, M ld Compute strength for local decoding. is the unloading delay of CAV i to offload data to RSU m, is the RSU m feature enhancement delay, which can be expressed as:

[0074]

[0075] Based on this, the delay for D-CAVi to complete the processing and fusion of perception data in time slot t is:

[0076] T i D =T i receive +T i feat +T i BEV +T i fuse (15)

[0077] At this point, the total latency for CAVi to make reasonable offloading decisions and complete the global BEV fusion task with the assistance of RSUm is:

[0078] T i total =T i R +T i D (16)

[0079] 3. Multimodal data fusion model based on BEV fusion

[0080] In order to build a collaborative vehicle-road perception system, a layered computing architecture is adopted: a lightweight BEV fusion model is deployed on the on-board terminal, and a deep BEV model is deployed on the RSU. First, multimodal data feature extraction is performed on the vehicle side, and the extracted features are mapped to the BEV space through the perspective transformation module to generate structured BEV features. Then, the perception information quality assessment module selects the optimal feature description based on the quantitative scoring mechanism, and formulates a dynamic offloading strategy accordingly. During the execution of the strategy, high-confidence features are decoded locally; features that need to be enhanced are unloaded to the RSU side, and feature optimization and reconstruction are performed through the deep model. Finally, the RSU transmits the enhanced feature sequence back to the vehicle side, performs multi-source fusion with local features, and outputs the final state of environmental perception. See the specific model structure diagram. Figure 2 , diagram of multimodal data fusion model based on BEV fusion.

[0081] 1) Quality Assessment Module

[0082] In the CPPLO model, D-CAV perceives fusion information Contains self-perception information P i , R-CAV perception information P j→i , perception fusion information It can be expressed as:

[0083]

[0084] Definition O u Contains the perception fusion information P i Related status information of the occluder u in the middle. u The elements of the blind spot entity type ty u (vehicles, pedestrians and road construction areas), the spatial coordinates of the obstruction in the vehicle coordinate system loc u , perception distance d u Required computing resources GPU req and 3D detection bounding box feature box u Equivalent to isomorphic information, it can be expressed as:

[0085] O u ={ty u ,loc u ,d u ,GPU req ,box u} (18)

[0086] Among them, loc u =(x u ,y u ) is the two-dimensional coordinate information of the occluder u; box u =(l u ,h u ,w u ) is the 3D detection bounding box of the occluder u. CAV is achieved by ty u ,loc u Get the type and location information of the occluder u, according to d u ,GPU req ,box u And the available computing resources GPU that this vehicle can provide avail To evaluate the quality of the perceptual information of the occluder u.

[0087] In order to establish a quantifiable perceptual information quality performance evaluation system, the present invention defines ROI perceptual fusion quality as a multi-dimensional joint optimization function. This quantitative indicator integrates the following evaluation dimensions: allocation of computing resources, observation distance, spatial geometric consistency, and perceptual confidence, namely, uThe description is defined as:

[0088]

[0089] where ω c ,ω d ,ω r Represent the weights of perception confidence, observation distance and relative computing power respectively, and ω c +ω d +ω r =1, the weight value is different in different scenarios. u is the perceived confidence, which can be obtained by Get, where conf max is the maximum response value of the feature map, reflecting the confidence scores of different occluders. γ is the distance attenuation coefficient.

[0090] Q local It directly reflects the reliability of the perception information. Specifically, the closer the observation distance is, the more complete the CAV perception information is. local The higher the quality score, the more computational resources CAV needs to process occlusions, and the more complex the feature expression. local The higher the quality score of Q, the higher the perception confidence, the larger the three-dimensional volume of the occluder u, and the higher the decision priority the system needs to assign. local The higher the quality score, the more likely it is to be processed locally. Ultimately, the quality score serves as the basis for offloading decisions; a higher value increases the likelihood of local computation. Generally speaking, higher perceived data credibility, closer distance to the perceived target, and lower required computing power favor local processing, leading to faster responses. Conversely, offloading to the RSU is more likely.

[0091] Taking into account the perception fusion information from multiple responding vehicles, where there may be multiple repeated descriptions of the same occluder. Therefore, D-CAV i needs to u ,loc u Perception fusion information Classify and select a description for each occluder u for fusion. Assume There are U occluders in , so formula (17) can be rewritten as Each of the O u There are K descriptions, namely Finally, D-CAV i from each O u Choose one to make Q local The description with the largest quality score is used to achieve perceptual fusion, thereby fusing ROI to achieve full-domain BEV.

[0092] Finally, the offloading decision is made dynamically based on the above D-CAV i’s perception information quality evaluation results:

[0093]

[0094] Among them, τ high is the upper limit of the perceived quality score, τ low is the lower limit of the perceived quality score.

[0095] In the perceptual quality driven offloading decision framework, the upper threshold τ is defined high and the lower bound τ low As the quality fraction Q local When the perceptual quality is greater than the threshold τ high When the perception quality is less than the threshold τ, the local full computing mode is executed and feature decoding and fusion are completed in the on-board computing unit; low When the full uninstallation policy is enabled, the system will uninstall the Perform feature enhancement calculations and send the enhancement results back to the vehicle for final fusion; in other cases, a dynamic proportional unloading mechanism is implemented, and the unloading weight changes with Q local Monotonically decreasing, thereby synergistically optimizing latency and fusion quality.

[0096] 2) Result fusion module

[0097] Based on the offloading decision made by the perceptual quality evaluation, the CAV end will eventually have local calculation results and RSU calculation results. Therefore, the result fusion strategy is divided into fully local result fusion, fully offloaded result fusion, and partially offloaded result fusion:

[0098]

[0099] in, is the result of local calculation, is the result of RSU calculation. θ is the scaling factor,

[0100] Q RSU is the quality evaluation score of the RSU calculation result, which is used to evaluate the calculation status of the RSU and can be expressed as:

[0101]

[0102] Among them, conf avg is the average confidence of RSU. RSU is the distance between RSU and D-CAV i, FR norm is the normalized refresh rate, 1-L RSU is the load factor, (1-L RSU )∈[0,1].

[0103] Specifically, the reliability of data fusion tasks depends on multiple factors, including data quality, spatiotemporal characteristics, and system status. The average confidence level reflects the accuracy of the fusion results, emphasizing the dependence of information reliability on data accuracy. The distance attenuation term addresses spatial factors: closer distances correspond to stronger signals and higher reliability. The refresh rate addresses temporal factors: higher update frequencies correspond to fresher data and improved real-time performance. Furthermore, the lower the load on the RSU, the higher the reliability, avoiding performance degradation under high RSU load.

[0104] 4. Modeling the resource allocation problem of multi-source perception data fusion

[0105] Based on the CPPLO model, multi-source perception fusion is considered. In order to improve the accuracy and robustness of perception fusion and minimize the average perception fusion delay of the system, the multi-source perception fusion and resource allocation optimization problem is modeled as follows:

[0106]

[0107] In the system model, T represents the total number of time slots, and constraints C1 to C3 are used to limit the value range of the decision variables. Constraint C4 requires that the data block size cannot exceed the original data size. Constraint C5 requires that the total computing resources allocated by RSU m to all its associated CAVs do not exceed its maximum available computing capacity to ensure the feasibility of resource allocation. Constraint C6 ensures that the ROI perception information collected by R-CAV i meets the latency requirements and is transmitted within the specified time τ, thereby maintaining the real-time performance of the system. Therefore, to achieve optimal, real-time ROI perception fusion quality under the CPPLO model, it is necessary to minimize the ROI perception fusion latency.

[0108] 5. Design of resource scheduling algorithm based on HDDPG

[0109] Since the objective function and constraints of the ROI fusion resource allocation optimization problem P1 are non-convex, traditional gradient descent algorithms fall into local optimality and it is difficult to ensure the global optimality of the solution. Therefore, this paper introduces a reinforcement learning (RL) method based on hybrid action space, and models the problem as PAMDPs, denoted as Among them, the state space S, action space A and reward function R are defined as follows:

[0110] 1) State space: In time slot t, the state is defined as Contains the available computing resources of CAV and RSU and the perception fusion information of each CAV, the state s of time slot t t ∈S.

[0111] 2) Action space: In time slot t, the action space Including the computing resources allocated by RSU m to CAV i, the offloading decision That is, the computation task at CAV i is offloaded to RSU m at time slot t. And the benchmark block parameters Overhead coefficient k t .

[0112] 3) Reward function:

[0113]

[0114] Through the DRL framework, RSU aims to maximize the long-term cumulative reward, so the reward function is:

[0115]

[0116] Use the discount factor γ∈[0,1] to balance short-term and long-term benefits. Strategy π θ The mapping from state to action is learned through trial and error mechanism to obtain the final optimal strategy:

[0117]

[0118] Based on the above Markov decision process, the resource allocation algorithm based on HDDPG proposed in this invention solves the problem as follows: Figure 3 As shown in Figure 2, the HDDPG algorithm is equipped with a three-branch Actor network and a global Critic network. Discrete Strategy Network Learning Strategy To select the discrete action offloading strategy, the benchmark block overhead and the overhead coefficient m, B0, k, the continuous strategy network learning strategy The resource f is calculated by selecting the continuous action RSU of resource allocation. The discrete policy network outputs 3i values ​​V={m1,...,m i ,B 0,1 ,...,B 0,i ,k1,...,k i}, obtain the discrete action v to be taken by random sampling of the sigmod(m) distribution. The continuous policy network generates a strategy for continuous actions by calculating the mean and variance of the Gaussian distribution of each parameter Therefore, the discrete action v selected by the discrete policy network and the continuous action f selected by the continuous policy network together form a hybrid action

[0119] The calculation formulas for discrete action v and continuous action f are as follows:

[0120]

[0121] Where S represents the state space; θ d and θc Represent the discrete strategy network parameters and the continuous strategy network parameters, respectively, which together constitute the current network and target network θ of the Actor network π and θ π' .

[0122] The HDDPG algorithm proposed in this paper can be regarded as a combination of the Actor-Critic algorithm based on the hybrid action space and the DQN algorithm. The action value function Q(·) is defined to approximate the long-term cumulative reward Use Q(s t ,a t ) to evaluate and improve the optimal strategy of the Actor network. The action value function can be expressed as:

[0123]

[0124] From this, we can see that the action value function of the Actor's current network can be given by the Bellman equation:

[0125]

[0126] Among them, R t (s t ,a t ) indicates that the agent is in a given state s t Next take action a t instant rewards; Indicates the Q-value estimation at the next moment based on the strategy π.

[0127] At the same time, the current network of the Critic in the HDDPG algorithm is the network in the DQN model, and the Q value is an evaluation of the strategy of the Actor's current network. The Q value function of the Critic network is:

[0128]

[0129] Similar to Q learning and DQN, the Critic network in the HDDPG algorithm uses TD error to update the Q value. Finally, Q(s t ,a t ) is updated by:

[0130] Q q (s t+1 ,a t+1 )=Q q (s t ,a t )+ηδ t (31)

[0131] Among them, η represents the learning rate; δ t is the TD error, which can be expressed as:

[0132]

[0133] Where γ represents the discount factor and Q(s',a') represents the historical Q value. Theoretically, the optimal Q value can be obtained by iterating through Equation (31) to |Q(s',a')-Q(s,a)|<ξ, where ξ is a very small positive number. Based on the DQN loss function, the loss in the current state is:

[0134]

[0135] Among them, y t Indicates the Q value obtained by the current network of Critic, which can be expressed as:

[0136] y t =R t (s t ,a t )+γQ(s t+1 ,π(s t+1 )|θ q ) (34)

[0137] Among them, π(·) and Q(·) represent the network parameters of the Actor current network and the Critic current network, respectively.

[0138] According to the loss function Loss(θ q ) to update the current network parameters θ of Critic q Finally, using the gradient ascent algorithm, the Actor's current network updates the Actor's current network parameters θ according to the policy gradient π , the loss gradient of the network is:

[0139]

[0140] RSU collects environmental information of the Internet of Vehicles (IoVs) scene, stores it in the experience replay pool, and uses the new-to-old strategy to keep the samples fresh. π' and Critic target network θ q 'Get y t Therefore, y in formula (34) t Can be changed to:

[0141] y t =R t (s t ,a t )+γQ'(s t+1 ,π'(s t+1 )|θ q') (36)

[0142] Finally, the HDDPG algorithm updates the Actor's current network and the Critic's current parameters using the policy gradient algorithm. The following formula is then used to update the target network's parameters:

[0143] θ π' ←τθ π +(1-τ)θ π' (37a)

[0144] θ q' ←τθ q +(1-τ)θ q' (37b) Where τ is the soft update parameter, which takes the weighted average of the current network parameters and the new target network parameters and assigns them to the target network.

[0145] In addition, in the deterministic strategy π(s t |θ π ) adds a random noise The strategy of the HDDPG algorithm is as follows:

[0146]

[0147] In summary, the resource allocation algorithm process based on HDDPG is designed as follows: first, the Actor-Critic network and its target network parameters are randomly initialized, and an empty experience replay pool is created; second, actions are generated through exploration strategies and environmental interactions are performed, and the generated state transition data, including state, action, reward and subsequent state, are stored in the experience pool; then, small batches of samples are sampled from the experience pool, and the Critic network parameters (minimizing the temporal difference error) and the Actor network parameters (maximizing the expected cumulative reward) are updated in sequence, and the target network parameters are synchronized with a soft update mechanism; finally, by optimizing the resource allocation strategy in the continuous-discrete hybrid action space, the system delay is minimized and the reward function converges to the optimal solution.

[0148] 6. System Flowchart

[0149] Figure 4 The figure shows the execution flow chart of the multi-source perception data fusion and vehicle-road resource allocation method based on the BEV fusion framework in a complex traffic environment. The specific steps are as follows:

[0150] S601-S602: System initialization, CAV activates the multimodal sensor array and completes parameter calibration, simultaneously initializes the on-board computing unit and loads the lightweight perception model instance. RSU deploys the scalable computing resource pool, initializes the dynamic resource allocator core module, and presets the maximum allocable resource threshold (f in Equation (23) m，max );

[0151] S603: Establish vehicle-to-vehicle (CAV-to-CAV) and vehicle-to-road (CAV-to-RSU) communication links, configure OFDM parameters, and allocate channel resources;

[0152] S604: The CAV receives the benchmark block parameters and system overhead coefficients generated by the previous iteration training from the RSU through the vehicle-road cooperative interface, and dynamically maintains the spatiotemporal consistency of the global BEV semantic map based on the parameter set;

[0153] S605: When the D-CAV local perception confidence is lower than the threshold, it is perceived that there is an occluder in the ROI in the BEV image;

[0154] S606: The D-CAV encapsulates the required information, such as ROI range, location, timestamp, etc., and broadcasts it to the neighboring CAVs via the V2V network to act as the R-CAV.

[0155] S607-S609: The R-CAV responds to the request command sent by the D-CAV, first collecting raw perception data in the form of LiDAR point clouds and images, then performing feature compression and ROI extraction on the raw data in a pipelined and parallel manner, and finally sending a data packet containing the occlusion description to the D-CAV;

[0156] S610-S611: D-CAV receives multimodal data from multi-vehicle collaborative perception and performs data preprocessing steps such as decapsulation and spatiotemporal alignment. After extracting features from image and LiDAR point cloud data, it converts their perspectives into a unified BEV space to generate BEV features.

[0157] S612: The D-CAV calculates the perceived data quality score based on the obtained BEV features and makes a dynamic offloading decision. If the score is greater than a threshold, S613 is executed; otherwise, S614 is executed.

[0158] S613: Execute local lightweight BEV model decoding of D-CAV, temporarily store the decoded data, and wait for execution of S621;

[0159] S614: The D-CAV initiates an offloading request to the nearby RSU and sends the offloading rate;

[0160] S615-S618: Execute the HDDPG resource allocation process. After the RSU triggers the HDDPG algorithm, it first generates and executes resource allocation decisions in the continuous-discrete hybrid action space based on the current policy network. Secondly, it stores the state transition tuples generated by environmental interactions in the experience replay pool. Then, it samples small batches of experience data and updates the Actor network parameters. After completing the main network update, it uses a soft update mechanism to synchronize the target network parameters. Finally, it generates a resource allocation decision matrix and feeds it back to the system controller to achieve dynamic resource optimization configuration. The resource allocation results are then transmitted back via OFDM.

[0161] S619-S620: Based on a dynamic resource allocation strategy, the D-CAV offloads data fusion computing tasks to the RSU, which then performs multi-level feature enhancement of the sensor data. To optimize communication efficiency, the enhanced feature tensors are serialized and encoded, and the serialized feature vectors are ultimately transmitted back to the D-CAV via a low-latency vehicle-to-road link.

[0162] S621: The D-CAV receives the serialized results from the RSU and performs spatiotemporal alignment of the local computation results with the edge computation results. Finally, it updates the ROI fusion model of the global BEV space. In addition, a Kalman filter is used to eliminate fusion noise and improve model robustness.

[0163] S622: The RSU collects the interaction data (state, action, reward) between the CAV and the environment and stores it in the experience replay pool for HDDPG training. At the same time, it soft-updates the Critic and Actor target network parameters to ensure training stability. The parameters such as the training benchmark block parameters and overhead coefficients of this stage are also transmitted to the CAV for use in the next mission.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-source perception data fusion and vehicle-road resource allocation method based on BEV, characterized by: Including the following step: S1: Build a vehicle-road dual-layer communication network architecture, which includes: The mobile layer consists of intelligent connected vehicles (CAVs) The edge layer consists of roadside units (RSUs); S2: Establishing a collaborative parallel perception delay optimization (CPPLO) model based on the vehicle-road dual-layer network architecture; S3: Construct a multimodal data fusion model based on BEV fusion; S4: Establish a multi-source perception data fusion resource allocation problem with the goal of minimizing the global perception fusion delay; S5: A resource scheduling algorithm based on hybrid action space deep deterministic policy gradient (HDDPG) is used to solve the resource allocation problem.

2. The BEV-based multi-source perception data fusion and vehicle-road resource allocation method according to claim 1 is characterized by: The S2 includes: When the demand vehicle D-CAV senses a blind spot, it generates a collaborative perception demand instruction and broadcasts it to the adjacent response vehicle R-CAV; R-CAV performs parallel execution on the raw perception information: (a) Perceptual compression: The calculation formula is (b) Region of interest (ROI) extraction: The calculation formula is: Where: D i is the amount of original perception data, To perceive the compression calculation intensity, is the compression power allocation ratio, f i t is the computational resource of CAVi in time slot t, η i→j is the ROI extraction ratio, Indicates the amount of compressed data, M ex Calculate intensity for ROI extraction; The ROI information is transmitted to D-CAV, and the transmission delay is: in Indicates the amount of ROI data, Indicates the vehicle-to-vehicle communication rate.

3. The BEV-based multi-source perception data fusion and vehicle-road resource allocation method according to claim 1 is characterized by: The quality assessment in S3 includes: Calculate multi-dimensional joint indicators: Where: s u represents the perceived confidence, d u Indicates the observation distance, GPU avail Indicates available computing resources, GPU req represents the required computing resources, ω c 、ω d 、ω r is the weight coefficient and satisfies ω c +ω d +ω r =1, γ represents the distance attenuation coefficient; Based on Q local Generate uninstall decisions:

4. The BEV-based multi-source perception data fusion and vehicle-road resource allocation method according to claim 1 is characterized by: The S5 includes: The resource allocation problem is modeled as a parameterized action space Markov decision process PAMDPs; Define the state space: Define the action space: in: Indicates the computing resources allocated by RSU, represents the uninstall decision, represents the benchmark block size, k t represents the overhead coefficient; Define the reward function:

5. A vehicle-road cooperative perception system implementing the method according to any one of claims 1 to 4, characterized in that: include: Mobile layer: contains multiple CAVs, each equipped with: (1) Multimodal sensor array (2) Embedded edge computing unit (3) Lightweight BEV fusion model; Edge layer: contains multiple RSUs, each RSU is configured as follows: (1) High-performance edge server (2) Complete BEV fusion model; Communication module: Orthogonal frequency division multiplexing (OFDM) mechanism is used to realize vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications.

6. The vehicle-road cooperative perception system according to claim 5, characterized in that: The CAV includes: a D-CAV module: configured to perform collaborative sensing demand broadcasting and result fusion; R-CAV module: used for pipelined parallel execution of perceptual compression and ROI extraction; Data block controller: Calculate the data block size according to the following formula: in Indicates the compression power, Indicates ROI extraction computing power.

7. The vehicle-road cooperative perception system according to claim 5, characterized in that: The RSU includes: a resource scheduler: executing the HDDPG algorithm to generate a resource allocation strategy; Feature Enhancer: performs feature enhancement calculations, the calculation formula is: in represents the amount of BEV characteristic data, Indicates increased computational intensity.

Images