A communication perception and edge computing integrated system for vehicle-road cooperation

Abstract

The application discloses a kind of communication perception and edge computing integration systems for vehicle-road cooperation, including perception and edge computing unit, road side communication unit, intelligent vehicle terminal and cloud platform, wherein perception and edge computing unit is composed of millimeter wave radar, camera and edge computing module;Road side communication unit is composed of control module, communication module and positioning module;Intelligent vehicle terminal obtains the category of nearby traffic target, distance information, judges whether it needs to brake or continue to drive operation in combination with its own traffic state;Cloud platform collects the road condition information uploaded by road side unit for analysis and processing, prints the driving vehicle and traffic target information accessed to cloud map, and comprehensively judges through traffic, speed, position, to realize macro traffic control.The application has the advantages of simple structure, convenient deployment, strong cooperation, suitable for various scenes, easy to expand, etc.In different scenes, traffic target information can be accurately obtained, and road traffic safety is improved.

Description

Technical Field

[0001] This invention relates to the field of vehicle-road cooperative technology, specifically to an integrated communication, sensing, and edge computing system for vehicle-road cooperative systems. Background Technology

[0002] Vehicle-to-everything (V2X) systems have different perception and communication requirements for different application scenarios. Because single sensors have perception limitations and cannot meet the precise perception needs of certain scenarios, existing multi-sensor fusion perception is mostly applied to single-vehicle intelligence. While this compensates for the perception capabilities of single sensors to some extent, it is limited to the perception blind spots of a single vehicle and cannot perform perception beyond the vehicle's line of sight. To compensate for these blind spots and improve the safety and reliability of vehicle operation, enhancing the perception capabilities of the roadside is particularly important. Summary of the Invention

[0003] The purpose of this invention is to provide an integrated communication, sensing, and edge computing system for vehicle-road cooperation, in order to solve the problems mentioned in the background art.

[0004] The technical solution for achieving the objective of this invention is: an integrated communication, sensing, and edge computing system for vehicle-road cooperation, comprising: a sensing and edge computing unit, a roadside communication unit, an intelligent vehicle terminal, and a cloud platform, wherein:

[0005] The perception and edge computing unit consists of millimeter-wave radar, camera and edge computing module. The millimeter-wave radar and camera acquire roadside traffic target data. The edge computing module receives the collected traffic target information through wired means, and performs real-time detection, identification and classification of strong and weak traffic participants, and obtains the type of traffic target within the roadside range and the relative distance information between the target and the driver vehicle.

[0006] The roadside communication unit consists of a control module, a communication module, and a positioning module. The communication module includes two types: an LTE communication module and a PC5 communication module, supporting V2X-based cellular communication and direct communication modes respectively, enabling communication interconnection with intelligent vehicle terminals within the roadside area. The roadside communication unit connects to the edge computing module in the sensing and edge computing unit via WiFi. The control module controls the communication module to send data processed by the edge computing module to the intelligent vehicle terminal via MQTT or PC5, switching communication modes in different application scenarios. Simultaneously, the intelligent vehicle terminal sends basic vehicle driving information to the roadside communication unit. The roadside communication unit uploads the data processed by the edge computing module and the basic vehicle driving information to the cloud platform via HTTP through the communication module. The roadside communication unit determines its latitude and longitude information through the positioning module. By uploading the latitude and longitude information to the cloud platform and sending it to the intelligent vehicle terminal, the roadside communication unit completes the printing of information on the cloud map and achieves the exchange of location information with the intelligent vehicle terminal.

[0007] The intelligent vehicle terminal obtains information on the type and distance of nearby traffic targets, and determines whether to brake to avoid danger or continue driving based on its own traffic conditions.

[0008] The cloud platform collects and analyzes road condition information uploaded by roadside units, prints the connected driving vehicle and traffic target information onto the cloud map, and makes a comprehensive judgment based on traffic flow, vehicle speed, and location to achieve macro-level traffic control.

[0009] Furthermore, in the perception and edge computing unit, the camera is connected to the edge computing module via a USB interface, the millimeter-wave radar is connected to the bus analyzer via a CAN bus, and the CAN bus analyzer is connected to the edge computing module via a CAN bus-to-USB interface. The millimeter-wave radar and the camera respectively acquire traffic target data and transmit the data to the edge computing module in real time via the USB interface.

[0010] Furthermore, the edge computing module, developed based on the Jetson Xavier NX suite, fuses point cloud data acquired by millimeter-wave radar and image data acquired by cameras using a radar-visual fusion target detection method to achieve real-time detection of traffic targets. The specific method is as follows:

[0011] The radar-visual fusion target detection method includes radar data preprocessing, radar-visual data spatiotemporal alignment, radar-visual data fusion, and fusion model detection steps. Radar data preprocessing and radar-visual data spatiotemporal alignment belong to the pre-fusion part, responsible for processing abnormal radar data and unifying the spatiotemporal coordinate systems of millimeter-wave radar and visual data. Radar-visual data fusion and fusion model detection belong to the post-fusion part, responsible for projecting radar data containing effective features onto the vertical plane of the image to generate an enhanced image containing radar feature information, which is finally fed into the fusion model for target detection and recognition. The specific process of radar-visual fusion is as follows:

[0012] The first step is to remove abnormal data generated by millimeter-wave radar: preprocess the raw data collected by millimeter-wave radar to remove empty targets, stationary targets, and false targets in order to avoid affecting the accuracy of fused target detection.

[0013] The second step is to ensure that the camera and radar maintain a consistent perspective: complete the fusion of the spatial coordinate systems of the camera and the millimeter-wave radar, and map the millimeter-wave radar data to the pixel coordinate system where the visual data is located. The mapping order is the millimeter-wave radar coordinate system, the three-dimensional world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The transformation from the millimeter-wave radar coordinate system to the two-dimensional pixel coordinate system is performed in the above order to make the radar and image data spatially consistent.

[0014] The third step is to ensure the real-time performance of the system: complete the time synchronization between the camera and the millimeter-wave radar, that is, to jointly calibrate the time frame rate of the two sensors according to the sampling frequency, calculate the common multiple of the time interval between the two samplings, and ensure synchronization in time.

[0015] The fourth step is to complete the deep fusion of the two types of sensor data: the millimeter-wave radar and visual data are fused, that is, the radar data containing effective features is projected on the vertical plane of the image, thereby expanding the dimension of the image data with RGB three-dimensional channels, adding a one-dimensional radar data channel, and generating an enhanced image containing radar feature information, which is used as the input of the fusion target detection model. The selected radar feature information is the millimeter-wave radar cross section (RCS) and distance information.

[0016] The fifth step involves target recognition using fused data from the camera and millimeter-wave radar: The fusion detection model is built on RetinaNet, and its backbone network includes a millimeter-wave radar data path and a fusion network output data path, consisting of 7 max-pooling layers and 5 VGG blocks. The input to the fusion detection model is denoted as C0, which is the initial enhanced image generated by fusing millimeter-wave radar and visual data once. The first 5 layers consist of 5 max-pooling layers and 5 VGG blocks. Within each layer, the millimeter-wave radar data is processed by the max-pooling layer at a corresponding ratio. After processing, the data is merged with the fused data output from the previous layer into the VGG block-based system. Fusion is performed on the data path of the fusion network. After fusion at this layer, the millimeter-wave radar data continues to enter the next layer's max pooling layer along the path to change its size. It is then merged with the fused data output from the previous layer into the data path of the fusion network composed of VGG blocks for fusion. The millimeter-wave radar data and C0 pass through 5 max pooling layers and VGG blocks respectively, that is, 5 fusion operations are performed. The output fused data is denoted as C1, C2, C3, C4 and C5. The remaining 2 max pooling layers form the last 2 layers. The millimeter-wave radar passes through the last two max pooling layers in sequence. At this time, the fused data is no longer merged with it. The output millimeter-wave radar data are C6 and C7 respectively.

[0017] Feature pyramids are introduced into the deep network of the fusion detection model, denoted as P3, P4, P5, P6 and P7 respectively. C3, C4, C5, C6 and C7 are fed into feature pyramids P3, P4, P5, P6 and P7 in sequence. At each level of the feature pyramid, the corresponding millimeter-wave radar data are merged to generate feature outputs denoted as X3, X4, X5, X6 and X7 respectively.

[0018] The target classification subnetwork and the bounding box regression subnetwork process the outputs X3, X4, X5, X6 and X7 of the feature pyramid, and finally generate the corresponding classification and regression results.

[0019] Furthermore, in the roadside communication unit, the control module is developed based on Raspberry Pi 4B, and the communication module includes two sub-modules: LTE and PC5. Corresponding to the two communication modes of the roadside communication unit, the Quectel EC20-LTE module and the Chenxin CX7101-PC5 module are used respectively.

[0020] By default, the control module of the roadside communication unit uploads the traffic target recognition results obtained by the edge computing module to the cloud platform and distributes them to the vehicle terminal App via a 4G cellular network, and the information transmission method is MQTT. In scenarios without network coverage, the communication method of the roadside communication unit will automatically switch to the direct connection mode based on the PC5 module.

[0021] Furthermore, the basic driving information of the vehicle is a custom message that includes some BSM message features. The message body contains the vehicle ID, vehicle position, vehicle speed, steering angle, and vehicle status.

[0022] Furthermore, the camera, millimeter-wave radar, edge computing module, and battery are placed in the same device box, maintaining the same detection angle for the camera and radar. The battery powers all three components through a step-down converter and is installed on the roadside. The edge computing module is connected to the camera and millimeter-wave radar via a wired connection. The roadside communication unit transmits information to the edge computing module via WiFi and is installed above the traffic light pole. The intelligent vehicle terminal app is mounted on a mobile phone or tablet and placed in the vehicle, allowing the driver to view the roadside information in real time.

[0023] A method for integrating communication sensing and edge computing for vehicle-road cooperation is provided, which realizes integrated communication sensing based on vehicle-road cooperation based on the aforementioned integrated communication sensing and edge computing system for vehicle-road cooperation.

[0024] Compared with existing technologies, the significant advantages of this invention are: 1) The sensing and edge computing units are separated from the roadside communication units, allowing multiple sets of sensing and edge computing units to be deployed across multiple lanes at intersections, sharing the same roadside communication unit for information uploading and downloading, reducing equipment redundancy and effectively lowering costs. 2) The system integrates various units, using the vehicle as the information distribution node on the roadside and the roadside as the communication and sensing center for the vehicle. Through the interconnection of various units, integrated communication, sensing, and edge computing can be achieved in different scenarios. 3) Compared with single-vehicle fusion sensing, this system adopts a roadside fusion sensing scheme, using millimeter-wave radar and cameras as sensing units, and employing a multi-level fusion feature-level radar-visual fusion target detection method as the core sensing algorithm. This not only compensates for the blind spots of single-vehicle perception beyond line-of-sight but also meets the sensing needs of various roadside scenarios.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of the integrated communication, sensing and edge computing system for vehicle-road cooperation of the present invention.

[0027] Figure 2 This is a schematic diagram of the workflow of the integrated communication, sensing and edge computing system for vehicle-road cooperation of the present invention.

[0028] Figure 3 This is a schematic diagram of the network structure of the fusion detection model of the integrated communication sensing and edge computing system for vehicle-road cooperation according to the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] To enhance perception capabilities within the roadside area for vehicle-road cooperative systems, it is necessary to accurately extract data from various sensors across different spatiotemporal dimensions, fuse the extracted data, and then detect and identify the fused data. The perception results are then transmitted to vehicles and the cloud within a certain time latency range via V2X communication technology, thereby enabling perception and communication in various traffic scenarios. This invention uses cameras and millimeter-wave radar as roadside sensors, integrating them with edge computing modules, roadside communication units, intelligent vehicle terminals, and cloud platforms into an integrated system for vehicle-road cooperation. This system aims to achieve communication, perception, and edge computing between the vehicle and the roadside. The vehicle acts as an information distribution node on the roadside, while the roadside serves as the communication and perception center for the vehicle. Through the interconnection of various units, communication, perception, and early warning can be achieved in different scenarios.

[0031] like Figure 1 As shown, an integrated method and system for communication, perception, and edge computing based on vehicle-road cooperation and radar-visual fusion includes the following components: a perception and edge computing unit, a roadside communication unit, an intelligent vehicle-mounted terminal, and a cloud platform. The perception and edge computing unit is used to collect raw data from video and millimeter-wave radar, and to perform real-time detection and identification of the data; the roadside communication unit is used for uploading and downloading data; the intelligent vehicle-mounted terminal is used to receive roadside information and upload vehicle driving information; the cloud platform collects the roadside information and vehicle driving information for macro-level traffic control.

[0032] The aforementioned units are as follows: the sensing and edge computing unit is installed on the roadside, facing the traffic lights; the roadside communication unit transmits information with the sensing and edge computing unit via WiFi and is installed above the traffic light pole; the intelligent vehicle terminal app is mounted on a mobile phone or tablet and placed in the vehicle, allowing the driver to view the roadside information in real time.

[0033] The composition and design of each unit will be described in detail below.

[0034] (I) Sensing and Edge Computing Unit

[0035] The perception and edge computing unit includes a millimeter-wave radar, a camera, and an edge computing module. The camera connects to the edge computing module via a USB interface, the millimeter-wave radar connects to a bus analyzer via a CAN bus, and the CAN bus analyzer connects to the edge computing module via a CAN bus-to-USB interface. Traffic target data is acquired by the millimeter-wave radar and the camera, and the data is transmitted to the edge computing module in real time via the USB interface. In this invention, the millimeter-wave radar used is the Continental 77GHz ARS408, the camera is a USB monocular camera, and the edge computing module uses the Jetson Xavier NX kit. The edge computing module utilizes the fusion perception model deployed on the Jetson Xavier NX development kit to fuse the video and radar raw data acquired by the perception and edge computing unit, and finally performs real-time detection and recognition to obtain traffic target type and distance information. The radar-visual fusion target detection process is briefly described below.

[0036] The first step, in order to remove abnormal data generated by millimeter-wave radar, is to preprocess the raw data collected by millimeter-wave radar to remove empty targets, stationary targets, and false targets, so as to avoid affecting the accuracy of fused target detection.

[0037] The second step, in order to keep the camera and radar perspectives consistent, is to first complete the fusion of the spatial coordinate systems of the camera and the millimeter-wave radar on the edge computing module. The millimeter-wave radar data is mapped to the pixel coordinate system where the visual data is located. The mapping order is the millimeter-wave radar coordinate system, the three-dimensional world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The transformation from the millimeter-wave radar coordinate system to the two-dimensional pixel coordinate system is performed in the above order so that the radar and image data can achieve spatial consistency.

[0038] The third step, in order to ensure the real-time performance of the system, is to complete the time synchronization between the camera and the millimeter-wave radar on the edge computing module. That is, to perform joint calibration of the time frame rate of the two sensors according to the sampling frequency and calculate the common multiple of the time interval between the two samplings. In this invention, 0.66s is used as the timestamp to ensure synchronization in time.

[0039] The fourth step, in order to achieve deep fusion of the two types of sensor data, is to fuse millimeter-wave radar and visual data. The main operation is to project the radar data containing effective features onto the vertical plane of the image, thereby expanding the dimensions of the image data with RGB three-dimensional channels, adding a one-dimensional radar data channel, and generating an enhanced image containing radar feature information as input to the fusion target detection model. The radar feature information selected in this invention are the millimeter-wave radar cross section (RCS) and range information.

[0040] The fifth step involves building a deep learning model to perform target recognition using fused data from the camera and millimeter-wave radar. The radar-visual fusion target detection model of this invention is based on RetinaNet and includes a Backbone module, a feature pyramid module, and a target classification and bounding box regression module. The Backbone module uses a VGG16 network, while the feature pyramid module and the target classification and bounding box regression module are consistent with RetinaNet. The network input is the enhanced image generated after fusing the millimeter-wave radar and visual data in the fourth step, and the output is the bounding box regression coordinates, the detected traffic target category, and the classification score.

[0041] The network input is denoted as C0, which is the initial enhanced image generated by fusing millimeter-wave radar and visual data once. The left side of the backbone network of the fusion detection model is the millimeter-wave radar data path, and the right side is the fusion network output data path, consisting of 7 max-pooling layers on the left and 5 VGG blocks on the right. Within each layer, the millimeter-wave radar data is processed by the max-pooling layer at a corresponding ratio. After processing, it is merged with the fused data output from the previous layer into the main pipeline on the right side of the fusion network, which consists of VGG blocks. After fusion at this layer, the millimeter-wave radar data continues along the left path into the next level's max-pooling layer to change its size. The fused data continues along the right path into the next level, and after the left-side millimeter-wave radar data completes max-pooling, it is fused with it in the main pipeline on the right. The millimeter-wave radar data and the initial network input C0 each pass through 5 max-pooling layers and VGG blocks, i.e., 5 fusion operations are performed. The output fused data are denoted as C1, C2, C3, C4, and C5. The millimeter-wave radar then passes through two more max-pooling layers. At this point, the fused data is no longer merged with the radar data, and the output millimeter-wave radar data are C6 and C7, respectively. In the fusion detection model, the output data of the fusion network is fused with the millimeter-wave radar data at each layer, and logistic regression is used to locate the optimal fusion layer. Finally, the network learns to determine the fusion weights for different layers.

[0042] Furthermore, a feature pyramid is introduced into the deep network of the fusion detection model, denoted as P3, P4, P5, P6, and P7 in the network structure. C3, C4, C5, C6, and C7 are sequentially fed into the feature pyramid, and the corresponding radar channels are merged at each level of the feature pyramid to generate feature outputs denoted as X3, X4, X5, X6, and X7. Finally, the outputs X3, X4, X5, X6, and X7 of the feature pyramid are processed by the target classification subnetwork and the bounding box regression subnetwork to generate the corresponding classification and regression results.

[0043] Verification has shown that the radar-visual fusion recognition model in this invention has high accuracy and robustness.

[0044] (ii) Roadside communication unit

[0045] The roadside communication unit includes a control module, a communication module, and a positioning module. The communication module and positioning module are connected to the control module via USB interfaces. In this invention, the control module is developed based on a Raspberry Pi 4B, the communication module uses a Quectel EC20-LTE module and a Chenxin CX7101-PC5 module, and the positioning module uses an ATGM332D dual-mode positioning module. The positioning module obtains the location information of the roadside unit. After the control module receives the processing results from the edge computing module, it sends the processed results to the intelligent vehicle terminal App via the communication module using MQTT or PC5. Simultaneously, it uploads the processing results and the location information of the roadside communication unit to the cloud platform via HTTP protocol.

[0046] (III) Intelligent Vehicle Terminal

[0047] The intelligent vehicle terminal app, installed on a mobile phone or tablet, can obtain the processing results from the roadside communication unit in real time. Drivers can use the intelligent vehicle terminal to obtain information on the type and distance of nearby traffic targets, determining whether to brake to avoid a hazard or continue driving. Simultaneously, the intelligent vehicle terminal app sends basic driving information of the driver's vehicle to the roadside communication unit, including vehicle ID, vehicle position, vehicle speed, azimuth angle, and vehicle status.

[0048] (iv) Cloud Platform

[0049] The cloud platform collects and analyzes road condition information uploaded by roadside units, prints the connected driving vehicle and traffic target information onto the cloud map, and makes a comprehensive judgment based on information such as traffic flow, vehicle speed, and location to carry out macro-level traffic control.

[0050] In summary, the integrated communication, perception, and edge computing system for vehicle-road cooperation uses millimeter-wave radar and cameras as sensors and is implemented based on edge computing modules and embedded devices. It provides functions such as radar and video raw data acquisition and preprocessing; spatiotemporal alignment of radar and video data; traffic target detection; data generation; data distribution and uploading; vehicle terminal information display; and cloud platform information display.

[0051] Based on this, the present invention also proposes an integrated communication perception and edge computing method for vehicle-road cooperation. First, the millimeter-wave radar and camera of the perception and edge computing unit collect traffic target data. The edge computing module acquires the raw radar and video data, performs data preprocessing, spatiotemporal synchronization, data fusion, target detection and recognition through a radar-visual fusion target detection method, and then sends the processed results to the roadside communication unit. The roadside communication unit sends the processing results to the intelligent vehicle terminal App via MQTT or PC5, and simultaneously uploads the processing results and the location information of the roadside communication unit to the cloud platform via HTTP protocol. The intelligent vehicle terminal displays traffic target information and distance information. The cloud platform summarizes, analyzes and processes the uploaded information and displays it on a web interface. A specific implementation scheme can be designed as follows:

[0052] Raw data acquisition and preprocessing for radar: The millimeter-wave radar and camera are connected to the Jetson XavierNX kit. A Python script is used to collect visual data and parse the millimeter-wave radar CAN messages. Then, the millimeter-wave radar data is preprocessed to remove abnormal signals, and finally the raw visual data and the preprocessed radar data are obtained.

[0053] Spatiotemporal alignment and fusion of radar data: The acquired data is fused in spatial coordinate system and synchronized in time using Python code. Then, the radar data containing RCS and distance information is projected onto the image plane after spatiotemporal alignment to generate an enhanced image containing radar feature information. The data is continuously processed through Python scripts to achieve spatiotemporal alignment and fusion of radar data.

[0054] Traffic target detection: A deep learning model based on RetinaNet and feature layer fusion is adopted, with VGG16 as the backbone network. The input of the network is the enhanced image generated after data fusion, and the output is the bounding box regression coordinates, the detected traffic target category and classification score. Radar and image data are fused in multiple layers of the network to ensure the real-time performance and accuracy of target recognition.

[0055] Data generation: The data is read and identified using a deep learning model based on radar-visual fusion deployed on the Jetson Xavier NX suite, generating a txt file that includes traffic target type and distance information.

[0056] Data delivery and upload: The roadside communication unit obtains its own location information through the positioning module, the control module obtains the final processed data from the Jetson XavierNX kit, and sends the result data to the intelligent vehicle terminal App via the communication module in the form of MQTT or PC5. At the same time, the result data and the location information of the roadside communication unit are uploaded to the cloud platform via the HTTP protocol.

[0057] Vehicle terminal information display: Users can obtain traffic target information and distance information sent by the roadside communication unit through the vehicle terminal APP, and make judgments on road conditions in real time. At the same time, the APP sends basic vehicle driving information to the roadside communication unit by default, including vehicle ID, vehicle position, vehicle speed, azimuth angle and vehicle status.

[0058] Cloud platform information display: The cloud platform collects packaged data and location information uploaded by roadside communication units, summarizes, analyzes and processes it, and displays it on the web interface, thereby realizing macro-level traffic control.

[0059] In summary, this invention has the following advantages: simple structure, convenient deployment, strong collaboration, applicability to various scenarios, good recognition effect, easy expansion, and suitability for real-time perception and communication of traffic targets. It can accurately acquire traffic target information in different scenarios, improve road traffic safety, strengthen macro-control of roads, and realize vehicle-road cooperation. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0060] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An integrated communication, sensing, and edge computing system for vehicle-road cooperation, characterized in that, include: The system comprises a sensing and edge computing unit, a roadside communication unit, an intelligent vehicle terminal, and a cloud platform, including: The perception and edge computing unit consists of millimeter-wave radar, camera and edge computing module. The millimeter-wave radar and camera acquire roadside traffic target data. The edge computing module receives the collected traffic target information through wired means, and performs real-time detection, identification and classification of strong and weak traffic participants, and obtains the type of traffic target within the roadside range and the relative distance information between the target and the driver vehicle. The roadside communication unit consists of a control module, a communication module, and a positioning module. The communication module includes two types: an LTE communication module and a PC5 communication module, supporting V2X-based cellular communication and direct communication modes respectively, enabling communication interconnection with intelligent vehicle terminals within the roadside area. The roadside communication unit connects to the edge computing module in the sensing and edge computing unit via WiFi. The control module controls the communication module to send data processed by the edge computing module to the intelligent vehicle terminal via MQTT or PC5, switching communication modes in different application scenarios. Simultaneously, the intelligent vehicle terminal sends basic vehicle driving information to the roadside communication unit. The roadside communication unit uploads the data processed by the edge computing module and the basic vehicle driving information to the cloud platform via HTTP through the communication module. The roadside communication unit determines its latitude and longitude information through the positioning module. By uploading the latitude and longitude information to the cloud platform and sending it to the intelligent vehicle terminal, the roadside communication unit completes the printing of information on the cloud map and achieves the exchange of location information with the intelligent vehicle terminal. The intelligent vehicle terminal obtains information on the type and distance of nearby traffic targets, and determines whether to brake to avoid danger or continue driving based on its own traffic conditions. The cloud platform collects and analyzes road condition information uploaded by roadside units, prints the connected driving vehicle and traffic target information onto the cloud map, and makes a comprehensive judgment based on traffic flow, vehicle speed, and location to achieve macro-level traffic control. in: The edge computing module, developed based on the Jetson Xavier NX suite, fuses point cloud data acquired by millimeter-wave radar and image data acquired by cameras using a radar-visual fusion target detection method to achieve real-time detection of traffic targets. The specific method is as follows: The radar-visual fusion target detection method includes radar data preprocessing, radar-visual data spatiotemporal alignment, radar-visual data fusion, and fusion model detection steps. Radar data preprocessing and radar-visual data spatiotemporal alignment belong to the pre-fusion part, responsible for processing abnormal radar data and unifying the spatiotemporal coordinate systems of millimeter-wave radar and visual data. Radar-visual data fusion and fusion model detection belong to the post-fusion part, responsible for projecting radar data containing effective features onto the vertical plane of the image to generate an enhanced image containing radar feature information, which is finally fed into the fusion model for target detection and recognition. The specific process of radar-visual fusion is as follows: The first step is to remove abnormal data generated by millimeter-wave radar: preprocess the raw data collected by millimeter-wave radar to remove empty targets, stationary targets, and false targets in order to avoid affecting the accuracy of fusion target detection. The second step is to ensure that the camera and radar maintain a consistent perspective: complete the fusion of the spatial coordinate systems of the camera and the millimeter-wave radar, and map the millimeter-wave radar data to the pixel coordinate system where the visual data is located. The mapping order is the millimeter-wave radar coordinate system, the three-dimensional world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system. The transformation from the millimeter-wave radar coordinate system to the two-dimensional pixel coordinate system is performed in the above order to make the radar and image data spatially consistent. The third step is to ensure the real-time performance of the system: complete the time synchronization between the camera and the millimeter-wave radar, that is, to jointly calibrate the time frame rate of the two sensors according to the sampling frequency, calculate the common multiple of the time interval between the two samplings, and ensure synchronization in time. The fourth step is to complete the deep fusion of the two types of sensor data: the millimeter-wave radar and visual data are fused, that is, the radar data containing effective features is projected on the vertical plane of the image, thereby expanding the dimension of the image data with RGB three-dimensional channels, adding a one-dimensional radar data channel, and generating an enhanced image containing radar feature information, which is used as the input of the fusion target detection model. The selected radar feature information is the millimeter-wave radar cross section (RCS) and distance information. The fifth step involves target recognition using fused data from the camera and millimeter-wave radar: The fusion detection model is built on RetinaNet, and its backbone network includes a millimeter-wave radar data path and a fusion network output data path, consisting of 7 max-pooling layers and 5 VGG blocks. The input to the fusion detection model is denoted as C0, which is the initial enhanced image generated by fusing millimeter-wave radar and visual data once. The first 5 layers consist of 5 max-pooling layers and 5 VGG blocks. Within each layer, the millimeter-wave radar data is processed by the max-pooling layer at a corresponding ratio. After processing, the data is merged with the fused data output from the previous layer into the fusion network composed of VGG blocks. The data is fused along the network data path. After the fusion at this layer is completed, the millimeter-wave radar data continues to enter the next layer of the max pooling layer along the path to change its size. Then it is merged with the fused data output from the previous layer into the fusion network data path composed of VGG blocks for fusion. The millimeter-wave radar data and C0 pass through 5 max pooling layers and VGG blocks respectively, that is, 5 fusion operations are performed. The output fused data is denoted as C1, C2, C3, C4 and C5. The remaining 2 max pooling layers form the last 2 layers. The millimeter-wave radar data passes through the two max pooling layers in sequence. At this time, the fused data is no longer merged with it. The output millimeter-wave radar data are C6 and C7 respectively. Feature pyramids are introduced into the deep network of the fusion detection model, denoted as P3, P4, P5, P6 and P7 respectively. C3, C4, C5, C6 and C7 are fed into feature pyramids P3, P4, P5, P6 and P7 in sequence. At each level of the feature pyramid, the corresponding millimeter-wave radar data are merged to generate feature outputs denoted as X3, X4, X5, X6 and X7 respectively. The target classification subnetwork and the bounding box regression subnetwork process the outputs X3, X4, X5, X6 and X7 of the feature pyramid, and finally generate the corresponding classification and regression results.

2. The integrated communication, sensing, and edge computing system for vehicle-road cooperation according to claim 1, characterized in that, In the perception and edge computing unit, the camera is connected to the edge computing module via a USB interface, the millimeter-wave radar is connected to the bus analyzer via a CAN bus, and the CAN bus analyzer is connected to the edge computing module via a CAN bus-to-USB interface. The millimeter-wave radar and the camera respectively acquire traffic target data and transmit the data to the edge computing module in real time via the USB interface.

3. The integrated communication, sensing, and edge computing system for vehicle-road cooperation according to claim 1, characterized in that, In the roadside communication unit, the control module is developed based on Raspberry Pi 4B, and the communication module includes two sub-modules: LTE and PC5. Corresponding to the two communication modes of the roadside communication unit, the Quectel EC20-LTE module and the Chenxin CX7101-PC5 module are used respectively. By default, the control module of the roadside communication unit uploads the traffic target recognition results obtained by the edge computing module to the cloud platform and distributes them to the vehicle terminal App via a 4G cellular network, and the information transmission method is MQTT. In scenarios without network coverage, the communication method of the roadside communication unit will automatically switch to the direct connection mode based on the PC5 module.

4. The integrated communication, sensing, and edge computing system for vehicle-road cooperation according to claim 1, characterized in that, The basic driving information of the vehicle is a custom message that includes some features of BSM messages. The message body contains the vehicle ID, vehicle position, vehicle speed, steering angle and vehicle status.

5. The integrated communication, sensing, and edge computing system for vehicle-road cooperation according to claim 1, characterized in that, The camera, millimeter-wave radar, edge computing module, and battery are housed in the same equipment box, ensuring consistent detection angles for the camera and radar. The battery powers all three components via a step-down converter. The box is installed on the roadside. The edge computing module is wired to the camera and millimeter-wave radar. The roadside communication unit transmits information to the edge computing module via WiFi and is installed above the traffic light pole. The intelligent vehicle terminal app is mounted on a mobile phone or tablet and placed in the vehicle, allowing the driver to view the roadside information in real time.

6. A traffic target perception and communication method for vehicle-road cooperation, characterized in that, Based on the integrated communication perception and edge computing system for vehicle-road cooperation as described in any one of claims 1-5, traffic target perception and communication for vehicle-road cooperation are realized.

Images