Vehicle holographic perception and risk behavior identification system based on deep fusion of radar, vision and multi-source data

Abstract

The application discloses a vehicle holographic perception and risk behavior identification system based on deep fusion of laser vision multi-source data, which comprises a laser vision fusion vehicle information perception layer integrated with a roadside laser radar, a video sensor and a high-definition snapshot camera; an edge computing layer of vehicle holographic identification and risk behavior identification based on an edge computing server; a cloud center processing server layer based on intelligent gantry cooperation and fusion operation; and an application software layer integrated with vehicle holographic perception, high-precision vehicle trajectory tracking, vehicle risk identification and statistical analysis. The application has the advantages that laser radar information and video image information can be fused, vehicle risk behavior identification in a complex road scene can be realized, corresponding vehicle holographic perception data can be obtained according to the risk behavior identification result, risk behavior area statistical analysis and risk vehicle tracking can be realized, and the road safety monitoring system can be provided with high-precision and high-reliability vehicle information perception and risk behavior identification functions.

Description

Technical Field

[0001] This invention patent relates to the field of vehicle information perception and risk identification in complex highway scenarios, and particularly to a vehicle holographic perception and risk behavior identification system based on deep fusion of multi-source data from radar vision. Background Technology

[0002] Currently, highway risk management systems mainly monitor vehicle speed. However, due to limitations imposed by road, environmental, and equipment factors, and the relatively limited types of risk monitoring, there are instances of monitoring omissions and failures in actual risk management, posing potential risks to traffic safety management and failing to meet the needs of vehicle risk monitoring in today's complex scenarios. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data from radar and vision. In terms of risk behavior monitoring, the system increases the types of risk behavior to be monitored, including frequent abnormal vehicle behaviors (speeding, low speed, driving in the wrong direction, illegal parking, crossing lanes) and occasional abnormal vehicle behaviors (abnormal trajectory). Under complex conditions, the system improves the accuracy of risk behavior recognition by fusing information from video and lidar. At the same time, the system can realize risk vehicle tracking and risk behavior area statistics functions to improve highway traffic capacity and traffic safety management level, and improve the level of intelligent transportation development and operation management.

[0004] To address the problems of existing highway risk monitoring systems, this invention proposes a vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data from radar and video sensors. It comprises: a radar-video fusion vehicle information perception layer integrating roadside LiDAR, video sensors, and high-definition capture cameras; an edge computing layer for vehicle holographic recognition and risk behavior identification; a cloud-based central processing server layer; and an application software layer. The radar-video fusion vehicle information perception layer includes roadside LiDAR, video sensors, high-definition capture cameras, and dedicated data transmission equipment. The edge computing layer for vehicle holographic recognition and risk behavior identification includes an edge computing server corresponding to each gantry. The cloud-based central processing server layer includes a cloud-based central computing server. The application software layer includes edge computing server application software and cloud-based central computing server application software. The specific process is as follows:

[0005] Within a certain range of the gantry, LiDAR and high-definition video cameras determine whether vehicles are passing on the highway. When a vehicle passes, the LiDAR, as the main sensor, collects the vehicle's outline dimensions and model information. Simultaneously, the high-definition video camera acquires holographic information composed of the vehicle's license plate, brand, type, and color. These two types of information are matched in the edge computing server. If the passing vehicle exhibits risky behavior, the corresponding lane's capture camera will be activated to capture an image of the vehicle's risky behavior, which is then transmitted to the edge computing server. The server then performs holographic information perception on the designated vehicle, acquiring its holographic data. By integrating the data obtained from each node, the system achieves statistical analysis of risky behavior areas across the entire road segment and tracks risky vehicles.

[0006] As a further improvement of the present invention, the system hardware composition of the edge computing server mainly includes an edge computing server, a data storage device, a high-resolution vehicle point cloud, video and image acquisition equipment, communication equipment, and energy and security equipment; one edge computing server is configured at each node and installed at an appropriate location on the node; the edge computing server is equipped with a high-performance CPU and GPU to meet the computing power requirements of various models, including point cloud perception processing and trajectory tracking, video perception processing, image holographic perception, and radar-visual data fusion; the data storage device includes a storage hard drive installed in a rack-mounted server or cabinet; the lidar and video sensor form a radar-visual fusion subsystem. The edge computing server system is equipped with a high-resolution vehicle point cloud, video, and image acquisition device installed on one side of the gantry. One such device is positioned at each lane and direction to sequentially acquire vehicle image information from the front and sides of vehicles passing through the gantry. Communication equipment refers to the information communication protocols, hardware, and facilities used for information exchange between edge computing servers and between edge computing servers and the cloud-based central computing server. Energy and safety equipment includes an all-weather power supply for the edge computing server system, a system operation monitoring sensor system, a rack and cabinet temperature and humidity monitoring system, and fault diagnosis and alarm equipment. As a further improvement of this invention, the edge computing server system includes an edge computing server system control module, a multi-source information recognition module, and a holographic perception module. The control module is responsible for monitoring the operating status of the edge computing server, receiving access requests, and issuing risk warnings. The multi-source information recognition module includes automatic reception of front-end collected data, invocation of point cloud data processing algorithms and 3D trajectory tracking algorithms, invocation of video trajectory tracking algorithms, display of multi-source data information, invocation of risk behavior recognition models, and judgment of risk behaviors. The main functions of the holographic perception module include invocation of vehicle holographic perception models and display of vehicle information.

[0007] As a further improvement of the present invention, the edge computing server receives 3D point cloud data and 2D image data pushed from the front-end information sensing device, wherein the 2D image data includes video and images; for the point cloud data of LiDAR, the PointNet-based Fusion algorithm is used to preprocess the raw point cloud information to realize different module functions; through multiple Voxel-based target detection methods including PointPillars and SECOND, the 3D point cloud data of the entire lane in the coverage area is transformed into vehicle perception data that meets different functions, including the displacement, size, position and unique ID code of all sensed vehicles in the coverage area; based on the extracted perception data, high-precision vehicle trajectory tracking based on 3D point cloud images is achieved through a series of Center-based target detection and tracking models, wherein the Center-based series includes CenterNet, CenterTrack and CenterPoint; for the collected video data, a video vehicle high-precision trajectory tracking model based on Deep-SORTMOT is called to complete the video vehicle high-precision trajectory tracking of highway scenes. The system performs high-precision trajectory tracking and outputs structured data. Utilizing deep learning network fusion algorithms, it achieves deep fusion of a series of center-based 3D high-precision trajectory tracking models and a Deep-SORT video trajectory tracking model, enabling high-precision, all-time vehicle trajectory tracking via radar-visual fusion. Based on the vehicle data from the radar-visual fusion, a deep learning vehicle abnormal behavior detection algorithm is used to ultimately achieve vehicle risk behavior identification through deep fusion of multi-source data in complex highway scenarios. For risky behaviors and their corresponding vehicles detected by vehicle risk behavior identification, the system drives the capture camera on the corresponding lane to capture images at specific locations, obtaining high-resolution images. It also calls the vehicle holographic perception module to acquire vehicle holographic perception data. Finally, all acquired structured data, including 3D vehicle perception data, trajectory data, holographic perception data, and risk behavior identification data, are merged and uploaded to the cloud-based central computing server.

[0008] As a further improvement of the present invention, the statistical analysis of risk behavior areas and the tracking function of risk vehicles in the whole road section are based on the control of each gantry by the cloud central computing server. According to the structured data uploaded by the edge computing server at each point, which includes three-dimensional vehicle perception data, trajectory data, holographic perception data and risk behavior recognition data, intelligent collaboration and integrated computing are realized.

[0009] As a further improvement of the present invention, the application software layer includes edge computing server application software and cloud central computing server application software. The cloud central computing server application software interface includes a server control interface and a cloud service statistical analysis interface. The server control interface includes three parts: a node status display module, a current node operation statistics module, and a risk vehicle details display module. The cloud service statistical analysis interface includes today's data statistics, seven-day data statistics, risk behavior vehicle node distribution statistics across the entire road segment, and cumulative risk behavior vehicle count statistics for each node across the entire road segment.

[0010] As a further improvement of this invention, in the edge computing server, the acquired raw 3D point cloud data is preprocessed. Multi-target detection transforms the 3D point cloud data of the entire lane within the coverage area into vehicle perception data that meets different functions, including the displacement, size, position, and unique ID code of all perceived vehicles within the coverage area. The target detection and tracking model achieves high-precision vehicle trajectory tracking based on 3D point cloud images. For the acquired video data, a high-precision video vehicle trajectory tracking model is used to complete high-precision trajectory tracking of video vehicles in highway scenes and output structured data. For the acquired 3D point cloud data and video data in complex environments, a network fusion model is used to achieve deep fusion of the 3D high-precision trajectory tracking model and the video trajectory tracking model, realizing high-precision all-time vehicle trajectory tracking through radar-visual fusion. A vehicle risk behavior recognition model suitable for the fusion model is established for 2D and 3D data. Finally, for the risky behaviors and corresponding vehicles detected by vehicle risk behavior recognition, the capture camera on the corresponding lane is driven to capture images at specific locations to obtain high-definition images. The vehicle holographic perception module is called to obtain its vehicle holographic perception data. Finally, all the acquired structured data, including 3D vehicle perception data, trajectory data, holographic perception data, and risk behavior recognition data, will be merged and uploaded to the cloud central computing server.

[0011] As a further improvement of this invention, the vehicle risk behavior recognition model in the edge computing server identifies both frequent abnormal vehicle behaviors (speeding, low speed, driving against traffic, illegal parking, crossing lanes) and occasional abnormal vehicle behaviors (abnormal trajectories). Radar and video equipment acquire vehicle speed α(x,y), displacement β(x,y), and position γ(x,y) information as input to the model. The model's default risk behavior recognition time period is t(α,β,γ). The input data types of the vehicle risk behavior recognition model include two-dimensional data and three-dimensional data.

[0012] As a further improvement of the present invention, the vehicle trajectory tracking selects the input type of the model under the judgment condition. When three-dimensional data is used as input information, it is assumed that the number of detections in a certain ID target trajectory is c, and the number of missing values ​​is c′. The trajectory tracking model sets a target detection threshold c1 (preferably between 3 and 6) and a lower limit c′ for the missing value threshold. 2min (Preferably between 2 and 4), upper limit of the loss threshold c 2max (Preferably between 7 and 9), when the number of times the target is continuously detected satisfies c>c1, the target is determined to be a temporary target. If the number of times the target is lost within time t satisfies c′ <c 2min The target is determined to be the target to be detected, and the detection of the target continues until the number of target losses satisfies c′>c. 2max When the target is considered lost, the number of times the target is lost is c. 2min <c′<c 2max Under certain conditions, it determines whether target tracking can be achieved by fusing two-dimensional video information; when using two-dimensional data as input, it determines whether target tracking can be achieved by fusing three-dimensional LiDAR information, also under certain conditions. Finally, it outputs the optimal trajectory for risk behavior identification.

[0013] As a further improvement of this invention, the fusion of lidar and video data involves spatial calibration, temporal calibration, and target matching. In spatial calibration, the radar coordinate system O... xyz Transformation to world coordinate system P xyz World coordinate system P xyz Transformation Q to the three-dimensional coordinate system of the video device xyz Finally, it is converted into a two-dimensional pixel coordinate system R. xy Calibration on the timeline, the average sampling period t of the lidar l The average sampling period of the video is t v , satisfying t v =nt l Sampling begins at time k-1, and after one sampling period t... l Afterwards, the fusion system satisfies kt v =(k-1)kt v +nt l During target matching, the 3D information acquired by the lidar and the 2D information from the video contain a large amount of independent target information. For a radar trajectory sequence S acquired at a certain sampling period... l and video trajectory sequence S v The radar-detected target obj is acquired at i consecutive positions (preferably between 3 and 5). l and video detection target obj vThe center coordinates and size information of the target bounding box are obtained, and the average intersection-union ratio is calculated. If the intersection-union ratio is greater than 80%, it is determined that they are the same target.

[0014] As a further improvement of the present invention, for the input type determination of the vehicle trajectory tracking model, within time t, for the radar trajectory sequence S l and video trajectory sequence S v The number of times the target is continuously detected satisfies c > c1, and the number of times the target is lost satisfies c′. <c 2min The number of times satisfies Input 3D data, otherwise input 2D data; for the judgment of vehicle trajectory tracking model fusion data, as mentioned above, if the number of target loss determined by the 3D point cloud within time t satisfies c... 2min <c′<c 2max At this point, based on the corresponding initial and final positions lost in the two-dimensional video data, the intersection-union ratio (IU / R) is calculated. The IU / R must satisfy a condition where the initial and final IU / R ratio is greater than 80%, and the number of target detections in the video is greater than the number of lost targets (c′). Then, the fusion model is used to complete the fusion of the video radar trajectory data.

[0015] The beneficial effects of this invention are: by using a vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data from radar vision to replace the traditional risk management system, the accuracy of vehicle risk behavior recognition can be effectively improved, the types of risk behavior recognition monitoring can be increased, and the functions of risk vehicle tracking and risk behavior area statistics can be realized, so as to improve the traffic capacity and traffic safety management level of highways, and improve the level of intelligent transportation development and operation management. Attached Figure Description

[0016] Figure 1 The overall system framework diagram for this invention.

[0017] Figure 2 The laser-visual fusion vehicle information perception workflow constructed for this invention.

[0018] Figure 3 The edge computing server workflow constructed for this invention.

[0019] Figure 4 Framework structure for vehicle risk behavior recognition model

[0020] Figure 5 Vehicle risk behavior identification and control process

[0021] Figure 6 The cloud-based central computing server control process constructed for this invention.

[0022] Figure 7 This is the control interface for the cloud-based central computing server constructed for this invention.

[0023] Figure 8 The risk vehicle details display interface constructed for this invention.

[0024] Figure 9 The cloud service statistical analysis interface constructed for this invention.

[0025] Figure 10 A schematic diagram of the edge computing server system constructed for this invention.

[0026] Figure 11 Service diagram of the edge computing server software system constructed for this invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0028] like Figure 1 As shown, the vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data from radar and vision includes: a radar-vision fusion vehicle information perception layer integrating roadside LiDAR, video sensors, and high-definition capture cameras; an edge computing layer for vehicle holographic recognition and risk behavior identification; a cloud-based central processing server layer; and an application software layer. The vehicle information perception layer integrating LiDAR and high-definition video cameras includes LiDAR, high-definition video cameras, high-definition capture cameras, and dedicated data transmission equipment. The edge computing layer for vehicle holographic recognition and risk behavior identification includes an edge computing server corresponding to each gantry. The cloud-based central processing server layer includes a cloud-based central computing server. The application software layer includes edge computing server application software and central computing server application software.

[0029] like Figure 2 As shown, within a certain range of the gantry, LiDAR and high-definition video cameras determine whether vehicles are passing on the highway. When a vehicle passes, the LiDAR, as the main sensor, collects the vehicle's outline dimensions and model information. Simultaneously, the high-definition video camera acquires holographic information composed of the vehicle's license plate, brand, type, and color. These two types of information are matched in the edge computing server. If the passing vehicle exhibits risky behavior, the corresponding lane's capture camera is activated to take a picture, acquiring an image of the vehicle's risky behavior and transmitting it to the edge computing server. The server then performs holographic information perception on the designated vehicle, acquiring its holographic data. By integrating the data obtained from each node, the system achieves statistical analysis of risky behavior areas across the entire road segment and tracks risky vehicles.

[0030] In this embodiment, the hardware components of the edge computing server system mainly include an edge computing server, a data storage device, a high-resolution vehicle point cloud, video and image acquisition equipment, communication equipment, energy and security equipment, etc. One edge computing server is configured at each gantry and installed at an appropriate location on the gantry.

[0031] like Figure 3 As shown, in the edge computing server, the acquired raw 3D point cloud data is preprocessed. Multi-target detection transforms the 3D point cloud data of the entire lane within the coverage area into vehicle perception data that meets different functions, including the displacement, size, position, and unique ID code of all perceived vehicles within the coverage area. The target detection and tracking model achieves high-precision vehicle trajectory tracking based on 3D point cloud images. For the acquired video data, a high-precision video vehicle trajectory tracking model is used to complete high-precision trajectory tracking of vehicles in highway scenes and output structured data. For the acquired 3D point cloud data and video data in complex environments, a network fusion model is used to achieve deep fusion of the 3D high-precision trajectory tracking model and the video trajectory tracking model, realizing high-precision, all-time vehicle trajectory tracking through radar-visual fusion. A vehicle risk behavior recognition model suitable for the fusion model is established for 2D and 3D data. Finally, for the risky behaviors and corresponding vehicles detected by vehicle risk behavior recognition, the capture camera on the corresponding lane is driven to capture images at specific locations to obtain high-definition images. The vehicle holographic perception module is called to obtain its vehicle holographic perception data. Finally, all the acquired structured data, including 3D vehicle perception data, trajectory data, holographic perception data, and risk behavior recognition data, will be merged and uploaded to the cloud central computing server.

[0032] In this implementation, such as Figure 4 As shown, the vehicle risk behavior recognition model in the edge computing server identifies both frequent abnormal vehicle behaviors (speeding, low speed, driving against traffic, illegal parking, crossing lanes) and occasional abnormal vehicle behaviors (abnormal trajectories). Radar and video equipment acquire vehicle speed α(x,y), displacement β(x,y), and position γ(x,y) information as input to the model. The model's default risk behavior recognition time period is t(α,β,γ). The input data types for the vehicle risk behavior recognition model include two-dimensional and three-dimensional data. The specific vehicle trajectory acquisition method is as follows:

[0033] like Figure 5 As shown, the vehicle trajectory tracking model selects the input type under the judgment condition. When using 3D data as input information, assuming that the number of detections in a certain ID target trajectory is c and the number of missing values ​​is c′, the trajectory tracking model sets a target detection threshold c1 (preferably between 3 and 6) and a lower limit c′ for the missing value threshold. 2min (Preferably between 2 and 4), upper limit of the loss threshold c2max (Preferably between 7 and 9), when the number of times the target is continuously detected satisfies c>c1, the target is determined to be a temporary target. If the number of times the target is lost within time t satisfies c′ <c 2min The target is determined to be the target to be detected, and the detection of the target continues until the number of target losses satisfies c′>c. 2max When the target is considered lost, the number of times the target is lost is c. 2min <c′<c 2max Under certain conditions, it determines whether target tracking can be achieved by fusing two-dimensional video information; when using two-dimensional data as input, it determines whether target tracking can be achieved by fusing three-dimensional LiDAR information, also under certain conditions. Finally, it outputs the optimal trajectory for risk behavior identification.

[0034] The fusion of lidar and video data involves spatial calibration, temporal calibration, and target matching. Spatial calibration includes the radar coordinate system O... xyz Transformation to world coordinate system P xyz World coordinate system P xyz Transformation Q to the three-dimensional coordinate system of the video device xyz Finally, it is converted into a two-dimensional pixel coordinate system R. xy Calibration on the timeline, the average sampling period t of the lidar l The average sampling period of the video is t v , satisfying t v =nt l Sampling begins at time k-1, and after one sampling period t... l Afterwards, the fusion system satisfies kt v =(k-1)kt v +nt l During target matching, the 3D information acquired by the lidar and the 2D information from the video contain a large amount of independent target information. For a radar trajectory sequence S acquired at a certain sampling period... l and video trajectory sequence S v The radar-detected target obj is acquired at i consecutive positions (preferably between 3 and 5). l and video detection target obj v The center coordinates and size information of the target bounding box are obtained, and the average intersection-union ratio is calculated. If the intersection-union ratio is greater than 80%, it is determined that they are the same target.

[0035] For the input type determination of the vehicle trajectory tracking model, within time t, for the radar trajectory sequence S l and video trajectory sequence S v The number of times the target is continuously detected satisfies c > c1, and the number of times the target is lost satisfies c′. <c 2min The number of times satisfies Input 3D data, otherwise input 2D data; for the judgment of vehicle trajectory tracking model fusion data, as mentioned above, if the number of target loss determined by the 3D point cloud within time t satisfies c... 2min <c′<c 2max At this point, based on the corresponding lost initial and ending positions on the two-dimensional video data, the intersection-union ratio is calculated. The intersection-union ratio between the initial and ending positions must be greater than 80%, and the number of video target detections must be greater than the number of lost targets c′. The fusion of video radar trajectory data is then completed through the fusion model.

[0036] like Figure 6 As shown, in the cloud-based central computing server, when a vehicle passes through a detection node, LiDAR and video sensors perceive the vehicle. At the node's edge computing server, the 3D point cloud data from the LiDAR is used to acquire the vehicle's perception information, and the data from the LiDAR and video sensors is deeply fused to detect whether the vehicle exhibits any risky behavior. If the vehicle does not exhibit risky behavior, the perception information data of the risk-free vehicle is directly transmitted to the application software platform for storage and display. If the vehicle exhibits risky behavior, the corresponding lane's capture camera is activated. When the vehicle reaches a specific location, a high-definition image of the risky vehicle is captured, and holographic perception is performed to acquire holographic information. This information is then uploaded to the central computing server to complete the retrieval and binding of the corresponding vehicle's 3D data perception information and holographic perception information. The central computing server compiles and statistically analyzes the areas with risky behavior along the entire road segment, simultaneously tracking the trajectory of vehicles exhibiting risky behavior across the entire road segment. Finally, the integrated information is uploaded to the corresponding location on the application software platform for storage, for subsequent use and display.

[0037] like Figure 7 , Figure 8As shown, based on the interface functional requirements of the cloud-based central computing server, the software interface is designed into two sub-interfaces: a cloud-based central computing server control interface and a cloud service statistical analysis interface. The cloud-based central computing server control interface comprises three parts: a node status display module, a current node operation statistics module, and a risk vehicle details display module. The node status display module in the cloud-based central computing server control interface mainly includes two parts. The first part contains the status of all edge computing servers and data transmission time, and allows selection of specific nodes to be displayed in the current node operation module. The second part statistically displays the number of online and offline nodes. The current node operation module in the cloud-based central computing server control interface mainly includes real-time transmission log information, risk behavior vehicle information, and a risk vehicle details button. The real-time transmission log information mainly receives vehicle information transmitted between edge computing servers and displays the transmission results, including vehicle type, transmission time (using 24-hour timing), and transmission result (success / failure). The risk behavior vehicle information includes risk type, vehicle ID, and recording time.

[0038] The "Risk Vehicle Details" button displays and confirms vehicles exhibiting risky behavior uploaded at the current node, facilitating subsequent processing. Clicking the "Risk Vehicle Details" button brings up the risk vehicle details display interface, which mainly includes the risk vehicle image, risk vehicle point cloud, risk vehicle information, the current risk vehicle's sequence number, the node where the information was uploaded, the recording time, and "Previous" and "Next" buttons. The risk vehicle information includes the vehicle license plate number, type, color, brand identified by the vehicle holographic perception model, and the vehicle speed and vehicle ID obtained from the 3D point cloud data processing module. The sequence number indicates the current risk vehicle's ranking among the risk vehicles uploaded at this node; for example, 3 / 15 means that 15 vehicles with risky behavior have been counted, and this vehicle is the third one. Node information and the recording time are used to complete the vehicle information display. The "Previous" and "Next" buttons allow viewing all risk vehicle details.

[0039] like Figure 9As shown, the cloud service statistical analysis interface mainly includes four parts: today's data statistics, seven-day data statistics, statistics on the distribution of vehicles exhibiting risky behavior across all road segments, and statistics on the cumulative number of vehicles exhibiting risky behavior at each node across the entire road segment. Today's data statistics include the total number of vehicles passing through that day, the total number of vehicles exhibiting risky behavior that day, the percentage of vehicles exhibiting risky behavior, the highest number of risky behaviors, and the top five nodes where risky behavior is most likely to occur. The seven-day data statistics are consistent with today's data statistics, recorded and displayed in tabular form from Sunday to Saturday. The statistics on the distribution of vehicles exhibiting risky behavior across all road segments have two parts. The first part is a statistical table containing vehicle type, vehicle license plate number, number of risky behaviors, and statistics on the nodes where the risky behavior occurred; the second part is a bar chart containing statistical data on the nodes where a specified vehicle exhibited risky behavior across the entire road segment. The statistics on the cumulative number of vehicles exhibiting risky behavior at each node across the entire road segment are presented as a bar chart, showing the number of vehicles exhibiting risky behavior at each node across the entire road segment within the storage update period. In summary, the cloud-based service statistical analysis interface not only visually displays which nodes along the entire road segment are prone to vehicle risk behaviors, but also provides statistical analysis of the frequency and location of specific vehicles exhibiting such behaviors. The "Today's Data Statistics" section allows personnel to directly understand the day's traffic flow and risk behavior information, while the "Seven-Day Data Statistics" section facilitates comparison and summarization.

[0040] like Figure 10 , Figure 11As shown, the edge computing server system includes an edge computing server system control module, a multi-source information recognition module, and a holographic perception module. The edge computing server control module is responsible for monitoring the edge computing server's operational status, receiving access requests, and issuing risk warnings. The operational status monitoring includes real-time display of the system's current time, node number, node working status, and node location information. The risk warning function receives remote access parameters from the central computing server for specified nodes through the access request receiving function, and sets risk warnings and provides safe driving guidance based on risk behavior information obtained from the central computing server in the edge computing server software interface. The edge computing server multi-source information recognition module includes automatic reception of front-end acquired data, invocation of point cloud data processing algorithms and 3D trajectory tracking algorithms, invocation of video trajectory tracking algorithms, display of multi-source data information, invocation of risk behavior recognition models, and risk behavior determination. The automatic front-end data acquisition function refers to the separate reception and reading of 3D point cloud data and video data acquired by the Rave-Vision fusion subsystem devices for subsequent algorithm implementation. The point cloud data processing algorithm and 3D trajectory tracking algorithm call refer to the process of first calling the point cloud data processing algorithm to obtain vehicle information in the highway scene, including vehicle type, lane, vehicle speed, and unique ID, and then calling the 3D trajectory tracking algorithm to obtain the vehicle's 3D trajectory throughout the acquisition period. The video trajectory tracking algorithm call refers to the process of calling the video trajectory tracking algorithm to obtain the vehicle trajectory in the highway scene from the received front-end video data. The multi-source data information display refers to the process of calling the multi-source data fusion model after the edge computing server completes trajectory tracking for different data types to obtain the final vehicle trajectory tracking data. Simultaneously, the fused vehicle trajectory information, combined with the acquired data, is displayed in the Rave-Vision data fusion display window and the real-time trajectory tracking window for risky behavior vehicles. The risk behavior recognition model call and risk behavior judgment refer to the process of calling the risk behavior recognition model to obtain the current operating status information of the detected vehicle, and simultaneously judging the risk behavior of the vehicle's operating status. If a risky behavior is confirmed, instructions need to be transmitted to the LiDAR system to trigger a pulse signal to the high-resolution camera in the lane where the vehicle is located. When the vehicle reaches the designated position, the camera captures a high-definition image of the vehicle, and the edge computing server's holographic perception module then performs subsequent operations. The main functions of the edge computing server's holographic perception module include invoking the vehicle holographic perception model and displaying vehicle information. Invoking the vehicle holographic perception model refers to the edge computing server sending instructions to the LiDAR to drive the camera to capture the vehicle, enabling the high-resolution camera in the lane where the risky vehicle is located to capture the image when the vehicle enters its capture range, thus obtaining high-definition image data of the vehicle.Then, the vehicle holographic perception model is invoked to perceive high-definition images of vehicles engaging in risky behavior, and information such as vehicle color, model, and brand is obtained. Vehicle information display refers to displaying the vehicle color, model, brand, and other information obtained by the vehicle holographic perception model in text form on the system interface, while simultaneously displaying the vehicle image in image form on the system interface.

[0041] In this embodiment, as a further improvement of the present invention, the statistical analysis of risk behavior areas and the tracking function of risky vehicles across the entire road segment are based on the control of each gantry by the cloud-based central computing server. The structured data, which includes three-dimensional vehicle perception data, trajectory data, holographic perception data and risk behavior recognition data, uploaded by the edge computing servers at each point, enables intelligent collaboration and fusion computing.

Claims

1. A vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data from radar vision, characterized in that, include: The system comprises a radar-visual fusion vehicle information perception layer integrating roadside lidar, video sensors, and high-definition capture cameras; an edge computing layer for vehicle holographic recognition and risk behavior identification; a cloud-based central processing server layer; and an application software layer. The radar-visual fusion vehicle information perception layer includes lidar, high-definition video cameras, high-definition capture cameras, and dedicated data transmission equipment. The edge computing layer for vehicle holographic recognition and risk behavior identification includes an edge computing server corresponding to each gantry. The cloud-based central processing server layer includes a cloud-based central computing server. The application software layer includes edge computing server application software and cloud-based central computing server application software. High-precision vehicle trajectory tracking satisfies the condition of selecting the model's input type. When using 3D data as input information, it is assumed that the number of detections in a certain ID target trajectory is... The number of times lost is Setting a target detection threshold in the trajectory tracking model and the lower limit of the loss threshold Loss threshold upper limit When the target is continuously detected a certain number of times The target is determined to be a temporary target. The number of times the target is lost within a time period meets the requirements. The target is determined to be the target to be detected, and the detection of the target continues until the target is lost a certain number of times. When the target is considered lost, the number of times the target is lost is determined. Under the given conditions, determine whether target tracking can be completed by fusing two-dimensional video information; When using two-dimensional data as input information, the system determines whether to complete target tracking by fusing three-dimensional information from lidar under certain conditions, and finally outputs the optimal trajectory for risk behavior identification. The fusion of lidar and video data includes spatial calibration, temporal calibration, and target matching; in spatial calibration, the radar coordinate system... Transformation to the world coordinate system World coordinate system Transformation to the three-dimensional coordinate system of the video device Finally, it is converted into a two-dimensional pixel coordinate system. ; Calibration on the timeline, average sampling period of the lidar The average sampling period of the video is ,satisfy ,exist Sampling begins at a specific time, and after one sampling period... Afterwards, the fusion system satisfies ; During target matching, the 3D information acquired by lidar and the 2D information from video contain a large amount of independent target information. For a radar trajectory sequence acquired at a certain sampling period... and video trajectory sequences Obtain continuous Radar detection targets at various locations and video detection targets The center coordinates and size information of the target bounding box are obtained, and the average intersection-union ratio is calculated. If the intersection-union ratio is greater than 80%, the target is determined to be the same target. For the input type determination of high-precision vehicle trajectory tracking models, in Within a time period, for the radar trajectory sequence and video trajectory sequences The target continuous detection number meets Meanwhile, the number of times the target is lost meets the requirement. If the number of times satisfies Input 3D data if necessary, otherwise input 2D data; for the judgment of high-precision vehicle trajectory tracking model fusion data, if the number of target loss determined by the 3D point cloud within time t meets the following condition... At this point, based on the corresponding lost initial and ending positions on the two-dimensional video data, the intersection-union ratio (IU / R) is calculated. The IU / R must satisfy a condition where the IU / R of the initial and ending positions is greater than 80%, and the number of detected trajectories is greater than [a certain value]. The fusion of video radar trajectory data is achieved through a fusion model.

2. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 1, characterized in that, The system's specific execution process is as follows: When a vehicle passes near the gantry, the LiDAR and high-definition video camera acquire the vehicle's 3D and 2D information, respectively. The LiDAR, as the main sensor, collects the vehicle's outline dimensions and model information, while simultaneously triggering the high-definition video camera to acquire holographic information composed of the vehicle's license plate, brand, type, and color. The two types of information are matched in the computing server. If the passing vehicle exhibits risky behavior, the corresponding lane's capture camera will be activated to capture images of the vehicle's risky behavior and transmit them to the edge computing server. The server will then perform holographic information perception on the designated vehicle, acquire its holographic data, and merge and upload all acquired structured data, including 3D vehicle perception data, trajectory data, holographic perception data, and risky behavior recognition data, to the cloud-based central computing server. The cloud-based central computing server will coordinate and control multiple edge computing servers, and based on the structured data uploaded by each edge computing server, which includes 3D vehicle perception data, trajectory data, holographic perception data, and risky behavior recognition data, it will achieve intelligent collaboration, fusion computing, statistical analysis of risky behavior areas across the entire road segment, and tracking of risky vehicles across the entire road segment.

3. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 1, characterized in that, The system hardware components of an edge computing server include edge computing servers, data storage devices, high-resolution vehicle point clouds, video and image acquisition equipment, communication equipment, and energy and security equipment. One edge computing server is configured at each node, installed in an appropriate location. The edge computing servers are equipped with high-performance CPUs and GPUs to meet the computing power requirements of various models, including point cloud perception processing and trajectory tracking, video perception processing, image holographic perception, and radar-visual data fusion. Data storage devices include hard drives installed in rack-mounted servers or server racks. LiDAR and video sensors form the radar-visual fusion subsystem equipment, installed on a gantry. On one side; high-resolution vehicle point cloud, video and image acquisition equipment is deployed at the gantry, with one high-resolution vehicle point cloud, video and image acquisition device deployed in each lane and each direction, to sequentially collect vehicle image information from the front and sides of vehicles passing through the gantry; communication equipment refers to the information communication protocols, communication hardware and communication facilities and equipment on which information exchange is achieved between edge computing servers and between edge computing servers and cloud central computing servers; energy and safety equipment includes the all-weather power supply device for the edge computing server system, the system operation monitoring sensor system, the temperature and humidity monitoring system in the rack and cabinet, and fault diagnosis and alarm equipment.

4. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 2, characterized in that, The edge computing server receives 3D point cloud data and 2D image data pushed from front-end information sensing devices, where the 2D image data includes video and images. For the LiDAR point cloud data, the PointNet-based Fusion algorithm is used to preprocess the raw point cloud information to enable different module functions. Multiple Voxel-based target detection methods, including PointPillars and SECOND, are used to transform the 3D point cloud data of the entire lane within the coverage area into vehicle perception data that meets different functions, including the displacement, size, position, and unique ID codes of all sensed vehicles within the coverage area. Based on the extracted perception data, a Center-based series of target detection and tracking models is used to achieve high-precision vehicle trajectory tracking based on 3D point cloud images, where the Center-based series includes CenterNet, CenterTrack, and CenterPoint. For the acquired video data, a Deep-SORT-based algorithm is called. The MOT video vehicle high-precision trajectory tracking model completes high-precision trajectory tracking of video vehicles in highway scenes and outputs structured data; using deep learning network fusion algorithms, it completes the deep fusion of the Center-based series of 3D high-precision trajectory tracking models and the Deep-SORT video trajectory tracking model, realizing high-precision vehicle trajectory tracking through radar-visual fusion. Based on the vehicle data information fused by radar and vision, and using a deep learning vehicle abnormal behavior detection algorithm, the vehicle risk behavior identification in complex highway scenarios is finally realized through deep fusion of multi-source data. For risky behaviors and their corresponding vehicles detected by vehicle risk behavior identification, the corresponding lane's capture camera will be activated to capture high-definition images. The vehicle holographic perception module is invoked to obtain its vehicle holographic perception data; finally, all the obtained structured data, including 3D vehicle perception data, trajectory data, holographic perception data and risk behavior recognition data, are merged and uploaded to the cloud central computing server.

5. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 2, characterized in that, The determination of whether a perceived vehicle exhibits risky behavior is as follows: If the vehicle does not exhibit risky behavior, the perception information data of the risk-free vehicle is directly transmitted to the application software platform for storage and display; if the vehicle exhibits risky behavior, the high-definition capture camera in the corresponding lane is first activated to capture a high-definition image of the vehicle exhibiting risky behavior and complete the holographic perception of the vehicle to obtain holographic information, which is then uploaded to the cloud central computing server to complete the retrieval and binding of the corresponding vehicle's three-dimensional data perception information and holographic perception information. The cloud-based central computing server compiles and statistically analyzes data to identify areas with risky behaviors along the entire road segment. It also enables high-precision vehicle trajectory tracking for these risky behaviors. Finally, the integrated information is uploaded to the corresponding location on the application software platform for storage, for future use and display.

6. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 5, characterized in that, The application software layer includes edge computing server application software and cloud central computing server application software. The interface of the cloud central computing server application software includes three parts: node status display module, current node operation statistics module, and risk vehicle details display module.

7. The vehicle holographic perception and risk behavior recognition system based on deep fusion of multi-source data as described in claim 4, characterized in that, The vehicle risk behavior recognition model in the edge computing server identifies both frequent and occasional abnormal vehicle behaviors. Frequent abnormal behaviors include speeding, slow driving, driving against traffic, illegal parking, and crossing lanes, while occasional abnormal behaviors include abnormal trajectories. LiDAR and video equipment are used to acquire vehicle speed. Displacement ,Location Information is used as input to the model, and the model's default time period for identifying risky behaviors is... .

Images