Highway Traffic Safety Monitoring Method Based on Multi-Source Data Intelligent Fusion

Abstract

This invention provides a highway traffic safety monitoring method based on intelligent fusion of multi-source data, belonging to the field of traffic safety monitoring technology. The method includes: acquiring vehicle modal data and road modal data; acquiring vehicle state characteristic information, road state characteristic information, and road state prediction characteristic information through multiple sub-models of a traffic safety monitoring model; predicting safety risk information for multiple zones; and generating traffic safety monitoring alerts. According to this invention, data from different modalities can be extracted and intelligently fused based on a traffic safety monitoring model. By comprehensively considering road conditions, vehicle conditions, and the mutual influence between vehicles, the method predicts the safety risks existing in each zone of the road. It can monitor the risks in the area in a timely and comprehensive manner and take necessary measures to reduce the potential hazards caused by safety risks, improving monitoring accuracy and reducing the impact of safety hazards.

Description

Technical Field

[0001] This invention relates to the field of traffic safety monitoring technology, and in particular to a method for monitoring highway traffic safety based on intelligent fusion of multi-source data. Background Technology

[0002] In related technologies, traffic safety on highways is usually improved by radar speed measurement and detection of whether vehicles cross the line. However, this method can only determine whether a vehicle has violated the rules, thereby indirectly determining whether a vehicle poses a safety risk. The monitoring of road safety risks is not timely or comprehensive, and it is difficult to take necessary measures in a timely manner when safety risks occur.

[0003] The information disclosed in the background section of this application is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a highway traffic safety monitoring method based on intelligent fusion of multi-source data, which can solve the technical problems of untimely and incomplete monitoring of road safety risks in related technologies.

[0005] According to a first aspect of the present invention, a highway traffic safety monitoring method based on intelligent fusion of multi-source data is provided, comprising: capturing monitoring video of a monitoring area using a camera and acquiring vehicle modal data and road modal data, wherein the road modal data is used to represent the vehicle occupancy status of multiple zones of multiple roads in the monitoring area, and the vehicle modal data is used to represent the movement status of multiple vehicles in the monitoring area; processing the vehicle modal data of multiple vehicles using a trained traffic safety monitoring model's vehicle modal feature extraction sub-model to obtain vehicle state feature information of each vehicle; and processing the road modal feature extraction sub-model of the trained traffic safety monitoring model to obtain vehicle state feature information of each vehicle. The road modal data is processed to obtain road state feature information; based on vehicle modal data, the correlation information of each vehicle is determined, wherein the correlation information is used to describe the mutual influence between the vehicles; based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model, road state prediction feature information is obtained; based on the road state prediction feature information, the safety risk information of multiple zones of multiple roads is determined; among the multiple zones of multiple roads, it is determined whether there are zones with safety risk information higher than a preset risk threshold; if so, traffic safety monitoring prompt information is generated based on the safety risk information.

[0006] According to the present invention, the method of capturing monitoring video of a monitoring area by a camera and acquiring vehicle modal data and road modal data includes: setting vehicle identification information for each vehicle; determining the position information, speed information, and acceleration information of each vehicle based on the monitoring video frame corresponding to the current monitoring time and its adjacent video frames; stitching the vehicle identification information, position information, speed information, and acceleration information together to obtain vehicle modal data; acquiring the boundary position information of each partition; determining the vehicle identification information within each partition, and the first ratio of a first area of ​​the portion of the vehicle within the partition to a second area of ​​the region where the vehicle is located in the monitoring video frame corresponding to the current monitoring time; and obtaining road modal data for each partition based on the boundary position information, the vehicle identification information, and the first ratio.

[0007] According to the present invention, a vehicle modal feature extraction sub-model of a trained traffic safety monitoring model is used to process vehicle modal data of multiple vehicles to obtain vehicle state feature information of each vehicle. This includes: processing the vehicle modal data of multiple vehicles through a first feature extraction layer of the vehicle modal feature extraction sub-model to obtain vehicle motion state feature information corresponding to each vehicle; obtaining the nearest neighbor vehicles of each vehicle based on the vehicle modal data; processing the vehicle motion state feature information corresponding to each vehicle and its nearest neighbor vehicles through a first self-attention mechanism of the vehicle modal feature extraction sub-model to obtain first vehicle fusion feature information, wherein the first vehicle fusion feature information is used to describe the vehicle's own motion state and the impact of the motion state of the nearest neighbor vehicles on the vehicle; and decoding each first vehicle fusion feature information through a temporal decoding layer of the vehicle modal feature extraction sub-model to obtain vehicle state feature information of each vehicle, wherein the vehicle state feature information is used to describe the vehicle's current motion state and motion trend, and the motion trend is used to predict the motion state at the next monitoring moment.

[0008] According to the present invention, a road modal feature extraction sub-model of a trained traffic safety monitoring model processes road modal data to obtain road state feature information, including: combining the road state feature information corresponding to each partition according to the location information of each partition to obtain a road state feature information tensor; performing cross-channel feature fusion processing on the road state feature information tensor through a 1×1 convolutional layer of the road modal feature extraction sub-model to obtain a partition state description feature tensor, wherein the partition state description feature tensor is used to describe the vehicle occupancy status of each partition; and processing the partition state description feature tensor through an encoding layer of the road modal feature extraction sub-model to obtain a road state description vector, wherein the road state description vector is used to describe... The system monitors the overall occupancy status of multiple zones within the monitoring area. Through the road temporal prediction layer of the road modal feature extraction sub-model, the road state description vector at the current monitoring time and the road state latent vector at the previous monitoring time are processed to obtain the road state latent vector at the current monitoring time. This latent vector describes the changing trend of occupancy status in each zone of the current road. Through the decoding layer of the road modal feature extraction sub-model, the road state latent vector at the current monitoring time is decoded to obtain road state feature information for each zone. This road state feature information describes the changing trend of vehicle occupancy status in each zone, and the changing trend of occupancy status in each zone is used to predict the vehicle occupancy status at the next monitoring time.

[0009] According to the present invention, determining the association information of each vehicle based on vehicle modal data includes: obtaining position information, velocity information, and acceleration information of each vehicle based on the vehicle modal data; obtaining a predicted motion path for each vehicle based on the position information, velocity information, and acceleration information of each vehicle; sampling along the predicted path to obtain multiple predicted sampling positions corresponding to each vehicle; obtaining the minimum distance between the multiple predicted sampling positions of the i-th vehicle and the multiple predicted sampling positions of the j-th vehicle; determining current mutual influence parameters based on the position information of the i-th and j-th vehicles; determining future predicted mutual influence parameters based on the minimum distance; and determining the association information of the i-th and j-th vehicles based on the current mutual influence parameters and the future predicted mutual influence parameters.

[0010] According to the present invention, road state feature information is obtained based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model. This includes: processing the vehicle state feature information and road state feature information through the first cross-attention mechanism of the multimodal fusion prediction sub-model to obtain first fused state feature information for each vehicle, wherein the first fused state feature information represents the occupancy trend of each vehicle in each zone, and the occupancy trend is used to predict the occupancy status of each vehicle in each zone at the next monitoring time; processing the first fused state feature information and correlation information of each vehicle through the graph model processing hierarchy of the multimodal fusion prediction sub-model to obtain second fused state feature information for each vehicle, wherein the second... The fused state feature information is used to represent the occupancy trend of each vehicle in each zone under the influence of other vehicles. The second fused state feature information of each vehicle is processed through the first fully connected layer and the first activation layer of the multimodal fusion prediction sub-model to obtain the predicted occupancy status information of each vehicle in each zone. The second fused state feature information is then weighted and summed using the predicted occupancy status information of each vehicle in each zone to obtain the comprehensive predicted state information of vehicles within each zone. This comprehensive predicted state information describes the predicted motion state of all vehicles within a zone under mutual influence. The comprehensive predicted state information of vehicles within each zone is processed through the second fully connected layer and the second activation layer of the multimodal fusion prediction sub-model to obtain the road state prediction feature information of each zone.

[0011] According to the present invention, the training steps of the traffic safety monitoring model include: acquiring training videos of the monitoring area, and acquiring training vehicle modal data and training road modal data; processing the training vehicle modal data through a vehicle modal feature extraction sub-model to obtain training vehicle state feature information for each vehicle; processing the training road modal data through a road modal feature extraction sub-model to obtain training road state feature information for each zone; processing the training vehicle state feature information and training road state feature information through a multimodal fusion prediction sub-model to obtain training fusion state feature information and training road state prediction feature information; decoding the training vehicle state feature information to obtain the training predicted vehicle motion state; determining the motion loss function based on the training vehicle modal data acquired in the training video at the next moment and the training predicted vehicle motion state; and decoding the training road state feature information to obtain... The training predicts the occupancy status of each zone; based on the training road modal data and training predictive occupancy status obtained from the training video at the next time step, the zone occupancy loss function is determined; the training fusion state feature information is decoded to obtain the training occupancy status information of each vehicle for each zone; based on the training occupancy status information and the training road modal data at the next time step obtained from the training video, the vehicle occupancy loss function is determined; the training target zone in the training video where there is a vehicle emergency action at the next time step is identified; based on the training target zone and the training road state prediction feature information, the zone safety risk loss function is determined; based on the motion loss function, zone occupancy loss function, vehicle occupancy loss function, and zone safety risk loss function, the loss function of the traffic safety monitoring model is determined; based on the loss function of the traffic safety monitoring model, the traffic safety monitoring model is trained to obtain the trained traffic safety monitoring model.

[0012] According to the present invention, the partition numbers are set sequentially according to the road direction. Based on the training occupancy information and the training road modal data for the next time step obtained from the training video, a vehicle occupancy loss function is determined, including: determining a penalty term using an index function, the proportion of vehicles in each partition determined based on the training occupancy information, and the proportion of vehicles in each partition determined based on the training road modal data for the next time step; and determining the penalty term vehicle occupancy loss function based on the penalty term, the proportion of vehicles in each partition determined based on the training occupancy information, and the proportion of vehicles in each partition determined based on the training road modal data for the next time step.

[0013] According to a second aspect of the present invention, a highway traffic safety monitoring system based on intelligent fusion of multi-source data is provided, comprising: an acquisition module, configured to capture monitoring video of a monitoring area using a camera, and acquire vehicle modal data and road modal data, wherein the road modal data is used to represent the vehicle occupancy status of multiple zones of multiple roads in the monitoring area, and the vehicle modal data is used to represent the movement status of multiple vehicles in the monitoring area; a vehicle state feature information module, configured to process the vehicle modal data of multiple vehicles using a vehicle modal feature extraction sub-model of a trained traffic safety monitoring model to obtain vehicle state feature information of each vehicle; and a road state feature information module, configured to process the road modal data using a road modal feature extraction sub-model of a trained traffic safety monitoring model. The system comprises the following modules: a road condition feature module, an association information module, and a road condition prediction feature information module. The road condition prediction feature information module is used to determine the association information of each vehicle based on vehicle modal data, wherein the association information describes the mutual influence between vehicles. The road condition prediction feature information module is used to obtain road condition prediction feature information based on the vehicle condition feature information, road condition feature information, association information, and the multimodal fusion prediction sub-model of the trained traffic safety monitoring model. The safety risk information module is used to determine the safety risk information of multiple zones on multiple roads based on the road condition prediction feature information. The judgment module is used to determine whether there are zones on multiple roads where the safety risk information exceeds a preset risk threshold. The prompting module is used to generate traffic safety monitoring prompt information based on the safety risk information if such a zone exists.

[0014] Technical Effects: According to the present invention, data from different modalities can be extracted from camera monitoring videos. Then, based on a traffic safety monitoring model, intelligent fusion processing of the data from different modalities is performed. Considering road conditions, vehicle conditions, and the mutual influence between vehicles, the safety risks in various sections of the road can be predicted, and timely warnings can be issued. This allows for timely and comprehensive monitoring of risks in the area and timely implementation of necessary measures to reduce potential safety hazards, improving monitoring accuracy and reducing the impact of safety risks. Furthermore, when determining the motion state and trend of a vehicle, the influence of its neighboring vehicles can be considered. Therefore, when using a self-attention mechanism for calculation, the correlation between distant vehicles is actively ignored. This not only reduces computational load but also reduces information noise interference from distant vehicles, improving the accuracy of calculating the motion state and trend of each vehicle under the influence of neighboring vehicles. Furthermore, cross-channel feature fusion of each road state feature information can be performed through a 1×1 convolutional layer to enhance feature representation capabilities. The U-NET encoding layer further integrates the feature vectors describing the state of multiple zones across zones, considering the interrelationships and mutual influences of changing trends between different zones during the fusion process. This allows a single vector to represent the overall situation of multiple zones, and the overall occupancy trend is calculated based on the road time-series prediction layer. After decoding, the changing trend of vehicle occupancy in each zone can be obtained, improving the accuracy of feature representation of road state information. Moreover, when determining the correlation between two vehicles, the mutual influence between the two vehicles at the current monitoring time is considered, and the potential future mutual influence can be determined by the distance and time difference between simplified routes. This makes the calculation of correlations more accurate and comprehensive. Furthermore, when determining the future movement trends and risks of two vehicles, combining potential future mutual influences improves the matching degree between data, enhancing the accuracy of movement trend prediction and risk identification. Furthermore, cross-attention mechanisms and graph models can be used to fuse the movement trends of vehicles interacting with each other and the occupancy trends of lanes. For each lane, the trend of vehicle occupancy within that lane can be analyzed to determine whether there are safety risks arising from the interaction of vehicles within that lane. Information from multiple modalities can be integrated during the analysis, improving the comprehensiveness and completeness of the information. Analyzing the vehicle movement within a lane can enhance the targeting and accuracy of safety risk prediction, facilitating safety risk monitoring. During model training, specific settings for lane numbers can be used to design penalty terms in the loss function corresponding to each vehicle. This allows for setting larger penalties for more serious situations such as lane prediction errors, thereby specifically increasing the training intensity and improving training efficiency and the accuracy of the fused state feature information.

[0015] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Other features and aspects of the invention will become clearer from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a highway traffic safety monitoring method based on intelligent fusion of multi-source data according to an embodiment of the present invention is shown as an example;

[0018] Figure 2 A schematic diagram illustrating partitioning according to an embodiment of the present invention is shown exemplarily;

[0019] Figure 3 A schematic diagram of a first proportion according to an embodiment of the present invention is shown exemplarily;

[0020] Figure 4 A schematic diagram of a traffic safety monitoring model according to an embodiment of the present invention is shown as an example;

[0021] Figure 5 An exemplary schematic diagram of the partition numbering according to an embodiment of the present invention is shown;

[0022] Figure 6 An exemplary training comparison diagram is shown according to an embodiment of the present invention;

[0023] Figure 7 An exemplary embodiment of the highway traffic safety monitoring system based on intelligent fusion of multi-source data according to an embodiment of the present invention is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0026] Figure 1 An exemplary flowchart illustrates a highway traffic safety monitoring method based on intelligent fusion of multi-source data according to an embodiment of the present invention, the method comprising:

[0027] Step S1: Capture monitoring video of the monitoring area using a camera, and acquire vehicle modal data and road modal data. The road modal data is used to represent the vehicle occupancy status of multiple zones of multiple roads in the monitoring area, and the vehicle modal data is used to represent the movement status of multiple vehicles in the monitoring area.

[0028] Step S2: The vehicle modal feature extraction sub-model of the trained traffic safety monitoring model is used to process the vehicle modal data of multiple vehicles to obtain the vehicle state feature information of each vehicle.

[0029] Step S3: The road modal feature extraction sub-model of the trained traffic safety monitoring model is used to process the road modal data and obtain road state feature information.

[0030] Step S4: Determine the association information of each vehicle based on the vehicle modal data, wherein the association information is used to describe the mutual influence between the vehicles.

[0031] Step S5: Based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model, obtain road state prediction feature information;

[0032] Step S6: Based on the road condition prediction feature information, determine the safety risk information of multiple zones for multiple roads;

[0033] Step S7: In multiple sections of multiple roads, determine whether there are sections with safety risk information higher than a preset risk threshold.

[0034] Step S8: If the risk exists, generate traffic safety monitoring alert information based on the safety risk information.

[0035] According to an embodiment of the present invention, the highway traffic safety monitoring method based on multi-source data intelligent fusion can extract data of different modalities through camera monitoring videos, and then perform intelligent fusion processing on the data of different modalities based on a traffic safety monitoring model. Taking into account the road conditions, the condition of the vehicles themselves, and the mutual influence between vehicles, the method can predict the safety risks existing in each zone of the road and issue timely warnings. It can monitor the risks in the area in a timely and comprehensive manner, and take necessary measures in a timely manner to reduce the hidden dangers caused by safety risks, thereby improving the monitoring accuracy and reducing the impact of safety hazards.

[0036] According to one embodiment of the present invention, in step S1, the monitoring area captured by the camera can be a section of a highway, for example, a section of a three-lane road in one direction, and multiple vehicles are traveling on the road section. The road modal data can be obtained from the video frames of the monitoring video, and the vehicle modal data can also be obtained from the video frames of the monitoring video, or can be obtained through other devices such as speed measuring radar. The present invention does not limit this.

[0037] According to one embodiment of the present invention, a monitoring video of a monitoring area is captured by a camera, and vehicle modal data and road modal data are obtained, including: setting vehicle identification information for each vehicle; determining the position information, speed information, and acceleration information of each vehicle based on the monitoring video frame corresponding to the current monitoring time and its adjacent video frames; stitching the vehicle identification information, position information, speed information, and acceleration information together to obtain vehicle modal data; obtaining the boundary position information of each partition; determining the vehicle identification information within each partition, and the first ratio of a first area of ​​the portion of the vehicle within the partition to a second area of ​​the region where the vehicle is located in the monitoring video frame corresponding to the current monitoring time; and obtaining road modal data for each partition based on the boundary position information, the vehicle identification information, and the first ratio.

[0038] According to one embodiment of the present invention, the vehicle identification information for each vehicle may be the vehicle's license plate number or descriptive information about its appearance, such as black, sedan, brand "XX", etc., and this information is converted into a specific code as vehicle identification information. The present invention does not limit the method of setting the vehicle identification information.

[0039] According to one embodiment of the present invention, the time interval between adjacent monitoring moments can be 1 second, 0.5 seconds, etc. Multiple monitoring video frames can be captured between adjacent monitoring moments. The position information of each vehicle can be determined based on the monitoring video frame corresponding to the current monitoring moment. For example, the coordinate position of the center point within the vehicle's outline in the monitoring video frame. Alternatively, the coordinate position in the monitoring video frame can be converted into actual geographic coordinates using camera calibration data. Vehicle speed and acceleration information can also be determined through multiple consecutive adjacent video frames. For example, the ratio of the position difference between the vehicle's position coordinates in adjacent video frames to the time difference between adjacent video frames can be used as speed information. This can include speed information in the x-direction and speed information in the y-direction. For instance, the ratio of the position difference between the monitoring video frame corresponding to the current monitoring moment and the previous monitoring video frame to the time difference between adjacent video frames can be used as the vehicle's speed information in the monitoring video frame corresponding to the current monitoring moment. Acceleration information can also be determined by changes in velocity information in consecutive adjacent video frames. This can include acceleration information in the x-direction and acceleration information in the y-direction. For example, the ratio of the difference between the vehicle's velocity information in the current monitoring video frame and the vehicle's velocity information in the previous monitoring video frame to the time difference between adjacent video frames can be used as the vehicle's acceleration information in the current monitoring video frame. Alternatively, as mentioned above, parameters such as velocity information can also be obtained through other methods such as radar speed measurement, and this invention does not limit this.

[0040] According to one embodiment of the present invention, the above vehicle identification information, location information, speed information and acceleration information are concatenated to obtain a vector for describing the vehicle's motion state, which is the vehicle modal data.

[0041] Figure 2 A schematic diagram of partitioning according to an embodiment of the present invention is shown as an example.

[0042] According to one embodiment of the present invention, the monitoring area can be divided into multiple zones. For example, along the direction of a road, the zone can be divided every 10 meters; along a perpendicular line from the road, the zone can be divided according to lanes, thus obtaining multiple zones. Furthermore, the boundary position information of each zone can be marked in the monitoring video frame; for example, the coordinate positions of the four corner points of each zone in the monitoring video frame can be marked. The present invention does not limit the method of zone division.

[0043] According to one embodiment of the present invention, a partition may accommodate multiple vehicles at the same time, and some vehicles (e.g., vehicles changing lanes) are not entirely located within a partition. Therefore, the identification information of vehicles within a partition and the first ratio of the first area of ​​the part of the vehicle within the partition to the second area of ​​the area where the vehicle is located can be statistically analyzed.

[0044] Figure 3A schematic diagram of a first proportion according to an embodiment of the present invention is shown as an example.

[0045] According to one embodiment of the present invention, vehicle 1 is entirely located within partition 1, while vehicle 2 is only partially located within partition 1, with the remainder located in other partitions. Therefore, when determining the road modal data of partition 1, the information related to vehicle 1 may include vehicle identification information and a first percentage (e.g., 1) of vehicle 1, and the information related to vehicle 2 may include vehicle identification information and a first percentage (e.g., 0.25) of vehicle 2. In the example, when determining the road modal data of a partition, the vehicle identification information and first percentage of a preset number of vehicles with the highest first percentage can be determined. For example, the vehicle identification information and first percentage of the top 3 vehicles with the highest first percentage can be determined. If the number of vehicles in the partition is less than the preset number, zeros can be added to the missing positions. For example, if the preset number is 3, the information related to the 3 vehicles in partition 1 includes the vehicle identification information and first percentage of vehicle 1, the vehicle identification information and first percentage of vehicle 2, and two zeros, totaling 6 bits of data. By concatenating the boundary location information (e.g., the coordinates of the four corner points) with the vehicle identification information of a preset number of vehicles and their first proportion, a vector describing the vehicle occupancy status of the partition can be obtained, that is, the road modal data of the partition.

[0046] Figure 4 A schematic diagram of a traffic safety monitoring model according to an embodiment of the present invention is shown as an example.

[0047] According to one embodiment of the present invention, in step S2, the traffic safety detection model can be a deep learning neural network model composed of multiple sub-models, which can be used to determine whether there is a safety risk in each zone. If a safety risk exists, not only can safety risk information be generated to alert staff, but the location of the zone with the risk can also be accurately located, facilitating timely action by staff.

[0048] According to one embodiment of the present invention, the traffic safety detection model may include a vehicle modal feature extraction sub-model, which can be used to obtain vehicle state feature information of each vehicle to describe the motion state and motion trend of the vehicle.

[0049] According to one embodiment of the present invention, a vehicle modal feature extraction sub-model of a trained traffic safety monitoring model is used to process vehicle modal data of multiple vehicles to obtain vehicle state feature information of each vehicle. This includes: processing the vehicle modal data of multiple vehicles through a first feature extraction layer of the vehicle modal feature extraction sub-model to obtain vehicle motion state feature information corresponding to each vehicle; obtaining the nearest neighbor vehicles of each vehicle based on the vehicle modal data; processing the vehicle motion state feature information corresponding to each vehicle and its nearest neighbor vehicles through a first self-attention mechanism of the vehicle modal feature extraction sub-model to obtain first vehicle fusion feature information, wherein the first vehicle fusion feature information is used to describe the vehicle's own motion state and the impact of the motion state of the nearest neighbor vehicles on the vehicle; and decoding each first vehicle fusion feature information through a temporal decoding layer of the vehicle modal feature extraction sub-model to obtain vehicle state feature information of each vehicle, wherein the vehicle state feature information is used to describe the current motion state and motion trend of the vehicle, and the motion trend is used to predict the motion state at the next monitoring moment.

[0050] According to an embodiment of the present invention, in the vehicle modal feature extraction sub-model of the traffic safety monitoring model, vehicle modal data of multiple vehicles can be input into the first feature extraction layer. The first feature extraction layer may include multiple fully connected layers and ReLU or GELU activation layers, which can be used to process the vehicle modal data to obtain its feature representation, that is, vehicle motion state feature information, which is used to express the motion state of the vehicle from more dimensions.

[0051] According to one embodiment of the present invention, during vehicle operation, its motion may be affected by other vehicles. For example, if a vehicle in an adjacent lane intends to change into the vehicle's lane, the vehicle may slow down to make room for the other vehicle. In this process, the vehicle's deceleration is a result of the influence of other vehicles. However, the vehicle is not affected by all vehicles in the monitoring area, but only by nearby vehicles. In this example, it may be affected by vehicles within a 20-meter radius. Therefore, the nearest neighbor vehicles of each vehicle can be searched. For example, other vehicles within a circle with a preset radius centered on the vehicle can be considered as the vehicle's nearest neighbors. This circular range can be mapped to the monitoring video frame using the camera's calibration parameters, thereby defining a range within the monitoring video frame and searching for the vehicle's nearest neighbors within the monitoring video frame.

[0052] According to one embodiment of the present invention, the first self-attention mechanism differs from the self-attention mechanism in related technologies. While the self-attention mechanism in related technologies calculates the association weights between all input vectors, the first self-attention mechanism calculates only the association weights between each vehicle and its nearest neighbors based on the search results of nearby vehicles within the aforementioned range. In the example, vehicle motion state feature information can be multiplied with the query matrix of the first self-attention mechanism to obtain a query vector; vehicle motion state feature information can be multiplied with the key-value matrix of the first self-attention mechanism to obtain a key-value vector; and vehicle motion state feature information can be multiplied with the weight matrix of the first self-attention mechanism to obtain a weight vector. A vehicle's query vector is multiplied only with its own and the key-value vectors of its nearest neighbors to obtain its own association weight and its association weight with its nearest neighbors. After scaling and softmax activation processing, the association coefficients corresponding to the aforementioned association weights are obtained. The weight vectors of the vehicle itself and its nearest neighbors are then weighted and summed using the association coefficients to obtain the first vehicle fusion feature information of the vehicle. By processing the vehicle motion state feature information of each vehicle as described above, the first vehicle fusion feature information of each vehicle can be obtained, which can be used to describe the current motion state of the vehicle under the influence of neighboring vehicles.

[0053] According to one embodiment of the present invention, the temporal decoding layer can be a network layer based on LSTM or RNN, which can be used to process the current motion state of a vehicle to predict the motion state at the next monitoring moment. In the example, the first vehicle fusion feature information of a vehicle and the hidden state feature information of the vehicle at the previous moment can be simultaneously input into the temporal decoding layer, outputting the hidden state feature information of the vehicle at the current moment. After processing by a fully connected layer and an activation layer (e.g., a ReLU or GELU activation layer), vehicle state feature information is obtained, which can be used to describe the current motion state and motion trend of the vehicle. Based on the current motion state and motion trend of the vehicle, the motion state at the next monitoring moment can be predicted. If the current monitoring moment is the first moment, the hidden state feature information of the previous moment is an all-zero vector. By performing the above processing on the first vehicle fusion feature information of all vehicles, the vehicle state feature information of each vehicle can be obtained.

[0054] In this way, when determining the motion state and trend of a vehicle, the influence of its neighboring vehicles can be considered. Thus, when using the self-attention mechanism for calculation, the correlation between distant vehicles can be actively ignored. This not only reduces the amount of computation but also reduces the information noise interference from distant vehicles, thereby improving the accuracy of calculating the motion state and trend of each vehicle under the influence of its neighboring vehicles.

[0055] According to an embodiment of the present invention, in step S3, the traffic safety detection model may include a road modal feature extraction sub-model, which can be used to obtain road state feature information of each zone to describe the changing trend of vehicle occupancy in each zone.

[0056] According to an embodiment of the present invention, a road modal feature extraction sub-model of a trained traffic safety monitoring model processes road modal data to obtain road state feature information, including: combining the road state feature information corresponding to each partition according to the location information of each partition to obtain a road state feature information tensor; performing cross-channel feature fusion processing on the road state feature information tensor through a 1×1 convolutional layer of the road modal feature extraction sub-model to obtain a partition state description feature tensor, wherein the partition state description feature tensor is used to describe the vehicle occupancy status of each partition; and processing the partition state description feature tensor through an encoding layer of the road modal feature extraction sub-model to obtain a road state description vector, wherein the road state description vector is used to describe the vehicle occupancy status of each partition. The system describes the overall occupancy status of multiple zones within a monitoring area. Through the road time-series prediction layer of the road modal feature extraction sub-model, the road state description vector at the current monitoring time and the road state latent vector at the previous monitoring time are processed to obtain the road state latent vector at the current monitoring time. This latent vector describes the changing trend of occupancy status in each zone of the current road. Through the decoding layer of the road modal feature extraction sub-model, the latent vector is decoded to obtain road state feature information for each zone. This feature information describes the changing trend of vehicle occupancy status in each zone, and the changing trend of occupancy status in each zone is used to predict the vehicle occupancy status at the next monitoring time.

[0057] According to one embodiment of the present invention, the partitions are distributed in a matrix in the monitoring area. For example, the monitoring area includes 3 rows and 3 columns of partitions. The road condition feature information of each partition is a vector including multiple data. Therefore, the dimension of the road condition feature information tensor is M×N×C, where M is the number of rows of the partitions in the monitoring area, N is the number of columns of the partitions in the monitoring area, and C is the number of data included in the road condition feature information of each partition.

[0058] According to one embodiment of the present invention, a 1×1 convolutional layer can perform cross-channel feature fusion processing on features from different channels. That is, it performs feature fusion processing on multiple data in the road state feature information of each partition to obtain the partition state description feature vector corresponding to each partition. This allows for a more in-depth description of the state features of each partition through more dimensions. The partition state description feature vectors of each partition are combined to obtain a partition state description feature tensor. The dimension of the partition state description feature tensor is M×N×B, where B is the number of data included in the partition state description feature vector.

[0059] According to an embodiment of the present invention, the encoding layer of the road modal feature extraction sub-model can be the encoding layer of a U-NET-based neural network model. The encoding layer may include convolutional layers, pooling layers and activation layers, which can process the partition state description feature tensor to obtain the road state description vector. That is, the partition state description feature vectors of multiple partitions can be extracted and fused, and the resulting road state description vector can be used to describe the overall occupancy status of multiple partitions.

[0060] According to one embodiment of the present invention, the road time series prediction layer can be a network layer based on LSTM or RNN, which can process the road state description vector at the current monitoring time and the road state latent vector at the previous monitoring time to obtain the road state latent vector at the current monitoring time, so as to describe the changing trend of the occupancy status of each zone in the current road.

[0061] According to one embodiment of the present invention, the decoding layer of the road modal feature extraction sub-model can be the decoding layer of U-NET. The decoding layer can decode the road state latent vector and the feature map of the same size that is connected to the input of the encoding layer to obtain a road state feature information tensor, which includes road state feature information for each partition, and can be used to describe the changing trend of vehicle occupancy in each partition. The dimension of the road state feature information tensor is M×N×A, where A is the number of data included in the road state feature information.

[0062] In this way, cross-channel feature fusion of each road state feature information can be performed through a 1×1 convolutional layer to improve feature representation capability. The U-NET encoding layer can be used to perform cross-region feature fusion of the partition state description feature vectors of multiple partitions. During the fusion process, the interrelationship and mutual influence of the changing trends between different partitions are considered so that the overall status of multiple partitions can be expressed by a single vector. Then, the overall occupancy change trend can be calculated based on the road time series prediction layer. After decoding, the vehicle occupancy change trend of each partition can be obtained, thereby improving the accuracy of feature representation of road state feature information.

[0063] According to one embodiment of the present invention, in step S4, the correlation information between vehicles can be determined based on vehicle modal data. That is, from the perspective of vehicle position and movement, the actual correlation relationship between vehicles and the actual mutual influence between vehicles can be determined. This differs from the above-mentioned search for neighboring vehicles, which aims to determine the impact of a vehicle's movement state on its neighboring vehicles. Here, the correlation information is used to represent the magnitude of the influence between vehicles, thereby calculating how the vehicle's movement trend will change and how its occupancy of each partition will change in subsequent processing.

[0064] According to one embodiment of the present invention, determining the association information of each vehicle based on vehicle modal data includes: obtaining position information, velocity information, and acceleration information of each vehicle based on the vehicle modal data; obtaining a predicted motion path for each vehicle based on the position information, velocity information, and acceleration information of each vehicle; sampling along the predicted path to obtain multiple predicted sampling positions corresponding to each vehicle; obtaining the minimum distance between the multiple predicted sampling positions of the i-th vehicle and the multiple predicted sampling positions of the j-th vehicle; determining current mutual influence parameters based on the position information of the i-th and j-th vehicles; determining future predicted mutual influence parameters based on the minimum distance; and determining the association information of the i-th and j-th vehicles based on the current mutual influence parameters and the future predicted mutual influence parameters.

[0065] According to one embodiment of the present invention, position information, velocity information, and acceleration information of each vehicle can be extracted from the modal data of each vehicle. Based on this information, existing influences and potential future influences between vehicles can be determined, thereby determining the correlation information between the vehicles. The predicted motion path of a vehicle can be a virtual path of the vehicle while maintaining its velocity and acceleration at the current monitoring moment and moving in a straight line to the next monitoring moment. This can be used to describe a simplified path of the vehicle between two monitoring moments without being affected. If the simplified path of one vehicle is too close to the simplified paths of other vehicles, or even intersects, the two vehicles may influence each other in the future. Existing influences can be determined based on the positional relationship of the vehicles at the current monitoring moment.

[0066] According to one embodiment of the present invention, sampling is performed on the predicted route (e.g., the time interval between two sampling points is 0.1 seconds) to obtain multiple predicted sampling positions corresponding to each vehicle, and the minimum distance between the multiple predicted sampling positions of the i-th vehicle and the multiple predicted sampling positions of the j-th vehicle can be determined. This minimum distance can be approximated as the shortest distance between two predicted motion routes.

[0067] According to an embodiment of the present invention, the association information between the i-th vehicle and the j-th vehicle is determined according to formula (1). ,

[0068] (1)

[0069] in, As a preset safe distance, Let be the distance between the i-th vehicle and the j-th vehicle at the current monitoring time. Let be the minimum distance between multiple predicted sampling locations of vehicle i and multiple predicted sampling locations of vehicle j. It is the maximum value among the minimum distances between multiple predicted sampling locations for each vehicle. This is the minimum of the minimum distances between multiple predicted sampling locations for each vehicle. To determine the time difference between the predicted sampling position of the i-th vehicle and the predicted sampling position of the j-th vehicle corresponding to the minimum distance, This represents the time difference between adjacent monitoring times.

[0070] According to an embodiment of the present invention, in formula (1), the associated information It is a dimensionless coefficient. The larger the value, the stronger the mutual influence (or comprehensive risk) between the i-th vehicle and the j-th vehicle. This correlation information can include both the spatial proximity at the current time and the potential impact at future times.

[0071] According to one embodiment of the present invention, The ratio between the preset safe distance and the distance between the i-th vehicle and the j-th vehicle at the current monitoring time can be used as the current mutual influence parameter. The larger the ratio, the smaller the distance between the i-th vehicle and the j-th vehicle, even smaller than the safe distance. Therefore, the greater the mutual influence between the two at the current monitoring time.

[0072] According to one embodiment of the present invention, This represents the normalization of the shortest distance between the future simplified routes of vehicle i and vehicle j. The larger the value, the smaller the distance between their simplified routes, and the greater the possibility that they will influence each other in the future. Conversely, the smaller the value, the larger the distance between their simplified routes, and the less likely that the two vehicles will change their simplified routes in the future due to their mutual influence. Therefore, the possibility that they will influence each other in the future is also smaller. This is the time difference between the closest sampling points on the simplified routes of the two vehicles. For example, if vehicle i arrives at sampling point 2 at 0.2 seconds and vehicle j arrives at sampling point 5 at 0.5 seconds, and the distance between sampling point 2 on vehicle i's simplified route and sampling point 5 on vehicle j's simplified route is the shortest, meaning the two simplified routes reach their shortest distance at these two points, then the time difference of 0.3 seconds between 0.5 seconds and 0.2 seconds is 0.3 seconds. The larger the time difference, the greater the time difference between the two vehicles at the closest point of their simplified routes. In other words, the greater the time difference between the two vehicles reaching the closest point, the greater the actual distance they travel, and the smaller the mutual influence between them. for and The ratio, that is, The ratio of the monitoring period to half the monitoring cycle duration. If this ratio is less than 1, it indicates that the mutual influence between the two vehicles is significant within a monitoring cycle, and there is even a risk of collision. Conversely, if the ratio is greater than 1, the distance between the two vehicles is relatively large within a monitoring cycle (i.e., the distance between the two vehicles is not expected to shrink to a very small extent within a monitoring cycle), the risk of collision is low, and the mutual influence is also small. Therefore, a ratio greater than 1 can be considered... As the denominator, and As a discount factor for the risk of future mutual influence or collision, The larger, the better The smaller the value, the less likely the two vehicles are to affect each other in the future, and the lower the risk from the current perspective. Therefore, a smaller discount factor can be assigned. This can represent the potential future interaction between two vehicles as observed from the current perspective, which is the future predicted interaction parameter. The current interaction... Interaction with the future as observed from the current perspective Averaging yields the correlation information between the i-th and j-th vehicles, representing the combined influence between them; that is, the combined influence coefficient considering both the current and future monitoring times. Although both use the same weights... An average was calculated, but due to the discount factor... The existence of makes the total weight of future mutual influences . The weight of the mutual influence at the current monitoring time is less than that of the current monitoring time. In other words, the current mutual influences are of greater importance in determining the overall influence. The aforementioned overall influence can then be used as the correlation information between the i-th vehicle and the j-th vehicle. The above calculations can be performed on each vehicle to obtain the correlation information between them.

[0073] In this way, when determining the relationship between two vehicles, not only can the mutual influence between the two vehicles at the current monitoring time be considered, but also the possible mutual influence between the two vehicles in the future can be determined by the distance and time difference between the simplified routes. This makes the calculation of the relationship more accurate and comprehensive. Furthermore, when determining the future movement trend and risk of two vehicles, combining the possible mutual influence in the future can make the matching degree between the data higher, thereby improving the accuracy of movement trend prediction and risk identification.

[0074] According to an embodiment of the present invention, in step S5, the multimodal fusion prediction sub-model can fuse vehicle state feature information of multiple vehicles and road state feature information of multiple zones, and analyze the fused information to analyze the probability of safety risks in each zone by combining the motion state of vehicles under mutual influence, that is, the possibility of a safety risk event occurring in each zone.

[0075] According to an embodiment of the present invention, road state feature information is obtained based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model. This includes: processing the vehicle state feature information and road state feature information through a first cross-attention mechanism of the multimodal fusion prediction sub-model to obtain first fused state feature information for each vehicle, wherein the first fused state feature information represents the occupancy trend of each vehicle in each zone, and the occupancy trend is used to predict the occupancy status of each vehicle in each zone at the next monitoring time; processing the first fused state feature information and correlation information of each vehicle through the graph model processing hierarchy of the multimodal fusion prediction sub-model to obtain second fused state feature information for each vehicle, wherein... The second fused state feature information is used to represent the occupancy trend of each vehicle in each zone under the influence of other vehicles. The second fused state feature information of each vehicle is processed through the first fully connected layer and the first activation layer of the multimodal fusion prediction sub-model to obtain the predicted occupancy status information of each vehicle in each zone. The second fused state feature information is weighted and summed using the predicted occupancy status information of each vehicle in each zone to obtain the comprehensive predicted state information of vehicles in each zone. This comprehensive predicted state information of vehicles is used to describe the predicted motion state information under the mutual influence of all vehicles in a zone. The comprehensive predicted state information of vehicles in each zone is processed through the second fully connected layer and the second activation layer of the multimodal fusion prediction sub-model to obtain the road state prediction feature information of each zone.

[0076] According to one embodiment of the present invention, vehicle state feature information and road state feature information can be fused through a first cross-attention mechanism to obtain first fused state feature information for each vehicle. This information can be used to represent the occupancy trend of each vehicle in each zone, that is, the movement trend of each vehicle between zones, providing a data basis for determining the safety risks within a zone. For example, based on the first fused state feature information of two vehicles, if it is determined that the first proportion of both vehicles in the same zone is likely to increase rapidly, then the probability of a safety risk occurring in a relatively small zone is higher.

[0077] According to one embodiment of the present invention, the graph model processing layer can be used to further fuse the first fused state feature information, thereby representing the change in the vehicle's occupancy trend of a partition under the influence of the movement trends of other vehicles and their occupancy trends. In the example, the adjacency matrix can be determined through the association relationships between the various vehicles determined above, that is, the association relationships... The data in the i-th row and j-th column of the adjacency matrix is ​​identified. Further, the sum of multiple data points in each row of the adjacency matrix can be used as the values ​​on the diagonal of the degree matrix to obtain the degree matrix. After obtaining the adjacency matrix and degree matrix, the parameters of the graph model processing hierarchy, along with the adjacency matrix and degree matrix, can be used to process the first fused state feature information of each vehicle to obtain the second fused state feature information of each vehicle.

[0078] According to one embodiment of the present invention, after combining the influence of other vehicles and its own occupancy trend in a partition, the second fused state feature information can be used to predict the future occupancy status of vehicles in each partition. For example, after processing through a first fully connected layer and a first activation layer (e.g., a ReLU or GELU activation layer), the predicted occupancy status information of vehicles in each partition in the future can be obtained. Further, by weighting and summing the predicted occupancy status information of each vehicle for each partition, the comprehensive predicted state information of vehicles in each partition can be obtained. For example, for partition 1, the second fused state feature information can be weighted by the predicted occupancy status information of the first vehicle for partition 1, the second fused state feature information can be weighted by the predicted occupancy status information of the second vehicle for partition 1, and so on, and the weighted results can be summed to obtain the comprehensive predicted state information of vehicles in partition 1 in the future. Similarly, similar processing can be performed on other partitions to determine the predicted occupancy trend of each vehicle for each partition. Thus, the occupancy trend changes of multiple vehicles for the same partition can be obtained.

[0079] According to one embodiment of the present invention, since the second fused state feature information contains both the vehicle's own motion trend information and its occupancy trend information for the same partition, the weighted summation of the vehicle's comprehensive predicted state information for a single partition can be used to represent the vehicle's motion trend and occupancy status changes within that partition. This can be used to determine whether there is a risk within that partition. For example, if two vehicles both show an increasing occupancy trend for the same partition, and the relative speed of the two vehicles is relatively fast, then the risk of a safety accident involving the two vehicles within that partition is relatively high. By processing the comprehensive predicted state information of the vehicles within each partition through the second fully connected layer and the second activation layer (RELU or GELU activation layer), the road state predicted feature information for that partition can be obtained. This information can be used to predict the comprehensive motion status of vehicles within that partition, thereby determining whether there is a safety risk within that partition.

[0080] In this way, the movement trends of vehicles interacting with each other and the occupancy trends of zones can be fused using cross-attention mechanisms and graph models. For each zone, the trend of vehicle occupancy in that zone can be analyzed to determine whether there are safety risks arising from the mutual movement of vehicles within that zone. Information from multiple modalities can be integrated during the analysis to improve the comprehensiveness and completeness of the information. Furthermore, analyzing the vehicle movement status within a zone can improve the targeting and accuracy of safety risk prediction, thus facilitating safety risk supervision.

[0081] According to an embodiment of the present invention, in step S6, the road state prediction feature information of each partition can be input into the fully connected layer and the activation layer (e.g., the sigmoid activation layer) for processing to obtain the security risk information of each partition, that is, the probability information of the existence of security risks in each partition.

[0082] According to one embodiment of the present invention, in step S7, a preset risk threshold can be set, for example, 0.5, and it is determined whether there are zones with safety risk information higher than the preset risk threshold. In step S8, traffic safety monitoring prompt information is generated based on the safety risk information. For example, the traffic safety monitoring prompt information may include the location of the zone and the specific value of the safety risk information of that zone. Further, the safety monitoring prompt information can be further analyzed. For example, if the frequency of safety monitoring prompt messages corresponding to a certain zone is high, such as frequent occurrences of vehicles illegally changing lanes causing other vehicles to brake suddenly, or vehicles decelerating abnormally leading to an increased probability of rear-end collisions, resulting in consistently high safety risk information, then the area near that zone can be under focused monitoring. For example, the reasons for the frequent occurrence of safety risks near that zone can be analyzed, such as signs being obscured by trees or other obstructions making them difficult for drivers to see, or unclear road guide lines. Necessary measures can be taken to address these reasons, such as adding warning signs, removing obstructions, or repainting lines, thereby guiding vehicles to reduce violations at that location.

[0083] According to an embodiment of the present invention, the traffic safety monitoring model described above can be trained before use. The training steps of the traffic safety monitoring model include: acquiring training videos of the monitoring area, and acquiring training vehicle modal data and training road modal data; processing the training vehicle modal data through a vehicle modal feature extraction sub-model to obtain training vehicle state feature information for each vehicle; processing the training road modal data through a road modal feature extraction sub-model to obtain training road state feature information for each zone; processing the training vehicle state feature information and training road state feature information through a multimodal fusion prediction sub-model to obtain training fusion state feature information and training road state prediction feature information; decoding the training vehicle state feature information to obtain the training predicted vehicle motion state; determining the motion loss function based on the training vehicle modal data acquired in the training video at the next moment and the training predicted vehicle motion state; and converting the training road... The state feature information is decoded to obtain the training predicted occupancy status of each zone; based on the training road modal data and training predicted occupancy status obtained from the training video at the next time step, the zone occupancy loss function is determined; the training fusion state feature information is decoded to obtain the training occupancy status information of each vehicle for each zone; based on the training occupancy status information and the training road modal data at the next time step obtained from the training video, the vehicle occupancy loss function is determined; the training target zone in the training video where there is a vehicle emergency action at the next time step is identified; based on the training target zone and the training road state prediction feature information, the zone safety risk loss function is determined; based on the motion loss function, zone occupancy loss function, vehicle occupancy loss function, and zone safety risk loss function, the loss function of the traffic safety monitoring model is determined; based on the loss function of the traffic safety monitoring model, the traffic safety monitoring model is trained to obtain the trained traffic safety monitoring model.

[0084] According to one embodiment of the present invention, the monitoring area during the training process may be the same as or different from the monitoring area in actual use, and the present invention does not impose any limitation thereon. Training vehicle modal data and training road modal data can be obtained in a manner similar to that described above for obtaining vehicle modal data and road modal data. Furthermore, training vehicle state feature information, training road state feature information, training fusion state feature information, and training road state prediction feature information can be obtained respectively in a manner similar to that described above for obtaining vehicle state feature information, road state feature information, second fusion state feature information, and road state prediction feature information; further details will not be elaborated here.

[0085] According to an embodiment of the present invention, as mentioned above, vehicle state feature information can describe the motion trend of a vehicle. Therefore, training vehicle state feature information obtained in the same way can also be used to describe the motion trend of a vehicle and predict the motion state of the vehicle at the next moment. However, the prediction result may contain errors. The training vehicle state feature information is decoded (e.g., processed through fully connected layers and ReLU or GELU activation layers), and converted into the training predicted vehicle motion state for the next time step. This is then compared with the training vehicle modal data for the next time step obtained from the training video (which describes the vehicle's actual motion state at the next time step) to determine the motion loss function. For example, each predicted data point in the training predicted vehicle motion state of each vehicle in the video frame (e.g., predicted position, predicted velocity, and predicted acceleration) can be compared with the corresponding real data in the vehicle's actual motion state at the next time step to obtain the mean square error between multiple predicted data points and real data for multiple vehicles. This mean square error is used as the motion loss function, thereby reducing the motion loss function during training and making the predicted data closer to the real data, thus improving the accuracy of motion state prediction. Using an independent motion loss function specifically enhances the training strength of the vehicle modal feature extraction sub-model, thereby improving the accuracy of the training vehicle state feature information and providing more accurate basic information for subsequent fusion calculations.

[0086] According to one embodiment of the present invention, as described above, road state feature information is used to predict the changing trend of occupancy status in different zones. Therefore, training road state feature information obtained in the same manner can also be used to describe the changing trend of occupancy status in each zone, thereby predicting the occupancy status of each zone at the next time step. However, this prediction result may contain errors. The training road state feature information is decoded (e.g., processed through a fully connected layer and a ReLU or GELU activation layer) to convert it into the training predicted occupancy status of each zone at the next time step. This is then compared with the training road modal data for the next time step obtained from the training video (which can describe the actual occupancy status of each zone at the next time step) to determine the zone occupancy loss function. For example, each data item of the training predicted occupancy status of each zone in the video frame (e.g., the identification information of vehicles occupying the zone and the predicted value of their first percentage) can be compared with the corresponding data in the training road modal data for the next time step (the actual value of the identification information of vehicles occupying the zone and their first percentage). The mean square error between the predicted values ​​and the actual values ​​of multiple data items for multiple zones can be obtained, which serves as the zone occupancy loss function. This allows the partition occupancy loss function to be reduced during training, making the predicted values ​​closer to the true values, thereby improving the prediction accuracy of partition occupancy status. Using an independent partition occupancy loss function specifically enhances the training strength of the road modal feature extraction sub-model, thereby improving the accuracy of training road state feature information and providing more accurate basic information for subsequent fusion calculations.

[0087] According to one embodiment of the present invention, as described above, the second fused state feature information is used to represent the occupancy trend of each vehicle in each zone under the influence of other vehicles, and can be used to predict the occupancy status of each vehicle in each zone at the next time step. Therefore, the training fused state feature information obtained in the same way can also be used to predict the occupancy status of each vehicle in each zone at the next time step, but the prediction result may contain errors. The training fused state feature information is decoded (e.g., processed through a fully connected layer and a ReLU or GELU activation layer), and converted into training occupancy status information of vehicles in each zone at the next time step. The training road modal data at the next time step is obtained to obtain the actual occupancy status information of each vehicle in each zone. Then, the vehicle occupancy loss function is determined based on the comparison between the training occupancy status information and the actual occupancy status information.

[0088] According to one embodiment of the present invention, the partition numbers are set sequentially according to the road direction. A vehicle occupancy loss function is determined based on training occupancy information and training road modal data for the next time step obtained from the training video. This includes: determining a penalty term using an index function, the proportion of vehicles in each partition determined based on the training occupancy information, and the proportion of vehicles in each partition determined based on the training road modal data for the next time step; and determining the penalty term vehicle occupancy loss function based on the penalty term, the proportion of vehicles in each partition determined based on the training occupancy information, and the proportion of vehicles in each partition determined based on the training road modal data for the next time step.

[0089] According to an embodiment of the present invention, the vehicle occupancy loss function is determined according to formula (2). ,

[0090] (2)

[0091] in, This refers to the percentage of the k-th vehicle that is located in the s-th partition, as determined based on the training occupancy information. This represents the percentage of the k-th vehicle located in the s-th partition, determined based on the training road modal data at the next time step, where n is the number of partitions and m is the number of vehicles. These are the preset parameters.

[0092] Figure 5 An exemplary schematic diagram of the partition numbering according to an embodiment of the present invention is shown.

[0093] According to one embodiment of the present invention, the partition numbers are sequentially set along the direction of the road. For example, the partition numbers in each lane increase sequentially in the opposite direction to the road direction, so that the partition number difference within the same lane is small, while the partition number difference between different lanes is large. That is, the partition number difference along the direction of the road is small; for example, the partition after partition 1 is partition 2, with a number difference of 1. However, in the direction perpendicular to the direction of the road, the partition number difference is large; for example, the partition to the right of partition 1 is partition 4, with a number difference of 3.

[0094] According to one embodiment of the present invention, For indexing functions, To determine the percentage of the k-th vehicle located in the s-th partition based on the training occupancy information, therefore... This is the partition number corresponding to the maximum percentage of the k-th vehicle in each partition. For example, if the k-th vehicle is in partition 1, its percentage in that partition is 60%, while its percentage in other partitions is less than that in partition 1. The value is the serial number of the partition corresponding to the maximum percentage, which is "1". This refers to the partition number corresponding to the maximum percentage determined based on training occupancy information (i.e., predicted values). This is the partition number corresponding to the maximum proportion determined based on the training road modality data (i.e., the true values) at the next time step. This indicates the difference between the two. This refers to the aforementioned penalty item. and If the values ​​are the same, it means that the multimodal fusion prediction sub-model is basically correct in predicting the occupancy trend of the vehicle in each zone. In this case, the value of the penalty term is 1, that is, no additional penalty is required.

[0095] According to one embodiment of the present invention, if and If they are different, there are two scenarios. If there is no error in predicting lane occupancy, that is, the zone number corresponding to the maximum proportion determined based on the predicted value and the zone number corresponding to the maximum proportion determined based on the actual value are in the same lane, then the difference between the two is small. For example, if there is an error in estimating the vehicle's speed, leading to an error in predicting the zone occupancy during the vehicle's movement, then the difference between the two is small, making the penalty term... A smaller value results in a smaller penalty value in the loss function; that is, for lane predictions that are correct, a smaller penalty value is used (but if it is greater than 1, there is an additional penalty). However, if a lane occupancy prediction error occurs, due to the partitioning method described above, the partition numbers on different lanes differ significantly, leading to a larger penalty value. A larger value means a larger penalty value in the loss function. That is, for more serious problems such as lane prediction errors, a larger penalty value is used to increase the training intensity.

[0096] According to one embodiment of the present invention, the loss value corresponding to the kth vehicle is ,in, This is a comparison between the predicted and actual values ​​of the proportion of the k-th vehicle in the s-th partition. To prevent decimals with a denominator or logarithmic term of 0, for example, 0.001, This is the sum of comparison terms between the predicted and actual values ​​of the k-th vehicle's proportion in each partition. During training, this term can be reduced, i.e., made smaller by ensuring that each... All are closer to 0, so that Closer to 1, in other words, makes Closer This improves the accuracy of training occupancy information, and consequently improves the accuracy of training fusion state feature information.

[0097] According to one embodiment of the present invention, the above-mentioned penalty item is... As The weights are calculated by summing the loss values ​​for each vehicle to obtain the vehicle occupancy loss function. This gives higher weight to lane prediction errors, thereby increasing training intensity, training efficiency, and the accuracy of training fusion state feature information.

[0098] In this way, by setting specific partition numbers, the penalty term of the loss function corresponding to the vehicle is designed, thereby setting a larger penalty term for more serious situations such as lane prediction errors, thus targeting and improving the training intensity, thereby improving training efficiency and the accuracy of training fusion state feature information.

[0099] According to one embodiment of the present invention, training target partitions where emergency vehicle actions are likely to occur in the next moment can be determined by methods such as manual annotation or video detection. For example, if a partition contains vehicles that are affected by other vehicles and perform emergency braking, lane changing, or even collisions, then that partition is the training target partition. For example, the training target partition is marked as 1, and the partitions where no emergency vehicle actions occur are marked as 0. If no emergency vehicle actions occur in any partition, then the markings of all partitions are set to 0.

[0100] According to one embodiment of the present invention, the training road state prediction feature information can be transformed into training safety risk information for multiple partitions in the same way as obtaining safety risk information for multiple partitions, that is, the probability of the existence of safety risks in each partition. A cross-entropy loss function is then constructed using the training safety risk information and the labels of each partition (the label of the training target partition is set to 1, and the label of partitions without vehicle emergency actions is set to 0). The partition safety risk loss function is obtained by summing the cross-entropy loss functions corresponding to each partition.

[0101] According to one embodiment of the present invention, the loss function of the traffic safety monitoring model can be obtained by weighted summing of the motion loss function, the zone occupancy loss function, the vehicle occupancy loss function, and the zone safety risk loss function. During training, the parameters of the traffic safety monitoring model can be adjusted using gradient descent to reduce the loss function, thereby reducing the overall loss function and improving the accuracy of the traffic safety monitoring model. Furthermore, each sub-model can be trained using individually set loss functions to further enhance its accuracy. After multiple training iterations until the model converges, a trained traffic safety monitoring model can be obtained and used in the aforementioned process of determining safety risk information for multiple zones.

[0102] According to an embodiment of the present invention, the traffic safety monitoring model includes a vehicle modal feature extraction sub-model, a road modal feature extraction sub-model, and a multimodal fusion prediction sub-model. The following ablation experiments were conducted in the absence of specific sub-models. Specifically, if the vehicle modal feature extraction sub-model is not used, a multilayer perceptron layer (e.g., a BP neural network layer composed of multiple fully connected layers and activation layers) is used to replace the vehicle modal feature extraction sub-model to obtain vehicle state feature information; if the road modal feature extraction sub-model is not used, a multilayer perceptron layer is used to replace the road modal feature extraction sub-model to obtain road state feature information; if the multimodal fusion prediction sub-model is not used, a basic transformer decoder is directly used to fuse vehicle state feature information into road state feature information to obtain road state prediction feature information for each zone, thereby determining the safety risk information for each zone. That is, in the ablation experiments, the vehicle modal feature extraction sub-model, road modal feature extraction sub-model, and multimodal fusion prediction sub-model of the present invention are compared with their respective basic models, and the results are shown in Table 1 below.

[0103] Table 1: Ablation Experiment Results of Traffic Safety Monitoring Model

[0104]

[0105] According to an embodiment of the present invention, as shown in Table 1, the results of the ablation experiment show that the vehicle modal feature extraction sub-model, the road modal feature extraction sub-model, and the multimodal fusion prediction sub-model all outperform their respective base models. The F1-score and AUC index are all higher than those in the case of missing any one or more sub-models. That is, the specially designed sub-models can bring a significant improvement in prediction accuracy compared to the base models.

[0106] According to one embodiment of the present invention, during the training process, a special setting method for the partition number and its corresponding vehicle occupancy loss function are used to adaptively adjust the penalty term and training intensity, which can also improve training efficiency and accuracy, as shown in Table 2 below. Figure 6 As shown:

[0107] Table 2: Comparison of Training Data

[0108]

[0109] Figure 6 An exemplary training comparison diagram is shown according to an embodiment of the present invention.

[0110] According to one embodiment of the present invention, Figure 6Table 2 describes the impact of whether or not the vehicle occupancy loss function is used on the partition safety risk loss function (i.e., the loss function that describes the accuracy of the final output of the model). Figure 6 The horizontal axis represents the training epoch, and the vertical axis represents the safety risk loss function value. When using the vehicle occupancy loss function, the partition safety risk loss function converges faster, yields a smaller converged loss function value, and the model's risk assessment for each partition is more accurate. Conversely, when using only the motion loss function, partition occupancy loss function, and partition safety risk loss function without the vehicle occupancy loss function, both the model's convergence speed and accuracy decrease. Therefore, using the vehicle occupancy loss function significantly improves both the model's training efficiency and accuracy.

[0111] The highway traffic safety monitoring method based on multi-source data intelligent fusion according to embodiments of the present invention can extract data of different modalities from camera monitoring videos, and then perform intelligent fusion processing on the data of different modalities based on a traffic safety monitoring model. By comprehensively considering road conditions, vehicle conditions, and mutual influences between vehicles, it predicts the safety risks existing in various sections of the road and issues timely warnings. This allows for timely and comprehensive monitoring of risks in the area and timely implementation of necessary measures to reduce potential safety hazards, improving monitoring accuracy and reducing the impact of safety risks. Furthermore, when determining the motion state and trend of a vehicle, the method considers the influence of its neighboring vehicles. Therefore, when using a self-attention mechanism for calculation, it actively ignores the correlation relationships of distant vehicles, which not only reduces computational load but also reduces information noise interference from distant vehicles, improving the accuracy of calculating the motion state and trend of each vehicle under the influence of neighboring vehicles. Furthermore, cross-channel feature fusion of each road state feature information can be performed through a 1×1 convolutional layer to enhance feature representation capabilities. The U-NET encoding layer further integrates the feature vectors describing the state of multiple zones across zones, considering the interrelationships and mutual influences of changing trends between different zones during the fusion process. This allows a single vector to represent the overall situation of multiple zones, and the overall occupancy trend is calculated based on the road time-series prediction layer. After decoding, the changing trend of vehicle occupancy in each zone can be obtained, improving the accuracy of feature representation of road state information. Moreover, when determining the correlation between two vehicles, the mutual influence between the two vehicles at the current monitoring time is considered, and the potential future mutual influence can be determined by the distance and time difference between simplified routes. This makes the calculation of correlations more accurate and comprehensive. Furthermore, when determining the future movement trends and risks of two vehicles, combining potential future mutual influences improves the matching degree between data, enhancing the accuracy of movement trend prediction and risk identification. Furthermore, cross-attention mechanisms and graph models can be used to fuse the movement trends of vehicles interacting with each other and the occupancy trends of lanes. For each lane, the trend of vehicle occupancy within that lane can be analyzed to determine whether there are safety risks arising from the interaction of vehicles within that lane. Information from multiple modalities can be integrated during the analysis, improving the comprehensiveness and completeness of the information. Analyzing the vehicle movement within a lane can enhance the targeting and accuracy of safety risk prediction, facilitating safety risk monitoring. During model training, specific settings for lane numbers can be used to design penalty terms in the loss function corresponding to each vehicle. This allows for setting larger penalties for more serious situations such as lane prediction errors, thereby specifically increasing the training intensity and improving training efficiency and the accuracy of the fused state feature information.

[0112] Figure 7 An exemplary embodiment of a highway traffic safety monitoring system based on intelligent fusion of multi-source data according to an embodiment of the present invention is shown, comprising:

[0113] The acquisition module is used to capture monitoring videos of the monitoring area through a camera and acquire vehicle modal data and road modal data. The road modal data is used to represent the vehicle occupancy status of multiple zones of multiple roads in the monitoring area, and the vehicle modal data is used to represent the movement status of multiple vehicles in the monitoring area.

[0114] The vehicle state feature information module is used to process the vehicle modal data of multiple vehicles through the vehicle modal feature extraction sub-model of the trained traffic safety monitoring model to obtain the vehicle state feature information of each vehicle.

[0115] The road condition feature information module is used to process road modal data and obtain road condition feature information through the road modal feature extraction sub-model of the trained traffic safety monitoring model.

[0116] The association information module is used to determine the association information of each vehicle based on the vehicle modal data, wherein the association information is used to describe the mutual influence between the vehicles.

[0117] The road condition prediction feature information module is used to obtain road condition prediction feature information based on the vehicle condition feature information, road condition feature information, and related information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model.

[0118] The safety risk information module is used to determine the safety risk information of multiple zones on multiple roads based on road condition prediction feature information;

[0119] The judgment module is used to determine whether there are any zones in multiple sections of multiple roads where the safety risk information is higher than a preset risk threshold.

[0120] The prompt module is used to generate traffic safety monitoring prompts based on safety risk information if such risks exist.

[0121] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0122] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and explained in the embodiments, and any variations or modifications may be made to the implementation of the present invention without departing from the stated principles.

Claims

1. A highway traffic safety monitoring method based on intelligent fusion of multi-source data, characterized in that, include: The system captures monitoring videos of the monitored area using cameras and acquires vehicle modal data and road modal data. The road modal data is used to represent the vehicle occupancy status of multiple zones on multiple roads within the monitored area, while the vehicle modal data is used to represent the movement status of multiple vehicles within the monitored area. The vehicle modal feature extraction sub-model of the trained traffic safety monitoring model is used to process the vehicle modal data of multiple vehicles to obtain the vehicle state feature information of each vehicle. By using the road modal feature extraction sub-model of the trained traffic safety monitoring model, road modal data is processed to obtain road state feature information; Based on vehicle modal data, the correlation information of each vehicle is determined, wherein the correlation information is used to describe the mutual influence between the vehicles. Based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model, road state prediction feature information is obtained. Based on road condition prediction feature information, safety risk information for multiple zones of multiple roads is determined; In multiple zones along multiple roads, determine whether there are zones with safety risk information exceeding a preset risk threshold; If present, traffic safety monitoring alerts will be generated based on the safety risk information.

2. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, The system captures monitoring video of the monitored area using cameras and acquires vehicle modal data and road modal data, including: Set vehicle identification information for each vehicle; Based on the monitoring video frame corresponding to the current monitoring time and its adjacent video frames, the position, speed and acceleration information of each vehicle are determined. Vehicle identification information, location information, speed information, and acceleration information are stitched together to obtain vehicle modal data; Obtain the boundary location information for each partition; Determine the vehicle identification information within each zone, and the first ratio of the first area of ​​the part of the vehicle within the zone to the second area of ​​the region where the vehicle is located in the monitoring video frame corresponding to the current monitoring time; Based on boundary location information, vehicle identification information, and the first proportion, road modal data for each zone are obtained.

3. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, The vehicle modal feature extraction sub-model of the trained traffic safety monitoring model processes the vehicle modal data of multiple vehicles to obtain the vehicle state feature information of each vehicle, including: The vehicle modal data of multiple vehicles are processed through the first feature extraction layer of the vehicle modal feature extraction sub-model to obtain the vehicle motion state feature information corresponding to each vehicle. Based on vehicle modal data, obtain the nearest neighbor vehicles for each vehicle; The first self-attention mechanism of the vehicle modal feature extraction sub-model is used to process the vehicle motion state feature information corresponding to each vehicle and its neighboring vehicles to obtain the first vehicle fusion feature information. The first vehicle fusion feature information is used to describe the motion state of the vehicle itself and the impact of the motion state of the neighboring vehicles on the vehicle. By using the temporal decoding layer of the vehicle modal feature extraction sub-model, the fused feature information of each first vehicle is decoded to obtain the vehicle state feature information of each vehicle. The vehicle state feature information is used to describe the current motion state and motion trend of the vehicle, and the motion trend is used to predict the motion state at the next monitoring moment.

4. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, The road modal feature extraction sub-model of the trained traffic safety monitoring model processes the road modal data to obtain road state feature information, including: Based on the location information of each partition, the road state feature information corresponding to each partition is combined to obtain the road state feature information tensor. By using the 1×1 convolutional layer of the road modal feature extraction sub-model, cross-channel feature fusion processing is performed on the road state feature information tensor to obtain the partition state description feature tensor, wherein the partition state description feature tensor is used to describe the vehicle occupancy status of each partition. By processing the partition state description feature tensor through the encoding level of the road modal feature extraction sub-model, a road state description vector is obtained. The road state description vector is used to describe the overall occupancy status of multiple partitions in the monitoring area. By extracting the road modal feature sub-model, the road time-series prediction level is obtained by processing the road state description vector at the current monitoring time and the road state latent vector at the previous monitoring time to obtain the road state latent vector at the current monitoring time. The road state latent vector at the current monitoring time is used to describe the changing trend of the occupancy status of each zone in the current road. By decoding the road state latent vector at the current monitoring time through the road modal feature extraction sub-model, the road state feature information of each zone is obtained. The road state feature information is used to describe the changing trend of vehicle occupancy in each zone, and the changing trend of vehicle occupancy in each zone is used to predict the vehicle occupancy at the next monitoring time.

5. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, Based on vehicle modal data, determine the correlation information of each vehicle, including: Based on vehicle modal data, obtain the position, speed, and acceleration information of each vehicle; Based on the position, speed, and acceleration information of each vehicle, the predicted trajectory of each vehicle is obtained. Sampling is performed on the predicted route to obtain multiple predicted sampling locations for each vehicle; Find the minimum distance between multiple predicted sampling positions of vehicle i and multiple predicted sampling positions of vehicle j; Based on the location information of the i-th vehicle and the j-th vehicle, determine the current mutual influence parameters; Based on the minimum distance, determine the parameters for future prediction mutual influence; Based on the current mutual influence parameters and the future predicted mutual influence parameters, the association information between the i-th vehicle and the j-th vehicle is determined.

6. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, Based on the vehicle state feature information, road state feature information, and correlation information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model, road state prediction feature information is obtained, including: The first cross-attention mechanism of the multimodal fusion prediction sub-model is used to process vehicle state feature information and road state feature information to obtain the first fused state feature information of each vehicle. The first fused state feature information is used to represent the occupancy trend of each vehicle in each zone. The occupancy trend is used to predict the occupancy status of each vehicle in each zone at the next monitoring time. By processing the graph model hierarchy of the multimodal fusion prediction sub-model, the first fusion state feature information and association information of each vehicle are processed to obtain the second fusion state feature information of each vehicle. The second fusion state feature information is used to represent the occupancy trend of each vehicle in each partition under the influence of other vehicles. By processing the second fusion state feature information of each vehicle through the first fully connected layer and the first activation layer of the multimodal fusion prediction sub-model, the predicted occupancy information of each vehicle for each partition is obtained. By using the predicted occupancy information of each vehicle for each zone, the second fused state feature information is weighted and summed to obtain the comprehensive predicted state information of vehicles in each zone. The comprehensive predicted state information of vehicles is used to describe the predicted motion state information of all vehicles in a zone under mutual influence. The comprehensive vehicle prediction state information within each partition is processed by the second fully connected layer and the second activation layer of the multimodal fusion prediction sub-model to obtain the road state prediction feature information of each partition.

7. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 1, characterized in that, The training steps for the traffic safety monitoring model include: Acquire training videos of the monitoring area, and acquire training vehicle modal data and training road modal data; The training vehicle modal data is processed by the vehicle modal feature extraction sub-model to obtain the training vehicle state feature information of each vehicle. The training road modal data is processed by the road modal feature extraction sub-model to obtain the training road state feature information of each zone; The training vehicle state feature information and training road state feature information are processed by the multimodal fusion prediction sub-model to obtain the training fused state feature information and the training road state prediction feature information. The training vehicle state feature information is decoded to obtain the training predicted vehicle motion state. Based on the training vehicle modal data for the next moment obtained from the training video, and the training predicted vehicle motion state, determine the motion loss function; The training road state feature information is decoded to obtain the training prediction occupancy status of each partition. Based on the training road modal data and training prediction occupancy status obtained from the training video at the next time step, determine the partition occupancy loss function; The training fusion state feature information is decoded to obtain the training occupancy information of each vehicle for each partition. Based on the training occupancy information and the training road modal data for the next time step obtained from the training video, determine the vehicle occupancy loss function; Identify the target training zones in the training videos where the vehicle is likely to take emergency action in the next moment. Based on the training target partition and the training road state prediction feature information, determine the partition safety risk loss function; Based on the motion loss function, zone occupancy loss function, vehicle occupancy loss function, and zone safety risk loss function, the loss function of the traffic safety monitoring model is determined. Based on the loss function of the traffic safety monitoring model, the traffic safety monitoring model is trained to obtain the trained traffic safety monitoring model.

8. The highway traffic safety monitoring method based on intelligent fusion of multi-source data according to claim 7, characterized in that, The zones are numbered sequentially according to the direction of the roads. Based on the training occupancy information and the next time-step training road modal data obtained from the training video, the vehicle occupancy loss function is determined, including: The penalty term is determined by the index function, the proportion of vehicles in each zone determined by the training occupancy information, and the proportion of vehicles in each zone determined by the training road modality data at the next time step. Based on the penalty term, the proportion of vehicles in each zone determined by the training occupancy information, and the proportion of vehicles in each zone determined by the training road modal data at the next time step, the penalty term vehicle occupancy loss function is determined.

9. A highway traffic safety monitoring system based on intelligent fusion of multi-source data, characterized in that, The system is used to perform the method as described in any one of claims 1-8, comprising: The acquisition module is used to capture monitoring videos of the monitoring area through a camera and acquire vehicle modal data and road modal data. The road modal data is used to represent the vehicle occupancy status of multiple zones of multiple roads in the monitoring area, and the vehicle modal data is used to represent the movement status of multiple vehicles in the monitoring area. The vehicle state feature information module is used to process the vehicle modal data of multiple vehicles through the vehicle modal feature extraction sub-model of the trained traffic safety monitoring model to obtain the vehicle state feature information of each vehicle. The road condition feature information module is used to process road modal data and obtain road condition feature information through the road modal feature extraction sub-model of the trained traffic safety monitoring model. The association information module is used to determine the association information of each vehicle based on vehicle modal data, wherein the association information is used to describe the mutual influence between the vehicles. The road condition prediction feature information module is used to obtain road condition prediction feature information based on the vehicle condition feature information, road condition feature information, and related information, as well as the multimodal fusion prediction sub-model of the trained traffic safety monitoring model. The safety risk information module is used to determine the safety risk information of multiple zones on multiple roads based on road condition prediction feature information; The judgment module is used to determine whether there are any zones in multiple sections of multiple roads where the safety risk information is higher than a preset risk threshold. The prompt module is used to generate traffic safety monitoring prompts based on safety risk information if such risks exist.

Images