Road traffic visualization display method and system based on virtual reality
Through the event implicit representation neural network based on virtual reality, the multi-source collaborative processing of road traffic data is solved, and a single data source is difficult to reflect complex traffic events is achieved, and efficient and accurate traffic event identification and display in virtual reality is achieved.
Patent Information
- Application Number
- CN202411406254.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-10-10
AI Technical Summary
The existing road traffic visual display technology relies on a single data source and is difficult to comprehensively and accurately reflect complex traffic events and their dynamic changes. The multi-source data fusion method lacks deep interaction and collaborative mining, resulting in poor identification accuracy and stability.
Using a virtual reality-based method, the road binocular visual data and millimeter-wave radar data are encoded implicitly through event implicit representation neural network, and feature random occlusion and collaborative implicit representation mining are performed to generate multi-source collaborative implicit representation. , realize accurate identification and visual display of traffic events.
In the absence of road millimeter wave radar data or binocular visual data, it can accurately index implicit representation of traffic events, improve the accuracy and stability of event classification, enhance the collaborative training effect between data sources, and ensure accurate identification in complex environments Traffic incident.
Smart Images

Figure CN118918548B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a road traffic visualization display method and system based on virtual reality. Background Art
[0002] With the acceleration of urbanization and the rapid development of intelligent transportation systems, the importance of road traffic management and visualization technology has become increasingly prominent. Traditional road traffic monitoring and management systems often rely on a single data source, such as video surveillance or radar detection. Although these data can provide information about road conditions to a certain extent, they are limited by the limitations of a single sensor and are difficult to fully and accurately reflect complex traffic events and their dynamic changes. In existing road traffic visualization technology, video image processing technology is usually used to analyze road monitoring videos to identify events such as traffic congestion, accidents, and illegal driving. However, this method is highly dependent on image quality and lighting conditions, and is easily affected by factors such as weather and obstructions, resulting in low recognition accuracy and poor stability. On the other hand, although millimeter-wave radar technology can provide relatively accurate information such as distance and speed, it is difficult to accurately determine the nature of traffic events and their spatial distribution in complex traffic environments by relying solely on radar data. In recent years, with the rapid development of artificial intelligence and machine learning technologies, multi-source data fusion and collaborative processing have become an important way to improve the recognition and visualization of traffic events. However, most of the existing multi-source data fusion methods remain at the simple superposition or concatenation of feature levels, lacking deep interaction and collaborative mining, resulting in limited fusion effects and unable to fully utilize the advantages of multi-source data. Summary of the invention
[0003] In view of this, the present application provides a road traffic visualization display method based on virtual reality.
[0004] The technical solution of this application is implemented as follows:
[0005] On the one hand, the present application provides a road traffic visualization display method based on virtual reality, the method comprising: obtaining debugging learning samples, each of the debugging learning samples comprising road binocular vision data, road millimeter-wave radar data matched with the road binocular vision data, and traffic event labels; based on a first branch component of an event implicit representation neural network, implicitly encoding an initial road binocular vision data implicit representation of the road binocular vision data to obtain an encoded binocular vision data implicit representation, and obtaining a first traffic event recognition result for the debugging learning sample according to the encoded binocular vision data implicit representation; based on a second branch component of the event implicit representation neural network, implicitly encoding an initial radar data implicit representation of the road millimeter-wave radar data to obtain an encoded radar data implicit representation, and obtaining a second traffic event recognition result for the debugging learning sample according to the encoded radar data implicit representation; based on a third branch component of the event implicit representation neural network, performing feature random masking on the encoded binocular vision data implicit representation and the encoded radar data implicit representation to obtain a multi-source fusion implicit representation. The method comprises the following steps: a first traffic event recognition method, a second traffic event recognition method, a third traffic event recognition method for the debugging learning sample, and a traffic event recognition error for the debugging learning sample. The first traffic event recognition method, the second traffic event recognition method, the third traffic event recognition method and the traffic event label are used to generate a traffic event recognition error. After adjusting the parameters according to the traffic event recognition error, the method jumps to the step of obtaining the debugging learning sample for iterative debugging. When the convergence state is reached, the method stops iterating and obtains a debugged event implicit representation neural network for traffic event recognition. The target radar-vision fusion data of the target road are obtained, and the target radar-vision fusion data include the road binocular vision data and the road millimeter-wave radar data of the target road. The target traffic event recognition result is obtained based on the debugged event implicit representation neural network recognition. The event update display is performed in a preset virtual reality twin based on the target traffic event recognition result of the target road.
[0006] On the other hand, the present application provides a computer system, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps in the above method when executing the program.
[0007] The beneficial effects of the present application include at least: the road traffic visualization display method and system based on virtual reality provided by the present application, after obtaining the debugging learning sample, encode the initial road binocular vision data implicit representation of the road binocular vision data of the debugging learning sample to obtain the corresponding encoded binocular vision data implicit representation, obtain the first traffic event recognition result for the debugging learning sample according to the encoded binocular vision data implicit representation, encode the initial radar data implicit representation of the road millimeter-wave radar data of the debugging learning sample to obtain the corresponding encoded radar data implicit representation, and obtain the second traffic event recognition result for the debugging learning sample according to the encoded radar data implicit representation. In addition, after the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data are subjected to feature random masking to obtain the multi-source fusion implicit representation, the traffic event implicit representation is performed according to the multi-source fusion implicit representation, and the third traffic event recognition result for the debugging learning sample is obtained based on the implicit representation of the traffic event. Subsequently, the parameter variables are adjusted according to the above three recognition results and the prior labels, which can ensure that when there is no road millimeter-wave radar data or no road binocular vision data, accurate implicit representation of traffic events can be indexed in the representation domain, and event classification according to the multi-source implicit representation can be completed. At the same time, collaborative training is performed based on the road millimeter-wave radar data and the road binocular vision data, so that the event and the road binocular vision data are associated in the same representation domain, and the accuracy of the event implicit representation neural network for implicit representation of traffic events for road binocular vision data or millimeter-wave radar is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.
[0009] Figure 1 A schematic diagram of the implementation flow of a road traffic visualization display method based on virtual reality provided in an embodiment of the present application.
[0010] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0011] The road traffic visualization display method based on virtual reality provided in the embodiment of the present application can be executed by a processor of a computer system. The computer system can refer to a device with data processing capabilities such as a server, a laptop, a tablet computer, a desktop computer, etc., which can receive images and radar data uploaded from the radar-visual integrated device. Figure 1 A schematic diagram of the implementation process of a road traffic visualization display method based on virtual reality provided in an embodiment of the present application is shown in FIG. Figure 1As shown, the method includes two stages. The first stage is the training stage of the network, including the following steps S100 to S500; the second stage is the application stage of the network, including the following steps S600 to S800. Specifically, the method includes:
[0012] Step S100: Acquire debugging learning samples, each debugging learning sample including road binocular vision data, road millimeter-wave radar data matched with the road binocular vision data, and traffic event labels.
[0013] In step S100, the computer system obtains the initial data set of the event implicit representation neural network for debugging and training. These data sets are debugging learning samples. Each debugging learning sample is a comprehensive data set that aims to fully reflect the actual situation of road traffic so that the model can learn the characteristic performance under different traffic conditions. Specifically, each debugging learning sample contains the following three core components:
[0014] Road binocular vision data: This part of data exists in the form of a collection, which contains a series of road binocular vision images corresponding to different time points. Binocular vision images refer to road scene images captured simultaneously from different angles by two cameras (simulating the binocular structure of the human eye). For example, the road binocular vision data in a debugging learning example may include images of traffic conditions at intersections taken from two different perspectives during peak hours such as 7:00, 8:00, and 9:00 in the morning. These images can capture key information such as vehicle trajectory, speed changes, lane occupancy, and the dynamics of pedestrians and non-motor vehicles.
[0015] Road millimeter-wave radar data: Matching the road binocular vision data, each debugging learning sample also contains data collected by the road millimeter-wave radar during the same time period. Millimeter-wave radar detects and measures information such as the distance, speed, and angle of surrounding objects by emitting and receiving millimeter-wave signals. In traffic scenarios, these data can accurately reflect the vehicle's real-time position, speed, and whether there are abnormal behaviors such as sudden lane changes and emergency stops. For example, radar data may show that a vehicle suddenly slows down and changes lanes at a certain moment, which may be related to a traffic accident or congestion ahead.
[0016] Traffic event labels: Each debugging learning example is also accompanied by a traffic event label as the target output of supervised learning. These labels can be binary or multi-classified, indicating whether a specific traffic event has occurred in the road scene during that time period, as well as the specific type of event. For example, a label is a simple binary value (0 or 1) indicating whether a congestion event has occurred; or a more specific classification label such as "congestion", "illegal lane change", "traffic accident", etc. These labels are generated based on actual traffic monitoring records or manual annotations, and are used to guide the model to learn how to identify different traffic events based on the input data.
[0017] After obtaining these debugging learning samples, the computer system will further process the data, including image preprocessing (such as resizing, normalization), radar data parsing and label encoding, to prepare the data for input into the event implicit representation neural network for training. Through such a dataset preparation process, the model can learn to extract useful features from road stereo vision images and millimeter wave radar data and accurately predict traffic events.
[0018] Step S200: Based on the first branch component of the event implicit representation neural network, the initial road binocular vision data implicit representation of the road binocular vision data is implicitly encoded to obtain the encoded binocular vision data implicit representation, and the first traffic event recognition result for the debugging learning sample is obtained according to the encoded binocular vision data implicit representation.
[0019] In step S200, the computer system starts the first branch component of the event implicit representation neural network, which is specially designed to process road binocular vision data. First, the system receives the road binocular vision data in the debugging learning sample, which usually exists in the form of a collection and contains binocular vision images at multiple time points, each of which contains rich road scene information.
[0020] In order to extract useful features for traffic event recognition from these raw images, the first branch component first needs to encode and transform the initial implicit representation of the road stereo vision data. This conversion process is achieved through a multi-layer structure in the neural network, each of which may contain convolution layers, pooling layers, activation layers, etc., which are used to gradually abstract and extract key features in the image. Taking the convolutional neural network (CNN) as an example, the first branch component can use multiple convolution kernels to perform convolution operations on the road stereo vision image to extract low-level features such as edges and textures in the image. Subsequently, the dimension of the feature map is reduced through the pooling layer, reducing the amount of calculation and extracting the main features. After multiple such convolution-pooling blocks, the feature map is further passed to the fully connected layers (FC), which map the learned feature representation to a high-dimensional space to form an implicit representation of the encoded stereo vision data. The implicit representation of the encoded stereo vision data is a high-dimensional vector that captures the information in the road stereo vision image that is crucial for traffic event recognition. This vector is then used as input for subsequent processing or classification. In the first branch component, this encoding vector may be used directly or through an additional network layer (such as a softmax layer) to predict the first traffic event recognition result corresponding to the debugging learning sample.
[0021] For example, if the traffic event label is multi-classified (such as "normal", "congestion", "accident"), the output layer of the first branch component may contain neurons equal to the number of categories, each neuron corresponding to the predicted probability of a category. By applying the softmax function, these predicted probabilities are normalized so that the sum of the predicted probabilities of all categories is 1. Based on these predicted probabilities, the system selects the category with the highest probability as the first traffic event recognition result.
[0022] Step S300: Based on the second branch component of the event implicit representation neural network, the initial radar data implicit representation of the road millimeter-wave radar data is implicitly encoded to obtain the encoded radar data implicit representation, and the second traffic event recognition result for the debugging learning sample is obtained according to the encoded radar data implicit representation.
[0023] Step S300 is a step of processing road millimeter wave radar data in the road traffic visualization display method based on virtual reality. At this stage, the computer system uses the second branch component of the event implicit representation neural network, which is specially designed to process and analyze the road traffic data collected by the millimeter wave radar.
[0024] First, the system receives the road millimeter-wave radar data in the debugging learning sample. These data usually exist in the form of time series, including the measurement values of physical quantities such as vehicle speed, distance, angle, etc. at multiple time points. Millimeter-wave radar plays an important role in traffic monitoring with its high precision and high reliability, and can accurately capture the dynamic changes of vehicles on the road. Next, the second branch component implicitly represents and encodes the initial implicit representation of the road millimeter-wave radar data (which can be understood as the original format of the radar data or the format after preliminary processing). This process is similar to the processing of road binocular vision data in step S200, but it is aimed at the characteristics of radar data. Since radar data is usually a one-dimensional time series, the second branch component may adopt a network structure suitable for processing sequence data, such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a gated recurrent unit (GRU). Taking LSTM as an example, the second branch component can effectively capture the time dependency and long-term dependency in radar data through its internal memory unit and forget gate, input gate, output gate and other mechanisms. At each time step, the LSTM unit receives the radar data at the current time point as input, combines the state and output of the previous time step, updates its internal state, and generates the output of the current time step. In this way, over time, LSTM can accumulate and transmit useful information about the state of road traffic. After being processed by the LSTM network (or other suitable sequence processing network), the radar data is encoded into a high-dimensional vector, namely the encoded radar data implicit representation. This vector captures the temporal pattern and spatial features in the radar data that are crucial for traffic event recognition. Finally, based on the encoded radar data implicit representation, the system can obtain the second traffic event recognition result for the debugging learning sample. This process may involve passing the encoded vector to one or more fully connected layers (FC layers), which map the encoded vector to the probability distribution of traffic event categories. If the traffic event is multi-classified (such as "normal driving", "slow driving", "emergency stop", etc.), the last layer is usually a softmax layer to output the predicted probability of each category. The system selects the category with the highest probability as the second traffic event recognition result.
[0025] The training process of the second branch component is carried out synchronously with the training of the entire event implicit representation neural network. During the training process, the system calculates the difference (i.e., loss) between the predicted traffic event category and the actual label, and uses the back-propagation algorithm to adjust the network parameters to minimize the loss function. Commonly used loss functions include cross entropy loss (for multi-classification problems), which is calculated as:
[0026] ;
[0027] Where N is the number of debugging learning samples, C is the number of traffic event categories, is an indicator variable (1 if the actual category of the i-th sample is c, otherwise 0), is the probability that the model predicts that the i-th sample belongs to category c. By minimizing this loss function, the system can continuously optimize the performance of the second branch component, thereby improving the accuracy of traffic event recognition.
[0028] Step S400: Based on the third branch component of the event implicit representation neural network, the encoded binocular vision data implicit representation and the encoded radar data implicit representation are subjected to random feature masking to obtain a multi-source fusion implicit representation, and the multi-source fusion implicit representation is subjected to collaborative implicit representation mining to obtain a multi-source collaborative implicit representation, and a third traffic event recognition result for the debugging learning sample is obtained according to the multi-source collaborative implicit representation; the multi-source collaborative implicit representation is an implicit representation of traffic events for the debugging learning sample.
[0029] Step S400 is a step of integrating multi-source data and performing advanced feature fusion in the road traffic visualization display method based on virtual reality. At this stage, the computer system uses the third branch component of the event implicit representation neural network to deeply fuse the implicit representation of encoded binocular vision data with the implicit representation of encoded radar data to capture the complementary information and interaction between the two, thereby generating a more comprehensive and accurate implicit representation of traffic events.
[0030] First, the third branch component receives the encoded data from the first and second branches: the implicit representation of encoded binocular vision data and the implicit representation of encoded radar data. These two types of encoded data represent the visual information and physical measurement information of the road scene, respectively, and each contains a unique set of features. In order to enhance the generalization and robustness of the model, the third branch component introduces a feature random masking mechanism. This mechanism randomly selects some features in the encoded data to mask (i.e., set to zero or a fixed value) to simulate data missing or noisy situations that may be encountered in actual applications. For example, at a certain moment, the binocular vision image may not capture key information due to occlusion or sensor failure, or the radar data may produce errors due to weather interference. Through feature random masking, the model can learn the ability to make accurate predictions under these uncertain conditions.
[0031] After the feature random masking process, the third branch component combines the remaining encoded data into a multi-source fused implicit representation. This step is to initially integrate data from different sources at the feature level, laying the foundation for subsequent collaborative implicit representation mining.
[0032] Next, the third branch component uses collaborative implicit representation mining technology to conduct in-depth analysis of multi-source fused implicit representations. This usually involves complex machine learning algorithms or neural network modules, such as attention mechanisms, graph neural networks (GNNs), or specific implicit representation collaborative mining modules. Taking the attention mechanism as an example, it allows the model to dynamically adjust the importance weights of data from different sources, thereby focusing on the features that are most critical to the current traffic event recognition. Through the allocation of attention weights, the model is able to capture the interaction and influence between the implicit representations of encoded binocular vision data and the implicit representations of encoded radar data, and achieve effective fusion of cross-modal information. In the collaborative implicit representation mining process, the model performs cross-feature mining tasks to explore the potential correlations and complementarities between different source data. These correlations and complementarities may be reflected in multiple dimensions such as spatial location, temporal dynamics, and behavioral patterns. For example, radar data may provide information about the speed and distance of a vehicle, while binocular vision images reveal the shape, color, and driving trajectory of the vehicle. Through cross-feature mining, the model is able to combine this information to form a more complete understanding of traffic events.
[0033] Finally, after mining the collaborative implicit representation, the third branch component outputs a multi-source collaborative implicit representation. This representation not only combines the advantages of the implicit representation of encoded binocular vision data and the implicit representation of encoded radar data, but also captures the interaction and influence between them. Based on this multi-source collaborative implicit representation, the model can generate the third traffic event recognition result for the debugging learning sample. This result is a comprehensive judgment of the traffic events in the debugging learning sample, and its accuracy benefits from the deep fusion of multi-source data and collaborative representation learning.
[0034] It should be noted that the training process of the third branch component is also carried out synchronously with the training of the entire event implicit representation neural network. During the training process, the system calculates the difference (i.e., loss) between the predicted traffic event category and the actual label, and uses the back-propagation algorithm to adjust the network parameters to minimize the loss function. The calculation of the loss function may involve multiple forms such as cross entropy loss and mean square error loss, depending on the classification method of the traffic event and the model output type. Through continuous iterative training and optimization adjustment, the model can gradually learn how to extract effective features from multi-source data and perform accurate event recognition.
[0035] Step S500: Generate a traffic event recognition error based on the first traffic event recognition result, the second traffic event recognition result, the third traffic event recognition result and the traffic event label. After adjusting the parameters based on the traffic event recognition error, jump to the step of obtaining debugging learning samples for iterative debugging. When the convergence state is reached, no more iteration is performed to obtain a debugged event implicit representation neural network for traffic event recognition.
[0036] Step S500 involves the calculation of traffic event recognition errors, the adjustment of model parameters, and the execution of iterative debugging. First, the computer system receives the traffic event recognition results from the first, second, and third branch components of the event implicit representation neural network, which are obtained based on road binocular vision data, road millimeter wave radar data, and the fusion data of the two. At the same time, the system also obtains the traffic event labels corresponding to each debugging learning sample, which are used as supervision information to evaluate the accuracy of model recognition.
[0037] In order to quantify the difference between the model's recognition results and the actual situation, the computer system calculates the traffic event recognition error. This process usually involves defining a loss function that measures the degree of inconsistency between the predicted results and the actual labels. Taking multi-classification problems as an example, the commonly used loss function is cross entropy loss.
[0038] For each debugging learning example, the computer system calculates the loss between the first, second and third traffic event recognition results and the corresponding labels, and performs weighted summation of these loss values (the weights can be set according to actual needs to reflect the importance of different data sources) to obtain the total loss. This total loss is the quantitative representation of the traffic event recognition error. After obtaining the traffic event recognition error, the computer system uses the backpropagation algorithm to adjust the parameters (including weights and biases) of the event implicit representation neural network. The backpropagation algorithm calculates the gradient of the loss function with respect to each parameter and updates the parameter value according to the gradient descent method (or other optimization algorithms, such as Adam, RMSprop, etc.) to minimize the loss function. After completing a parameter update, the computer system jumps to the step of obtaining the debugging learning example and starts a new round of iterative debugging. This process is repeated until the stopping condition is met (such as the loss value no longer decreases significantly, reaches the preset number of iterations, or the performance on the validation set begins to degrade, etc.), at which time the model is considered to have converged and the training process is over. Finally, the computer system obtained a debugged event implicit representation neural network, which can accurately identify traffic events based on road binocular vision data and millimeter-wave radar data, providing strong support for the subsequent road traffic visualization display in virtual reality.
[0039] Step S600: acquiring target radar-visual fusion data of the target road, where the target radar-visual fusion data includes road binocular vision data and road millimeter-wave radar data of the target road.
[0040] In step S600, the computer system is responsible for acquiring the target radar-visual fusion data of the target road, which is collected in real time or recently and is used to display the current traffic conditions of the road in virtual reality. Specifically, the computer system first determines the target road that needs to be monitored and displayed. This road is, for example, a city trunk road, a transportation hub, an accident-prone area, and other places with important traffic management significance. Once the target road is determined, the system begins to collect the radar-visual fusion data on the road, that is, a combination of the road binocular vision data and the road millimeter-wave radar data. The road binocular vision data is collected by binocular cameras installed on the target road. These cameras can simulate the principle of binocular vision of the human eye and capture high-definition images of road scenes from different angles at the same time. These images contain rich visual information, such as vehicle type, color, driving direction, lane occupancy, etc., which are an important basis for understanding road traffic dynamics.
[0041] At the same time, road millimeter-wave radar data is collected by millimeter-wave radar equipment installed on the roadside or on vehicles. Millimeter-wave radar uses high-frequency electromagnetic waves to detect the surrounding environment, and can accurately measure physical quantities such as the distance, speed, and angle of the target. It is particularly effective in monitoring dynamic targets. In road traffic, millimeter-wave radar can track the vehicle's driving trajectory in real time, detect the vehicle's speed changes, acceleration, whether there is an emergency stop or illegal lane change, and provide key data support for the prediction and identification of traffic events.
[0042] In the process of acquiring the target radar-visual fusion data, the computer system needs to communicate with the data acquisition equipment to ensure the real-time and accuracy of the data. Once the data is successfully received and stored, the system will pre-process the raw data, including but not limited to data cleaning, format conversion, time synchronization and other operations, so that the event implicit representation neural network in the subsequent steps can be processed smoothly.
[0043] After data preprocessing, the target radar-visual fusion data is ready to be input into the debugged event implicit representation neural network for traffic event recognition. This step will use the neural network to extract key features from binocular vision images and millimeter-wave radar data, and generate implicit representations of traffic events on the target road through multi-source data fusion and collaborative representation learning. Ultimately, these implicit representations will be used to display the traffic conditions of the target road in real time in a virtual reality environment, helping traffic managers and the public better understand and respond to the current traffic situation.
[0044] Step S700: Obtain target traffic event recognition results based on the debugged event implicit representation neural network recognition.
[0045] Step S700 uses the debugged neural network model to process the radar-visual fusion data of the target road to identify the current road traffic event.
[0046] Specifically, the computer system first calls the debugged event implicit representation neural network model, which has been trained through a large number of debugging learning examples and has the ability to efficiently fuse and analyze road binocular vision data and millimeter-wave radar data. Subsequently, the system takes the target road's radar-visual fusion data (including road binocular vision images and millimeter-wave radar data) as input and passes it to the neural network model.
[0047] Inside the neural network, the input data first passes through the preprocessing module to perform necessary format conversion, noise suppression, and data enhancement operations to ensure data consistency and stability of model processing. Then, the data is sent to the various branch components of the neural network for implicit representation encoding. These branch components correspond to the first branch (processing binocular vision data), the second branch (processing millimeter-wave radar data), and possible fusion branches (for collaborative processing of multi-source data) mentioned above. During the encoding process, the neural network uses structures such as convolutional layers, pooling layers, and fully connected layers to extract high-level features from the input data and generate corresponding coded representations. These coded representations capture the driving status, speed changes, lane occupancy, and potential traffic event characteristics of vehicles in road scenes.
[0048] After the encoding is completed, the fusion and recognition module of the neural network starts to work. This module receives the encoding representations from different branches, and realizes the deep fusion and interaction of multi-source data through technical means such as attention mechanism and implicit representation collaborative mining. In this process, the model can automatically learn the correlation and complementarity between different data sources, and then generate a more comprehensive and accurate implicit representation of traffic events. Finally, based on this implicit representation, the neural network outputs the recognition result of the target traffic event through a classifier (such as a softmax layer). This result is usually a probability distribution vector, which indicates the probability that the input data belongs to each preset traffic event category. Based on this probability distribution vector, the computer system selects the category with the highest probability as the final target traffic event recognition result. For example, if the preset traffic event categories include "normal driving", "congestion", "accident", etc., the probability distribution vector output by the neural network may be [0.1, 0.8, 0.1], indicating that there is an 80% probability that the current road condition is in a "congested" state. Based on this, the computer system judges and displays the corresponding traffic event information, providing intuitive visual feedback to traffic managers and the public in a virtual reality environment.
[0049] Step S800: Based on the target traffic event recognition result of the target road, the event update display is performed in the preset virtual reality twin.
[0050] Step S800 realizes seamless connection from data processing to result presentation. In this step, the computer system uses the target traffic event recognition results obtained in step S700 to update and intuitively display the road traffic conditions in real time in the preset virtual reality twin.
[0051] First, the computer system receives the target traffic event recognition results from the event implicit representation neural network. This result is a highly summarized and accurate judgment of the current traffic conditions of the target road, such as labels such as "congestion", "accident", and "normal driving". These labels not only reflect the basic status of road traffic, but also imply potential risks and changing trends. Then, the computer system maps these recognition results to the preset virtual reality twin. The virtual reality twin is a highly simulated digital model that is built based on information such as the road network, traffic facilities, and vehicle dynamics in the real world, and can reflect the actual situation of road traffic in real time. In this model, every road, every intersection, and every vehicle is accurately simulated and has physical properties and behavior rules corresponding to the real world. In order to integrate the target traffic event recognition results into the virtual reality twin, the computer system can use any feasible existing visualization technology. For example, for congestion events, the system can highlight the congested section in red in the virtual reality twin, and display detailed information such as the degree of congestion and duration; for accident events, the system will set up warning signs around the accident site and warn users through pop-up windows or voice prompts. These visualization elements are not only intuitive and easy to understand, but also can quickly attract the user's attention, helping them to quickly understand the road traffic conditions and make corresponding decisions. Through the execution of step S800, the computer system successfully converts the target traffic event recognition results into an intuitive display in the virtual reality twin. This display method not only improves the readability and usability of traffic information, but also provides users with a more immersive and interactive experience. In the road traffic visualization display scene based on virtual reality, users can enter the virtual reality world by wearing VR helmets and other equipment, and experience the changes and dynamics of road traffic conditions in an immersive way.
[0052] As an implementation mode, the first branch component includes a pre-adjusted binocular vision implicit representation mining component and a binocular vision implicit representation encoding component; based on this, step S200, based on the first branch component of the event implicit representation neural network, performs implicit representation encoding on the initial road binocular vision data implicit representation of the road binocular vision data to obtain the encoded binocular vision data implicit representation, including:
[0053] Step S210: based on the pre-adjusted binocular vision implicit representation mining component, data sampling is performed on the road binocular vision data to obtain a road binocular vision image set, and the initial road binocular vision data implicit representation of each road binocular vision image in the road binocular vision image set is extracted to obtain an initial road binocular vision data implicit representation set;
[0054] Step S220: Based on the binocular vision implicit representation encoding component, perform implicit representation aggregation on the initial road binocular vision data implicit representation set to obtain an aggregated implicit representation, and perform implicit representation encoding on the aggregated implicit representation to obtain an encoded binocular vision data implicit representation.
[0055] In step S210, the computer system processes the original road binocular vision data. These data usually contain a large number of continuous video frames or static images, each of which captures a certain moment of the road scene. Since directly processing all the data is time-consuming and may introduce unnecessary computational burden, the system first performs data sampling to reduce the amount of data while retaining key information.
[0056] Assume that the road binocular vision dataset contains a pair of visual images (left eye and right eye views) taken every 5 minutes within 24 hours a day. For efficient processing, the computer system may choose to retain only the data during peak hours (such as 7:00-9:00 in the morning and 17:00-19:00 in the evening), and further sample at intervals of 15 minutes. In this way, the large number of image pairs that could have been processed are streamlined to a more manageable scale. Next, the system uses the pre-tuned binocular vision implicit representation mining component to extract the initial implicit representation of each pair of images. The pre-tuning here means that the component has been pre-trained on one or more related large datasets before being used for the current task, so that it has a certain general visual feature extraction capability.
[0057] For each pair of binocular vision images, the pre-adjusted binocular vision implicit representation mining component may first extract the underlying features of the image, such as edges, textures, etc., through a series of convolutional layers. Then, these underlying features are used to further generate higher-level abstract representations that can capture complex information such as objects, shapes, and spatial relationships in the image. Since this is binocular vision data, the component may also pay special attention to the disparity information between the left and right views to infer depth information. After this process, each pair of binocular vision images is converted into one or more high-dimensional vectors, which constitute the initial road binocular vision data implicit representation set. These implicit representations not only contain the visual features of the image itself, but may also imply key information such as road conditions and vehicle behavior.
[0058] In step S220, the computer system receives the initial road binocular vision data implicit representation set outputted in step S210 and prepares for further aggregation and encoding processing. The purpose of this step is to merge multiple initial implicit representations into a unified, more compact encoding representation to facilitate subsequent analysis and recognition tasks.
[0059] If the initial implicit representation set contains implicit representations of images at multiple time points, the system can use pooling technology to aggregate these representations. Pooling is a dimensionality reduction technique that reduces the spatial size of data by statistically summarizing local areas (such as taking the maximum value, average value, etc.) while retaining important features. For example, for the implicit representation vector of each time point, the system can apply one-dimensional maximum pooling or average pooling along the time dimension to obtain an aggregated implicit representation that integrates information from multiple time points.
[0060] Although the aggregated implicit representation integrates the information of multiple initial implicit representations, it may still be too large or not compact enough to be directly used for subsequent tasks. Therefore, the system needs to use the binocular vision implicit representation encoding component to further encode it. The encoding component, for example, is a neural network composed of multiple layers of fully connected layers (Fully Connected Layers), which are connected by nonlinear activation functions (such as ReLU) to map the aggregated implicit representation to a lower-dimensional but more expressive space. During the training process, the weights of these layers are gradually adjusted so that the encoded representation can best retain the key information in the original data while removing redundancy and noise. Specifically, the encoding process can be expressed as a combination of a series of matrix multiplications and nonlinear transformations.
[0061] After this series of aggregation and encoding processes, the computer system finally obtains a compact and efficient implicit representation of the encoded binocular vision data. This representation not only captures the key features and information in the road binocular vision data, but also provides a solid foundation for subsequent tasks such as event recognition and traffic condition analysis.
[0062] As an implementation method, the second branch component includes a pre-adjusted radar data implicit representation mining component and a radar data implicit representation encoding component. Based on this, step S300, based on the second branch component of the event implicit representation neural network, performs implicit representation encoding on the initial radar data implicit representation of the road millimeter wave radar data to obtain the encoded radar data implicit representation, including:
[0063] Step S310: extracting the initial radar data implicit representation of the road millimeter wave radar data based on the pre-adjusted radar data implicit representation mining component;
[0064] Step S320: Based on the radar data implicit representation encoding component, implicit representation encoding is performed on the initial radar data implicit representation to obtain an encoded radar data implicit representation.
[0065] In step S310, the core task of the computer system is to use the pre-adjusted radar data implicit representation mining component to extract the initial radar data implicit representation from the original road millimeter wave radar data. The pre-adjusted radar data implicit representation mining component is a pre-trained machine learning model or neural network that has the ability to automatically extract key features from radar data. The pre-adjusted radar data implicit representation mining component may adopt structures such as convolutional neural network (CNN), recurrent neural network (RNN) or its variants (such as LSTM, GRU), depending on the characteristics of radar data and task requirements. Since radar data usually exists in the form of one-dimensional time series, RNN and its variants may be a better choice when processing such data. For example, assume that the road millimeter wave radar collects physical quantity information such as vehicle speed, distance, angle, etc. at fixed time intervals (such as 10 times per second). These data constitute time series data, in which each time point corresponds to a set of measurement values. The pre-adjusted radar data implicit representation mining component first receives these raw radar data and gradually extracts features through its internal network structure. After being processed by the pre-tuned radar data implicit representation mining component, the raw radar data is converted into a series of high-dimensional vectors, which constitute the initial radar data implicit representation. Each vector contains the key feature information of the radar data at a certain time point or time window. These implicit representations not only retain the core features of the original data, but also enhance the data's expressiveness through nonlinear transformations, providing strong support for subsequent data encoding and event recognition tasks.
[0066] After successfully extracting the initial radar data implicit representation, the computer system enters step S320 to further encode these initial representations using the radar data implicit representation encoding component. The purpose of the encoding component is to convert the initial implicit representation into a more compact and expressive encoded radar data implicit representation. The radar data implicit representation encoding component can use a multi-layer fully connected neural network (FCN), an autoencoder, or other types of neural network structures. The core idea is to map the high-dimensional initial implicit representation into a low-dimensional but information-rich space by learning the intrinsic structure of the data.
[0067] For example, assume that the encoding component adopts a simple multi-layer fully connected neural network structure. The network consists of multiple hidden layers, each of which contains a certain number of neurons and is connected by nonlinear activation functions (such as ReLU). During the training process, the network gradually adjusts its weights and bias parameters by learning the mapping relationship between the input data (i.e., the initial radar data implicit representation) and the output target (i.e., the desired encoded representation). Specifically, the encoding process can be represented as a combination of a series of matrix multiplications and nonlinear transformations. Given an initial implicit representation vector x, the encoding component generates the encoded radar data implicit representation z in the following way:
[0068] ;
[0069] Where W is the weight matrix, b is the bias vector, and f is the nonlinear activation function. By stacking multiple such layers, the encoding component is able to capture the complex patterns and structures in the initial implicit representation and generate a more compact and efficient encoding representation.
[0070] In practical applications, directly using a fully connected network for encoding may face the risk of overfitting, especially when the dimension of the input data is high and the sample size is limited. To alleviate this problem, regularization techniques (such as L1 / L2 regularization), dropout, or early stopping can be used to control the complexity of the model. In addition, as an unsupervised learning method, autoencoders can also be used for implicit representation encoding of radar data. Autoencoders learn low-dimensional representations of data by training an encoder and a decoder simultaneously. The encoder compresses the input data into a low-dimensional encoded representation, while the decoder attempts to reconstruct the original input from the encoded representation. During the training process, the parameters of the encoder and decoder are optimized by minimizing the reconstruction error. The resulting encoded representation not only retains the key information of the original data, but also removes redundancy and noise.
[0071] The implicit representation of radar data after encoding has lower dimensions and higher information density, which makes it more efficient and practical in subsequent tasks such as event recognition and traffic condition analysis. In the road traffic visualization display scene based on virtual reality, the implicit representation of encoded radar data can be used to generate simulated images or animations of road traffic conditions in real time, helping users to intuitively understand information such as vehicle dynamics, speed changes, and congestion on the road. At the same time, these encoded representations can also be fused with other data sources (such as binocular vision data) to further improve the accuracy and real-time performance of traffic event recognition.
[0072] As an implementation method, the third branch component includes a feature random masking module and an implicit representation collaborative mining module. Based on this, step S400, based on the third branch component of the event implicit representation neural network, performs feature random masking on the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data to obtain a multi-source fusion implicit representation, performs collaborative implicit representation mining on the multi-source fusion implicit representation, and obtains a multi-source collaborative implicit representation, including:
[0073] Step S410: Based on the feature random masking module, arbitrarily retain one or all of the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data;
[0074] Step S420: Based on the implicit representation collaborative mining module, the retained implicit representation is determined as a multi-source fusion implicit representation, and collaborative implicit representation mining is performed on the multi-source fusion implicit representation to obtain a multi-source collaborative implicit representation of the debugging learning example.
[0075] In step S410, the computer system uses the feature random masking module to process the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data. This step is intended to simulate the data missing or sensor failure that may occur in actual applications, and to improve the robustness of the model to incomplete data by randomly retaining or masking some data features. The feature random masking module determines whether each feature is retained or masked through a certain probability distribution (such as Bernoulli distribution). In actual operation, the module generates a random number for each feature. If the number is less than the preset masking probability (for example, 0.5), the feature value is set to 0 (or a predefined fill value), indicating that the feature is masked; otherwise, the feature value remains unchanged.
[0076] For example, suppose the encoded stereo visual data is implicitly represented as a vector v containing 100 features visual =[v1,v2,...,v 100 ], the encoded radar data is implicitly represented as another vector v containing 50 features radar =[r1,r2,...,r 50 ]. The feature random masking module first generates random numbers for each feature of the two vectors and decides whether to keep the feature based on the masking probability. For example, if the masking probability is 0.3, then about 30% of the features will be masked. After the feature random masking process, two partially masked vectors can be obtained, such as v visual′ =[v1,0,v3,...,0,v 99 ,v 100 ] and v radar′ =[r1,0,r3,...,r 48 ,0,r 50], where 0 represents an obscured feature. It is worth noting that since the obscuration is random, the location and amount of obscuration may be different each time it is processed. However, in step S410, the feature random obscuration module does not obscure all corresponding features in the two vectors at the same time, but allows at least one of the vectors to retain complete or partial information. This means that in some cases, only the visual data may be obscured while the radar data is retained, or vice versa; in rare cases, both may be partially obscured, but usually at least one of the vectors will retain enough features for subsequent processing.
[0077] In step S420, the data after feature random masking (i.e., multi-source fusion implicit representation) is sent to the implicit representation collaborative mining module for further processing. This module aims to mine and integrate complementary information between different data sources through collaborative learning methods to generate more expressive and robust multi-source collaborative implicit representations. The implicit representation collaborative mining module may use a variety of machine learning techniques to achieve collaborative processing of multi-source data, including but not limited to attention mechanisms, graph neural networks (GNNs), tensor decomposition, etc. These technologies can capture the complex relationships between different data sources and generate new representation spaces through nonlinear transformations.
[0078] For example, suppose that the implicit representation co-mining module uses an attention mechanism to process multi-source fusion implicit representations. The attention mechanism allows the model to dynamically adjust the importance weights of different data sources in the fusion process, thereby focusing on the most useful information for the current task.
[0079] In the case of random masking of features, some features may have been set to 0 or filler values, which will affect the calculation of the attention score. To handle this situation, the module can normalize or reweight the features before calculating the attention score to ensure that the masked features do not have too much impact on the final result. After processing by the attention mechanism, the module will generate a multi-source collaborative implicit representation that combines the implicit representation of encoded stereo vision data and the implicit representation of encoded radar data. This representation not only contains the key information from the two data sources, but also captures the interaction and complementarity between them through the attention mechanism.
[0080] Multi-source collaborative implicit representation has important application value in the road traffic visualization display scene based on virtual reality. It can not only improve the accuracy and real-time performance of traffic event recognition, but also provide more intuitive and comprehensive road traffic status information for traffic managers and the public. For example, in a virtual reality environment, multi-source collaborative implicit representation can be used to dynamically render elements such as vehicles, pedestrians, and traffic lights on the road, as well as to display information such as traffic congestion and accident warnings in real time. This information not only helps traffic managers make timely and effective decisions, but also improves the public's traffic safety awareness and travel efficiency. In summary, step S400 realizes the effective fusion of the implicit representation of coded binocular vision data and the implicit representation of coded radar data through two key steps: feature random masking and implicit representation collaborative mining. This process not only enhances the robustness of the model to incomplete data, but also improves the efficiency and accuracy of multi-source data fusion, providing a solid foundation for subsequent road traffic visualization.
[0081] As an implementation mode, the third branch component further includes a noise adding module. Based on this, the method further includes:
[0082] Step S110: based on the first denoising module between the binocular vision implicit representation encoding component and the feature random masking module, denoising the coded binocular vision data implicit representation to obtain the coded binocular vision data implicit representation after denoising;
[0083] Step S120: Based on the second denoising module between the radar data implicit representation coding component and the feature random masking module, denoising is performed on the coded radar data implicit representation to obtain the noisy coded radar data implicit representation.
[0084] In step S110, the first task of the computer system is to perform noise processing on the encoded implicit representation of binocular vision data. This step is performed immediately after the encoding of the implicit representation of binocular vision data, that is, after the binocular vision implicit representation encoding component completes its work. The purpose of noise addition is to simulate the data noise that may be encountered in practical applications, so that the model can learn how to remain robust in such a noisy environment during the training process. The first noise addition module is designed as a component that can inject noise into the encoded implicit representation of binocular vision data. The form of noise can be random, conforming to a specific distribution (such as Gaussian noise, salt and pepper noise, etc.), or it can be a carefully designed noise pattern designed to test the limits of the model.
[0085] For example, suppose the encoded binocular vision data is implicitly represented as a high-dimensional vector v visual =[v1,v2,...,v n ], where n is the dimension of the vector. The first noise module first generates a vector with v visualThe noise vector of the same dimension n=[n1,n2,...,n n ], where each noise component n i The noise vector is then added to the original vector to obtain the implicit representation v of the encoded binocular vision data after adding noise. visual′ =v visual +n.
[0086] The choice of noise parameters (such as the standard deviation σ of Gaussian noise) is an important design decision. Too small noise may not be enough to significantly improve the robustness of the model, while too large noise may cause the model to be unable to learn useful information from the noise. Therefore, in practical applications, it is usually necessary to optimize the noise parameters through methods such as cross-validation.
[0087] Similar to step S110, step S120 is for the implicit representation of the encoded radar data. In this step, the computer system uses the second noise addition module to perform noise addition processing on the encoded implicit representation of the radar data. Since radar data and binocular vision data are essentially different (for example, radar data is usually a one-dimensional time series, while binocular vision data is a two-dimensional image or image sequence), the second noise addition module can adopt different noise injection strategies.
[0088] For radar data, due to its time series characteristics, the noise addition module can consider the temporal correlation. This means that the injected noise may not only be random, but also have a certain temporal pattern. However, in order to simplify the description, an independent and identically distributed noise model similar to step S110 can still be used. For example, assuming that the encoded radar data is implicitly represented as a one-dimensional vector r=[r1,r2,...,r m ], where m is the length of the vector (i.e., the length of the time series). The second noise adding module also generates a noise vector n′=[n 1′ ,n 2′ ,...,n m′ ] and add it to the original vector to obtain the implicit representation of the noisy coded radar data r′=r+n′.
[0089] Integrating the denoising module into the overall method framework means that after encoding the binocular vision data and radar data and before performing feature random masking, the two encoded data representations are denoised separately. The noisy data representation will be used as the input of the feature random masking module, and then participate in the subsequent multi-source data fusion and collaborative implicit representation mining process. The benefit of this design is that it can introduce more variability and challenges to the model during the training process, forcing the model to learn how to extract useful information from noisy data. In this way, when the model is deployed in practical applications, it can maintain high recognition accuracy and robustness even in the face of noisy real data. In addition, the denoising module can also be used in combination with other data augmentation techniques (such as rotation, scaling, cropping, etc.) to further enrich the diversity of the training data set.
[0090] As an implementation mode, step S500 generates a traffic event recognition error according to the first traffic event recognition result, the second traffic event recognition result, the third traffic event recognition result and the traffic event label, and adjusts the parameter according to the traffic event recognition error, including:
[0091] Step S510: combining the first traffic event recognition result and the traffic event label to generate a first recognition error;
[0092] Step S520: combining the second traffic event recognition result and the traffic event label to generate a second recognition error;
[0093] Step S530: combining the third traffic event recognition result and the traffic event label to generate a third recognition error;
[0094] Step S540: Integrate the first recognition error, the second recognition error and the third recognition error to obtain a traffic event recognition error.
[0095] In step S510, the computer system first focuses on the first traffic event recognition result, which is the result obtained by processing the road binocular vision data through the first branch component (including the pre-adjusted binocular vision implicit representation mining component and the binocular vision implicit representation encoding component). In order to evaluate the accuracy of this recognition result, the system needs to compare it with the corresponding traffic event label.
[0096] Assume that the traffic event labels are in the form of multi-classification, and the category set is {0: 'normal', 1: 'congestion', 2: 'accident', 3: 'illegal driving'}. For each debugging learning example, there is a corresponding label, such as label 2, which means that the example describes an accident scene.
[0097] The output of the first branch component is, for example, a probability distribution vector, which indicates the probability that the input road binocular vision data belongs to each traffic event category. For example, for a certain sample, the output vector may be [0.1, 0.2, 0.6, 0.1], indicating that the model believes that the probability of the accident described by the sample is 60%, while the probabilities of congestion, normal and illegal driving are relatively low. In order to quantify the difference between the first traffic event recognition result and the true label, the computer system uses the cross entropy loss function as the error metric. The cross entropy loss function is a common method to measure the difference between two probability distributions, and its calculation formula is as follows:
[0098] ;
[0099] Where y is the one-hot encoding vector of the true label (for example, the one-hot encoding corresponding to the accident label 2 is [0, 0, 1, 0]), is the probability distribution vector predicted by the model (such as [0.1, 0.2, 0.6, 0.1] above). C is the total number of categories, and are the values of the true label and predicted probability on category c, respectively. Substituting the true label and predicted probability in the above example into the formula, the first recognition error can be calculated. This error value reflects the accuracy of the first branch component in identifying traffic events in road binocular vision data. The smaller the error, the closer the recognition result is to the actual situation.
[0100] Similar to step S510, step S520 focuses on the second traffic event recognition result, which is obtained based on the road millimeter wave radar data processed by the second branch component (including the pre-adjusted radar data implicit representation mining component and the radar data implicit representation encoding component). The system also uses the cross entropy loss function to calculate the second recognition error, but at this time, the difference between the output of the second branch component and the real traffic event label is compared. Similar to the first branch component, the output of the second branch component is also a probability distribution vector, which indicates the probability that the input radar data belongs to each traffic event category. For example, for the same sample, the output vector may be [0.05, 0.7, 0.2, 0.05], indicating that the model believes that the probability of congestion described by the sample is 70%. The process of generating the second recognition error: Similar to step S510, the system uses the cross entropy loss function to calculate the second recognition error. The second recognition error can be obtained by substituting the one-hot encoding vector of the real label and the output probability distribution vector of the second branch component into the cross entropy loss function formula. This error value reflects the accuracy of the second branch component in identifying traffic events in road millimeter wave radar data.
[0101] Step S530 focuses on the third traffic event recognition result, which is obtained by fusing the implicit representation of encoded binocular vision data and the implicit representation of encoded radar data through the third branch component (including the feature random masking module and the implicit representation collaborative mining module). Similar to the previous two steps, the system still uses the cross entropy loss function to calculate the third recognition error.
[0102] The output of the third branch component is also a probability distribution vector, which indicates the probability that the fused multi-source data belongs to each traffic event category. For example, for the same example, the output vector may be [0.03, 0.85, 0.08, 0.04], indicating that the model believes that the probability of congestion described by the fused data is 85%. The system uses the cross entropy loss function to calculate the third recognition error. The third recognition error can be obtained by substituting the one-hot encoding vector of the true label and the output probability distribution vector of the third branch component into the cross entropy loss function formula. This error value reflects the accuracy of the third branch component in identifying traffic events after fusing binocular vision data and radar data, and is also an important indicator of the overall model performance.
[0103] In step S540, the computer system integrates the first recognition error, the second recognition error and the third recognition error to obtain a comprehensive traffic event recognition error. This comprehensive error not only reflects the performance of a single branch component, but also reflects the effectiveness of the multi-source data fusion strategy. Method for integrating recognition errors: The specific method for integrating recognition errors can be designed according to actual needs. A simple and effective method is to perform a weighted summation of the three recognition errors:
[0104] ;
[0105] Among them, L1, L2, and L3 are the first, second, and third recognition errors respectively. , , is the corresponding weight coefficient. The weight coefficient can be determined according to the importance of different branch components, the reliability of data, and the requirements of actual application scenarios.
[0106] In practical applications, the choice of weight coefficient is an important design decision. If binocular vision data is considered more important in traffic event recognition, Set to a higher value; if radar data is more reliable in certain scenarios, such as at night or in bad weather conditions, At the same time, because the third branch component integrates multi-source data and performs complex collaborative mining processing, The value of should also be given enough attention. By integrating the recognition error and selecting appropriate weight coefficients, the system can comprehensively evaluate the performance of the event implicit representation neural network in multiple dimensions and provide strong guidance for subsequent parameter adjustment.
[0107] After calculating the comprehensive traffic event recognition error, the computer system will guide the parameter adjustment process of the event implicit representation neural network based on this error value. Parameter adjustment is one of the core links in neural network training. It adjusts the parameter values such as weights and biases in the network through optimization algorithms (such as gradient descent method, Adam optimizer, etc.) to minimize the recognition error. Parameter adjustment is an iterative process. In each iteration, the system first calculates the recognition error based on the current parameter value (i.e., the forward propagation process), and then calculates the gradient based on the recognition error and updates the parameter value (i.e., the back propagation process). Through multiple iterative debugging, the system gradually approaches the optimal solution and minimizes the recognition error. When the recognition error converges to a stable value or reaches the preset number of iterations, the training process ends and the debugged event implicit representation neural network model is output.
[0108] As an implementation manner, step S700, obtaining a target traffic event recognition result based on the debugged event implicit representation neural network recognition, may include:
[0109] Step S701: calling the debugged event implicit representation neural network to perform implicit representation mining on the target radar-visual fusion data to obtain the traffic event implicit representation of the target radar-visual fusion data;
[0110] Step S702: performing traffic event recognition on the implicit representation of traffic events in the target radar-visual fusion data based on the classifier to obtain a target traffic event recognition result.
[0111] In step S701, the target radar-visual fusion data includes road binocular vision data and road millimeter-wave radar data. The fusion of these two data sources provides a rich information basis for the accurate identification of traffic events.
[0112] The event implicit representation neural network is a complex model that has been trained with a large number of debugging learning examples and successfully converged. It contains multiple branch components, each of which is optimized for a specific data type. When calling the model, the computer system first needs to preprocess the target radar-visual fusion data in the format required by the network to ensure that the dimension and type of the data meet the input requirements. Once the target radar-visual fusion data is successfully input into the neural network, the model begins to perform the task of implicit representation mining. This process involves multiple steps, including data encoding, feature extraction, multi-source data fusion, and implicit representation generation. The binocular vision data and millimeter-wave radar data are first processed by their respective encoding components. For binocular vision data, this may include the extraction of image features by a convolutional neural network (CNN); for millimeter-wave radar data, a recurrent neural network (RNN) or a long short-term memory network (LSTM) may be used to process time series data. These encoding components convert the raw data into a high-dimensional implicit representation vector. During the encoding process, the neural network automatically learns and extracts key features from the data. For example, for binocular vision data, the model may learn features such as the shape, color, and direction of travel of the vehicle; for millimeter-wave radar data, it may learn features such as the speed, acceleration, and distance of the vehicle. After obtaining the implicit representation of binocular vision data and millimeter-wave radar data, the neural network effectively fuses these data from different sources through mechanisms such as feature random masking and collaborative implicit representation mining. This process not only considers the feature relationship within a single data source, but also explores the interaction and influence between different data sources. Ultimately, the neural network generates an implicit representation of traffic events that integrates multi-source information. This representation not only contains rich feature information, but also implies the inherent logic and laws of traffic events. It is the basis for subsequent traffic event recognition.
[0113] Assume that the target radar-visual fusion data contains a set of binocular visual images and corresponding millimeter-wave radar data. Through the processing of the event implicit representation neural network, the system may generate an implicit representation vector containing hundreds of elements. Each element in this vector represents a specific feature or attribute, such as vehicle speed, lane occupancy, traffic flow density, etc. These features together constitute a comprehensive description of the current traffic conditions. In step S702, the computer system uses a classifier to further process the implicit representation of traffic events generated in step S701 to identify specific traffic events.
[0114] During the training process of the event implicit representation neural network, the classifier is usually jointly optimized as part of the network. This means that parameters such as the weights and biases of the classifier are automatically adjusted during the training process of the entire network. However, in some cases, the event implicit representation neural network can also be trained first to generate a high-quality implicit representation vector, and then a classifier is trained separately to identify traffic events. Commonly used classifiers include support vector machines (SVM), decision trees, random forests, and fully connected neural networks (FCN) in deep learning. In this embodiment, since the implicit representation of traffic events is already a high-dimensional vector representation, using a fully connected neural network as a classifier is, for example, a suitable choice.
[0115] During the traffic event recognition process, the classifier receives the implicit representation vector from the event implicit representation neural network as input. Then, it maps this vector to the predefined traffic event category based on the learned classification boundary or decision rule. For example, if the predefined traffic event categories include "normal driving", "congestion", "accident" and "illegal driving", the classifier will output a four-dimensional probability distribution vector, indicating the probability that the input data belongs to these four categories. Finally, the computer system determines the recognition result of the target traffic event based on the output of the classifier. For example, if the probability distribution vector output by the classifier is [0.1, 0.8, 0.05, 0.05], the system can judge that the current road condition is "congested" and use this result in subsequent virtual reality visualization or other traffic management tasks.
[0116] As another implementation, step S700, obtaining a target traffic event recognition result based on the debugged event implicit representation neural network recognition, includes:
[0117] Step S710: determining road binocular vision data having the same quality as the target radar vision fusion data in the high-frequency event radar vision fusion database;
[0118] Step S720: Based on the debugged event implicit representation neural network, corresponding multi-source collaborative implicit representations are respectively extracted for the road binocular vision data with homogeneity, and the multi-source collaborative implicit representation is the traffic event implicit representation of the road binocular vision data;
[0119] Step S730: determining the implicit representation of traffic events of target radar-vision fusion data according to the implicit representation of traffic events of binocular vision data of each road with homogeneous conditions;
[0120] Step S740: According to the implicit representation of traffic events in the target radar-visual fusion data and the implicit representation library of traffic events, determine the road binocular vision data that has a commonality threshold with the target radar-visual fusion data in the traffic event fusion database; the implicit representation of traffic events in the implicit representation library of traffic events is obtained by extracting the corresponding multi-source collaborative implicit representation from the binocular vision data of each road in the traffic event fusion database based on the debugged event implicit representation neural network;
[0121] Step S750: The traffic events corresponding to the road binocular vision data greater than the commonality threshold are taken as the traffic events of the target radar-vision fusion data.
[0122] In step S710, the primary task of the computer system is to search the high-frequency event radar fusion database for road binocular vision data with similar characteristics to the target radar fusion data. The high-frequency event radar fusion database is a warehouse that stores a large amount of historical traffic event data. These data cover various common traffic conditions, such as congestion, accidents, illegal driving, etc., and these events appear frequently in the database, so they are called "high-frequency events". When constructing the high-frequency event radar fusion database, it is necessary to collect a large amount of road binocular vision data and corresponding millimeter-wave radar data, and classify and index these data according to dimensions such as time, location, and event type. At the same time, each event needs to be annotated in detail, including the specific time and location of the event, the number of vehicles involved, casualties, traffic impact range, and other information. This information is crucial for subsequent data retrieval and event analysis.
[0123] In order to determine the road binocular vision data that is homogeneous with the target radar-vision fusion data, the computer system needs to adopt an effective similarity measurement method. A common method is based on the cosine similarity calculation of feature vectors. First, the system converts the road binocular vision data in the target radar-vision fusion data into feature vectors (for example, by extracting deep features of the image through a pre-trained CNN model). Then, the feature vectors of all road binocular vision data in the high-frequency event radar-vision fusion database are traversed, and the cosine similarity between them and the target feature vector is calculated. Finally, the data with a similarity higher than the set threshold is selected as the data set of homogeneity.
[0124] For example, suppose that the road binocular vision image in the target radar-vision fusion data shows a traffic jam scene at an intersection. The computer system first extracts the feature vector of the image and searches for similar images in the high-frequency event radar-vision fusion database. Through cosine similarity calculation, the system finds several historical images that also describe the traffic jam scene at the intersection. The similarity between the feature vectors of these images and the target feature vector exceeds the set threshold (such as 0.8). The data sets corresponding to these images are identified as data sets with the same homogeneity as the target radar-vision fusion data.
[0125] In step S720, after determining the homogeneous data set, the computer system will then extract the multi-source collaborative implicit representation of the road binocular vision data (and the matching millimeter-wave radar data) in these data sets based on the debugged event implicit representation neural network. After debugging and training, the event implicit representation neural network has the ability to extract key features from the road binocular vision data and millimeter-wave radar data and generate implicit representations. In this step, the system inputs each set of data in the homogeneous data set (including the road binocular vision image and millimeter-wave radar data) into the neural network and obtains its corresponding multi-source collaborative implicit representation.
[0126] Multi-source collaborative implicit representation is the result of deep fusion and collaborative processing of road binocular vision data and millimeter-wave radar data by the neural network. It not only contains the key feature information from the two data sources, but also captures the intrinsic connection and interaction between these information through the complex interaction mechanism within the neural network. This representation method makes subsequent analysis and recognition tasks more accurate and efficient.
[0127] In step S730, after obtaining the implicit representation of traffic events of binocular vision data of all roads in the homogeneous situation data set, the computer system needs to comprehensively judge the implicit representation of traffic events corresponding to the target radar-visual fusion data based on this information. A simple and effective method is to perform aggregation processing on the implicit representation of traffic events of all homogeneous situation data. The aggregation processing method can be designed according to specific needs, for example, the average method, the median method, the weighted average method, etc. can be used. In this embodiment, it is assumed that the average method is used for aggregation processing. That is, firstly, the average values of the implicit representation of traffic events of all homogeneous situation data in each dimension are calculated, and then these average values are used as the implicit representation of traffic events of the target radar-visual fusion data.
[0128] In step S740, after obtaining the implicit representation of the traffic event of the target radar-visual fusion data, the computer system needs to further search for similar event cases in the traffic event fusion database. The traffic event fusion database is a comprehensive database containing representative data of various traffic events, which not only covers common traffic event types, but also records the specific circumstances and characteristics of each event in detail.
[0129] Before searching for similar events, the computer system builds a traffic event implicit representation library. This library is the result of multi-source collaborative implicit representation extraction of all road binocular vision data in the traffic event fusion database based on the debugged event implicit representation neural network. Each event in the library corresponds to a multi-dimensional implicit representation vector, which reflects the key features and internal connections of the event.
[0130] In the implicit representation library of traffic events, the computer system uses a method similar to that in step S710 to calculate the similarity between the implicit representation of the traffic event in the target radar-visual fusion data and the implicit representation of each event in the library. Similarly, cosine similarity can be used as a similarity metric. By calculating the similarity and sorting the result set, the system can find several event cases with the highest similarity to the target event.
[0131] In order to ensure the accuracy and reliability of the search results, the computer system needs to set a commonality threshold to screen the similarity. Only when the similarity is higher than this threshold, the corresponding event case will be considered as a case with significant commonality with the target event. This threshold can be set and adjusted according to actual needs and database characteristics.
[0132] In step S750, the traffic event type corresponding to the target radar-visual fusion data is determined according to the similar event cases found in step S740. Since these similar event cases have been clearly labeled and classified in the traffic event fusion database, their category labels can be directly used as the category labels of the target event. If the system finds multiple similar event cases with high similarity (i.e., event cases whose similarity is higher than the commonality threshold), the majority voting method can be used to determine the final event category. That is, the category label distribution of all similar event cases is counted, and the category label with the largest number of occurrences is selected as the category label of the target event. For example, suppose that three similar event cases A, B, and C with high similarity are found in the traffic event fusion database. Among them, A and B are both labeled as "congestion" events, and C is labeled as "accident" events. Since the number of occurrences of "congestion" events (2 times) is greater than that of "accident" events (1 time), the system finally determines the traffic event type corresponding to the target radar-visual fusion data as "congestion".
[0133] As an implementation mode, step S710, determining road binocular vision data having the same properties as target radar vision fusion data in a high-frequency event radar vision fusion database, includes: based on a pre-adjusted binocular vision implicit representation mining component, performing data sampling on the target radar vision fusion data to obtain a road binocular vision image set, extracting the road binocular vision data implicit representation of each road binocular vision image in the road binocular vision image set, for each road binocular vision data implicit representation, determining the corresponding common road binocular vision data implicit representation in the road binocular vision data implicit representation library of the high-frequency event radar vision fusion database, and determining the road binocular vision data corresponding to the common road binocular vision data implicit representation as road binocular vision data having the same properties as the target radar vision fusion data.
[0134] First, the computer system uses the pre-adjusted binocular vision implicit representation mining component to perform data sampling on the target radar-vision fusion data. Data sampling is an important step in data processing, which aims to select a representative subset from the original data for analysis to reduce the amount of calculation and improve processing efficiency. In this embodiment, the sampling operation is mainly performed on the road binocular vision part in the target radar-vision fusion data.
[0135] The pre-adjusted binocular vision implicit representation mining component is a pre-trained machine learning model or neural network component that has the ability to extract key features from binocular vision images and generate implicit representations. These implicit representations are usually in the form of high-dimensional vectors that can capture complex information and intrinsic structures in the image. By pre-adjusting the binocular vision implicit representation mining component, the computer system samples the road binocular vision images in the target radar fusion data to obtain a road binocular vision image set containing multiple images. Then, for each image in the set, the component extracts its road binocular vision data implicit representation. These implicit representations serve as the basis for subsequent searches for homogeneous situation data.
[0136] After extracting the implicit representation of road binocular vision data from the target radar-vision fusion data, the computer system then needs to perform search and matching operations in the implicit representation library of road binocular vision data in the high-frequency event radar-vision fusion database. The high-frequency event radar-vision fusion database is a warehouse containing a large amount of historical traffic event data, of which the implicit representation library of road binocular vision data is an important part of the database. Each entry in the library corresponds to a set of implicit representations of road binocular vision data and its related traffic event information. This information is obtained by processing and analyzing a large amount of historical data in advance, which provides strong support for subsequent data retrieval and event recognition.
[0137] In order to find road binocular vision data that is homogeneous with the target radar-vision fusion data, the computer system first compares the implicit representation of the target data with the entries in the implicit representation library one by one. This comparison process usually involves calculating the similarity or distance between the target implicit representation and the implicit representation in the library. Commonly used similarity measurement methods include cosine similarity, Euclidean distance, etc.
[0138] In this embodiment, the computer system may use cosine similarity as a similarity metric. Cosine similarity evaluates the similarity of two vectors by calculating the degree of proximity in direction, and its calculation formula is as follows:
[0139] ;
[0140] Among them, A and B are two high-dimensional vectors that need to be compared. and are the components of vectors A and B respectively, and n is the dimension of the vector. The computer system traverses all entries in the implicit representation library and calculates the cosine similarity between the target implicit representation and the implicit representation in each library. Then, they are sorted according to the similarity value, and the entries with similarity higher than the set threshold are selected as candidate results. Finally, the computer system determines the road binocular vision data that has homogeneous conditions with the target radar fusion data based on the common road binocular vision data implicit representation in the candidate results. Common road binocular vision data implicit representations refer to those implicit representations that are highly similar to the target implicit representation in the implicit representation library. Since the road binocular vision data corresponding to these implicit representations have significant similarities with the target data in terms of features, structure or content, they can be considered to have homogeneous conditions with the target data. In order to determine these homogeneous data, the computer system first checks each entry in the candidate results to confirm whether the corresponding road binocular vision data is indeed similar to the target data. This process may involve further analysis and comparison of the image content to ensure the accuracy of the results. Once homogeneous situation data are confirmed, the computer system extracts these data and prepares them for subsequent multi-source collaborative implicit representation extraction and traffic event recognition tasks.
[0141] Assume that the target radar-vision fusion data contains a set of binocular vision images describing the traffic conditions at the intersection. The computer system first extracts the road binocular vision data implicit representation of these images by pre-adjusting the binocular vision implicit representation mining component. Then, it searches in the road binocular vision data implicit representation library of the high-frequency event radar-vision fusion database to find several entries with high similarity to the target implicit representation. The road binocular vision data corresponding to these entries may also describe the traffic conditions at the intersection, but may involve different time periods, weather conditions, or traffic flow. Through further analysis and comparison, the computer system confirms that several of the entries do have homogeneous conditions with the target data. These homogeneous condition data are then used in the extraction of multi-source collaborative implicit representations and the identification of traffic events.
[0142] As an implementation mode, the method further includes a process of generating a traffic event implicit representation library, specifically including:
[0143] Step S74A: Acquire a traffic event fusion database;
[0144] Step S74B: Based on the debugged event implicit representation neural network, extract the multi-source collaborative implicit representation of each road binocular vision data in the traffic event fusion database respectively, and the multi-source collaborative implicit representation is the traffic event implicit representation of the road binocular vision data;
[0145] Step S74C: Generate a traffic event implicit representation library according to the traffic event implicit representation of the binocular vision data of each road in the traffic event fusion database.
[0146] In step S74A, a traffic event fusion database is obtained. This database is a comprehensive resource library that integrates various types of traffic event data. It not only contains road binocular vision data, but may also cover multiple information sources such as millimeter-wave radar data, traffic flow data, and weather condition data. These data come from different acquisition systems and sensor networks, and after cleaning, integration and annotation, a structured data set is formed. The data of the traffic event fusion database may come from multiple channels such as the urban traffic management center, traffic police department, road monitoring system, vehicle GPS tracking system, etc. These data are collected and transmitted to the data center for processing through specific interfaces or protocols. In the data integration stage, the computer system will unify the format, synchronize time, and remove duplicates of data from different sources to ensure the accuracy and consistency of the data. At the same time, the data will be annotated to provide supervision signals for subsequent machine learning model training.
[0147] In step S74B, after obtaining the traffic event fusion database, the computer system will then use the debugged event implicit representation neural network to extract the multi-source collaborative implicit representation of each road binocular vision data. This process is actually a re-application of the neural network. Through the powerful feature extraction and fusion capabilities of the neural network, the key information in the road binocular vision data is converted into a higher-level and more abstract representation. The debugged event implicit representation neural network already has the ability to extract multi-source collaborative implicit representations from road binocular vision data and millimeter-wave radar data. In this step, although only road binocular vision data is involved (assuming that millimeter-wave radar data is not required or has been integrated into the road binocular vision data as a preprocessing step), the neural network can still use the feature extraction and fusion strategies learned during the training process to generate high-quality implicit representations.
[0148] Multi-source collaborative implicit representation is a high-dimensional vector or matrix obtained by the neural network after in-depth analysis of road binocular vision data. It not only contains the visual features in the image (such as vehicle shape, color, driving trajectory, etc.), but also may imply the interactions and relationships between these features. This representation makes subsequent analysis and recognition tasks more efficient and accurate.
[0149] For example, suppose the traffic event fusion database contains a set of road binocular vision data describing the scene of a traffic accident. Through the processing of the event implicit representation neural network, these data are converted into a series of multi-dimensional implicit representation vectors. Each vector corresponds to a specific moment or perspective in a set of road binocular vision images, and contains the key feature information of the traffic accident at that moment or perspective.
[0150] In step S74C, after extracting the multi-source collaborative implicit representations of all road binocular vision data, the computer system stores these implicit representations in the traffic event implicit representation library. This library is a database system specifically used to store and retrieve implicit representations of traffic events, which supports efficient indexing, querying and matching operations. The traffic event implicit representation library usually includes multiple tables or sets to store different types of data. For example, a main table can be designed to store the identifier of each implicit representation (such as a unique ID), the corresponding road binocular vision data identifier, the event type, the timestamp and other information; at the same time, an auxiliary table can also be designed to store the data of the implicit representation vector itself. In order to improve the retrieval efficiency, the traffic event implicit representation library will index the implicit representation vector. Commonly used indexing methods include K-means clustering, LSH (local sensitive hashing), etc. These methods can group similar implicit representations together, so that candidate sets similar to the target implicit representation can be quickly located during retrieval.
[0151] For example, suppose the implicit representation library of traffic events already contains a large number of implicit representation vectors of historical traffic accidents. When the computer system receives a new set of road binocular vision data, it will first extract the multi-source collaborative implicit representation vector of the data; then, perform a search operation in the implicit representation library of traffic events to find a set of historical implicit representations similar to the vector; finally, infer the possible type and occurrence time of the current traffic event based on the event type and timestamp information corresponding to these similar implicit representations.
[0152] As an implementation mode, step S740, based on the implicit representation of traffic events of target radar-visual fusion data and the implicit representation library of traffic events, determines in the traffic event fusion database the road binocular vision data having a commonality threshold with the target radar-visual fusion data, including:
[0153] Step S741: based on the implicit representation of the traffic event of the target radar-visual fusion data, determine the implicit representation of the traffic event in the traffic event implicit representation library to obtain a set of implicit representations of the traffic event;
[0154] Step S742: binning the implicit representation set of traffic events to obtain multiple feature bins and the bin centroid of each feature bin, where the bin centroid is the average value of the implicit representation of traffic events in the feature bin;
[0155] Step S743: Determine the implicit representation of the traffic event that is most similar to the centroid of the corresponding bin in each of the multiple feature bins, and determine the road binocular vision data corresponding to the determined multiple implicit representations of traffic events as the road binocular vision data that has a commonality threshold with the target radar vision fusion data.
[0156] In step S741, the computer system searches the implicit representation library of traffic events based on the implicit representation of the traffic event in the target radar-vision fusion data to find a set of implicit representations similar to it.
[0157] The traffic event implicit representation library is a collection of implicit representations of a large amount of historical traffic event data. These data implicit representations are obtained by processing the road binocular vision data in the traffic event fusion database through the debugged event implicit representation neural network. Each implicit representation captures the key features and traffic event information in the corresponding road binocular vision data.
[0158] In order to find a set of implicit representations that are similar to the implicit representation of the target radar-vision fusion data, the computer system needs to adopt a similarity measurement method. Commonly used similarity measures include cosine similarity, Euclidean distance, etc. Here, it can be assumed that cosine similarity is used as a metric because it can effectively evaluate the degree of proximity between two vectors in direction.
[0159] The computer system traverses all implicit representations in the traffic event implicit representation library and calculates the cosine similarity between them and the target radar-visual fusion data implicit representation. Then, they are sorted according to the similarity value, and implicit representations with similarity higher than a set commonality threshold (such as 0.8) are selected as candidate sets. This commonality threshold is set according to the actual application scenario and data characteristics to ensure that the selected implicit representation has sufficient similarity with the target implicit representation.
[0160] After determining a set of implicit representations that are similar to the implicit representation of the target radar-vision fusion data, the computer system then performs binning on these implicit representations. Binning is a commonly used data preprocessing method that can divide a continuous data range into several discrete intervals, with each box containing data points within a certain range. The purpose of binning is to classify similar implicit representations into the same box for more efficient subsequent analysis and processing. Through binning, the computer system can reduce the dimensionality and complexity of the data while retaining the main features and information of the data. Common binning methods include equal-width binning, equal-frequency binning, and so on. Here, we can assume that the equal-width binning method is used as an example. Equal-width binning divides each dimension of the implicit representation into intervals of the same width, and then assigns them to the corresponding boxes according to the values of the implicit representation in each dimension. For the implicit representation set Each implicit representation in , the computer system bins its values in each dimension. Assuming that the implicit representation has n dimensions and each dimension is divided into k bins, a total of n×k bins will be generated. Then, for all implicit representations in each feature bin, the computer system calculates their average (i.e., bin centroid) to represent the central tendency of the implicit representation in the bin. For example, assuming the implicit representation set The implicit representations in are all two-dimensional vectors (for simplicity), and each dimension is divided into three boxes of equal width. A total of six boxes are generated. For each implicit representation in a box, the computer system calculates its average value as the centroid of the box. For example, in a feature box, there are 3 implicit representations , , , then the centroid of the box is:
[0161] ;
[0162] In step S743, the ultimate goal of the computer system is to find road binocular vision data that is greater than the commonality threshold with the target radar vision fusion data. This process involves finding the historical implicit representation that is most similar to the target implicit representation in the feature bins, and outputting the corresponding road binocular vision data as the result. For each feature bin, the computer system first calculates the similarity between the bin centroid and the target radar vision fusion data implicit representation. Then, all implicit representations are traversed in the bin, and the implicit representation with the highest similarity to the bin centroid is found as the representative of the bin. This ensures that the selected implicit representation is not only similar to the target implicit representation, but also has a certain representativeness in the bin. After determining the representative implicit representation of each bin, the computer system needs to apply the commonality threshold again to filter the results. That is, only when the similarity between the representative implicit representation and the target implicit representation is higher than the set commonality threshold (here, for example, a lower threshold than that in step S741), the corresponding road binocular vision data will be determined as the final result. This step is intended to further ensure that the selected data has significant commonality with the target data.
[0163] Through the implementation of the above step S740, the computer system can efficiently find road binocular vision data that has significant commonalities with the target radar-vision fusion data in the traffic event fusion database. This process not only relies on the implicit representation of the target data and the rich resources in the traffic event implicit representation library, but also makes full use of the efficiency and accuracy of data binning technology. In practical applications, this method can provide powerful data support and analysis means for the visualization of road traffic based on virtual reality. At the same time, by adjusting the hyperparameter settings such as commonality thresholds and binning parameters, the computer system can further optimize the quality and efficiency of the search results.
[0164] As an embodiment, the method further includes:
[0165] Step S7401: obtaining the target road binocular vision data determined from the road binocular vision data having a commonality threshold with the target radar vision fusion data;
[0166] Step S7402: Obtain the target feature bins where the implicit representation of the traffic event in the binocular vision data of the target road is located;
[0167] Step S7403: determining the average value of the implicit representation of traffic events in the road binocular vision data having the same condition as the target radar-visual fusion data and the implicit representation of traffic events in the target feature bin as the updated implicit representation of traffic events in the target radar-visual fusion data;
[0168] Step S7404: According to the updated implicit representation of traffic events and the implicit representation library of traffic events, determine the road binocular vision data having a commonality threshold greater than the target radar vision fusion data from the traffic event fusion database.
[0169] In step S7401, the primary task of the computer system is to obtain part or all of the road binocular vision data determined in the previous step (such as step S740) and having a commonality threshold with the target radar vision fusion data as the target road binocular vision data. These data are selected because they show a high degree of similarity with the target radar vision fusion data in terms of traffic event characteristics, and therefore can provide valuable information for updating the implicit representation of traffic events in the target radar vision fusion data. For example, assume that in step S740, by comparing the implicit representation of traffic events in the target radar vision fusion data with the entries in the implicit representation library of traffic events, the computer system finds multiple implicit representations with similarities above the commonality threshold. These implicit representations correspond to different road binocular vision data in the traffic event fusion database. From them, the system selects a set of the most representative road binocular vision data as the target road binocular vision data, which may contain multiple images at different times and different perspectives but all describing similar traffic conditions.
[0170] In step S7402, after the target road binocular vision data is determined, the next step of the computer system is to find the feature bins corresponding to the implicit representations of traffic events of these data in the implicit representation library of traffic events. The feature bins are obtained by binning the implicit representation set in the previous step (such as step S742), and each bin contains a group of implicit representations that are similar in feature space.
[0171] In step S7403, after determining the target feature bin, the computer system will perform an update operation to generate an updated implicit representation of traffic events in the target radar-visual fusion data. This update process combines the implicit representation of traffic events in the road binocular vision data with the target radar-visual fusion data and the implicit representation of traffic events in the target feature bin.
[0172] One possible update strategy is to perform a weighted average of these implicit representations. Specifically, the computer system first calculates the average of the implicit representations of traffic events in the road binocular vision data that has homogeneous conditions with the target radar fusion data as the representative of the implicit representation of homogeneous conditions. Then, it calculates the average of all implicit representations in the target feature bin as the central tendency of the bin. Finally, the two averages are weighted and summed according to a certain weight ratio to obtain the updated implicit representation of traffic events.
[0173] The weight distribution can be adjusted according to the actual situation and needs. For example, if the data volume of homogeneous situations is large and the similarity with the target radar fusion data is very high, then these data can be given a higher weight; conversely, if the data in the target feature bin is more extensive and represents a more general traffic situation, then it can be given a lower weight but ensure its contribution in the updated representation.
[0174] In step S7404, after obtaining the updated implicit representation of the traffic event, the computer system will execute a process similar to step S740, but this time it is based on the updated representation to search the traffic event fusion database for road binocular vision data that has a commonality threshold with the target radar vision fusion data.
[0175] The computer system uses the updated traffic event implicit representation v updated As the query vector, it searches in the implicit representation library of traffic events. It calculates the correlation between each implicit representation in the library and v updated The similarity between the two images (such as cosine similarity) is calculated, and implicit representations with similarity higher than the newly set commonality threshold (this threshold may be different from that in step S740) are screened out. Then, the road binocular vision data corresponding to these implicit representations are output as the final result.
[0176] As an implementation method, the steps of extracting corresponding multi-source collaborative implicit representation of road binocular vision data by the debugged event implicit representation neural network include:
[0177] Step S71: Based on the first branch component of the debugged event implicit representation neural network, implicit representation encoding is performed on the initial road binocular vision data implicit representation of the road binocular vision data to obtain the encoded binocular vision data implicit representation;
[0178] Step S72: Based on the debugged second branch component of the event implicit representation neural network, implicit representation encoding is performed on the initial radar data implicit representation of the road millimeter wave radar data matched with the road binocular vision data to obtain the encoded radar data implicit representation;
[0179] Step S73: Based on the debugged third branch component of the event implicit representation neural network, a multi-source fusion implicit representation is obtained according to the encoded binocular vision data implicit representation and the encoded radar data implicit representation, and collaborative implicit representation mining is performed on the multi-source fusion implicit representation to obtain a multi-source collaborative implicit representation; the multi-source collaborative implicit representation is an implicit representation of traffic events in road binocular vision data.
[0180] In step S71, the first task of the computer system is to process the road binocular vision data. These data usually contain a series of binocular image pairs, each pair of images capturing the road scene from two different perspectives. In order to extract useful information from these data, the system first uses the first branch component of the debugged event implicit representation neural network to encode the initial road binocular vision data implicit representation.
[0181] The first branch component is a neural network part specially designed for processing binocular vision data. It may contain multiple convolutional layers, pooling layers, activation layers, etc., which are used to extract low-level to high-level features in the image. These features may include edges, textures, shapes, object recognition, etc., which together constitute a preliminary understanding of the road scene. The initial implicit representation of road binocular vision data is, for example, a low-dimensional vector or matrix containing the basic information of the original image. In the first branch component, this initial representation first undergoes a series of convolution operations, and each convolution layer slides on the input data through a set of learnable convolution kernels to generate a series of feature maps. These feature maps are then downsampled through the pooling layer to reduce the amount of calculation and extract the main features. The activation layer (such as the ReLU layer) is used to introduce nonlinear factors to enhance the expression ability of the network. After multiple layers of convolution, pooling and activation operations, the first branch component outputs an encoded implicit representation of binocular vision data. This encoded representation is a high-dimensional vector that captures the key features in the road binocular vision data and provides a basis for subsequent fusion and mining.
[0182] In step S72, the computer system then processes the road millimeter wave radar data matched with the road binocular vision data. These data exist in the form of time series and provide information about the distance, speed, direction, etc. of vehicles and other objects on the road. In order to extract useful information from these data, the system uses the second branch component of the debugged event implicit representation neural network to encode the initial radar data implicit representation. The second branch component is a neural network part specially designed for processing radar data. It may use a recurrent neural network (RNN) or its variants (such as LSTM, GRU) to capture dynamic changes and long-term dependencies in time series data. These network structures can effectively process the timestamp information in the radar data and extract key features such as the vehicle's motion state, acceleration, trajectory, etc. The initial radar data implicit representation is, for example, a one-dimensional array or a two-dimensional matrix, which contains information such as the distance and speed of all points scanned by the radar. In the second branch component, this initial representation is first fed into the RNN or its variant network. The network iteratively processes the data points of each time step, gradually updates its internal state, and generates a series of hidden state vectors. These hidden state vectors together constitute the encoded representation of the radar data.
[0183] After being processed by the RNN or its variant network, the second branch component outputs an encoded radar data implicit representation. This encoded representation is also a high-dimensional vector that captures the key features in the radar data, especially the information about the dynamic behavior of the vehicle.
[0184] In step S73, the computer system fuses the implicit representation of the encoded binocular vision data obtained in steps S71 and S72 with the implicit representation of the encoded radar data to generate a multi-source collaborative implicit representation. This process involves the functional implementation of the third branch component, which is responsible for processing the fusion and collaborative mining tasks of multi-source data.
[0185] The third branch component may include multiple sub-modules such as feature random masking module, attention mechanism module, implicit representation collaborative mining module, etc. The feature random masking module is used to simulate data missing conditions and improve the robustness of the model; the attention mechanism module is used to dynamically adjust the importance weights of different data sources in the fusion process; and the implicit representation collaborative mining module is responsible for performing specific fusion and mining tasks.
[0186] First, the feature random masking module randomly masks the implicit representation of encoded binocular vision data and the implicit representation of encoded radar data to simulate the incomplete data that may occur in actual applications. Then, the attention mechanism module calculates the attention scores (i.e., correlation weights) between different data sources based on the masked implicit representations to determine their contribution in the fusion process.
[0187] Next, the implicit representation collaborative mining module performs weighted fusion processing on the masked implicit representation according to the attention score to generate a multi-source fused implicit representation. This representation contains both spatial information in the binocular vision data (such as vehicle position and shape, etc.) and temporal information in the radar data (such as vehicle speed and acceleration, etc.), achieving a comprehensive description of road traffic conditions.
[0188] Finally, the implicit representation collaborative mining module further mines the multi-source fusion implicit representation to extract the key features and internal laws. This process may involve complex nonlinear transformations and feature interaction analysis, and finally generates a multi-source collaborative implicit representation. This representation is an implicit representation of traffic events in road binocular vision data, which accurately reflects the type and severity of traffic events on the current road.
[0189] Figure 2 A hardware entity diagram of a computer system provided in an embodiment of the present application is shown in FIG. Figure 2As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and the processor 1001 implements the steps in the method of any of the above embodiments when executing the program.
[0190] The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A road traffic visualization display method based on virtual reality, characterized in that: The method comprises: Acquire debugging learning samples, each of which includes road binocular vision data, road millimeter-wave radar data matched with the road binocular vision data, and traffic event labels; Based on the first branch component of the event implicit representation neural network, the initial road binocular vision data implicit representation of the road binocular vision data is implicitly encoded to obtain the encoded binocular vision data implicit representation, and the first traffic event recognition result for the debugging learning example is obtained according to the encoded binocular vision data implicit representation; Based on the second branch component of the event implicit representation neural network, the initial radar data implicit representation of the road millimeter-wave radar data is implicitly represented and encoded to obtain an encoded radar data implicit representation, and a second traffic event recognition result for the debugging learning example is obtained according to the encoded radar data implicit representation; Based on the third branch component of the event implicit representation neural network, the encoded binocular vision data implicit representation and the encoded radar data implicit representation are subjected to feature random masking to obtain a multi-source fusion implicit representation, the multi-source fusion implicit representation is subjected to collaborative implicit representation mining to obtain a multi-source collaborative implicit representation, and a third traffic event recognition result for the debugging learning example is obtained according to the multi-source collaborative implicit representation; the multi-source collaborative implicit representation is an implicit representation of traffic events for the debugging learning example; Generate a traffic event recognition error according to the first traffic event recognition result, the second traffic event recognition result, the third traffic event recognition result and the traffic event label, adjust the parameters according to the traffic event recognition error, jump to the step of obtaining the debugging learning sample to perform iterative debugging, stop iterating when the convergence state is reached, and obtain a debugged event implicit representation neural network for traffic event recognition; Acquire target radar-visual fusion data of a target road, wherein the target radar-visual fusion data includes road binocular vision data and road millimeter-wave radar data of the target road; Obtaining a target traffic event recognition result based on the event implicit representation neural network recognition completed by the debugging; Based on the target traffic event recognition result of the target road, the event is updated and displayed in a preset virtual reality twin.
2. The method according to claim 1, characterized in that The first branch component includes a pre-adjusted binocular vision implicit representation mining component and a binocular vision implicit representation encoding component; the first branch component based on the event implicit representation neural network performs implicit representation encoding on the initial road binocular vision data implicit representation of the road binocular vision data to obtain the encoded binocular vision data implicit representation, including: Based on the pre-adjusted binocular vision implicit representation mining component, data sampling is performed on the road binocular vision data to obtain a road binocular vision image set, and the initial road binocular vision data implicit representation of each road binocular vision image in the road binocular vision image set is extracted to obtain an initial road binocular vision data implicit representation set; Based on the binocular vision implicit representation encoding component, the initial road binocular vision data implicit representation set is implicitly aggregated to obtain an aggregated implicit representation, and the aggregated implicit representation is implicitly encoded to obtain an encoded binocular vision data implicit representation; The second branch component includes a pre-adjusted radar data implicit representation mining component and a radar data implicit representation encoding component; the second branch component based on the event implicit representation neural network performs implicit representation encoding on the initial radar data implicit representation of the road millimeter wave radar data to obtain the encoded radar data implicit representation, including: Extracting the initial radar data implicit representation of the road millimeter wave radar data based on the pre-adjusted radar data implicit representation mining component; Based on the radar data implicit representation encoding component, implicitly represent and encode the initial radar data implicit representation to obtain an encoded radar data implicit representation; The third branch component includes a feature random masking module and an implicit representation collaborative mining module; the third branch component based on the event implicit representation neural network performs feature random masking on the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data to obtain a multi-source fusion implicit representation, and performs collaborative implicit representation mining on the multi-source fusion implicit representation to obtain a multi-source collaborative implicit representation, including: Based on the feature random masking module, arbitrarily retain one or all of the implicit representation of the encoded binocular vision data and the implicit representation of the encoded radar data; Based on the implicit representation collaborative mining module, the retained implicit representation is determined as a multi-source fusion implicit representation, and collaborative implicit representation mining is performed on the multi-source fusion implicit representation to obtain a multi-source collaborative implicit representation of the debugging learning example.
3. The method according to claim 2, characterized in that The third branch component further includes a noise adding module; the method further includes: Based on the first denoising module between the binocular vision implicit representation encoding component and the feature random masking module, denoising the coded binocular vision data implicit representation to obtain the coded binocular vision data implicit representation after denoising; Based on a second denoising module between the radar data implicit representation encoding component and the feature random masking module, the coded radar data implicit representation is denoised to obtain a noisy coded radar data implicit representation.
4. The method according to claim 1, characterized in that: The method of generating a traffic event recognition error according to the first traffic event recognition result, the second traffic event recognition result, the third traffic event recognition result and the traffic event label, and adjusting parameters according to the traffic event recognition error includes: combining the first traffic event recognition result and the traffic event label to generate a first recognition error; combining the second traffic event recognition result and the traffic event label to generate a second recognition error; combining the third traffic event recognition result and the traffic event label to generate a third recognition error; The first recognition error, the second recognition error and the third recognition error are integrated to obtain the traffic event recognition error.
5. The method according to any one of claims 1 to 4, characterized in that: The target traffic event recognition result is obtained based on the event implicit representation neural network recognition completed by the debugging, including: Determine the road binocular vision data having the same quality as the target radar vision fusion data in the high-frequency event radar vision fusion database; Based on the debugged event implicit representation neural network, corresponding multi-source collaborative implicit representations are respectively extracted for the road binocular vision data with homogeneity, and the multi-source collaborative implicit representations are implicit representations of traffic events of the road binocular vision data; Determining the implicit representation of traffic events of the target radar-vision fusion data according to the implicit representation of traffic events of the binocular vision data of each road with homogeneous conditions; According to the implicit representation of traffic events of the target radar-visual fusion data and the implicit representation library of traffic events, the road binocular vision data having a commonality threshold with the target radar-visual fusion data is determined in the traffic event fusion database; the implicit representation of traffic events in the implicit representation library of traffic events is obtained by extracting corresponding multi-source collaborative implicit representations from binocular vision data of each road in the traffic event fusion database based on the debugged event implicit representation neural network; The traffic events corresponding to the road binocular vision data greater than the commonality threshold are used as the traffic events of the target radar-vision fusion data.
6. The method according to claim 5, characterized in that The determining of the road binocular vision data having the same quality as the target radar vision fusion data in the high-frequency event radar vision fusion database comprises: Based on the pre-adjusted binocular vision implicit representation mining component, data sampling is performed on the target radar vision fusion data to obtain a road binocular vision image set, and the road binocular vision data implicit representation of each road binocular vision image in the road binocular vision image set is extracted. For each road binocular vision data implicit representation, the corresponding common road binocular vision data implicit representation is determined in the road binocular vision data implicit representation library of the high-frequency event radar vision fusion database, and the road binocular vision data corresponding to the common road binocular vision data implicit representation is determined as the road binocular vision data having the same condition as the target radar vision fusion data; The method further comprises: Obtain traffic incident fusion database; Based on the debugged event implicit representation neural network, respectively extract the multi-source collaborative implicit representation of the binocular vision data of each road in the traffic event fusion database, and the multi-source collaborative implicit representation is the implicit representation of the traffic event of the binocular vision data of the road; A traffic event implicit representation library is generated according to the traffic event implicit representation of the binocular vision data of each road in the traffic event fusion database.
7. The method according to claim 5, characterized in that The method of determining the road binocular vision data having a commonality threshold with the target radar-visual fusion data in the traffic event fusion database based on the traffic event implicit representation of the target radar-visual fusion data and the traffic event implicit representation library includes: Based on the implicit representation of the traffic event of the target radar-visual fusion data, the implicit representation of the traffic event is determined in the implicit representation library of the traffic event to obtain a set of implicit representations of the traffic event; Binning the implicit representation set of traffic events to obtain a plurality of feature bins and a bin centroid of each feature bin, wherein the bin centroid is an average value of the implicit representation of traffic events in the feature bin; In each of the plurality of feature bins, an implicit representation of a traffic event that is most similar to the centroid of the corresponding bin is determined, and the road binocular vision data corresponding to the determined plurality of implicit representations of traffic events is determined as the road binocular vision data having a commonality greater than a threshold with the target radar vision fusion data.
8. The method according to claim 7, characterized in that The method further comprises: Acquire target road binocular vision data determined from road binocular vision data having a commonality greater than a threshold value with the target radar vision fusion data; Obtaining a target feature bin where an implicit representation of a traffic event in the binocular vision data of the target road is located; Determine the average value of the implicit representation of traffic events in the road binocular vision data having the same condition as the target radar-visual fusion data and the implicit representation of traffic events in the target feature bin as the implicit representation of traffic events updated by the target radar-visual fusion data; According to the updated implicit representation of traffic events and the implicit representation library of traffic events, road binocular vision data having a commonality threshold greater than that of the target radar-vision fusion data is determined from a traffic event fusion database.
9. The method according to claim 5, characterized in that The step of extracting corresponding multi-source collaborative implicit representation of road binocular vision data by the debugged event implicit representation neural network includes: Based on the first branch component of the debugged event implicit representation neural network, implicitly encoding the initial road binocular vision data implicit representation of the road binocular vision data to obtain an encoded binocular vision data implicit representation; Based on the debugged event implicit representation neural network's second branch component, implicitly represent and encode the initial radar data implicit representation of the road millimeter-wave radar data matched with the road binocular vision data to obtain the encoded radar data implicit representation; Based on the third branch component of the debugged event implicit representation neural network, a multi-source fusion implicit representation is obtained according to the encoded binocular vision data implicit representation and the encoded radar data implicit representation, and collaborative implicit representation mining is performed on the multi-source fusion implicit representation to obtain a multi-source collaborative implicit representation; the multi-source collaborative implicit representation is an implicit representation of traffic events in the road binocular vision data.
10. A computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the program, the steps in the method according to any one of claims 1 to 9 are implemented.
