Virtual-real linkage highway transaction special situation disposal method and device

Abstract

The invention discloses a virtual-real linkage highway transaction special situation disposal method and device, and relates to the technical field of three-dimensional modeling, and the device comprises a front-end collection module which is used for collecting the real-time state of a toll station dynamic element; the three-dimensional modeling module is used for creating a virtual model of a real toll station physical system by using a digital twinning technology, creating a model with various attributes of dynamic and static elements of a toll station, introducing an AI analysis and prediction function, performing early warning on events and providing disposal scheme suggestions; and the twin rendering module is used for realizing scene rendering closely associated with the transaction special situation disposal business, so that remote special situation disposal personnel can handle the disposal business in an immersive manner. Through the AI algorithm and the three-dimensional modeling technology, the generated disposal scheme has high pertinence and accuracy, the special situation processing efficiency can be remarkably improved, the manual intervention time is shortened, and the overall passing efficiency and safety of the toll station are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a method and device for handling special situations in highway transactions with virtual-real linkage. Background Art

[0002] With the development of intelligent transportation systems, the intelligent management of highway toll stations has gradually become a hot topic of research. In the existing technology, highway toll stations mainly rely on manual monitoring and processing of transaction anomalies (special situations), which is inefficient and easily leads to traffic congestion, affecting the efficiency of vehicle traffic. At the same time, with the increase in the number of vehicles and the increase in traffic complexity, the traditional manual processing method is difficult to meet the actual needs, and a method that can improve processing efficiency and accuracy is urgently needed.

[0003] In practical applications, the real-time collection and processing of dynamic elements at toll stations is a key link. The traditional single-mode data collection method is difficult to fully reflect the actual situation of toll stations, and it is easy to cause incomplete or inaccurate data. In the existing technology, single technical means such as video surveillance and infrared sensing are usually used for data collection, which cannot achieve the fusion and comprehensive analysis of multi-modal data, limiting the comprehensive perception of the toll station environment and vehicle status.

[0004] In addition, existing 3D modeling technologies are rarely used in toll station applications, making it difficult to achieve virtual simulation and real-time rendering of the physical system of toll stations. Even if some 3D modeling technologies are introduced, they are often limited to static display and lack real-time linkage with the dynamic elements of actual toll stations, making it impossible to achieve real-time prediction and solution push for special situation handling. Summary of the invention

[0005] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0006] In view of the problems existing in the above-mentioned existing method and device for handling special situations in highway transactions with virtual and real linkage, the present invention is proposed.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] A virtual-real linkage highway transaction special situation handling device, the device comprising:

[0009] The front-end acquisition module is used to collect the real-time status of dynamic elements of toll stations;

[0010] 3D modeling module: The 3D modeling module uses digital twin technology to create a virtual model of the physical system of the real toll station, creates a model with various attributes of the dynamic and static elements of the toll station, and introduces AI analysis and prediction functions to warn of events and provide disposal plan suggestions;

[0011] Twin Rendering Module: The Twin Rendering Module is used to realize scene rendering closely related to the special situation handling business of transactions, so that remote special situation handling personnel can handle the handling business in an immersive way, including real-time rendering of perspective, real-time rendering of vehicles, and rendering of interactive video stream access;

[0012] Virtual-reality linkage module: By matching the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system, when operating each element on the twin model, the operation content will be directly projected onto the real scene, controlling the equipment in the real scene, interacting with the real scene, and completing the handling of special transaction situations.

[0013] As a preferred solution of the virtual-real linkage highway transaction special situation handling device of the present invention, wherein: the front-end acquisition module includes: a panoramic acquisition submodule, a side position acquisition submodule, a front position acquisition submodule, and a transaction information acquisition submodule;

[0014] The panoramic acquisition submodule is used to acquire dynamic and static elements from the perspective of the entire lane;

[0015] The side position acquisition submodule is used to collect the side elements of the vehicle, focusing on the vehicle model and the number of axles, and the interaction status between the driver and the remote platform;

[0016] The front position acquisition module is used to collect the front-end features of the vehicle, focusing on the license plate information and the status information of the driver and passengers in the vehicle; the transaction information acquisition submodule is used to collect the transaction status of the vehicle.

[0017] A method for handling special situations in highway transactions with virtual and real linkage, based on the above-mentioned special situation handling device for highway transactions with virtual and real linkage, the method comprises the following steps:

[0018] S1: The front-end acquisition module is started, and the panoramic acquisition submodule, the side acquisition submodule, the front acquisition submodule and the transaction information acquisition submodule are used to collect the real-time status of the dynamic elements of the toll station;

[0019] S2: The multi-modal collected data is fused through a data fusion algorithm to form a unified data set, using image processing, infrared imaging, LiDAR data, and data obtained from LiDAR, fusion technology to achieve comprehensive perception of the vehicle and environment;

[0020] S3: The 3D modeling module uses the fused data to create a virtual model of the physical system of the real toll station and construct a 3D model with various attributes of the dynamic and static elements of the toll station;

[0021] S4: Introducing real-time AI analysis and prediction functions in the 3D modeling module, and performing real-time analysis on the fused data and 3D models through machine learning algorithms;

[0022] S5: The twin rendering module is started to realize scene rendering closely related to the special transaction situation handling business;

[0023] S6: The virtual-reality linkage module matches the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system one by one.

[0024] As a preferred solution of the method for handling special situations of highway transactions with virtual-real linkage described in the present invention, in which: In the S2 process, the specific process of fusion processing is:

[0025] Define a multimodal dataset, let D = {D v , D r , D ι}, D v Denotes the video image dataset, D r represents the infrared dataset, D ι Represents the LiDAR dataset;

[0026] Normalize the data of different modes to make their data range consistent. is the data of the ith mode, and the normalized data is expressed as

[0027] The normalized multimodal data are fused using the fusion algorithm F(·) to form a unified data set F, then:

[0028]

[0029] Among them, α, β are fusion weight coefficients, λ is the attenuation coefficient, x is the data point, and N is the total number of data.

[0030] As a preferred solution of the method and device for handling special situations of highway transactions in virtual-real linkage of the present invention, in which: in the process of S4, input data is obtained from the fused data set F, and feature extraction is performed on the input data to generate a feature vector X i , annotate historical data, identify and record various special events, including: vehicle anomalies, transaction problems, and use these annotations as label data Y label , used for model training and prediction;

[0031] X i =[X v , X c , X r , X f , X t ];

[0032] Then the dataset {X i , Y label} Divide into training set and test set, with 80% of the data as training set and 20% of the data as test set;

[0033] A method combining random forest and deep neural network is selected to train the model, and the prediction results of random forest and neural network are integrated to generate the final prediction results.

[0034] As a preferred solution of the method for handling special situations of highway transactions with virtual and real linkage described in the present invention, the prediction results of random forest and neural network are integrated to generate the final prediction result: Y final =θ·Y RF +(1-θ)·Y NN

[0035] Among them, θ is the weight coefficient, Y RF is the prediction result of random forest, Y NN is the prediction result of the neural network.

[0036] As a preferred solution of the virtual-real linkage highway transaction special situation handling method of the present invention, wherein: judging whether a special situation occurs according to the prediction result of the model, and giving an early warning;

[0037] The probability of a special situation is:

[0038] Among them, P alert The value range of is [0, 1], λ is the attenuation coefficient, and T is the warning time window;

[0039] Generate corresponding disposal plan suggestions based on the warning results and push them to remote special situation disposal personnel;

[0040] The treatment plan formula is:

[0041] Among them, w i is the weight of the treatment plan, and N is the number of possible special situations.

[0042] As a preferred solution of the method for handling special situations of highway transactions in virtual and real linkage described in the present invention, the value range of the handling solution is S solution is a positive number, and the weights of different special situations are w iThe weight of serious accidents is higher, while that of minor transaction problems is lower. According to different comprehensive score values, corresponding solutions are given:

[0043]

[0044] The above mapping ensures that the comprehensive score can correspond to a specific treatment plan.

[0045] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, the steps of the above-mentioned method for handling special situations in highway transactions with virtual and real linkage are implemented.

[0046] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method and device for handling special situations in highway transactions in a virtual-real linkage.

[0047] Beneficial effects of the present invention:

[0048] 1. The present invention can achieve comprehensive perception of the toll station environment and vehicle status and improve the integrity and accuracy of data by introducing multimodal data collection and fusion technology. It uses image processing, infrared imaging, LiDAR data fusion and other technologies to form a unified data set and provide more comprehensive real-time monitoring and analysis capabilities;

[0049] 2. The present invention introduces real-time AI analysis and prediction functions in the 3D modeling module, and uses machine learning algorithms to perform real-time analysis on the fused data and 3D models. Based on historical data and current status, it predicts possible special situations (such as vehicle anomalies, transaction problems, etc.), issues early warnings, and generates disposal proposals, which are pushed to remote special situation disposal personnel;

[0050] 3. The present invention uses digital twin technology to create a virtual model of the physical system of a real toll station, and realizes the three-dimensional modeling of the dynamic and static elements of the toll station. The twin rendering module realizes the scene rendering closely related to the transaction special situation handling business, allowing remote special situation handling personnel to handle the handling business in an immersive way, and realizes real-time control and interaction of the actual toll station equipment through the virtual-real linkage module;

[0051] 4. The disposal plan generated by the present invention through AI algorithm and three-dimensional modeling technology is highly targeted and accurate, which can significantly improve the efficiency of special situation handling, reduce the time of manual intervention, and improve the overall traffic efficiency and safety of toll stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0053] Figure 1 This is a schematic diagram of the overall structure of a virtual-real linkage highway transaction special situation handling device proposed by the present invention;

[0054] Figure 2 This is a schematic diagram of the effect of the fusion of prediction models in the virtual-real linkage highway transaction special situation handling method proposed by the present invention;. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0058] Reference Figure 1-Figure 2 , is an embodiment of the present invention, and provides a method and device for handling special situations of highway transactions with virtual and real linkage, the device comprising:

[0059] The front-end acquisition module is used to collect the real-time status of dynamic elements of toll stations;

[0060] Specifically, the front-end acquisition module includes: a panoramic acquisition submodule, a side acquisition submodule, a front acquisition submodule, and a transaction information acquisition submodule;

[0061] The panoramic acquisition submodule is used to acquire dynamic and static elements from the perspective of the entire lane;

[0062] The side position acquisition submodule is used to collect the side elements of the vehicle, focusing on the vehicle model and the number of axles, and the interaction status between the driver and the remote platform;

[0063] The front position acquisition module is used to collect the front-end features of the vehicle, focusing on the license plate information and the status information of the driver and passengers in the vehicle; the transaction information acquisition submodule is used to collect the transaction status of the vehicle.

[0064] 3D modeling module: The 3D modeling module uses digital twin technology to create a virtual model of the physical system of the real toll station, creates a model with various attributes of the dynamic and static elements of the toll station, and introduces AI analysis and prediction functions to warn of events and provide disposal plan suggestions;

[0065] Twin Rendering Module: The Twin Rendering Module is used to realize scene rendering closely related to the special situation handling business of transactions, so that remote special situation handling personnel can handle the handling business in an immersive way, including real-time rendering of perspective, real-time rendering of vehicles, and rendering of interactive video stream access;

[0066] Virtual-reality linkage module: By matching the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system, when operating each element on the twin model, the operation content will be directly projected onto the real scene, controlling the equipment in the real scene, interacting with the real scene, and completing the handling of special transaction situations.

[0067] The virtual-real linkage highway transaction special situation handling method applied to the above-mentioned virtual-real linkage highway transaction special situation handling device comprises the following steps:

[0068] S1: The front-end acquisition module is started, and the panoramic acquisition submodule, the side acquisition submodule, the front acquisition submodule and the transaction information acquisition submodule are used to collect the real-time status of the dynamic elements of the toll station;

[0069] S2: The multi-modal collected data is fused through a data fusion algorithm to form a unified data set, and the image processing, infrared imaging, and LiDAR data [data obtained from laser radar] fusion technology are used to achieve comprehensive perception of the vehicle and the environment;

[0070] The specific process of fusion processing is as follows:

[0071] Define a multimodal dataset, let D = {D v , D r , D ι}, D v Denotes the video image dataset, D r represents the infrared dataset, D ι Represents the LiDAR dataset;

[0072] Normalize the data of different modes to make their data range consistent. is the data of the ith mode, and the normalized data is expressed as

[0073] The normalized multimodal data are fused using the fusion algorithm F(·) to form a unified data set F, then:

[0074]

[0075] Among them, α, β are fusion weight coefficients, λ is the attenuation coefficient, x is the data point, and N is the total number of data.

[0076] S3: The 3D modeling module uses the fused data to create a virtual model of the physical system of the real toll station and construct a 3D model with various attributes of the dynamic and static elements of the toll station;

[0077] Specifically, it includes: data analysis, extracting various types of information from the fused data set F, including vehicle location, lane status, vehicle side and front features, and transaction information, cleaning and filtering the extracted data, removing noise data, and ensuring data accuracy and integrity

[0078] Initialize the 3D modeling environment: Use 3D modeling software (such as Unity, Unreal Engine, etc.) to initialize the 3D modeling environment and set up a basic virtual scene.

[0079] Create the basic framework of the virtual toll station, including static elements such as lanes, toll booths, signs and markings.

[0080] Adding dynamic elements: Add dynamic elements such as vehicles to the 3D scene.

[0081] That is, it includes: Vehicle model: According to the side and front feature data, select the corresponding vehicle model from the vehicle model library and scale it according to the actual size. Vehicle position and status: According to the vehicle position data, place the vehicle model at the corresponding position in the virtual scene, and set its motion trajectory and speed according to the lane status data. Static element addition: Add the static elements of the toll station to the three-dimensional scene. Toll booth model: According to the layout and design of the actual toll station, select or customize the toll booth model from the model library. Signs and markings: According to the lane status data, set the signs and markings in the virtual scene, including lane lines, stop lines, traffic signs, etc.;

[0082] Attribute binding: bind various attributes in the fused dataset to the corresponding parts of the 3D model, and data mapping: map the dynamic data in the fused dataset to the virtual model to achieve real-time update and interaction:

[0083] Final scene rendering: Use a high-quality rendering engine to render the three-dimensional scene to ensure the authenticity and visual effects of the scene.

[0084] S4: Introducing real-time AI analysis and prediction functions in the 3D modeling module, and performing real-time analysis on the fused data and 3D models through machine learning algorithms;

[0085] Furthermore, we obtain input data from the fused dataset F, extract features from the input data, and generate a feature vector X i , annotate historical data, identify and record various special events, including: vehicle anomalies, transaction problems, and use these annotations as label data Y label , used for model training and prediction;

[0086] X i =[X v , X c , X r , X f , X t ];

[0087] Then the dataset {X i , Y label} Divide into training set and test set, with 80% of the data as training set and 20% of the data as test set;

[0088] A method combining random forest and deep neural network is selected to train the model, and the prediction results of random forest and neural network are integrated to generate the final prediction results.

[0089] Model training process:

[0090] Random forest model training, setting the parameters of the random forest, including the number of trees n trees and the maximum depth d max , train the random forest model, using the training data {X i , Y label}Train a random forest model.

[0091] Then: RF = RandomForestClassifier (n trees , d max );

[0092] RF.fit(X train , Y train );

[0093] Deep neural network training, neural network structure design: Design the structure of deep neural network, including input layer, hidden layer and output layer.

[0094] Input layer: feature vector X i

[0095] Hidden layer: multiple fully connected layers, each layer contains a certain number of neurons, and the activation function is ReLU

[0096] Output layer: prediction result Y NN , and the activation function uses Softmax;

[0097] Network structure diagram: Y NN =Softmax(W out ·ReLU(W2·ReLU(W1·X+b1)+b2)+b out )

[0098] Model parameter initialization: Initialize the parameters W1, W2, W of the neural network out , b1, b2, b out ;

[0099] Training the neural network model: Use the training data to train the neural network model and optimize the loss function (such as cross entropy loss): Loss = -∑ i Y label log(Y i );

[0100] Update the parameters using the gradient descent method:

[0101] The prediction results of random forest and neural network are combined to generate the final prediction result: Y final =θ·Y RF +(1-θ)·Y NN

[0102] Among them, θ is the weight coefficient, Y RF is the prediction result of random forest, Y NN is the prediction result of the neural network.

[0103] ginseng Figure 2 It can be seen that the prediction value of the fusion model (black dotted line) is closer to the actual value (red solid line) in most areas. Compared with the prediction results of random forest (blue dashed line) and neural network (green dotted line), the fusion model significantly reduces the error. This is because the fusion model combines the advantages of two different models and can better capture the complex patterns and changes in the data, while a single model may sometimes be sensitive to specific samples or noise data, resulting in unstable prediction results. By fusing the prediction results of random forest and neural network, the fluctuations of a single model can be smoothed, making the prediction results more stable and reliable.

[0104] What needs to be specially explained is that according to the prediction results of the model, it is judged whether a special situation occurs and an early warning is issued;

[0105] The probability of a special situation is:

[0106] Among them, P alert The value range of is [0, 1], λ is the attenuation coefficient, and T is the warning time window;

[0107] Generate corresponding disposal plan suggestions based on the warning results and push them to remote special situation disposal personnel;

[0108] The treatment plan formula is:

[0109] Among them, w i is the weight of the treatment plan, N is the number of possible special situations;

[0110] :The treatment solution value range S solution is a positive number, and the weights of different special situations are w i The weight of serious accidents is higher, while that of minor transaction problems is lower. According to different comprehensive score values, corresponding solutions are given:

[0111]

[0112] The above mapping ensures that the comprehensive score can correspond to a specific treatment plan.

[0113] S5: The twin rendering module is started to realize scene rendering closely related to the special transaction situation handling business;

[0114] S6: The virtual-reality linkage module matches the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system one by one.

[0115] In summary, the present invention can achieve comprehensive perception of the toll station environment and vehicle status and improve the integrity and accuracy of the data by introducing multimodal data collection and fusion technology. Use image processing, infrared imaging, LiDAR data fusion and other technologies to form a unified data set, providing more comprehensive real-time monitoring and analysis capabilities; introduce real-time AI analysis and prediction functions in the three-dimensional modeling module, and use machine learning algorithms to perform real-time analysis on the fused data and three-dimensional models. Based on historical data and current status, predict possible special situations (such as vehicle anomalies, transaction problems, etc.), give early warnings, and generate disposal plan suggestions, which are pushed to remote special situation disposal personnel.

[0116] The present invention uses digital twin technology to create a virtual model of the physical system of a real toll station, and realizes three-dimensional modeling of dynamic and static elements of the toll station. The twin rendering module realizes scene rendering closely related to the special transaction handling business, allowing remote special situation handling personnel to handle the handling business in an immersive way, and realize real-time control and interaction of actual toll station equipment through the virtual-real linkage module. Through AI algorithms and three-dimensional modeling technology, the generated handling plan is highly targeted and accurate, which can significantly improve the efficiency of special situation handling, reduce manual intervention time, and improve the overall traffic efficiency and safety of toll stations.

[0117] This embodiment also provides a computer device, which is suitable for a method and device for handling special situations in highway transactions with virtual and real linkage, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement a method and device for handling special situations in highway transactions with virtual and real linkage as proposed in the above embodiment.

[0118] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0119] The present embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, a method and device for handling special situations in highway transactions with virtual and real linkage as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A virtual-real linkage highway transaction special situation handling device, characterized in that: The device includes: The front-end acquisition module is used to collect the real-time status of dynamic elements of toll stations; 3D modeling module: The 3D modeling module uses digital twin technology to create a virtual model of the physical system of the real toll station, creates a model with various attributes of the dynamic and static elements of the toll station, and introduces AI analysis and prediction functions to warn of events and provide disposal plan suggestions; Twin Rendering Module: The Twin Rendering Module is used to realize scene rendering closely related to the special situation handling business of transactions, so that remote special situation handling personnel can handle the handling business in an immersive way, including real-time rendering of perspective, real-time rendering of vehicles, and rendering of interactive video stream access; Virtual-reality linkage module: By matching the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system, when operating each element on the twin model, the operation content will be directly projected onto the real scene, controlling the equipment in the real scene, interacting with the real scene, and completing the handling of special transaction situations.

2. A virtual-real linkage highway transaction special situation handling device according to claim 1, characterized in that: The front-end acquisition module includes: a panoramic acquisition submodule, a side acquisition submodule, a front acquisition submodule, and a transaction information acquisition submodule; The panoramic acquisition submodule is used to acquire dynamic and static elements from the perspective of the entire lane; The side position acquisition submodule is used to collect the side elements of the vehicle, focusing on the vehicle model and the number of axles, and the interaction status between the driver and the remote platform; The front position acquisition module is used to collect the front-end features of the vehicle, focusing on the license plate information and the status information of the driver and passengers in the vehicle; the transaction information acquisition submodule is used to collect the transaction status of the vehicle.

3. A virtual-real linkage highway transaction special situation handling method, based on the virtual-real linkage highway transaction special situation handling device according to claim 2, characterized in that: The method comprises the following steps: S1: The front-end acquisition module is started, and the panoramic acquisition submodule, the side acquisition submodule, the front acquisition submodule and the transaction information acquisition submodule are used to collect the real-time status of the dynamic elements of the toll station; S2: The multi-modal collected data is fused through a data fusion algorithm to form a unified data set, and image processing, infrared imaging, and LiDAR data fusion technology are used to achieve comprehensive perception of the vehicle and the environment; S3: The 3D modeling module uses the fused data to create a virtual model of the physical system of the real toll station and construct a 3D model with various attributes of the dynamic and static elements of the toll station; S4: Introducing real-time AI analysis and prediction functions in the 3D modeling module, and performing real-time analysis on the fused data and 3D models through machine learning algorithms; S5: The twin rendering module is started to realize scene rendering closely related to the special transaction situation handling business; S6: The virtual-reality linkage module matches the data interfaces of the dynamic and static elements of the toll station mapped and rendered on the three-dimensional model with the toll collection system, electromechanical system, and monitoring system one by one.

4. A method and device for handling special situations in highway transactions with virtual and real linkage according to claim 3, characterized in that: In the S2 process, the specific process of fusion processing is as follows: Define a multimodal dataset, let D = {D v ,D r ,D ι }, D v Denotes the video image dataset, D r represents the infrared dataset, D ι Represents LiDAR dataset; Normalize the data of different modes to make their data range consistent. is the data of the ith mode, and the normalized data is expressed as The normalized multimodal data are fused using the fusion algorithm F(·) to form a unified data set F, then: Among them, α, β are fusion weight coefficients, λ is the attenuation coefficient, x is the data point, and N is the total number of data.

5. A method and device for handling special situations in highway transactions with virtual and real linkage according to claim 4, characterized in that: In the S4 process, input data is obtained from the fused data set F, and features are extracted from the input data to generate a feature vector X i , annotate historical data, identify and record various special events, including: vehicle anomalies, transaction problems, and use these annotations as label data Y label , used for model training and prediction; X i =[X v ,X c ,X r ,X f ,X t ]; Then the dataset {X i , Y label } Divide into training set and test set, with 80% of the data as training set and 20% of the data as test set; A method combining random forest and deep neural network is selected to train the model, and the prediction results of random forest and neural network are integrated to generate the final prediction results.

6. A method and device for handling special situations in highway transactions with virtual and real linkage according to claim 5, characterized in that: The prediction results of random forest and neural network are combined to generate the final prediction result: Y final =θ·Y RF +(1-θ)·Y NN Among them, θ is the weight coefficient, Y RF is the prediction result of random forest, Y NN is the prediction result of the neural network.

7. A method and device for handling special situations in highway transactions with virtual and real linkage according to claim 6, characterized in that: Based on the prediction results of the model, determine whether a special situation has occurred and issue an early warning; The probability of a special situation is: Among them, P alert The value range of is [0,1], λ is the attenuation coefficient, and T is the warning time window; Generate corresponding disposal plan suggestions based on the warning results and push them to remote special situation disposal personnel; The treatment plan formula is: Among them, w i is the weight of the treatment plan, and N is the number of possible special situations.

8. A method and device for handling special situations in highway transactions with virtual and real linkage according to claim 7, characterized in that: The treatment solution value range S solution is a positive number, and the weights of different special situations are w i The weight of serious accidents is higher, while that of minor transaction problems is lower. According to different comprehensive score values, corresponding solutions are given: The above mapping ensures that the comprehensive score can correspond to a specific treatment plan.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method and device for handling special situations in highway transactions in a virtual-real linkage manner as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method and device for handling special situations in highway transactions in a virtual-real linkage manner as described in any one of claims 1 to 7 are implemented.

Images