Dynamic path identification method and device based on multi-sensor fusion, and ETC system

Abstract

The invention relates to a dynamic path identification method and device based on multi-sensor fusion and an ETC system, and the method comprises the steps: carrying out the data collection through multiple sensors, and obtaining multi-mode original data; processing the multi-modal original data to obtain multi-modal input data; performing quality evaluation and calculation to obtain a quality evaluation value and a quality evaluation result of each modal; based on the quality evaluation value and the quality evaluation result of each modal, carrying out dynamic weight distribution on the multi-modal input data by adopting a dynamic weight distribution algorithm to obtain weighted multi-source observation data; and performing fusion and prediction according to the multi-source observation data, the spatial-temporal characteristic data and the environmental parameters to obtain a target motion track of the target vehicle. Through a multi-sensor dynamic fusion mechanism, the weight of each sensor is adjusted, and the problems of poor environmental sensitivity, insufficient spatial resolution, data processing bottleneck and multi-source data missing are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and more specifically, to a dynamic path recognition method and system based on multi-sensor fusion. Background Art

[0002] Existing ETC systems mainly rely on RFID technology, which has the following defects:

[0003] Poor environmental sensitivity: In rainy and foggy weather, the 2.4GHz signal attenuates by 6.8dB / km, and the positioning error exceeds 1.5 meters, failing to meet the standard requirements.

[0004] Insufficient spatial resolution: Due to the design limitation of the minimum recognition distance of 2.4 meters, the misjudgment rate is relatively high in the following vehicle scenario.

[0005] Data processing bottleneck: The traditional solution based on the ARM Cortex-A53 processor has a delay of 82ms, which cannot meet the real-time requirements.

[0006] Lack of multi-source data: The performance of a single sensor drops sharply under extreme conditions. For example, the false alarm rate of a millimeter-wave radar exceeds 15%, and a visible light camera fails completely in rainy and foggy weather. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a dynamic path recognition method, device and ETC system based on multi-sensor fusion in view of the problems existing in the prior art.

[0008] The technical solution adopted by the present invention to solve its technical problems is: to construct a dynamic path recognition method based on multi-sensor fusion, including the following steps:

[0009] Collect data through multi-sensors to obtain multi-modal raw data;

[0010] Process the multi-modal raw data to obtain multi-modal input data;

[0011] Adopt a quality assessment model to conduct quality assessment on the multi-modal to obtain the quality assessment results of each modality in the multi-modal;

[0012] Based on the quality assessment results of each modality, adopt a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data;

[0013] Construct spatio-temporal feature data according to the multi-source observation data, and perform fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

[0014] In the dynamic path recognition method based on multi-sensor fusion according to the present invention, the multi-modal raw data includes: raw point cloud data, visual image data, and temperature data;

[0015] The data acquisition through multi-sensors to obtain multi-modal raw data includes:

[0016] Target detection is performed by a millimeter-wave radar to obtain the raw point cloud data;

[0017] Image target detection and positioning are performed by a vision module to obtain the visual image data;

[0018] Infrared detection is performed by an infrared sensor to obtain the temperature data.

[0019] In the dynamic path recognition method based on multi-sensor fusion according to the present invention, the multi-modal input data includes: structured target list data, target detection result set data, and corrected temperature matrix data;

[0020] The processing of the multi-modal raw data to obtain multi-modal input data includes:

[0021] The raw point cloud data is processed using a point cloud algorithm to obtain structured target list data;

[0022] The visual image data is processed based on the YOLOv6 detection model to obtain target detection result set data;

[0023] The temperature data is processed using a thermal radiation correction algorithm to obtain corrected temperature matrix data.

[0024] In the dynamic path recognition method based on multi-sensor fusion according to the present invention, based on the quality assessment results of each modality, a dynamic weight allocation algorithm is used to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data, including:

[0025] Determine the weight adjustment scheme for each modality according to the quality assessment results;

[0026] Perform dynamic weight allocation on the multi-modal input data according to the weight adjustment scheme of each modality to obtain the weighted multi-source observation data.

[0027] In the dynamic path recognition method based on multi-sensor fusion according to the present invention, constructing spatio-temporal feature data according to the multi-source observation data, and performing fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle, including:

[0028] Integrate based on the multi-source observation data in combination with spatio-temporal dimension data, environmental data, trajectory feature data, and system status data to obtain the spatio-temporal feature data;

[0029] According to the spatio-temporal feature data and the environmental parameters, use a preset path prediction model to perform prediction to obtain the initial motion trajectory of the target vehicle;

[0030] Perform filtering and fusion on the initial motion trajectory of the target vehicle through the Kalman filter algorithm to obtain the target motion trajectory of the target vehicle.

[0031] The present invention also provides a dynamic path recognition device based on multi-sensor fusion, including:

[0032] A data acquisition unit, which is used to collect data through multi-sensors to obtain multi-modal raw data;

[0033] A roadside intelligent base station, which is connected to the data acquisition unit and is used to collect and transmit the multi-modal raw data;

[0034] An edge computing unit, which is connected to the roadside intelligent base station unit and is used to perform the following actions:

[0035] Process the multi-modal raw data to obtain multi-modal input data;

[0036] Use a quality assessment model to perform quality assessment and calculation on the multi-modal to obtain the quality assessment results of each modality in the multi-modal;

[0037] Based on the quality assessment results of each modality, use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data;

[0038] A cloud platform, which is connected to the edge computing unit and is used to construct spatio-temporal feature data according to the multi-source observation data, and perform fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

[0039] In the dynamic path recognition device based on multi-sensor fusion of the present invention, the multi-modal raw data includes: raw point cloud data, visual image data, and temperature data;

[0040] The data acquisition unit includes: a millimeter radar group, a visual module, and an infrared sensor group;

[0041] The millimeter radar group is used to perform target detection to obtain the raw point cloud data;

[0042] The visual module is used for image target detection and positioning to obtain the visual image data;

[0043] The infrared sensor group is used for infrared detection to obtain the temperature data.

[0044] In the dynamic path recognition device based on multi-sensor fusion according to the present invention, the edge computing unit includes: an FPGA computing module and an ARM auxiliary control module;

[0045] The FPGA computing module is used to process and perform quality assessment calculations on the input data of each modality in a parallel computing manner, and perform dynamic weight allocation on the multi-modal input data by using a dynamic weight allocation algorithm to obtain weighted multi-source observation data;

[0046] The ARM auxiliary control module is used to load an algorithm configuration file into the FPGA computing module and monitor the computing status of the FPGA computing module.

[0047] In the dynamic path recognition device based on multi-sensor fusion according to the present invention, the cloud platform includes: a trajectory database, a path prediction module, and a toll decision engine module;

[0048] The trajectory database is used to store the multi-source observation data, spatio-temporal feature data, and the environmental parameters;

[0049] The path prediction module is connected to the trajectory database and is used to perform fusion and prediction based on the multi-source observation data, the spatio-temporal feature data, and the environmental parameters to obtain the target motion trajectory of the target vehicle;

[0050] The toll decision engine module is connected to the path prediction module and is used to generate a toll decision based on the target motion trajectory.

[0051] The present invention also provides an ETC system, including: the dynamic path recognition device based on multi-sensor fusion described above.

[0052] Implementing the dynamic path recognition method based on multi-sensor fusion of the present invention has the following beneficial effects: The dynamic path recognition method includes the following steps: collecting data through multi-sensors to obtain multi-modal raw data; processing the multi-modal raw data to obtain multi-modal input data; performing quality evaluation and calculation on the multi-modal input data to obtain the quality evaluation values and quality evaluation results of each modality; based on the quality evaluation values and quality evaluation results of each modality, using a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data; performing fusion and prediction based on the multi-source observation data, spatio-temporal feature data, and environmental parameters to obtain the target motion trajectory of the target vehicle. Through the multi-sensor dynamic fusion mechanism of the present invention, the adjustment of the weights of each sensor is realized, effectively solving the problems of poor environmental sensitivity, insufficient spatial resolution, data processing bottleneck, and multi-source data loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The following will further illustrate the present invention in conjunction with the drawings and embodiments. In the drawings:

[0054] Figure 1 is a schematic flowchart of a preferred embodiment of the dynamic path recognition method based on multi-sensor fusion provided by the present invention;

[0055] Figure 2 is a logical block diagram of the dynamic path recognition device based on multi-sensor fusion provided by the present invention;

[0056] Figure 3 is an embedded heterogeneous computing architecture diagram provided by the present invention;

[0057] Figure 4 is a configuration diagram of the edge computing unit provided by the present invention;

[0058] Figure 5 is a structural diagram of the ARM control domain and the FPGA computing domain provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0060] To solve the problems existing in the prior art, the present invention proposes a dynamic path recognition method and system based on multi-sensor integration through a multi-sensor dynamic weight integration algorithm and a heterogeneous computing architecture, which can be applied to the ETC system and can effectively solve the problems of poor environmental sensitivity, insufficient spatial resolution, data processing bottleneck, and multi-source data loss existing in the existing solutions.

[0061] Referring to Figure 1 , the present invention provides a dynamic path recognition method based on multi-sensor fusion, and the method includes the following steps:

[0062] Step S101: Collect data through multi-sensors to obtain multi-modal raw data.

[0063] Specifically, in some embodiments, the multi-sensors include but are not limited to millimeter-wave radars, vision sensors, and infrared sensors. The multi-modal raw data includes but is not limited to raw point cloud data, visual image data, and temperature data. Among them, collecting data through multi-sensors to obtain multi-modal raw data includes: detecting targets through millimeter-wave radars to obtain raw point cloud data; detecting and positioning image targets through vision sensors to obtain visual image data; performing infrared detection through infrared sensors to obtain temperature data.

[0064] By using multiple sensors to collect multi-source data and input it into the system, the system can receive the raw point cloud data from millimeter-wave radars, the visual image data from vision sensors (such as camera modules), and the temperature data from infrared sensors, forming multi-source heterogeneous data input.

[0065] Step S102: Process the multi-modal raw data to obtain multi-modal input data.

[0066] In some embodiments, the multi-modal input data includes: structured target list data, target detection result set data, and calibrated temperature matrix data; processing the multi-modal raw data to obtain multi-modal input data includes: processing the raw point cloud data using a point cloud algorithm to obtain structured target list data; processing the visual image data based on the YOLOv6 detection model to obtain target detection result set data; processing the temperature data using a thermal radiation correction algorithm to obtain calibrated temperature matrix data. Specifically, for the raw point cloud data input through the millimeter radar channel, the visual image data input through the visual channel, and the temperature data input through the infrared channel, the data of these three channels can be processed in parallel using a parallel feature processing method. Among them, the raw point cloud data input through the millimeter radar channel generates data in the form of a structured target list after spatially aggregating the radar reflection points through point cloud clustering; the visual image data input through the visual channel performs image target detection and positioning using YOLOv6 to obtain data in the form of a target detection result set; the temperature data input through the infrared channel is processed to generate a calibrated temperature matrix, which can be used to perform radiation correction to eliminate environmental temperature interference. The processed data of each channel is shown in Table 1.

[0067] Table 1. Data of Each Channel

[0068]

[0069] Step S103: Use a quality assessment model to assess the quality of the multi-modal data and obtain the quality assessment results of each modality in the multi-modal data.

[0070] Specifically, the data output after processing of the three channels (i.e., the millimeter radar channel, the visual sensor channel, and the infrared sensor channel, each channel corresponding to a modality) is uniformly subjected to data quality assessment to obtain the quality assessment results of each modality in the multi-modal data, so as to quantify the confidence of each sensor group by calculating the quality assessment values of each sensor group.

[0071] Optionally, in the embodiments of the present invention, the quality assessment model is as follows:

[0072] Q = 0.4×SNR + 0.3×Lux + 0.3×Vis (1).

[0073] In formula (1), Q is the quality assessment result (i.e., the quality assessment value), SNR is the signal-to-noise ratio (dB), Lus is the ambient illuminance (lx), and Vis is the visibility (km). Among them, for the weight decay factor: when the ambient illuminance < 50 lux, the visual weight is automatically reduced by 30%.

[0074] The Q values of each sensor can be calculated through the quality assessment model of formula (1). Specifically, as follows:

[0075] Q radar = 0.4 × SNR radar + 0.3 × Lux + 0.3 × Vis(2).

[0076] Q vision = 0.4 × SNR vision + 0.3 × Lux + 0.3 × Vis(3).

[0077] Q ir = 0.4 × SNR ir + 0.3 × Lux + 0.3 × Vis(4).

[0078] (In equation (2), Q radar is the quality evaluation value of the millimeter-wave radar, and SNR radar is the signal-to-noise ratio of the millimeter-wave radar; in equation (3), Q vision is the quality evaluation value of the vision sensor, and SNR vision is the signal-to-noise ratio of the vision sensor; in equation (4), Q ir is the quality evaluation value of the infrared sensor, and SNR ir is the signal-to-noise ratio of the infrared sensor.)

[0079] Step S104: Based on the quality evaluation results of each modality, use the dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain the weighted multi-source observation data.

[0080] Optionally, in some embodiments, based on the quality evaluation results of each modality, using the dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain the weighted multi-source observation data includes: determining the weight adjustment scheme for each modality according to the quality evaluation results; performing dynamic weight allocation on the multi-modal input data according to the weight adjustment scheme for each modality to obtain the weighted multi-source observation data.

[0081] Specifically, after calculating the quality evaluation values of each modality in step S103, use the dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to achieve dynamic adjustment of the weights of each modality. That is, first perform weight adjustment, and then perform fusion (i.e., normalization calculation) to obtain the final weights of each modality. For example, through equations (2), (3), and (4), it is calculated respectively: Q radar = 0.8, Q vision = 0.9, Q ir = 0.7, and the current environment is sunny noon (high brightness). If the environment is too dark (<50 lux): # For example, inside a tunnel; then the camera score = the original Q value × 0.7 # 30% off. If entering the tunnel and the brightness changes to 30 lux, then Q visionThe value is adjusted from 0.9 to 0.63, that is, 0.9×0.7 = 0.63.

[0082] After the weight adjustment, perform normalization calculation:

[0083] Convert the Q value to percentage weight:

[0084] The total score (Q 总 ) = Q radar +Q vision +Q ir (5).

[0085] The weight of the millimeter-wave radar (denoted by α) is:

[0086] α = Q radar / total score (6).

[0087] The weight of the vision sensor (denoted by β) is:

[0088] β = Q vision / total score (7).

[0089] The weight of the infrared sensor (denoted by γ) is:

[0090] γ = Q ir / total score (8).

[0091] Through equation (5), the total score can be calculated as: Q 总 = 0.8 + 0.63 + 0.7 = 2.13;

[0092] By calculating through equations (6), (7), and (8) respectively, we can obtain: α = 0.375 (37.5%), β = 0.296 (29.6%), γ = 0.329 (32.9%), and this is the weight adjustment scheme. The final effect is: on sunny days, the weight of the camera (vision sensor) is the highest (clear image); in dark environments, the weights of the radar and infrared are automatically increased (compensating for the performance degradation of the camera); in foggy weather, the weight of the radar is automatically increased (better penetration).

[0093] After calculating the weights of each sensor, the multi-modal input data can be weighted based on the calculated weights to obtain the weighted multi-source observation data. Optionally, in the embodiments of the present invention, the Q-Learning algorithm can be used to optimize the sensor weights.

[0094] It should be noted that α (weight of millimeter-wave radar), its physical meaning: represents the reliability weight of millimeter-wave radar signals. Dynamic range: 0.2 - 0.7. Adjustment logic: When the signal-to-noise ratio (S / N) is high (such as clear target, less interference), increase the value of α (close to 0.7) to enhance the contribution of the millimeter-wave radar; when the signal-to-noise ratio is low (such as rainy, snowy weather, significant multipath effect), decrease the value of α (close to 0.2) to weaken its influence. β (weight of visual sensor), its physical meaning: represents the reliability weight of the visual sensor (such as a camera). Dynamic range: 0.1 - 0.6. Adjustment logic: When the light intensity is high (such as during the day, good lighting), increase the value of β (close to 0.6) and rely on the visual perception result; when the light is low (such as at night, in a tunnel), decrease the value of β (close to 0.1) to avoid misjudgment. γ (weight of infrared sensor), its physical meaning: represents the reliability weight of the infrared sensor. Dynamic range: 0.1 - 0.3. Adjustment logic: When the visibility is low (such as in a haze, smoky environment), increase the value of γ (close to 0.3) and rely on the infrared penetration performance. When the visibility is high (such as on a clear day), decrease the value of γ (close to 0.1).

[0095] Step S105: Construct spatio-temporal feature data based on multi-source observation data, and perform fusion and prediction based on the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

[0096] Optionally, in some embodiments, constructing spatio-temporal feature data based on multi-source observation data, and performing fusion and prediction based on the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle includes: integrating based on multi-source observation data in combination with spatio-temporal dimension data, environmental data, trajectory feature data, and system state data to obtain spatio-temporal feature data; according to the spatio-temporal feature data and environmental parameters, using a preset path prediction model to perform prediction to obtain the initial motion trajectory of the target vehicle; filtering and fusing the initial motion trajectory of the target vehicle through the Kalman filter algorithm to obtain the target motion trajectory of the target vehicle.

[0097] Specifically, the spatio-temporal feature data can be obtained by integrating multi-source observation data, spatio-temporal dimension data (time dimension data and space dimension data), environmental data, trajectory feature data, and system state data, etc. Specifically as shown in Table 2. Environmental parameters include but are not limited to: light intensity, visibility, temperature, humidity, precipitation intensity, wind speed, road surface state data, etc., specifically as shown in Table 3.

[0098] Table 2. Spatio-temporal feature data

[0099]

[0100]

[0101] Table 3. Environmental parameters

[0102]

[0103]

[0104] Optionally, in the embodiments of the present invention, the preset path prediction model may be an LSTM prediction model. The spatio-temporal feature data and environmental parameters are fused through the LSTM prediction model for prediction to obtain the initial motion trajectory of the target vehicle, and then filtered and fused through the Kalman filter algorithm to obtain the target motion trajectory of the target vehicle. The present invention performs spatio-temporal fusion on the weighted multi-source observation data based on the Kalman filter, and finally outputs a smooth and reliable motion trajectory of the target vehicle. The entire process embodies the technical features of sensor redundancy design, data complementary fusion, and dynamic reliability assessment.

[0105] Reference Figure 2 In order to implement the above dynamic path recognition method based on multi-sensor fusion, the present invention provides a dynamic path recognition device based on multi-sensor fusion.

[0106] Specifically, as Figure 2 shown, the dynamic path recognition device based on multi-sensor fusion includes:

[0107] A data acquisition unit 100, which is used to acquire data through multi-sensors to obtain multi-modal raw data.

[0108] Optionally, in some embodiments, the data acquisition unit 100 includes: a millimeter radar group 101, a vision module 102, and an infrared sensor group 103. Among them, the millimeter radar group 101 is used for target detection to obtain raw point cloud data; the vision module 102 is used for image target detection and positioning to obtain visual image data; the infrared sensor group 103 is used for infrared detection to obtain temperature data. Among them, the millimeter radar group 101 may include multiple millimeter wave radars, the vision module 102 may include multiple cameras (i.e., multiple vision sensors), and the infrared sensor group 103 may include multiple infrared sensors.

[0109] The data acquisition unit 100 of the present invention consists of a multi-modal sensing unit composed of multiple millimeter-wave radars, multiple cameras, and multiple infrared sensors. Among them, the millimeter-wave radar can adopt the TI AWR2944 series (77 GHz, maximum detection range 300 m), which can detect targets within a range of 300 m through the FMCW waveform and output 3D point clouds of 2048 points / frame. The vision module 102 can adopt a 4K vision module 102, such as the Sony IMX585 series (4K HDR, 120 dB dynamic range), which can run the lightweight YOLOv6 model to achieve vehicle feature recognition at 100 fps. The infrared sensor can adopt the FLIR ADK series (resolution 640×512, temperature measurement range -40~550 °C), and the infrared sensor can provide thermal radiation compensation with an accuracy of ±1 °C to enhance the perception in rainy and foggy environments.

[0110] The roadside intelligent base station 200 is connected to the data acquisition unit 100 and is used to collect and transmit multi-modal raw data. By setting up this roadside intelligent base station 200, the acquisition and transmission of data can be realized.

[0111] The edge computing unit 300 is connected to the roadside intelligent base station 200 unit and is used to perform the following actions: process the multi-modal raw data to obtain multi-modal input data; use a quality assessment model to perform quality assessment and calculation on the multi-modal to obtain the quality assessment results of each modality in the multi-modal; based on the quality assessment results of each modality, use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data.

[0112] In some embodiments, the edge computing unit 300 includes: an FPGA computing module 301 and an ARM auxiliary control module 302. Among them, the FPGA computing module 301 is used to process and perform quality assessment calculations on the input data of each modality in a parallel computing manner, and use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data; the ARM auxiliary control module 302 is used to load the algorithm configuration file into the FPGA computing module 301 and monitor the computing status of the FPGA computing module 301.

[0113] In the embodiment of the present invention, the edge computing unit 300 is based on an ARM+FPGA heterogeneous architecture, and its embedded heterogeneous computing architecture is as Figure 3 shown, and its configuration diagram is as Figure 4 shown. Specifically, the ARM auxiliary control module 302 includes an ARM processor, and the operating mechanism of its ARM processor is as follows: run the RT-Linux 5.10 real-time operating system; sensor data preprocessing pipeline; hardware resource monitoring and dynamic scheduling. The specific indicators and action responses are shown in Table 4.

[0114] Table 4

[0115] Monitoring metrics Response actions CPU load > 90% Trigger FPGA task offloading Memory latency > 200 ns Enable 2MB large page memory allocation

[0116] The real-time scheduling system runs through ARM Cortex-A72, executes the dynamic weight decision algorithm (updated in a 10ms cycle), deploys the hardware acceleration core using FPGA, and realizes radar signal processing (clustering speed of 50Mpts / s) and LSTM inference acceleration.

[0117] As Figure 5 shown, in the embodiment of the present invention, the FPGA computing module 301 processes the data of the millimeter wave channel, the vision channel, and the infrared channel in parallel. Among them, the point cloud clustering algorithm of the millimeter wave channel is hardware-accelerated by FPGA; the YOLOv6 detection model of the vision channel deploys the quantized neural network through FPGA. The spatio-temporal attention LSTM network adopted has 4.8M parameters, the model size is 48MB, and the neural network architecture adopts the search (NAS) compression technology. The thermal radiation correction algorithm of the infrared channel is realized through the FPGA pipeline. The ARM auxiliary control module 302 is mainly responsible for loading the algorithm configuration file (such as YOLO weights, clustering threshold parameters) into the FPGA; monitoring the FPGA computing status (through the AXI-Lite bus).

[0118] The cloud platform 400 is connected to the edge computing unit 300, and is used to construct spatio-temporal feature data based on multi-source observation data, and perform fusion and prediction based on the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

[0119] Optionally, in some embodiments, the cloud platform 400 includes: a trajectory database 401, a path prediction module 402, and a toll decision engine module 403. Among them, the trajectory database 401 is used to store multi-source observation data, spatio-temporal feature data, and environmental parameters; the path prediction module 402 is connected to the trajectory database 401 and is used to perform fusion and prediction based on multi-source observation data, spatio-temporal feature data, and environmental parameters to obtain the target motion trajectory of the target vehicle; the toll decision engine module 403 is connected to the path prediction module 402 and is used to generate a toll decision based on the target motion trajectory.

[0120] Specifically, the trajectory database 401 is used to store spatio-temporal feature data, which supports SQL spatio-temporal queries and 30-day data backtracking. The path prediction module 402 supports the output of 5-second trajectory predictions (with a 10Hz update frequency). By dynamically outputting dynamic path data through the path prediction module 402, it can be used as the basis for accurate road network settlement splitting, promoting the industry's transformation from "shortest path billing" to "actual path billing". The charging decision engine module 403 can achieve real-time charging decisions with an accuracy of 99.3% based on lane topology matching.

[0121] Specifically, the specific cooperation operation process among the units in the dynamic path recognition device based on multi-sensor fusion here can specifically refer to the above-mentioned dynamic path recognition method based on multi-sensor fusion, which will not be elaborated here.

[0122] Furthermore, the present invention also provides an ETC system, which may include the dynamic path recognition device based on multi-sensor fusion disclosed in the embodiments of the present invention.

[0123] Compared with the traditional ETC system, the ETC system of the present invention has the following advantages:

[0124] Higher positioning accuracy, with a positioning accuracy of up to ±0.3m, while the traditional solution can only reach ±1.5m;

[0125] Higher lane-changing recognition accuracy, with a lane-changing recognition accuracy that can reach 99.2%, while the lane-changing recognition accuracy of the traditional solution can only reach 87.3%;

[0126] The system response delay is significantly improved, with the system response delay being only 23ms or even lower, while the system response delay of the traditional solution is as high as 82ms;

[0127] Higher reliability in extreme temperatures, with a reliability in extreme temperatures that can reach -40 to 85°C, while the traditional solution is only -20 to 70°C.

[0128] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0129] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0130] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of both. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0131] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and cannot limit the protection scope of the present invention. All equivalent changes and modifications made to the scope of the claims of the present invention shall fall within the scope covered by the claims of the present invention.

Claims

1. A dynamic path recognition method based on multi-sensor fusion, characterized in that, It includes the following steps: Collect data through multiple sensors to obtain multi-modal raw data; Process the multi-modal raw data to obtain multi-modal input data; Use a quality assessment model to perform quality assessment on the multi-modal data to obtain the quality assessment results of each modality in the multi-modal data; Based on the quality assessment results of each modality, use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data; Construct spatio-temporal feature data according to the multi-source observation data, and perform fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

2. The dynamic path recognition method based on multi-sensor fusion according to claim 1, wherein The multi-modal raw data includes: raw point cloud data, visual image data, and temperature data; The step of collecting data through multiple sensors to obtain multi-modal raw data includes: Detect targets through a millimeter-wave radar to obtain the raw point cloud data; Perform image target detection and positioning through a vision module to obtain the visual image data; Perform infrared detection through an infrared sensor to obtain the temperature data.

3. The dynamic path recognition method based on multi-sensor fusion according to claim 2, characterized in that, The multi-modal input data includes: structured target list data, target detection result set data, and corrected temperature matrix data; The step of processing the multi-modal raw data to obtain multi-modal input data includes: Use a point cloud algorithm to process the raw point cloud data to obtain structured target list data; Based on the YOLOv6 detection model, process the visual image data to obtain target detection result set data; Use a thermal radiation correction algorithm to process the temperature data to obtain a corrected temperature matrix data.

4. The dynamic path recognition method based on multi-sensor fusion according to claim 1, wherein, The step of, based on the quality assessment results of each modality, using a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data includes: Determine the weight adjustment scheme for each modality according to the quality assessment results; Perform dynamic weight allocation on the multi-modal input data according to the weight adjustment scheme of each modality to obtain the weighted multi-source observation data.

5. The dynamic path recognition method based on multi-sensor fusion according to claim 1, characterized in that The step of constructing spatio-temporal feature data according to the multi-source observation data, and performing fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle includes: Integrate based on the multi-source observation data in combination with spatio-temporal dimension data, environmental data, trajectory feature data, and system state data to obtain the spatio-temporal feature data; According to the spatio-temporal feature data and the environmental parameters, use a preset path prediction model to perform prediction to obtain the initial motion trajectory of the target vehicle; Filter and fuse the initial motion trajectory of the target vehicle through a Kalman filter algorithm to obtain the target motion trajectory of the target vehicle.

6. A dynamic path recognition device based on multi-sensor fusion, characterized in that, It includes: A data collection unit, which is used to collect data through multiple sensors to obtain multi-modal raw data; A roadside intelligent base station, which is connected to the data collection unit and is used to collect and transmit the multi-modal raw data; An edge computing unit, which is connected to the roadside intelligent base station unit and is used to perform the following actions: Process the multi-modal raw data to obtain multi-modal input data; Use a quality assessment model to perform quality assessment and calculation on the multi-modal data to obtain the quality assessment results of each modality in the multi-modal data; Based on the quality assessment results of each modality, use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data; A cloud platform, which is connected to the edge computing unit, is used to construct spatio-temporal feature data according to the multi-source observation data, and perform fusion and prediction according to the spatio-temporal feature data and environmental parameters to obtain the target motion trajectory of the target vehicle.

7. The dynamic path recognition device based on multi-sensor fusion according to claim 6, wherein The multi-modal raw data includes: raw point cloud data, visual image data, and temperature data; The data acquisition unit includes: a millimeter radar group, a vision module, and an infrared sensor group; The millimeter radar group is used to perform target detection to obtain the raw point cloud data; The vision module is used to perform image target detection and positioning to obtain the visual image data; The infrared sensor group is used to perform infrared detection to obtain the temperature data.

8. The dynamic path recognition device based on multi-sensor fusion according to claim 6, characterized in that, The edge computing unit includes: an FPGA computing module and an ARM auxiliary control module; The FPGA computing module is used to process and perform quality assessment calculations on the input data of each modality in a parallel computing manner, and use a dynamic weight allocation algorithm to perform dynamic weight allocation on the multi-modal input data to obtain weighted multi-source observation data; The ARM auxiliary control module is used to load an algorithm configuration file into the FPGA computing module and monitor the computing status of the FPGA computing module.

9. The dynamic path recognition device based on multi-sensor fusion according to claim 6, characterized in that, The cloud platform includes: a trajectory database, a path prediction module, and a toll decision engine module; The trajectory database is used to store the multi-source observation data, spatio-temporal feature data, and the environmental parameters; The path prediction module is connected to the trajectory database and is used to perform fusion and prediction based on the multi-source observation data, the spatio-temporal feature data, and the environmental parameters to obtain the target motion trajectory of the target vehicle; The toll decision engine module is connected to the path prediction module and is used to generate a toll decision based on the target motion trajectory.

10. An ETC system, characterized in that, Includes: The dynamic path recognition device based on multi-sensor fusion according to any one of claims 6-9.

Images