Intelligent induction control method and system based on multi-modal sensor fusion

Abstract

The invention relates to the technical field of intelligent sensing control, in particular to an intelligent sensing control method and system based on multi-modal sensor fusion. According to the method, vision, millimeter-wave radar, geomagnetic and other multi-mode sensors are deployed to collect traffic data, feature information is extracted after preprocessing, an adaptive weighted fusion algorithm is adopted for fusion, and an intelligent control decision is generated according to a fusion result. The method comprises the steps of signal lamp timing optimization based on fuzzy logic, traffic flow induction based on reinforcement learning and traffic event detection and response based on event rules, and finally an instruction is sent to traffic control equipment to adjust the flow. The advantages of various sensors are integrated, the limitation of a single sensor is overcome, traffic information is accurately and comprehensively obtained, intelligent decision is made on the basis, the traffic condition can be comprehensively and accurately sensed, traffic control is intelligently optimized, the self-adaptive adjusting capacity is high, the traffic operation efficiency can be improved, and the traffic safety can be enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent sensing control, and specifically to an intelligent sensing control method and system based on multi-modal sensor fusion. Background Art

[0002] With the acceleration of urbanization and the rapid growth of the motor vehicle ownership, traffic problems such as traffic congestion and traffic accidents have become increasingly prominent, posing higher requirements for intelligent transportation management systems. Traditional traffic management means often rely on a single type of sensor to obtain traffic information. For example, only rely on cameras to obtain image data, or only use radars to detect vehicle speed and distance and other information.

[0003] However, single sensors have many limitations. Taking cameras as an example, in bad weather conditions, the clarity of image acquisition will drop significantly, thereby affecting the accuracy of identifying traffic participants and judging traffic conditions. While radar sensors can better detect dynamic information such as the distance and speed of target objects, their ability to identify the specific types of target objects is weak, and it is difficult to accurately distinguish different types of traffic participants. In addition, although geomagnetic sensors can sense the passing and staying of vehicles, the traffic information they provide is relatively single and cannot comprehensively reflect the overall situation of the traffic scene.

[0004] In view of these deficiencies of single sensors, in order to achieve more accurate and comprehensive traffic information perception, and then improve the intelligent level and control efficiency of traffic management, a method and system that can fuse data of multiple types of sensors and make intelligent decisions based on the fused information are needed. Summary of the Invention

[0005] The present invention provides an intelligent sensing control method and system based on multi-modal sensor fusion. By integrating the advantages of multiple sensors, overcoming the limitations of single sensors, accurately and comprehensively obtaining traffic information, and making intelligent decisions based on this, it realizes the effective regulation of traffic flow, the timely response to traffic events, and the overall improvement of traffic operation efficiency.

[0006] The intelligent sensing control method based on multi-modal sensor fusion includes:

[0007] S1. Deployment of multi-modal sensors and data acquisition: Deploy multi-modal sensors in the intelligent traffic management area. The multi-modal sensors include visual sensors, millimeter-wave radar sensors, and geomagnetic sensors, and collect multi-modal information through the multi-modal sensors.

[0008] Preferably, the collection of multi-modal information through the multi-modal sensors specifically includes:

[0009] The vision sensor collects image data. The vision sensor includes a high-definition camera and an infrared camera. When there is sufficient daylight, the high-definition camera is used to collect the image data of the traffic scene. In the night or low-light environment, the infrared camera is switched to for image data collection.

[0010] The millimeter-wave radar sensor collects the distance and speed data of the target object. By transmitting millimeter waves and receiving the reflected waves, it accurately measures the distance between the vehicle and the radar, the driving speed of the vehicle, and the acceleration information.

[0011] The geomagnetic sensor is installed under the road and is used to collect the passing and staying data of the vehicle. When a vehicle is detected passing by, the passing time of the vehicle is recorded. When the vehicle stays above the sensor, the staying duration of the vehicle is recorded, and thus the passing frequency and staying situation of the vehicle are counted.

[0012] S2. Data preprocessing: Preprocess the collected multi-modal data to improve the quality and usability of the data.

[0013] Preferably, the preprocessing of the collected multi-modal data to improve the quality and usability of the data specifically includes:

[0014] Image data preprocessing:

[0015] Noise reduction processing: The median filtering algorithm is used to perform noise reduction on the image data to remove the interference in the image while retaining the edge information of the image, making the image clearer.

[0016] Sharpening processing: The Laplace operator is used to perform sharpening processing on the denoised image to enhance the edges and details of the image, highlight the contours of traffic participants, and facilitate subsequent feature extraction.

[0017] Radar data preprocessing:

[0018] Filtering processing: The Kalman filtering algorithm is used to filter the radar data, remove the noise interference, smooth the data curve, and improve the accuracy of distance and speed measurement.

[0019] Calibration processing: According to the installation position and parameters of the radar, the measured data is calibrated to eliminate the measurement deviation.

[0020] Geomagnetic sensor data preprocessing: The moving average smoothing processing method is used to smooth the collected geomagnetic sensor data, reduce the data fluctuation, and improve the data reliability.

[0021] S3. Feature information extraction: Extract representative feature information from the preprocessed data.

[0022] S4. Multimodal Feature Information Fusion: The feature information extracted by different sensors is fused using an adaptive weighted fusion algorithm.

[0023] Preferably, the fusion of the feature information extracted by different sensors using the adaptive weighted fusion algorithm specifically includes:

[0024] Data Normalization: The feature information extracted from different sensors is normalized using the min-max normalization method, mapping the feature information to the interval [0, 1] to eliminate the differences in dimension and numerical range of each feature information;

[0025] Sensor Accuracy Evaluation: The measurement errors of each sensor are monitored in real time, and the measurement accuracy of the sensor is evaluated by calculating the mean square error of the measurement errors;

[0026] Weight Dynamic Adjustment: According to the evaluated measurement accuracy of the sensors, the weights of the feature information of each sensor in the fusion process are dynamically assigned;

[0027] Feature Information Fusion: According to the adjusted weights and the normalized feature information of each sensor, the fused feature information is calculated;

[0028] Verification and Correction of Fusion Results: The fused feature information is verified to check whether it conforms to the actual traffic conditions. If obvious deviations are found in the fusion results, the possible reasons are analyzed and the fusion process is corrected;

[0029] The formula for calculating the fused feature information is:

[0030]

[0031] In the formula, m is the number of sensors, w i is the weight of the normalized feature information of the i-th sensor, and x i is the normalized feature information of the i-th sensor.

[0032] Preferably, the dynamic assignment of the weights of the feature information of each sensor in the fusion process according to the evaluated measurement accuracy of the sensors specifically includes:

[0033] Obtain the measurement values of each sensor in real time;

[0034] Calculate the mean square error between the measurement value and the true value of each sensor as the error value of each sensor;

[0035] Calculate the weights of each sensor according to the error values of each sensor;

[0036] The formula for calculating the weights of each sensor is:

[0037]

[0038] where w i is the weight of the i-th sensor, m is the number of sensors, and e i is the error value of the i-th sensor.

[0039] S5. Intelligent control decision generation: Generate an intelligent control decision based on the fused feature information. The intelligent control decision includes a dynamic traffic signal timing optimization scheme based on fuzzy logic control, a traffic flow guidance strategy based on reinforcement learning, and a traffic event automatic detection and response mechanism based on event rules.

[0040] Preferably, the dynamic traffic signal timing optimization scheme based on fuzzy logic control specifically includes:

[0041] Determine input variables: Obtain traffic flow, vehicle queue length, and average vehicle waiting time from the feature information as input variables for fuzzy logic control;

[0042] Define fuzzy sets and membership functions: Define fuzzy sets for each input variable. Define three fuzzy sets of "low", "medium", and "high" for traffic flow, three fuzzy sets of "short", "medium", and "long" for vehicle queue length, and three fuzzy sets of "short", "medium", and "long" for average vehicle waiting time; and determine membership functions for each fuzzy set;

[0043] Formulate fuzzy rules: Formulate fuzzy rules based on traffic management experience and traffic operation laws;

[0044] Fuzzy inference: Calculate the membership degrees of the current actual values of the input variables belonging to each fuzzy set through the membership functions, and then perform fuzzy inference based on the fuzzy rules to obtain a fuzzy output regarding the adjustment of signal timing;

[0045] Defuzzification: Use the centroid method for defuzzification to convert the fuzzy output into a specific adjustment value of the green light duration of the signal;

[0046] Signal timing adjustment: Dynamically adjust the current signal timing scheme according to the adjustment value obtained from defuzzification;

[0047] The formula for defuzzification using the centroid method is:

[0048]

[0049] where z is the adjustment value obtained after defuzzification, n is the number of fuzzy outputs, z i is the element of the i-th output, and μ(z i ) is the membership degree corresponding to the element z i .

[0050] Preferably, the traffic flow induction strategy based on reinforcement learning specifically includes:

[0051] Define the state space: The state space is composed of traffic information after multi-modal sensor fusion;

[0052] Define the action space: Define traffic flow induction actions, including recommended routes, estimated travel times, ramp control on or off, and lane restriction rule adjustments;

[0053] Design the reward function: Set the reward function, and the goal of the reward function is to reduce traffic congestion and vehicle travel time;

[0054] Policy selection: Use the DQN algorithm to learn the optimal traffic flow induction strategy. At each time step, the algorithm selects an action from the action space according to the current state to execute. The environment updates the state according to this action and feedbacks the corresponding reward;

[0055] Policy update and optimization: As time goes by and traffic conditions change, continuously update the parameters of the DQN network and optimize the traffic flow induction strategy;

[0056] Apply the induction strategy: When the strategy optimization is completed, apply the optimized traffic flow induction strategy to traffic management.

[0057] Preferably, the traffic event automatic detection and response mechanism based on event rules specifically includes:

[0058] Rule setting: Define traffic event rules, specifically:

[0059] Traffic accident detection rule: If multiple vehicles on a section suddenly drop to zero speed within a short time, and the vehicle positions are relatively fixed, and at the same time, it is recognized through image data that the vehicles have collision deformation, it is determined that a traffic accident has occurred;

[0060] Traffic congestion detection rule: If the average speed of vehicles on a certain section is lower than the set threshold, and the vehicle queue length exceeds a certain value and lasts for a set duration, it is determined that traffic congestion has occurred on that section;

[0061] Vehicle abnormal parking rule: In a non-parking area, if a vehicle stays for more than the preset time, it is determined that the vehicle is parked abnormally;

[0062] Event detection: Continuously compare the fused traffic information with the pre-set rules in real time to determine whether the rule conditions are met;

[0063] Response measure trigger: If the fused traffic information meets a certain rule condition, automatically trigger the corresponding response measure, specifically:

[0064] For traffic accidents, immediately send alarm information to the traffic management department, and at the same time publish accident information on the surrounding variable message signs to prompt drivers to take a detour; adjust the signal timing of the surrounding intersections to guide the orderly evacuation of vehicles;

[0065] For traffic congestion, publish information on congested sections and recommended detour routes through variable message signs; activate ramp control to limit the number of vehicles entering the congested section; adjust the signal timing of the surrounding areas to increase the green light duration in the relevant directions of the congested section;

[0066] For abnormal vehicle parking, send reminder information to nearby law enforcement officers, and at the same time display reminder information on the variable message signs near the abnormal parking point, asking the vehicle to leave as soon as possible.

[0067] S6. Control instruction sending: Send corresponding control instructions to traffic control devices to achieve the regulation of traffic flow.

[0068] An intelligent induction control system based on multi-modal sensor fusion includes:

[0069] A multi-modal sensor module, including a vision sensor, a millimeter wave radar sensor, and a geomagnetic sensor, is used to collect image data of the intelligent traffic management area, distance and speed data of target objects, and passing and staying data of vehicles;

[0070] A data preprocessing module, connected to the multi-modal sensor module, is used to preprocess the data collected by each sensor, including denoising and sharpening the image data, filtering and calibrating the radar data, and smoothing the geomagnetic sensor data;

[0071] A feature extraction module, connected to the data preprocessing module, is used to extract feature information from the preprocessed data;

[0072] A multi-modal data fusion module, connected to the feature extraction module, uses an adaptive weighted fusion algorithm to fuse the feature information extracted by different sensors;

[0073] An intelligent decision-making and control module, connected to the multi-modal data fusion module, generates intelligent control decisions based on the fused traffic information. The intelligent control decisions include a dynamic traffic signal timing optimization scheme based on fuzzy logic control, a traffic flow guidance strategy based on reinforcement learning, and a traffic event automatic detection and response mechanism based on event rules;

[0074] An instruction sending module, connected to the intelligent decision-making and control module, is used to send corresponding control instructions to traffic control devices to regulate traffic flow.

[0075] Compared with the prior art, the advantages of the present invention are:

[0076] Comprehensively and accurately perceive traffic conditions: By fusing multi-modal sensor data, the limitations of single sensors are overcome, enabling comprehensive and accurate acquisition of traffic information, including the types, quantities, motion states of traffic participants, as well as the passing and staying situations of vehicles, etc., providing a more reliable basis for traffic management decisions.

[0077] Intelligently optimize traffic control: The dynamic traffic signal timing optimization scheme based on fuzzy logic control can dynamically adjust signal timing according to real-time traffic conditions, improving the traffic efficiency at intersections; the traffic flow guidance strategy based on reinforcement learning can guide vehicles to reasonably choose driving routes and balance traffic flow; the traffic event automatic detection and response mechanism based on event rules can timely detect and handle abnormal situations such as traffic accidents and traffic jams, reducing the impact on traffic.

[0078] Adaptive adjustment ability: The adaptive weighted fusion algorithm can dynamically adjust weights according to the measurement accuracy of each sensor under different environmental conditions, ensuring the accuracy and reliability of the fused traffic information. At the same time, intelligent control decisions can continuously optimize and adjust according to real-time traffic conditions, with strong adaptive ability. Specific implementation methods

[0079] Embodiment

[0080] In one embodiment, an implementation method and system of intelligent induction control based on multi-modal sensor fusion in the intelligent traffic management of intersections are provided, including:

[0081] Deployment of multi-modal sensor modules:

[0082] Vision sensors: Install high-definition cameras and infrared cameras in four directions at intersections. The high-definition cameras are used to collect clear traffic scene image data during the day, and the infrared cameras automatically switch to work at night or in low-light environments to ensure accurate image information is obtained all day long.

[0083] Millimeter-wave radar sensors: Install millimeter-wave radar sensors above each lane to real-time monitor data such as the distance, speed, and acceleration of vehicles, and can accurately capture the movement trajectories of vehicles.

[0084] Geomagnetic sensors: Bury geomagnetic sensors under the lane ground to detect the passing and staying situations of vehicles, and count the passing frequency and staying duration of vehicles.

[0085] Deployment of other modules:

[0086] Data preprocessing module: Set up a server in the control room near the intersection, install data preprocessing software, and perform real-time preprocessing on the data collected by each sensor.

[0087] Feature extraction module, multi-modal data fusion module, intelligent decision-making and control module, and instruction sending module: All are integrated in the server in the control room and implement corresponding functions through software programming.

[0088] Data acquisition:

[0089] During the day: The high-definition camera is turned on to continuously collect traffic scene image data at the intersection, including information such as the types, quantities, and movement directions of vehicles, pedestrians, and non-motor vehicles. The millimeter-wave radar sensor real-time monitors the distance, speed, and acceleration of vehicles, and the geomagnetic sensor detects the passing and staying of vehicles.

[0090] At night or in low-light periods: The system automatically switches to the infrared camera for operation to ensure that the quality of image data collection is not affected. Other sensors remain in normal working condition and continue to collect relevant data.

[0091] Data preprocessing:

[0092] Image data: Use the median filtering algorithm to denoise the collected images and remove noise interference in the images; use the Laplacian operator to sharpen the images, enhance the edge information of the images, and improve the clarity of the images.

[0093] Radar data: Filter the data collected by the millimeter-wave radar sensor to remove noise and interference signals; at the same time, perform calibration processing to ensure the accuracy of the measurement data.

[0094] Geomagnetic sensor data: Smooth the collected geomagnetic sensor data to eliminate fluctuations and outliers in the data and make the data more stable and reliable.

[0095] Feature information extraction:

[0096] Feature extraction of image data: Use computer vision algorithms to extract feature information such as the types, quantities, and movement directions of traffic participants from the preprocessed image data. For example, identify the types and quantities of vehicles, pedestrians, and non-motor vehicles through object detection algorithms, and determine their movement directions through motion tracking algorithms.

[0097] Feature extraction of radar data: Extract feature information such as the real-time speed, acceleration, and movement trajectory of vehicles from the preprocessed radar data. Calculate the speed and acceleration of vehicles based on the measured distance and time changes of the radar; draw the movement trajectory of vehicles through continuous measurement data.

[0098] Feature extraction of geomagnetic sensor data: Extract feature information such as the passing frequency and staying duration of vehicles from the preprocessed geomagnetic sensor data. Count the number of vehicles passing through in a unit time to obtain the passing frequency of the vehicles; record the time when the vehicles enter and leave the detection area and calculate the staying duration of the vehicles.

[0099] Multi-modal feature information fusion:

[0100] Data normalization: Perform min-max normalization on the feature information extracted from different sensors. For example, for the feature of vehicle quantity, find its minimum value x min and maximum value x max , and convert each vehicle quantity value to Map all feature information to the interval [0, 1].

[0101] Sensor accuracy evaluation: Monitor the measurement errors of each sensor in real time. For the vision sensor, calculate its recognition error by comparing with a known standard scene; for the millimeter-wave radar sensor, regularly measure target objects with known distances and speeds and calculate its measurement error; for the geomagnetic sensor, count the accuracy of vehicle passing and staying times and evaluate its error. Use the mean square error as the error evaluation index.

[0102] Dynamic weight adjustment: Dynamically allocate the weights of the feature information of each sensor in the fusion process according to the results of sensor accuracy evaluation. Assume there are three sensors (vision sensor, millimeter-wave radar sensor, and geomagnetic sensor), and the weight of the i-th sensor is w i , and calculate the weight through the formula , where m is the number of sensors, and e i is the error value of the i-th sensor.

[0103] Feature information fusion: Perform weighted summation on the normalized feature information of each sensor according to the adjusted weights to obtain the fused feature information. Let the normalized feature information of the i-th sensor be x i , and the fused feature information

[0104] Verification and correction of fusion results: Compare the fused feature information with historical data and manual observation results to check whether it conforms to the actual traffic situation. If obvious deviations are found in the fusion results, analyze possible reasons such as sensor failures and unreasonable weight allocation, and correct the fusion process.

[0105] Intelligent control decision generation:

[0106] Dynamic traffic signal timing optimization scheme based on fuzzy logic control:

[0107] Determine input variables: Obtain traffic flow, vehicle queue length, and average vehicle waiting time from the feature information as the input variables for fuzzy logic control. The traffic flow is obtained by counting the number of vehicles passing through a certain lane at an intersection per unit time through sensor data; the vehicle queue length is measured by the length of the queue formed by the vehicles waiting to pass in front of the stop line at the intersection; the average vehicle waiting time is calculated by recording the time experienced by the vehicle from entering the queue area to starting to pass and calculating the average value.

[0108] Define fuzzy sets and membership functions: Define multiple fuzzy sets for each input variable respectively. Define three fuzzy sets of "low", "medium", and "high" for traffic flow, three fuzzy sets of "short", "medium", and "long" for vehicle queue length, and three fuzzy sets of "short", "medium", and "long" for average vehicle waiting time. Determine triangular membership functions for each fuzzy set to describe the degree to which the input variable belongs to each fuzzy set.

[0109] Formulate fuzzy rules: Formulate a series of fuzzy rules based on traffic management experience and traffic operation laws. For example:

[0110] If the traffic flow is high, the vehicle queue length is long, and the average vehicle waiting time is long, then the green light duration of the signal light should be increased.

[0111] If the traffic flow is low, the vehicle queue length is short, and the average vehicle waiting time is short, then the green light duration of the signal light should be decreased.

[0112] Fuzzy inference: According to the actual values of the current input variables, calculate the membership degrees of their belonging to each fuzzy set through the membership function, and then perform fuzzy inference based on the fuzzy rules to obtain a fuzzy output regarding the adjustment of signal light timing.

[0113] Defuzzification: Use the centroid method to convert the fuzzy output into a specific adjustment value for the green light duration of the signal light. The calculation method is where z is the adjustment value obtained after defuzzification, n is the number of fuzzy outputs, z i is the i-th element of the output, and μ(z i ) is the membership degree corresponding to the element z i .

[0114] Adjustment of signal light timing: Dynamically adjust the current signal light timing plan according to the adjustment value obtained from defuzzification. For example, if the calculated adjustment value is to increase the green light duration by 10 seconds, then increase the green light duration of the signal light in the corresponding direction by 10 seconds.

[0115] Traffic flow guidance strategy based on reinforcement learning:

[0116] Define the state space: The state space is composed of traffic information after multi-modal sensor fusion, including traffic flow, average vehicle speed, vehicle density on each road section, and congestion status at intersections, etc. Each state is a combination of these traffic information, used to describe the specific situation of the current traffic network.

[0117] Define the action space: The action space contains a series of traffic flow guidance actions that can be selected, such as publishing different guidance information (recommended routes, estimated travel times, etc.) on variable message signs, turning on or off ramp control, adjusting lane restriction rules, etc.

[0118] Design the reward function: The reward function is designed with traffic congestion level and vehicle travel time as the core elements. When a certain action is taken, if the traffic congestion level decreases (such as the average vehicle speed on the road section increases, the vehicle queue length shortens), and the overall vehicle travel time decreases, a positive reward is given; conversely, if the traffic condition deteriorates, a negative reward is given. At the same time, considering the implementation cost of the guidance action, such as the display cost of the variable message sign, the equipment operation cost of ramp control, etc., the reward function is appropriately adjusted.

[0119] Adopt the Deep Q-Network (DQN) algorithm: Use the DQN algorithm to learn the optimal traffic flow guidance strategy. At each time step, the algorithm selects an action from the action space according to the current state and executes it. The environment updates the state according to this action and feedbacks the corresponding reward. The algorithm continuously interacts with the environment, collects state-action-reward sequences, and uses these data to train the DQN network to maximize the long-term cumulative reward.

[0120] Policy update and optimization: As time goes by and traffic conditions change, continuously update the parameters of the DQN network and optimize the traffic flow guidance strategy. At the same time, adopt the experience replay mechanism, store the historical state-action-reward sequences in the experience replay pool, and randomly sample from it for learning during training to improve the stability and convergence speed of the algorithm.

[0121] Apply the guidance strategy: After the strategy optimization is completed, apply the optimized traffic flow guidance strategy to traffic management.

[0122] Traffic event automatic detection and response mechanism based on event rules:

[0123] Preset traffic event rules in advance:

[0124] Traffic accident detection rule: When the speeds of multiple vehicles drop to zero within a short time and their positions are fixed, and at the same time the visual sensor detects that the vehicles are deformed due to collision, it is determined that a traffic accident has occurred.

[0125] Traffic congestion detection rule: When the average speed of vehicles on a certain road section is lower than a certain threshold (e.g., 10 km / h), and the vehicle queue length exceeds a certain value (e.g., 200 meters), and it lasts for a certain duration (e.g., 5 minutes), it is determined that traffic congestion has occurred on this road section.

[0126] Abnormal vehicle parking rule: When a vehicle stays in a non-parking area for more than a preset time (e.g., 3 minutes), it is determined that the vehicle has parked abnormally.

[0127] Real-time monitoring and detection: The system compares the fused traffic information with the preset traffic event rules in real time, and uses multi-modal data for detailed analysis and judgment. For example, combining the image information of the vision sensor and the speed information of the millimeter-wave radar sensor to accurately determine whether a traffic accident has occurred.

[0128] Response measure triggering: Once the fused traffic information meets the rule conditions, the system automatically triggers the corresponding response measures.

[0129] If a traffic accident is detected, the system immediately reports to the traffic management department and publishes the accident information on the variable message sign to prompt vehicles to detour; at the same time, adjusts the signal timing of the surrounding areas to guide vehicles to evacuate in an orderly manner.

[0130] If traffic congestion is detected, the system publishes congestion information and recommended detour routes on the variable message sign, activates ramp control to limit the number of vehicles entering the congested road section; adjusts the signal timing to increase the green light duration in the congested direction and improve the vehicle passing efficiency.

[0131] If abnormal vehicle parking is detected, the system sends a reminder message to the law enforcement personnel and displays a reminder message on the nearby display screen, asking the vehicle to leave as soon as possible.

[0132] Effect evaluation and adjustment: After triggering the response measures, the system continuously monitors the traffic conditions and evaluates the effectiveness of the response measures. If the traffic conditions do not improve, the system will dynamically adjust the subsequent strategies, such as further optimizing the signal timing, updating the guidance information, etc.

[0133] Control instruction sending:

[0134] The instruction sending module sends the generated control instructions to the corresponding traffic control devices. Sends the signal timing adjustment instruction to the signal controller to achieve dynamic adjustment of signal timing; sends the traffic flow guidance information to the variable message sign to provide real-time traffic information and guidance suggestions for drivers; sends the ramp control instruction to the ramp control device to control the opening and closing of the ramp and regulate the vehicle flow entering the main road.

[0135] In the description of this specification, the descriptions referring to the terms "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. An intelligent sensing control method based on multimodal sensor fusion, characterized in that: include: S1. Multimodal sensor deployment and data collection: deploying a multimodal sensor in the intelligent traffic management area, wherein the multimodal sensor includes a visual sensor, a millimeter wave radar sensor and a geomagnetic sensor, and collecting multimodal information through the multimodal sensor; S2, data preprocessing: preprocess the collected multimodal data to improve the quality and availability of the data; S3, feature information extraction: extract representative feature information from the preprocessed data; S4, multi-modal feature information fusion: Adopt adaptive weighted fusion algorithm to fuse the feature information extracted by different sensors; S5, intelligent control decision generation: Generate intelligent control decisions based on the fused feature information, including dynamic traffic signal timing optimization scheme based on fuzzy logic control, traffic flow induction strategy based on reinforcement learning, and automatic detection and response mechanism of traffic events based on event rules; S6. Sending control instructions: Sending corresponding control instructions to the traffic control equipment to achieve regulation of traffic flow.

2. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The collecting multimodal information by the multimodal sensor specifically includes: The visual sensor collects image data. The visual sensor includes a high-definition camera and an infrared camera. When there is sufficient light during the day, the high-definition camera is used to collect image data of the traffic scene; at night or in a low-light environment, the infrared camera is switched to collect image data; The millimeter-wave radar sensor collects the distance and speed data of the target object. By emitting millimeter waves and receiving reflected waves, it accurately measures the distance between the vehicle and the radar, the vehicle's speed, and acceleration information. The geomagnetic sensor is installed under the road to collect vehicle passing and staying data. When a vehicle is detected passing, the vehicle's passing time is recorded; when the vehicle stays above the sensor, the vehicle's staying time is recorded, so as to count the vehicle's passing frequency and staying situation.

3. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The preprocessing of the collected multimodal data to improve the quality and usability of the data specifically includes: Image data preprocessing: Noise reduction processing: Use the median filter algorithm to reduce the noise of the image data to remove the interference in the image, while retaining the edge information of the image to make the image clearer; Sharpening: Use the Laplacian operator to sharpen the denoised image to enhance the edges and details of the image, highlight the contours of traffic participants, and facilitate subsequent feature extraction; Radar data preprocessing: Filtering: Use Kalman filtering algorithm to filter radar data, remove noise interference, smooth data curves, and improve the accuracy of distance and speed measurement; Calibration processing: calibrate the measurement data according to the installation position and parameters of the radar to eliminate measurement deviation; Geomagnetic sensor data preprocessing: The moving average smoothing method is used to smooth the collected geomagnetic sensor data to reduce data fluctuations and improve data reliability.

4. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The adaptive weighted fusion algorithm is used to fuse the feature information extracted by different sensors, specifically including: Data normalization: The feature information extracted from different sensors is normalized using the minimum-maximum normalization method, and the feature information is mapped to the [0,1] interval to eliminate the differences in dimension and value range of each feature information; Sensor accuracy evaluation: monitor the measurement error of each sensor in real time, and evaluate the measurement accuracy of the sensor by calculating the mean square error of the measurement error; Dynamic weight adjustment: Dynamically assign the weight of each sensor feature information in the fusion process according to the measurement accuracy of the evaluated sensor; Feature information fusion: Calculate the fused feature information based on the adjusted weights and normalized feature information of each sensor; Verification and correction of fusion results: Verify the fused feature information to check whether it conforms to the actual traffic conditions. If obvious deviations are found in the fusion results, analyze the possible causes and correct the fusion process. The formula for calculating the fusion feature information is: Where m is the number of sensors, w is i is the weight of the normalized feature information of the i-th sensor, x i is the normalized feature information of the i-th sensor.

5. The intelligent sensing control method based on multimodal sensor fusion according to claim 4 is characterized in that: The step of dynamically allocating the weight of each sensor feature information in the fusion process according to the evaluated measurement accuracy of the sensor specifically includes: Get the measurement value of each sensor in real time; Calculate the mean square error between the measured value and the true value of each sensor as the error value of each sensor; Calculate the weight of each sensor according to the error value of each sensor; The formula for calculating the weight of each sensor is: In the formula, w i is the weight of the i-th sensor, m is the number of sensors, e i is the error value of the i-th sensor.

6. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The dynamic traffic light timing optimization scheme based on fuzzy logic control specifically includes: Determine input variables: Obtain traffic flow, vehicle queue length and average vehicle waiting time from characteristic information as input variables for fuzzy logic control; Define fuzzy sets and membership functions: define fuzzy sets for each input variable, define three fuzzy sets of "low", "medium", and "high" for traffic flow, define three fuzzy sets of "short", "medium", and "long" for vehicle queue length, and define three fuzzy sets of "short", "medium", and "long" for average vehicle waiting time; and determine the membership function for each fuzzy set; Formulate fuzzy rules: formulate fuzzy rules based on traffic management experience and traffic operation rules; Fuzzy reasoning: According to the actual value of the current input variable, the membership degree of each fuzzy set is calculated through the membership function, and then fuzzy reasoning is performed according to the fuzzy rules to obtain the fuzzy output about the traffic light timing adjustment; Defuzzification: Use the centroid method to defuzzify and convert the fuzzy output into a specific adjustment value for the green light duration of the traffic light; Traffic light timing adjustment: Dynamically adjust the current traffic light timing plan based on the adjustment value obtained by defuzzification; The formula for defuzzification using the centroid method is: Where z is the adjusted value after defuzzification, n is the number of fuzzy outputs, and z i is the i-th output element, μ(z i ) is the element z i The corresponding membership degree.

7. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The traffic flow induction strategy based on reinforcement learning specifically includes: Define the state space: The state space is composed of traffic information after multimodal sensor fusion; Define action space: Define traffic flow induction actions, including recommended routes, estimated travel time, ramp control opening or closing, and lane restriction rule adjustment; Design reward function: Set the reward function, the goal of which is to reduce traffic congestion and reduce vehicle travel time; Strategy selection: Use the DQN algorithm to learn the optimal traffic flow induction strategy. At each time step, the algorithm selects an action from the action space based on the current state. The environment updates the state based on the action and feeds back the corresponding reward. Strategy update and optimization: As time goes by and traffic conditions change, the parameters of the DQN network are continuously updated to optimize the traffic flow induction strategy; Application of induction strategy: After the strategy optimization is completed, the optimized traffic flow induction strategy will be applied to traffic management.

8. The intelligent sensing control method based on multimodal sensor fusion according to claim 1 is characterized in that: The traffic incident automatic detection and response mechanism based on event rules specifically includes: Rule setting: define traffic event rules, specifically: Traffic accident detection rules: If the speed of multiple vehicles on a road section drops sharply to zero in a short period of time, and the vehicle positions are relatively fixed, and the image data identifies that the vehicles have collided and deformed, it is determined that a traffic accident has occurred; Traffic congestion detection rules: If the average speed of vehicles on a certain road section is lower than the set threshold, and the length of the vehicle queue exceeds a certain value and the duration reaches the set time, then it is determined that traffic congestion occurs on the road section; Abnormal parking rules for vehicles: If a vehicle stays in a non-parking area for longer than a preset time, it will be considered abnormal parking; Event detection: compare the integrated traffic information with the pre-set rules in real time to determine whether the rule conditions are met; Response measures trigger: If the fused traffic information meets a certain rule condition, the corresponding response measures will be automatically triggered, specifically: In case of traffic accidents, an alarm message is immediately sent to the traffic management department, and accident information is posted on surrounding variable information boards to prompt drivers to take detours; the signal lights at surrounding intersections are adjusted to guide vehicles to evacuate in an orderly manner; For traffic congestion, the congested road section information and recommended detour routes are released through variable information boards; ramp control is turned on to limit the number of vehicles entering the congested road section; the timing of surrounding traffic lights is adjusted to increase the green light duration in the relevant directions of the congested road section; For abnormal parking of vehicles, a reminder message is sent to nearby law enforcement officers, and a prompt message is displayed on a variable information board near the abnormal parking spot, requiring the vehicle to leave as soon as possible.

9. Intelligent sensing control system based on multimodal sensor fusion, characterized in that: include: Multimodal sensor module, including visual sensor, millimeter wave radar sensor and geomagnetic sensor, used to collect image data of intelligent traffic management area, distance and speed data of target objects, and passing and staying data of vehicles; A data preprocessing module, connected to the multimodal sensor module, for preprocessing the collected sensor data, including denoising and sharpening the image data, filtering and calibrating the radar data, and smoothing the geomagnetic sensor data; A feature extraction module, connected to the data preprocessing module, for extracting feature information from the preprocessed data; A multimodal data fusion module is connected to the feature extraction module and uses an adaptive weighted fusion algorithm to fuse feature information extracted by different sensors; An intelligent decision-making and control module, connected to the multimodal data fusion module, generates intelligent control decisions based on the fused traffic information, wherein the intelligent control decisions include a dynamic traffic signal timing optimization scheme based on fuzzy logic control, a traffic flow induction strategy based on reinforcement learning, and an automatic detection and response mechanism for traffic events based on event rules; The instruction sending module is connected to the intelligent decision and control module and is used to send corresponding control instructions to the traffic control equipment to adjust the traffic flow.

Images