Electric motorcycle intelligent safety control system based on multi-sensor fusion

Abstract

The invention discloses an electric motorcycle intelligent safety control system based on multi-sensor fusion. The system comprises a sensor module, a data processing unit, a target detection and classification module, a danger level evaluation module and a motor control module, and realizes all-weather and all-around environmental perception and target detection through data fusion of a millimeter wave radar, a visible light camera and an infrared camera. And the power output of the motor is intelligently regulated and controlled according to the detected danger level, so that the driving safety of the electric motorcycle is improved. The problem of limitation of a single sensor of an existing electric motorcycle safety system is solved, the stability and reliability of the system in a complex environment are improved, and early warning and intelligent prevention and control of potential risks are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric vehicle safety control, and in particular relates to an intelligent safety control system for electric motorcycles based on multi-sensor fusion. Background Art

[0002] In recent years, with growing environmental awareness and the development of new energy technologies, electric motorcycles, as a green means of transportation, have gradually increased their share in urban traffic. However, due to their fast acceleration and low noise levels, electric motorcycles can easily distract pedestrians and other vehicles during operation, leading to traffic accidents. Furthermore, electric motorcycles also limit the driver's ability to perceive their surroundings, making accidents more likely to occur, especially at night or in adverse weather conditions such as rain and fog.

[0003] Existing electric motorcycle safety systems primarily rely on a single type of sensor, such as visible light cameras or radar for environmental perception. These systems significantly degrade in performance under certain conditions, such as at night or in inclement weather. For example, visible light cameras perform poorly at night or in fog; radar, while unaffected by lighting conditions, has limited target classification capabilities; and infrared cameras, while effective at night, perform poorly in bright sunlight. Furthermore, most existing systems only provide warnings and lack active control over the vehicle's powertrain, making them unable to automatically adjust the vehicle's power output in dangerous situations.

[0004] Therefore, there is an urgent need to develop an electric motorcycle safety control system that can perceive the environment in all weather and all directions and can intelligently adjust the motor power output according to the level of danger, so as to improve the driving safety of electric motorcycles. Summary of the Invention

[0005] The present invention aims to provide an electric motorcycle intelligent safety control system based on multi-sensor fusion to improve the driving safety of the electric motorcycle.

[0006] To achieve the purpose of the present invention, the present invention provides an electric motorcycle intelligent safety control system based on multi-sensor fusion, including the following modules:

[0007] A sensor module that integrates millimeter-wave radar, visible light camera, and infrared camera to simultaneously collect 3D point cloud data, visible light images, and thermal imaging data;

[0008] The data processing module is used to run the main processor of the real-time operating system and directly connect to each sensor via Ethernet. It calibrates the heterogeneous sensor data in time and space, builds a unified Cartesian coordinate system, and generates an environmental point cloud map that integrates time and space information.

[0009] The target detection and classification module deploys the YOLOv5 network to detect targets on the fused point cloud image and combines it with the Kalman filter to achieve multi-target tracking and motion trajectory prediction.

[0010] A control execution module collects target type, distance, relative speed, and predicted trajectory, calculates the dynamic hazard level using a multi-factor hazard assessment model, and regulates motor power output based on the hazard level to provide graded driving warnings.

[0011] The motor control module is used to adopt corresponding control strategies according to the risk level.

[0012] Compared with the existing technology, the significant progress of the present invention is as follows: (1) The present invention realizes the multi-sensor fusion of millimeter wave radar, visible light camera and infrared camera, overcomes the limitations of a single sensor under different environmental conditions, and realizes all-weather and all-round environmental perception; (2) The present invention adopts a deep learning algorithm to accurately classify the detected targets, and combines a multi-factor comprehensive evaluation model to achieve an accurate assessment of the danger level; (3) The present invention intelligently adjusts the motor power output according to the assessed danger level, thereby improving the driving safety of the electric motorcycle; (4) The present invention enhances the driver's safety awareness and response ability through a multi-modal driver feedback system.

[0013] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0015] Figure 1 It is a system architecture diagram of the present invention;

[0016] Figure 2 is a flowchart of the steps of the target detection and classification module of the present invention;

[0017] Figure 3 is a flow chart of the steps for evaluating the danger level of the control execution module of the present invention;

[0018] Figure 4 It is a sensor installation diagram of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0020] The present invention is an electric motorcycle intelligent safety control system based on multi-sensor fusion, combined with Figure 1 , including the following modules:

[0021] A sensor module that integrates millimeter-wave radar, visible light camera, and infrared camera to simultaneously collect 3D point cloud data, visible light images, and thermal imaging data;

[0022] The data processing module is used to run the main processor of the real-time operating system and directly connect to each sensor via Ethernet. It calibrates the heterogeneous sensor data in time and space, builds a unified Cartesian coordinate system, and generates an environmental point cloud map that integrates time and space information.

[0023] The target detection and classification module deploys the YOLOv5 network to detect targets on the fused point cloud image and combines it with the Kalman filter to achieve multi-target tracking and motion trajectory prediction.

[0024] A control execution module collects target type, distance, relative speed, and predicted trajectory, calculates the dynamic hazard level using a multi-factor hazard assessment model, and regulates motor power output based on the hazard level to provide graded driving warnings.

[0025] The motor control module is used to adopt corresponding control strategies according to the risk level.

[0026] The sensor module includes a millimeter wave radar unit, a visible light camera unit, and an infrared camera unit;

[0027] The millimeter-wave radar unit is used to detect the distance, speed, and general outline of the target, and is not restricted by light and weather conditions;

[0028] The visible light camera unit is used to finely identify and classify targets under good lighting conditions and provide rich texture and color information;

[0029] The infrared camera unit is used for target detection at night or under insufficient lighting conditions, especially for detection of heat sources.

[0030] The data processing module includes a data acquisition unit, a sensor calibration unit, and a data fusion processor unit;

[0031] The data acquisition unit collects raw data from the millimeter-wave radar, visible light camera, and infrared camera in real time through a Gigabit Ethernet interface to minimize transmission delays;

[0032] The sensor calibration unit performs spatiotemporal synchronization calibration on multi-source heterogeneous data to eliminate spatiotemporal misalignment between sensors;

[0033] The data fusion processing unit performs multimodal data fusion and feature enhancement to output a four-dimensional environment point cloud map ,in Represents the horizontal pose data of the point cloud, Represents the depth direction pose data of the point cloud, Represents the height direction pose data of the point cloud, Contains infrared intensity and millimeter wave velocity vectors.

[0034] The target detection and classification module is implemented using a deep learning framework, combined with Figure 2 , the method specifically comprises the following steps:

[0035] Step 1: Radar data processing: The radar echo signal is processed using the constant false alarm rate (CFAR) algorithm to extract the target's range, speed, and position information, providing an initial motion state estimate for target tracking.

[0036] Step 2: Image preprocessing: Image data synchronized with the radar data is obtained from the data processing module. De-noising, enhancement, and geometric correction are performed on the visible light image and infrared image respectively to improve image quality, reduce noise, and enhance contrast, providing high-quality input for subsequent target feature extraction.

[0037] Step 3: Feature extraction: ResNet50 (Residual Network 50, 50 layers) is used as the backbone network to extract deep semantic features from the preprocessed image. The deep semantic features include texture, shape, and context information of the target.

[0038] Step 4: Object Detection: Detection is performed on the preprocessed image using an improved object detection algorithm based on YOLOv5 (You Only Look Once v5, a single-stage object detection algorithm). This algorithm detects multiple types of objects in the image, including pedestrians, cars, motorcycles, bicycles, animals, and stationary obstacles, in real time, and outputs information such as the location, category, and confidence level of each object.

[0039] Step 5: Multimodal fusion: Fuse the radar data processing results with the image target detection results to improve the accuracy and robustness of detection and output more reliable target detection data results;

[0040] Step 6, target tracking: Using the distance, speed and position information of the target extracted in step 1, Kalman filtering and Hungarian algorithm are used to achieve continuous tracking of the target, establish an ID association to identify the target, analyze its motion trajectory, and finally output the target trajectory information with a unique ID, including current position, speed, acceleration and future position prediction results.

[0041] The control execution module's risk level assessment, combined with Figure 3 , assess the level of potential danger based on the following factors:

[0042] Factor 1, target attribute (O): Different weights are assigned according to the type of dangerous target. In this embodiment, the weight of pedestrians is 1.0, the weight of vehicles is 0.8, and the weight of fixed obstacles is 0.6;

[0043] Factor 2: Time to Collision Risk (TTC): Calculates the time to potential collision based on the relative speed and distance between the target and the electric motorcycle:

[0044] ;

[0045] in, is the target distance, is the relative speed, positive value means approaching, negative value means moving away; when hour, ;when hour, Calculate according to the formula.

[0046] Factor 3: Spatial Collision Risk (CP): Calculate the collision probability based on the relative position and predicted trajectory of the target and the electric motorcycle:

[0047] ;

[0048] in, and Represent the minimum distances between the target predicted trajectory and the electric motorcycle predicted trajectory on the x-axis and y-axis respectively; and represents the standard deviation of uncertainty on the x-axis and y-axis, respectively, and is related to sensor accuracy and prediction time. When CP is close to 1, the collision probability is high; when CP is close to 0, the collision probability is low.

[0049] Factor 4: Road Conditions (RCF): Combined with GPS and map data, taking into account factors such as road type and weather conditions:

[0050] ;

[0051] Where T represents the road type factor, which in this example is 0.5 for expressways, 0.7 for urban roads, and 0.9 for rural roads; W represents the weather condition factor, which in this example is 0.5 for sunny days, 0.8 for rainy days, and 1.0 for snowy / foggy days; C represents the road complexity factor, which in this example is 0.5 for straight roads, 0.8 for curved roads, and 1.0 for intersections;

[0052] 、 and are the weights of the corresponding factors, and ;

[0053] Finally, the comprehensive risk index CHI is calculated based on the above factors. The calculation formula is as follows:

[0054] ;

[0055] in, Indicates the target weight, which is determined by the target type; 、 、 、 represent the weight of each factor, and + ;

[0056] Hazard levels are determined according to CHI:

[0057] Low risk level: ;

[0058] Medium risk level: ;

[0059] High risk level: ;

[0060] The motor control module adopts corresponding control strategies according to the risk level:

[0061] Low danger level: Provides warning information to the driver but does not interfere with motor control;

[0062] Medium danger level: gradually reduce motor power output by up to 30% and provide a warning message;

[0063] High danger level: Rapidly reduces motor power output by up to 70%, activates the emergency braking system if necessary, and provides a strong warning signal.

[0064] The motor control communicates with the motor controller of the electric motorcycle via a CAN bus (Controller Area Network) to control the power output of the motor. The motor power output control is shown in the following formula:

[0065] ;

[0066] in, Indicates the output power, Indicates the maximum power, represents the comprehensive risk factor, and Indicates the upper and lower limits of the current level. Indicates the maximum power reduction ratio, which is related to the current evaluation level.

[0067] A sensor installation method is also provided for the above system, combined with Figure 4 , using the GNSS PPS (pulse per second) signal as the global clock source to trigger the synchronous acquisition of the sensors including the millimeter wave radar, visible light camera, and infrared camera, facilitating data alignment;

[0068] The millimeter-wave radar has an operating frequency of 77 GHz, a detection range of 0.2-150 meters, a horizontal field of view angle of ±60°, a vertical field of view angle of ±10°, a distance resolution of 0.1 meter, and a velocity resolution of 0.1 m / s.

[0069] The visible light camera has a resolution of 1920×1080, a frame rate of 30fps, a horizontal field of view of 120°, a vertical field of view of 70°, and an HDR function to adapt to different lighting conditions.

[0070] The infrared camera has a resolution of 640×480, a frame rate of 30fps, a horizontal field of view of 90°, a vertical field of view of 60°, a temperature resolution of 0.05°C, and can operate effectively in a completely dark environment.

[0071] Step 1: The sensor is installed at the front of the vehicle to obtain a sufficiently large field of view;

[0072] Step 2: The millimeter-wave radar is installed in the middle of the front of the vehicle due to its wide range;

[0073] Step 3: The visible light camera and the infrared camera have limited viewing angles and are installed on top of the millimeter wave radar to ensure sufficient overlapping directions and ensure target detection accuracy.

[0074] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0075] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent safety control system for electric motorcycles based on multi-sensor fusion, characterized in that: Includes the following modules: A sensor module that integrates millimeter-wave radar, visible light camera, and infrared camera to simultaneously collect 3D point cloud data, visible light images, and thermal imaging data; The data processing module is used to run the main processor of the real-time operating system and directly connect to each sensor via Ethernet. It calibrates the heterogeneous sensor data in time and space, builds a unified Cartesian coordinate system, and generates an environmental point cloud map that integrates time and space information. The target detection and classification module deploys the YOLOv5 network to detect targets on the fused point cloud image and combines it with the Kalman filter to achieve multi-target tracking and motion trajectory prediction. A control execution module collects target type, distance, relative speed, and predicted trajectory, calculates the dynamic hazard level using a multi-factor hazard assessment model, and regulates motor power output based on the hazard level to provide graded driving warnings. The motor control module is used to adopt corresponding control strategies according to the risk level.

2. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 1 is characterized in that: The sensor module includes a millimeter wave radar unit, a visible light camera unit, and an infrared camera unit; The millimeter-wave radar unit is used to detect the distance, speed, and general outline of the target, and is not restricted by light and weather conditions; The visible light camera unit is used to finely identify and classify targets under good lighting conditions and provide rich texture and color information; The infrared camera unit is used for target detection at night or under insufficient lighting conditions, especially for detection of heat sources.

3. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 1 is characterized in that: The data processing module includes a data acquisition unit, a sensor calibration unit, and a data fusion processor unit; The data acquisition unit collects raw data from the millimeter-wave radar, visible light camera, and infrared camera in real time through a Gigabit Ethernet interface to minimize transmission delays; The sensor calibration unit performs spatiotemporal synchronization calibration on multi-source heterogeneous data to eliminate spatiotemporal misalignment between sensors; The data fusion processing unit performs multimodal data fusion and feature enhancement, and outputs a four-dimensional environment point cloud map.

4. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 1 is characterized in that: The target detection and classification module is implemented using a deep learning framework method, which specifically includes the following steps: Step 1: Radar data processing: Use a constant false alarm rate algorithm to process the radar echo signal and extract the target's distance, speed, and position information; Step 2: Image preprocessing: Obtain image data synchronized with the radar data from the data processing module, and preprocess the visible light image and infrared image separately to provide high-quality input for subsequent target feature extraction; Step 3: Feature extraction: ResNet50 is used as the backbone network to extract deep semantic features from the preprocessed image. The deep semantic features include the texture, shape, and context information of the target. Step 4: Target detection: Perform detection operations on the preprocessed image, using the improved target detection algorithm based on YOLOv5 to complete real-time detection of multiple types of targets in the image, and output the location, category, and confidence information of each target; Step 5: Multimodal fusion: Fuse the radar data processing results and the image target detection results to output the target detection data results; Step 6, target tracking: Using the distance, speed and position information of the target extracted in step 1, Kalman filtering and Hungarian algorithm are used to achieve continuous tracking of the target, establish an ID association to identify the target, analyze its motion trajectory, and finally output the target trajectory information with a unique ID, including the current position, speed, acceleration and future position prediction results.

5. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 1 is characterized in that: The control execution module's risk level assessment evaluates the potential risk level based on the following factors: Factor 1, target attribute 0: assign different weights according to the type of dangerous target; Factor 2: Time to Collision Risk (TTC): Calculates the time to potential collision based on the relative speed and distance between the target and the electric motorcycle: ； in, is the target distance, is the relative speed, positive value means approaching, negative value means moving away; when hour, ;when hour, Then calculate according to the formula; Factor 3: Spatial Collision Risk (CP): Calculate the collision probability based on the relative position and predicted trajectory of the target and the electric motorcycle: ； in, and Represent the minimum distances between the target predicted trajectory and the electric motorcycle predicted trajectory on the x-axis and y-axis respectively; and represents the standard deviation of uncertainty on the x-axis and y-axis, respectively, and is related to sensor accuracy and prediction time. When CP is close to 1, the collision probability is high; when CP is close to 0, the collision probability is low. Factor 4: Road Conditions (RCF): Combined with GPS and map data, taking into account road type and weather conditions: ； Among them, T represents the road type factor; W represents the weather condition factor; C represents the road complexity factor; 、 and are the weights of the corresponding factors, and ; Finally, the comprehensive risk index CHI is calculated based on the above factors. The calculation formula is as follows: ； in, Indicates the target weight, which is determined by the target type; 、 、 、 represent the weight of each factor, and + ; Hazard levels are determined according to CHI: Low risk level: ; Medium risk level: ; High risk level: .

6. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 5 is characterized in that: The motor control module adopts corresponding control strategies according to the risk level: Low danger level: Provides warning information to the driver but does not interfere with motor control; Medium danger level: gradually reduce motor power output by up to 30% and provide a warning message; High danger level: Rapidly reduces motor power output by up to 70%, activates the emergency braking system if necessary, and provides a strong warning signal.

7. The electric motorcycle intelligent safety control system based on multi-sensor fusion according to claim 6 is characterized in that: The motor control communicates with the motor controller of the electric motorcycle via the CAN bus to control the power output of the motor. The motor power output control is shown in the following formula: ； in, Indicates the output power, Indicates the maximum power, represents the comprehensive risk factor, and Indicates the upper and lower limits of the current level. Indicates the maximum power reduction ratio, which is related to the current evaluation level.

Images