Multi-sensor fusion traffic light identification method and device

Abstract

The invention discloses a traffic light identification method and device based on multi-sensor fusion, and is used for solving the technical problems that the current traffic light color identification accuracy and reliability are low, driver state monitoring is lacked, and a reminding mechanism is not intelligent enough. The method comprises the steps of obtaining an infrared reflection intensity image by scanning an area in front of a vehicle, and generating three-dimensional point cloud data containing traffic lights; performing space-time alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and an aligned infrared image; according to the aligned point cloud data and the aligned infrared image, color state comprehensive identification based on feature matching fusion is carried out, and a color state identification result of the traffic light can be rapidly and accurately obtained; when it is detected that the traffic light is changed from the red light to the green light, the attention state of the driver is analyzed to judge whether reminding information is sent to the driver or not, and therefore traffic light change intelligent reminding can be achieved according to the actual analysis condition of driver state monitoring.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a multi-sensor fusion traffic light recognition method, a multi-sensor fusion traffic light recognition device, an electronic device and a storage medium. Background Art

[0002] Traffic light recognition and driver status monitoring are important means to ensure traffic safety. With the rapid development of autonomous driving technology and Advanced Driver Assistance Systems (ADAS), more and more vehicles are equipped with various sensors and cameras to improve driving safety and convenience. At present, some systems can recognize traffic lights (hereinafter referred to as traffic lights) and send reminders to drivers.

[0003] However, the current traffic light recognition system still lacks accuracy and stability in bad weather conditions (such as fog, rain, night, etc.). At the same time, the driver's attention state is also an important factor affecting driving safety. Traditional technology usually uses a single sensor (such as a camera) to recognize traffic lights. However, in extreme or bad weather conditions, the recognition effect of the camera will be significantly reduced. At the same time, there is still a lack of means to monitor the driver's status, and the reminder mechanism is not smart enough. Summary of the invention

[0004] The present invention provides a multi-sensor fusion traffic light recognition method, a multi-sensor fusion traffic light recognition device, an electronic device and a storage medium, which are used to solve or partially solve the current technical problems of low accuracy and reliability of traffic light color recognition, lack of driver status monitoring and insufficient intelligence of the reminder mechanism.

[0005] The present invention provides a multi-sensor fusion traffic light recognition method, the method comprising:

[0006] By scanning the area in front of the vehicle, an infrared reflection intensity image is obtained to generate three-dimensional point cloud data including traffic lights;

[0007] Performing spatiotemporal alignment processing on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image;

[0008] Performing comprehensive color state recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light;

[0009] When it is detected that the traffic light changes from red to green, the driver's attention state is analyzed to determine whether to send a reminder message to the driver.

[0010] Optionally, the three-dimensional point cloud data is generated by a laser radar, and the infrared reflection intensity image is obtained by scanning with an infrared sensor; the step of performing spatiotemporal alignment processing on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image includes:

[0011] Determining whether the three-dimensional point cloud data and the infrared reflection intensity image are collected within the same time frame;

[0012] If so, the relative position and posture between the laser radar and the infrared sensor are determined using external parameter calibration based on the installation position and viewing angle of the laser radar and the infrared sensor, and the three-dimensional point cloud data is spatially aligned with the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image.

[0013] Optionally, performing comprehensive color state recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light includes:

[0014] extracting traffic light features from the aligned point cloud data, and extracting light intensity features from the aligned infrared images;

[0015] Perform feature matching according to the traffic light feature and the light intensity feature to obtain a feature matching result, wherein the feature matching result is used to determine whether the traffic light feature and the light intensity feature belong to the same traffic light;

[0016] According to the light intensity characteristics, determining the color state of the traffic light to obtain a preliminary recognition result;

[0017] Based on the dynamic adjustment of confidence, the feature matching result and the preliminary recognition result are combined to perform a weighted fusion calculation to obtain the final color state recognition result of the traffic light.

[0018] Optionally, the dynamic adjustment based on confidence, combining the feature matching result and the preliminary recognition result to perform weighted fusion calculation to obtain the final color state recognition result of the traffic light, includes:

[0019] Obtaining a first basic confidence level corresponding to the laser radar and a second basic confidence level corresponding to the infrared sensor;

[0020] Determine the current environmental conditions around the vehicle, and determine whether to dynamically adjust the first basic confidence level and / or the second basic confidence level based on the result of the current environmental conditions determination;

[0021] When there is no need to dynamically adjust the confidence, a weighted fusion calculation is performed based on the first basic confidence and the second basic confidence, in combination with the feature matching result and the preliminary recognition result, to obtain a final color state recognition result of the traffic light;

[0022] When dynamic adjustment of the confidence is required, a weighted fusion calculation is performed based on the confidence obtained after the dynamic adjustment in combination with the feature matching result and the preliminary recognition result to obtain the final color state recognition result of the traffic light.

[0023] Optionally, the current environmental condition judgment result includes a weather condition judgment result and a lighting condition judgment result; and determining whether to dynamically adjust the first basic confidence and / or the second basic confidence based on the current environmental condition judgment result includes:

[0024] When the weather condition judgment result indicates that the current weather condition is extreme weather, lowering the first basic confidence level;

[0025] When the weather condition judgment result indicates that the current weather condition is non-extreme weather, keeping the first basic confidence level unchanged;

[0026] When the illumination condition judgment result indicates that the illumination around the current vehicle is within the strong light range, lowering the second basic confidence level;

[0027] When the illumination condition judgment result indicates that the illumination around the current vehicle is within a low-light range, increasing the second basic confidence level;

[0028] When the lighting condition judgment result indicates that the lighting around the current vehicle is within the normal light range, the second basic confidence level is kept unchanged.

[0029] Optionally, the analyzing the driver's attention state to determine whether to send a reminder message to the driver includes:

[0030] Capturing a driver's facial image and sight direction, and analyzing the driver's attention state based on the facial image and the sight direction based on an image processing algorithm to determine whether the driver has noticed the traffic light change;

[0031] When it is determined based on the attention state analysis result that the driver did not notice that the green light was on, a reminder message is sent to the driver in the form of an in-car voice broadcast, and / or a reminder message is displayed on a user display interface in the car;

[0032] When it is determined based on the attention state analysis result that the driver has noticed that the green light has turned on, no reminder information is sent to the driver.

[0033] Optionally, analyzing the driver's attention state based on the facial image and the line of sight direction and based on an image processing algorithm to determine whether the driver notices the traffic light change includes:

[0034] Extracting facial key points, left eye coordinate information, and right eye coordinate information from the facial image;

[0035] Calculating the sight line direction offset according to the facial key points, and calculating the sight line center coordinate information according to the left eye coordinate information and the right eye coordinate information;

[0036] Converting the sight direction into an initial sight direction vector, and calculating a target sight direction vector according to the initial sight direction vector and the sight direction offset;

[0037] Acquire the current coordinate information of the traffic light, and calculate the target position direction vector according to the current coordinate information and the sight center coordinate information;

[0038] Calculate the angle between the target sight line direction vector and the target position direction vector;

[0039] When the angle value is less than a preset angle threshold, it is determined that the driver has noticed the change in the traffic light;

[0040] When the angle value is greater than or equal to a preset angle threshold, it is determined that the driver has not noticed the change in the traffic light.

[0041] The present invention also provides a multi-sensor fusion traffic light recognition device, comprising:

[0042] A data acquisition unit, used to acquire an infrared reflection intensity image by scanning the area in front of the vehicle, and generate three-dimensional point cloud data including traffic lights;

[0043] A spatiotemporal alignment processing unit, configured to perform spatiotemporal alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared images;

[0044] A color state comprehensive recognition unit is used to perform color state comprehensive recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light;

[0045] The attention state analysis unit is used to analyze the driver's attention state when detecting that the traffic light changes from red to green, so as to determine whether to send a reminder message to the driver.

[0046] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0047] The memory is used to store program code and transmit the program code to the processor;

[0048] The processor is used to execute the traffic light recognition method based on multi-sensor fusion as described in any one of the above items according to the instructions in the program code.

[0049] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the multi-sensor fusion traffic light recognition method as described in any one of the above items.

[0050] It can be seen from the above technical solutions that the present invention has the following advantages:

[0051] A multi-sensor fusion traffic light recognition method is provided. First, by scanning the area in front of the vehicle, an infrared reflection intensity image is obtained to generate three-dimensional point cloud data containing traffic lights; based on the three-dimensional point cloud data and the infrared reflection intensity image, spatiotemporal alignment processing is performed to obtain aligned point cloud data and aligned infrared images; based on the aligned point cloud data and aligned infrared images, a comprehensive color state recognition based on feature matching fusion is performed to quickly and accurately obtain the color state recognition result of the traffic light; when it is detected that the traffic light changes from red to green, the driver's attention state is analyzed to determine whether to send a reminder message to the driver, so that intelligent reminders of traffic light changes can be realized according to the actual analysis of driver status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a structural diagram of a multi-sensor fusion traffic light recognition system;

[0054] Figure 2 A flowchart of the steps of a multi-sensor fusion traffic light recognition method;

[0055] Figure 3 It is a schematic diagram of the overall process of a multi-sensor fusion traffic light recognition method;

[0056] Figure 4 This is a structural block diagram of a multi-sensor fusion traffic light recognition device. DETAILED DESCRIPTION

[0057] The embodiments of the present invention provide a multi-sensor fusion traffic light recognition method, a multi-sensor fusion traffic light recognition device, an electronic device and a storage medium, which are used to solve or partially solve the current technical problems of low accuracy and reliability of traffic light color recognition, lack of driver status monitoring and insufficient intelligence of the reminder mechanism.

[0058] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] As an example, the current traffic light recognition system still lacks accuracy and stability in bad weather conditions (such as fog, rain, night, etc.). At the same time, the driver's attention state is also an important factor affecting driving safety. Traditional technology usually uses a single sensor (such as a camera) to recognize traffic lights. However, in extreme or bad weather conditions, the recognition effect of the camera will be significantly reduced. At the same time, there is still a lack of means to monitor the driver's status, and the reminder mechanism is not smart enough.

[0060] Through further analysis, the present invention believes that the problems existing in the conventional technology are as follows:

[0061] Single sensor reliance: Many current systems rely primarily on a single type of sensor, such as a camera or lidar. This single sensor reliance may result in reduced recognition accuracy in certain environments (such as bad weather, strong light interference, etc.). For example, a camera may not be able to accurately recognize the status of a traffic light under strong direct sunlight or low light conditions at night.

[0062] Lack of driver status monitoring: Currently, most traffic light recognition systems only focus on the external environment and ignore the monitoring of the driver's status. This means that even if the system can accurately recognize that the green light is on, it cannot determine whether the driver has noticed this change, and thus cannot provide timely and effective reminders. In this case, the driver may delay starting due to failure to notice the green light in time, affecting traffic flow.

[0063] The reminder mechanism is not smart enough: Some current reminder systems will directly send reminder information to the driver after detecting the green light, without considering whether the driver has noticed the green light. This method may cause unnecessary reminders, increase the driver's burden and irritability, and affect the driving experience.

[0064] In summary, current technologies have certain limitations in traffic light recognition and driver status monitoring, and the current means are also difficult to achieve efficient and intelligent reminder functions.

[0065] Therefore, one of the core invention points of the embodiment of the present invention is: in view of the shortcomings of the current technology, a method for traffic light recognition and driver status monitoring based on multi-sensor fusion of laser radar, infrared sensor and camera is provided. On the one hand, by comprehensively utilizing multiple sensor technologies and fusing the data of laser radar and infrared sensor, high-precision recognition of the color status of traffic lights is achieved, effectively reducing the misjudgment that may be caused by a single sensor, thereby improving the safety and reliability of the automatic driving system. At the same time, laser radar and infrared sensor each have different anti-interference characteristics, and can maintain high recognition accuracy even under different severe weather conditions. For example, on rainy and foggy days, laser radar can penetrate a certain degree of water vapor and fog, while infrared sensor is not limited by visible light conditions. The fusion of multiple sensors can effectively resist the influence of external factors such as changes in ambient light and weather conditions, and is widely applicable to a variety of practical application scenarios. On the other hand, the driver's attention is monitored by the camera, so that reminder information can be sent to the driver at the right time to improve driving safety and efficiency.

[0066] Reference Figure 1 , showing a schematic structural diagram of a multi-sensor fusion traffic light recognition system provided by an embodiment of the present invention.

[0067] Combination Figure 1 The multi-sensor fusion traffic light recognition system 100 may mainly include a data processing unit 101, namely an electronic control unit (ECU), and a laser radar (Light Detection and Ranging, LiDAR) 102, an infrared sensor 103, a camera 104 and a reminder device 105 which are respectively connected to the data processing unit 101 for communication.

[0068] The laser radar 102, the infrared sensor 103 and the camera 104 are respectively connected to the data processing unit 101 through a data bus. The data processing unit 101 is connected to the reminder device 105, and the reminder is triggered by a control signal.

[0069] The laser radar 102 is mainly used to detect the position and status of the traffic light ahead. The infrared sensor 103 is mainly used to assist the laser radar 102 in identifying traffic lights in low light or bad weather conditions. The camera 104 is installed in the car to capture the driver's facial expression and line of sight. The data processing unit 101 is responsible for data processing and decision logic. The reminder device 105 can mainly include a speaker and a display screen (user display interface) to send reminder information to the driver.

[0070] In order to combine the laser radar 102 and the infrared sensor 103 to identify the color state of the traffic light, it is necessary to comprehensively utilize the data of the two sensors. That is, higher accuracy is achieved through data fusion technology. In the actual traffic light color state recognition process, the laser radar 102 and the infrared sensor 103 scan the road ahead in real time to detect the position and state of the traffic light. The data processing unit 101 receives the sensor data and analyzes the current color state of the traffic light (such as red light, green light) through an algorithm.

[0071] Combined with the multi-sensor fusion traffic light recognition system introduced earlier, refer to Figure 2 , shows a flowchart of a multi-sensor fusion traffic light recognition method provided by an embodiment of the present invention, which may specifically include the following steps:

[0072] Step 201, by scanning the area in front of the vehicle, obtaining an infrared reflection intensity image, and generating three-dimensional point cloud data including traffic lights;

[0073] The area ahead can be scanned by LiDAR to generate 3D point cloud data containing traffic lights. Assume that the position of the traffic light detected by LiDAR is , or you can write .

[0074] The infrared sensor scans the front area to obtain the infrared reflection intensity image. Assume that the infrared sensor detects that the infrared reflection intensity image includes the wavelength of the light source and the corresponding light intensity .

[0075] Step 202, performing spatiotemporal alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image;

[0076] In some embodiments, the process of obtaining the aligned point cloud data and the aligned infrared image based on the spatiotemporal alignment of the three-dimensional point cloud data and the infrared reflection intensity image includes the following sub-steps S01 to S02:

[0077] Step S01: determining whether the three-dimensional point cloud data and the infrared reflection intensity image are collected within the same time frame;

[0078] First, to improve the accuracy of data fusion, it is necessary to ensure that the data of the lidar and infrared sensor are collected in the same time frame for subsequent fusion. Therefore, when the data processing unit receives the three-dimensional point cloud data and the infrared reflection intensity image, it can first determine whether the two are collected in the same time frame. If the two are collected in the same time frame, the subsequent data processing operation is performed. If the two are not collected in the same time frame, the data collection operation is performed again until the collection time of the two is synchronized.

[0079] Step S02: If yes, then use external parameter calibration to determine the relative position and posture between the laser radar and the infrared sensor according to their installation position and viewing angle, spatially align the three-dimensional point cloud data with the infrared reflection intensity image, and obtain aligned point cloud data and aligned infrared image.

[0080] In step S02 , the relative position and posture between the two sensors are determined mainly by using an extrinsic calibration method.

[0081] Assume that the transformation matrix between the laser radar coordinate system and the infrared sensor coordinate system is , then the expression of the lidar point cloud in the infrared sensor coordinate system is: .

[0082] in, It is the three-dimensional point cloud data of LiDAR; It is the three-dimensional point cloud data in the converted infrared sensor coordinate system, that is, the aligned point cloud data.

[0083] Step 203, performing comprehensive color state recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light;

[0084] In some embodiments, the implementation process of performing comprehensive color state recognition based on feature matching fusion according to the aligned point cloud data and the aligned infrared image to obtain the color state recognition result of the traffic light includes the following sub-steps S11 to S14:

[0085] Step S11: extracting traffic light features from the aligned point cloud data, and extracting light intensity features from the aligned infrared images;

[0086] Extracting traffic light features from aligned point cloud data , including the location of traffic lights and shape characteristics.

[0087] Extracting light intensity features from aligned infrared images , including the wavelength of the light source and light intensity feature.

[0088] Step S12: performing feature matching according to the traffic light feature and the light intensity feature to obtain a feature matching result, wherein the feature matching result is used to determine whether the traffic light feature and the light intensity feature belong to the same traffic light;

[0089] Traffic light features Light intensity characteristics Perform feature matching to obtain feature matching results. Assume that the matching function is:

[0090]

[0091] in, Represents the difference between two features feature comparison points.

[0092] The judgment result of feature matching can be expressed by the following formula:

[0093]

[0094] Indicates feature matching, which can correspond to the judgment score of the laser radar . Indicates feature mismatch, which can correspond to the judgment score of the laser radar .

[0095] Therefore, by adopting efficient feature extraction and matching algorithms, the system can complete data processing and decision-making in a very short time, allowing the system to run efficiently on the embedded platform.

[0096] Step S13: judging the color state of the traffic light according to the light intensity characteristics and obtaining a preliminary recognition result;

[0097] Assume that the wavelength ranges of traffic lights are and Among them, the wavelength range of red light . Green light wavelength range .

[0098] The initial judgment conditions for the color status of traffic lights are:

[0099]

[0100] Among them, when the light intensity characteristics (mainly the wavelength of the light source) When , the traffic light is initially judged to be red. When the traffic light is initially judged to be green, the color status is initially judged to be green. Both of these situations correspond to the judgment score of the infrared sensor. .

[0101] When the wavelength of the light source When it does not fall within these two wavelength ranges, the traffic light is initially judged to be in a non-red or green light state (i.e., unkown state). This situation corresponds to the judgment score of the infrared sensor. When in the unkown state, the system does not process it, and the driver needs to make a judgment. For example, if the traffic light is not on, it means that the current traffic light is faulty or blocked, and the system cannot further detect its traffic light status and judges it as unkown.

[0102] Step S14: Based on the dynamic adjustment of confidence, a weighted fusion calculation is performed on the comprehensive feature matching results and the preliminary recognition results to obtain the final color state recognition result of the traffic light.

[0103] Step S14 essentially combines the above two judgment results, and fuses the data of the laser radar and infrared sensor through a weighted fusion method to finally determine the color status of the traffic light.

[0104] In practical applications, confidence refers to the degree of trust or certainty of a judgment or prediction result. In the comprehensive judgment of the embodiment of the present invention, confidence is used to represent the reliability weight of the respective judgment results of the laser radar and the infrared sensor. The value of confidence can be between [0,1]. The higher the value, the more reliable the judgment result.

[0105] The embodiment of the present invention mainly relies on laser radar to generate the shape of the object and relies on infrared sensor to generate point map to judge the color state. Therefore, the adjustment of confidence can be based on the following considerations:

[0106] The first is sensor performance. LiDAR is mainly used to generate the shape and position of an object. Its confidence can be set based on indicators such as resolution, accuracy, and noise level. Infrared sensors are mainly used to detect color wavelengths. Its confidence can be set based on indicators such as spectral resolution, sensitivity, and anti-interference ability.

[0107] The second is environmental conditions. Environmental conditions can be further divided into lighting conditions and weather conditions. Among them, lighting conditions refer to the fact that the performance of infrared sensors may be affected under strong or weak light conditions, and its confidence needs to be adjusted dynamically. Weather conditions refer to extreme weather (bad weather) such as rain, fog, snow, sandstorms, etc., which may affect the accuracy of LiDAR and its confidence needs to be adjusted accordingly.

[0108] The next thing is historical data and experience. We can mainly consider historical accuracy. Based on past data statistics, we calculate the accuracy of each sensor under different conditions to determine its confidence.

[0109] The confidence level can also be adjusted dynamically based on real-time feedback of system performance. By monitoring the system's performance in real time, the confidence level can be adjusted dynamically based on the actual situation. For example, if the reading of the infrared sensor fluctuates greatly at a certain moment, its confidence level can be temporarily lowered.

[0110] Based on the above introduction, assume that the confidence parameter symbols of the feature matching result (corresponding to the laser radar) and the preliminary recognition result of the traffic light color state (corresponding to the infrared sensor) are , .

[0111] The weighted fusion calculation can be performed through the following comprehensive judgment formula:

[0112]

[0113] in, Indicates the judgment score of the laser radar (0 or 1); Indicates the judgment score of the infrared sensor (0 or 1).

[0114] When the comprehensive score Greater than a threshold When , it is determined that a traffic light is detected, and the color state is the color state of the traffic light recognized by the infrared sensor.

[0115] Through the above calculation process, the data from the lidar and infrared sensors can be effectively combined to accurately identify the color status of traffic lights, thereby improving the accuracy and reliability of the system's traffic light recognition.

[0116] Based on the above introduction, in a specific implementation, based on dynamic adjustment of confidence, weighted fusion calculation is performed on the comprehensive feature matching results and the preliminary recognition results to obtain the final color state recognition result of the traffic light, which may include the following sub-steps S141 to S143:

[0117] Step S141: obtaining a first basic confidence level corresponding to the laser radar and a second basic confidence level corresponding to the infrared sensor;

[0118] Step S142: judging the current environmental conditions around the vehicle, and determining whether to dynamically adjust the first basic confidence level and / or the second basic confidence level based on the current environmental conditions judgment result;

[0119] Combined with the above description, environmental conditions can be mainly divided into weather conditions and lighting conditions, and the current environmental condition judgment result can mainly include weather condition judgment result and lighting condition judgment result. On this basis, based on the current environmental condition judgment result, it is determined whether to dynamically adjust the first basic confidence and / or the second basic confidence, which can mainly include the following situations:

[0120] Dynamic adjustment of the first basic confidence level:

[0121] Case 1: When the weather condition judgment result indicates that the current weather condition is extreme weather, the first basic confidence level is lowered.

[0122] Case 2: When the weather condition judgment result indicates that the current weather condition is non-extreme weather, the first basic confidence level remains unchanged.

[0123] Dynamic adjustment of the second basic confidence level:

[0124] Case 1: When the illumination condition judgment result indicates that the illumination around the current vehicle is within the strong light range, the second basic confidence level is lowered.

[0125] Case 2: When the lighting condition judgment result indicates that the lighting around the current vehicle is within the low-light range, the second basic confidence level is increased.

[0126] Case 3: When the lighting condition judgment result indicates that the lighting around the current vehicle is within the normal light range, the second basic confidence level is kept unchanged.

[0127] Step S143: When there is no need to dynamically adjust the confidence level, a weighted fusion calculation is performed based on the first basic confidence level and the second basic confidence level, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light; when dynamic adjustment of the confidence level is required, a weighted fusion calculation is performed based on the confidence level obtained after the dynamic adjustment, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0128] When it is determined based on the current environmental conditions that there is no need to dynamically adjust the confidence level, a weighted fusion calculation is performed directly based on the original first basic confidence level and the second basic confidence level, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0129] When it is determined that dynamic adjustment of confidence is required based on current environmental conditions, a weighted fusion calculation is performed based on the confidence obtained after dynamic adjustment, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0130] For example, if the first updated confidence is obtained only by adjusting the first basic confidence, the first updated confidence will replace the original first basic confidence, and a weighted fusion calculation will be performed based on the first updated confidence and the second basic confidence, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0131] For another example, the second updated confidence is obtained only by adjusting the second basic confidence, then the second updated confidence replaces the original second basic confidence, and a weighted fusion calculation is performed based on the first basic confidence and the second updated confidence, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0132] For another example, the first updated confidence is obtained by adjusting the first basic confidence, and the second updated confidence is obtained by adjusting the second basic confidence. The first updated confidence replaces the original first basic confidence, and the second updated confidence replaces the original second basic confidence. A weighted fusion calculation is performed based on the first updated confidence and the second updated confidence, combined with the feature matching results and the preliminary recognition results, to obtain the final color state recognition result of the traffic light.

[0133] For example, assuming that the basic confidence of the lidar , the basic confidence of the infrared sensor , the laser radar's judgment score , the judgment score of the infrared sensor , threshold The comprehensive score is for:

[0134]

[0135] because Greater than threshold , it is determined that the detected one is a traffic light.

[0136] The confidence level is dynamically adjusted to take into account current environmental conditions.

[0137] For example, in bright light conditions, the confidence of the infrared sensor may decrease. .

[0138] For example, in rainy and foggy weather, the confidence of the LiDAR may be reduced. .

[0139] by , For example, if the score remains unchanged, the comprehensive score can be recalculated. :

[0140]

[0141] Even in this case, Still greater than the threshold Therefore, it can still be determined that a traffic light is detected.

[0142] In this way, the confidence level can be dynamically adjusted to improve the accuracy and reliability of the system under different environmental conditions.

[0143] In some cases, based on the above-mentioned traffic light judgment and recognition, in order to effectively reduce the possibility of misdetection and further improve the accuracy of traffic light color state judgment, additional misdetection detection measures can be set according to actual needs. For example, before data fusion, the sensor data is filtered and denoised to reduce the influence of ambient light interference and noise. For example, a Kalman filter or a mean filter is used to smooth the data. For another example, multiple infrared sensors are used for redundant detection, and reliability is improved by comparing the readings of multiple sensors. When the readings of multiple sensors are consistent, the credibility of the detection results can be improved. Historical data and pattern recognition technology can also be used to verify the current detection results. If the current detection result is significantly different from the historical data, it can be marked as a suspected misdetection and further checked. In addition, each sensor can be regularly calibrated and maintained to maintain its operation in the best state, reduce misdetection caused by sensor aging or damage, and so on. It is understandable that the present invention is not limited to this.

[0144] Therefore, by optimizing data processing and fusion algorithms, not only can the real-time requirements of the autonomous driving system be met, but the consumption of computing resources can also be reduced.

[0145] Step 204: when it is detected that the traffic light changes from red to green, the driver's attention state is analyzed to determine whether to send a reminder message to the driver.

[0146] In this step, when it is detected that the traffic light changes from red to green, the driver's facial image and sight direction are captured by the camera. The data processing unit analyzes the driver's attention state through an image processing algorithm to determine whether the driver has noticed the change of the traffic light.

[0147] In some embodiments, the process of analyzing the driver's attention state to determine whether to send a reminder message to the driver includes the following sub-steps S21 to S22:

[0148] Step S21: capturing the driver's facial image and line of sight direction, and analyzing the driver's attention state based on the facial image and line of sight direction and an image processing algorithm to determine whether the driver has noticed the traffic light change;

[0149] Furthermore, the implementation process of step S21 may mainly include the following sub-steps S211 to S216:

[0150] Step S211: extracting facial key points, left eye coordinate information and right eye coordinate information from the facial image;

[0151] Step S212: calculating the sight line direction offset according to the facial key points, and calculating the sight line center coordinate information according to the left eye coordinate information and the right eye coordinate information;

[0152] Assume that the facial key points are , which can be obtained by facial key points The calculated sight line offset is recorded as . Sight direction offset The calculation process of can be implemented by referring to the relevant technology and will not be elaborated here.

[0153] Assume that the left eye coordinate information and the right eye coordinate information are and , then the center point position of the driver's eyes (i.e. the coordinate information of the center of sight) can be calculated by the following formula:

[0154]

[0155] Step S213: converting the sight direction into an initial sight direction vector, and calculating the target sight direction vector according to the initial sight direction vector and the sight direction offset;

[0156] Convert the view direction to the initial view direction vector , according to the initial sight direction vector And the sight direction offset , the target sight direction vector is calculated by the following formula :

[0157]

[0158] Step S214: obtaining the current coordinate information of the traffic light, and calculating the target position direction vector according to the current coordinate information and the sight center coordinate information;

[0159] To determine whether the driver is looking at the traffic light, it is necessary to calculate the target sight direction vector The direction vector of the target position (i.e., the angle between the traffic light position vector). Combined with the previous content, we can know that the traffic light position detected by the lidar is , then the target position direction vector It can be calculated by the following formula:

[0160]

[0161] Step S215: Calculate the angle between the target sight line direction vector and the target position direction vector;

[0162] Specifically, the angle value The calculation steps are as follows:

[0163] First calculate the target sight direction vector The direction vector of the target position The dot product of .

[0164] Then calculate the target sight direction vector The direction vector of the target position The modulus (length) and .

[0165] Then calculate the cosine. That is, take the dot product Divide by the product of the magnitudes of two vectors , we can get the cosine value between two vectors .

[0166] Finally, the target sight direction vector is calculated by the arccos function The direction vector of the target position The angle between : .

[0167] Step S216: When the angle value is less than the preset angle threshold, it is determined that the driver has noticed the change in the traffic light; when the angle value is greater than or equal to the preset angle threshold, it is determined that the driver has not noticed the change in the traffic light.

[0168] The results of attention state analysis can be expressed by the following formula:

[0169]

[0170] When the angle value Less than the preset angle threshold When the driver notices the traffic light change, When the angle value Greater than or equal to the preset angle threshold When the driver does not notice the traffic light change, .

[0171] Through the above algorithm and related calculation formulas, it is possible to monitor the driver's attention state and determine whether he notices the changes in traffic lights.

[0172] For example, it is assumed that the target sight direction vector can be obtained by calculation , target position direction vector .

[0173] Target sight direction vector The direction vector of the target position The angle between The calculation steps are as follows:

[0174] First, calculate the dot product of the two:

[0175]

[0176] Then calculate the magnitude of the two vectors separately:

[0177]

[0178]

[0179] Then calculate the cosine:

[0180]

[0181] Finally, the angle value is calculated by the arc cosine function:

[0182]

[0183] Through the above calculation process, the angle between the driver's line of sight and the position of the traffic light can be determined, so as to further judge whether the driver is paying attention to the traffic light. Assuming the angle value If the angle is less than a preset threshold (such as 45°), it is considered that the driver is looking at the traffic light. Otherwise, it is considered that the driver did not pay attention to the traffic light change, corresponding to .

[0184] Step S22: When it is determined based on the attention status analysis results that the driver has not noticed the green light coming on, a reminder message is sent to the driver in the form of an in-car voice broadcast, and / or the reminder message is displayed on the user display interface in the car; when it is determined based on the attention status analysis results that the driver has noticed the green light coming on, no reminder message is sent to the driver.

[0185] When the traffic light changes from red to green, the data processing unit first determines whether the driver has noticed the change. If it is determined that the driver has not noticed the green light, the data processing unit sends a reminder sound or displays a reminder message to the driver through the reminder device. If the driver has noticed the green light, the data processing unit will not send a reminder to the driver to avoid unnecessary interference.

[0186] In an embodiment of the present invention, a method for traffic light recognition and driver status monitoring based on multi-sensor fusion of laser radar, infrared sensor and camera is provided. On the one hand, by comprehensively utilizing multiple sensor technologies and fusing the data of laser radar and infrared sensor, high-precision recognition of the color status of traffic lights can be achieved, effectively reducing the misjudgment that may be caused by a single sensor, thereby improving the safety and reliability of the automatic driving system. At the same time, laser radar and infrared sensor each have different anti-interference characteristics, and can maintain high recognition accuracy even under different severe weather conditions. The fusion of multiple sensors can effectively resist the influence of external factors such as changes in ambient light and weather conditions, and is widely applicable to a variety of practical application scenarios. On the other hand, the driver's attention is monitored by the camera, so that reminder information can be sent to the driver at the appropriate time to improve driving safety and efficiency.

[0187] For better explanation, refer to Figure 3 , showing a schematic diagram of the overall process of a multi-sensor fusion traffic light recognition method provided by an embodiment of the present invention. It should be pointed out that this embodiment only briefly describes the general process of multi-sensor fusion traffic light recognition, and the specific implementation process of each step can be understood by referring to the relevant content in the aforementioned embodiment, which will not be described here. It can be understood that the present invention is not limited to this.

[0188] Step 301: Scan the area in front of the vehicle through an infrared sensor and a laser radar to obtain an infrared reflection intensity image and generate three-dimensional point cloud data including traffic lights;

[0189] Step 302: When the data processing unit determines that the three-dimensional point cloud data and the infrared reflection intensity image are collected in the same time frame, the relative position and posture between the laser radar and the infrared sensor are determined by using external parameter calibration according to the installation position and viewing angle of the laser radar and the infrared sensor, and the three-dimensional point cloud data and the infrared reflection intensity image are spatially aligned to obtain aligned point cloud data and aligned infrared image;

[0190] Step 303: extracting traffic light features from the aligned point cloud data, and extracting light intensity features from the aligned infrared images;

[0191] Step 304: performing feature matching according to the traffic light feature and the light intensity feature to obtain a feature matching result, and judging the color state of the traffic light according to the light intensity feature to obtain a preliminary recognition result;

[0192] Step 305: Based on the dynamic adjustment of confidence, a weighted fusion calculation is performed on the comprehensive feature matching result and the preliminary recognition result to obtain the color state recognition result of the traffic light;

[0193] Step 306: When it is detected that the traffic light changes from red to green, the driver's facial image and sight direction are captured by the camera, and the driver's attention state is analyzed based on the facial image and sight direction based on the image processing algorithm to determine whether the driver notices the change of the traffic light;

[0194] Step 307-1: when it is determined based on the attention state analysis result that the driver has not noticed the green light, a reminder message is sent to the driver in the form of an in-car voice broadcast through a reminder device, and / or a reminder message is displayed on a user display interface in the car;

[0195] Step 307 - 2 : When it is determined based on the attention state analysis result that the driver has noticed that the green light has turned on, no reminder information is sent to the driver.

[0196] Reference Figure 4 , shows a structural block diagram of a multi-sensor fusion traffic light recognition device provided by an embodiment of the present invention, which may specifically include:

[0197] The data acquisition unit 401 is used to acquire an infrared reflection intensity image by scanning the area in front of the vehicle, and generate three-dimensional point cloud data including traffic lights;

[0198] A spatiotemporal alignment processing unit 402 is used to perform spatiotemporal alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image;

[0199] A color state comprehensive recognition unit 403 is used to perform color state comprehensive recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light;

[0200] The attention state analysis unit 404 is used to analyze the driver's attention state when detecting that the traffic light changes from red to green, so as to determine whether to send a reminder message to the driver.

[0201] In an optional embodiment, the three-dimensional point cloud data is generated by a laser radar, and the infrared reflection intensity image is obtained by scanning with an infrared sensor; the spatiotemporal alignment processing unit 402 includes:

[0202] A time synchronization judgment unit, used to judge whether the three-dimensional point cloud data and the infrared reflection intensity image are collected in the same time frame;

[0203] The spatial alignment unit is used to determine the relative position and posture between the laser radar and the infrared sensor according to the installation position and viewing angle of the laser radar and the infrared sensor using external parameter calibration, and to spatially align the three-dimensional point cloud data with the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image.

[0204] In an optional embodiment, the color state comprehensive identification unit 403 includes:

[0205] a feature extraction unit, configured to extract traffic light features from the aligned point cloud data and light intensity features from the aligned infrared images;

[0206] a feature matching unit, configured to perform feature matching according to the traffic light feature and the light intensity feature to obtain a feature matching result, wherein the feature matching result is used to determine whether the traffic light feature and the light intensity feature belong to the same traffic light;

[0207] A traffic light color state preliminary recognition unit, used to determine the color state of the traffic light according to the light intensity feature and obtain a preliminary recognition result;

[0208] The weighted fusion calculation unit is used to dynamically adjust based on the confidence level, and to perform weighted fusion calculation based on the feature matching result and the preliminary recognition result to obtain the final color state recognition result of the traffic light.

[0209] In an optional embodiment, the weighted fusion calculation unit includes:

[0210] A basic confidence acquisition unit, used to acquire a first basic confidence corresponding to the laser radar and a second basic confidence corresponding to the infrared sensor;

[0211] a current environmental condition judgment unit, configured to judge the current environmental condition around the vehicle, and determine whether to dynamically adjust the first basic confidence level and / or the second basic confidence level based on the current environmental condition judgment result;

[0212] A first weighted fusion calculation unit, configured to perform a weighted fusion calculation based on the first basic confidence level and the second basic confidence level, in combination with the feature matching result and the preliminary recognition result, to obtain a final color state recognition result of the traffic light when no dynamic adjustment of the confidence level is required;

[0213] The second weighted fusion calculation unit is used to perform weighted fusion calculation according to the confidence obtained after dynamic adjustment, combined with the feature matching result and the preliminary recognition result, to obtain the final color state recognition result of the traffic light when dynamic adjustment of the confidence is required.

[0214] In an optional embodiment, the current environmental condition judgment result includes a weather condition judgment result and a lighting condition judgment result; and the current environmental condition judgment unit is specifically used to:

[0215] When the weather condition judgment result indicates that the current weather condition is extreme weather, lowering the first basic confidence level;

[0216] When the weather condition judgment result indicates that the current weather condition is non-extreme weather, keeping the first basic confidence level unchanged;

[0217] When the illumination condition judgment result indicates that the illumination around the current vehicle is within the strong light range, lowering the second basic confidence level;

[0218] When the illumination condition judgment result indicates that the illumination around the current vehicle is within a low-light range, increasing the second basic confidence level;

[0219] When the lighting condition judgment result indicates that the lighting around the current vehicle is within the normal light range, the second basic confidence level is kept unchanged.

[0220] In an optional embodiment, the attention state analysis unit 404 includes:

[0221] an attention state analysis subunit, for capturing a driver's facial image and a line of sight direction, and analyzing the driver's attention state based on the facial image and the line of sight direction and an image processing algorithm to determine whether the driver has noticed a change in a traffic light;

[0222] A first control unit for sending a reminder message is used to send a reminder message to the driver in the form of an in-vehicle voice broadcast and / or display the reminder message on a user display interface in the vehicle when it is determined based on the attention state analysis result that the driver did not notice that the green light was on.

[0223] The second control unit for sending the reminder information is used to not send the reminder information to the driver when it is determined based on the attention state analysis result that the driver has noticed that the green light is on.

[0224] In an optional embodiment, the attention state analysis subunit includes:

[0225] A facial information extraction unit, used to extract facial key points, left eye coordinate information and right eye coordinate information from the facial image;

[0226] A sight line center coordinate information calculation unit, used to calculate the sight line direction offset according to the facial key points, and calculate the sight line center coordinate information according to the left eye coordinate information and the right eye coordinate information;

[0227] a target sight line direction vector calculation unit, configured to convert the sight line direction into an initial sight line direction vector, and calculate a target sight line direction vector according to the initial sight line direction vector and the sight line direction offset;

[0228] A target position direction vector calculation unit, used to obtain the current coordinate information of the traffic light, and calculate the target position direction vector according to the current coordinate information and the sight center coordinate information;

[0229] An angle value calculation unit, used to calculate the angle value between the target sight line direction vector and the target position direction vector;

[0230] A first angle value judgment unit, used for judging that the driver has noticed the traffic light change when the angle value is less than a preset angle threshold;

[0231] The second angle value judgment unit is used to judge that the driver has not noticed the change of the traffic light when the angle value is greater than or equal to a preset angle threshold.

[0232] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0233] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, some technical features are distinguished by the first and the second in the embodiments of the present invention. The first and the second are only used for data distinction and have no other special meanings. It can be understood that the present invention is not limited to this.

[0234] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0235] The memory is used to store the program code and transmit the program code to the processor;

[0236] The processor is used to execute the traffic light recognition method using multi-sensor fusion according to any embodiment of the present invention according to the instructions in the program code.

[0237] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the multi-sensor fusion traffic light recognition method of any embodiment of the present invention.

[0238] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0239] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0243] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion traffic light recognition method, characterized in that: include: By scanning the area in front of the vehicle, an infrared reflection intensity image is obtained to generate three-dimensional point cloud data including traffic lights; Performing spatiotemporal alignment processing on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image; Performing comprehensive color state recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light; When it is detected that the traffic light changes from red to green, the driver's attention state is analyzed to determine whether to send a reminder message to the driver.

2. The multi-sensor fusion traffic light recognition method according to claim 1, characterized in that: The three-dimensional point cloud data is generated by a laser radar, and the infrared reflection intensity image is obtained by scanning with an infrared sensor; performing spatiotemporal alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image includes: Determining whether the three-dimensional point cloud data and the infrared reflection intensity image are collected within the same time frame; If so, the relative position and posture between the laser radar and the infrared sensor are determined using external parameter calibration based on the installation position and viewing angle of the laser radar and the infrared sensor, and the three-dimensional point cloud data is spatially aligned with the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared image.

3. The multi-sensor fusion traffic light recognition method according to claim 2, characterized in that: The step of performing comprehensive color state recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light includes: extracting traffic light features from the aligned point cloud data, and extracting light intensity features from the aligned infrared images; Perform feature matching according to the traffic light feature and the light intensity feature to obtain a feature matching result, wherein the feature matching result is used to determine whether the traffic light feature and the light intensity feature belong to the same traffic light; According to the light intensity characteristics, determining the color state of the traffic light to obtain a preliminary recognition result; Based on the dynamic adjustment of confidence, the feature matching result and the preliminary recognition result are combined to perform a weighted fusion calculation to obtain the final color state recognition result of the traffic light.

4. The traffic light recognition method based on multi-sensor fusion according to claim 3 is characterized in that: The dynamic adjustment based on confidence, combining the feature matching result and the preliminary recognition result to perform weighted fusion calculation to obtain the final color state recognition result of the traffic light, includes: Obtaining a first basic confidence level corresponding to the laser radar and a second basic confidence level corresponding to the infrared sensor; Determine the current environmental conditions around the vehicle, and determine whether to dynamically adjust the first basic confidence level and / or the second basic confidence level based on the result of the current environmental conditions determination; When there is no need to dynamically adjust the confidence, a weighted fusion calculation is performed based on the first basic confidence and the second basic confidence, in combination with the feature matching result and the preliminary recognition result, to obtain a final color state recognition result of the traffic light; When dynamic adjustment of the confidence is required, a weighted fusion calculation is performed based on the confidence obtained after the dynamic adjustment in combination with the feature matching result and the preliminary recognition result to obtain the final color state recognition result of the traffic light.

5. The multi-sensor fusion traffic light recognition method according to claim 4, characterized in that: The current environmental condition judgment result includes a weather condition judgment result and a lighting condition judgment result; and determining whether to dynamically adjust the first basic confidence level and / or the second basic confidence level based on the current environmental condition judgment result includes: When the weather condition judgment result indicates that the current weather condition is extreme weather, lowering the first basic confidence level; When the weather condition judgment result indicates that the current weather condition is non-extreme weather, keeping the first basic confidence level unchanged; When the illumination condition judgment result indicates that the illumination around the current vehicle is within the strong light range, lowering the second basic confidence level; When the illumination condition judgment result indicates that the illumination around the current vehicle is within a low-light range, increasing the second basic confidence level; When the lighting condition judgment result indicates that the lighting around the current vehicle is within the normal light range, the second basic confidence level is kept unchanged.

6. The multi-sensor fusion traffic light recognition method according to claim 1, characterized in that: The analyzing the driver's attention state to determine whether to send a reminder message to the driver includes: Capturing a driver's facial image and sight direction, and analyzing the driver's attention state based on the facial image and the sight direction based on an image processing algorithm to determine whether the driver has noticed the traffic light change; When it is determined based on the attention state analysis result that the driver did not notice that the green light was on, a reminder message is sent to the driver in the form of an in-car voice broadcast, and / or a reminder message is displayed on a user display interface in the car; When it is determined based on the attention state analysis result that the driver has noticed that the green light has turned on, no reminder information is sent to the driver.

7. The multi-sensor fusion traffic light recognition method according to claim 6, characterized in that: Analyzing the driver's attention state based on the facial image and the sight line direction based on an image processing algorithm to determine whether the driver notices the traffic light change includes: Extracting facial key points, left eye coordinate information, and right eye coordinate information from the facial image; Calculating the sight line direction offset according to the facial key points, and calculating the sight line center coordinate information according to the left eye coordinate information and the right eye coordinate information; Converting the sight direction into an initial sight direction vector, and calculating a target sight direction vector according to the initial sight direction vector and the sight direction offset; Acquire the current coordinate information of the traffic light, and calculate the target position direction vector according to the current coordinate information and the sight center coordinate information; Calculate the angle between the target sight line direction vector and the target position direction vector; When the angle value is less than a preset angle threshold, it is determined that the driver has noticed the change in the traffic light; When the angle value is greater than or equal to a preset angle threshold, it is determined that the driver has not noticed the change in the traffic light.

8. A multi-sensor fusion traffic light recognition device, characterized in that: include: A data acquisition unit, used to acquire an infrared reflection intensity image by scanning the area in front of the vehicle, and generate three-dimensional point cloud data including traffic lights; A spatiotemporal alignment processing unit, configured to perform spatiotemporal alignment processing based on the three-dimensional point cloud data and the infrared reflection intensity image to obtain aligned point cloud data and aligned infrared images; A color state comprehensive recognition unit is used to perform color state comprehensive recognition based on feature matching and fusion according to the aligned point cloud data and the aligned infrared image to obtain a color state recognition result of the traffic light; The attention state analysis unit is used to analyze the driver's attention state when detecting that the traffic light changes from red to green, so as to determine whether to send a reminder message to the driver.

9. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the multi-sensor fusion traffic light recognition method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the multi-sensor fusion traffic light recognition method according to any one of claims 1 to 7.

Images