Multi-sensor fusion unmanned vehicle detection method suitable for complex environment

Abstract

The invention discloses a multi-sensor fusion unmanned vehicle detection method suitable for a complex environment. The method comprises the following steps: obtaining road surface data through a laser radar and an infrared thermal imaging camera; detecting rain and snow weather conditions in real time, and calculating weather environment factors based on rainfall data and snowfall data; dynamically adjusting data weight coefficients of the laser radar and the infrared thermal imaging camera based on the weather environment factors; using a computer vision algorithm to identify an obstacle area, and extracting the distance and height of the obstacle according to the corresponding data weight coefficient; and the obstacle type is analyzed, the minimum safe distance or the minimum torque between the unmanned vehicle and the obstacle is calculated, and a driving decision is provided for the unmanned vehicle. According to the invention, the self-adaptive capability and safety of the unmanned vehicle in a complex environment are improved, the accident risk caused by sudden obstacles is effectively reduced through an intelligent early warning and intervention mechanism, and a solid safety foundation is laid for the wide application of an unmanned driving technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence autonomous driving technology, and in particular to an unmanned vehicle detection method suitable for multi-sensor fusion in complex environments. Background Art

[0002] With the continuous advancement of science and technology, driverless cars, as a new type of intelligent transportation, are gradually changing our lifestyle. They can not only provide efficient and accurate delivery services, but also effectively alleviate urban traffic pressure and improve road use efficiency.

[0003] However, how to accurately detect and warn of road obstacles and ensure driving safety while unmanned vehicles are driving has always been the focus of scientific researchers. Especially in complex environments, the complexity of the driving environment of unmanned vehicles is particularly complex due to the uncertainty of road properties, climate types, types of road users, movement status, number, traffic rules, etc.

[0004] Because of this, currently unmanned vehicles are basically running at low speeds on roads with excellent road conditions and good weather conditions. Although unmanned vehicles have made significant progress in pedestrian detection and path optimization in recent years, research on obstacle detection and early warning in complex environments is still relatively lagging behind. Summary of the invention

[0005] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0006] The purpose of the present invention is to provide a multi-sensor fusion unmanned vehicle detection method suitable for complex environments. Based on the data obtained by multiple sensors, the method integrates and calculates road conditions, provides driving decisions for unmanned vehicles, and ensures the driving safety of unmanned vehicles.

[0007] In order to achieve the above-mentioned object, the present invention provides a multi-sensor fusion unmanned vehicle detection method suitable for complex environments, comprising the following steps:

[0008] S100, when the unmanned vehicle is in operation, the laser radar scans the road ahead with high precision to obtain the three-dimensional structural information of the road surface, and triggers the infrared thermal imaging camera to start when the obstacle features are detected;

[0009] S200, infrared thermal imaging camera captures the infrared radiation signal of the scene in front, converts the infrared signal into a digital signal, and performs emissivity correction and temperature calculation based on Planck's radiation law to create an infrared image;

[0010] S300, preprocessing the infrared image;

[0011] S400, detecting rainy and snowy weather conditions in real time through a rain sensor and a snow sensor, and calculating weather environment factors based on the rain data and the snow data;

[0012] S500, dynamically adjusts the data weight coefficients of the laser radar and infrared thermal imaging camera based on weather and environmental factors;

[0013] S600, synchronize the measurement data of the laser radar and the infrared thermal imaging camera, and fuse the acquired data; identify the obstacle area using a computer vision algorithm, and extract the distance and height of the obstacle according to the corresponding data weight coefficient;

[0014] S700: Analyze the obstacle type based on the identified obstacle area and the distance and height of the obstacle, and calculate the minimum safe distance or minimum torque between the unmanned vehicle and the obstacle; and provide driving decisions for the unmanned vehicle based on the obstacle type and the corresponding minimum safe distance or minimum torque.

[0015] A further preferred technical solution of the present invention is that step S100 specifically comprises:

[0016] When the unmanned vehicle is running, the laser radar equipped on the unmanned vehicle emits laser pulses to the road ahead and measures the time for reflection, thereby calculating the precise position and distance of each point on the road surface, completing high-precision scanning, and obtaining the three-dimensional structural information of the road surface; when the laser radar detects the characteristics of an obstacle on the road surface, the laser radar will transmit the corresponding key data points to the unmanned vehicle's on-board control system, and the on-board control system will control the infrared thermal imaging camera to start.

[0017] Preferably, step S200 is specifically as follows:

[0018] The infrared thermal imaging camera of the unmanned vehicle is activated at the same time as the laser radar detects an obstacle, capturing the infrared radiation in the scene ahead, converting the infrared signal into a digital signal, and calculating the temperature value T, which is calculated as follows:

[0019]

[0020] Where h is Planck's constant, c is the speed of light, and k is the speed of light. B is the Boltzmann constant, λ is the wavelength, and M is the radiation intensity, which is calculated as follows:

[0021]

[0022] Where B(λ,T) is the blackbody radiation intensity at wavelength λ and temperature T, and its calculation formula is:

[0023]

[0024] The temperature value T is mapped to the pixel value of the image to reconstruct the image.

[0025] Preferably, the specific method of preprocessing the infrared image in step S300 includes denoising and contrast enhancement operations.

[0026] Preferably, the denoising operation includes:

[0027] After Gaussian filtering, it is expressed as:

[0028]

[0029] Where G(x,y) is the value of the Gaussian function at the point (x,y); σ is the standard deviation of the Gaussian distribution, which determines the width of the curve, that is, the range of the distribution; is the normalization factor, ensuring that the integral of the entire function is equal to 1;

[0030] Bilateral filtering, expressed as:

[0031]

[0032] Among them, F(x,y) is the pixel value at position (x,y) after bilateral filtering; I(x+i,y+j) is the pixel value at position (x,y) of the original image; w(i,j,x,y) is the weight function, which is the product of the similarity function and the distance function, that is, f r (i, j) is a similarity function, which is a Gaussian function used to measure the difference in pixel values. Its weight decreases as the difference in pixel values ​​increases. is a distance function, which is a Gaussian function used to measure the spatial distance between pixels, and its weight decreases as the distance increases; k defines the size of the filter, that is, the neighborhood range considered.

[0033] Preferably, the contrast enhancement operation includes:

[0034] Enhance the contrast of the image, expressed as:

[0035]

[0036] Where g(x,y) is the pixel value of the output image at position (x,y); f(x,y) is the pixel value of the input image at the same position; L is the total number of gray levels, for a standard 8-bit image, L = 256; P -f(s) is the probability density function of the gray level s of the input image, which is defined as the ratio of the frequency of occurrence of gray level s to the sum of the frequencies of occurrence of all possible gray levels; T is a function that represents contrast enhancement of the input image f(x,y); ds is a small change in the integral variable s, which represents the integration of the continuous change of s from 0 to 1;

[0037] Histogram equalization is expressed as:

[0038]

[0039] Where I′(x,y) represents the pixel value of the original image at position (x,y); I″(x,y) represents the new pixel value at position (x,y) after equalization; H(I′(x,y)) is the value of the histogram cumulative distribution function corresponding to the pixel value I′(x,y); H min is the minimum cumulative distribution function value in the histogram: H max is the maximum cumulative distribution function value in the histogram.

[0040] Preferably, in step S400, the weather environment factor is calculated based on the rainfall data and the snow data, and the specific method is:

[0041] S410. For the rainfall R and snow amount S, calculate their standard definition differences SD(R) and SD(S) to measure the degree of data dispersion:

[0042]

[0043] Where N is the number of data points, R i and S i are the rainfall and snowfall amounts of the ith data point, and are the average amounts of rainfall and snow, respectively;

[0044] S420, calculate the impact factors R and S, which are used to adjust the proportional constant to reflect the actual impact of rain and snow on the sensor performance. The calculation formula of the impact factors R and S is:

[0045]

[0046] Among them, the performance degradation percentage R and the percentage of performance degradation S The performance degradation percentage of radar and infrared sensors in rain and snow conditions, and the sensitivity score R and sensitivity scores S is the corresponding sensitivity score;

[0047] S430. Use linear regression to analyze the relationship between rain, snow and performance indicators, determine the slope as a preliminary estimate of the proportional constant, and adjust it according to the influencing factors. The proportional constant is calculated as follows:

[0048]

[0049]

[0050] S440, calculate the attenuation constant λ using the standard definition difference and the proportional constant R and λ S :

[0051]

[0052] Among them, max(R) and max(S) are the maximum rainfall and snowfall recorded in the experiment, respectively;

[0053] S450. Calculate the environmental factor E using the formula:

[0054]

[0055] λ R and λ S are the attenuation constants associated with rain and snow, respectively; R is the current rain measurement; S is the current snow measurement.

[0056] As a preferred embodiment, the specific method of S500 is:

[0057] After identifying the current environment as rain or snow, the following formula is used to gradually reduce the radar data weight and increase the infrared thermal imaging data weight:

[0058]

[0059] Among them, w LDAR and w IR are the weights of radar data and infrared thermal imaging data, respectively, LiDAR and P IR is the sensor performance indicator, and E is the weather environment factor, which is determined based on the measured value of rainfall or snow.

[0060] Preferably, the distance and height of the obstacle in step S600 are calculated by the following formula:

[0061]

[0062] Where t is the reference time point; Z fused (t) and H fused (t) is the distance and height of the obstacle after fusion at time point t; Z LiDAR and H LiDARZ is the distance and height of the obstacle measured by the LiDAR; IR and H IR The distance and height of obstacles measured by the infrared thermal imaging camera: δt LiDAR and δt IR is the time offset of the laser radar and infrared thermal imaging camera relative to the reference time point; w LiDAR and w IR are the weights of radar data and infrared thermal imaging data respectively.

[0063] Preferably, in step S700, when the obstacle is a speed bump, the width D of the speed bump is first calculated using the following formula:

[0064]

[0065] Where D is the width of the speed bump, B is the distance between the binocular cameras, f is the focal length of the camera, and x is l1 and x r1 Respectively represent the horizontal positions of feature points in the left and right cameras;

[0066] Then calculate the minimum distance d for the unmanned vehicle to safely pass the speed bump min , the calculation formula is:

[0067]

[0068] Among them, d min is the minimum safety distance required by the unmanned vehicle, v is the current speed of the vehicle, and a max is the maximum braking acceleration of the vehicle, t is the reaction time, g is the acceleration of gravity, D is the width of the speed bump, h is the height of the vehicle's center of gravity, G is the height of the speed bump, and K and S are both safety factors;

[0069] By comparing the minimum safe distance with the actual distance between the unmanned vehicles, the unmanned vehicle's onboard control system determines whether there is enough space and time to safely pass or slow down to pass the speed bump.

[0070] Preferably, in step S700, when the obstacle is a pedestrian or a vehicle, the minimum torque T1 required for the unmanned vehicle to successfully pass the pedestrian or the vehicle is calculated, and the specific formula is as follows:

[0071]

[0072] Where T1 is the minimum torque required, which is the torque required for the unmanned vehicle to maintain a turn at a specific speed and steering angle, m is the mass of the vehicle, v is the speed of the vehicle, θ is the steering angle of the wheel, L is the wheelbase of the vehicle, that is, the distance from the front wheel to the rear wheel, and k is the load transfer coefficient;

[0073] By comparing the minimum torque with the maximum torque of the unmanned vehicle, the on-board control system of the unmanned vehicle determines whether there is enough space and time to safely slow down or detour to avoid pedestrians or vehicles.

[0074] Preferably, in step S700, when the calculated actual distance is less than the minimum safety distance, the on-board control system of the unmanned vehicle issues a command for the unmanned vehicle to slow down or bypass the speed bump; when the calculated required minimum torque is greater than the maximum torque of the unmanned vehicle, the on-board control system of the unmanned vehicle issues a command for the unmanned vehicle to perform an emergency stop.

[0075] Beneficial effects: The present invention improves the adaptive ability and safety of unmanned vehicles in complex environments, and effectively reduces the risk of accidents caused by sudden obstacles through intelligent early warning and intervention mechanisms, laying a solid safety foundation for the widespread application of unmanned driving technology;

[0076] The present invention utilizes multi-sensor fusion technology, combined with laser radar, infrared thermal imaging camera, light sensor, rain sensor, snow sensor, etc., to accurately obtain detailed information about the vehicle's surrounding environment. After these data are processed by advanced algorithms, they can accurately identify and measure the size of obstacles. Subsequently, the identified obstacles are evaluated using a preset parameter database to determine whether they are within the acceptable range of the unmanned vehicle. If the evaluation results show that the height, length or slope of the obstacle exceeds the safe passing standard of the unmanned vehicle, the system will automatically start the emergency procedure to ensure that the vehicle can pass safely or stop if necessary. At the same time, an immediate alert can be sent to the operator, providing detailed fault diagnosis information and recommended countermeasures for manual intervention to further ensure the safety of the unmanned vehicle.

[0077] In summary, the present invention significantly enhances the operational stability of unmanned vehicles when facing unknown road conditions. It has an advanced prediction mechanism that can perform real-time environmental perception and analysis when the unmanned vehicle approaches possible obstacles or special road sections. Through this mechanism, it is possible to pre-evaluate whether the road conditions ahead are suitable for the unmanned vehicle to continue driving, so as to take corresponding preventive measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 The present invention is a flow chart of a multi-sensor fusion unmanned vehicle detection method suitable for use in complex environments. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0080] Combine the following Figure 1 The present invention describes an unmanned vehicle detection method based on multi-sensor fusion suitable for use in complex environments.

[0081] Embodiment: This embodiment provides a multi-sensor fusion unmanned vehicle detection method suitable for complex environments. The overall steps include:

[0082] S100, when the unmanned vehicle is in operation, the laser radar scans the road ahead with high precision to obtain the three-dimensional structural information of the road surface, and triggers the infrared thermal imaging camera to start when the obstacle features are detected;

[0083] S200, infrared thermal imaging camera captures the infrared radiation signal of the scene in front, converts the infrared signal into a digital signal, and performs emissivity correction and temperature calculation based on Planck's radiation law to create an infrared image;

[0084] S300, preprocessing the infrared image;

[0085] S400, detecting rainy and snowy weather conditions in real time through a rain sensor and a snow sensor, and calculating weather environment factors based on the rain data and the snow data;

[0086] S500, dynamically adjusts the data weight coefficients of the laser radar and infrared thermal imaging camera based on weather and environmental factors;

[0087] S600, synchronize the measurement data of the laser radar and the infrared thermal imaging camera, and fuse the acquired data; identify the obstacle area using a computer vision algorithm, and extract the distance and height of the obstacle according to the corresponding data weight coefficient;

[0088] S700: Analyze the obstacle type based on the identified obstacle area and the distance and height of the obstacle, and calculate the minimum safe distance or minimum torque between the unmanned vehicle and the obstacle; and provide driving decisions for the unmanned vehicle based on the obstacle type and the corresponding minimum safe distance or minimum torque.

[0089] Each step is described in detail below.

[0090] Step S100 is to perform high-precision scanning of the road ahead through a laser radar to obtain the three-dimensional structural information of the road surface, and trigger the infrared thermal imaging camera to start when obstacle features are detected.

[0091] When the unmanned vehicle is running, the laser radar equipped on the unmanned vehicle is responsible for high-precision scanning of the road ahead to obtain the three-dimensional structural information of the road. The laser radar emits laser pulses and measures the time of reflection, thereby calculating the precise position and distance of each point on the road surface, completing high-precision scanning, and obtaining the three-dimensional structural information of the road surface; when the laser radar detects the characteristics of obstacles on the road surface (such as obstacles, pedestrians and vehicles parked randomly on the road, hereinafter collectively referred to as obstacles), the laser radar transmits the corresponding key data points to the on-board control system of the unmanned vehicle, and the on-board control system controls the infrared thermal imaging camera to start.

[0092] Step S200 is that the infrared thermal imaging camera of the unmanned vehicle is activated at the same time as the laser radar detects an obstacle, captures the infrared radiation in the scene ahead, converts the infrared signal into a digital signal, calculates the temperature value, maps it to the pixel value of the image, and reconstructs the image.

[0093] In this step, the infrared thermal imaging camera performs emissivity correction and temperature calculation based on Planck's radiation law to reconstruct the image, specifically:

[0094] The infrared thermal imaging camera of the unmanned vehicle is activated at the same time as the laser radar detects an obstacle, capturing the infrared radiation in the scene ahead, converting the infrared signal into a digital signal, and calculating the temperature value T, which is calculated as follows:

[0095]

[0096] Where h is Planck's constant, c is the speed of light, and k is the speed of light. B is the Boltzmann constant, λ is the wavelength, and M is the radiation intensity, which is calculated as follows:

[0097]

[0098] Where B(λ,T) is the blackbody radiation intensity at wavelength λ and temperature T. According to Planck's radiation law, its calculation formula is:

[0099]

[0100] The temperature value T is mapped to the pixel value of the image to reconstruct the image.

[0101] Step S300 is to preprocess the infrared image, wherein the preprocessing method includes denoising and contrast enhancement.

[0102] The denoising process includes:

[0103] One is through Gaussian filtering, expressed as:

[0104]

[0105] Where G(x,y) is the value of the Gaussian function at the point (x,y); σ is the standard deviation of the Gaussian distribution, which determines the width of the curve, that is, the range of the distribution; is the normalization factor, ensuring that the integral of the entire function is equal to 1;

[0106] The second is to perform bilateral filtering, expressed as:

[0107]

[0108] Among them, F(x,y) is the pixel value at position (x,y) after bilateral filtering; I(x+i,y+j) is the pixel value at position (x,y) of the original image; w(i,j,x,y) is the weight function, which is the product of the similarity function and the distance function, that is, f r (i, j) is a similarity function, which is a Gaussian function used to measure the difference in pixel values. Its weight decreases as the difference in pixel values ​​increases. is a distance function, which is a Gaussian function used to measure the spatial distance between pixels, and its weight decreases as the distance increases; k defines the size of the filter, that is, the neighborhood range considered.

[0109] The contrast enhancement process includes:

[0110] One is to enhance the contrast of the image, expressed as:

[0111]

[0112] Where g(x,y) is the pixel value of the output image at position (x,y); f(x,y) is the pixel value of the input image at the same position; L is the total number of gray levels, for a standard 8-bit image, L = 256; P - f(s) is the probability density function of the gray level s of the input image, which is defined as the ratio of the frequency of occurrence of gray level s to the sum of the frequencies of occurrence of all possible gray levels; T is a function that represents contrast enhancement of the input image f(x,y); ds is a small change in the integral variable s, which represents the integration of the continuous change of s from 0 to 1;

[0113] The second is histogram equalization, expressed as:

[0114]

[0115] Where I′(x,y) represents the pixel value of the original image at position (x,y); I″(x,y) represents the new pixel value at position (x,y) after equalization; H(I′(x,y)) is the value of the histogram cumulative distribution function corresponding to the pixel value I′(x,y); H min is the minimum cumulative distribution function value in the histogram: H max is the maximum cumulative distribution function value in the histogram.

[0116] Step S400 is to detect rainy and snowy weather conditions in real time through a rain sensor and a snow sensor, and calculate weather environment factors based on the rain data and the snow data.

[0117] Rain sensors use capacitive or photoelectric principles to detect the presence of rain. Capacitive rain sensors sense rain by measuring the change in capacitance caused by water droplets. When rain falls on the sensor surface, the dielectric constant of the water droplets is different from that of air, causing the capacitance value between the sensor electrodes to change. By monitoring the change in capacitance value, the sensor is able to detect the presence of rain.

[0118] Snow sensors detect the presence of snow by measuring the weight or height of the snow. A common method is to use a piezoelectric element to measure the change in weight of the snow. When snow falls on the sensor, the piezoelectric element is pressed, generating an electrical signal proportional to the weight of the snow. By analyzing the strength of the electrical signal, the sensor is able to accurately measure the weight of the snow.

[0119] The weather environment factors are calculated based on rainfall data and snow data. The specific method is as follows:

[0120] SD difference calculation:

[0121] For the rainfall R and snow amount S, calculate their standard definition differences SD(R) and SD(S) to measure the degree of dispersion of the data:

[0122]

[0123] Where N is the number of data points, R i and S i are the rainfall and snowfall amounts of the ith data point, and are the average amounts of rainfall and snow, respectively;

[0124] Impact Factor Calculation:

[0125] The influence factors R and S are calculated to adjust the proportional constant to reflect the actual impact of rain and snow on sensor performance. The calculation formulas for the influence factors R and S are:

[0126]

[0127] Among them, the performance degradation percentage R and the percentage of performance degradation S The performance degradation percentage of radar and infrared sensors in rain and snow conditions, and the sensitivity score R and sensitivity scores S is the corresponding sensitivity score;

[0128] The proportionality constant is determined by:

[0129] Linear regression is used to analyze the relationship between rain, snow and performance indicators. The slope is determined as a preliminary estimate of the proportional constant and adjusted according to the influencing factors. The proportional constant is calculated as follows:

[0130]

[0131] Attenuation constant calculation:

[0132] Calculate the attenuation constant λ using the standard definition difference and the proportionality constant R and λ S :

[0133]

[0134] Among them, max(R) and max(S) are the maximum rainfall and snowfall recorded in the experiment, respectively;

[0135] Environmental factor calculation:

[0136] Calculate the environmental factor E using the formula:

[0137]

[0138] λ R and λ S are the attenuation constants associated with rain and snow, respectively; R is the current rain measurement (mm / hour); S is the current snow measurement (mm / hour).

[0139] Step S500 is to dynamically adjust the data weight coefficients of the laser radar and the infrared thermal imaging camera based on weather environment factors.

[0140] After identifying the current environment as rain or snow, the following formula is used to gradually reduce the radar data weight and increase the infrared thermal imaging data weight:

[0141]

[0142] Among them, w LDAR and w IR are the weights of radar data and infrared thermal imaging data, respectively, LiDAR and P IRis the sensor performance indicator, and E is the weather environment factor, which is determined based on the measured value of rainfall or snow.

[0143] Step S600 is to synchronize the measurement data of the laser radar and the infrared thermal imaging camera and fuse the acquired data; use the computer vision algorithm to identify the obstacle area and extract the distance and height of the obstacle according to the corresponding data weight coefficient.

[0144] In this step, the computer vision algorithm is first used to detect obstacles and calculate the height value:

[0145] Determine the baseline length of the cameras, which is the distance between the two cameras.

[0146] The intrinsic parameters and extrinsic parameters of the camera are calibrated to obtain the intrinsic parameter matrix and extrinsic parameter matrix of the camera.

[0147] Match the pixel coordinates of the corresponding points in the images of the left and right cameras, and calculate the distance Z and height H of the highest point of the obstacle protrusion from the unmanned vehicle based on the camera baseline length and pixel coordinates. The specific formula is as follows:

[0148]

[0149] Among them, Z represents the distance from the highest point of the obstacle to the unmanned vehicle, B represents the distance between the binocular cameras, f represents the focal length of the camera, and x represents the distance from the obstacle to the unmanned vehicle. l1 and x r1 Respectively represent the horizontal positions of feature points in the left and right cameras;

[0150] According to the similarity theorem, we can get:

[0151]

[0152] Convert to:

[0153]

[0154] Among them, H represents the height of the highest point of the obstacle protrusion from the ground, B represents the distance between the binocular cameras, f represents the focal length of the camera, y1 and y r Represent the vertical positions of feature points in the left and right cameras respectively.

[0155] Then the acquired data is fused, including:

[0156]

[0157] Where t is the reference time point; Z fused (t) and H fused (t) is the distance and height of the obstacle after fusion at time point t; Z LiDAR and HLiDAR Z is the distance and height of the obstacle measured by the LiDAR; IR and H IR The distance and height of obstacles measured by the infrared thermal imaging camera: δt LiDAR and δ IR is the time offset of the laser radar and infrared thermal imaging camera relative to the reference time point; w LiDAR and w IR are the weights of radar data and infrared thermal imaging data respectively.

[0158] Step S700 is to analyze the obstacle type and provide a safe passage strategy based on the identified obstacle area and the distance and height of the obstacle.

[0159] When the obstacle is a speed bump, first calculate the width D of the speed bump using the following formula:

[0160]

[0161] Where D is the width of the speed bump, B is the distance between the binocular cameras, f is the focal length of the camera, and x is l1 and x r1 Respectively represent the horizontal positions of feature points in the left and right cameras;

[0162] Then calculate the minimum distance d for the unmanned vehicle to safely pass the speed bump min , the calculation formula is:

[0163]

[0164] Among them, d min is the minimum safety distance required by the unmanned vehicle, v is the current speed of the vehicle, and a max is the maximum braking acceleration of the vehicle, t is the reaction time, g is the acceleration due to gravity, D is the width of the speed bump, h is the height of the vehicle's center of gravity, H is the height of the speed bump, K is the safety factor, which is determined based on vehicle design, suspension system characteristics, driving environment and regulatory requirements, and S is the safety factor, a value set based on experience and design standards to take into account uncertainty and risk to ensure safe driving;

[0165] By comparing the minimum safe distance with the actual distance between the unmanned vehicles, the unmanned vehicle's onboard control system determines whether there is enough space and time to safely pass or slow down to pass the speed bump.

[0166] When the obstacle is a pedestrian or a vehicle, the minimum torque T1 required for the unmanned vehicle to successfully pass the pedestrian or vehicle is calculated. The specific formula is as follows:

[0167]

[0168] Where T1 is the minimum torque required, which is the torque required for the unmanned vehicle to maintain a turn at a specific speed and steering angle, m is the mass of the vehicle, v is the speed of the vehicle, θ is the steering angle of the wheel, L is the wheelbase of the vehicle, that is, the distance from the front wheel to the rear wheel, and k is the load transfer coefficient, which is a dimensionless coefficient used to adjust for the additional centripetal force caused by weight transfer when the vehicle accelerates or brakes;

[0169] By comparing the minimum torque with the maximum torque of the unmanned vehicle, the on-board control system of the unmanned vehicle determines whether there is enough space and time to safely slow down or detour to avoid pedestrians or vehicles.

[0170] Finally, if the calculated actual distance is less than the minimum safe distance, the control system will issue a command and the unmanned vehicle will take appropriate avoidance measures, such as slowing down or detouring, to ensure that the vehicle passes the speed bump smoothly. If the calculated minimum torque is greater than the maximum torque of the unmanned vehicle, the control system will issue a command and the unmanned vehicle will take emergency braking measures to ensure the driving safety of the unmanned vehicle.

[0171] The core of the invention is to significantly enhance the operational stability of unmanned vehicles when facing unknown road conditions. It has an advanced prediction mechanism that can perform real-time environmental perception and analysis when the unmanned vehicle approaches possible obstacles or special road sections, such as obstacles. Through this mechanism, the system can pre-evaluate whether the road conditions ahead are suitable for the unmanned vehicle to continue driving, and take corresponding preventive measures.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion unmanned vehicle detection method suitable for complex environments, characterized in that: The following steps are involved: S100, when the unmanned vehicle is in operation, the laser radar scans the road ahead with high precision to obtain the three-dimensional structural information of the road surface, and triggers the infrared thermal imaging camera to start when the obstacle features are detected; S200, infrared thermal imaging camera captures the infrared radiation signal of the scene in front, converts the infrared signal into a digital signal, and performs emissivity correction and temperature calculation based on Planck's radiation law to create an infrared image; S300, preprocessing the infrared image; S400, detecting rainy and snowy weather conditions in real time through a rain sensor and a snow sensor, and calculating weather environment factors based on the rain data and the snow data; S500, dynamically adjusts the data weight coefficients of the laser radar and infrared thermal imaging camera based on weather and environmental factors; S600, synchronize the measurement data of the laser radar and the infrared thermal imaging camera, and fuse the acquired data; identify the obstacle area using a computer vision algorithm, and extract the distance and height of the obstacle according to the corresponding data weight coefficient; S700: Analyze the obstacle type based on the identified obstacle area and the distance and height of the obstacle, and calculate the minimum safe distance or minimum torque between the unmanned vehicle and the obstacle; and provide driving decisions for the unmanned vehicle based on the obstacle type and the corresponding minimum safe distance or minimum torque.

2. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 1 is characterized in that: The specific method of preprocessing the infrared image in step S300 includes denoising and contrast enhancement operations.

3. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 2 is characterized in that: The denoising operation includes: After Gaussian filtering, it is expressed as: Where G(x,y) is the value of the Gaussian function at the point (x,y); σ is the standard deviation of the Gaussian distribution, which determines the width of the curve, that is, the range of the distribution; is the normalization factor, ensuring that the integral of the entire function is equal to 1; Bilateral filtering, expressed as: Among them, F(x,y) is the pixel value at position (x,y) after bilateral filtering; I(x+i,y+j) is the pixel value at position (x,y) of the original image; w(i,j,x,u) is the weight function, which is the product of the similarity function and the distance function, that is, f , (i, j) is a similarity function, which is a Gaussian function used to measure the difference in pixel values. Its weight decreases as the difference in pixel values ​​increases. is a distance function, which is a Gaussian function used to measure the spatial distance between pixels, and its weight decreases as the distance increases; k defines the size of the filter, that is, the neighborhood range considered.

4. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 2 is characterized in that: The contrast enhancement operation includes: Enhance the contrast of the image, expressed as: Where g(x,y) is the pixel value of the output image at position (x,y); f(x,y) is the pixel value of the input image at the same position; L is the total number of gray levels, and for a standard 8-bit image, L=256; P_f(s) is the probability density function of the gray level s of the input image, which is defined as the ratio of the frequency of occurrence of gray level s to the sum of the frequencies of occurrence of all possible gray levels; T is a function that represents contrast enhancement of the input image f(x,y); ds is a small change in the integral variable s, which represents the integration of the continuous change of s from 0 to 1; Histogram equalization is expressed as: Where I′(x,y) represents the pixel value of the original image at position (x,y); I″(x,y) represents the new pixel value at position (x,y) after equalization; H(I′(x,y)) is the value of the histogram cumulative distribution function corresponding to the pixel value I′(x,y); H min is the minimum cumulative distribution function value in the histogram: H max is the maximum cumulative distribution function value in the histogram.

5. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 1 is characterized in that: In step S400, the weather environment factor is calculated based on the rainfall data and the snow data. The specific method is: S410. For the rainfall R and snow amount S, calculate their standard definition differences SD(R) and SD(S) to measure the degree of data dispersion: Where N is the number of data points, R i and S i are the rainfall and snowfall amounts of the ith data point, and are the average amounts of rainfall and snow, respectively; S420, calculate the impact factors R and S, which are used to adjust the proportional constant to reflect the actual impact of rain and snow on the sensor performance. The calculation formula of the impact factors R and S is: Among them, the performance degradation percentage R and the percentage of performance degradation S The performance degradation percentage of radar and infrared sensors in rain and snow conditions, and the sensitivity score R and sensitivity scores S is the corresponding sensitivity score; S430. Use linear regression to analyze the relationship between rain, snow and performance indicators, determine the slope as a preliminary estimate of the proportional constant, and adjust it according to the influencing factors. The proportional constant is calculated as follows: S440, calculate the attenuation constant λ using the standard definition difference and the proportional constant R and λ s : Among them, max(R) and max(S) are the maximum rainfall and snowfall recorded in the experiment, respectively; S450. Calculate the environmental factor E using the formula: λ R and λ s are the attenuation constants associated with rain and snow, respectively; R is the current rain measurement; S is the current snow measurement.

6. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 1 is characterized in that: The specific method of S500 is: After identifying the current environment as rain or snow, the following formula is used to gradually reduce the radar data weight and increase the infrared thermal imaging data weight: Among them, w LDAR and w IR are the weights of radar data and infrared thermal imaging data, respectively, LiDAR and P IR is the sensor performance indicator, and E is the weather environment factor, which is determined based on the measured value of rainfall or snow.

7. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 6, characterized in that: The distance and height of the obstacle in step S600 are calculated as follows: Where t is the reference time point; Z fused (t) and H fused (t) is the distance and height of the obstacle after fusion at time point t; Z LiDAR and H LiDAR Z is the distance and height of the obstacle measured by the LiDAR; IR and H IR The distance and height of obstacles measured by the infrared thermal imaging camera: δt LiDAR and δt IR is the time offset of the lidar and infrared thermal imaging camera relative to the reference time point; w LiDAR and w IR are the weights of radar data and infrared thermal imaging data respectively.

8. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 1 is characterized in that: In step S700, when the obstacle is a speed bump, the width D of the speed bump is first calculated using the following formula: Where D is the width of the speed bump, B is the distance between the binocular cameras, f is the focal length of the camera, and x is l1 and x r1 Respectively represent the horizontal positions of feature points in the left and right cameras; Then calculate the minimum distance d for the unmanned vehicle to safely pass the speed bump min , the calculation formula is: Among them, d min is the minimum safety distance required by the unmanned vehicle, v is the current speed of the vehicle, and a max is the maximum braking acceleration of the vehicle, t is the reaction time, g is the acceleration of gravity, D is the width of the speed bump, h is the height of the vehicle's center of gravity, H is the height of the speed bump, and K and S are both safety factors; By comparing the minimum safe distance with the actual distance between the unmanned vehicles, the unmanned vehicle's onboard control system determines whether there is enough space and time to safely pass or slow down to pass the speed bump.

9. The unmanned vehicle detection method of multi-sensor fusion suitable for complex environments according to claim 1, characterized in that: In step S700, when the obstacle is a pedestrian or a vehicle, the minimum torque T1 required for the unmanned vehicle to successfully pass the pedestrian or the vehicle is calculated. The specific formula is as follows: Where T1 is the minimum torque required, which is the torque required for the unmanned vehicle to maintain a turn at a specific speed and steering angle, m is the mass of the vehicle, v is the speed of the vehicle, θ is the steering angle of the wheel, L is the wheelbase of the vehicle, that is, the distance from the front wheel to the rear wheel, and k is the load transfer coefficient; By comparing the minimum torque with the maximum torque of the unmanned vehicle, the on-board control system of the unmanned vehicle determines whether there is enough space and time to safely slow down or detour to avoid pedestrians or vehicles.

10. The unmanned vehicle detection method using multi-sensor fusion suitable for complex environments according to claim 8 or 9, characterized in that: In step S700, when the calculated actual distance is less than the minimum safety distance, the on-board control system of the unmanned vehicle issues a command for the unmanned vehicle to slow down or bypass the speed bump; when the calculated required minimum torque is greater than the maximum torque of the unmanned vehicle, the on-board control system of the unmanned vehicle issues a command for the unmanned vehicle to perform an emergency stop.

Images