An intelligent pavement recognition system based on multi-modal sensor data fusion

Abstract

The application relates to the technical field of road surface identification, and discloses an intelligent road surface identification system based on multi-modal sensor data fusion, which comprises a solving module, a region screening module, a data extraction module, a rotation inversion module, a depolarization module, a correction module and a road surface state judgment module. The orientation reference of sky polarization is obtained by solving the sun geometry, multi-spectral polarization images are fused to extract road surface features, the spectral polarization orientation split quantity is calculated, polarization parameter reverse rotation is carried out, the rotating linear polarization degree and the depolarization intensity are obtained, and the water body absorption optical thickness is determined. The system corrects the optical thickness by using an adaptive scale coefficient, and finally realizes intelligent identification of a dry state, a film state and a water film state of a road surface.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road surface recognition, more particularly, it relates to an intelligent road surface recognition system based on multi-modal sensor data fusion. BACKGROUND

[0002] New energy vehicle intelligent driving and active safety system widely rely on multi-modal sensors (including visible light camera, near-infrared camera, short-wave infrared imaging unit and polarization imaging module) to perceive road surface state to adapt to different light, weather and ground conditions. Under clear sky sunshine, the incident light of the road surface is mainly sky scattered light, and the polarization direction and intensity are significantly affected by the sun's position and the observation angle, and the imaging results of the sensor depend on the road surface material, humidity and optical transmission chain characteristics.

[0003] In order to reduce the cabin heat load and improve energy efficiency, new energy vehicles generally use windshield and camera cover plates with infrared reflection or low radiation multilayer coating. Such coating has different transmission and reflection characteristics for different wavelengths of light, and responds differently to the amplitude and phase of the two orthogonal polarization components of light, resulting in inconsistent polarization information received by each waveband sensor; this difference is amplified with changes in vehicle attitude and solar elevation angle. Existing image fusion algorithms and hardware calibration are mostly completed in laboratory static conditions, based on constant optical transmission characteristics, and cannot cope with the dynamic drift of coating phase response over time, temperature and sunlight conditions in real driving scenarios.

[0004] The most significant technical problem in the above scenario is that under clear sky sunshine, the coating structure causes the multi-spectral imaging system to produce directional deviation in the polarization information of the same road surface, causing multi-modal fusion errors, and leading to misjudgment of the dry and wet state or the reflection characteristics of the road surface. The formation mechanism of the technical problem is that the polarization of sky scattered light follows the natural law (the polarization degree is highest when the observation angle is about 90 degrees), while the rotation difference of the coating for different waveband polarization orientations is significant at large incident angles. This systematic deviation is misidentified as an optical change of the road surface itself, and traditional calibration and algorithms cannot eliminate it, which becomes the core problem of multi-modal fusion landing. SUMMARY

[0005] The present application provides an intelligent road surface recognition system based on multi-modal sensor data fusion, which solves the technical problems raised in the background art.

[0006] An intelligent road surface recognition system based on multi-modal sensor data fusion, comprising:

[0007] A solving module for solving the sun geometry to obtain a sky polarization orientation reference;

[0008] The region filtering module acquires the first-band polarization image and the second-band polarization image, calculates the total light intensity, linear polarization degree and polarization orientation angle of each spectral band based on the polarization images, and fuses and filters the region of interest on the road surface.

[0009] The data extraction module extracts the polarization orientation angle characterization values ​​of the first-band polarization image and the second-band polarization image within the region of interest on the road surface, calculates the spectral polarization orientation split based on the characterization values, and determines the sign parameter in conjunction with the sky polarization orientation reference.

[0010] The rotation inversion module rotates the first band polarization image and the second band polarization image in the same reverse order within the preset polarization parameter plane according to the spectral polarization orientation split and the sign parameter, to obtain the rotational linear polarization degree of the second band polarization image.

[0011] The depolarization module calculates the depolarization intensity of the water body characteristic absorption spectrum and the reference spectrum based on the total light intensity and the degree of rotational linear polarization of the second-band polarization image, and determines the water body absorption optical thickness based on the depolarization intensity.

[0012] The correction module constructs an adaptive scaling coefficient based on the spectral polarization orientation split, and uses the adaptive scaling coefficient to calibrate the water absorption optical thickness to obtain the corrected optical thickness.

[0013] The road surface condition determination module maps the corrected optical thickness to the corresponding road surface condition; the road surface condition includes: dry state, thin film state and water film state.

[0014] The beneficial effects of this invention include: by introducing a multimodal polarization sensing fusion and solar geometry calculation mechanism, high-precision identification of road surface conditions (dry, thin film, and water film) under different lighting, weather, and vehicle attitude conditions is achieved. Using a sky polarization orientation reference, multi-spectral polarization deviations caused by differences in the transmission characteristics of multi-layer coatings are dynamically corrected; by constructing spectral polarization orientation splitting and adaptive scaling coefficients, the effects of inconsistent polarization responses and time drift between sensor bands are effectively eliminated; and a stable optical thickness mapping model is established by combining the depolarization intensity of the water body's characteristic absorption spectrum. Therefore, this invention can maintain polarization characteristic consistency and spectral response under complex natural lighting conditions, improving the robustness and accuracy of road surface dry / wet identification, and providing state switching support for intelligent driving perception systems. Attached Figure Description

[0015] Fig. 1 This is a block diagram of the present invention;

[0016] Fig. 2 This is a flowchart of the present invention;

[0017] Fig. 3 This is a flowchart of the road surface recognition and determination process of the present invention;

[0018] Fig. 4 This is a schematic diagram of polarization direction sampling according to the present invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, features described in some examples may be combined in other examples.

[0020] like Figs. 1 to 4 As shown, an intelligent road surface recognition system based on multimodal sensor data fusion includes:

[0021] The solution module calculates solar geometry to obtain a sky polarization orientation reference.

[0022] The region filtering module acquires the first-band polarization image and the second-band polarization image, calculates the total light intensity, linear polarization degree and polarization orientation angle of each spectral band based on the polarization images, and fuses and filters the region of interest on the road surface.

[0023] The data extraction module extracts the polarization orientation angle characterization values ​​of the first-band polarization image and the second-band polarization image within the region of interest on the road surface, calculates the spectral polarization orientation split based on the characterization values, and determines the sign parameter in conjunction with the sky polarization orientation reference.

[0024] The rotation inversion module rotates the first band polarization image and the second band polarization image in the same reverse order within the preset polarization parameter plane according to the spectral polarization orientation split and the sign parameter, to obtain the rotational linear polarization degree of the second band polarization image.

[0025] The depolarization module calculates the depolarization intensity of the water body characteristic absorption spectrum and the reference spectrum based on the total light intensity and the degree of rotational linear polarization of the second-band polarization image, and determines the water body absorption optical thickness based on the depolarization intensity.

[0026] The correction module constructs an adaptive scaling coefficient based on the spectral polarization orientation split, and uses the adaptive scaling coefficient to calibrate the water absorption optical thickness to obtain the corrected optical thickness.

[0027] The road surface condition determination module maps the corrected optical thickness to the corresponding road surface condition; the road surface condition includes: dry state, thin film state and water film state.

[0028] In one embodiment of the present invention, solving the solar geometry to obtain a sky polarization orientation reference includes:

[0029] Get geographic latitude Geographical longitude World Time Day number Camera principal ray elevation angle Camera principal ray azimuth angle ;

[0030] Calculate the time equation :

[0031] ;

[0032] Calculate the angle ;

[0033] Calculate the solar declination angle ;

[0034] Calculate the sine component of the solar altitude angle The sine component of the solar azimuth angle Sum and cosine components :

[0035] ;

[0036] ;

[0037] ;

[0038] Construct the unit vector of the observation direction Unit vector in the direction of the sun :

[0039] ;

[0040] ;

[0041] Unit vector of the observation direction Unit vector in the direction of the sun The cross product of the scattering plane is used as the normal vector. , scattering plane normal vector and observation direction unit vector The cross product of the vectors is used as the unit vector for the polarization direction of the sky. ;

[0042] unit vector of sky polarization direction unit vector of sky polarization direction The standard sky polarization vector is obtained by normalizing the vector magnitude. ;

[0043] Determine the standard sky polarization vector lateral component and longitudinal components ;

[0044] Calculate the sky polarization orientation reference angle .

[0045] Geographic latitude / longitude: determines the specific location of the vehicle on the Earth's surface. The variation patterns of the sun's altitude and azimuth differ in different latitude regions of the Earth, and it serves as a spatial reference for calculating the sun's position.

[0046] Coordinated Universal Time (UTC): Standard time based on the Prime Meridian.

[0047] Day number: refers to the day of the year.

[0048] Camera principal ray elevation / azimuth angle: The principal ray refers to the ray of light from the center of the camera lens pointing to the road surface. The elevation angle is the angle between the ray and the horizontal plane (positive for upward and negative for downward, usually negative when observing the road surface). The azimuth angle is the horizontal angle between the ray and due south, used to determine the specific direction of the vehicle camera relative to the road surface.

[0049] Earth's orbit around the sun is elliptical rather than circular, and its axis of rotation is tilted at an angle to the plane of its orbit. This causes a discrepancy between the actual time the sun reaches local noon (solar apparent time) and UT. The time equation is used to correct this discrepancy and obtain accurate solar apparent time.

[0050] The hour angle is the angle measuring the sun's position relative to local noon. At local noon, the sun is due south (in the Northern Hemisphere), and the hour angle is 0. In the morning, the sun is in the east, and the hour angle is negative; in the afternoon, the sun is in the west, and the hour angle is positive. The hour angle reflects the sun's east-west position throughout the day.

[0051] The solar declination angle is the angle between the point where the sun is directly overhead and the Earth's equatorial plane (positive in the Northern Hemisphere and negative in the Southern Hemisphere). It varies with the seasons (approximately +23.45 degrees in summer and approximately -23.45 degrees in winter). The solar declination angle can be calculated using the day sequence number. The solar declination angle reflects the sun's north-south position in the sky.

[0052] The solar altitude angle is the angle between the sun's rays and the horizontal plane (the larger the altitude angle, the more direct the sunlight). Its sine component can be directly used to determine the sun's altitude. The solar azimuth angle is the angle between the sun's rays and the local due south horizontal plane. Its sine and cosine components can jointly determine the sun's position on the horizontal plane.

[0053] A unit vector is a vector with a length of 1 that only represents direction (eliminating the interference of distance on the direction representation). That is, the observation direction unit vector is constructed by the elevation angle and azimuth angle of the camera's principal ray, describing the three-dimensional direction of the camera looking at the road surface; the solar direction unit vector is constructed by the components of the solar altitude angle and azimuth angle, describing the three-dimensional direction of the sunlight hitting the ground. The two vectors together constitute the spatial geometric relationship between the sun, the observation point, and the road surface.

[0054] The scattering plane is a plane formed by the direction of the sun, the observation direction, and the observation point (skylight scattered from the sun must pass through this plane to reach the observation point). The normal vector is a vector perpendicular to this plane. The normal vector is used to determine the spatial orientation of the scattering plane. Since the polarization direction of Rayleigh scattering from the sky is necessarily perpendicular to the scattering plane, the normal vector is used to derive the polarization direction.

[0055] The polarization direction of sky-scattered light must simultaneously satisfy two conditions: it must be perpendicular to the scattering plane (i.e., parallel to the normal vector) and perpendicular to the observation direction (the polarization direction lies in a plane perpendicular to the observation direction). Therefore, by taking the cross product of the scattering plane normal vector and the observation direction unit vector, a unique sky polarization direction vector can be obtained. This vector can then be converted into a unit vector (eliminating the influence of length) to obtain the standard sky polarization vector, retaining only the polarization direction information.

[0056] The standard sky polarization vector is a three-dimensional vector, which needs to be decomposed into two orthogonal directions (such as east-west for the horizontal direction and north-south for the vertical direction) on the camera imaging plane (or the local horizontal coordinate system) to obtain the horizontal component and the vertical component.

[0057] By analyzing the horizontal and vertical components of the standard sky polarization vector, the angle of the sky-scattered light polarization direction relative to the local horizontal reference can be obtained; this angle is the sky polarization orientation reference angle. Since the incident light on the road surface mainly comes from Rayleigh scattering from the sky, its polarization direction should theoretically be consistent with this reference angle. If a deviation between the road surface polarization direction and this reference angle is subsequently detected, it is determined that the deviation originates from interference such as windshield coating.

[0058] In one embodiment of the present invention, a first-band polarization image and a second-band polarization image are acquired. Based on the polarization images, the total light intensity, linear polarization degree, and polarization orientation angle of each spectral band are calculated, and the region of interest on the road surface is fused and selected, including:

[0059] The first band polarization image is a near-infrared spectral polarization image, containing pixels. Intensity in the 0-degree polarization direction Intensity in the 45-degree polarization direction Intensity in the 90-degree polarization direction Intensity in the 135-degree polarization direction ;

[0060] The second-band polarization image is a polarization image of the characteristic absorption spectrum of the water body and the reference spectrum, containing pixels. Intensity in the 0-degree polarization direction Intensity in the 45-degree polarization direction Intensity in the 90-degree polarization direction Intensity in the 135-degree polarization direction ;

[0061] in, Indicates the position in the first band polarization image and the second band polarization image. Pixels;

[0062] Calculate the pixels of the first band polarization image The polarization parameters include:

[0063] Step 1, Linear polarization degree of the first band :

[0064] ;in, The total brightness of the first band polarization image , The first band, first interval difference component , First band, second interval difference component ;

[0065] Step 2, first band polarization orientation angle :

[0066] ;

[0067] For the pixels of the first band polarization image Repeat steps 1 and 2 to obtain the corresponding linear polarization degree of the second band. Second band polarization orientation angle Total brightness of the second band polarization image The first interval difference component of the second band Second band second interval difference component ;

[0068] Determine the first lower limit threshold of brightness for the road surface area in the first-band polarized image and the second-band polarized image. Second brightness lower limit threshold ;

[0069] Determine the first linear polarization degree lower limit threshold for the road surface region in the first-band polarization image and the second-band polarization image. Second linear polarization degree lower limit threshold ;

[0070] Define the first band road surface candidate region set :

[0071] ;

[0072] Define the second-band road surface candidate region set :

[0073] ;

[0074] Pick and The intersection of these points yields the region of interest for the road surface. .

[0075] The first band polarization image uses the near-infrared spectral band, which is less affected by ambient light interference and can stably capture the basic polarization characteristics of the road surface.

[0076] Second-band polarization image: covering the characteristic absorption spectrum of water (such as the strong absorption band of water in shortwave infrared) and the reference spectrum (the band of weak absorption by water).

[0077] Both polarization images contain light intensity data for each pixel in four polarization directions (0°, 45°, 90°, and 135°). These four directions were chosen because they are orthogonal or obliquely distributed, which can completely cover all directional information of linearly polarized light.

[0078] Total brightness (total luminous intensity): It is obtained by summing the luminous intensities of the pixel in the four polarization directions and then averaging them. Averaging can reduce noise interference in a single polarization direction and ensure the reliability of the total luminous intensity data.

[0079] Two interval difference components: the first interval difference component is the light intensity difference between the 0-degree and 90-degree polarization directions, and the second interval difference component is the light intensity difference between the 45-degree and 135-degree polarization directions. These two differences reflect the polarization intensity differences in two orthogonal directions: horizontal-vertical and oblique-anti-oblique.

[0080] Linear polarization degree: Calculated by dividing the magnitude of the polarization intensity vector constructed from the difference components of two intervals by the total brightness. It reflects the proportion of linearly polarized light to the total light intensity; a larger value indicates a more significant optical polarization characteristic of the pixel (e.g., high polarization degree of specular reflection light in wet paths and low polarization degree of diffuse reflection light in dry paths).

[0081] The polarization orientation angle of the first band is calculated based on the arctangent of the difference components between two intervals, reflecting the angle of the vibration direction of linearly polarized light relative to the reference direction (such as the 0-degree polarization direction). For example, if the light intensity difference between 0 degrees and 90 degrees is the largest, and the light intensity difference between 45 degrees and 135 degrees is 0, then the polarization orientation angle is close to 0 degrees or 90 degrees, intuitively reflecting the spatial orientation of the polarization direction.

[0082] The lower limit threshold for brightness is based on the typical brightness range of the road surface area (obtained through statistical analysis of a large number of samples, excluding brightness values ​​from non-road surface areas such as shadows and dark areas). It is used to filter out pixels with sufficient light intensity: if the brightness of a pixel is lower than the threshold, it indicates that the signal quality in that area is poor (e.g., high noise ratio), and the polarization parameter calculation results are unreliable, so it needs to be excluded.

[0083] The lower limit threshold for linear polarization is based on the typical polarization range of the road surface area (especially specular reflection areas where water film may exist), excluding polarization values ​​from weakly polarized or unpolarized areas such as the sky and trees. This is used to filter out pixels with significant polarization characteristics; that is, the specular reflection light from road surface water film has a high polarization degree, while the diffuse reflection areas of the main road have a low polarization degree. Setting a lower limit allows focusing on pixels with polarization analysis value, excluding interference areas without polarization information.

[0084] The first band road surface candidate region set includes all pixels in the first band whose brightness is greater than or equal to the first brightness lower limit threshold and whose linear polarization degree is greater than or equal to the first linear polarization lower limit threshold. In other words, it is the set of pixels in this band with qualified signal quality and significant polarization characteristics.

[0085] The second band road surface candidate region set includes all pixels in the second band whose brightness is greater than or equal to the second lower limit threshold and whose linear polarization degree is greater than or equal to the second lower limit threshold. In other words, it is the set of pixels in this band with qualified signal quality and significant polarization characteristics.

[0086] The region of interest for the road surface must simultaneously satisfy the conditions of reliable signal in both spectral bands and polarization characteristics: only the intersection of the two candidate regions can be taken to ensure that the region has effective data in both spectral bands.

[0087] In one embodiment of the present invention, the polarization orientation angles of the first-band polarization image and the second-band polarization image within the region of interest of the road surface are extracted, respectively. Based on these values, the spectral polarization orientation split is calculated, and the sign parameter is determined in conjunction with a sky polarization orientation reference, including:

[0088] Extracting the region of interest on the road surface The median value among all the first-band polarization orientation angles corresponding to the first-band image is used as the characterization value of the first band. ;

[0089] Extracting the region of interest on the road surface The median value among all second-band polarization orientation angles corresponding to the second-band image is used as the characterization value of the second band. ;

[0090] Using the angle principal value function ;in, For input variables, The principal polarization orientation angle difference is calculated.

[0091] Taking the absolute value of the principal polarization orientation angle difference yields the spectral polarization orientation splitting factor. ;

[0092] Calculate symbolic parameters :

[0093] ;in, For a symbolic function: when the independent variable of the symbolic function is positive, the symbolic parameter is 1; when the independent variable of the symbolic function is negative, the symbolic parameter is -1; when the independent variable of the symbolic function is 0, the symbolic parameter is 0.

[0094] The polarization orientation angle exhibits periodicity (e.g., when the angle difference is 180 degrees, the actual polarization direction is equivalent; when the angle difference is 190 degrees, the actual minimum difference is 10 degrees). Directly calculating the difference between the two band values ​​can easily lead to calculation errors due to periodicity (e.g., mistakenly calculating the difference between 10 degrees and 190 degrees as 180 degrees, when the actual minimum difference is 10 degrees). The principal value function of the angle eliminates the interference of angle periodicity: through calculation, it maps the difference between the two band values ​​(i.e., the input variable) to the minimum angle difference interval (from -90 degrees to +90 degrees), ensuring that the obtained principal value polarization orientation angle difference is the actual minimum difference in polarization orientation between the two bands, thus avoiding calculation errors caused by periodicity.

[0095] The spectral polarization orientation splitting factor is a parameter obtained by taking the absolute value of the principal-valued polarization orientation angle difference, and is used to quantify the degree of splitting of the dual-band polarization orientation:

[0096] The larger the spectral polarization orientation split, the greater the difference in the polarization orientation observation results of the first band and the second band for the same road surface area, that is, the more significant the cross-band polarization deviation caused by multi-layer coating; the smaller the value, the more consistent the polarization orientation of the two bands, and the slighter the deviation.

[0097] The sign parameter is obtained by combining the difference between the two band characterization values ​​and the sky polarization orientation reference with the sign function. It is used to determine the direction of the polarization parameter's reverse rotation. The specific logic is as follows:

[0098] The difference between the characterization value of the first band and the sky polarization orientation reference reflects the offset direction of the first band's polarization orientation relative to the absolute reference; the difference between the characterization value of the second band and the sky polarization orientation reference reflects the offset direction of the second band; the difference between the two can determine which band is offset further forward and which is offset further back relative to the sky reference, thus determining the direction relationship of the split; the sign function outputs 1, -1, or 0 according to the positive or negative of the above difference, corresponding to three rotation direction instructions respectively. When outputting 1, the first band needs to rotate in the opposite direction to the sign, and the second band rotates in the same direction; when outputting -1, the rotation directions are opposite; when outputting 0, the offsets of the two bands relative to the sky reference are consistent, and no rotation is required;

[0099] In one embodiment of the present invention, within a preset polarization parameter plane, the first band polarization image and the second band polarization image are rotated in opposite directions by an equal amount according to the spectral polarization orientation split and in combination with the sign parameter to obtain the rotational linear polarization degree of the second band polarization image, including:

[0100] The preset polarization parameter plane is constructed with polarization difference components as the horizontal axis and polarization difference components as the vertical axis.

[0101] Based on the spectral polarization orientation splitting amount With symbolic parameters Calculate the rotation angle, including:

[0102] First band rotation angle ;in, This indicates the rotation angle of the first band polarization parameter within the preset polarization parameter plane; the negative sign indicates the direction of rotation and the sign of the parameter. Pointing to the opposite;

[0103] Second band rotation angle ;in, This indicates the rotation angle of the second band polarization parameter within the preset polarization parameter plane; the plus sign indicates the direction of rotation and the sign parameter. Pointing to the same thing;

[0104] Step 3: Within the preset polarization parameter plane, rotate the first interval difference component and the second interval difference component of the first band using a rotation matrix while maintaining the total light intensity unchanged. This includes:

[0105] ;

[0106] ;

[0107] in, This represents the rotational difference component of the first band and first interval. This represents the rotational difference component of the second interval of the first band. This represents the total rotational light intensity of the first band;

[0108] Repeat step 3 for the difference component of the first interval of the second band and the difference component of the second interval of the second band to obtain the corresponding rotational difference component of the first interval of the second band. Second band, second interval rotational difference component Second band rotation total light intensity ;

[0109] Calculate the rotational linear polarization degree corresponding to the polarization image of the second band. .

[0110] The horizontal axis of the preset polarization parameter plane represents the first interval difference component (the light intensity difference between the 0° and 90° polarization directions), and the vertical axis represents the second interval difference component (the light intensity difference between the 45° and 135° polarization directions). These two difference components can completely describe the linear polarization state of light, meaning that each point in the plane corresponds to a unique polarization direction and polarization intensity.

[0111] Dual-band rotation angle, including:

[0112] Equal scalar rotation: The rotation angle is related to the amount of spectral polarization orientation split (half of the split amount), the purpose of which is to align the two bands from the split state towards the center. For example, when the split amount is a certain angle, rotating each band by half of that angle can eliminate the split and ensure that the polarization orientation of the two bands is consistent after correction.

[0113] Reverse rotation: The first band's rotation angle is negative, indicating that its rotation direction is opposite to the sign parameter; the second band's rotation angle is positive, indicating that its rotation direction is the same as the sign parameter. Reverse rotation is to ensure that the two bands rotate towards each other around a common reference (sky polarization orientation reference), rather than shifting in the same direction. This ensures that after rotation, both bands are aligned with the absolute reference, rather than merely eliminating relative splitting while still maintaining overall offset.

[0114] Rotation matrix: In the plane of preset polarization parameters, the two interval difference components of the first band are geometrically rotated by the rotation matrix. Essentially, it corrects the deviation of the polarization direction of the band. That is, the first interval rotation difference component and the second interval rotation difference component after rotation correspond to the difference components of the corrected polarization state.

[0115] The total light intensity remains unchanged: The total light intensity is the overall energy of light and is independent of the polarization direction (polarization only changes the direction of light vibration, not the magnitude of energy). Therefore, the rotation operation only corrects the difference component related to the polarization direction. The total light intensity of the first band after rotation is completely consistent with the total light intensity before rotation.

[0116] After correcting for the polarization direction deviation caused by the coating, the true polarization intensity ratio of the road surface area under the second band is reflected.

[0117] In one embodiment of the present invention, based on the total light intensity and rotational linear polarization degree of the second-band polarized image, the depolarization intensity of the water body characteristic absorption spectrum and the reference spectrum of the second-band polarized image is calculated, and the water body absorption optical thickness is determined by the depolarization intensity, including:

[0118] Rotate the total light intensity of the second band The total light intensity of the water body characteristic absorption spectrum bands in the second band polarization image are respectively used as the total light intensity. Total light intensity of the reference spectral band of the second-band polarization image ;

[0119] Rotate the second band polarization image by linear polarization degree The degree of linear polarization of the characteristic absorption spectrum of water bodies is rotated respectively. Rotational linear polarization degree of the reference spectrum ;

[0120] Calculate the depolarization intensity of the characteristic absorption spectrum of water in the second band. ;

[0121] Calculate the depolarization intensity of the second-band reference spectrum. ;

[0122] Calculate the optical thickness of water absorption .

[0123] The second band itself includes water-characteristic absorption spectrum (such as the spectrum near 1450nm for strong water absorption) and reference spectrum (such as the spectrum near 1650nm for weak water absorption). Both belong to the second band. The total light intensity of the second band obtained by the rotation inversion module (after correcting for polarization deviation caused by coating) can reflect the unified basic characteristics of light intensity in this band. Therefore, it is used as the total light intensity of these two sub-spectral bands respectively.

[0124] The rotation inversion module has eliminated the transspectral polarization bias in the second band, and its output rotational linear polarization degree in the second band accurately reflects the corrected state of polarization characteristics within this band. Since the two sub-bands belong to the same second band, their polarization characteristics are affected by rotational correction in the same way. Therefore, this rotational linear polarization degree is used as the rotational linear polarization degree of the water body characteristic absorption spectrum and the rotational linear polarization degree of the reference spectrum, respectively.

[0125] Specular reflection from road surfaces (especially wet roads) generates strong polarized signals, interfering with the accurate acquisition of water absorption signals. The physical meaning of depolarization intensity is the light intensity after removing specular polarization components, calculated as total light intensity of the sub-spectral band × (1 - sub-spectral band rotational linear polarization degree). By removing the proportion of polarization components using 1 - rotational linear polarization degree, and then combining it with the total light intensity, we obtain the light intensity that only reflects the contributions of diffuse reflection and water absorption. The depolarization intensity of the water characteristic absorption spectrum and the reference spectrum are calculated separately.

[0126] Water exhibits strong absorption in its characteristic absorption spectrum and weak absorption in a reference spectrum (which can be considered a reference with no absorption). Based on the Lambert-Beer law, the degree of water absorption is related to the ratio of the depolarization intensities of the two sub-spectral bands: the stronger the absorption, the smaller the ratio of the depolarization intensities of the characteristic absorption spectrum to the reference spectrum. By taking the negative logarithm of this ratio, the absorption optical thickness of the water body is obtained. This thickness monotonically increases with the thickness of the water film on the road surface, allowing direct quantification of the actual water content on the road surface.

[0127] In one embodiment of the present invention, an adaptive scaling coefficient is constructed based on the spectral polarization orientation split, and the water absorption optical thickness is calibrated using the adaptive scaling coefficient to obtain a corrected optical thickness, including:

[0128] Within the historical time period, there are M sub-time periods. The spectral polarization orientation splitting amount is obtained for each sub-time period, and the nominal splitting amount is obtained by averaging the spectral polarization orientation splitting amounts of the M sub-time periods. ;

[0129] The physical calibration constant is set by the water absorption optical thickness corresponding to the standard wet target surface. ;

[0130] Based on the spectral polarization orientation splitting factor and the nominal splitting factor, an adaptive scaling coefficient is constructed. ;in, for ;

[0131] The optical thickness of water absorption is calibrated based on adaptive scaling coefficients and physical calibration constants:

[0132] ;in, This indicates a correction for optical thickness.

[0133] The spectral polarization orientation splitting amount changes dynamically with the scene, so it is necessary to first determine the baseline splitting level under typical scenarios, i.e., the nominal splitting amount. By dividing the historical time period into multiple sub-time periods (M), the spectral polarization orientation splitting amount of each sub-time period is collected, and then the average of these splitting amounts is taken as the nominal splitting amount. This average value can reflect the splitting properties of the vehicle-windshield-sensor combination under normal conditions.

[0134] The physical calibration constant is the benchmark for the calibration process and must be determined based on a standard target surface in a known state. That is, a standard wet target surface is selected (its water film thickness is known, and the corresponding water absorption optical thickness can also be accurately measured). The physical calibration constant is set in reverse by using the water absorption optical thickness of the standard target surface.

[0135] The adaptive scaling coefficient reflects the dynamic adjustment of the current spectral polarization orientation splitting amount from the nominal splitting amount: the current splitting amount is used as the numerator, and the nominal splitting amount plus a minimum value (to avoid the denominator being zero and causing calculation failure) is used as the denominator to obtain the scaling coefficient. The greater the deviation of the current splitting amount from the nominal splitting amount, the larger the scaling coefficient, and vice versa, thus quantifying the strength of the coating dispersion residual under the current scenario.

[0136] The optical thickness of water absorption is affected by the dispersion residual of the coating (the greater the splitting, the stronger the residual, and the greater the optical thickness deviation). By combining the physical calibration constant with the adaptive scaling coefficient, a calibration formula is constructed to adjust the original optical thickness of water absorption. That is, the stronger the residual (the larger the scaling coefficient), the larger the calibration amplitude, thereby offsetting the influence of the residual and obtaining the corrected optical thickness. This corrected value can truly reflect the optical properties of the water film on the road surface.

[0137] In one embodiment of the present invention, mapping the corrected optical thickness to the corresponding road surface state includes:

[0138] The first corrected optical thickness and the second corrected optical thickness corresponding to the standard wet target surface and the standard dry target surface are determined respectively, and calibrated as the first threshold and the second threshold.

[0139] If the corrected optical thickness is less than or equal to the second threshold, the road surface condition is dry.

[0140] If the corrected optical thickness is greater than or equal to the first threshold, the road surface state is a water film state;

[0141] If the corrected optical thickness is between the first threshold and the second threshold, the road surface state is a thin film state.

[0142] The magnitude of the corrected optical thickness is monotonically increasing with the thickness of the water film on the road surface (the thicker the water film, the greater the corrected optical thickness). It is necessary to first determine the boundary threshold using a standard target surface in a known state: select a standard dry target surface (simulating a completely dry road surface without moisture) and measure its corresponding corrected optical thickness as the second threshold (the boundary between dry and non-dry states); select a standard wet target surface (simulating a wet road surface covered with a continuous water film) and measure its corresponding corrected optical thickness as the first threshold (the boundary between water film state and non-water film state).

[0143] If the corrected optical thickness of the current road surface is less than or equal to the second threshold, it indicates that its moisture content is comparable to or lower than that of the standard dry target surface (no obvious water film), which is consistent with the properties of a dry road surface and is therefore determined to be dry.

[0144] If the corrected optical thickness of the current road surface is greater than or equal to the first threshold, it indicates that its moisture content is comparable to or higher than that of the standard wet target surface (a continuous water film has been formed), which is consistent with the properties of a water film road surface, and therefore it is determined to be a water film surface.

[0145] If the corrected optical thickness of the current road surface is between the first threshold and the second threshold, it indicates that its moisture content is between the dry state and the continuous water film state (only a very thin water film that does not form a continuous layer exists), which is consistent with the properties of a thin film road surface, so it is determined to be a thin film state.

[0146] The embodiments of this example have been described above. However, this example is not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms based on the guidance of this example, and all of them are within the protection scope of this example.

Claims

1. An intelligent road surface recognition system based on multimodal sensor data fusion, characterized in that, include: The solution module calculates solar geometry to obtain a sky polarization orientation reference. The region filtering module acquires the first-band polarization image and the second-band polarization image, calculates the total light intensity, linear polarization degree and polarization orientation angle of each spectral band based on the polarization images, and fuses and filters the region of interest on the road surface. The data extraction module extracts the polarization orientation angle characterization values ​​of the first-band polarization image and the second-band polarization image within the region of interest on the road surface, calculates the spectral polarization orientation split based on the characterization values, and determines the sign parameter in conjunction with the sky polarization orientation reference. The rotation inversion module rotates the first band polarization image and the second band polarization image in the same reverse order within the preset polarization parameter plane according to the spectral polarization orientation split and the sign parameter, to obtain the rotational linear polarization degree of the second band polarization image. The depolarization module calculates the depolarization intensity of the water body characteristic absorption spectrum and the reference spectrum based on the total light intensity and the degree of rotational linear polarization of the second-band polarization image, and determines the water body absorption optical thickness based on the depolarization intensity. The correction module constructs an adaptive scaling coefficient based on the spectral polarization orientation split, and uses the adaptive scaling coefficient to calibrate the water absorption optical thickness to obtain the corrected optical thickness. The road surface condition determination module maps the corrected optical thickness to the corresponding road surface condition; the road surface condition includes: dry state, thin film state and water film state.

2. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 1, characterized in that, Solving for solar geometry yields a sky polarization orientation reference, including: Get geographic latitude Geographical longitude World Time Day number Camera principal ray elevation angle Camera principal ray azimuth angle ; Calculate the time equation : ； Calculate the angle ; Calculate the solar declination angle ; Calculate the sine component of the solar altitude angle The sine component of the solar azimuth angle Sum and cosine components : ； ； ； Construct the unit vector of the observation direction Unit vector in the direction of the sun : ； ； Unit vector of the observation direction Unit vector in the direction of the sun The cross product of the scattering plane is used as the normal vector. , scattering plane normal vector and observation direction unit vector The cross product of the vectors is used as the unit vector for the polarization direction of the sky. ; unit vector of sky polarization direction unit vector of sky polarization direction The standard sky polarization vector is obtained by normalizing the vector magnitude. ; Determine the standard sky polarization vector lateral component and longitudinal components ; Calculate the sky polarization orientation reference angle .

3. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 2, characterized in that, Acquire polarization images of the first and second bands, calculate the total light intensity, linear polarization degree, and polarization orientation angle of each spectral band based on the polarization images, and fuse and filter regions of interest on the road surface, including: The first band polarization image is a near-infrared spectral polarization image, containing pixels. Intensity in the 0-degree polarization direction Intensity in the 45-degree polarization direction Intensity in the 90-degree polarization direction Intensity in the 135-degree polarization direction ; The second-band polarization image is a polarization image of the characteristic absorption spectrum of the water body and the reference spectrum, containing pixels. Intensity in the 0-degree polarization direction Intensity in the 45-degree polarization direction Intensity in the 90-degree polarization direction Intensity in the 135-degree polarization direction ; in, Indicates the position in the first band polarization image and the second band polarization image. Pixels; Calculate the pixels of the first band polarization image The polarization parameters include: Step 1, Linear polarization degree of the first band : ;in, The total brightness of the first band polarization image , The first band, first interval difference component , First band, second interval difference component ; Step 2, first band polarization orientation angle : ； For the pixels of the first band polarization image Repeat steps 1 and 2 to obtain the corresponding linear polarization degree of the second band. Second band polarization orientation angle Total brightness of the second band polarization image The first interval difference component of the second band Second band second interval difference component ; Determine the first lower limit threshold of brightness for the road surface area in the first-band polarized image and the second-band polarized image. Second brightness lower limit threshold ; Determine the first linear polarization degree lower limit threshold for the road surface region in the first-band polarization image and the second-band polarization image. Second linear polarization degree lower limit threshold ; Define the first band road surface candidate region set : ； Define the second-band road surface candidate region set : ； Pick and The intersection of these points yields the region of interest for the road surface. .

4. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 3, characterized in that, The polarization orientation angles of the first-band and second-band polarization images within the region of interest on the road surface are extracted, respectively. Based on these values, the spectral polarization orientation split is calculated, and the sign parameter is determined using a sky polarization orientation reference, including: Extracting the region of interest on the road surface The median value among all the first-band polarization orientation angles corresponding to the first-band image is used as the characterization value of the first band. ; Extracting the region of interest on the road surface The median value among all second-band polarization orientation angles corresponding to the second-band image is used as the characterization value of the second band. ; Using the angle principal value function ;in, For input variables, The principal polarization orientation angle difference is calculated. Taking the absolute value of the principal polarization orientation angle difference yields the spectral polarization orientation splitting factor. ; Calculate symbolic parameters : ;in, For a symbolic function: when the independent variable of the symbolic function is positive, the symbolic parameter is 1; when the independent variable of the symbolic function is negative, the symbolic parameter is -1; when the independent variable of the symbolic function is 0, the symbolic parameter is 0.

5. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 4, characterized in that, Within a preset polarization parameter plane, the first-band polarization image and the second-band polarization image are rotated in opposite directions by equal amounts based on the spectral polarization orientation split and the sign parameter, resulting in the rotational linear polarization degree of the second-band polarization image, including: The preset polarization parameter plane is constructed with polarization difference components as the horizontal axis and polarization difference components as the vertical axis. Based on the spectral polarization orientation splitting amount With symbolic parameters Calculate the rotation angle, including: First band rotation angle ;in, This indicates the rotation angle of the first band polarization parameter within the preset polarization parameter plane; the negative sign indicates the direction of rotation and the sign of the parameter. Pointing to the opposite; Second band rotation angle ;in, This indicates the rotation angle of the second band polarization parameter within the preset polarization parameter plane; the plus sign indicates the direction of rotation and the sign parameter. Pointing to the same thing; Step 3: Within the preset polarization parameter plane, rotate the first interval difference component and the second interval difference component of the first band using a rotation matrix while maintaining the total light intensity unchanged. This includes: ； ； in, This represents the rotational difference component of the first band and first interval. This represents the rotational difference component of the second interval of the first band. This represents the total rotational light intensity of the first band; Repeat step 3 for the difference component of the first interval of the second band and the difference component of the second interval of the second band to obtain the corresponding rotational difference component of the first interval of the second band. Second band, second interval rotational difference component Second band rotation total light intensity ; Calculate the rotational linear polarization degree corresponding to the polarization image of the second band. .

6. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 5, characterized in that, Based on the total light intensity and rotational linear polarization degree of the second-band polarization image, the depolarization intensity of the water body's characteristic absorption spectrum and the reference spectrum in the second-band polarization image is calculated. The water body's absorption optical thickness is determined using the depolarization intensity, including: Rotate the total light intensity of the second band The total light intensity of the water body characteristic absorption spectrum bands in the second band polarization image are respectively used as the total light intensity. Total light intensity of the reference spectral band of the second-band polarization image ; Rotate the second band polarization image by linear polarization degree The degree of linear polarization of the characteristic absorption spectrum of water bodies is rotated respectively. Rotational linear polarization degree of the reference spectrum ; Calculate the depolarization intensity of the characteristic absorption spectrum of water in the second band. ; Calculate the depolarization intensity of the second-band reference spectrum. ; Calculate the optical thickness of water absorption .

7. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 6, characterized in that, An adaptive scaling coefficient is constructed based on the spectral polarization orientation split, and the water absorption optical thickness is calibrated using the adaptive scaling coefficient to obtain the corrected optical thickness, including: Within the historical time period, there are M sub-time periods. The spectral polarization orientation splitting amount is obtained for each sub-time period, and the nominal splitting amount is obtained by averaging the spectral polarization orientation splitting amounts of the M sub-time periods. ; The physical calibration constant is set by the water absorption optical thickness corresponding to the standard wet target surface. ; Based on the spectral polarization orientation splitting factor and the nominal splitting factor, an adaptive scaling coefficient is constructed. ;in, for ; The optical thickness of water absorption is calibrated based on adaptive scaling coefficients and physical calibration constants: ;in, This indicates a correction for optical thickness.

8. The intelligent road surface recognition system based on multimodal sensor data fusion according to claim 7, characterized in that, Mapping the corrected optical thickness to the corresponding road surface condition includes: The first corrected optical thickness and the second corrected optical thickness corresponding to the standard wet target surface and the standard dry target surface are determined respectively, and calibrated as the first threshold and the second threshold. If the corrected optical thickness is less than or equal to the second threshold, the road surface condition is dry. If the corrected optical thickness is greater than or equal to the first threshold, the road surface state is a water film state; If the corrected optical thickness is between the first threshold and the second threshold, the road surface state is a thin film state.

Images