Road surface sensing method and equipment based on multi-modal data, and medium

Abstract

The embodiment of the invention discloses a road surface sensing method and device based on multi-modal data and a medium, and relates to the technical field of road surface sensing, and the method comprises the steps: obtaining real-time curve road parameters of a curve region through a curve edge node corresponding to the curve region in a target road, and obtaining a target road surface based on the real-time curve road parameters; road curve feature analysis is carried out, and curve deployment information of each curve area is determined; according to the curve deployment information, performing multi-stage trigger sensing on the curve area, and determining multi-modal road abnormal data corresponding to the curve area, including primary vibration sensing, second-stage radar focusing sensing and third-stage optical verification sensing; according to pre-acquired real-time meteorological data, carrying out multi-modal cooperative verification on the multi-modal road abnormal data, and determining a region abnormal confidence coefficient corresponding to the curve region; and carrying out multi-modal feature fusion on the multi-modal road abnormal data based on the region abnormal confidence coefficient corresponding to the curve region so as to determine abnormal risk information corresponding to the curve region.

Description

Technical Field

[0001] The present specification relates to the field of road surface perception technology, and in particular to a road surface perception method, device and medium based on multimodal data. Background Art

[0002] In the field of road management, real-time and accurate road condition monitoring is the core requirement for improving traffic safety and traffic efficiency. Among them, road waterlogging is a high-risk abnormal scenario that can easily cause accidents such as vehicle skidding and brake failure, especially in curved areas. A curve may consist of multiple arc segments with different curvature radii. Due to the centrifugal force, the water accumulation is unevenly distributed, further magnifying the safety hazard.

[0003] In the current road surface perception method, image recognition technology is used, which relies on the reflective characteristics of accumulated water. However, due to problems such as curved surface reflection and light refraction in curved areas, metal signs or wet roads may be misjudged as accumulated water. The indiscriminate collection method is adopted in curved areas and non-curved areas, which adds an additional burden of data processing, resulting in the perception efficiency of the target road to be improved. In addition, in order to solve the technical problem of one-sided perception results produced by a single type of sensor perception, there is an existing technology that fuses multiple modal data. However, in the current multi-sensor fusion perception method, multiple sensors collect synchronously at all times, and the perception advantages of different types of sensors are not considered, which increases the power consumption of the sensor equipment. Therefore, in the traditional road surface water perception method, the particularity of the curved area and the perception advantages of different types of perception are not considered, the perception efficiency needs to be improved, and there is a problem of delayed warning of water accumulation risk, which increases the hidden danger of traffic safety. Summary of the invention

[0004] One or more embodiments of the present specification provide a road surface perception method, device and medium based on multimodal data, which are used to solve the following technical problems: In the traditional road surface waterlogging perception method, the particularity of the curved area and the perception advantages of different types of perception are not taken into account, the perception efficiency needs to be improved, and there is a problem of delayed warning of waterlogging risk, which increases the potential safety hazard of traffic.

[0005] One or more embodiments of this specification adopt the following technical solutions: One or more embodiments of the present specification provide a road surface perception method based on multimodal data, the method comprising: obtaining real-time curve road parameters of the curve area through curve edge nodes corresponding to the curve area in the target road, performing road curve feature analysis based on the real-time curve road parameters, and determining curve deployment information of each curve area, wherein the curve deployment information includes a deployment working mode corresponding to the multimodal sensor; performing multi-level trigger perception on the curve area through the curve deployment information to determine multi-modal road abnormality data corresponding to the curve area, wherein the multi-level trigger perception includes primary vibration perception, secondary radar focus perception and tertiary optical verification perception; performing multi-modal collaborative verification on the multi-modal road abnormality data according to pre-acquired real-time meteorological data to determine the regional abnormality confidence corresponding to the curve area; performing multi-modal feature fusion on the multi-modal road abnormality data based on the regional abnormality confidence corresponding to the curve area to determine the abnormal risk information corresponding to the curve area.

[0006] One or more embodiments of this specification provide a road surface perception device based on multimodal data, including: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0007] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0008] At least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: obtaining real-time curve road parameters through curve edge nodes, analyzing road curve characteristics, determining curve deployment information, and taking corresponding configuration parameter settings according to the particularities of the curve area, such as the composition of arc segments with different curvature radii, uneven distribution of water accumulation due to centrifugal force, etc., thereby avoiding misjudgment caused by problems such as reflection of curve surfaces and refraction of light in traditional methods, and improving the source accuracy of road condition perception data in curve areas; changing the indiscriminate collection method in curve areas and non-curve areas, and performing targeted multi-level trigger perception of curve areas according to curve deployment information, thereby avoiding unnecessary data collection and reducing the data processing burden, thereby improving the perception efficiency of target roads and being able to discover curves faster. The abnormal road conditions in the area have been detected, overcoming the problem of low perception efficiency in traditional methods; the perception advantages of different types of sensors have been taken into consideration, such as the multi-level trigger perception method of primary vibration perception, secondary radar focus perception and tertiary optical verification perception, which avoids the synchronous collection of multiple sensors at all times and reduces the power consumption of sensor equipment. While ensuring the perception effect, it improves energy utilization efficiency; multi-modal collaborative verification of multi-modal road abnormality data is carried out according to real-time meteorological data to determine the regional abnormality confidence, and then multi-modal feature fusion is performed to determine the abnormal risk information. The multi-modal data fusion and collaborative verification method makes full use of the data of multiple sensors, which can judge the abnormal risk of the curve area more comprehensively and accurately, improve the reliability of risk judgment, reduce the problem of delayed warning of water accumulation risk, and reduce the hidden dangers of traffic safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings: Figure 1 A schematic diagram of a flow chart of a road surface perception method based on multimodal data provided in an embodiment of this specification; Figure 2 A schematic diagram of the structure of a road surface perception device based on multimodal data provided in an embodiment of this specification. DETAILED DESCRIPTION

[0010] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0011] The embodiments of this specification provide a road surface perception method based on multimodal data. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A schematic diagram of a road surface perception method based on multimodal data provided in an embodiment of this specification, such as Figure 1 As shown, it mainly includes the following steps: Step S101, obtaining real-time curve road parameters of the curve area through the curve edge nodes corresponding to the curve area in the target road, performing road curve feature analysis based on the real-time curve road parameters, and determining the curve deployment information of each curve area.

[0012] The curve deployment information includes a deployment working mode corresponding to the multimodal sensor; In one embodiment of the present specification, in order to achieve road surface perception of the target road, curve edge nodes are pre-set in multiple curve areas of the target road, and road surface perception in the corresponding curve area is achieved through the curve edge nodes. For a target road with a long length, when perceiving the road surface in this road, the amount of perception data is large. The traditional centralized perception processing method increases the data processing burden of the central processing equipment, and the real-time perception cannot be guaranteed. To solve this problem, the embodiment of the present specification sets a curve edge node in the corresponding curve area. This curve edge node is only used to obtain the sensor data in this area for real-time perception, which shares the data processing pressure, thereby improving the perception efficiency of the curve area and ensuring the real-time perception. The real-time curve road parameters of the curve area are obtained through the curve edge node corresponding to the curve area. The data collected by each curve edge node is only the data in this curve area, which reduces the amount of data at the data acquisition level.

[0013] Based on the real-time curve road parameters, a road curve feature analysis is performed to determine the curve deployment information of each curve area, specifically including: obtaining real-time road data and curve geometry feature data in the real-time curve road parameters, wherein the real-time road data includes real-time traffic flow and real-time average speed data, and the curve geometry feature data includes curve curvature parameters corresponding to the curve area; through the curve curvature parameters corresponding to the curve area and the real-time road data, setting acquisition parameters for a pre-set ground vibration sensor to determine the real-time curve deployment information, wherein the real-time curve deployment information includes an effective monitoring radius and a sampling frequency.

[0014] In one embodiment of the present specification, the vehicle flow (vehicles / minute) and average vehicle speed (km / h) are collected in real time through roadside cameras, geomagnetic sensors or vehicle-mounted terminals, and the curvature parameters of the curves of the target road during the road construction stage are obtained. In general road perception systems, sensor equipment is pre-arranged in the early stage, but the arrangement is usually uniform, that is, both the straight area and the curve area are arranged with a fixed spacing, for example, a piezoelectric vibration sensor is deployed every 5 meters along the edge of the road to form a grid monitoring array. In this scenario, it is difficult to adjust the physical position parameters of the uniformly arranged sensor network, and a one-size-fits-all parameter setting method is usually adopted, ignoring the particularity of the curve area. Therefore, in the embodiment of the present specification, without changing the physical parameters such as the position of the sensor network in the road, through spatial adaptive parameter mapping, the fixedly deployed sensor network can dynamically configure parameter deployment for the curve geometry characteristics and real-time traffic information of the curve area to meet the curve structure of the curve area and the road perception requirements of the real-time traffic status.

[0015] The acquisition parameters of the pre-set ground vibration sensor are set through the curve curvature parameters corresponding to the curve area and the real-time road data to determine the real-time curve deployment information, specifically including: determining the curvature adjustment factor corresponding to the curve area through the curve curvature parameters corresponding to the curve area and the reference curvature parameters corresponding to the pre-set reference spacing, and correcting the preset reference monitoring radius based on the curvature adjustment factor to determine the effective monitoring radius corresponding to the curve area; determining the traffic flow level corresponding to the real-time traffic flow according to the real-time road data to match the reference frequency under the traffic flow level; and performing vehicle speed compensation on the reference frequency based on the real-time average vehicle speed to determine the sampling frequency corresponding to the ground vibration sensor in the curve area.

[0016] In one embodiment of the present specification, the deployment parameters of the ground vibration sensor are dynamically optimized by integrating the curve curvature parameters and the real-time traffic data. First, the curvature adjustment factor is calculated based on the ratio of the actual curve curvature to the preset reference curvature, and the reference monitoring radius is dynamically compressed to reduce the effective monitoring radius of the sharp curve area to the high-risk area (such as the outer lane). It should be noted that the preset reference curvature here refers to the curvature parameter matched by the sensor at the corresponding reference spacing during the layout and installation stage. This curvature radius is used as the reference curvature, and the curvature adjustment factor is calculated by the ratio of the actual curve area of ​​the target curve to the reference curvature. The reference monitoring radius under the reference curvature is corrected. The reference monitoring radius can be understood as the monitoring radius corresponding to the unified sensor configuration parameters set in advance for the entire road. The dynamic compression of the reference monitoring radius is achieved by multiplying the curvature adjustment factor and the reference monitoring radius to obtain the effective monitoring radius corresponding to the curve area.

[0017] According to the real-time traffic flow classification, the benchmark sampling frequency is matched, and then the frequency compensation is performed in combination with the average vehicle speed, so that the higher the vehicle speed, the denser the sampling interval. According to the real-time traffic flow in the real-time road data, the traffic flow level corresponding to the real-time traffic flow is determined, and the benchmark frequency under the traffic flow level is matched in combination with the traffic flow level. Based on the real-time average vehicle speed, the speed compensation coefficient is calculated, and the speed compensation is performed on the benchmark frequency by multiplying the speed compensation coefficient and the benchmark frequency to determine the sampling frequency corresponding to the ground vibration sensor in the curve area. The following table is an example table of a traffic flow level, a reference frequency, and a vehicle speed compensation coefficient provided in an embodiment of this specification. As shown in Table 1, when the traffic flow level is <30 vehicles / minute, the reference frequency is 1kHz, and the vehicle speed compensation coefficient is 1.0. At this time, the corresponding sampling frequency is 1 kHz; when the traffic flow level corresponds to a traffic flow of 30-100 vehicles / minute, the reference frequency is 2kHz, and the vehicle speed compensation coefficient is 1+0.1(v / 80). The sampling frequency is obtained by multiplying the vehicle speed compensation coefficient and the reference frequency; when the traffic flow level corresponds to a traffic flow of >100 vehicles / minute, the reference frequency is 5kHz, and the vehicle speed compensation coefficient is , then the sampling frequency obtained is the product of the vehicle speed compensation coefficient corresponding to the vehicle speed at this time and the reference frequency.

[0018] Table 1 Example table of traffic flow level, reference frequency and vehicle speed compensation coefficient

[0019] In the above embodiments of this specification, by focusing on high-risk area perception resources in the spatial dimension and matching traffic state changes in the temporal dimension, the high incidence of local waterlogging caused by the centrifugal effect of curves can be solved, and data redundancy uniformly deployed on the entire road section can be avoided. The curve detection resolution can be increased by more than 3 times, and the amount of invalid data can be reduced by 70% in a scenario with a traffic volume of 200 vehicles / hour. At the same time, the speed adaptation mechanism ensures that the vibration waveform is fully captured when a vehicle passes at 100km / h, significantly improving the real-time and accuracy of road waterlogging identification in bad weather. It is suitable for complex curves with a curvature radius of less than 300 meters and large traffic speed fluctuations, such as urban interchange ramps and sharp turns on mountain highways, providing a high-precision, low-energy curve safety monitoring solution for intelligent transportation systems.

[0020] Step S102: Perform multi-level triggering perception on the curve area through the curve deployment information to determine the multi-modal road abnormality data corresponding to the curve area.

[0021] Among them, the multi-level trigger perception includes primary vibration perception, secondary radar focus perception and tertiary optical verification perception; In one embodiment of the present specification, the following key parameters of the vibration sensor are adjusted through the curve deployment information to achieve dynamic control of the monitoring radius and sampling frequency. When adjusting the monitoring radius, the effective monitoring radius can be expanded by lowering the sensor signal trigger threshold. Conversely, increasing the threshold can narrow the monitoring range and focus on high-risk areas. In addition, the frequency domain filtering range can be adjusted, and the passband frequency of the sensor filter can be adjusted to suppress low-frequency environmental noise, indirectly narrowing the effective monitoring radius to the high-frequency vibration characteristic area, such as the near-field water impact signal. When adjusting the sampling frequency, high-frequency signal acquisition can be achieved by directly adjusting the sampling rate of the analog-to-digital converter; at a fixed sampling rate, the effective sampling frequency can also be equivalently improved by shortening the data acquisition time window, but the signal splicing algorithm needs to be optimized to prevent waveform breakage.

[0022] Through the curve deployment information, multi-level trigger perception is performed on the curve area to determine the multi-modal road abnormality data corresponding to the curve area, specifically including: through the curve deployment information, real-time curve vibration data collected by multiple ground vibration sensors are collected to determine the time-space correlation trigger matrix according to the real-time curve vibration data and the sensor layout information of the ground vibration sensor; when a first preset number of adjacent sensors in the time-space correlation trigger matrix simultaneously detect an abnormal vibration spectrum within a second preset time length, the abnormal vibration time-space information corresponding to the abnormal vibration spectrum is determined, and a first trigger instruction corresponding to the secondary radar focus perception is generated; under the triggering of the first trigger instruction, a priority scanning area is determined according to the abnormal vibration time-space information, and a radar focus scan is performed on the priority scanning area to determine the real-time scanning data in the priority scanning area; The reflection intensity gradient of the scanned data is calculated. When there is an abnormal reflection intensity gradient in the priority scanning area, the second trigger instruction corresponding to the third-level optical verification perception is triggered, and the abnormal area point cloud coordinate data corresponding to the abnormal reflection intensity gradient is determined; under the triggering of the second trigger instruction, the optical sensing device is subjected to digital zoom lock operation and strobe fill light operation to determine the abnormal optical feature of the optical sensing device and the abnormal optical feature coordinate data corresponding to the abnormal optical feature; the multimodal abnormal feature information is determined through the abnormal vibration spectrum, the abnormal reflection intensity gradient and the abnormal optical feature, and the multimodal abnormal position data is determined according to the abnormal vibration time-space information, the abnormal area point cloud coordinate data and the abnormal optical feature coordinate data; based on the multimodal abnormal feature information and the multimodal abnormal position data, the multimodal road abnormal data is determined.

[0023] In one embodiment of the present specification, abnormal detection of the curve area is achieved through a multi-level trigger perception mechanism. First, the preset ground vibration sensor network is used to collect real-time vibration data, and a time-space correlation trigger matrix is ​​constructed in combination with the sensor's geographic coordinates. When three adjacent sensors in the matrix detect an abnormal vibration spectrum at the same time within 2 seconds, a first-level alarm is generated and the millimeter-wave radar is triggered to perform a directional focus scan on the target area. It should be noted that the abnormal vibration spectrum here refers to the frequency domain energy spectrum generated by Fourier transforming the original vibration waveform collected by the piezoelectric sensor, and the monitored 20-50Hz frequency band, which is the characteristic frequency band of tire water impact.

[0024] When the millimeter-wave radar performs a directional focused scan of the target area, the radar system identifies abnormal areas through reflection intensity gradient calculation. For example, if the reflection intensity mutation value of the waterlogged area exceeds 4dB / m², the third-level optical verification is started after the coordinates are locked, and the camera is controlled to perform digital zoom and strobe fill light. The digital zoom can be optically magnified 8 times, and the strobe fill light can be set to a 100Hz short pulse to eliminate motion blur. Through the above focusing operation, abnormal optical features are captured, such as the mirror reflection spot area > 0.2㎡. Finally, the vibration spectrum, radar reflection gradient and optical feature data are integrated to generate a multi-modal abnormal data set through spatial coordinate mapping. The hierarchical trigger mechanism can filter out false alarms layer by layer. Vibration sensing serves as a low-cost first screening layer, radar provides millimeter-level spatial positioning, and optical equipment completes high-precision verification. The three form a "funnel-type" verification system, which effectively improves the accuracy of abnormal detection on curves. Compared with traditional single-modal methods, it reduces the operating time of radar and optical equipment, and can effectively compress the response speed of water accumulation identification in rainstorm scenarios. It can solve the three major pain points of excessive energy consumption, misjudgment of reflection on curved surfaces, and data processing delays caused by the synchronous operation of multiple sensors at all times in the background technology. It is suitable for scenarios with visual blind spots and high real-time requirements, such as sharp turns on highways and urban interchange ramps, and provides a reliable multi-modal collaborative perception solution for road safety management systems.

[0025] Step S103: performing multimodal collaborative verification on the multimodal road anomaly data based on the real-time meteorological data acquired in advance, and determining the regional anomaly confidence level corresponding to the curve area.

[0026] According to the real-time meteorological data acquired in advance, the multimodal road anomaly data is subjected to multimodal collaborative verification to determine the regional anomaly confidence corresponding to the curve area, specifically including: obtaining the abnormal feature information and abnormal position data in the multimodal road anomaly data, wherein the abnormal position data includes abnormal vibration spatiotemporal information, the abnormal area point cloud coordinate data and the abnormal optical feature coordinate data; mapping the abnormal vibration spatiotemporal information and the abnormal area point cloud coordinate data to the abnormal optical feature coordinates through feature point matching to determine the multimodal anomaly feature corresponding to each abnormal feature point; according to the real-time meteorological data, the multimodal anomaly feature corresponding to each abnormal feature point is verified in the spatiotemporal dimension to generate the regional anomaly confidence corresponding to the curve area. According to the real-time meteorological data, the multimodal abnormal features corresponding to each abnormal feature point are verified in time and space dimensions to generate the regional abnormality confidence corresponding to the curve area, specifically including: according to the real-time meteorological data, a modal weight combination corresponding to multiple modes is determined, and based on the modal weight combination, a weighted calculation is performed on the multimodal abnormal features corresponding to each of the abnormal feature points to determine the multimodal verification confidence corresponding to each of the abnormal feature points; the average value of the multimodal verification confidence of each of the abnormal feature points is calculated to determine the regional abnormality confidence corresponding to the curve area.

[0027] In one embodiment of the present specification, by fusing real-time meteorological data with multimodal road abnormality features, dynamic weight allocation is performed to achieve accurate calculation of abnormal confidence in the curve area, and solve the technical problems of high misjudgment rate and rigid multimodal data fusion under complex meteorological conditions in traditional methods. First, the multimodal road abnormality data collected by vibration sensors, millimeter wave radars and optical devices are time-space aligned and feature matched. The time-space stamp information of the vibration sensor, the radar point cloud coordinates and the pixel-level coordinates of the optical device are spatially mapped using a high-precision coordinate conversion algorithm. The high-precision coordinate conversion algorithm can be WGS84 to a local plane coordinate system. The sub-meter alignment of the trimodal data is achieved through SIFT feature extraction and RANSAC algorithm. The vibration abnormal point and the radar point cloud are matched through an iterative nearest point algorithm to calculate the transformation matrix; the aligned radar-vibration data are then affine transformed with the optical image feature points, such as the edge contour of the water accumulation, and finally the trimodal coordinate fusion is achieved to ensure that each abnormal feature point has an associated abnormal vibration spectrum, abnormal reflection gradient and abnormal optical texture features under a unified time-space reference.

[0028] Real-time access to meteorological data streams to quantify the weight of the impact of parameters such as rainfall intensity, wind speed, and visibility on multimodal sensors. For example, when the rainfall intensity exceeds 30 mm / h, the reliability of the mirror reflection feature of the optical sensor decreases due to raindrop interference, and its weight coefficient is dynamically reduced from the baseline value of 0.3 to 0.2, while the penetration characteristics of the millimeter-wave radar in rain and fog increase its weight from 0.4 to 0.6. At the same time, the vibration sensor needs to perform band energy compensation calculations due to the increase in rain impact noise.

[0029] For each aligned abnormal feature point, a dynamic weighted confidence is calculated. Here, the multimodal verification confidence of each abnormal feature point is Through the dynamic weight formula Real-time calculation, in which the vibration spectrum anomaly score Sv, radar reflection gradient anomaly score Sr, and optical texture consistency anomaly score So are normalized respectively. , and They are vibration weight, radar weight and optical weight respectively. The initial values ​​are 0.3, 0.4 and 0.3 respectively. They change dynamically after adjustment according to real-time meteorological data. When determining the vibration spectrum anomaly score Sv, radar reflection gradient anomaly score Sr and optical texture consistency anomaly score So, each feature can be assigned a different value based on whether it is an abnormal feature. If the feature is abnormal, it is assigned a value of 1. On the contrary, it is assigned a value of 0. Assuming that there is an abnormal feature point A, among the corresponding vibration spectrum, radar reflection gradient and optical texture features, only one abnormal feature is the optical texture feature. The corresponding optical texture feature abnormal score is 1, and the rest of the vibration spectrum anomaly scores and radar reflection gradient anomaly scores are 0. In addition to the above methods, the abnormal feature ratio can also be obtained by the ratio of the abnormal quantity corresponding to the abnormal feature to the standard quantity. The purpose is to determine whether multiple modes have the same abnormal conclusion for the same feature point. Finally, the confidence of all feature points in the region is averaged, and the spatial continuity correction can be superimposed. For example, when the proportion of continuous abnormal points is >60%, the confidence is increased by 15% to generate the regional abnormal confidence.

[0030] By utilizing the complementary characteristics of multimodal data under complex meteorological conditions, vibration sensing can capture high-frequency shock waves when a vehicle passes through water, but it is easily disturbed by crosswinds; radar reflection gradients can penetrate rain and fog to identify the shape of water accumulation, but are sensitive to small ups and downs on the road surface; optical texture analysis is extremely accurate in sunny weather, but is easily ineffective in strong light or rain and fog. For example, in foggy weather with visibility below 50m, the radar weight is increased to 0.7 to dominate the decision-making, vibration data is used to verify the frequency of vehicle passing, and optical equipment is only used for auxiliary verification, which effectively solves the problems of high misjudgment rate caused by meteorological interference, rigid multimodal data utilization, and delayed response in extreme weather. Through meteorological-driven dynamic weight allocation, the dominant modal data in the current environment is automatically strengthened, and the dominant modality is associated with the confidence score in the data collaborative verification stage, ensuring the accuracy of the collaborative verification results between multimodal data, and eliminating invalid data before risk warning, which not only avoids the misjudgment results caused by invalid data, but also reduces the amount of data processing.

[0031] Step S104: Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area.

[0032] Based on the regional abnormality confidence corresponding to the curve area, multimodal feature fusion is performed on the multimodal road abnormality data to determine the abnormal risk information corresponding to the curve area, specifically including: when the regional abnormality confidence is not less than a preset confidence threshold, abnormal feature information in the multimodal road abnormality data is obtained, wherein the abnormal feature information includes an abnormal vibration spectrum, an abnormal reflection intensity gradient and an abnormal optical feature; based on the abnormal feature information, multimodal feature fusion is performed on the curve area to identify road water accumulation abnormality information in the curve area, wherein the road water accumulation abnormality information includes road water accumulation depth data; real-time road data is obtained to perform dynamic risk quantification on the curve area based on the real-time road data and the road water accumulation abnormality information to determine a dynamic risk index; and abnormal risk information corresponding to the curve area is determined through the dynamic risk index, wherein the abnormal risk information includes an abnormal risk level and a risk disposal strategy.

[0033] In one embodiment of the present specification, the regional anomaly confidence is converted into executable traffic risk decision information through multimodal feature fusion and dynamic risk quantification mechanism, effectively solving the core problems of inaccurate water depth estimation and static risk management strategy in traditional methods. When it is determined that the anomaly confidence of a certain curve area exceeds the preset threshold (such as ≥0.85), it means that the collected multimodal data is accurate and available data. After confirming the available accurate data, the subsequent risk identification process is executed. If the anomaly confidence of a certain curve area does not exceed the preset threshold, it means that the accuracy of the abnormal features collected in this curve area is not high, and the target curve area continues to be monitored, and the subsequent multimodal feature fusion and dynamic risk analysis process is not executed.

[0034] First, the key features in the multimodal sensor data are extracted, and the abnormal vibration spectrum energy distribution map is obtained from the vibration sensor to identify the propagation range of the shock wave in the waterlogged area. The reflection intensity gradient matrix of the millimeter-wave radar is analyzed simultaneously. By comparing the preset dry road reflection baseline value (such as -5dB) with the current measurement value, the reflection attenuation difference of each grid unit is calculated, and the surface morphology of the waterlogged area is inverted by combining the three-dimensional coordinates of the point cloud; at the same time, the high-resolution multispectral image captured by the optical device is called, and the water absorption characteristics of the near-infrared band (850nm) and the visible light mirror reflection characteristics are used to construct a pixel-level waterlogging depth estimation model. The fusion of the three modal data adopts a hierarchical calibration strategy. The vibration data provides the trigger mark of the waterlogging event in the time dimension, the radar point cloud establishes a millimeter-level spatial coordinate frame, and the optical image performs sub-meter-level texture analysis within this frame. Finally, the waterlogging depth surface model is fitted through the weighted least squares algorithm.

[0035] The depth of water directly affects the attenuation rate of the tire-road friction coefficient, while the speed and vehicle mass jointly determine the probability of brake failure, and the radius of curvature increases the risk of vehicle skidding through centrifugal force. By cross-modal calibration of the spatiotemporal distribution characteristics of the vibration spectrum reflecting instantaneous impact energy, the radar reflection gradient characterizing the stability of the water morphology, and the optical depth inversion reflecting the surface hydrological characteristics, the perception limitations of a single sensor in rain and fog interference or low-light conditions at night can be overcome. For example, in a rainstorm scene, the vibration sensor can capture the high-frequency impact signal of dense vehicles passing through the water area, the radar locks the expansion trend of the water boundary through the sudden change of reflection intensity, and the optical device estimates the depth growth rate based on the change of near-infrared absorption rate. The fusion of the three enables the system to maintain a high accuracy rate of water recognition in low visibility.

[0036] Based on the real-time road data and the abnormal information on waterlogging on the road, the dynamic risk quantification is performed on the curved area, and a dynamic risk index is determined, which specifically includes: performing risk correction on the road waterlogging depth data according to the real-time average vehicle speed data in the real-time road data and the pre-acquired curve curvature parameters, and determining the corrected waterlogging depth data; using the real-time road data, counting the proportion of designated large vehicles in the curved area within a preset time period, and obtaining the current visibility data of the curved area; based on the current visibility data, the proportion of designated large vehicles and the corrected waterlogging depth data, the risk quantification is performed on the curved area, and the dynamic risk index is determined.

[0037] In one embodiment of the present specification, in the dynamic risk quantification stage, the road traffic volume, average speed and large vehicle proportion data are accessed in real time to construct a multi-factor coupled risk index model. Based on the water depth as the basic parameter, the road water depth data is risk-corrected according to the curve curvature radius and the real-time average speed to determine the corrected water depth data. If the average speed in the curve area is high, the risk is doubled at high speed, while the risk is attenuated at low speed; similarly, there is water accumulation of the same depth in the curve area with different curve curvature radii, and the corresponding risks are also different. Therefore, the road water depth data is risk-corrected according to the curve curvature radius and the real-time average speed. When the real-time average speed is greater than 80km / h, the corresponding risk increases. Therefore, 1.1 times the road water depth is taken as the water depth after speed correction; when the real-time average speed is less than 50km / h, the corresponding risk decreases. Therefore, 0.9 times the road water depth is taken as the water depth after speed correction; when the real-time average speed is not greater than 80km / h and not less than 50km / h, no correction is performed. When correcting the road water depth according to the radius of curve curvature, based on the relationship between the benchmark curve curvature and the curve curvature radius corresponding to this curve, if the curve curvature radius is larger than the benchmark curve curvature, it means that the curve is smaller, and 0.9 times the road water depth is taken as the corrected water depth of the curve; if the curve curvature radius is not larger than the benchmark curve curvature, it means that the sharper the curve, the more significant the risk amplification effect, and 1.1 times the road water depth is taken as the corrected water depth of the curve.

[0038] By integrating visibility data, the proportion of large vehicles and the corrected water depth, a multi-dimensional risk quantification model is constructed to accurately assess the real-time safety risks in the curved area and generate a dynamic risk index. This model combines environmental perception, traffic flow characteristics and the physical state of the road, and achieves intelligent determination of risk levels through a hierarchical weighting and dynamic correction mechanism.

[0039] First, the three key parameters are normalized and mapped to a unified risk contribution scale (0-1). The corrected water depth is used as the basic risk parameter to directly reflect the degree of physical danger on the road surface. For example, water accumulation above 8 cm is marked as "extremely high risk" with a corresponding scale value of 1.0, while water accumulation below 3 cm is "low risk" (scale 0.3). Visibility data is converted into a perceived risk coefficient through an exponential decay model. When visibility is less than 50 meters, the driver's reaction time is significantly extended and the risk scale is increased to 0.9; when visibility exceeds 200 meters, the risk scale is reduced to 0.2. The proportion of large vehicles is converted in a linear ratio. For every 10% increase in heavy vehicle flow, the risk scale increases by 0.2 to reflect its characteristics of long braking distance and serious accident consequences.

[0040] Based on the sensitivity differences of various parameters in different scenarios, weights are dynamically allocated. When visibility is less than 100 meters, the visibility weight is increased to 60%, and the depth of water accumulation and the proportion of large vehicles are 20% each. For example, in a rainstorm, the visibility is 30 meters, the correction value of the water depth is 6 centimeters, and the proportion of large vehicles is 30%. At this time, the visibility risk scale is 0.9×60% = 0.54, the water depth scale is 0.75×20% = 0.15, and the large vehicle scale is 0.6×20% = 0.12. The total risk index is 0.81 (81 points), triggering a high-risk warning.

[0041] When visibility is good (>500 meters) but the proportion of large vehicles exceeds 40%, the weight of large vehicles rises to 50%, the depth of water accounts for 30%, and visibility accounts for 20%. For example, if the water level on the bend of the freight channel is 4 cm (corrected value), the proportion of large vehicles is 50%, and visibility is 800 meters, the total risk index is (0.5×50%) + (0.5×30%) + (0.1×20%) = 0.42 (42 points), which is in the medium risk range.

[0042] The dynamic risk index is divided into four levels, among which 0-30 is low risk, yellow warning, and the corresponding risk handling strategy is to issue speed limit reminders on the corresponding roads, such as "slippery curves, recommended speed ≤ 60km / h"; 31-60 is medium risk, orange warning, and the corresponding risk handling strategy is to send lane closure suggestions, such as closing the outer lane, starting the variable speed limit sign to dynamically reduce the speed, such as from 100km / h to 80km / h; 61-85 is high risk, red warning, and the corresponding risk handling strategy is forced diversion, and linking the navigation software to re-plan the route, and simultaneously notifying the maintenance unit for emergency drainage; 86-100 is extremely high risk, deep red warning, and the corresponding risk handling strategy is to close the entire road section, activate the emergency lane as a temporary evacuation channel, and guide vehicles to detour through roadside broadcasting. The depth of water directly affects the adhesion decay rate between tires and the road surface. The high kinetic energy characteristics of large vehicles aggravate the severity of accidents, while low visibility shortens the driver's effective reaction time. Through dynamic weight allocation, the current dominant risk factors can be automatically identified. For example, in dense fog, the weight of visibility is increased to prioritize the decline in environmental perception; and during peak freight hours, the weight of large vehicles is increased to prevent chain accidents caused by heavy vehicles skidding. In addition, the correction of water depth has pre-integrated the effects of vehicle speed and curve curvature to ensure the comparability of risks under different curve geometry conditions.

[0043] The dynamic risk index model integrates multi-dimensional data such as road geometry parameters, real-time traffic flow and environmental visibility, which improves the fit between risk warning and actual conditions. The graded disposal strategy can be uploaded through the edge node to achieve real-time linkage with the roadside variable information board and the vehicle-mounted terminal. When the risk index reaches the red level, it can automatically trigger lane control 2km upstream to guide vehicles to change lanes in advance. The response speed is improved compared to manual decision-making, which effectively solves the false alarm problem caused by the inability of the static threshold model to adapt to dynamic traffic scenarios, and the response delay problem caused by the disconnection between the risk disposal strategy and the real-time road conditions.

[0044] Through the technical solution of the embodiments of this specification, real-time curve road parameters are obtained through curve edge nodes, and road curve characteristics are analyzed to determine curve deployment information. According to the particularities of the curve area, such as the composition of arc segments with different curvature radii and uneven distribution of water accumulation due to centrifugal force, corresponding configuration parameter settings are adopted to avoid misjudgment caused by problems such as reflection of curve surfaces and refraction of light in traditional methods, and improve the source accuracy of road condition perception data in curve areas; the indiscriminate collection method in curve areas and non-curve areas is changed, and targeted multi-level trigger perception is performed on curve areas according to curve deployment information, avoiding unnecessary data collection and reducing data processing burden, thereby improving the perception efficiency of the target road and being able to discover abnormal road conditions in curve areas more quickly. The problem of low perception efficiency in traditional methods is overcome; the perception advantages of different types of sensors are taken into consideration, such as the multi-level trigger perception method of primary vibration perception, secondary radar focus perception and tertiary optical verification perception, which avoids the synchronous collection of multiple sensors at all times and reduces the power consumption of sensor equipment. While ensuring the perception effect, it improves energy utilization efficiency; multi-modal collaborative verification is performed on multi-modal road abnormal data according to real-time meteorological data to determine the regional abnormality confidence, and then multi-modal feature fusion is performed to determine the abnormal risk information. The multi-modal data fusion and collaborative verification method makes full use of the data of various sensors, which can judge the abnormal risk of the curve area more comprehensively and accurately, improve the reliability of risk judgment, reduce the problem of delayed warning of water accumulation risk, and reduce the hidden dangers of traffic safety.

[0045] The embodiment of this specification also provides a road surface perception device based on multimodal data, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0046] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0047] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0048] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0049] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0050] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0052] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0054] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0055] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0056] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0057] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0058] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A road surface perception method based on multimodal data, characterized in that: The method comprises: Acquire real-time curve road parameters of the curve area through the curve edge nodes corresponding to the curve area in the target road, perform road curve feature analysis based on the real-time curve road parameters, and determine curve deployment information of each curve area, wherein the curve deployment information includes a deployment working mode corresponding to the multimodal sensor; Through the curve deployment information, multi-level trigger perception is performed on the curve area to determine multi-modal road abnormality data corresponding to the curve area, wherein the multi-level trigger perception includes primary vibration perception, secondary radar focus perception and tertiary optical verification perception; According to the real-time meteorological data acquired in advance, multi-modal collaborative verification is performed on the multi-modal road anomaly data to determine the regional anomaly confidence level corresponding to the curve area; Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area.

2. The road surface perception method based on multimodal data according to claim 1, characterized in that: Based on the real-time curved road parameters, a road curve feature analysis is performed to determine the curve deployment information of each curved area, specifically including: Acquire real-time road data and curve geometric feature data in the real-time curve road parameters, wherein the real-time road data includes real-time traffic flow and real-time average vehicle speed data, and the curve geometric feature data includes curve curvature parameters corresponding to the curve area; The collection parameters of the pre-set ground vibration sensor are set according to the curve curvature parameters corresponding to the curve area and the real-time road data to determine the real-time curve deployment information, wherein the real-time curve deployment information includes the effective monitoring radius and the sampling frequency.

3. The road surface perception method based on multimodal data according to claim 2, characterized in that: The pre-set ground vibration sensor is set to collect parameters according to the curve curvature parameter corresponding to the curve area and the real-time road data to determine the real-time curve deployment information, specifically including: Determine a curvature adjustment factor corresponding to the curved area by using a curve curvature parameter corresponding to the curved area and a reference curvature parameter corresponding to a preset reference interval, so as to correct a preset reference monitoring radius based on the curvature adjustment factor and determine an effective monitoring radius corresponding to the curved area; Determine, according to the real-time traffic volume in the real-time road data, a traffic volume level corresponding to the real-time traffic volume to match a reference frequency under the traffic volume level; Based on the real-time average vehicle speed, the reference frequency is compensated for the vehicle speed, and a sampling frequency corresponding to the ground vibration sensor in the curve area is determined.

4. The road surface perception method based on multimodal data according to claim 1, characterized in that: The curve area is subjected to multi-level triggering perception through the curve deployment information to determine the multi-modal road abnormality data corresponding to the curve area, specifically including: Collecting real-time curve vibration data collected by a plurality of ground vibration sensors through the curve deployment information, so as to determine a spatiotemporal correlation trigger matrix according to the real-time curve vibration data and the sensor layout information of the ground vibration sensors; When a first preset number of adjacent sensors in the spatiotemporal correlation trigger matrix simultaneously detect an abnormal vibration spectrum within a second preset time length, determining the spatiotemporal information of abnormal vibrations corresponding to the abnormal vibration spectrum, and generating a first trigger instruction corresponding to the secondary radar focus perception; Under the triggering of the first trigger instruction, a priority scanning area is determined according to the abnormal vibration spatiotemporal information, so as to perform radar focused scanning on the priority scanning area and determine real-time scanning data in the priority scanning area; Calculating the reflection intensity gradient of the real-time scanning data, and when there is an abnormal reflection intensity gradient in the priority scanning area, triggering a second trigger instruction corresponding to the third-level optical verification perception, and determining the abnormal area point cloud coordinate data corresponding to the abnormal reflection intensity gradient; Under the triggering of the second trigger instruction, a digital zoom lock operation and a stroboscopic fill light operation are performed on the optical sensing device to determine an abnormal optical feature of the optical sensing device and abnormal optical feature coordinate data corresponding to the abnormal optical feature; Determine multimodal abnormal feature information through the abnormal vibration spectrum, the abnormal reflection intensity gradient and the abnormal optical feature, and determine multimodal abnormal position data according to the abnormal vibration spatiotemporal information, the abnormal area point cloud coordinate data and the abnormal optical feature coordinate data; The multimodal road anomaly data is determined based on the multimodal anomaly feature information and the multimodal anomaly position data.

5. The road surface perception method based on multimodal data according to claim 1, characterized in that: According to the real-time meteorological data acquired in advance, multi-modal collaborative verification is performed on the multi-modal road abnormality data to determine the regional abnormality confidence corresponding to the curve area, specifically including: Acquire abnormal feature information and abnormal position data in the multimodal road abnormal data, wherein the abnormal position data includes abnormal vibration spatiotemporal information, abnormal area point cloud coordinate data and abnormal optical feature coordinate data; By matching feature points, the abnormal vibration spatiotemporal information and the abnormal area point cloud coordinate data are mapped to the abnormal optical feature coordinates to determine the multimodal abnormal features corresponding to each abnormal feature point; According to the real-time meteorological data, the multimodal abnormal features corresponding to each abnormal feature point are verified in time and space dimensions to generate the regional abnormality confidence corresponding to the curve area.

6. The road surface perception method based on multimodal data according to claim 5, characterized in that: According to the real-time meteorological data, the multimodal abnormal features corresponding to each abnormal feature point are verified in time and space dimensions to generate the regional abnormality confidence corresponding to the curve area, specifically including: Determine a modal weight combination corresponding to a plurality of modalities according to the real-time meteorological data, and perform weighted calculation on the multimodal abnormal feature corresponding to each of the abnormal feature points based on the modal weight combination to determine the multimodal verification confidence corresponding to each of the abnormal feature points; The multimodal verification confidence of each abnormal feature point is averaged to determine the regional abnormality confidence corresponding to the curve area.

7. The road surface perception method based on multimodal data according to claim 1, characterized in that: Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area, specifically including: When the regional abnormality confidence is not less than a preset confidence threshold, acquiring abnormal feature information in the multimodal road abnormality data, wherein the abnormal feature information includes an abnormal vibration spectrum, an abnormal reflection intensity gradient, and an abnormal optical feature; According to the abnormal feature information, multimodal feature fusion is performed on the curve area to identify abnormal information of road water accumulation in the curve area, wherein the abnormal information of road water accumulation includes road water accumulation depth data; Acquire real-time road data, and perform dynamic risk quantification on the curve area based on the real-time road data and the abnormal road water accumulation information to determine a dynamic risk index; The abnormal risk information corresponding to the curve area is determined by the dynamic risk index, wherein the abnormal risk information includes an abnormal risk level and a risk handling strategy.

8. The road surface perception method based on multimodal data according to claim 7, characterized in that: Based on the real-time road data and the abnormal road water accumulation information, dynamic risk quantification is performed on the curve area to determine a dynamic risk index, specifically including: According to the real-time average vehicle speed data in the real-time road data and the pre-acquired curve curvature parameters, risk correction is performed on the road water depth data to determine the corrected water depth data; By using the real-time road data, the proportion of designated large vehicles in the curved area within a preset time period is counted, and current visibility data of the curved area is obtained; Based on the current visibility data, the proportion of designated large vehicles and the corrected water depth data, the risk of the curved area is quantified to determine the dynamic risk index.

9. A road surface perception device based on multimodal data, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images