A Road Surface Sensing Method, Device and Medium Based on Multimodal Data

Abstract

The embodiments of this specification disclose a road surface perception method, device, and medium based on multimodal data, which relate to the technical field of road surface perception. The method includes: obtaining real-time curved road parameters of a curved road area through curved edge nodes corresponding to the curved road area in a target road, analyzing road curve characteristics based on the real-time curved road parameters, and determining the curved road deployment information of each curved road area; performing multi-level trigger perception on the curved road area through the curved road deployment information to determine multimodal road abnormal data corresponding to the curved road area, including primary vibration perception, secondary radar focusing perception, and tertiary optical verification perception; performing multimodal collaborative verification on the multimodal road abnormal data according to the pre-obtained real-time meteorological data to determine the regional abnormal confidence level corresponding to the curved road area; and performing multimodal feature fusion on the multimodal road abnormal data based on the regional abnormal confidence level corresponding to the curved road area to determine the abnormal risk information corresponding to the curved road area.

Description

Technical Field

[0001] This specification relates to the technical field of road surface perception, and particularly to a road surface perception method, device, and medium based on multi-modal data. Background Art

[0002] In the field of road management, real-time and accurate monitoring of road surface conditions is a core requirement for improving traffic safety and traffic efficiency. Among them, road surface water accumulation, as a high-risk abnormal scenario, is likely to cause accidents such as vehicle skidding and braking failure. Especially in a curved road section, a curve may consist of multiple circular arc segments with different radii of curvature. Due to the action of centrifugal force, the water accumulation is unevenly distributed, further amplifying the potential safety hazards.

[0003] In the current road surface perception methods, through image recognition technology, it relies on the reflective characteristics of water accumulation. However, in the curved road section, due to problems such as curved surface reflection and light refraction, there are situations where metal signs or wet road surfaces are misjudged as water accumulation. The non-discriminatory acquisition method is adopted in both the curved road section and the non-curved road section, which additionally increases the data processing burden, resulting in the need to improve the perception efficiency of the target road. In addition, in order to solve the technical problem of one-sided perception results generated by a single type of sensor, there are existing technologies that fuse multiple modal data. However, in the current multi-sensor fusion perception method, multiple sensors collect data synchronously all the time, without considering the perception advantages of different types of sensors, increasing the power consumption of the sensing devices. Therefore, in the traditional road surface water accumulation perception method, the particularity of the curved road section and the perception advantages of different types of perception are not considered, the perception efficiency needs to be improved, there is a problem of lag in water accumulation risk warning, and the traffic safety hidden danger is increased. Summary of the Invention

[0004] One or more embodiments of this specification provide a road surface perception method, device, and medium based on multi-modal data to solve the following technical problems: In the traditional road surface water accumulation perception method, the particularity of the curved road section and the perception advantages of different types of perception are not considered, the perception efficiency needs to be improved, there is a problem of lag in water accumulation risk warning, and the traffic safety hidden danger is increased.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of this specification provide a road surface perception method based on multimodal data. The method includes: obtaining real-time curve road parameters of a curve area through curve edge nodes corresponding to the curve area in a target road, and based on the real-time curve road parameters, analyzing road curve characteristics to determine curve deployment information for each curve area, where the curve deployment information includes the deployment working mode corresponding to a multimodal sensor; performing multi-level trigger perception on the curve area through the curve deployment information to determine multimodal road abnormal data corresponding to the curve area, where the multi-level trigger perception includes primary vibration perception, secondary radar focusing perception, and tertiary optical verification perception; performing multimodal collaborative verification on the multimodal road abnormal data according to pre-obtained real-time meteorological data to determine the regional abnormal confidence level corresponding to the curve area; and performing multimodal feature fusion on the multimodal road abnormal data based on the regional abnormal confidence level corresponding to the curve area to determine abnormal risk information corresponding to the curve area.

[0007] One or more embodiments of this specification provide a road surface perception device based on multimodal data, including:

[0008] At least one processor; and,

[0009] A memory communicatively connected to the at least one processor; where,

[0010] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.

[0011] A non-volatile computer storage medium provided by one or more embodiments of this specification stores computer-executable instructions, and the computer-executable instructions are set to: execute the above method.

[0012] The above at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By obtaining real-time curved road parameters through curved-edge nodes and analyzing the characteristics of road curves, the deployment information for curves is determined. In view of the particularities of the curved area, such as being composed of arc segments with different curvature radii and uneven water accumulation distribution caused by centrifugal force, corresponding configuration parameter settings are taken, avoiding misjudgments caused by problems such as curved surface reflection and light refraction in traditional methods, and improving the accuracy of the source of road surface state perception data in the curved area; It changes the non-discriminatory acquisition method in the curved area and non-curved area, and conducts targeted multi-level trigger perception of the curved area according to the deployment information of the curve, avoiding unnecessary data acquisition, reducing the data processing burden, thereby improving the perception efficiency of the target road, being able to discover road surface anomalies in the curved area faster, and overcoming the problem of low perception efficiency in traditional methods; It takes into account the perception advantages of different types of sensors, such as adopting a multi-level trigger perception method of primary vibration perception, secondary radar focusing perception, and tertiary optical verification perception, avoiding synchronous acquisition by multiple sensors at all times, reducing the power consumption of sensing devices, and improving the energy utilization efficiency while ensuring the perception effect; It conducts multi-modal collaborative verification on multi-modal road anomaly data according to real-time meteorological data to determine the regional anomaly confidence level, and then conducts multi-modal feature fusion to determine the anomaly risk information. The method of multi-modal data fusion and collaborative verification makes full use of the data of multiple sensors, can more comprehensively and accurately judge the anomaly risk in the curved area, improves the reliability of risk judgment, reduces the problem of lag in water accumulation risk warning, and reduces the potential safety hazard of passing. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0014] Figure 1 It is a schematic flowchart of a road surface perception method based on multi-modal data provided by an embodiment of this specification;

[0015] Figure 2 It is a schematic structural diagram of a road surface perception device based on multi-modal data provided by an embodiment of this specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all of them. Based on the embodiments of this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0017] The embodiments of this specification provide a road surface perception method based on multimodal data. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 As shown in the flowchart of a road surface perception method based on multimodal data provided by the embodiments of this specification, Figure 1 it mainly includes the following steps:

[0018] Step S101: Obtain the real-time curved road parameters of the curved road area through the curved edge nodes corresponding to the curved road area in the target road, and based on the real-time curved road parameters, conduct road curve feature analysis to determine the curve deployment information of each curved road area.

[0019] Among them, the curve deployment information includes the deployment working mode corresponding to the multimodal sensor;

[0020] In an embodiment of this specification, in order to achieve road surface perception of the target road, curved edge nodes are pre-set in multiple curved road areas of the target road, and road surface perception within the corresponding curved road area is realized through the curved edge nodes. For a target road with a long length, when perceiving the road surface within this road, the amount of perception data is large. The traditional centralized perception processing method increases the data processing burden on the central processing device, and the real-time performance of perception cannot be guaranteed. To solve this problem, the embodiments of this specification set curved edge nodes in the corresponding curved road area. These curved edge nodes are only used to obtain the sensing data within this area for real-time perception, sharing the data processing pressure, thereby improving the perception efficiency of the curved road area and ensuring the real-time performance of perception. The real-time curved road parameters of the curved road area are obtained through the curved edge nodes corresponding to the curved road area. The data collected by each curved edge node is only the data within this curved road area, reducing the amount of data at the data acquisition level.

[0021] Based on the real-time curved road parameters, road curve feature analysis is carried out to determine the curve deployment information for each such curve area, specifically including: obtaining the real-time road data and curve geometric feature data in the real-time curved road parameters, where the real-time road data includes real-time traffic flow and real-time average vehicle speed data, and the curve geometric feature data includes the curve curvature parameter corresponding to the curve area; through the curve curvature parameter corresponding to the curve area and the real-time road data, the acquisition parameter setting of the pre-set ground vibration sensor is carried out to determine the real-time curve deployment information, where the real-time curve deployment information includes the effective monitoring radius and the sampling frequency.

[0022] In one embodiment of the present specification, the traffic flow (vehicles / minute) and the average vehicle speed (km / h) are collected in real time through roadside cameras, geomagnetic sensors or in-vehicle terminals, and the curve curvature parameter of the target road in the road construction stage is obtained. In general road perception systems, sensing devices are pre-arranged in the early stage, but the usual arrangement method is uniform arrangement, that is, whether it is a straight area or a curve area, a fixed-spacing arrangement method is adopted. For example, a piezoelectric vibration sensor is deployed every 5 meters along the road edge to form a grid monitoring array. In this scenario, it is difficult to adjust the physical position parameters of the uniformly arranged sensing network, and usually a one-size-fits-all parameter setting method is adopted, ignoring the particularity of the curve area. Therefore, in the embodiment of the present specification, without changing the physical parameters such as the position of the sensing network in the road, through spatial adaptive parameter mapping, the fixed-deployment sensor network can perform dynamic configuration parameter deployment for the curve geometric features and real-time traffic flow information in the curve area to meet the road perception requirements of the curve structure and real-time traffic state in the curve area.

[0023] Through the curve curvature parameter corresponding to the curve area and the real-time road data, the acquisition parameter setting of the pre-set ground vibration sensor is carried out to determine the real-time curve deployment information, specifically including: determining the curvature adjustment factor corresponding to the curve area through the curve curvature parameter corresponding to the curve area and the reference curvature parameter corresponding to the pre-set reference spacing, so as to correct the pre-set reference monitoring radius based on the curvature adjustment factor to determine the effective monitoring radius corresponding to the curve area; determining the traffic flow level corresponding to the real-time traffic flow in the real-time road data to match the reference frequency at this traffic flow level; compensating the reference frequency based on the real-time average vehicle speed to determine the sampling frequency corresponding to the ground vibration sensor in the curve area.

[0024] In one embodiment of this specification, the deployment parameters of ground vibration sensors are dynamically optimized by fusing the curve curvature parameters and real-time traffic data. First, a curvature adjustment factor is calculated based on the ratio of the actual curve curvature to the preset reference curvature, and the reference monitoring radius is dynamically compressed, so that the effective monitoring radius in the sharp curve area is reduced to a high-risk area (such as the outer lane). It should be noted that the preset reference curvature here refers to the curvature parameter matched at the reference spacing corresponding to the sensor during the layout and installation stage. Using this curvature radius as the reference curvature, the curvature adjustment factor is calculated through the ratio of the actual curve area of the target curve to the reference curvature. The reference monitoring radius under the reference curvature is corrected. The reference monitoring radius can be understood as the monitoring radius corresponding to the unified sensor configuration parameters set for the entire road in advance. Through the product of the curvature adjustment factor and the reference monitoring radius, the dynamic compression of the reference monitoring radius is achieved, and the effective monitoring radius corresponding to the curve area is obtained.

[0025] The sampling frequency is matched according to the real-time traffic flow level, and frequency compensation is performed in combination with the average vehicle speed to make the sampling interval denser as the vehicle speed is higher. According to the real-time traffic flow in the real-time road data, the traffic flow level corresponding to the real-time traffic flow is determined, and the reference frequency at this traffic flow level is matched in combination with the traffic flow level. Based on the real-time average vehicle speed, the vehicle speed compensation coefficient is calculated, and the reference frequency is compensated for vehicle speed by the product of the vehicle speed compensation coefficient and the reference frequency to determine the sampling frequency corresponding to the ground vibration sensor in this curve area. The following table is an example table of the traffic flow level, reference frequency, and vehicle speed compensation coefficient provided by the embodiment of this specification. As shown in Table 1, when the traffic flow level is <30 vehicles / minute, the reference frequency is 1 kHz, and the vehicle speed compensation coefficient is 1.0. At this time, the corresponding sampling frequency is 1 kHz; when the traffic flow corresponding to the traffic flow level is 30 - 100 vehicles / minute, the reference frequency is 2 kHz, and the vehicle speed compensation coefficient is 1 + 0.1(v / 80). Through the product of the vehicle speed compensation coefficient and the reference frequency, the sampling frequency is obtained; when the traffic flow corresponding to the traffic flow level is >100 vehicles / minute, the reference frequency is 5 kHz, and the vehicle speed compensation coefficient is , and the obtained sampling frequency is the product of the vehicle speed compensation coefficient corresponding to the vehicle speed at this time and the reference frequency.

[0026] Table 1 Example table of traffic flow level, reference frequency, and vehicle speed compensation coefficient

[0027]

[0028] In the above embodiments of this specification, by focusing on high-risk area perception resources in the spatial dimension and matching traffic state changes in the time dimension, it not only solves the problem of high incidence of local water accumulation caused by the centrifugal effect of curves, but also avoids data redundancy caused by uniform deployment across the entire road section. It can increase the curve detection resolution by more than 3 times, reduce the amount of invalid data by 70% in the scenario of 200 vehicles per hour traffic flow, and at the same time ensure the complete capture of vibration waveforms when vehicles with a speed of 100 km / h pass through through the vehicle speed adaptive mechanism, significantly improving the real-time performance and accuracy of road water accumulation identification in bad weather. It is applicable to complex curve scenarios with a radius of curvature less than 300 meters and large fluctuations in vehicle flow speed, such as urban interchange ramps and sharp turns on mountain expressways, providing a high-precision and low-energy consumption curve safety monitoring solution for intelligent transportation systems.

[0029] Step S102, through the curve deployment information, perform multi-level trigger perception on the curve area to determine the multi-modal road abnormal data corresponding to the curve area.

[0030] Among them, the multi-level trigger perception includes primary vibration perception, secondary radar focusing perception, and tertiary optical verification perception;

[0031] In an embodiment of this specification, through the curve deployment information, the following key parameters of the vibration sensor are adjusted to achieve dynamic control of the monitoring radius and sampling frequency. When adjusting the monitoring radius, the effective monitoring radius can be expanded by reducing the sensor signal trigger threshold. Conversely, increasing the threshold can narrow the monitoring range and focus on high-risk areas. In addition, the frequency domain filtering range can be adjusted, and the passband frequency of the sensor filter can be adjusted to suppress low-frequency environmental noise, indirectly narrowing the effective monitoring radius to the high-frequency vibration feature area, such as the near-field water accumulation impact signal. When adjusting the sampling frequency, high-frequency signal acquisition can be achieved by directly adjusting the sampling rate of the analog-to-digital converter; at a fixed sampling rate, the effective sampling frequency can also be equivalently increased by shortening the data acquisition time window, but the signal splicing algorithm needs to be optimized to prevent waveform breakage.

[0032] Deploy information through this bend to perform multi-level trigger perception on this bend area to determine multi-modal road anomaly data corresponding to this bend area, specifically including: collecting real-time bend vibration data collected by multiple ground vibration sensors through this bend deployment information, and determining a spatio-temporal correlation trigger matrix based on the real-time bend vibration data and the sensor layout information of the ground vibration sensors; when the first preset number of adjacent sensors in the spatio-temporal correlation trigger matrix simultaneously detect an abnormal vibration spectrum within the second preset duration, determining the abnormal vibration spatio-temporal information corresponding to the abnormal vibration spectrum and generating a first trigger command corresponding to secondary radar focused perception; under the trigger of the first trigger command, determining a priority scanning area based on the abnormal vibration spatio-temporal information, performing radar focused scanning on the priority scanning area to determine real-time scanning data within the priority scanning area; calculating a reflection intensity gradient for the real-time scanning data, and when there is an abnormal reflection intensity gradient in the priority scanning area, triggering a second trigger command corresponding to tertiary optical verification perception and determining abnormal area point cloud coordinate data corresponding to the abnormal reflection intensity gradient; under the trigger of the second trigger command, performing digital zoom locking operation and stroboscopic fill light operation on the optical sensing device to determine abnormal optical characteristics of the optical sensing device and abnormal optical characteristic coordinate data corresponding to the abnormal optical characteristics; determining multi-modal abnormal feature information through the abnormal vibration spectrum, the abnormal reflection intensity gradient, and the abnormal optical characteristics, and determining multi-modal abnormal position data based on the abnormal vibration spatio-temporal information, the abnormal area point cloud coordinate data, and the abnormal optical characteristic coordinate data; determining the multi-modal road anomaly data based on the multi-modal abnormal feature information and the multi-modal abnormal position data.

[0033] In one embodiment of this specification, abnormal detection of the bend area is achieved through a multi-level trigger perception mechanism. First, real-time vibration data is collected using a pre-set ground vibration sensor network, and a spatio-temporal correlation trigger matrix is constructed by combining the geographical coordinates of the sensors. When 3 adjacent sensors in the matrix simultaneously detect an abnormal vibration spectrum within 2 seconds, a first-level alarm is generated and a millimeter-wave radar is triggered to perform directional focused scanning on the target area. It should be noted that the abnormal vibration spectrum here refers to the frequency-domain energy spectrum generated by performing Fourier transform on the original vibration waveform collected by the piezoelectric sensor, and the monitored frequency band is 20 - 50 Hz, which is the tire water-hitting characteristic frequency band.

[0034] Under the trigger of the millimeter-wave radar for directional focusing scanning of the target area, the radar system identifies abnormal areas by calculating the reflection intensity gradient. For example, if the mutation value of the reflection intensity in the water accumulation area exceeds 4 dB / m², after locking the coordinates, a three-level optical verification is initiated, and the camera is controlled to perform digital zoom and stroboscopic fill light. The digital zoom can be an optical magnification of 8 times, and the stroboscopic fill light can be set to a 100 Hz short pulse to eliminate motion blur. Through the above focusing operations, abnormal optical features are captured, such as a specular reflection spot area > 0.2 m². Finally, the vibration spectrum, radar reflection gradient, and optical feature data are fused, and a multi-modal abnormal data set is generated through spatial coordinate mapping. The hierarchical trigger mechanism can filter false alarms layer by layer. Vibration perception serves as a low-cost first screening layer, the radar provides millimeter-level spatial positioning, and the optical device completes high-precision verification. The three form a "funnel-shaped" verification system, effectively improving the accuracy of curve anomaly detection, reducing the operating time of the radar and optical devices compared with traditional single-modal methods, and effectively compressing the response speed of water accumulation recognition in heavy rain scenarios. It can solve the three major pain points in the background technology, namely, excessive energy consumption caused by full-time multi-sensor synchronous operation, misjudgment of curve surface reflection, and data processing delay, and is applicable to scenarios with visual blind spots and high real-time requirements, such as sharp turns on highways and ramp roads of urban interchanges, providing a reliable multi-modal collaborative perception solution for the road safety management system.

[0035] Step S103: According to the pre-acquired real-time meteorological data, perform multi-modal collaborative verification on the multi-modal road abnormal data to determine the regional abnormal confidence level corresponding to the curve area.

[0036] According to the pre-acquired real-time meteorological data, perform multi-modal collaborative verification on the multi-modal road abnormal data to determine the regional abnormal confidence level corresponding to the curve area, specifically including: obtaining the abnormal feature information and abnormal position data in the multi-modal road abnormal data, where the abnormal position data includes abnormal vibration spatio-temporal information, point cloud coordinate data of the abnormal area, and coordinate data of the abnormal optical feature; through feature point matching, map the abnormal vibration spatio-temporal information and the point cloud coordinate data of the abnormal area to the coordinate of the abnormal optical feature to determine the multi-modal abnormal feature corresponding to each abnormal feature point; according to the real-time meteorological data, perform spatio-temporal dimension verification on the multi-modal abnormal feature corresponding to each abnormal feature point to generate the regional abnormal confidence level corresponding to the curve area. According to the real-time meteorological data, perform spatio-temporal dimension verification on the multi-modal abnormal feature corresponding to each abnormal feature point to generate the regional abnormal confidence level corresponding to the curve area, specifically including: according to the real-time meteorological data, determine the modal weight combination corresponding to multiple modalities, and based on the modal weight combination, perform weighted calculation on the multi-modal abnormal feature corresponding to each abnormal feature point to determine the multi-modal verification confidence level corresponding to each abnormal feature point; calculate the average value of the multi-modal verification confidence levels of each abnormal feature point to determine the regional abnormal confidence level corresponding to the curve area.

[0037] In one embodiment of the present specification, by fusing real-time meteorological data with multi-modal road anomaly features, dynamic weight allocation is performed to accurately calculate the anomaly confidence level in the curve area, solving the technical problems of high misjudgment rate and rigid multi-modal data fusion of traditional methods under complex meteorological conditions. First, spatio-temporal alignment and feature matching are performed on the multi-modal road anomaly data collected by vibration sensors, millimeter-wave radars, and optical devices. The spatio-temporal stamp information of the vibration sensor, the radar point cloud coordinates, and the pixel-level coordinates of the optical device are spatially mapped using a high-precision coordinate conversion algorithm. The high-precision coordinate conversion algorithm can be from WGS84 to the local plane coordinate system. Sub-meter alignment of the three-modal data is achieved through SIFT feature extraction and the RANSAC algorithm. The vibration anomaly points and the radar point cloud are matched through the iterative closest point algorithm to calculate the transformation matrix. The aligned radar-vibration data is then affine-transformed with the feature points of the optical image, such as the edge contour of water accumulation, and finally, three-modal coordinate fusion is achieved to ensure that each anomaly feature point has an associated anomaly vibration spectrum, anomaly reflection gradient, and anomaly optical texture feature under a unified spatio-temporal reference.

[0038] Real-time access to the meteorological data stream is performed to quantify the influence weights of parameters such as rainfall intensity, wind speed, and visibility on the multi-modal sensors. For example, when the rainfall intensity exceeds 30 mm / h, the reliability of the specular reflection feature of the optical sensor decreases due to raindrop interference, and its weight coefficient is dynamically adjusted from the reference value of 0.3 to 0.2. The penetration characteristics of the millimeter-wave radar in rain and fog increase its weight from 0.4 to 0.6. At the same time, the vibration sensor needs to perform frequency band energy compensation calculation due to increased rain impact noise.

[0039] For each aligned anomaly feature point, the dynamic weighted confidence level is calculated. Here, the multi-modal verification confidence level of each anomaly feature point is calculated in real time through the dynamic weight formula where the anomaly score Sv of the vibration spectrum, the anomaly score Sr of the radar reflection gradient, and the anomaly score So of the optical texture consistency are each normalized. 、 and They are vibration weight, radar weight and optical weight respectively. The initial values ​​are 0.3, 0.4 and 0.3 respectively. They change dynamically after adjustment according to real-time meteorological data. When determining the vibration spectrum anomaly score Sv, radar reflection gradient anomaly score Sr and optical texture consistency anomaly score So, each feature can be assigned a different value based on whether it is an abnormal feature. If the feature is abnormal, it is assigned a value of 1. On the contrary, it is assigned a value of 0. Assuming that there is an abnormal feature point A, among the corresponding vibration spectrum, radar reflection gradient and optical texture features, only one abnormal feature is the optical texture feature. The corresponding optical texture feature abnormal score is 1, and the rest of the vibration spectrum anomaly scores and radar reflection gradient anomaly scores are 0. In addition to the above methods, the abnormal feature ratio can also be obtained by the ratio of the abnormal quantity corresponding to the abnormal feature to the standard quantity. The purpose is to determine whether multiple modes have the same abnormal conclusion for the same feature point. Finally, the confidence of all feature points in the region is averaged, and the spatial continuity correction can be superimposed. For example, when the proportion of continuous abnormal points is >60%, the confidence is increased by 15% to generate the regional abnormal confidence.

[0040] By utilizing the complementary characteristics of multimodal data under complex meteorological conditions, vibration sensing can capture high-frequency shock waves when a vehicle passes through water, but it is easily disturbed by crosswinds; radar reflection gradients can penetrate rain and fog to identify the shape of water accumulation, but are sensitive to small ups and downs on the road surface; optical texture analysis is extremely accurate in sunny weather, but is easily ineffective in strong light or rain and fog. For example, in foggy weather with visibility below 50m, the radar weight is increased to 0.7 to dominate the decision-making, vibration data is used to verify the frequency of vehicle passing, and optical equipment is only used for auxiliary verification, which effectively solves the problems of high misjudgment rate caused by meteorological interference, rigid multimodal data utilization, and delayed response in extreme weather. Through meteorological-driven dynamic weight allocation, the dominant modal data in the current environment is automatically strengthened, and the dominant modality is associated with the confidence score in the data collaborative verification stage, ensuring the accuracy of the collaborative verification results between multimodal data, and eliminating invalid data before risk warning, which not only avoids the misjudgment results caused by invalid data, but also reduces the amount of data processing.

[0041] Step S104: Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area.

[0042] Based on the regional anomaly confidence corresponding to the bend area, perform multimodal feature fusion on the multimodal road anomaly data to determine the anomaly risk information corresponding to the bend area, specifically including: when the regional anomaly confidence is not less than the preset confidence threshold, obtain the anomaly feature information in the multimodal road anomaly data, where the anomaly feature information includes an abnormal vibration spectrum, an abnormal reflection intensity gradient, and abnormal optical features; according to the anomaly feature information, perform multimodal feature fusion on the bend area to identify the road water accumulation anomaly information in the bend area, where the road water accumulation anomaly information includes road water accumulation depth data; obtain real-time road data to perform dynamic risk quantification on the bend area based on the real-time road data and the road water accumulation anomaly information, and determine the dynamic risk index; through the dynamic risk index, determine the anomaly risk information corresponding to the bend area, where the anomaly risk information includes an anomaly risk level and a risk handling strategy.

[0043] In one embodiment of this specification, through a multimodal feature fusion and dynamic risk quantification mechanism, the regional anomaly confidence is converted into executable traffic risk decision-making information, effectively solving core problems such as inaccurate estimation of water accumulation depth and static risk handling strategies in traditional methods. When it is determined that the anomaly confidence of a certain bend area exceeds the preset threshold (e.g., ≥0.85), it indicates that the collected multimodal data is accurate and available data. After confirming the available accurate data, then execute the subsequent risk identification process. If the anomaly confidence of a certain bend area does not exceed the preset threshold, it indicates that the accuracy of the abnormal features collected in this bend area is not high, continue to monitor the target bend area, and do not execute the subsequent multimodal feature fusion and dynamic risk analysis process.

[0044] First, extract the key features in the multimodal sensor data. Obtain the abnormal vibration spectrum energy distribution map from the vibration sensor, which is used to identify the shock wave propagation range of the water accumulation area. Synchronously analyze the reflection intensity gradient matrix of the millimeter-wave radar. By comparing the preset dry road surface reflection baseline value (e.g., -5dB) with the current measured value, calculate the reflection attenuation difference of each grid unit, and combine the three-dimensional coordinates of the point cloud to invert the water accumulation surface morphology; at the same time, call the high-resolution multispectral image captured by the optical device, and use the water absorption characteristics of the near-infrared band (850nm) and the visible light specular reflection characteristics to construct a pixel-level water accumulation depth estimation model. The fusion of the three-modal data adopts a hierarchical calibration strategy. The vibration data provides a trigger mark for the water accumulation event in the time dimension, the radar point cloud establishes a millimeter-level spatial coordinate framework, and the optical image performs sub-meter-level texture analysis within this framework. Finally, a water accumulation depth surface model is fitted through the weighted least squares algorithm.

[0045] The water accumulation depth directly affects the attenuation rate of the tire-road friction coefficient, while the vehicle speed and vehicle mass jointly determine the probability of braking failure, and the radius of curvature exacerbates the risk of vehicle sideslip through the action of centrifugal force. By performing cross-modal calibration on the spatio-temporal distribution characteristics of the vibration spectrum reflecting instantaneous impact energy, the radar reflection gradient characterizing the stability of water accumulation morphology, and the optical depth inversion reflecting surface hydrological characteristics, the perception limitation of a single sensor under rain and fog interference or low light conditions at night is broken through. For example, in a heavy rain scenario, the vibration sensor can capture high-frequency impact signals when dense vehicles pass through the water accumulation area, the radar locks the expansion trend of the water accumulation boundary through sudden changes in reflection intensity, and the optical device estimates the depth growth rate based on changes in near-infrared absorption rate. After the three are fused, the system can still maintain a high water accumulation recognition accuracy when the visibility is low.

[0046] Based on the real-time road data and the abnormal road water accumulation information, the dynamic risk of the curve area is quantified to determine the dynamic risk index, which specifically includes: according to the real-time average vehicle speed data in the real-time road data and the pre-acquired curve curvature parameters, the risk of the road water accumulation depth data is corrected to determine the corrected water accumulation depth data; through the real-time road data, the proportion of specified large vehicles in the curve area within a preset time period is statistically calculated, and the current visibility data of the curve area is obtained; based on the current visibility data, the proportion of the specified large vehicles, and the corrected water accumulation depth data, the risk of the curve area is quantified to determine the dynamic risk index.

[0047] In one embodiment of this specification, during the dynamic risk quantification phase, road traffic flow, average vehicle speed, and the proportion of large vehicles are accessed in real time to construct a risk index model with multi-factor coupling. Using the water accumulation depth as a basic parameter, the risk of the road water accumulation depth data is corrected according to the curve radius of the bend and the real-time average vehicle speed to determine the corrected water accumulation depth data. If the average vehicle speed in the bend area is high, the risk doubles at high speed and attenuates at low speed; similarly, for the same depth of water accumulation in bend areas with different curve radii of the bend, the corresponding risks are also different. Therefore, the risk of the road water accumulation depth data is corrected according to the curve radius of the bend and the real-time average vehicle speed. When the real-time average vehicle speed is greater than 80 km / h, the corresponding risk increases. Therefore, 1.1 times the road water accumulation depth is taken as the water accumulation depth after vehicle speed correction; when the real-time average vehicle speed is less than 50 km / h, the corresponding risk decreases. Therefore, 0.9 times the road water accumulation depth is taken as the water accumulation depth after vehicle speed correction; when the real-time average vehicle speed is not greater than 80 km / h and not less than 50 km / h, no correction is made. When correcting the road water accumulation depth according to the curve radius of the bend, according to the size relationship between the reference curve radius of the bend and the curve radius of this bend, if the curve radius of the bend is greater than the reference curve radius of the bend, it means the bend is smaller, and 0.9 times the road water accumulation depth is taken as the water accumulation depth after bend correction; if the curve radius of the bend is not greater than the reference curve radius of the bend, it means the bend is sharper and the risk amplification effect is more significant, and 1.1 times the road water accumulation depth is taken as the water accumulation depth after bend correction.

[0048] By integrating visibility data, the proportion of large vehicles, and the corrected water accumulation depth, a multi-dimensional risk quantification model is constructed to accurately evaluate the real-time safety risk of the bend area and generate a dynamic risk index, combining environmental perception, traffic flow characteristics, and road physical conditions, and realizing the intelligent determination of risk levels through a hierarchical weighting and dynamic correction mechanism.

[0049] First, the three key parameters are normalized and mapped to a unified risk contribution scale (0 - 1). The corrected water accumulation depth is used as the basic risk parameter, directly reflecting the physical danger level of the road surface. For example, water accumulation above 8 cm is marked as "extremely high risk", corresponding to a scale value of 1.0, while below 3 cm is "low risk" (scale 0.3). Visibility data is converted into a perceived risk coefficient through an exponential decay model. When the visibility is less than 50 meters, the driver's reaction time significantly extends, and the risk scale increases to 0.9; when the visibility exceeds 200 meters, the risk scale drops to 0.2. The proportion of large vehicles is converted linearly. For every 10% increase in the heavy vehicle traffic flow, the risk scale increases by 0.2 to reflect its characteristics of long braking distance and serious accident consequences.

[0050] Based on the sensitivity differences of various parameters in different scenarios, weights are dynamically allocated. When visibility is less than 100 meters, the visibility weight is increased to 60%, and the depth of water accumulation and the proportion of large vehicles are 20% each. For example, in a rainstorm, the visibility is 30 meters, the correction value of the water depth is 6 centimeters, and the proportion of large vehicles is 30%. At this time, the visibility risk scale is 0.9×60% = 0.54, the water depth scale is 0.75×20% = 0.15, and the large vehicle scale is 0.6×20% = 0.12. The total risk index is 0.81 (81 points), triggering a high-risk warning.

[0051] When visibility is good (>500 meters) but the proportion of large vehicles exceeds 40%, the weight of large vehicles rises to 50%, the depth of water accounts for 30%, and visibility accounts for 20%. For example, if the water level on the bend of the freight channel is 4 cm (corrected value), the proportion of large vehicles is 50%, and visibility is 800 meters, the total risk index is (0.5×50%) + (0.5×30%) + (0.1×20%) = 0.42 (42 points), which is in the medium risk range.

[0052] The dynamic risk index is divided into four levels, among which 0-30 is low risk, yellow warning, and the corresponding risk handling strategy is to issue speed limit reminders on the corresponding roads, such as "slippery curves, recommended speed ≤ 60km / h"; 31-60 is medium risk, orange warning, and the corresponding risk handling strategy is to send lane closure suggestions, such as closing the outer lane, starting the variable speed limit sign to dynamically reduce the speed, such as from 100km / h to 80km / h; 61-85 is high risk, red warning, and the corresponding risk handling strategy is forced diversion, and linking the navigation software to re-plan the route, and simultaneously notifying the maintenance unit for emergency drainage; 86-100 is extremely high risk, deep red warning, and the corresponding risk handling strategy is to close the entire road section, activate the emergency lane as a temporary evacuation channel, and guide vehicles to detour through roadside broadcasting. The depth of water directly affects the adhesion decay rate between tires and the road surface. The high kinetic energy characteristics of large vehicles aggravate the severity of accidents, while low visibility shortens the driver's effective reaction time. Through dynamic weight allocation, the current dominant risk factors can be automatically identified. For example, in dense fog, the weight of visibility is increased to prioritize the decline in environmental perception; and during peak freight hours, the weight of large vehicles is increased to prevent chain accidents caused by heavy vehicles skidding. In addition, the correction of water depth has pre-integrated the effects of vehicle speed and curve curvature to ensure the comparability of risks under different curve geometry conditions.

[0053] The dynamic risk index model integrates multi-dimensional data such as road geometric parameters, real-time traffic flow, and environmental visibility, improving the fit between risk early warning and actual situations; the hierarchical disposal strategy can be uploaded through edge nodes to achieve real-time linkage with roadside variable message signs and in-vehicle terminals. When the risk index reaches the red level, it can automatically trigger lane control 2 km upstream to guide vehicles to change lanes in advance, improving the response speed compared to manual decision-making and effectively solving the false alarm problem caused by the inability of the static threshold model to adapt to dynamic traffic scenarios and the response delay problem caused by the disconnection between risk disposal strategies and real-time road conditions.

[0054] Through the technical solution of the embodiments of this specification, real-time curved road parameters are obtained through curved edge nodes, and road curve feature analysis is carried out to determine the curved road deployment information. In view of the particularity of the curved road area, such as being composed of arc segments with different curvature radii and uneven water accumulation distribution caused by centrifugal force, corresponding configuration parameter settings are taken to avoid misjudgment caused by problems such as curved surface reflection and light refraction in traditional methods, improving the source accuracy of the road surface state perception data in the curved road area; changing the non-discriminatory acquisition method in the curved road area and non-curved road area, and carrying out targeted multi-level trigger perception of the curved road area according to the curved road deployment information, avoiding unnecessary data acquisition and reducing the data processing burden, thereby improving the perception efficiency of the target road, being able to discover road surface anomalies in the curved road area faster, and overcoming the problem of low perception efficiency in traditional methods; considering the perception advantages of different types of sensors, such as adopting a multi-level trigger perception method of primary vibration perception, secondary radar focusing perception, and tertiary optical verification perception, avoiding all-time synchronous acquisition of multiple sensors, reducing the power consumption of sensing devices, and improving the energy utilization efficiency while ensuring the perception effect; performing multi-modal collaborative verification on multi-modal road anomaly data according to real-time meteorological data to determine the regional anomaly confidence level, and then performing multi-modal feature fusion to determine the anomaly risk information. The multi-modal data fusion and collaborative verification method makes full use of the data of multiple sensors, can more comprehensively and accurately judge the anomaly risk in the curved road area, improves the reliability of risk judgment, reduces the problem of lag in water accumulation risk warning, and reduces the hidden danger of traffic safety.

[0055] The embodiments of this specification also provide a road surface perception device based on multi-modal data, such as Figure 2 shown. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0056] The embodiments of this specification also provide a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set to: execute the above method.

[0057] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the key point of each embodiment is to illustrate the differences from other embodiments. In particular, for the embodiments of the device, equipment, and non-volatile computer storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for relevant content.

[0058] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0059] The devices and media provided in the embodiments of this specification correspond one-to-one with the methods. Therefore, the devices and media also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be elaborated here.

[0060] Those skilled in the art should understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This specification is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0062] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one process Figure 1 or more processes and / or blocks Figure 1 or more blocks specified in the block.

[0063] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 or more processes and / or blocks Figure 1 or more blocks specified in the block.

[0064] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0065] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

[0066] Computer-readable media includes permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0067] It should also be noted that the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0068] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various changes and modifications can be made to one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of one or more embodiments of this specification shall be included within the scope of the claims of this specification.

Claims

1. A road surface perception method based on multimodal data, characterized in that: The method comprises: Acquire real-time curve road parameters of the curve area through the curve edge nodes corresponding to the curve area in the target road, perform road curve feature analysis based on the real-time curve road parameters, and determine curve deployment information of each curve area, wherein the curve deployment information includes a deployment working mode corresponding to the multimodal sensor; Through the curve deployment information, multi-level trigger perception is performed on the curve area to determine multi-modal road abnormality data corresponding to the curve area, wherein the multi-level trigger perception includes primary vibration perception, secondary radar focus perception and tertiary optical verification perception; According to the real-time meteorological data acquired in advance, multi-modal collaborative verification is performed on the multi-modal road anomaly data to determine the regional anomaly confidence level corresponding to the curve area; Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area; Based on the real-time curved road parameters, a road curve feature analysis is performed to determine the curve deployment information of each curved area, specifically including: Acquire real-time road data and curve geometric feature data in the real-time curve road parameters, wherein the real-time road data includes real-time traffic flow and real-time average vehicle speed data, and the curve geometric feature data includes curve curvature parameters corresponding to the curve area; By using the curve curvature parameter corresponding to the curve area and the real-time road data, the pre-set ground vibration sensor is set to collect parameters to determine the real-time curve deployment information, wherein the real-time curve deployment information includes an effective monitoring radius and a sampling frequency; The pre-set ground vibration sensor is set to collect parameters according to the curve curvature parameter corresponding to the curve area and the real-time road data to determine the real-time curve deployment information, specifically including: Determine a curvature adjustment factor corresponding to the curved area by using a curve curvature parameter corresponding to the curved area and a reference curvature parameter corresponding to a preset reference interval, so as to correct a preset reference monitoring radius based on the curvature adjustment factor and determine an effective monitoring radius corresponding to the curved area; Determine, according to the real-time traffic volume in the real-time road data, a traffic volume level corresponding to the real-time traffic volume to match a reference frequency under the traffic volume level; Based on the real-time average vehicle speed, the reference frequency is compensated for the vehicle speed to determine the sampling frequency corresponding to the ground vibration sensor in the curve area; According to the real-time meteorological data acquired in advance, multi-modal collaborative verification is performed on the multi-modal road abnormality data to determine the regional abnormality confidence corresponding to the curve area, specifically including: Acquire abnormal feature information and abnormal position data in the multimodal road abnormal data, wherein the abnormal position data includes abnormal vibration spatiotemporal information, abnormal area point cloud coordinate data and abnormal optical feature coordinate data; By matching feature points, the abnormal vibration spatiotemporal information and the abnormal area point cloud coordinate data are mapped to the abnormal optical feature coordinates to determine the multimodal abnormal features corresponding to each abnormal feature point; According to the real-time meteorological data, the multimodal abnormal features corresponding to each abnormal feature point are verified in the time and space dimensions to generate the regional abnormality confidence corresponding to the curve area; According to the real-time meteorological data, the multimodal abnormal features corresponding to each abnormal feature point are verified in time and space dimensions to generate the regional abnormality confidence corresponding to the curve area, specifically including: Determine a modal weight combination corresponding to a plurality of modalities according to the real-time meteorological data, and perform weighted calculation on the multimodal abnormal feature corresponding to each of the abnormal feature points based on the modal weight combination to determine the multimodal verification confidence corresponding to each of the abnormal feature points; Calculating an average value of the multimodal verification confidence of each of the abnormal feature points to determine the regional abnormality confidence corresponding to the curved area; Based on the regional anomaly confidence level corresponding to the curve area, multimodal feature fusion is performed on the multimodal road anomaly data to determine the abnormal risk information corresponding to the curve area, specifically including: When the regional abnormality confidence is not less than a preset confidence threshold, acquiring abnormal feature information in the multimodal road abnormality data, wherein the abnormal feature information includes an abnormal vibration spectrum, an abnormal reflection intensity gradient, and an abnormal optical feature; According to the abnormal feature information, multimodal feature fusion is performed on the curve area to identify abnormal information of road water accumulation in the curve area, wherein the abnormal information of road water accumulation includes road water accumulation depth data; Acquire real-time road data, and perform dynamic risk quantification on the curve area based on the real-time road data and the abnormal road water accumulation information to determine a dynamic risk index; The abnormal risk information corresponding to the curve area is determined by the dynamic risk index, wherein the abnormal risk information includes an abnormal risk level and a risk handling strategy.

2. The road surface perception method based on multimodal data according to claim 1, characterized in that: The curve area is subjected to multi-level triggering perception through the curve deployment information to determine the multi-modal road abnormality data corresponding to the curve area, specifically including: Collecting real-time curve vibration data collected by a plurality of ground vibration sensors through the curve deployment information, so as to determine a spatiotemporal correlation trigger matrix according to the real-time curve vibration data and sensor layout information of the ground vibration sensors; When a first preset number of adjacent sensors in the spatiotemporal correlation trigger matrix simultaneously detect an abnormal vibration spectrum within a second preset time length, determining the spatiotemporal information of abnormal vibrations corresponding to the abnormal vibration spectrum, and generating a first trigger instruction corresponding to the secondary radar focus perception; Under the triggering of the first trigger instruction, a priority scanning area is determined according to the abnormal vibration spatiotemporal information, so as to perform radar focused scanning on the priority scanning area and determine real-time scanning data in the priority scanning area; Calculating the reflection intensity gradient of the real-time scanning data, and when there is an abnormal reflection intensity gradient in the priority scanning area, triggering a second trigger instruction corresponding to the third-level optical verification perception, and determining the abnormal area point cloud coordinate data corresponding to the abnormal reflection intensity gradient; Under the triggering of the second trigger instruction, a digital zoom lock operation and a stroboscopic fill light operation are performed on the optical sensing device to determine an abnormal optical feature of the optical sensing device and abnormal optical feature coordinate data corresponding to the abnormal optical feature; Determine multimodal abnormal feature information through the abnormal vibration spectrum, the abnormal reflection intensity gradient and the abnormal optical feature, and determine multimodal abnormal position data according to the abnormal vibration spatiotemporal information, the abnormal area point cloud coordinate data and the abnormal optical feature coordinate data; The multimodal road anomaly data is determined based on the multimodal anomaly feature information and the multimodal anomaly position data.

3. The road surface perception method based on multimodal data according to claim 1, characterized in that: Based on the real-time road data and the abnormal road water accumulation information, dynamic risk quantification is performed on the curve area to determine a dynamic risk index, specifically including: According to the real-time average vehicle speed data in the real-time road data and the pre-acquired curve curvature parameters, risk correction is performed on the road water depth data to determine the corrected water depth data; By using the real-time road data, the proportion of designated large vehicles in the curved area within a preset time period is counted, and current visibility data of the curved area is obtained; Based on the current visibility data, the proportion of designated large vehicles and the corrected water depth data, the risk of the curved area is quantified to determine the dynamic risk index.

4. A road surface perception device based on multimodal data, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 3.

5. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 3.

Images