A road state recognition method based on multi-modal fusion and physical constraint

Abstract

The application relates to the technical field of automatic driving and discloses a road state recognition method based on multi-modal fusion and physical constraints, which synchronously acquires an image stream and heterogeneous characteristic signals covering an environment temperature and a wiper working condition through a vehicle-mounted visual collection device and a chassis CAN bus, then extracts a road surface region of interest and carries out multi-scale feature coding, subsequently introduces a time lag compensation mechanism to realize accurate alignment of cross-modal data in a time and space domain, dynamically suppresses conflict feature weights based on cross-modal logic deviation, outputs an initial state probability through a fusion perception network, and finally combines a physical common sense rule base to execute mutual exclusion verification and negative feedback reconstruction, so that perception black box misjudgment is avoided; the application effectively eliminates time and space dislocation of heterogeneous data under adverse working conditions, and significantly improves fault tolerance robustness and closed-loop safety of road surface state recognition.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a road state recognition method based on multimodal fusion and physical constraints. Background Technology

[0002] In the field of intelligent connected vehicles and autonomous driving, the road surface condition (such as dry, waterlogged, icy, snowy, etc.) directly determines the adhesion characteristics between the tires and the ground. Accurate identification of this condition has become an indispensable core prerequisite for realizing vehicle dynamics control and active safety defense.

[0003] Existing road condition recognition methods are mainly divided into visual image-based perception methods and vehicle dynamics feedback-based estimation methods. Visual perception methods use onboard cameras to capture texture features to achieve long-distance prediction. However, under complex lighting conditions such as strong light reflection, nighttime, and alternating light and dark, feature extraction is easily affected by environmental interference, leading to severe fluctuations in recognition confidence. Dynamics feedback estimation methods use sensor data such as wheel speed and acceleration to infer road adhesion characteristics. Although these methods can reflect the real physical response, they are limited by the physical condition that the wheels must contact the road surface before feedback can be generated, resulting in significant perception lag and failing to provide effective control lead time.

[0004] Current research attempts to fuse visual information with sensor data, but in practical engineering applications, the following technical bottlenecks remain: misalignment of spatiotemporal features; existing technologies often assume that images and dynamic signals at the same sampling time are naturally aligned in space, ignoring the actual displacement difference between visual advance perception and physical feedback lag, which can easily lead to logical jitter in fusion decisions and the risk of "black box" misjudgment in deep models; most existing algorithms lack physical boundary constraints, and when the visual network outputs abnormal results due to environmental noise, there is a lack of cross-modal physical consistency verification and self-correction mechanisms, resulting in poor robustness of the overall perception closed loop; and the allocation of computing resources is rigid, which can lead to a waste of computing power when high-precision models run at full load all time under stable road conditions, lacking a dynamic adjustment mechanism based on operating conditions and making it difficult to adapt to the limited computing resources of in-vehicle embedded terminals.

[0005] Therefore, how to construct a road condition identification method that can achieve accurate spatiotemporal alignment of heterogeneous data, possess physical logic verification capabilities, and adaptively adjust computational load is a technical problem that needs to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a road state recognition method based on multimodal fusion and physical constraints. This method constructs an overall perception framework that includes multi-source perception, spatiotemporal alignment, and physical logic verification. It synchronously inputs vehicle video streams and heterogeneous data covering wiper status, ambient temperature, and dynamic feedback. It solves the asynchronous perception mismatch problem through displacement compensation and introduces physical common sense constraints to correct the recognition results, thereby outputting highly robust road state information to solve the problems mentioned in the background art.

[0007] The technical solution of this invention is: a road state recognition method based on multimodal fusion and physical constraints, comprising the following steps: S1: Captures road video image streams through vehicle-mounted vision equipment and performs multi-dimensional preprocessing to address environmental interference. This involves processing from the bottom pixel level, eliminating negative interference from severe light and shadow fluctuations and motion blur through dynamic noise reduction, brightness stabilization, and texture enhancement operations, and outputting a standardized base map that highlights the microscopic physical details of the road surface.

[0008] S2: Through the vehicle sensor and CAN bus interface, it collects multi-source heterogeneous data in real time that reflects the vehicle's driving environment and dynamic state, including wiper status, ambient temperature, road surface adhesion coefficient feedback and instantaneous vehicle speed, providing physical dimension information support for visual perception.

[0009] S3: The heterogeneous physical information obtained in step S2 is timestamped and reconstructed in format, encapsulated into a multimodal guidance parameter package, and sent to the visual perception layer, thereby converting discrete physical quantities into guidance instructions to drive subsequent visual processing algorithms to adaptively and dynamically adjust parameters.

[0010] S4: Receive the guidance parameter packet information from step S3, and dynamically correct the longitudinal sampling depth by combining it with the real-time vehicle speed signal, so that the perception line of sight can adaptively expand and contract with the vehicle speed. Then, extract the region of interest on the road surface on the standardized base map in step S1, and extract the spatial frequency and smoothness features through multi-scale convolution operators to generate a visual feature vector.

[0011] S5: Perform feature enhancement processing on the visual feature vector input from S4. That is, by constructing an adaptive feature gating operator, the weights of high-contribution texture components in visual encoding are amplified in a targeted manner, while suppressing redundant background interference, and outputting a more discriminative enhanced visual feature vector.

[0012] S6: Synchronously receives the guidance parameter packet from S3 and the enhanced visual feature vector from S5, performs validity detection on the sensor signal to eliminate abnormal jumps, and combines vehicle speed to construct a spatial displacement compensation and nonlinear time delay correction mechanism, and translates the visual forward features and under-wheel feedback features on the time axis to achieve accurate alignment of cross-modal data in the physical space coordinate system.

[0013] S7: Perform multimodal conflict detection and dynamic weighting on the spatiotemporally aligned data, calculate prior confidence based on environmental parameters, and quantify the logical deviation between visual perception trends and underlying physical feedback; when cross-modal conflicts are detected, suppress abnormal feature weights through dynamic weighting mechanism, output weighted feature vectors after temporal smoothing, and feed them back for calibration to avoid the risk of environmental deception by a single sensor.

[0014] S8: Input the weighted feature vector and the corresponding physical sensing features into the multi-feature space fusion perception network, perform cross-modal high-dimensional mapping in a unified decision space, and output the initial recognition probability distribution vector representing the state category of each road surface.

[0015] S9: Utilizes a constraint mechanism based on a physical common sense verification rule base to logically verify the initial identification results. If the identification state conforms to macroscopic physical laws, the final road surface state result is output. If an abnormal judgment that violates the mutual exclusion boundary is detected, the abnormal output is intercepted and negative feedback adjustment is triggered, and the data is returned to S6 for reconstruction to ensure the safety and stability of the final decision.

[0016] In one possible implementation, step S1 specifically includes the following steps: 1): Perform image acquisition and color space normalization processing. Use the vehicle-mounted visual perception device to acquire real-time road video image stream and extract the original RGB color image of the current frame from it. Then convert it to the more robust HSV color space for visual perception to achieve decoupling of luminance and chrominance components. For the uneven exposure caused by drastic changes in light and shadow on the road surface and the alternation of light and dark in the tunnel, perform luminance stabilization operation. Use the limited contrast adaptive histogram equalization operator to reconstruct the luminance channel. Its corresponding mathematical expression is defined as shown in equation (1): (1) In the formula: The output pixel brightness value after stabilization processing. The pixel brightness distribution of the original input. To process the grayscale level within a local window The number of pixels, The total number of pixels within the window. The preset peak brightness adjustment factor is used to achieve steady-state enhancement of image contrast through this operator.

[0017] 2): Dynamic noise reduction is performed. To reduce thermal noise and speckle interference during image acquisition, an adaptive bilateral filtering algorithm based on local pixel statistical characteristics is adopted. This algorithm can retain key high-frequency edges of the road surface while smoothing noise. The corresponding filter weight distribution function is shown in Equation (2): (2) In the formula: Center pixel Its neighboring pixels The combined filtering weights between them For pixels With pixels The Euclidean distance between them The preset spatial smoothing parameters, The difference in brightness between two pixels. The standard deviation is adjusted to reflect the range of values ​​that reflect changes in local variance of the image.

[0018] 3) Perform targeted texture enhancement processing to compensate for the loss of detail caused by the denoising process and enhance the physical features of the road surface. Use a multi-scale Laplacian operator to perform high-frequency sharpening on the denoised base image to highlight key textures that reflect the road surface roughness, graininess, and overburden morphology, thus obtaining the final standardized preprocessed road surface image. The feature enhancement process satisfies equation (3): (3) In the formula The preprocessed image is the final output. The input is a denoised image. The preset feature enhancement gain coefficient; The Laplacian edge operator is used to describe the second-order spatial partial derivative of image brightness.

[0019] In one possible implementation, step S2 specifically includes the following steps: 1) Perform real-time heterogeneous data acquisition. Through the vehicle sensor array and CAN bus interface, capture multi-dimensional raw feature signals reflecting the vehicle's dynamic state and external driving environment in real time. Under the synchronous triggering of the image acquisition frequency, acquire heterogeneous data streams containing ambient temperature components, wiper operating status components, current road surface adhesion coefficient feedback components, and instantaneous vehicle speed components. To simplify the unified scheduling of subsequent computing units, construct an initial heterogeneous feature vector from the acquired discrete signals. The initial heterogeneous feature vector satisfies equation (4): (4) In the formula: For the original heterogeneous feature vectors, These are discrete values ​​corresponding to the wiper activation state. This is the feedback value of the road surface adhesion coefficient. The instantaneous vehicle speed The ambient temperature is in Celsius.

[0020] 2) Perform clock alignment and frequency resampling processing. Due to the physical differences in the sampling periods of different sensor hardware, linear interpolation compensation is performed on asynchronous input sensor signals based on the visual acquisition frequency to ensure that each frame of video image can be associated with the physical feature components at the corresponding time. Then, a moving average filter operator is used to remove random electromagnetic interference noise in the transmission process of the bus signal to obtain a smoothed sensor feature vector. The filtering process is shown in equation (5): (5) In the formula: For the first The feature vector after smoothing at the sampling time. The preset sliding window length, For the first The raw input data at any given time.

[0021] 3) Perform sensor data unit normalization and feature space mapping. To eliminate the influence of different sensor unit dimensions on subsequent feature fusion weights, a min-max normalization algorithm is used to perform a linear mapping on the smoothed data, uniformly scaling its numerical range to a uniform size. Intervals are used to generate standardized sensor feature components. The standardization process satisfies equation (6): (6) In the formula: This is the standardized sensor feature vector. Preset the maximum range value for the sensor in the corresponding dimension. Set the minimum range value for the sensor in the corresponding dimension.

[0022] In one possible implementation, step S3 specifically includes the following sub-steps: 1): Perform heterogeneous data timestamp synchronization, call the global clock, and assign high-precision time tags to the wiper working status, ambient temperature, road surface adhesion coefficient and instantaneous vehicle speed signals obtained in step S2. Since there is a physical difference between the sampling frequency of the video image stream and each physical sensor, the sampling timestamp of the video frame is used as the reference to interpolate and align the asynchronous input sensor signals. This operation establishes a strict mapping relationship between a single frame road image and transient physical features on the time axis.

[0023] 2): Perform feature information format reconstruction, standardize and convert the synchronized physical sensing data, and reconstruct it into a guide instruction format compatible with the visual perception module.

[0024] 3): Establish a real-time cascaded transmission link. Relying on the internal high-speed data bus, the encapsulated multimodal guidance parameter package is directed to step S4. This low-latency transmission link ensures that step S4 can simultaneously extract the vehicle speed and environmental parameters associated with the current video frame at the moment of reading the current video frame, thereby providing underlying data support for the adaptive dynamic extraction of the region of interest.

[0025] In one possible implementation, step S4 specifically includes the following sub-steps: 1) Perform dynamic mapping of the region of interest, receive the guidance parameter packet sent in step S3, extract the instantaneous vehicle speed signal, and combine it with the physical installation parameters such as the height and pitch angle of the image acquisition device to delineate the basic field of view frame on the preprocessed image output in step S1. In order to achieve adaptive adjustment of the perception line of sight, the longitudinal sampling depth of the road surface is dynamically calculated using real-time vehicle speed feedback. The longitudinal sampling depth satisfies equation (7): (7) In the formula: This is the dynamically adjusted longitudinal sensing distance. As a set basic safe line of sight, To sense the lead gain coefficient, The real-time vehicle speed signal is transmitted synchronously in step S3. It is the acceleration due to gravity. This represents the prior adhesion coefficient of the current road surface.

[0026] 2): Perform dynamic cropping of the road surface pixel matrix. Based on the longitudinal sampling depth calculated by equation (7) above, generate a binary spatial mask matrix with boundary adaptive capability in the two-dimensional image coordinate system. By performing a spatial dot product operation between the mask matrix and the preprocessed base image, interference from non-road backgrounds such as the sky and guardrails is reduced. This operation can accurately extract the pixel matrix of the high-interest region of interest on the road surface. The extraction process is shown in equation (8): (8) In the formula: This is the pixel matrix of the extracted region of interest (ROI) on the road surface. The standardized road surface preprocessed image output from step S1. It is a dynamic spatial mask matrix that is strictly constrained by the vertical sampling depth. This represents the Hadamard product operation, which multiplies the corresponding elements of a matrix.

[0027] 3) Perform deep encoding of visual features, call the multi-scale convolutional feature extraction operator, perform deep scanning on the extracted region of interest pixel matrix, and obtain compressed redundant image pixels by performing nonlinear feature mapping on the scanning results. Generate a high-dimensional representation that can be used for multimodal fusion, and finally complete the adaptive deep encoding and dimensionality reduction vectorization of visual features, outputting a visual feature vector. The feature encoding process satisfies equation (9): (9) In the formula: This is the encoded visual feature vector of the road surface. It is a non-linear activation function. This represents the total number of multi-scale convolution kernels. For the first Adaptive fusion weights for features at each scale Extract convolution kernels for the corresponding two-dimensional features. This represents a two-dimensional discrete convolution operation. This is the bias matrix for each feature dimension.

[0028] In one possible implementation, step S5 specifically includes the following sub-steps: 1): Construct an adaptive feature gating operator and receive the visual feature vector output from step S4. By evaluating the distribution differences of this feature vector across various sensory channels, a dual spatial and channel enhancement weight matrix is ​​dynamically generated. The weight matrix satisfies equation (10): (10) In the formula: For adaptive enhancement weight matrix, To control the preset nonlinear hyperparameter of the gating activation hardness, The road surface visual feature vector is the input in step S4. The mean of global activation of the feature vector within the current effective sampling period is used as the reference value. A dynamic mean is introduced here. The purpose of using the threshold benchmark is to utilize the nonlinear mapping law to attenuate the weight of conventional background noise below the mean, and correspondingly enhance the activation expression of road surface micro-features above the mean.

[0029] 2) Perform residual gating modulation: Based on the calculated enhancement weight matrix, perform residual modulation processing on the original input features, fuse the gated modulated features with the original input features, and output the final purified enhanced visual feature vector. The feature enhancement process satisfies equation (11): (11) in This is the visual feature vector output after feature enhancement processing. This feature vector is used to pass to step S6 to enter the multimodal alignment stage. The residual structure represents the element-level Hadamard product operation of the feature tensor in the corresponding spatial dimension. The purpose of this residual structure design is to retain the original transmission path of the basic road topology information while improving the weight of high-contribution feature representation, so as to avoid the risk of feature dissipation during the forward propagation of the network and provide a stable visual discrimination benchmark for subsequent multimodal fusion.

[0030] In one possible implementation, step S6 specifically includes the following sub-steps: 1): Sensor effective signal detection: Receive the multimodal guidance parameter package provided in step S3 and the enhanced visual feature vector output in step S5. In order to eliminate the jump interference of discrete sensor signals, the amplitude jump rate of the original data is calculated in real time using a first-order differential operator. If the jump rate exceeds the preset physical boundary threshold, the current sampling point is determined as abnormal noise, and the moving average of the previous effective period is used for smooth replacement, thereby constructing a continuous underlying physical reference feature package.

[0031] 2): Spatial displacement time compensation. There is an inherent longitudinal spatial span between the visual acquisition device and the vehicle contact point. In order to eliminate the time phase difference between the visual forward-looking features and the immediate feedback features under the wheels, the instantaneous driving speed in the multimodal guidance parameter package is combined to dynamically calculate the time deviation caused by the spatial distance. The target reference timestamp compensation equation after displacement compensation satisfies equation (12): (12) In the formula: The target reference timestamp after performing the compensation alignment operation. The original timestamp when the current visual features were captured. The inherent longitudinal physical distance between the pre-calibrated visual acquisition point and the vehicle's front axle contact point. For step S3, the data passed in The instantaneous speed of a vehicle at any given moment.

[0032] 3): Nonlinear time-delay secondary calibration: The chassis actuators and dynamic sensors exhibit nonlinear lag in signal transmission and mechanical response. A time-delay compensation factor is introduced, and secondary time-series calibration is performed on the target reference timestamp to achieve steady-state mapping of cross-modal characteristics in the physical space coordinate system. The reconstructed final synchronized reference timestamp... Satisfying equation (13) (13) In the formula: This is the final synchronization reference timestamp after nonlinear time delay correction; The inherent total time delay constant of sensors and actuators is obtained by dynamically looking up a table based on the current physical conditions of the vehicle.

[0033] 4): Dynamic adjustment of computational load, real-time statistical analysis of the spatial variance fluctuation trend of enhanced visual feature vectors within adjacent sampling periods. When the fluctuation is lower than the preset steady-state threshold, the current road condition is determined to be stable, and the refresh frequency of the spatiotemporal alignment module is actively reduced to suppress computational redundancy. When the fluctuation exceeds the steady-state threshold, the spatiotemporal alignment processing is activated across the entire frequency range to achieve dynamic gating adjustment of the computational load of the vehicle-mounted embedded terminal.

[0034] In one possible implementation, step S7 specifically includes the following sub-steps: 1): Perform environmental prior confidence mapping and retrieve the real-time environmental temperature from the multimodal guidance parameter package passed in step S3. Wiper on status signal To construct a prior reliability mapping operator based on a multidimensional environmental attenuation mechanism, it is necessary to address the fact that frequent operation of windshield wipers caused by precipitation can physically obstruct visual perception, and extreme temperatures can easily cause distortion of road surface optical features (such as interference from icing or fog). Therefore, an exponential penalty model is introduced to quantitatively attenuate the baseline confidence of the visual sensor, and the environmental prior confidence factor is calculated. Its mapping calculation process satisfies equation (14): (14) In the formula: To calculate the environmental prior confidence factor of the output; This represents the baseline visual confidence level under pre-set ideal working conditions. The parameters for real-time wiper operation (the value is positively correlated with the transient intensity of precipitation or snowfall). Real-time ambient temperature; The ideal optical reference temperature with no risk of frost, as pre-calibrated; and These are the pre-calibrated wiper obstruction penalty coefficient and temperature distortion penalty coefficient, respectively. 2) Perform logical conflict detection by projecting heterogeneous features onto a unified metric space using a physical-logic mapping operator, and calculating the L2 distance between the visual enhancement feature vector and the sensor feature vector to quantitatively generate a logical deviation parameter characterizing the degree of cross-modal perception contradiction. The calculation process of the logical deviation parameter satisfies equation (15): (15) In the formula: For cross-modal logic deviation parameter, The enhanced visual feature vector output from step S5 The synchronous sensor feature vector passed in step S3, For visual feature semantic projection function, This is a sensor feature logic mapping function. Here, a projection function is used to map heterogeneous features to the same dimension, realizing a numerical measurement of the logical consistency between visual discrimination trends and underlying physical facts.

[0035] 3): Perform dynamic weight allocation, combining it with the environmental prior confidence factor calculated in step A1. The logical deviation parameter calculated in step A2 The decision contribution of target features is dynamically reconstructed using a nonlinear gated exponential decay operator, generating corrected dynamic weight scores. Its redistribution process satisfies equation (16): (16) In the formula: The corrected target feature weights; This is the preset conflict penalty gain coefficient; Generate instantaneous feature vectors to determine the tolerance threshold for cross-modal logical consistency. The modulation process satisfies equation (17): (17) In the formula: This is the spatiotemporal alignment feature vector output in step S6.

[0036] 4): Invoke the spatiotemporal continuity state smoothing operator and introduce a first-order low-pass time-domain smoothing matrix to perform temporal filtering on the instantaneous eigenvector. To eliminate transient jumps caused by local noise, the final weighted feature vector after conflict elimination is output. The time-domain smoothing process satisfies equation (18): (18) In the formula: This is the final weighted feature vector output at the current sampling time; This is the final weighted feature vector output at the previous sampling time. This is a preset time-domain smoothing coefficient matrix; identity matrix In one possible implementation, step S8 specifically includes the following sub-steps: 1): Perform high-dimensional implicit feature space mapping and receive the weighted feature vector output from step S7. The multi-feature space fusion perception network performs a nonlinear mapping on the feature vector, transforming it from a high-dimensional implicit feature space to a road surface physical state space. To adapt to the comprehensive classification requirements of various complex road conditions, a perception decision model containing multi-level mapping operators and adaptive activation boundaries is constructed. Using the feature projection matrix, the input comprehensive feature vector is transformed into deep decision latent variables representing the joint modality distribution. The latent space mapping process satisfies equation (19): (19) In the formula: Deep decision latent variables generated for feature mapping, To fuse the weight transformation matrix of the sensing network, For the corresponding feature bias vector, The purpose of using this feature projection mechanism, which is a preset nonlinear activation function, is to explore the deep nonlinear correlation between visual enhancement features and physical sensing features under cross-modal effects through matrix transformation in high-dimensional space, so as to provide a decision basis for subsequent state classification.

[0037] 2): Perform joint probability calculation for multiple categories of road surface states. Based on the latent variables generated by the mapping, perform joint probability calculation for multiple categories of road surface states. Use the normalized exponential function to decode the latent variables and calculate the confidence probability score of the current sensing area belonging to each preset physical state category (such as dry, wet, snow, ice, etc.). From this, output the initial recognition probability distribution vector. The probability calculation process satisfies equation (20): (20) In the formula: This is the initial recognition probability distribution vector characterizing the multi-class road surface conditions. and These are the state classification weight matrix and bias vector of the output layer of the perceptron network, respectively. The total number of preset road physical state classification categories. and They correspond to the first The local weight vector and bias term of each road surface condition are calculated, and the output probability distribution vector is used as the initial prediction conclusion and passed to step S9 for physical common sense constraints and final decision calibration.

[0038] In one possible implementation, step S9 specifically includes the following sub-steps: 1) Construct a physical commonsense logic verification rule base, extract the road surface state category corresponding to the peak of the probability distribution vector as the preliminary prediction result, and then synchronously call the underlying physical sensor parameters (such as ambient temperature) processed in step S3. Windshield wiper operating conditions Wheel end slip ratio Lateral acceleration (In conjunction with the anti-lock braking system state), a multi-dimensional macroscopic mechanical boundary review mechanism is established. This involves establishing a logical verification rule base of macroscopic physical common sense and using this rule base to perform objective boundary condition review on the preliminary prediction results, thereby verifying whether the mathematical model output by the perception network conforms to real physical laws.

[0039] The logic verification rule base includes cross-exclusive error correction logic, and its specific judgment conditions include: temperature mutual exclusion verification: if the real-time ambient temperature... Higher than the preset freezing point threshold If the icing condition is not met, the determination of the icing state is rejected; longitudinal slip mutual exclusion check: if the wheel end slip ratio calculated by the chassis is not met, the determination of the icing state is rejected. Breaking the preset slip threshold The conclusion regarding dry, high-adhesion road surfaces is then rejected; Lateral dynamics mutual exclusion verification: if lateral acceleration... If the low-adhesion physical limit is exceeded and the vehicle body does not experience sideslip instability, the icy / snowy road surface classification is deemed invalid; braking performs a mutual exclusion check: if the longitudinal deceleration... If the system is within the preset slight braking range and the anti-lock braking system is abnormally triggered, then the conclusion of the dry state is overturned.

[0040] 2): Execute differentiated decision routing and negative feedback adjustment. Based on the review conclusion of physical logic verification, execute differentiated decision routing. If the preliminary prediction result passes the physical consistency verification of the rule base, that is, there is no logical mutual exclusion between the network prediction state and the real-time multimodal sensing physical quantity, it is confirmed as high-reliability perception, and the final road state recognition result is directly output. Conversely, if the preliminary prediction result is detected to violate the physical common sense boundary, it is determined that there is a local failure in the current fusion perception link, and then the negative feedback adjustment closed loop is triggered. At this time, the abnormal decision link is intercepted, and the feature packet and conflict marker corresponding to the time segment are returned to step S6 to drive the second iteration of data reconstruction and perception weight. The latency of a single closed-loop reconstruction is controlled within 10-80ms and it recovers to the convergence state after a maximum of 5 feature iterations.

[0041] Compared with the prior art, the present invention has the following beneficial effects: 1. By introducing multimodal heterogeneous sensing information, physical parameters such as ambient temperature, wiper condition and adhesion coefficient are deeply integrated with visual perception, which effectively overcomes the perception blind spot and misjudgment risk of a single visual sensor in severe weather.

[0042] 2. By constructing a spatiotemporal alignment mechanism and combining vehicle speed for spatial displacement compensation and nonlinear time delay correction, the spatiotemporal phase difference between visual forward perception and chassis feedback is eliminated, and precise synchronization of cross-modal data under unified physical coordinates is achieved.

[0043] 3. By using a verification rule base based on physical common sense to perform logical verification on the fusion recognition results, and setting common sense constraints such as temperature-phase mutual exclusion conditions and mechanical boundary conditions, it can effectively intercept black box misjudgments caused by relying solely on visual features, and automatically trigger negative feedback reconstruction when logical conflicts are detected, which significantly improves the fault tolerance robustness and closed-loop security under extreme working conditions.

[0044] 4. By statistically analyzing the spatial variance fluctuation trend of visual features within adjacent sampling periods in real time, the calling frequency of the spatiotemporal alignment module and the fusion perception module is dynamically adjusted, realizing adaptive computational load scheduling based on the complexity of the working conditions, thereby reducing computational power consumption while ensuring recognition accuracy. Attached Figure Description

[0045] Figure 1 This is a diagram of the overall logical architecture of the present invention; Figure 2 This is a diagram showing the composition of the core functional modules of the present invention; Figure 3 This is a diagram showing the sub-functional structure distribution of each core functional module of the present invention; Figure 4 This is a flowchart illustrating the core steps of the road condition recognition method of the present invention. Figure 5 This is a graph showing the cross-modal conflict response and dynamic weight time-domain smoothing verification of the present invention. Figure 6 This is a comparison chart of the probability test for road surface condition recognition between the physical constraint-free mechanism of this invention and the method of this invention; Figure 7 This is a local time-domain magnified view of the negative feedback reconstruction and feature convergence process of this invention; Figure 8 This is a verification curve of physical boundary mutual exclusion triggering under the condition of sudden increase in slip ratio of the present invention. Detailed Implementation

[0046] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0047] Specifically, the road state recognition system based on multimodal fusion and physical constraints constructed using this method includes: an onboard vision acquisition device for acquiring road video image streams; a multimodal sensor array for real-time acquisition of heterogeneous data including wiper on / off status, ambient temperature, and road surface adhesion coefficient via a CAN bus; a vision perception processing module for performing image preprocessing and multi-scale feature encoding; an adaptive feature gating module for enhancing key texture components and suppressing background interference; a heterogeneous data spatiotemporal alignment module for achieving spatiotemporal alignment between visual features and sensor feedback; a multimodal conflict detection and dynamic weighting module for evaluating cross-modal logical deviation and dynamically allocating fusion weights; a multi-feature spatial fusion perception network for performing high-dimensional nonlinear mapping and outputting an initial recognition probability distribution; and a physical constraint verification module for performing logical verification and negative feedback reconstruction based on a physical commonsense rule base.

[0048] The multi-feature spatial fusion perception network, the gated sub-network in the adaptive feature gating module, and the logical deviation calculation parameters in the multimodal conflict detection module involved in this invention are all determined through the offline training phase. The following provides a preferred training configuration method, but the scope of protection of this invention is not limited to this specific training process.

[0049] 1.1 Training Data Composition The training dataset consists of two parts: (1) Visual-physical paired samples: Based on RSCD (Road Surface Classification Dataset), vehicle dynamics simulation software is used to synchronously generate corresponding multimodal guidance parameter packages for each frame of road surface image. The dataset contains about 1 million 360x240 pixel road surface image samples, covering six road surface adhesion levels such as dry, wet, water accumulation, new snow, melting snow and ice, as well as four road materials such as asphalt road, cement road, soil / mud road and gravel road. It also labels three road surface unevenness levels: smooth, slightly uneven and severely uneven, which are combined to form 27 refined classification categories.

[0050] Specifically, based on the road surface state label and vehicle motion state corresponding to each frame of image, the sensor feature vector at that moment is generated in reverse using vehicle dynamics simulation software. This vector includes: instantaneous vehicle speed v, ambient temperature T, windshield wiper operating state W, and road adhesion coefficient calculated from the tire model. The final sample format is: (image, [v, T, W, ...) ]) to the road surface status label, where .

[0051] (2) Negative Samples and Boundary Samples: To enhance the model's ability to identify and handle physical conflicts, a batch of counterfactual samples are generated through simulation. For example, an image labeled as frozen is set to an ambient temperature T of 10 degrees Celsius; or an image labeled as dry is set to a different adhesion coefficient. These samples are set to 0.2 and used to train the conflict detection and dynamic weighting module in step S7.

[0052] These counterfactual samples are unlikely to appear or have a low probability in the physical world. They are used to train the multimodal conflict detection module so that it can learn the continuous distribution of cross-modal logical deviations, thereby making a sensitive response to visual-physical contradictions during the inference stage. Boundary samples are generated by setting critical parameters (such as temperature 1℃, μ=0.4, etc.) to calibrate the tolerance threshold of conflict detection and avoid oversensitivity.

[0053] 1.2 Network Structure and Training Strategy The structure of each component is as follows: Visual feature encoder: ImageNet pre-trained ResNet-34 (with the last fully connected layer removed), outputting a 512-dimensional feature vector; Adaptive Feature Gating Module: A lightweight fully connected network that outputs a weight vector with the same dimension as the input features; Multi-feature spatial fusion perception network: 3-layer fully connected network (512 -> 256 -> N), N=5 categories (dry, water accumulation, snow accumulation, ice, others); The activation function used is ReLU, and Dropout (with a ratio between 0.3 and 0.7) is used after the intermediate layers to prevent overfitting.

[0054] 1.3 Loss Function Design The total training loss is composed of three weighted components: Classification loss Using cross-entropy loss, supervise the final output of the fusion network against the true labels.

[0055] Deviation constraint loss Forced logical deviation for normal samples Less than the threshold For counterfactual negative samples, forced If the value exceeds the threshold, a hinge loss is applied.

[0056] Feature consistency loss The visual and sensor features, after spatiotemporal alignment, should be kept consistent in the same latent space, using mean squared error loss.

[0057] Total loss function:

[0058] in , all .

[0059] 1.4 Training Process 1. Input the constructed visual-physical pairing samples into the network.

[0060] 2. Perform steps S1 to S8 sequentially to obtain the initial recognition probability distribution.

[0061] 3. Calculate the total loss. .

[0062] 4. Perform backpropagation to update all weight parameters of the visual encoder, gating module, and fusion perception network.

[0063] 5. Repeat steps 1-4 until the classification accuracy on the validation set no longer improves.

[0064] 1.5 Reference Basis for Physics Common Sense Verification Rule Base The physical common sense logic verification rule library constructed in this invention is based on standard theories in the field of vehicle dynamics (such as the magic formula of tire-road adhesion characteristics, the grading standard of typical road adhesion coefficient, ABS / ESP triggering logic, etc.). Those skilled in the art can establish it themselves based on the publicly available vehicle dynamics principles.

[0065] Example 1: Normal dry road surface condition This embodiment combines Figure 1 The overall logical architecture shown Figure 2 and Figure 3 The core functional module structure, and Figure 4 The method execution flow is described in detail, providing a detailed explanation of the road condition identification process under normal dry road surface conditions. Figure 5 The cross-modal conflict response and dynamic weight smoothing curves shown are consistent with the fusion process in this embodiment.

[0066] This embodiment uses a city expressway under clear daytime weather in winter as the application scenario. The vehicle speed is about 70km / h, the road surface is dry, and the visibility is good. The vehicle-mounted visual acquisition device uses a 2-megapixel CMOS industrial camera (30fps, 1920×1080 resolution, 120dB dynamic range), which is installed on the upper center of the inside of the vehicle's windshield. The ambient temperature sensor transmits data at T=10℃. The domain controller uses NVIDIA DRIVEOrin (AI computing power 254TOPS). The average load when running the algorithm of this invention is about 35%.

[0067] Step 1: Image acquisition and multidimensional preprocessing.

[0068] In this embodiment, the vehicle-mounted high-definition visual acquisition device captures the forward road video stream in real time at a fixed frame rate of 30Hz. The domain controller extracts the original RGB image of the current frame, first converts the RGB image to the HSV color space to achieve decoupling of brightness and chromaticity, and then calls the contrast-limited adaptive histogram reconstruction operator to reconstruct the brightness channel components. The cropping limit value is set to 3.0, and the window size is 8×8 pixels to complete the road surface brightness stabilization process, thereby effectively avoiding the problem of loss of local dark details caused by global processing. Its reconstruction process satisfies equation (1): (1) Then, the brightness-stable image is used as the base image and input into the adaptive bilateral filter operator for adaptive noise reduction processing. While smoothing the noise, the high-frequency edges of the road surface are preserved. The comprehensive filter weight distribution of this operator satisfies equation (2): (2) Among them, the spatial smoothing parameter is set. =15, standard deviation of the range =30, Then, using a multi-scale Laplacian operator, in this embodiment, a three-scale Laplacian operator is selected to perform high-frequency sharpening and detail compensation on the denoised image, as shown in equation (3), and output a standardized road surface preprocessing base map. .

[0069] (3) Among them, setting =1.0, 2.0, 4.0; gain coefficient λ=1.2.

[0070] Step 2: Sensor data synchronization and dynamic ROI extraction.

[0071] In this embodiment, the vehicle synchronously collects underlying physical characteristics in real time via an onboard sensor array and the chassis CAN bus. These characteristics include ambient temperature, windshield wiper status, anti-lock braking system feedback road adhesion coefficient, and instantaneous vehicle speed. Due to the inherent sampling frequency difference between the camera and the CAN bus, the high-frequency discrete sensor signals are linearly interpolated, resampled, and smoothed using a sliding window based on the timestamp of visual acquisition. This ensures strict alignment of heterogeneous data on the time axis. Furthermore, a moving average filter operator is used to remove electromagnetic interference noise from the bus signals during transmission. The reconstructed multimodal guidance parameter package includes: instantaneous vehicle speed. =70km / h, ambient temperature T=10℃, wiper status =0, and the prior adhesion coefficient =0.80; after being linearly mapped to the [0,1] interval by min-max normalization, the parameter packet is then sent to the visual perception layer.

[0072] After receiving the parameter packet, the visual perception layer uses the real-time vehicle speed. Dynamically calculate the longitudinal sampling depth of the road surface under the current working conditions. The forward-looking perception distance is adaptively adjusted, satisfying equation (7): (7) Based on the calculation Generate a boundary-adaptive binary spatial mask matrix. Perform Hadamard product calculations on the standardized pavement preprocessing base map: ; Accurately crop and strip the sky and irrelevant lanes, extract the region of interest on the road surface within the effective driving lanes, and then input the cropped image into a multi-scale convolutional network (for example, in this embodiment, the three convolutional kernels are 3×3, stride 2, and the number of channels are 32, 64, and 128 respectively), and then according to equation (9): (9) Perform feature encoding and output the dimensionality-reduced high-dimensional digital visual feature vector.

[0073] Step 3: Feature enhancement and alignment with heterogeneous spatiotemporal domains.

[0074] The system evaluates the variance of the distribution of visual feature vectors in each perceptual channel, and then... Input the adaptive feature gating module and construct the adaptive enhancement weight matrix according to equation (10). (10) Among them, the nonlinear hyperparameter γ=3.0 is set to control the gating activation hardness, and the high contribution texture components are amplified in a directional manner through the residual modulation mechanism. The enhanced visual feature vector is output according to Equation (11). .

[0075] (11) The visual image captured by the camera represents an advanced perception of the road conditions far ahead, while the chassis wheels are still traveling on the current road segment. Based on this, the system activates the heterogeneous spatiotemporal alignment module to extract the instantaneous vehicle speed from the multimodal guidance parameter package. Distance from inherent longitudinal physical distance The physical displacement condition compensation is calculated according to equation (12): (12) Because the chassis actuators and sensors have inherent nonlinear response time delays, further considering the inherent nonlinear time delay constant of the chassis sensors... A second-order fine calibration is performed according to formula (13).

[0076] (13) Based on the above calculations, the system postpones the advanced visual features captured in the current frame to the moment when the wheel actually rolls over the road surface slice. This operation eliminates the phase difference caused by spatial distance, and completes the timing mapping of heterogeneous modes in the same physical space coordinate system.

[0077] Specifically, the process involves: first, anomaly detection and removal are performed; then, the amplitude change rate of the sensor signal is calculated in real time using a first-order difference operator; and the physical boundary threshold is set to ±3. If the jump rate exceeds the threshold, the moving average of the previous effective period is used instead; then, spatial displacement time compensation is performed. There is an inherent longitudinal physical distance d between the visual acquisition calibration point and the front axle contact point. The time phase difference caused by spatial displacement is calculated in combination with the current instantaneous vehicle speed. Then, nonlinear time delay correction is performed, taking into account the inherent nonlinear time delay constant of the chassis sensor. A second time-series calibration is performed on the target reference timestamp to complete the time-series alignment of visual prospective features and physical sensor feedback features.

[0078] Step 4: Multimodal conflict detection and dynamic weighting.

[0079] In this embodiment, the system enters the quality and reliability assessment and dynamic weighting processing stage based on multimodal conflict detection. First, it retrieves the real-time temperature T=10℃ and wiper operating conditions from the guidance package. =0, calculate the environmental prior confidence factor according to equation (14): (14) The system performs conflict detection through the physical logic mapping operator and calculates the cross-modal logical divergence parameter between visual perception trends and underlying physical facts according to Equation (15).

[0080] (15) In this embodiment, the road surface texture features extracted by the visual network are "dry and rough," which is consistent with the sensor feedback adhesion coefficient μ=0.80 (typical value for dry road surfaces is 0.7-0.9), and the cross-modal logic deviation is... =0.08 (far below the conflict threshold) =0.3), the system calls the first-order low-pass time-domain smoothing operator (cutoff frequency 5Hz) to smooth transient feature jitter and output the final weighted feature vector.

[0081] Step 5: Integrate perception with physical common sense correction.

[0082] Weighted eigenvectors In this embodiment, a multi-feature space fusion perception network is constructed by inputting physical sensing features. The network adopts a two-layer fully connected structure (hidden layer dimension 512→256), ReLU activation function, and Dropout rate of 0.4. After high-dimensional nonlinear mapping, the normalized exponential function is then called to perform feature decoding according to equation (20). (20) Then, the initial recognition probability distribution vector containing multiple road surface conditions is output through the Softmax layer. .

[0083] The physical constraint verification module performs logical checks on the above probability distribution based on a physical common sense rule base. The system sets clear cross-exclusion error correction logic for various working conditions: Specifically, in this embodiment, at the temperature level, if Higher than the preset freezing point threshold The system directly rejects the icing determination; at the longitudinal dynamics level, when the chassis calculates the slip ratio... Breaking the slip threshold At that time, the prediction of a dry, high-adhesion road surface is forcibly rejected. At the lateral force level, as long as the inertial unit feedback... Exceeding the physical limit of low adhesion Furthermore, since the vehicle did not experience sideslip or instability, the system determined that the "icy and snowy road surface" classification was invalid; and in terms of braking feedback, if the longitudinal deceleration... The fact that even a slight braking action triggers the ABS to operate at high frequency indicates that the actual adhesion of the underlying material is extremely poor. Based on this, the system will completely overturn the original conclusion of a dry state. This cross-validation mechanism uses the actual physical boundary feedback of the chassis to block the potential for misjudgment caused by a single visual feature. By setting common sense mutually exclusive conditions, it examines whether the prediction result is incorrectly classified as an icy state when the ambient temperature is significantly higher than the freezing point threshold. This mechanism can effectively intercept the risk of black-box misjudgment caused by relying solely on visual features.

[0084] The peak category of the probability distribution vector is extracted and used as the initial prediction result to input the verification rule base: Current environmental parameters: T=10℃ (>freezing point threshold 0℃), wiper status=0 (off), adhesion coefficient μ=0.80 (normal dry range 0.7-0.9). All parameters do not trigger physical mutual exclusion rules, the initial recognition probability distribution conforms to physical laws, and finally outputs the "dry road surface" recognition result, and links the vehicle electronic stability program (ESP) to perform normal mode state maintenance.

[0085] Example 2: Icy and Snowy Road Conditions This embodiment takes urban expressways under low temperature and snowfall weather in winter as the application scenario. Combined with the actual driving scenario of mixed ice and snow road surface in winter (including hidden "black ice" reflective deception working conditions), this section elaborates on the data flow and workflow of this system under the real vehicle hardware architecture. The algorithm is set to be deployed on the vehicle intelligent driving domain controller. The instantaneous driving speed of the current test vehicle is v=60km / h, the ambient temperature sensor returns data T=-2℃, and the windshield wipers are triggered to turn on due to light snow conditions, and the corresponding status setting signal W=1.

[0086] Step 1: Image acquisition and multidimensional preprocessing.

[0087] The vehicle-mounted high-definition visual acquisition device captures forward road video streams in real time at a fixed frame rate of 30Hz. To address the issues of uneven exposure caused by low light in winter and snow reflections, the domain controller extracts the original RGB image of the current frame, converts it to the HSV color space, and calls the contrast-limited adaptive histogram reconstruction operator to locally enhance and reconstruct the separated luminance channels, thus stabilizing the road surface brightness. This operation effectively avoids the loss of details in local dark areas caused by global processing. The reconstruction process satisfies the following:

[0088] Because photosensitive elements are prone to thermal noise at low temperatures, the system inputs the aforementioned brightness-stabilized image into an adaptive bilateral filter operator for smoothing and noise reduction. The overall filter weight distribution of this operator satisfies:

[0089] After filtering, to compensate for detail dissipation and highlight the microscopic morphology of the "black ice" edges, a multi-scale Laplacian operator is used based on the following formula:

[0090] Perform second-order spatial partial derivative high-frequency sharpening on the base map to output a standardized road surface preprocessed base map. .

[0091] Step 2: Sensor data synchronization and dynamic ROI extraction.

[0092] The vehicle synchronously acquires underlying physical features at a frequency of 100Hz via the chassis CAN bus interface. Since there is an inherent difference in sampling frequency between the camera and the CAN bus, the system uses the timestamp of the visual acquisition as a reference to perform linear interpolation resampling and sliding window smoothing on the high-frequency discrete sensor signals, achieving strict alignment of heterogeneous data on the time axis. The reconstructed multimodal guidance parameter package includes: instantaneous vehicle speed. =60km / h, ambient temperature T=-2℃, wiper status =1 and prior adhesion coefficient =0.25, After receiving the parameter packet, the vision control core uses the real-time vehicle speed Dynamically calculate the longitudinal sampling depth of the road surface under the current working conditions. Adaptively adjust the forward-looking perception distance, satisfying the following:

[0093] Based on the calculation Generate a boundary-adaptive binary spatial mask matrix. Then, the Hadamard product operation is performed:

[0094] The system precisely crops and removes the sky and irrelevant lanes, extracts the region of interest (ROI) on the road surface within the valid driving lanes, and inputs this matrix into a multi-scale convolutional network to perform feature encoding according to the following formula.

[0095] Output the high-dimensional digital visual feature vector after dimensionality reduction.

[0096] Step 3: Feature enhancement and alignment with heterogeneous spatiotemporal domains.

[0097] The system evaluates the distribution variance of visual feature vectors in each sensory channel. In order to enhance the dark light specular scattering characteristics of the concealed black ice, an adaptive enhancement weight matrix is ​​constructed according to the following formula.

[0098]

[0099] High-contribution texture components are amplified in a targeted manner using a residual modulation mechanism, and the enhanced visual feature vector is output according to the following formula. .

[0100]

[0101] The visual image captured by the camera represents an advanced perception of the road conditions far ahead, while the chassis wheels are still traveling on the current snowboard section. Based on this, the system activates the heterogeneous spatiotemporal alignment module to extract the instantaneous vehicle speed from the multimodal guidance parameter package. Distance from inherent longitudinal physical distance The physical displacement condition compensation is calculated based on the following formula:

[0102] Because the chassis actuators and sensors have inherent nonlinear response time delays, further considering the inherent nonlinear time delay constant of the chassis sensors... A second-order fine-tuning of the timing is performed according to the following formula.

[0103]

[0104] Based on the above calculations, the system postpones the advanced visual features captured in the current frame to the moment when the wheel actually rolls over the road surface slice. This operation eliminates the phase difference caused by spatial distance, and completes the timing mapping of heterogeneous modes in the same physical space coordinate system.

[0105] Step 4: Multimodal conflict detection and dynamic weighting.

[0106] The system enters the quality and reliability assessment and dynamic weighting processing stage based on multimodal conflict detection. First, it retrieves the real-time temperature T=-2℃ and wiper operating conditions from the guidance package. =1, calculate the environmental prior confidence factor according to the following formula:

[0107] The diffuse reflection from the black ice surface easily interferes with the visual network, causing it to extract incorrect texture features classified as "dry road surface," while the measured adhesion coefficient of the underlying layer... The feedback indicates a low-attachment state. The system performs conflict detection through the physical logic mapping operator and calculates the cross-modal logical divergence parameter between the visual perception trend and the underlying physical facts according to the following formula.

[0108]

[0109] When detected Exceeding the tolerance threshold The system triggers nonlinear gated exponential decay, and the decision contribution is reconstructed according to the following formula:

[0110] Through reconstruction calculation, the weights of conflict features The weights are significantly reduced after calculation. The system uses these weights to perform element-wise multiplication modulation on the feature vectors, generating instantaneous feature vectors. Based on the following formula, a first-order low-pass time-domain smoothing operator is invoked to suppress transient jitter, ultimately outputting a final weighted feature vector that eliminates logical conflicts. .

[0111]

[0112] Step 5: Integrate perception with physical common sense correction.

[0113] Multi-feature spatial fusion sensing network receives weighted feature vectors The system performs latent variable feature projection transformation within a unified depth decision space, then calls the normalized exponential function to perform feature decoding according to the following formula, outputting an initial recognition probability distribution vector containing multiple road surface states. .

[0114]

[0115] The system performs closed-loop correction of physical constraints, extracts the peak category of the probability distribution vector, and inputs it as the preliminary prediction result into the verification rule base. The system then verifies the current ambient temperature. Confirm that the value is below the set freezing point threshold. ; Real-time wheel end slip ratio at this time Lateral acceleration of the vehicle body Neither the ABS feedback state nor the aforementioned mechanical mutual exclusion boundary was triggered, thus proving that the preliminary prediction results have passed the cross-verification of multi-dimensional macroscopic physical common sense. The current fused perception data is highly reliable. In the case of occasional strong reflective black ice causing the visual network to output abnormal sampling frames that violate common sense, the system will quickly intercept and initiate negative feedback reconstruction according to the above mutual exclusion conditions. Relying on the parallel computing power of the domain controller, the system can force the feature to reconverge within 10-80ms and at most n iterations. In this embodiment, the system can force the feature to reconverge within 30ms and at most 3 iterations. Regardless of normal judgment or through negative feedback reconstruction, the system sends the final identification result of "black ice road surface" to the chassis drive-by-wire unit and links the vehicle electronic stability program (ESP) to perform active torque limiting and braking prefill, thereby achieving highly reliable closed-loop intervention in complex road conditions.

[0116] like Figure 6 As shown, without physical constraints, the visual network misclassifies black ice as dry road surface (probability > 0.8), while the method of this invention successfully corrects this after physical mutual exclusion verification. Figure 7 It demonstrates that the features converge to a stable state within 30ms and after 3 iterations during the negative feedback reconstruction process; Figure 8 It was verified that the physical boundary mutual exclusion condition was correctly triggered when the slip ratio suddenly increased.

[0117] Example 3 3. Anomaly Handling Mechanism and Supplementary Physical Verification Rules 3.1 Sensor Failure Degradation Logic When any sensor malfunctions or its output signal exceeds the physically reasonable range, the system automatically executes a degradation strategy to ensure the stability of the identification process. The specific degradation logic is as follows: For temperature sensors, if the measured ambient temperature T < -50°C Or T>80 If the sensor fails, it is considered to be faulty, and a preset default ambient temperature value, such as T=20, is used. As an alternative.

[0118] For the wiper status signal, if the signal is lost or remains unchanged for multiple consecutive frames for a duration exceeding a preset threshold (e.g., t=10s), its confidence factor is set to the default value. The weight of this signal is reduced to 0.5, and its weight is further reduced in subsequent fusion calculations.

[0119] For vehicle speed signals, if the vehicle speed jumps to zero or the rate of change between adjacent frames exceeds a preset physical limit, for example... m / If the vehicle speed value of the previous valid frame is used for extrapolation compensation until the signal returns to normal.

[0120] Regarding the road surface adhesion coefficient, if the adhesion coefficient feedback value obtained via the CAN bus exceeds 0.1... If the physical range is reasonable, then the default value will be used. 0.5 and mark the feature of this dimension as untrustworthy.

[0121] The aforementioned degradation processing is performed in real time in the sensor effective signal detection sub-step of step S6 to ensure that multimodal fusion will not collapse due to the failure of a single sensor.

[0122] 3.2 Frequency Down-Synchronization Processing In step S6, the dynamic adjustment of the computational load will reduce the refresh frequency of the spatiotemporal alignment module according to the stability of the operating conditions. During this frequency reduction period, step S7 adopts a maintenance strategy: if the current frame does not trigger the alignment update of step S6, then step S7 inherits and uses the alignment parameters of the previous frame, including the spatial displacement compensation amount and the time delay correction value; at the same time, the inference frequency of step S7 is reduced synchronously with step S6, and does not run at the full frame rate alone, so as to avoid generating invalid conflict detection calculations and ensure further saving of the embedded platform's computing resources while maintaining the fusion effect.

[0123] 3.3 Iteration Limitations of Negative Feedback Reconstruction When step S9 triggers the negative feedback adjustment mechanism and reverts the feature packet back to step S6 for reconstruction, the following iteration constraints apply: a maximum of three feature iterations are performed during a single negative feedback reconstruction process, and the time t for each iteration is controlled within a specified timeframe. 10ms Within a 35ms range; if the feature does not converge to a stable state within the limited number of iterations, i.e., the cross-modal logic deviation changes less than a preset threshold for two consecutive iterations, the current iteration result is output and marked as a low-confidence state for reference by the upper-level decision module; to reduce the computational load of a single iteration, the iteration process can bypass the nonlinear time-delay secondary calibration submodule in step S6 and only perform spatial displacement compensation; thus ensuring the real-time performance and system stability of the negative feedback closed loop.

[0124] 3.4 Supplementary Physical Verification Rules 3.4.1 Supplementary Verification Rules for Black Ice Scene In low-temperature winter environments, black ice roads visually resemble dry asphalt, and when vehicles are traveling at a constant speed, it is difficult to trigger mutually exclusive rules through slip ratio or anti-lock braking system. Therefore, a supplementary verification rule, denoted as rule R4, is added to the physical common sense verification rule base in step S9, which is the suspected low adhesion alarm rule.

[0125] The rule is triggered when the following three conditions are met simultaneously: ambient temperature T < 3°C. The degree is the threshold near the freezing point; in the initial recognition probability distribution output by step S8, dry road surface is the peak category; the wheel speed sensor shows no abnormal fluctuations, meaning the slip ratio is calculated in real time. <3%.

[0126] Upon triggering this alarm, the system does not directly output "dry road surface," but instead outputs "potentially low-friction road surface," and may optionally perform one of the following actions: issue a warning message to the driver, or automatically apply a deceleration 'a' by the autonomous driving system. A tiny braking pulse of 0.1g is applied to stimulate the slip ratio, and the low adhesion state is then confirmed based on the kinetic response.

[0127] This rule has a lower priority than hard mutual exclusion rules such as those that have triggered anti-lock braking systems, longitudinal slip mutual exclusion, or lateral dynamic mutual exclusion, but a higher priority than the results of pure vision-based single-modal recognition.

[0128] 3.4.2 Optimization for low-light nighttime scenes To address the issue of road surface texture feature attenuation caused by low-light environments such as nighttime or tunnels, an adaptive enhancement mechanism based on illumination is added to the preprocessing stage in step S1.

[0129] First, the ambient light intensity is estimated in real time by reading the exposure time and ISO gain value of the camera sensor. When the light intensity is lower than a preset threshold, i.e., low light conditions, the cropping and limiting parameters of the contrast adaptive histogram equalization operator in step S1 are adjusted. From 0.2 dynamically reduced to To avoid over-enhancing noise, a lightweight low-light pre-enhancement network is added. This network is built based on Retinex theory, adopts a single-layer convolutional structure, has fewer than 20,000 parameters, and is pre-trained on the nighttime subset of the road surface classification dataset. This network is executed after brightness stabilization in step S1 and before adaptive noise reduction. The actual processing time per frame on the NVIDIA DRIVE Orin platform does not exceed two milliseconds. Through the above optimizations, the road surface texture discernibility under low-light conditions at night is significantly improved, thereby reducing the dependence of multimodal fusion on feedback from a single physical sensor.

[0130] 4. Experimental Design and Data for Blank Control Group To verify the technical effect of the method of the present invention, the following three sets of control experiments were designed. Each set of test samples consisted of 1,000 independent test images (from the independent test set of the RSCD dataset, not the training set samples), covering five road surface conditions: dry, waterlogged, snow-covered, icy, and slippery, with 200 images for each condition.

[0131] 4.1 Experimental group design Control group A (pure visual baseline): only ResNet-34 visual encoder was used for single-modal road classification, without multimodal fusion or physical constraints.

[0132] Control group B (multimodal without physical constraints): uses simple fusion of vision and sensor multimodal (feature stitching + full connection), without spatiotemporal alignment, conflict detection, or physical constraint verification.

[0133] This invention group: Complete use of all steps S1 to S9.

[0134] 4.2 Test Sample Distribution Normal operating condition samples (clear weather, normal road surface): 400 images / set (200 dry images + 200 wet images) Severe working conditions sample (rain, snow, ice, low light, sensor noise): 400 images / set (100 images of water accumulation + 100 images of snow accumulation + 200 images of ice accumulation) Boundary condition samples (black ice, partial sensor failure, nighttime): 200 images / set 4.3 Evaluation Indicators Overall accuracy, recall per class, false positive rate, and average processing time per frame.

[0135] 4.4 Experimental Conditions Hardware platform: NVIDIA DRIVE Orin (254 TOPS) Software framework: PyTorch 2.0 + TensorRT 8.5 Visual input resolution: 360×240 (consistent with the RSCD dataset) CAN bus sampling frequency: 100 Hz Visual capture frame rate: 30 fps 4.5 Simulation Results: Table 1

[0136] The above experimental data are obtained based on simulation verification of the method of the present invention on a test subset of the RSCD dataset. They are used to illustrate the performance improvement trend of the present invention compared with existing methods. The actual deployment effect may vary due to factors such as hardware platform, sensor accuracy, and training data distribution. This does not constitute an absolute guarantee of actual performance, but it does not affect the core beneficial effect of the present invention.

[0137] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A road state recognition method based on multimodal fusion and physical constraints, characterized in that, Includes the following steps: S1: Road surface perception information acquisition and feature extraction: Obtain road video image stream through vehicle-mounted vision acquisition device, perform brightness stabilization, adaptive noise reduction and texture enhancement preprocessing on the image stream, and output a standardized road surface preprocessing base map; S2: Synchronous acquisition of multimodal heterogeneous signals: Through vehicle sensors and CAN bus, heterogeneous characteristic signals including wiper on status, ambient temperature and road adhesion coefficient are acquired in real time. S3: Multimodal guidance parameter encapsulation and transmission: The heterogeneous feature signals obtained in step S2 are time-stamped and reconstructed, encapsulated into multimodal guidance parameter packets, and sent to the visual perception layer; S4: Adaptive localization and feature encoding of visual perception region: The visual perception layer receives the real-time vehicle speed signal in the multimodal guidance parameter package, dynamically corrects the longitudinal sampling depth according to the real-time vehicle speed signal, crops the standardized road preprocessing base map, and extracts multi-scale convolutional features to generate a visual feature vector. S5: Perceptual Feature Adaptive Gating Enhancement: Input the visual feature vector generated in step S4 into the adaptive feature gating module, construct an adaptive feature gating operator, amplify the weight of key texture components and suppress irrelevant background features according to the operator, and output the enhanced visual feature vector. S6: Heterogeneous data cross-modal spatiotemporal alignment: Using the heterogeneous data spatiotemporal domain alignment module, the multimodal guidance parameter package and enhanced visual feature vector are received, anomaly detection and deletion are performed on the data input by the vehicle sensor, and spatial displacement time compensation and nonlinear time delay correction are performed in combination with vehicle speed to achieve temporal alignment of visual forward features and sensor feedback features. S7: Multimodal conflict detection and dynamic weighted fusion: The multimodal conflict detection module receives time-aligned data, calculates the environmental prior confidence by combining the wiper on state and ambient temperature, calculates the cross-modal logical divergence between visual features and physical sensing facts, reallocates fusion weights based on the cross-modal logical divergence and the environmental prior confidence to suppress conflicting features, and eliminates transient feature jitter through a state smoothing operator, outputting the final weighted feature vector; S8: Multi-feature space fusion perception: Input the weighted feature vector output in step S7 and the physical sensing features into the multi-feature space fusion perception network, perform high-dimensional nonlinear mapping through the multi-feature space fusion perception network, and output the initial recognition probability distribution vector of the road surface state. S9: Physical constraint verification and negative feedback reconstruction: The initial recognition probability distribution vector output in step S8 is logically checked based on the physical common sense verification rule base through the physical constraint mechanism. If it conforms to physical laws, the final road state recognition result is output. If a logical violation is detected, the negative feedback adjustment mechanism is triggered to return the contradictory information to step S6 for data reconstruction processing. The data reconstruction processing is completed within the preset number of iterations and time delay threshold.

2. The road state recognition method based on multimodal fusion and physical constraints according to claim 1, characterized in that: The road surface perception information acquisition and feature extraction in step S1 includes the following steps: 1) Brightness stabilization processing: The RGB image captured by the current vehicle vision acquisition device is converted to the HSV color space to achieve decoupling of brightness and chromaticity. The brightness channel is reconstructed by limiting the contrast adaptive histogram equalization operator, and the brightness-stabilized image is output. 2) Adaptive noise reduction processing: The brightness-stabilized image is used as the base image and input to the adaptive bilateral filter operator to smooth the noise while preserving the high-frequency edges of the road surface, and output a noise-reduced image; 3): Texture enhancement processing: Introduce a multi-scale Laplacian operator to perform high-frequency sharpening and detail compensation on the denoised image, and output the standardized road surface preprocessing base map.

3. The road state recognition method based on multimodal fusion and physical constraints according to claim 2, characterized in that: The synchronous acquisition of multimodal heterogeneous signals in step S2 specifically includes the following steps: 1) Real-time acquisition of heterogeneous data: Through the vehicle sensor array and CAN bus interface, heterogeneous data streams containing ambient temperature components, wiper working status components, current road surface adhesion coefficient feedback components, and instantaneous vehicle speed components are captured in real time and constructed into an initial heterogeneous feature vector. 2) Clock alignment and frequency resampling: Based on the visual acquisition frequency, linear interpolation compensation is performed on the asynchronous input sensor signal to ensure that each frame of video image can be associated with the physical feature components at the corresponding time. The moving average filter operator is used to remove random electromagnetic interference noise in the transmission process of the bus signal to obtain the smoothed sensor feature vector. 3): Sensor data dimension normalization and feature space mapping: The min-max normalization algorithm is used to perform linear mapping on the smoothed data, and its numerical range is uniformly scaled to a preset interval to generate standardized sensor feature components.

4. The road state recognition method based on multimodal fusion and physical constraints according to claim 3, characterized in that: The encapsulation and transmission of multimodal guidance parameters in step S3 includes the following steps: 1) Heterogeneous data timestamp synchronization: Based on the sampling timestamp of the video frame, interpolation and alignment compensation are performed on asynchronous input sensor signals to establish a strict mapping relationship between single-frame road surface images and transient physical features on the time axis. 2) Feature information format reconstruction: The synchronized physical sensing data is standardized and reconstructed into a guide instruction format compatible with the visual perception module; 3): Establishment of real-time cascaded transmission link: Relying on the internal high-speed data bus, the encapsulated multimodal guidance parameter package is sent to step S4 to ensure that step S4 can simultaneously extract the vehicle speed and environmental parameters associated with the current video frame at the moment of reading the current video frame.

5. The road state recognition method based on multimodal fusion and physical constraints according to claim 4, characterized in that: The adaptive localization and feature encoding of the visual perception region in step S4 includes the following steps: 1): Adaptive adjustment of longitudinal sampling depth: Extract the real-time vehicle speed signal from the multimodal guidance parameter package, combine it with the physical installation parameters of the image acquisition device to delineate the basic field of view frame, and use real-time vehicle speed feedback to dynamically calculate the longitudinal sampling depth of the road surface to adaptively adjust the perception distance. 2): Region of Interest Extraction: Based on the longitudinal sampling depth, a boundary-adaptive binary spatial mask matrix is ​​generated in the two-dimensional image coordinate system. This matrix is ​​then multiplied with the preprocessed base image to extract the pixel matrix of the region of interest on the road surface. 3): Multi-scale feature encoding: The multi-scale convolution feature extraction operator is called to perform depth scanning and non-linear feature mapping on the pixel matrix of the region of interest of the road surface to generate a visual feature vector.

6. The road state recognition method based on multimodal fusion and physical constraints according to claim 5, characterized in that: The adaptive gating enhancement of the perception features in step S5 specifically includes: constructing an adaptive feature gating operator, dynamically generating a spatial and channel dual enhancement weight matrix by evaluating the distribution differences of the visual feature vector in each perception channel, using nonlinear mapping law to attenuate the conventional background noise weight below the mean and enhance the activation expression of road surface micro-features above the mean, and then outputting the enhanced visual feature vector after performing residual modulation processing on the original input features.

7. The road state recognition method based on multimodal fusion and physical constraints according to claim 6, characterized in that: The heterogeneous data cross-modal spatiotemporal alignment in step S6 includes the following steps: 1) Sensor signal detection: Effective signal detection is performed on the heterogeneous data input by the sensor. The amplitude jump rate of the original sensor signal is calculated in real time through the first-order differential operator. If the jump rate exceeds the preset physical boundary threshold, it is determined to be abnormal noise interference. The moving average value of the preceding effective period is used to eliminate and smooth the abnormal noise interference, thereby constructing a continuous underlying physical reference feature. 2): Spatial displacement compensation: Combining the instantaneous driving speed in the multimodal guidance parameter package, dynamically calculate the time phase difference caused by the spatial displacement between the vehicle-mounted visual acquisition device and the vehicle contact point, and perform timing alignment based on displacement compensation; 3): Nonlinear time delay secondary correction: For the inherent nonlinear time delay of chassis actuators and dynamic sensors in signal transmission and mechanical response, a time delay compensation factor is introduced to perform secondary time-series calibration, and complete the time-series mapping of heterogeneous modes in the same physical space coordinate system; 4) Dynamic gating of computational load: Real-time statistical analysis of the spatial variance fluctuation trend of enhanced visual feature vectors within adjacent sampling periods. If the current fluctuation is lower than the preset steady-state threshold, the refresh call frequency of the heterogeneous data cross-modal spatiotemporal alignment module is reduced. If the fluctuation is higher than the steady-state threshold, the module is activated at full frequency to achieve dynamic adjustment of the computational load of the vehicle-mounted embedded terminal.

8. The road state recognition method based on multimodal fusion and physical constraints according to claim 7, characterized in that: The multimodal conflict detection and dynamic weighted fusion in step S7 includes the following steps: 1) Environmental prior confidence assessment: The real-time temperature and wiper conditions in the multimodal guidance parameter package are retrieved to construct a prior reliability mapping operator based on a multidimensional environmental attenuation mechanism, and an exponential penalty model is introduced to quantitatively attenuate the basic confidence of the vision sensor to obtain the environmental prior confidence factor. 2): Cross-modal logic divergence calculation: Calculate the cross-modal logic divergence parameter between visual perception trends and underlying physical facts through a physical logic mapping operator; 3): Conflict feature suppression: When the cross-modal logic deviation exceeds the preset tolerance threshold, a nonlinear gated exponential decay mechanism is triggered to reconstruct the decision contribution of conflict features. The original fused feature vector is dynamically redistributed and fused weights are applied through element-level multiplication modulation to suppress conflict features. 4) State smoothing output: Suppress transient feature jitter by using a first-order low-pass time-domain smoothing operator to output the final weighted feature vector.

9. The road state recognition method based on multimodal fusion and physical constraints according to claim 8, characterized in that: The multi-feature space fusion perception in step S8 includes: projecting the weighted feature vector and physical sensing features onto a unified depth decision space, performing latent variable feature projection transformation, and then decoding the features through a normalized exponential function to output an initial recognition probability distribution vector containing multiple road surface states.

10. The road state recognition method based on multimodal fusion and physical constraints according to claim 9, characterized in that: The physical constraint verification and negative feedback reconstruction include the following steps: 1) Construct a logical verification rule base for physical common sense, extract the road surface state category corresponding to the peak of the probability distribution vector as the preliminary prediction result, and then, by synchronously calling the underlying physical sensing parameters processed in step S3, establish a multi-dimensional macroscopic mechanical boundary review mechanism, that is, establish a logical verification rule base for macroscopic physical common sense and use the rule base to perform objective boundary condition review on the preliminary prediction result to check whether the mathematical model output by the perception network conforms to the real physical laws. 2): Execute differentiated decision routing and negative feedback adjustment. Based on the review conclusion of physical logic verification, execute differentiated decision routing. If the preliminary prediction result passes the physical consistency verification of the rule base, that is, there is no logical mutual exclusion between the network prediction state and the real-time multimodal sensing physical quantity, it is confirmed as high-reliability perception, and the final road state recognition result is directly output. Otherwise, if the preliminary prediction result is detected to violate the physical common sense boundary, it is determined that there is a partial failure in the current fusion perception link, and then the negative feedback adjustment closed loop is triggered. At this time, the abnormal decision link is intercepted, and the feature packet and conflict marker corresponding to the time segment are returned to step S6.

Images