Off-road Surface Classification System Based on Deep Fusion of Dynamics and Vision

Abstract

This invention relates to a road surface classification system, and more particularly to an off-road road surface classification system based on deep fusion of dynamics and vision, comprising a multi-source data synchronization module, a deep fusion evaluation module, and an adaptive decision-making module. The multi-source data synchronization module ensures precise temporal synchronization between visual images and dynamic data through timestamp alignment and sliding window matching, serving as input to the deep fusion evaluation module. The deep fusion evaluation module designs a dual evaluation system: visual temporal continuity evaluation assesses the stability of the detection and segmentation results in the time dimension, and dynamic consistency evaluation verifies the degree of consistency between visual perception and actual vehicle response. The two are weighted and fused to form a final quality score. The adaptive decision-making module dynamically adjusts the fusion strategy based on real-time evaluation results, prioritizing dynamics at high confidence levels and increasing visual weight at low confidence levels, achieving intelligent complementary advantages of sensors.

Description

Technical Field

[0001] This invention relates to a road surface classification system, and more particularly to an off-road road surface classification system based on the deep fusion of dynamics and vision. Background Technology

[0002] With the development of intelligent driving vehicles, their application scenarios are becoming increasingly complex and extreme. Off-road vehicles need to navigate smoothly for extended periods on various road surfaces, including asphalt, dirt, sand, snow, rocks, brick, and gravel. Perception of road environment information, such as road type identification and road elevation information recognition, is gradually becoming an important research area for intelligent driving. Therefore, accurately identifying environmental information such as road type and unevenness is crucial for the autonomous driving functions of off-road vehicles, including intelligent decision-making and planning, and assessment of driving capability risks.

[0003] Currently, the road perception of autonomous vehicles in off-road environments mainly relies on the fusion of information from visual sensors and vehicle dynamics sensors. Visual sensors (such as cameras) can provide rich information on road texture and color, and have good foresight, but are sensitive to environmental interference such as lighting and weather. Dynamics sensors (such as IMUs and wheel speed sensors) can realistically reflect the physical interaction between the vehicle and the road surface and have high robustness, but their perception lacks foresight, and the recognition results are often lagging. Although existing methods attempt to fuse the two, they mostly adopt preset, fixed-strategy fusion mechanisms, failing to fully leverage the deep complementary advantages of the two types of sensors at the feature level. Summary of the Invention

[0004] To address the aforementioned technical challenges, this invention proposes an off-road surface classification system based on deep fusion of dynamics and vision. The system comprises a multi-source data synchronization module, a deep fusion evaluation module, and an adaptive decision-making module. The multi-source data synchronization module ensures precise temporal synchronization between visual images and dynamic data through timestamp alignment and sliding window matching, serving as input to the deep fusion evaluation module. The deep fusion evaluation module employs a dual evaluation system: visual temporal continuity evaluates the stability of the detection and segmentation results over time, while dynamic consistency evaluates the degree of agreement between visual perception and actual vehicle response. These two evaluations are weighted and fused to form the final quality score. The adaptive decision-making module dynamically adjusts the fusion strategy based on real-time evaluation results, prioritizing dynamics at high confidence levels and increasing visual weight at low confidence levels, thus achieving intelligent complementary advantages of the sensors.

[0005] Furthermore, a multi-source data synchronization module is used to synchronize the vehicle chassis signal and binocular camera signal required for off-road surface classification. The multi-source data synchronization module includes: a data input layer, a core processing layer, and an output interface layer.

[0006] The data input layer receives visual data streams, dynamic data streams, and raw timestamps. The visual data streams are continuous visual image sequences from onboard visual sensors, containing road texture and color information. Each frame is accompanied by a timestamp for subsequent timestamp alignment. The dynamic data streams are vehicle status information output from dynamic sensors on the vehicle's CAN bus, including IMU (Inertial Measurement Unit), wheel speed sensors, and steering angle sensors. Vehicle status information is acquired at a higher frequency, reflecting the physical interaction between the vehicle and the road surface. The raw timestamps are time stamps inherent to the data from each sensor, based on a unified system clock, providing a reference for timestamp alignment.

[0007] The core processing layer comprises three parts: timestamp alignment calculation, sliding window matching mechanism, and anti-duplicate matching mechanism. The timestamp alignment calculation establishes a unified time reference system, converting the timestamps of the visual and dynamic sensors into absolute system time, and uses linear interpolation to resample high-frequency dynamic data, synchronizing the dynamic data time points with the visual data time points. For the visual data time points... and dynamic data time points The synchronization condition is:

[0008] (1)

[0009] in, This represents the maximum permissible time difference.

[0010] The sliding window matching mechanism is used to maintain a fixed-size data buffer window to solve the problem of data flow rate mismatch; the visual data window capacity is 10-20 frames, corresponding to a time range of 1-2 seconds; the dynamic data window capacity is 50-100 data points, corresponding to a time range of 0.5-1 seconds; within the window, the unmatched dynamic data with the closest time is found for each visual data, forming a one-to-one visual-dynamic data combination;

[0011] The anti-duplicate matching mechanism ensures that each piece of data is used only once by generating a unique identifier for each input data and recording the matching history. The unique identifier includes a timestamp, sensor ID, and serial number. A hash table is used to store the information of matched visual-dynamic data combinations, and the validity period of the matching records is set, with expired records being cleaned up periodically.

[0012] The output interface layer comprises three parts: aligned data pairs, time difference statistics, and quality assessment. The aligned data pairs, through alignment in the frequency and time domains, ensure that the visual-dynamic data combinations formed after time synchronization processing by the core processing layer are precisely time-corresponding, providing a foundation for subsequent fusion processing. The time difference statistics record the actual time difference of each successfully matched visual-dynamic data combination for system performance monitoring and quality assessment. The quality assessment performs real-time quality evaluation of the time synchronization processing results of the core processing layer, including indicators such as dynamic confidence and the number of road surface pixels.

[0013] Furthermore, the parameters configured for the multi-source data synchronization module include:

[0014] Time window size: Defines the time range of the sliding window in the sliding window matching step in the core processing layer. By conducting experiments with different window sizes, the matching success rate and latency are compared to balance real-time performance and data coverage integrity.

[0015] Maximum time difference: Sets the maximum allowable time difference for data matching. The setting is based on the minimum sampling frequency of the visual sensor and the maximum sampling frequency of the dynamic sensor to ensure that the matching error does not exceed the time interval of one frame of visual data, thus ensuring synchronization accuracy.

[0016] Cache capacity: Controls the cache size of each data stream. Based on the system, a dynamic caching strategy is adopted. When the cache utilization rate is >80%, the earliest unmatched data is prioritized for elimination to ensure stable system operation and optimize memory usage efficiency.

[0017] The performance monitoring includes:

[0018] Synchronization success rate: The percentage of successfully matched visual-dynamic data combinations out of the total data volume;

[0019] Mean alignment error: Calculate the average time difference of all matched visual-dynamic data combinations;

[0020] Matching rate: measures the system's ability to process data matching per unit of time. Matching efficiency = number of successfully matched pairs / total theoretically possible number of matches.

[0021] Furthermore, the input data of the deep fusion evaluation module includes aligned visual data, aligned dynamic data, and historical frame data. The aligned visual data is visual image data from the multi-source data synchronization module, which has been precisely aligned with the dynamic data in time and includes pixel-level classification results output by the semantic segmentation network. The aligned dynamic data is vehicle state information synchronized from the multi-source data synchronization module, including road surface type recognition results and confidence scores, providing physical supervision signals for consistency evaluation. The historical frame data stores the visual segmentation results of the most recent few frames for temporal continuity analysis.

[0022] The input data is processed and analyzed by the evaluation calculation layer, which includes three parts: visual temporal continuity evaluation, dynamic consistency evaluation, and dual reward fusion.

[0023] The visual temporal continuity evaluation is used to quantify the stability of semantic segmentation results in the temporal dimension. The reward for visual temporal continuity evaluation mainly consists of two parts: a base reward (ensuring sufficient road surface information in the image for analysis) and a temporal reward (rewarding segments similar to those in previous frames). for:

[0024] = base_reward1 + 0.4 ×temporal_reward (2)

[0025] Base_reward1 is the basic visual reward, which reflects the effectiveness evaluation and requires at least 1000 road surface pixels to be evaluated effectively; temporal_reward is the temporal reward, which reflects the road surface type similarity evaluation, and the higher the similarity, the higher the reward.

[0026] The dynamic consistency evaluation is used to quantify the degree of consistency between visual segmentation results and dynamic recognition results. Based on a preset visual-dynamic mapping relationship, 17 types of visual segmentation results are mapped to 6 types of dynamic road surface types. The 17 types of visual segmentation results include background, buildings, dry concrete road surface, dry asphalt, trees, vehicles, people, shrubs, sky, dirt road, snow, sand, wet ground, rock, brick road, puddles, and gravel road. The 6 types of dynamic road surface types include normal road surface, snow, gravel, grass, mud, ruts, sand, rock road, and wading road. The consistency judgment is weighted according to the dynamic confidence level.

[0027] =base_reward2×confidence_weight (3)

[0028] in, The base_reward2 is the basic reward for consistency evaluation of dynamics, which is assigned a value based on whether the dynamics and visual results are consistent; confidence_weight is the confidence weight of dynamics, which is dynamically adjusted according to different confidence levels. The confidence weight of dynamics is set to 1, 2, and 3 points respectively for low, medium, and high.

[0029] The dual-reward fusion intelligently integrates the results of visual temporal continuity evaluation and dynamic consistency evaluation, dynamically adjusting the weights of the two evaluation indicators based on the proportion of road surface pixels: when the proportion of road surface pixels to the total number of pixels is sufficient (>30%), emphasis is placed on dynamic consistency; when the proportion of road surface pixels to the total number of pixels is insufficient (≤30%), emphasis is placed on visual temporal continuity; the total reward after fusion... The calculation formula is:

[0030] (4)

[0031] in and The dynamic weights for visual temporal continuity evaluation and dynamic consistency evaluation are respectively, satisfying... + =1;

[0032] The output of the deep fusion evaluation module includes a total reward score, sub-item rewards, and a quality assessment. The total reward score is the comprehensive quality score after fusion, reflecting the reliability of the current perception results. The sub-item rewards are detailed scores for visual temporal continuity evaluation and dynamic consistency evaluation. The quality assessment is a comprehensive evaluation of the current data quality, including dynamic confidence and the number of road surface pixels.

[0033] Furthermore, the deep fusion evaluation module also includes a mapping knowledge base and an anomaly detection module. The mapping knowledge base includes visual-dynamic mapping relationships, confidence weights, and scene rules. The visual-dynamic mapping relationships store the correspondence between 17 visual segmentation categories and 6 dynamic road surface types, providing a semantic basis for the consistency evaluation of the evaluation calculation layer. The confidence weights define the credibility of dynamic results under different road surface conditions and are used for weighted processing during reward calculation in the evaluation calculation layer. The scene rules include evaluation adjustment rules under different environmental conditions. The anomaly detection module includes sensor anomaly detection, physical rationality detection, and temporal continuity detection. Sensor anomaly detection monitors sensor data quality in real time and identifies abnormal states such as missing data, abnormal numerical ranges, and jumps. Physical rationality detection verifies the rationality of perception results based on physical common sense, such as the dynamics reflecting the characteristics of a water-covered road surface when visual detection detects a large amount of water. Temporal continuity detection combines historical frame data to detect abnormal abrupt change patterns in the data over time.

[0034] The beneficial effects of this invention are:

[0035] This invention collects dynamic and visual data through a dynamics and vision data acquisition platform. It aligns visual and dynamic information temporally using a frequency alignment strategy. Based on this, it proposes a dual-reward mechanism combining visual temporal continuity (stability is confirmed by judging the consistency of road surface type segmentation results between visual frames) and dynamic consistency (matching degree is confirmed by judging the consistency between the visually determined road surface type and the dynamically identified road surface type). This provides a quantitative quality evaluation index for multimodal fusion, achieving the fusion of dynamic features and visual information. The proposed dynamics and vision deep fusion method based on this dual-reward mechanism, through the design of an intelligent fusion evaluation system, achieves complementary advantages and mutual verification of the two types of sensor information, significantly improving the accuracy and robustness of off-road road surface classification. Furthermore, this invention constructs a cross-modal mapping relationship library, realizing intelligent conversion from 17 types of visual segmentation results to 6 types of dynamic road surface types, providing accurate semantic correspondence for fusion. This invention also develops a real-time consistency verification algorithm, which can quickly identify sensor anomalies and perception conflicts, ensuring the safe operation of the system. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall architecture of the off-road surface classification system based on the deep fusion of dynamics and vision of the present invention.

[0037] Figure 2 This is a schematic diagram of the multi-source data synchronization module structure of the present invention;

[0038] Figure 3 This is a schematic diagram of the deep fusion evaluation module structure of the present invention;

[0039] Figure 4 The image is a semantically segmented image according to an embodiment of the present invention;

[0040] Figure 5 This is an original road surface image from an embodiment of the present invention;

[0041] Figure 6 This is a statistical summary of the final results of the embodiments of the present invention. Detailed Implementation

[0042] like Figure 1As shown, this invention proposes an off-road surface classification system based on deep fusion of dynamics and vision, including a multi-source data synchronization module, a deep fusion evaluation module, and an adaptive decision-making module. The multi-source data synchronization module ensures that visual images and dynamic data are precisely synchronized in time through timestamp alignment and sliding window matching, serving as the input to the deep fusion evaluation module. The deep fusion evaluation module designs a dual evaluation system: visual temporal continuity evaluates the stability of the detection and segmentation results in the time dimension, and dynamic consistency evaluates the degree of consistency between visual perception and actual vehicle response. The two are weighted and fused to form the final quality score. The adaptive decision-making module dynamically adjusts the fusion strategy according to the real-time evaluation results, prioritizing dynamics when the confidence level is high and increasing the visual weight when the confidence level is low, achieving intelligent complementary advantages of sensors.

[0043] Furthermore, classification of off-road terrain requires vehicle chassis signals and binocular camera signals, but these two signals are heterogeneous and cannot be directly synchronized. This invention employs a multi-source data synchronization module for data synchronization, such as... Figure 2 As shown, the multi-source data synchronization module includes: a data input layer, a core processing layer, and an output interface layer;

[0044] The data input layer includes a visual data stream, a dynamic data stream, and raw timestamps. The visual data stream comes from a continuous sequence of visual images from onboard vision sensors (cameras), containing rich road texture and color information. Each frame of the image is accompanied by a timestamp for subsequent timestamp alignment. The dynamic data stream comes from dynamic sensor data on the vehicle's CAN bus, including vehicle status information output from IMU (Inertial Measurement Unit), wheel speed sensors, steering angle sensors, etc. This data is acquired at a higher frequency, reflecting the physical interaction between the vehicle and the road surface. The raw timestamps are time stamps inherent to each sensor's data, based on a unified system clock, providing a reference for timestamp alignment.

[0045] The core processing layer comprises three parts: timestamp alignment calculation, sliding window matching mechanism, and anti-duplicate matching mechanism. The core function of the timestamp alignment calculation is to resolve the sampling frequency difference between the visual sensor (10-30Hz) and the dynamic sensor (100-1000Hz). This is achieved by establishing a unified time reference system, converting all sensor timestamps into the system's absolute time, and using linear interpolation to resample the high-frequency dynamic data, thus synchronizing the dynamic data time points with the visual data time points. For the visual data time points… and dynamic data time points The synchronization condition is:

[0046] (1)

[0047] in, The maximum allowable time difference is usually set to 100ms.

[0048] The sliding window matching mechanism is used to maintain a fixed-size data buffer window to solve the problem of data flow rate mismatch; the visual data window capacity is 10-20 frames, corresponding to a time range of 1-2 seconds; the dynamic data window capacity is 50-100 data points, corresponding to a time range of 0.5-1 seconds; within the window, the unmatched dynamic data with the closest time is found for each visual data, forming a one-to-one visual-dynamic data combination;

[0049] The anti-duplicate matching mechanism ensures that each piece of data is used only once by generating a unique identifier (combining timestamp, sensor ID, and serial number) for each input data and recording the matching history; it also uses a hash table to store information on matched data pairs, sets the validity period of matching records, and periodically cleans up expired records.

[0050] The output interface layer comprises three parts: aligned data pairs, time difference statistics, and quality assessment. The aligned data pairs ensure precise temporal correspondence between the visual-dynamic data combinations formed after time synchronization processing by the core processing layer, through alignment in the frequency and time domains, providing a foundation for subsequent fusion processing. The time difference statistics record the actual time difference of each successfully matched visual-dynamic data combination for system performance monitoring and quality assessment. The quality assessment evaluates the real-time quality of the core processing layer's time synchronization processing results, including indicators such as dynamic confidence and the number of road surface pixels. The dynamic confidence is the true value flowing in with the dynamic data stream, and the number of road surface pixels is the number of pixels belonging to the road surface in an image, identified by pixel recognition codes to identify the road surface portion of each pixel among 17 types.

[0051] Furthermore, the parameters configured for the multi-source data synchronization module include:

[0052] Time window size: Defines the time range of the sliding window in the sliding window matching step in the core processing layer. By conducting experiments under different window sizes, the matching success rate and latency are compared. Finally, 10-20 frames are selected as the visual window to balance real-time performance (latency ≤ 2 seconds) and data coverage integrity.

[0053] Maximum time difference: Set the maximum allowable time difference for data matching, set to 100ms. This setting is based on the lowest sampling frequency (10Hz) of the visual sensor and the highest sampling frequency (1000Hz) of the dynamic sensor to ensure that the matching error does not exceed the time interval of one frame of visual data, thus ensuring synchronization accuracy.

[0054] Cache capacity: Controls the cache size of each data stream. Based on the system, a dynamic caching strategy is adopted. When the cache utilization rate is >80%, the earliest unmatched data is prioritized for elimination to ensure stable system operation and optimize memory usage efficiency.

[0055] The performance monitoring includes:

[0056] Synchronization success rate: The percentage of successfully matched visual-dynamic data combinations out of the total data volume;

[0057] Mean alignment error: Calculate the average time difference of all matched visual-dynamic data combinations;

[0058] Matching rate: measures the system's ability to process data matching per unit of time. Matching efficiency = number of successfully matched pairs / total theoretically possible number of matches.

[0059] Furthermore, such as Figure 3 As shown, the input data of the deep fusion evaluation module includes aligned visual data, aligned dynamic data, and historical frame data. The aligned visual data is visual image data from the multi-source data synchronization module, which has been precisely aligned with the dynamic data in time and includes pixel-level classification results output by the semantic segmentation network. The aligned dynamic data is vehicle state information synchronized from the multi-source data synchronization module, including road surface type recognition results and confidence scores, providing physical supervision signals for consistency evaluation. The historical frame data stores the visual segmentation results of the most recent few frames for temporal continuity analysis.

[0060] The input data is processed and analyzed by the evaluation calculation layer, which includes three parts: visual temporal continuity evaluation, dynamic consistency evaluation, and dual reward fusion.

[0061] The visual temporal continuity evaluation is used to quantify the stability of semantic segmentation results in the temporal dimension. The reward for visual temporal continuity evaluation mainly consists of two parts: a base reward (ensuring sufficient road surface information in the image for analysis) and a temporal reward (rewarding segments similar to those in previous frames). for:

[0062] = base_reward1 + 0.4 ×temporal_reward (2)

[0063] Base_reward1 is the basic visual reward, reflecting the effectiveness evaluation. It requires at least 1000 road surface pixels to be considered for effective evaluation. If the number of road surface pixels is greater than 1000, base_reward1 = 0.3; if the number of road surface pixels is less than 1000, base_reward1 = 0.1. Temporal_reward is the temporal reward, reflecting the road surface type similarity evaluation. The higher the similarity, the higher the reward. If the number of road surface pixels is greater than 1000, temporal_reward = number of dominant category pixels / total number of road surface pixels; if the number of road surface pixels is less than 1000, temporal_reward = 0.

[0064] The dynamic consistency evaluation is used to quantify the degree of consistency between visual segmentation results and dynamic recognition results. Based on a preset visual-dynamic mapping relationship, 17 types of visual segmentation results (background, buildings, dry concrete pavement, dry asphalt, trees, vehicles, people, shrubs, sky, dirt roads, snow, sand, wet ground, rocks, brick roads, puddles, and gravel roads) are mapped to 6 types of dynamic road surfaces (normal road surface, snow, gravel, grass, mud, ruts, sand, rock roads, and wading roads), and the consistency judgment is weighted according to the dynamic confidence level.

[0065] =base_reward2×confidence_weight (3)

[0066] in, The base_reward2 is the basic reward for consistency evaluation of dynamics. It is assigned a value based on whether the dynamics and visual results are consistent. If consistent, base_reward2 = 1.0; if inconsistent, base_reward2 = 0.2. The confidence_weight is the confidence weight of dynamics. It is dynamically adjusted according to different confidence levels. The confidence weight of dynamics is set to 1, 2, and 3 points for low, medium, and high confidence levels, respectively.

[0067] The dual-reward fusion intelligently integrates the results of visual temporal continuity evaluation and dynamic consistency evaluation, dynamically adjusting the weights of the two evaluation indicators based on the proportion of road surface pixels: when the proportion of road surface pixels to the total number of pixels is sufficient (>30%), emphasis is placed on dynamic consistency, with a weight of 0.7; when the proportion of road surface pixels to the total number of pixels is insufficient (≤30%), emphasis is placed on visual temporal continuity, with a weight of 0.6; the total reward after fusion... The calculation formula is:

[0068] (4)

[0069] in and The dynamic weights for visual temporal continuity evaluation and dynamic consistency evaluation are respectively, satisfying... + =1;

[0070] The output of the deep fusion evaluation module includes a total reward score, sub-item rewards, and a quality assessment. The total reward score is the comprehensive quality score after fusion, reflecting the reliability of the current perception results. The sub-item rewards are detailed scores for visual temporal continuity evaluation and dynamic consistency evaluation. The quality assessment is a comprehensive evaluation of the current data quality, including dynamic confidence and the number of road surface pixels.

[0071] Furthermore, the deep fusion evaluation module also includes a mapping knowledge base and an anomaly detection module. The mapping knowledge base includes visual-dynamic mapping relationships, confidence weights, and scene rules. The visual-dynamic mapping relationships store the correspondence between 17 visual segmentation categories and 6 dynamic road surface types, providing a semantic basis for the consistency evaluation of the evaluation calculation layer. The confidence weights define the credibility of dynamic results under different road surface conditions and are used for weighted processing during reward calculation in the evaluation calculation layer. The scene rules include evaluation adjustment rules under different environmental conditions. The anomaly detection module includes sensor anomaly detection, physical rationality detection, and temporal continuity detection. Sensor anomaly detection monitors sensor data quality in real time and identifies abnormal states such as missing data, abnormal numerical ranges, and jumps. Physical rationality detection verifies the rationality of perception results based on physical common sense, such as the dynamics reflecting the characteristics of a water-covered road surface when visual detection detects a large amount of water. Temporal continuity detection combines historical frame data to detect abnormal abrupt change patterns in the data over time.

[0072] This embodiment uses a camera and a dynamic sensor to collect data from a real environment. The experimental data includes various road surface types such as asphalt roads, dirt roads, sandy areas, snowy areas, rocky areas, brick roads, and gravel roads. The results obtained after visual semantic segmentation and the original images are shown below. Figure 4 and Figure 5 As shown, the dynamic data is used to obtain road surface classification results and confidence levels through the model. This invention will run on the above visual and dynamic data.

[0073] The final result is as follows Figure 6 As shown, the alignment of 3058 dynamic data and 366 visual data was achieved through the synchronization of multi-source data, with a matching efficiency of 98.9%. A total of 351 pairs of data were processed, with 215 pairs of inconsistent samples and a statistically consistent sample pair count of 38.7%.

Claims

1. A cross-country surface classification system based on deep fusion of dynamics and vision, characterized in that: It includes a multi-source data synchronization module, a deep fusion evaluation module, and an adaptive decision-making module; the multi-source data synchronization module ensures that visual images and dynamic data are precisely synchronized in time through timestamp alignment and sliding window matching, serving as the input to the deep fusion evaluation module; The deep fusion evaluation module designs a dual evaluation system: visual temporal continuity evaluation assesses the stability of the detection and segmentation results in the time dimension, and dynamic consistency evaluation verifies the degree of consistency between visual perception and actual vehicle response. The two are weighted and fused to form the final quality score. The adaptive decision module dynamically adjusts the fusion strategy based on the real-time evaluation results, prioritizing dynamics when the confidence level is high and increasing visual weight when the confidence level is low.

2. The off-road surface classification system based on deep fusion of dynamics and vision as described in claim 1, characterized in that: A multi-source data synchronization module is used to synchronize the vehicle chassis signal and binocular camera signal required for off-road surface classification. The multi-source data synchronization module includes: a data input layer, a core processing layer and an output interface layer. The data input layer receives visual data streams, dynamic data streams, and raw timestamps. The visual data streams originate from a continuous sequence of visual images from onboard vision sensors. These image sequences contain road texture and color information, and each frame is accompanied by a timestamp for subsequent timestamp alignment. The dynamic data streams consist of vehicle status information output from dynamic sensors on the vehicle's CAN bus. These dynamic sensors include an IMU, wheel speed sensors, and steering angle sensors. The vehicle status information is acquired at a higher frequency, reflecting the physical interaction between the vehicle and the road surface. The raw timestamps are time stamps inherent to the data from each sensor, based on a unified system clock, providing a reference for timestamp alignment. The core processing layer comprises three parts: timestamp alignment calculation, sliding window matching mechanism, and anti-duplicate matching mechanism. The timestamp alignment calculation establishes a unified time reference system, converting the timestamps of the visual and dynamic sensors into absolute system time, and uses linear interpolation to resample high-frequency dynamic data, synchronizing the dynamic data time points with the visual data time points. For the visual data time points... and dynamic data time points The synchronization condition is: （1） in, This is the maximum permissible time difference; The sliding window matching mechanism is used to maintain a fixed-size data buffer window to solve the problem of data flow rate mismatch; the visual data window capacity is 10-20 frames, corresponding to a time range of 1-2 seconds; the dynamic data window capacity is 50-100 data points, corresponding to a time range of 0.5-1 seconds; within the window, the unmatched dynamic data with the closest time is found for each visual data, forming a one-to-one visual-dynamic data combination; The anti-duplicate matching mechanism ensures that each piece of data is used only once by generating a unique identifier for each input data and recording the matching history. The unique identifier includes a timestamp, sensor ID, and serial number. A hash table is used to store the information of matched visual-dynamic data combinations, and the validity period of the matching records is set and expired records are cleaned up periodically. The output interface layer comprises three parts: aligned data pairs, time difference statistics, and quality assessment. The aligned data pairs ensure that the visual-dynamic data combinations formed after time synchronization processing by the core processing layer are precisely time-corresponding through alignment in the frequency and time domains, providing a foundation for subsequent fusion processing. The time difference statistics record the actual time difference of each successfully matched visual-dynamic data combination for system performance monitoring and quality assessment. The quality assessment evaluates the real-time quality of the time synchronization processing results from the core processing layer.

3. A cross-country surface classification system based on dynamics and visual depth fusion according to claim 1 or 2, characterized in that: The parameters configured for the multi-source data synchronization module include: Time window size: Defines the time range of the sliding window in the sliding window matching step in the core processing layer. By conducting experiments with different window sizes, the matching success rate and latency are compared to balance real-time performance and data coverage integrity. Maximum time difference: Sets the maximum allowable time difference for data matching. The setting is based on the minimum sampling frequency of the visual sensor and the maximum sampling frequency of the dynamic sensor to ensure that the matching error does not exceed the time interval of one frame of visual data, thus ensuring synchronization accuracy. Cache capacity: Controls the cache size of each data stream. Based on the system, a dynamic caching strategy is adopted. When the cache utilization rate is >80%, the earliest unmatched data is prioritized for elimination to ensure stable system operation and optimize memory usage efficiency.

4. The off-road surface classification system based on deep fusion of dynamics and vision according to claim 2, characterized in that: The performance monitoring includes: Synchronization success rate: The percentage of successfully matched visual-dynamic data combinations out of the total data volume; Mean alignment error: Calculate the average time difference of all matched visual-dynamic data combinations; Matching rate: measures the system's ability to process data matching per unit of time. Matching efficiency = number of successfully matched pairs / total theoretically possible number of matches.

5. The off-road surface classification system based on deep fusion of dynamics and vision according to claim 1, characterized in that: The input data of the deep fusion evaluation module includes aligned visual data, aligned dynamic data, and historical frame data. The aligned visual data is visual image data from the multi-source data synchronization module, which has been precisely aligned with the dynamic data in time and includes pixel-level classification results output by the semantic segmentation network. The aligned dynamic data is vehicle state information synchronized from the multi-source data synchronization module, including road surface type recognition results and confidence scores, providing physical supervision signals for consistency evaluation. The historical frame data stores the visual segmentation results of the most recent few frames for temporal continuity analysis; The input data is processed and analyzed by the evaluation calculation layer, which includes three parts: visual temporal continuity evaluation, dynamic consistency evaluation, and dual reward fusion. The visual temporal continuity evaluation is used to quantify the stability of semantic segmentation results in the temporal dimension. The reward for the visual temporal continuity evaluation mainly consists of two parts: a basic reward and a temporal reward. for: = base_reward1 + 0.4 ×temporal_reward (2) Base_reward1 is the basic visual reward, which reflects the effectiveness evaluation and requires at least 1000 road surface pixels to be evaluated effectively; temporal_reward is the temporal reward, which reflects the road surface type similarity evaluation, and the higher the similarity, the higher the reward. The dynamic consistency evaluation is used to quantify the degree of consistency between visual segmentation results and dynamic recognition results; Based on a pre-defined visual-dynamic mapping relationship, 17 visual segmentation results are mapped to 6 dynamic road surface types. The 17 visual segmentation results include background, buildings, dry concrete pavement, dry asphalt, trees, vehicles, people, shrubs, sky, dirt roads, snow, sand, wet ground, rocks, brick roads, puddles, and gravel roads. The 6 dynamic road surface types include normal roads, snow, gravel, and grass, mud, ruts, sand, rocky roads, and wading roads. Consistency judgments are weighted according to dynamic confidence levels. =base_reward2×confidence_weight (3) in, The base_reward2 is the basic reward for consistency evaluation of dynamics, which is assigned a value based on whether the dynamics and visual results are consistent; confidence_weight is the confidence weight of dynamics, which is dynamically adjusted according to different confidence levels. The confidence weight of dynamics is set to 1, 2, and 3 points respectively for low, medium, and high. The dual-reward fusion intelligently integrates the results of visual temporal continuity evaluation and dynamic consistency evaluation, dynamically adjusting the weights of the two evaluation indicators based on the proportion of road surface pixels: when the proportion of road surface pixels to the total number of pixels is sufficient (greater than 30%), emphasis is placed on dynamic consistency; when the proportion of road surface pixels to the total number of pixels is insufficient (no more than 30%), emphasis is placed on visual temporal continuity; the total reward after fusion... The calculation formula is: （4） in and The dynamic weights for visual temporal continuity evaluation and dynamic consistency evaluation are respectively, satisfying... + =1; The output of the deep fusion evaluation module includes a total reward score, sub-item rewards, and a quality assessment. The total reward score is the comprehensive quality score after fusion, reflecting the reliability of the current perception results. The sub-item rewards are detailed scores for visual temporal continuity evaluation and dynamic consistency evaluation. The quality assessment is a comprehensive evaluation of the current data quality, including dynamic confidence and the number of road surface pixels.

6. The off-road surface classification system based on dynamics and visual depth fusion according to claim 5, characterized in that: The deep fusion evaluation module also includes a mapping knowledge base and an anomaly detection module. The mapping knowledge base includes visual-dynamic mapping relationships, confidence weights, and scene rules. The visual-dynamic mapping relationships store the correspondence between 17 visual segmentation categories and 6 dynamic road surface types, providing a semantic basis for the consistency evaluation of the evaluation calculation layer. The confidence weights define the credibility of the dynamic results under different road surface conditions and are used for weighted processing during reward calculation in the evaluation calculation layer. The scenario rules include evaluation adjustment rules under different environmental conditions; The anomaly detection module includes: sensor anomaly detection, physical rationality detection, and temporal continuity detection. The sensor anomaly detection monitors sensor data quality in real time and identifies abnormal states. The physical rationality detection is based on common sense to verify the rationality of the perception results; the temporal continuity detection combines historical frame data to detect abnormal mutation patterns in the data over time.

Images