Method for evaluating road traffic marking legibility based on visual-acoustic multi-modal perception

Abstract

The application discloses a kind of road traffic marking legibility evaluation methods based on visual-acoustic multi-modal perception.The method is through vehicle-mounted multi-modal perception unit to collect road traffic marking and corresponding road surface image information and the rolling noise information generated when tire rolling, the road surface element is separated from traffic marking element by machine vision semantic segmentation, color feature is extracted and color contrast is calculated;At the same time, the noise signal collected is extracted and pattern recognized, and the marking material and physical state characteristics are obtained.The visual contrast and acoustic characteristics are fused and analyzed, and the multi-modal threshold standard determined by the driving simulator is combined to quickly determine the marking visual recognition level.The efficient quantitative evaluation of the marking recognition level effectively solves the problem of limited visual evaluation in bad weather, and can diagnose the marking state from the material level, quickly identify low-recognition marking safety hazards, and provide data support for road maintenance and marking optimization.

Description

Technical Field

[0001] This invention belongs to the field of road traffic detection technology, specifically involving a rapid evaluation method for the recognition of road traffic markings based on visual-acoustic multimodal perception. It is applicable to various road scenarios. Through vehicle-mounted multimodal perception acquisition, machine vision semantic segmentation, acoustic feature analysis, color contrast quantification analysis, and large-scale simulation experiment multimodal threshold calibration, it can quickly evaluate the visual recognition effect of road markings and provide technical support for road traffic safety assurance and maintenance management. Background Technology

[0002] Road markings, as key traffic facilities guiding vehicle movement and regulating traffic order, directly affect drivers' judgment and reaction efficiency in terms of visual recognition, and are an important factor in ensuring road traffic safety. Currently, the assessment of marking recognition largely relies on manual visual inspection, which is highly subjective, lacks standardized criteria, and makes it difficult to generate quantitative assessment results. Some technical solutions use specialized equipment to collect data, which is detached from actual driving scenarios and lacks scientifically unified qualification standards, making them unsuitable for complex usage scenarios under different road surfaces, lighting conditions, and vehicle speeds.

[0003] The existing technology has key shortcomings: First, the evaluation method is highly subjective, and the results of manual evaluation are easily affected by factors such as the evaluator's eyesight, experience, and subjective judgment, resulting in poor reliability and consistency. Second, there is a lack of data collection methods adapted to actual driving scenarios, and the data collected by professional equipment is difficult to reflect the road marking recognition effect under the vehicle's visual perspective. Third, the threshold is difficult to determine and has not been verified through large-scale experiments, and a single standard cannot meet the usage needs of different scenarios. Fourth, the existing solution relies solely on visual information, and the evaluation effect drops significantly under adverse conditions such as rain, snow, fog, haze, and night, and it cannot effectively diagnose the physical state of road markings such as wear and aging at the material level, resulting in one-sided and insufficiently in-depth evaluation results.

[0004] To address the aforementioned issues, this invention proposes a "visual-acoustic multimodal joint perception" technical solution. Through multimodal acquisition methods simulating actual driving scenarios, precise element separation technology, acoustic physical state diagnosis, and scientific experimental verification processes, it achieves all-weather, multi-dimensional, rapid, and accurate quantitative evaluation of the visual recognition of road markings, thus overcoming the shortcomings of existing technologies. Summary of the Invention

[0005] In view of the shortcomings of the above-mentioned background technology, the present invention provides a method for rapid evaluation of the recognizability of road traffic markings based on visual-acoustic multimodal perception.

[0006] The road traffic marking recognition evaluation method based on visual-acoustic multimodal perception includes multimodal information acquisition, image semantic segmentation, acoustic feature extraction, color feature extraction and contrast calculation, multimodal threshold calibration, and fusion recognition determination. The multimodal information acquisition uses an onboard camera and microphone array to acquire image information of the target road markings and corresponding road surface, as well as rolling noise information generated when tires roll over the markings or road surface. The image semantic segmentation, based on machine vision technology, performs semantic segmentation of road surface elements and traffic marking elements in the acquired images, achieving accurate separation of the two types of elements. The acoustic feature extraction preprocesses the acquired noise signals. The process involves several steps: First, acoustic feature extraction and spectral analysis are performed to obtain acoustic feature parameters associated with the type of road marking material, wear level, and adhesion state. Second, color feature extraction and contrast calculation extract color features of the road marking and the road surface from the segmented element images, and calculate the color contrast between the two based on a color space model. Third, multimodal threshold calibration is performed by simultaneously simulating the visual and acoustic environments using a driving simulator, and the joint pass standard for visual contrast and acoustic features in each scenario is determined through experimental testing. Fourth, fusion recognition determination compares the actually calculated visual contrast and acoustic feature parameters with the multimodal threshold of the corresponding scenario to output the visual recognition evaluation result of the road marking.

[0007] In the multimodal information acquisition, the vehicle-mounted camera and microphone array acquire data synchronously. When the camera acquires images, it adapts to the lighting conditions of the actual driving scene. The microphone array is installed in the area near the wheels to acquire the rolling noise signal generated when the tires run over the marking area and the road surface area during vehicle travel, ensuring that the image and noise signal are synchronously corresponding in the spatiotemporal dimension.

[0008] In the image semantic segmentation, machine vision-related algorithms are used to process the acquired images, accurately identify and segment the road surface element region and traffic marking element region in the image, eliminate other irrelevant interference elements, and ensure the complete separation of the two types of core elements.

[0009] In the color feature extraction and contrast calculation, color information is separated from the segmented road surface element image and traffic marking element image, core color feature parameters are extracted, and data processing is performed through a color space model to obtain the color contrast quantification value of the marking and the road surface.

[0010] The acoustic feature extraction includes: preprocessing the collected tire rolling noise signal by filtering and noise reduction; performing time-frequency analysis on the preprocessed signal to extract acoustic feature parameters including the spectral centroid, Mel frequency cepstral coefficients, and energy proportion of a specific frequency band; and identifying the physical state of the current marking based on a pre-established mapping relationship library between different marking material types, wear levels, adhesion states, and acoustic features.

[0011] In the multimodal threshold calibration, the driving simulator constructs a virtual scene that includes different road surface types, marking types, lighting conditions, driving speeds, and different degrees of wear and material types of markings. It simultaneously simulates the visual and acoustic environments, allowing the experimenter to obtain visual and auditory information at the same time during the driving process.

[0012] The multimodal threshold calibration involves recruiting a sufficient number of participants with different characteristics to participate in simulated driving tests. The participants cover different age groups, visual acuity levels, hearing levels, and driving experience to ensure the representativeness and breadth of the experimental sample. During the simulated driving test, the participants' recognition indicators for various road markings are recorded, including but not limited to first recognition distance, recognition reaction speed, and recognition accuracy. At the same time, the participants' subjective recognition experience under different visual-acoustic combination conditions is also recorded.

[0013] Statistical analysis was conducted based on the experimental data to determine the visual contrast threshold and acoustic feature threshold for each scenario, and a joint judgment rule for visual and acoustic indicators was established: when the visual contrast meets the threshold requirements, visual judgment takes precedence; when the visual contrast is lower than the threshold due to factors such as bad weather or occlusion, acoustic judgment results are used as a supplement or alternative basis; when there is a significant contradiction between visual and acoustic results, a verification mechanism is triggered.

[0014] The method also has multimodal data fusion verification and anomaly handling functions: when the vehicle camera is obstructed, the image is blurry, or the acquisition environment does not meet the basic recognition conditions, the system automatically switches to the acoustic evaluation mode; when the acoustic signal acquisition is severely interfered with by environmental noise, the system automatically switches to the visual evaluation mode; when both the visual and acoustic evaluation results are unreliable, they are marked as invalid data, prompting the user to reacquire the image and noise signal or suspend the evaluation.

[0015] The road surface types include mainstream road surfaces such as asphalt pavement and cement pavement; the marking types include common road traffic markings such as solid white / yellow lines, dashed lines, and directional arrows; and the marking material types include hot-melt type, cold-paint type, and two-component type marking materials, suitable for various road scenarios.

[0016] Compared with the prior art, the present invention has the following advantages: (1) Stronger scene adaptability: The vehicle-mounted multimodal perception unit is used to collect images and acoustic information, which fully simulates the driver's actual visual perspective and driving scene. The evaluation results are more in line with actual use needs, solving the problem of traditional evaluation being divorced from the actual scene.

[0017] (2) The evaluation indicators are supported by experimental samples: the multimodal thresholds are calibrated through large-scale simulation experiments and sufficient experimental data. The experimental samples are extensive and the threshold standards have sufficient practical support, avoiding the limitations of a single standard.

[0018] (3) More comprehensive evaluation dimensions: By integrating information from both visual and acoustic modalities, it can not only evaluate the color contrast of the marking surface, but also diagnose the wear and aging status of the marking from the material level, making the evaluation results more in-depth and forward-looking.

[0019] (4) Stronger all-weather applicability: In scenarios where visual conditions are limited, such as rain, snow, fog, haze, and night, acoustic modalities can serve as an effective supplement or replacement, solving the application blind spots of pure vision solutions and greatly improving the robustness and applicable scenario range of the method.

[0020] (5) Higher evaluation efficiency: The whole process is simple to operate, and multimodal data collection, processing and calculation can be completed quickly. There is no need for long-term on-site observation or complex professional equipment operation. It can achieve rapid screening of large-scale road sections and greatly improve the efficiency of road detection. Attached Figure Description

[0021] Figure 1 This is a system architecture diagram. Detailed Implementation

[0022] The road traffic marking recognition evaluation method based on visual-acoustic multimodal perception of the present invention includes the following five steps, the specific process of which is as follows: S1: Multimodal Information Acquisition Equipment preparation: The vehicle's built-in camera is used, and the equipment is calibrated in advance to ensure the clarity and color accuracy of the captured images; at the same time, a high-sensitivity microphone array is installed in the area near the vehicle's wheels, and the acoustic sensor is calibrated to ensure that the rolling noise generated when the tires roll over the road surface and markings can be clearly captured.

[0023] Scene adaptation: Synchronous data acquisition is performed in the actual driving scenario of the target road section, matching the natural lighting conditions of the road section during acquisition, without the need for additional lighting equipment, ensuring that the acquired images can truly reflect the visual effects of road markings and road surface from the driver's perspective; Acoustic acquisition automatically adapts to vehicle speed and environmental background noise.

[0024] Multimodal acquisition: The vehicle is driven at a constant speed on the target road section, and road images are continuously acquired through the on-board camera. At the same time, tire rolling noise signals are continuously acquired through the microphone array. The acquisition range covers the complete area of ​​each marking and the adjacent road area, ensuring that the marking edges in the image are clear and the color features are not distorted. It also ensures that the noise signal can clearly distinguish the sound difference between the area where the tire is rolling over the marking and the area where the tire is rolling over the road surface, avoiding acquisition failures caused by camera obstruction or microphone damage from mud and water.

[0025] Anomaly Handling: If, during the data acquisition process, the camera is detected to be obstructed by dirt, the image is blurred, or the color is distorted due to extreme lighting conditions, or the microphone array signal is abnormal, or the ambient noise is too loud and the acoustic features are unusable, the data acquired during that period will be marked as invalid data and will be reacquired after the interference factors are eliminated.

[0026] S2: Multimodal Data Preprocessing and Feature Extraction 2.1 Image Semantic Segmentation Image preprocessing: The acquired valid images are optimized in the early stage, including noise reduction, contrast enhancement and other operations, to improve the distinguishability of road surface and marking elements in the image, and provide a high-quality data foundation for subsequent semantic segmentation.

[0027] Element segmentation: The preprocessed image is processed using a machine vision semantic segmentation algorithm. Based on the differences in color, texture, and shape between the road surface and the road markings, the road surface element region and the traffic marking element region in the image are accurately identified. The two types of regions are completely separated from the image, while irrelevant interference elements such as pedestrians, vehicles, and road stains are excluded.

[0028] Segmentation Verification: Verify the validity of the segmented feature images, checking whether the road surface and marking areas are completely separated and whether the boundaries are clear. If there are problems such as feature adhesion or missing segmentation, mark it as an invalid segmentation result and return to the previous step for reprocessing or trigger image re-acquisition.

[0029] 2.2 Acoustic Feature Extraction Acoustic signal preprocessing: The collected tire rolling noise signal is filtered and denoised to remove interference components such as wind noise, engine noise, and environmental background noise, and to extract pure tire-road / marking contact noise.

[0030] Time-frequency analysis and feature extraction: Time-frequency analysis is performed on the preprocessed acoustic signal. Acoustic feature parameters are extracted using methods such as short-time Fourier transform or wavelet transform, including but not limited to spectral centroid, spectral bandwidth, Mel frequency cepstral coefficients, and energy proportion of specific frequency bands.

[0031] Feature mapping and state recognition: Based on a pre-established mapping relationship library between different marking material types (hot melt type, cold paint type, two-component type), different wear levels (new marking, slight wear, severe wear), different adhesion states (clean, lightly contaminated, heavily contaminated) and acoustic feature parameters, the physical state of the current marking is identified, providing supplementary information for visual assessment.

[0032] S3: Visual contrast calculation Color information separation: For the segmented road surface feature image and traffic marking feature image, the color information of the two types of areas is extracted separately to ensure that the color data comes only from the target feature and is not affected by other interference factors.

[0033] Color feature extraction: Using digital image technology, core color feature parameters are extracted from the separated color information to establish color feature databases for both road surface and road markings, ensuring that the feature parameters accurately reflect the essential color differences between the two.

[0034] Contrast calculation: Based on the selected color space model, the extracted color feature parameters are standardized and quantified. The color contrast value between the road markings and the road surface is obtained through data processing. This value serves as the core indicator for measuring the visual difference between the two.

[0035] S4: Multimodal threshold calibration Simulator Scene Setup: A high-fidelity driving simulator is used to construct a multi-dimensional virtual road scene system, covering different road surface types such as asphalt and cement, different marking types such as solid white / yellow lines and dashed lines, different marking materials such as hot melt, cold paint, and two-component, different wear levels such as newly painted, slightly worn, and severely worn, different lighting conditions such as sunny daytime, cloudy daytime, and nighttime, and driving speeds corresponding to different scenarios such as urban roads and highways. Each scene is configured with markings of different color differences and corresponding acoustic environments for different wear levels, forming a complete visual-acoustic joint gradient simulation environment.

[0036] Participant Recruitment: Recruit a sufficient number of representative participants to participate in the simulated driving test. Participants will be from different age groups (20-60 years old), including people with different visual acuity levels (0.8-1.5 without glasses), while also taking into account differences in hearing level, as well as professional drivers and ordinary drivers, to ensure that the experimental sample can cover the characteristics of the mainstream driver group.

[0037] Simulated driving test: Participants complete standardized simulated driving tasks in various virtual scenarios. During the driving process, participants need to identify road markings ahead and react accordingly. The system records the participants' key indicators in real time, including the distance of the first identification of the markings, the time from the discovery of the markings to the reaction (reaction speed), the accuracy of marking recognition, and other data. At the same time, it records the participants' subjective identification experience and load evaluation under different visual-acoustic combination conditions.

[0038] Threshold determination: Statistical analysis was performed on all experimental data, and the thresholds were determined separately for each scenario combination (road surface type + marking type + marking material + wear level + lighting conditions + vehicle speed): 1) Visual contrast threshold: The lowest contrast value that more than 90% of the participants can accurately identify the markings; 2) Acoustic feature reference range: Statistically analyze the recognition performance of the experimenters under different acoustic feature conditions, and establish the acoustic feature reference range corresponding to each physical state; 3) Joint Judgment Rules: Establish joint judgment rules for visual and acoustic indicators, and clarify the decision-making logic of prioritizing visual indicators when visual conditions are good, supplementing or substituting acoustic indicators when visual conditions are limited, and triggering a review when the two contradict each other.

[0039] Integrate thresholds and rules from all scenarios to form a complete multimodal scenario-based threshold system.

[0040] S5: Multimodal fusion recognition accuracy determination Threshold matching: Based on the actual conditions of the target road segment, including road surface type, marking type, marking material type, estimated wear degree, lighting conditions when the image was acquired, and the design driving speed of the road segment, the corresponding qualified threshold and acoustic feature reference range are matched from the multimodal threshold system.

[0041] Fusion determination: 1) Compare the actual visual contrast value calculated in step S3 with the matched visual contrast threshold; 2) Compare the acoustic feature parameters identified in step S2.2 with the matched acoustic feature reference range; 3) Conduct a comprehensive evaluation based on the joint judgment rules: ① If the visual contrast value is greater than or equal to the threshold and the acoustic features are within the normal range, it is judged as "the recognition is qualified and the marking is in good condition"; ② If the visual contrast value is greater than or equal to the threshold, but the acoustic features show that the markings are severely worn or the materials are aged, it is judged as "the recognition is qualified but there is a potential risk, and regular re-inspection is recommended"; ③ If the visual contrast value is less than the threshold, but the acoustic features show that the physical condition of the marking is good (such as visual limitation due to weather conditions), it is judged as "the recognition is temporarily limited, and it is recommended to re-examine in combination with weather conditions or refer to the acoustic assessment results". ④ If the visual contrast value is less than the threshold and the acoustic features show that the markings are severely worn or the materials are aged, it is judged as "the recognition is unqualified and it is recommended to repaint the markings"; ⑤ If there is a significant discrepancy between the visual and acoustic assessment results, the review mechanism will be triggered, prompting for data to be collected again.

[0042] Report generation: Automatically generates a complete assessment report, including basic road segment information, multimodal acquisition status, image semantic segmentation results, color feature extraction and contrast calculation process, acoustic feature extraction and physical state recognition results, threshold matching results, and multimodal fusion assessment conclusions. It supports direct export for road maintenance decision-making and management. Example 1

[0043] A rapid evaluation method for the recognizability of road traffic markings based on visual-acoustic multimodal perception is presented, and its execution process is as follows: Multimodal information acquisition was conducted on a section of asphalt pavement along a main urban road. Images were captured using a vehicle-mounted camera under natural daylight conditions, while tire rolling noise signals were simultaneously collected via a microphone array installed near the wheels. The vehicle traveled at a constant speed of 40 km / h, continuously capturing images of the solid white lines marking the road markings and adjacent asphalt pavement. A total of 20 valid images and 20 valid noise signal segments were acquired simultaneously, ensuring that the markings and road surface features in each image were clear and undistorted, and that each noise signal could clearly distinguish the sound difference between the tires rolling over the markings and the tires rolling over the road surface.

[0044] Image semantic segmentation preprocessed 20 valid images, removing environmental noise and enhancing image contrast to improve feature differentiation. A machine vision semantic segmentation algorithm was then used to further process the preprocessed images, accurately separating the asphalt pavement feature area from the white solid line marking feature area, eliminating interference factors such as scattered stains on the road surface and distant pedestrians. Verification showed that 18 images achieved complete and accurate segmentation, while 2 images were incompletely segmented due to slight wear on the marking edges and were marked as invalid data; two additional valid images were then collected.

[0045] Acoustic feature extraction involved filtering and denoising preprocessing 20 synchronously acquired effective noise signals to remove wind and engine noise interference. Short-time Fourier transform was then used to perform time-frequency analysis on the preprocessed signals, extracting acoustic feature parameters such as the spectral centroid and Mel-frequency cepstral coefficients. The extracted features were compared with a pre-established acoustic feature database, identifying the road marking material as heat-fused, with slight wear, clean adhesion, and acoustic feature parameters within the normal range.

[0046] Visual contrast was calculated by extracting color information from the road surface area and the road marking area from 18 effectively segmented images. Image recognition technology was used to extract the Lab color space feature parameters of the two types of areas. A standardized calculation process was employed to obtain the color contrast value for each image, with an average contrast value of 1.05 for the 18 samples.

[0047] Multimodal threshold calibration was performed using a driving simulator to create a virtual scenario with "asphalt road surface + solid white lines + hot-melt markings + slight wear + sunny daytime + 60 km / h driving speed," configuring different color difference gradients for the road markings and corresponding acoustic environments with varying degrees of wear. 150 participants aged 20-60 were recruited for simulated driving tests, including individuals with uncorrected visual acuity ranging from 0.8 to 1.5 and normal hearing levels. Among them were 60 professional drivers and 90 ordinary drivers. After completing the simulated driving, the system recorded the participants' initial recognition distance, reaction speed, and recognition accuracy, as well as their subjective perceptions. Statistical analysis determined that the visual contrast threshold for this scenario was 0.88, and the acoustic feature reference range matched the current recognition results.

[0048] The multimodal fusion recognition accuracy assessment matched the visual contrast threshold of 0.88 for the target road segment. The actual calculated average contrast value was 1.05 ≥ 0.88. The acoustic feature recognition result indicated slight wear and normal condition. Based on the joint judgment rule, the assessment was "recognition accuracy qualified, road marking condition good"; an evaluation report was generated, clarifying that the visual recognition accuracy of the road markings met the normal recognition needs of drivers, the material condition was good, and no additional optimization was required.

[0049] Anomaly Test 1 (Supplementary Verification in Severe Weather): Images and noise signals were collected on a cement road section at night (design speed 80km / h). Due to the nighttime conditions and slight fog / haze, the calculated visual contrast was only 0.65, lower than the corresponding scene threshold of 0.72. However, acoustic feature extraction identified the road marking material as a cold-paint type with slight wear and good physical condition. Based on the joint judgment rule, acoustic assessment is used as a supplement when visual conditions are limited. The system judged it as "Recognition is temporarily limited; it is recommended to re-examine in conjunction with weather conditions or refer to the acoustic assessment results," and output the suggestion "The physical condition of the road marking is good; recognition should return to normal after the weather improves."

[0050] Anomaly Test 2 (Material Aging Diagnosis): Data was collected on a certain road section. The calculated visual contrast value was 1.02, higher than the threshold of 0.88, indicating visual compliance. However, acoustic feature extraction identified the marking material as a two-component type with severe wear, and spectral features showed significant detachment of reflective beads. Based on the joint judgment rule, although visual compliance was acceptable, the acoustic indicators showed potential risks. The system determined that "the visibility is acceptable but there is a potential risk, and regular re-inspection is recommended," and output the diagnostic opinion that "the surface color of the marking is acceptable, but the material is severely worn, and the visibility is expected to decline rapidly in the short term. It is recommended to include it in the priority maintenance plan." The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for evaluating the recognizability of road traffic markings based on visual-acoustic multimodal perception, characterized in that, It includes multimodal information acquisition, image semantic segmentation, acoustic feature extraction, color feature extraction and contrast calculation, multimodal threshold calibration and fusion recognition determination; the multimodal information acquisition obtains image information of the target road section markings and corresponding road surface and rolling noise information generated when the tires roll over the markings or road surface through vehicle-mounted cameras and microphone arrays; The image semantic segmentation, based on machine vision technology, performs semantic segmentation of road surface elements and traffic marking elements in the acquired images, achieving accurate separation of the two types of elements. The acoustic feature extraction preprocesses, performs spectrum analysis and feature extraction on the acquired noise signals to obtain acoustic feature parameters associated with the marking material type, wear degree, and adhesion state. The color feature extraction and contrast calculation extracts the color features of the markings and road surface from the segmented element images and calculates their color contrast based on a color space model. The multimodal threshold calibration simulates the visual and acoustic environments simultaneously through a driving simulator, and the joint pass standard of visual contrast and acoustic features under each scenario is determined by experimental personnel. The fusion recognition judgment compares the actually calculated visual contrast and acoustic feature parameters with the multimodal threshold of the corresponding scenario to output the visual recognition evaluation result of the markings.

2. The method according to claim 1, characterized in that, In the multimodal information acquisition, the vehicle-mounted camera and microphone array acquire data synchronously. When the camera acquires images, it adapts to the lighting conditions of the actual driving scene. The microphone array is installed in the area near the wheels to acquire the rolling noise signal generated when the tires run over the marking area and the road surface area during vehicle travel, ensuring that the image and noise signal are synchronously corresponding in the spatiotemporal dimension.

3. The method according to claim 1, characterized in that, In the image semantic segmentation, machine vision-related algorithms are used to process the acquired images, accurately identify and segment the road surface element region and traffic marking element region in the image, eliminate other irrelevant interference elements, and ensure the complete separation of the two types of core elements.

4. The method according to claim 1, characterized in that, In the color feature extraction and contrast calculation, color information is separated from the segmented road surface element image and traffic marking element image, core color feature parameters are extracted, and data processing is performed through a color space model to obtain the color contrast quantification value of the marking and the road surface.

5. The method according to claim 1, characterized in that, The acoustic feature extraction includes: preprocessing the collected tire rolling noise signal by filtering and noise reduction; performing time-frequency analysis on the preprocessed signal to extract acoustic feature parameters including the spectral centroid, Mel frequency cepstral coefficients, and energy proportion of a specific frequency band; and identifying the physical state of the current marking based on a pre-established mapping relationship library between different marking material types, wear levels, adhesion states, and acoustic features.

6. The method according to claim 1, characterized in that, In the multimodal threshold calibration, the driving simulator constructs a virtual scene that includes different road surface types, marking types, lighting conditions, driving speeds, and different degrees of wear and material types of markings. It simultaneously simulates the visual and acoustic environments, allowing the experimenter to obtain visual and auditory information at the same time during the driving process.

7. The method according to claim 6, characterized in that, Based on the experimental data, statistical analysis was conducted to determine the visual contrast threshold and acoustic feature threshold for each scenario, and a joint judgment rule for visual and acoustic indicators was established: when the visual contrast meets the threshold requirements, visual judgment takes precedence. When visual contrast falls below the threshold due to factors such as severe weather or occlusion, acoustic judgment results are used as a supplementary or alternative basis; when there is a significant contradiction between visual and acoustic results, a review mechanism is triggered.

8. The method according to claim 1, characterized in that, It also features multimodal data fusion verification and anomaly handling functions: when the vehicle camera is obstructed, the image is blurry, or the acquisition environment does not meet the basic recognition conditions, the system automatically switches to an acoustic evaluation-based mode; when the acoustic signal acquisition is severely interfered with by environmental noise, the system automatically switches to a visual evaluation-based mode; when both visual and acoustic evaluation results are unreliable, the data is marked as invalid, prompting the user to reacquire the image and noise signal or suspend the evaluation.

9. The method according to claim 1, characterized in that, The road surface types include mainstream road surfaces such as asphalt pavement and cement pavement; the marking types include common road traffic markings such as solid white / yellow lines, dashed lines, and directional arrows; and the marking material types include hot-melt type, cold-paint type, and two-component type marking materials, suitable for various road scenarios.

Images