Road disease detection system and method based on multi-sensor fusion and multi-modal large model

Abstract

The invention relates to the technical field of intelligent traffic infrastructure detection, in particular to a road disease detection system and method based on multi-sensor fusion and a multi-modal large model, and the system comprises a multi-sensor fusion module which integrates various different types of sensors. The sensor comprises a visible light camera, a laser radar and a three-dimensional geological radar; synchronous acquisition of multi-dimensional data is realized through sensor layout and a data synchronization mechanism, the visible light camera is used for acquiring image information of a road surface, the laser radar is used for acquiring three-dimensional shape information of the road surface, and the three-dimensional geological radar is used for acquiring road roadbed structure disease information; the method has the beneficial effects that the multi-dimensional data of the road appearance and the underground structure are synchronously obtained through a multi-sensor fusion technology, and the alignment error of a multi-source data physical space is reduced, so that the precision of road disease detection is remarkably improved. The application of the multi-modal large model can identify different types of diseases more accurately, and reduces the false detection rate and the omission ratio.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation infrastructure detection, and particularly to a road disease detection system and method based on multi-sensor fusion and multi-modal large model. Background Art

[0002] There are three core pain points in the current road disease detection technology: First, the ability to collect multi-dimensional data is insufficient. Traditional detection means rely on a single sensor (such as a visible light camera or lidar), resulting in difficulty in synchronously obtaining key disease characteristics such as crack depth and underground structure anomalies. Second, the adaptability to dynamic environments is weak. Existing algorithms have low disease recognition accuracy and high false detection rates in complex scenarios such as light changes (such as at night / strong light), extreme weather (rain / fog / dust), and material aging, and it is difficult to meet the high standards of road disease detection accuracy and positioning error. There is an urgent need to develop a multi-sensor fusion acquisition and multi-modal large model intelligent analysis technology to achieve synchronous acquisition and real-time accurate analysis of road surface and underground structure diseases. Summary of the Invention

[0003] The purpose of the present invention is to provide a road disease detection system and method based on multi-sensor fusion and multi-modal large model to solve the problems raised in the above background art.

[0004] To achieve the above purpose, the present invention provides the following technical solution: A road disease detection system based on multi-sensor fusion and multi-modal large model, including a multi-sensor fusion module. The multi-sensor fusion module integrates multiple different types of sensors, and the sensors include visible light cameras, lidars, and three-dimensional geological radars. Through sensor layout and data synchronization mechanism, synchronous acquisition of multi-dimensional data is realized. The visible light camera is used to collect image information on the road surface, the lidar is used to obtain three-dimensional shape information of the road surface, and the three-dimensional geological radar is used to detect anomalies in the underground structure. Advanced sensor calibration and data fusion algorithms are adopted to fuse the data collected by different sensors, reduce the physical space alignment error of multi-source data, and ensure the accuracy and consistency of the data.

[0005] Preferably, it further includes a multi-modal large model analysis and dynamic environment adaptation module.

[0006] The multi-modal large model analysis and dynamic environment adaptation module constructs a multi-modal large model, which fuses data features of multiple modalities such as images, point clouds, and radar signals to realize automatic extraction of disease features, depth, position, and environment information.

[0007] The dynamic environment adaptive module automatically adapts the AI algorithm model according to the environmental characteristics, trains and analyzes the fused multi-dimensional data, automatically identifies the disease categories, extracts the quantitative information of the diseases, compares and correlates the relative positions of the diseases, and achieves a disease positioning accuracy of less than 3 cm and a disease recognition accuracy of greater than 90%. At the same time, based on the multi-modal large model, the twin reconstruction of road appearance and structural diseases is realized, and the relative relationship of the diseases is visually presented.

[0008] Preferably, it further includes a real-time data processing and feedback module. The real-time data processing and feedback module processes the collected and analyzed data in real time, and timely feeds back the detected disease information to relevant departments and personnel; adopts efficient data processing algorithms and communication technologies to ensure the real-time and reliability of the data; provides a visual interface to facilitate users to view and manage disease information, and provides decision-making support for road maintenance and management.

[0009] Preferably, in the multi-modal large model analysis and dynamic environment adaptive module, the multi-modal large model performs fusion learning on the data features of multiple modalities through deep learning algorithms to accurately extract disease-related information; the dynamic environment adaptive module automatically adjusts the parameters and structure of the AI algorithm model according to the real-time collected environmental data, such as light intensity, weather conditions, complexity of the road surrounding environment, etc., to adapt to different detection environments and improve the accuracy and stability of disease recognition and positioning.

[0010] Preferably, in the real-time data processing and feedback module, the visual interface has a function of classifying and displaying disease information, and classifies and displays the disease information according to dimensions such as disease type, severity, location area, etc.; at the same time, it provides data query and export functions, which facilitate users to query the disease information of a specific time period and specific area according to their needs, and export the relevant information as a file in a specified format for further analysis and processing.

[0011] A detection method for a road disease detection system based on multi-sensor fusion and multi-modal large model. The multi-sensor fusion module is used for data collection. The multi-sensor fusion module integrates multiple different types of sensors, including visible light cameras, lidars, and three-dimensional geological radars; through a reasonable sensor layout and data synchronization mechanism, the synchronous collection of multi-dimensional data is realized. The visible light camera collects the image information of the road surface, the lidar obtains the three-dimensional shape information of the road surface, and the three-dimensional geological radar detects the abnormal conditions of the underground structure; then, advanced sensor calibration and data fusion algorithms are adopted to fuse and process the data collected by different sensors, reduce the physical space alignment error of multi-source data, and ensure the accuracy and consistency of the data.

[0012] Preferably, after data fusion, a multi-modal large model analysis and dynamic environment adaptation module are used for processing to construct a multi-modal large model. This model fuses data features of multiple modalities such as images, point clouds, and radar signals, automatically extracts disease features, depth, location, and environmental information. The dynamic environment adaptation module automatically adapts the AI algorithm model according to environmental features, trains and analyzes the fused multi-dimensional data, automatically identifies disease categories, extracts disease quantification information, and compares and correlates the relative positions of diseases, achieving a disease localization accuracy of less than 3 cm and a disease recognition accuracy of greater than 90%.

[0013] Preferably, the multi-modal large model analysis and dynamic environment adaptation module also realizes the twin reconstruction of road appearance and structural diseases based on the multi-modal large model. By constructing a virtual road disease model, the relative relationship of diseases is visually presented to more clearly understand the distribution and impact of diseases.

[0014] Preferably, it also includes real-time data processing and feedback steps. The data after collection and analysis is processed in real time, and efficient data processing algorithms and communication technologies are adopted to ensure the real-time and reliability of the data. The detected disease information is timely fed back to relevant departments and personnel, and at the same time, a visualization interface is provided to facilitate users to view and manage disease information. This visualization interface has functions such as classified display, query, and export of disease information, providing decision-making support for road maintenance and management.

[0015] Preferably, in the dynamic environment adaptation module, the environmental features relied on include light intensity, weather conditions, and the complexity of the road surrounding environment. According to these environmental features, the parameters and structure of the AI algorithm model are automatically adjusted to adapt to different detection environments, ensuring accurate disease recognition and localization in different environments and improving the robustness and reliability of disease detection.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] The road disease detection system and method based on multi-sensor fusion and multi-modal large model proposed by the present invention can synchronously obtain multi-dimensional data of road appearance and underground structure through multi-sensor fusion technology, reduce the physical space alignment error of multi-source data, thereby significantly improving the accuracy of road disease detection. At the same time, the application of the multi-modal large model can more accurately identify different types of diseases, reducing the false detection rate and missed detection rate.

[0018] The dynamic environment adaptation module can adjust and optimize the detection data and analysis results according to different environmental conditions, improve the disease recognition accuracy and robustness in complex scenarios such as light changes, extreme weather, and material aging, and ensure the reliability of the detection results.

[0019] By adopting a real-time data processing and feedback module, it can complete data collection, analysis, and feedback in a short time, achieving real-time and accurate analysis of road diseases. This helps to promptly detect road diseases, take effective maintenance measures, and ensure the safety and smoothness of the road. Brief Description of the Drawings

[0020] Figure 1 Schematic diagram of the multi-sensor fusion module of the present invention;

[0021] Figure 2 Flow chart of the multi-modal large model analysis module of the present invention;

[0022] Figure 3 Schematic diagram of the disease visualization management interface of the present invention. Detailed Description of the Preferred Embodiments

[0023] In order to clearly and completely describe the objectives, technical solutions, and advantages of the present invention, the following further details the embodiments of the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only a part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present invention.

[0024] Embodiment 1. Please refer to Figures 1 to 3 , the present invention provides a technical solution: A road disease detection system based on multi-sensor fusion and multi-modal large model, including:

[0025] 1. Multi-sensor fusion module

[0026] This intelligent agent integrates a variety of different types of sensors, including but not limited to visible light cameras, lidar, 3D geological radars, etc. Through reasonable sensor layout and data synchronization mechanism, synchronous acquisition of multi-dimensional data is achieved. For example, a visible light camera is used to collect image information of the road surface, lidar is used to obtain the three-dimensional shape information of the road surface, and a subsurface detection radar is used to detect abnormal conditions of the subsurface structure. At the same time, advanced sensor calibration and data fusion algorithms are adopted to fuse the data collected by different sensors, reducing the physical space alignment error of multi-source data and ensuring the accuracy and consistency of the data. The multi-sensor fusion module is as Figure 1 shown, and the carrier form is not limited.

[0027] 2. Multi-modal large model analysis and dynamic environment adaptive module

[0028] Construct a multi-modal large model that integrates data features of multiple modalities such as images, point clouds, and radar signals, and realizes the automatic extraction of disease features, depth, location, and environmental information; a dynamic environment adaptation module that automatically adapts the AI algorithm model according to environmental features, trains and analyzes the fused multi-dimensional data, automatically identifies disease categories, extracts quantitative disease information, compares and correlates the relative positions of diseases, and achieves a disease localization accuracy of <3 cm and a disease recognition accuracy of >90%. At the same time, based on the multi-modal large model, a twin reconstruction of road appearance and structural diseases is realized, intuitively presenting the relative relationship of diseases. The specific process is as Figure 2 shown.

[0029] 3. Real-time data processing and feedback module

[0030] Perform real-time processing on the collected and analyzed data, and timely feedback the detected disease information to relevant departments and personnel. This module adopts efficient data processing algorithms and communication technologies to ensure the real-time and reliability of data. At the same time, it provides a visual interface for users to view and manage disease information, providing decision-making support for road maintenance and management. The visual interface is as Figure 3 shown.

[0031] In the second embodiment, based on the first embodiment, a detection method for a road disease detection system based on multi-sensor fusion and multi-modal large model is proposed. A multi-sensor fusion module is used for data collection. The multi-sensor fusion module integrates multiple different types of sensors, including visible light cameras, lidars, and three-dimensional geological radars; through a reasonable sensor layout and data synchronization mechanism, synchronous collection of multi-dimensional data is realized. The visible light camera collects image information on the road surface, the lidar obtains three-dimensional shape information of the road surface, and the three-dimensional geological radar detects abnormal conditions in the underground structure; then, advanced sensor calibration and data fusion algorithms are used to fuse and process the data collected by different sensors, reducing the physical space alignment error of multi-source data and ensuring the accuracy and consistency of data.

[0032] After data fusion, use the multi-modal large model analysis and dynamic environment adaptation module for processing. Construct a multi-modal large model that integrates data features of multiple modalities such as images, point clouds, and radar signals, and automatically extracts disease features, depth, location, and environmental information; the dynamic environment adaptation module automatically adapts the AI algorithm model according to environmental features, trains and analyzes the fused multi-dimensional data, automatically identifies disease categories, extracts quantitative disease information, compares and correlates the relative positions of diseases, and achieves a disease localization accuracy of less than 3 cm and a disease recognition accuracy of greater than 90%.

[0033] The multi-modal large model analysis and dynamic environment adaptation module also realizes the twin reconstruction of road appearance and structural diseases based on the multi-modal large model. By constructing a virtual road disease model, the relative relationship of diseases is visually presented to more clearly understand the distribution and impact of diseases.

[0034] It also includes real-time data processing and feedback steps. The collected and analyzed data is processed in real time, using efficient data processing algorithms and communication technologies to ensure the real-time and reliability of the data; the detected disease information is timely fed back to relevant departments and personnel, and at the same time, a visualization interface is provided to facilitate users to view and manage disease information. This visualization interface has functions of classified display, query, and export of disease information, providing decision-making support for road maintenance and management.

[0035] In the dynamic environment adaptation module, the environmental features considered include light intensity, weather conditions, and the complexity of the road surrounding environment; according to these environmental features, the parameters and structure of the AI algorithm model are automatically adjusted to adapt to different detection environments, ensuring accurate disease recognition and positioning in different environments and improving the robustness and reliability of disease detection.

[0036] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A road disease detection system based on multi-sensor fusion and multi-modal large model, characterized in that: It includes a multi-sensor fusion module. The multi-sensor fusion module integrates multiple different types of sensors, including visible light cameras, lidars, and 3D geological radars. Through sensor layout and data synchronization mechanisms, synchronous acquisition of multi-dimensional data is achieved. The visible light camera is used to collect image information of the road surface, the lidar is used to obtain the three-dimensional shape information of the road surface, and the 3D geological radar is used to detect abnormal conditions in the underground structure. Advanced sensor calibration and data fusion algorithms are adopted to fuse the data collected by different sensors, reducing the physical space alignment error of multi-source data and ensuring the accuracy and consistency of the data.

2. The road disease detection system based on multi-sensor fusion and multi-modal large model according to claim 1, characterized in that: It also includes a multi-modal large model analysis and dynamic environment adaptation module. The multi-modal large model analysis and dynamic environment adaptation module constructs a multi-modal large model, which fuses the data features of multiple modalities such as images, point clouds, and radar signals to automatically extract disease characteristics, depth, location, and environmental information. The dynamic environment adaptation module automatically adapts the AI algorithm model according to environmental characteristics, trains and analyzes the fused multi-dimensional data, automatically identifies disease categories, extracts disease quantification information, compares and correlates the relative positions of diseases, achieving a disease positioning accuracy of less than 3 cm and a disease recognition accuracy of greater than 90%. At the same time, based on the multi-modal large model, twin reconstruction of road appearance and structural diseases is realized, intuitively presenting the relative relationship of diseases.

3. The road disease detection system based on multi-sensor fusion and multi-modal large model according to claim 2, characterized in that: It also includes a real-time data processing and feedback module. The real-time data processing and feedback module performs real-time processing on the collected and analyzed data, and timely feedbacks the detected disease information to relevant departments and personnel. Efficient data processing algorithms and communication technologies are adopted to ensure the real-time and reliability of the data. A visualization interface is provided to facilitate users to view and manage disease information, providing decision-making support for road maintenance and management.

4. The road disease detection system based on multi-sensor fusion and multi-modal large model according to claim 3, wherein: In the multi-modal large model analysis and dynamic environment adaptation module, the multi-modal large model performs fusion learning on the data features of multiple modalities through deep learning algorithms to accurately extract disease-related information. The dynamic environment adaptation module automatically adjusts the parameters and structure of the AI algorithm model according to the real-time collected environmental data, such as light intensity, weather conditions, and road surrounding environment complexity, to adapt to different detection environments and improve the accuracy and stability of disease recognition and positioning.

5. The road disease detection system based on multi-sensor fusion and multi-modal large model according to claim 4, characterized in that: In the real-time data processing and feedback module, the visualization interface has a function of classifying and displaying disease information, classifying and displaying disease information according to dimensions such as disease type, severity, and location area. At the same time, it provides data query and export functions, facilitating users to query disease information for a specific time period and specific area according to their needs, and exporting the relevant information as a file in a specified format for further analysis and processing.

6. A detection method for a road disease detection system based on multi-sensor fusion and multi-modal large model according to claim 5, characterized in that: Data acquisition is carried out using a multi-sensor fusion module, which integrates various types of sensors, including visible light cameras, lidar, and 3D geological radars. Through a reasonable sensor layout and data synchronization mechanism, synchronous acquisition of multi-dimensional data is achieved. The visible light camera captures image information of the road surface, the lidar obtains 3D shape information of the road surface, and the 3D geological radar detects anomalies in the underground structure. Subsequently, advanced sensor calibration and data fusion algorithms are used to fuse the data collected by different sensors, reducing the physical space alignment error of multi-source data and ensuring the accuracy and consistency of the data.

7. The detection method according to claim 6, characterized in that: After data fusion, processing is performed using a multi-modal large model analysis and dynamic environment adaptation module. A multi-modal large model is constructed, which fuses data features of multiple modalities such as images, point clouds, and radar signals, and automatically extracts disease characteristics, depth, location, and environmental information. The dynamic environment adaptation module automatically adapts the AI algorithm model according to environmental characteristics, trains and analyzes the fused multi-dimensional data, automatically identifies disease categories, extracts disease quantification information, and compares and correlates the relative positions of diseases, achieving a disease localization accuracy of less than 3 cm and a disease recognition accuracy of greater than 90%.

8. A detection method according to claim 7, characterized in that: The multi-modal large model analysis and dynamic environment adaptation module also realizes the twin reconstruction of road appearance and structural diseases based on the multi-modal large model. By constructing a virtual road disease model, the relative relationship of diseases is visually presented to better understand the distribution and impact of diseases.

9. The detection method according to claim 8, characterized in that: It also includes real-time data processing and feedback steps. The data after acquisition and analysis is processed in real time, using efficient data processing algorithms and communication technologies to ensure the real-time and reliability of the data. The detected disease information is timely fed back to relevant departments and personnel, and at the same time, a visualization interface is provided to facilitate users to view and manage disease information. This visualization interface has functions such as classified display, query, and export of disease information, providing decision-making support for road maintenance and management.

10. A detection method according to claim 9, characterized in that: In the dynamic environment adaptation module, the environmental characteristics relied on include light intensity, weather conditions, and the complexity of the road surrounding environment. According to these environmental characteristics, the parameters and structure of the AI algorithm model are automatically adjusted to adapt to different detection environments, ensuring accurate disease recognition and localization in different environments and improving the robustness and reliability of disease detection.

Images