Road inspection method and system based on data fusion and edge computing

Abstract

The present application discloses a road inspection method based on data fusion and edge computing, which relates to the field of road inspection. The method includes: obtaining pre-marked weak-signal sections according to a preset road inspection route; collecting real-time point cloud map information and real-time driving information before the inspection unmanned vehicle enters the weak-signal section, and constructing a road simulation model and a driving state model; after entering the weak-signal section, updating the road simulation model and the driving state model in real time; fusing the updated road simulation model and the driving state model to obtain a road driving model; collecting real-time environmental image information after the inspection unmanned vehicle enters the weak-signal section; determining the type of disease feature based on the real-time point cloud map information and the real-time environmental image information, and obtaining the section disease location information of the weak-signal section based on the disease feature type and the road driving model. The present application has the effect of efficiently and accurately detecting the type of pavement disease and the precise location of the pavement disease.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of road inspection, and in particular to a road inspection method and system based on data fusion and edge computing. Background Art

[0002] With the rapid development of society, transportation networks play a vital role in this process. Road quality significantly impacts the efficiency of urban transportation networks. Factors such as sunlight, weathering, and vehicle friction can damage roads, leading to pavement defects such as cracks and potholes. These defects can affect the speed of passing vehicles and, in severe cases, even cause traffic accidents. Therefore, timely detection of pavement defects and the implementation of appropriate measures are crucial.

[0003] Currently, most road inspections are conducted manually, with human-driven inspection vehicles conducting road inspections. When a road defect is discovered, inspectors must dismount and photograph the defect. The inspectors then use the navigation system to locate the area where the defect is located, uploading the captured images and navigation information via handheld devices. This method is not only inefficient and prone to omissions, but also highly susceptible to accidents, disrupting normal traffic flow. While some areas have adopted unmanned inspection vehicles for intelligent road inspections, addressing the inefficiency and omissions of manual inspections, in areas where navigation systems cannot accurately locate road defects, such as tunnels and remote roads, weak signals make it difficult to pinpoint their precise location, even if a road defect is detected. Summary of the Invention

[0004] The embodiments of the present application provide a road inspection method and system based on data fusion and edge computing, which is used to solve the problem in the prior art that for sections with weak signals, it is difficult to accurately locate the precise location of road surface disease areas even if road surface disease areas are detected.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a road inspection method based on data fusion and edge computing is provided, the method comprising:

[0007] Controlling the unmanned inspection vehicle to perform road inspection tasks along a preset inspection route according to a preset road inspection plan, wherein the inspection route includes pre-marked weak signal sections;

[0008] Before the unmanned inspection vehicle enters the weak signal section, the laser radar module is used to continuously collect real-time point cloud information and simultaneously obtain the real-time driving information of the unmanned inspection vehicle. The edge computing module is used to preliminarily construct a road simulation model and a driving state model based on the real-time point cloud information and the real-time driving information. After the unmanned inspection vehicle enters the weak signal section, the road simulation model and the driving state model are updated in real time.

[0009] The road simulation model and the driving state model that are updated in real time are fused in real time to obtain a road driving model;

[0010] After entering the weak-signal road section, the unmanned inspection vehicle is controlled to continuously collect road section environment information using the image acquisition module to obtain real-time environment image information.

[0011] The real-time environmental image information and the real-time point cloud map information are subjected to real-time image recognition using image recognition technology, and the section defect location information of the weak signal section is obtained by combining the image recognition result with the road driving model.

[0012] Optionally, before the unmanned inspection vehicle enters the weak signal section, the laser radar module is used to continuously collect real-time point cloud information, and real-time driving information of the unmanned inspection vehicle is obtained at the same time, and the edge computing module is used to preliminarily construct a road simulation model and a driving state model based on the real-time point cloud information and the real-time driving information. When the unmanned inspection vehicle enters the weak signal section, the road simulation model and the driving state model are updated in real time, including the following steps:

[0013] Before entering the weak signal section, continuously scan the environment of the weak signal section using the laser radar module to obtain real-time point cloud information of the weak signal section;

[0014] Using the edge computing module to preliminarily construct a road simulation model based on the real-time point cloud information obtained before entering the weak signal road section;

[0015] Acquire real-time driving information of the unmanned inspection vehicle, and use the satellite positioning module installed on the unmanned road inspection vehicle to acquire the satellite positioning information of the unmanned road inspection vehicle in real time;

[0016] Preliminarily constructing a driving state model based on the real-time driving information and the satellite positioning information;

[0017] After entering the weak signal section, the road simulation model is updated in real time based on the real-time point cloud information acquired after entering the weak signal section;

[0018] The driving state model is updated in real time based on the real-time driving information acquired after entering the weak signal road section.

[0019] Optionally, the using the edge computing module to preliminarily construct a road simulation model based on the real-time point cloud information acquired before entering the weak signal road section includes the following steps:

[0020] Preprocessing the real-time point cloud information;

[0021] Segmenting the real-time point cloud information to obtain road point cloud information;

[0022] Fitting the road point cloud information to obtain road feature information of the weak signal section;

[0023] The edge computing module is used to preliminarily construct a road simulation model based on the road feature information.

[0024] Optionally, the real-time updating of the driving state model based on the real-time driving information acquired after entering the weak signal road section comprises the following steps:

[0025] Constructing a position prediction model for the road inspection unmanned vehicle based on a recursive neural network;

[0026] Training the position prediction model using the real-time driving information and the satellite positioning information before entering the weak signal road section as a training set;

[0027] After entering the weak signal section, the driving state model obtained after entering the weak signal section is input into the trained position prediction model to obtain the predicted position of the road inspection unmanned vehicle;

[0028] The driving state model is continuously updated based on the predicted position.

[0029] Optionally, the performing real-time image recognition on the real-time environmental image information and the real-time point cloud image information using image recognition technology, and obtaining the road section defect location information of the weak signal section by combining the image recognition result with the road driving model includes the following steps:

[0030] Registering the collected real-time environmental image information and the real-time point cloud image information;

[0031] Performing image enhancement on the registered real-time environment image information using an image enhancement algorithm;

[0032] Extracting a region of interest containing road section damage features from the real-time environmental image information using an image recognition algorithm, wherein the region of interest is represented as damage image information;

[0033] Mapping the region of interest from the real-time environmental image information to the real-time point cloud image information at the same time node to obtain disease point cloud image information;

[0034] Performing edge feature comparison between the defect point cloud information and the defect image information, and obtaining the defect feature type of the weak signal road section according to the edge feature comparison result;

[0035] The road section defect location information of the weak signal road section is obtained by combining the defect feature type and the road driving model.

[0036] Optionally, performing edge feature comparison between the defect point cloud information and the defect image information, and obtaining the defect feature type of the weak signal road section according to the edge feature comparison result comprises the following steps:

[0037] Performing denoising processing on the disease point cloud image information;

[0038] Extracting the edge points of the diseased point cloud image after denoising based on the adaptive multi-feature fusion method;

[0039] Clustering the extracted disease edge points and performing fitting to obtain the point cloud edge features of the disease point cloud information;

[0040] grayscale the disease image information to obtain grayscale disease image information;

[0041] Extracting all edge points of the grayscale defect image information using an edge enhancement operator, and integrating all the edge points to obtain image edge features of the grayscale defect image information;

[0042] Performing edge matching on the edge features of the point cloud image and the edge features of the image, and determining whether the edge features of the point cloud image and the edge features of the image are the same according to the edge matching result;

[0043] If the edge feature of the point cloud image is the same as the edge feature of the image, determining that the defect point cloud image information of the weak signal road section is three-dimensional defect information;

[0044] If the edge feature of the point cloud image is different from the edge feature of the image, it is determined that the defect image information of the weak signal section is plane defect information.

[0045] Optionally, the denoising process on the disease point cloud image information includes the following steps:

[0046] Calculating the point cloud density of each point in the disease point cloud image information;

[0047] Removing points whose point cloud density is less than a preset point cloud density threshold to obtain the defect point cloud image information after preliminary denoising;

[0048] Performing plane fitting on the defect point cloud image information after preliminary denoising to obtain a defect point cloud plane;

[0049] Calculating the plane distance between each point in the disease point cloud image information and the point cloud plane;

[0050] The points whose plane distance is greater than the preset point cloud distance are eliminated to obtain the denoised point cloud image information of the defect.

[0051] Optionally, the obtaining of the road section defect location information of the weak signal road section by combining the defect feature type and the road driving model comprises the following steps:

[0052] If the defect point cloud image information of the weak signal section is three-dimensional defect information, the center point of the three-dimensional defect information is selected as the three-dimensional defect point;

[0053] Select the center point of the inspection unmanned vehicle as the disease inspection point;

[0054] Calculating the spatial distance between the three-dimensional defect point and the defect inspection point in the road driving model using the Euclidean distance formula;

[0055] Acquire the unmanned vehicle position information of the unmanned inspection vehicle on the weak signal road section based on the road driving model;

[0056] The spatial distance and the position information of the unmanned vehicle are combined to obtain the section defect position information of the three-dimensional defect information in the weak signal section.

[0057] Optionally, the method further includes:

[0058] If the defect image information of the weak signal road section is plane defect information, determining the position information of the plane defect information in the road driving model according to the position information of the unmanned vehicle;

[0059] Obtaining device parameters of the image acquisition device in the real-time image acquisition module;

[0060] The position information is corrected according to the equipment parameters to obtain the section defect position information of the plane defect information in the weak signal section.

[0061] Secondly, this application provides a road inspection system based on data fusion and edge computing, including:

[0062] a memory configured to store instructions; and

[0063] A processor is configured to call the instructions from the memory and to implement the road inspection method based on data fusion and edge computing according to any one of the first aspects when executing the instructions.

[0064] The present invention specifically adopts the following technical solutions: pre-marked weak signal sections are obtained from a preset inspection route; real-time point cloud information and real-time driving information are obtained before the inspection unmanned vehicle enters the weak signal section; a road simulation model and a driving state model are constructed based on the real-time point cloud information and real-time driving information; after entering the weak signal section, the road simulation model and the driving state model are updated in real time, and the updated road simulation model and driving state model are integrated to obtain a road driving model; after controlling the unmanned vehicle to enter the weak signal section, an image acquisition module is used to capture images of the weak signal section to obtain real-time environmental information. Image recognition technology is used to perform image recognition on the real-time environmental information; based on the image recognition results, the defect point cloud information and the defect image information in the real-time point cloud information and the real-time environmental information are respectively extracted for edge feature extraction, and the edge features of the two are compared; the type of defect feature of the defect point cloud information and the defect image information is determined based on whether the edge features of the two are the same; and different methods are used to determine the location information of the road section defect in the weak signal section according to the different types of defect features. The present invention, through the above technical solution, can realize efficient and accurate detection of the type and precise location of pavement defects in weak signal sections.

[0065] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a road inspection method based on data fusion and edge computing provided in an embodiment of the present application.

[0067] Figure 2 A schematic diagram of a flow chart for obtaining the type of damage features of a weak signal section based on edge feature comparison results provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0070] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0071] Figure 1 The following schematically shows a flow chart of a road inspection method based on data fusion and edge computing according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a road inspection method based on data fusion and edge computing, which may include the following steps:

[0072] S101. According to a preset road inspection plan, the unmanned inspection vehicle is controlled to perform a road inspection task along a preset inspection route, where the inspection route includes pre-marked weak signal sections.

[0073] In this embodiment, weak signal sections refer to sections where the unmanned inspection vehicle cannot receive satellite signals and perform satellite positioning, such as tunnels and mountain roads. Before executing the inspection mission, the weak signal sections will be marked on the inspection route. Subsequently, the position information of the unmanned inspection vehicle in the weak signal section will be calculated based on the real-time point cloud information of the weak signal section and the real-time driving information of the unmanned inspection vehicle. Based on the position information of the unmanned inspection vehicle, the location information of the road section defect collected by the image acquisition module at the current time node will be calculated.

[0074] S102. Before the inspection unmanned vehicle enters a weak signal section, the laser radar module is used to continuously collect real-time point cloud map information, and the real-time driving information of the inspection unmanned vehicle is obtained at the same time. The edge computing module is used to preliminarily build a road simulation model and a driving status model based on the real-time point cloud map information and real-time driving information. When the inspection unmanned vehicle enters a weak signal section, the road simulation model and the driving status model are updated in real time.

[0075] In this embodiment, an edge computing module is installed inside the unmanned inspection vehicle. This module boasts a high computing power of 100 TOPS (processor operations per second), enabling real-time, high-speed processing of data acquired by other modules. Before entering a weak-signal road section, the vehicle's internal laser radar (LiDAR) remains powered on, continuously scanning the surrounding environment and collecting real-time point cloud information. The LiDAR emits laser pulses, which, because they contain a large amount of energy, form a long, thin beam. When an obstacle appears near the unmanned inspection vehicle, the laser pulse is immediately reflected back. The reflected beam is received by the LiDAR receiver, which generates real-time point cloud information based on the reflection of the laser pulse. After denoising the real-time point cloud information and segmenting the road, the edge computing module is used to initially construct a road simulation model. This initially constructed road simulation model is a three-dimensional point cloud model that reflects basic road conditions, such as the width of the current weak-signal road section and road smoothness.

[0076] While the LiDAR collects real-time point cloud information, the unmanned inspection vehicle's inertial measurement module and satellite positioning module also collect real-time driving information and satellite positioning information. Real-time driving information includes the vehicle's angular velocity, acceleration, and vehicle yaw direction, as collected by the inertial measurement module. Satellite positioning information is the vehicle's position information obtained in real time by the satellite positioning module through receiving satellite information. The vehicle's real-time driving information is used to construct a driving state model that reflects information such as the vehicle's angular velocity, acceleration, and vehicle yaw direction during driving. Upon entering a weak signal section, the preliminarily constructed road simulation model and driving state model are updated using the real-time point cloud information and real-time driving information collected during entry into the weak signal section.

[0077] S103: Fusing the updated road simulation model and the driving state model in real time to obtain a road driving model.

[0078] In this embodiment, whenever the road simulation model and driving state model are updated, the edge computing module immediately fuses the two models in real time to generate a comprehensive road driving model. This model fusion process utilizes multi-source data fusion technology, primarily consisting of two phases: data-layer fusion and feature-layer fusion. First, in the data-layer fusion phase, time synchronization and spatial registration are performed on the point cloud data from the lidar and the motion data from the inertial measurement unit. Time synchronization ensures that the data collected by different sensors corresponds to the same moment in time, while spatial registration unifies the data from different coordinate systems into a common reference coordinate system. This is achieved using the Kalman filter algorithm, which effectively handles noise and errors in sensor data, improving data accuracy. In the feature-layer fusion phase, key features are extracted from the synchronized and registered data. For the road simulation model, key features are extracted, such as road geometric features, such as road width, curvature, and slope; for the driving state model, key features are extracted, such as vehicle motion features, such as speed, acceleration, and heading angle. These features are fused using a pre-defined deep learning model, which has been trained with extensive historical data and can automatically learn the complex relationships between different features. The result is a comprehensive road driving model that includes the current position, speed, direction, and predicted trajectory. This model not only reflects the vehicle's immediate state, but also includes the perception of the surrounding environment and the prediction of future movement.

[0079] S104 , after entering the weak-signal road section, the inspection unmanned vehicle is controlled to continuously collect road section environment information using the image acquisition module in the weak-signal road section to obtain real-time environment image information.

[0080] In this embodiment, after entering a weak signal section, the unmanned inspection vehicle uses the image acquisition device within the image acquisition module, such as a high-definition camera, to continuously capture images of the road conditions and surrounding environment in the weak signal section, generating real-time environmental image information. This step is to capture road surface defects, road spills, and other road defects that could affect the travel of passing vehicles.

[0081] S105. Performing real-time image recognition on the real-time environmental image information and the real-time point cloud image information using image recognition technology, and obtaining the location information of the road section defects on the weak signal section by combining the image recognition results with the road driving model.

[0082] In this embodiment, after entering a weak signal section, the collected real-time environmental image information and real-time point cloud information are first aligned. This is because the laser radar module and the image acquisition module have different installation angles when collecting information, resulting in an angle difference in the collected information. Alignment can make the information collected by the laser radar module and the image acquisition module consistent in spatial position, facilitating subsequent further analysis and processing. After the alignment step is completed, the real-time environmental image information is enhanced using an image enhancement algorithm to make the real-time environmental image information clearer, emphasize certain features of interest, expand the differences between the features of different objects in the image, and suppress features of no interest, thereby improving image quality, enriching the amount of information, and enhancing image interpretation and recognition effects. This facilitates reducing recognition errors and improving the accuracy of image recognition when subsequently performing image recognition on the real-time environmental image information.

[0083] After completing the image enhancement step, image recognition technology is used to extract regions of interest (ROIs) from the real-time environmental image information. These regions of interest are areas containing road defects, such as pavement defects and spilled objects. These regions of interest in the real-time environmental image information are designated as defect image information. Simultaneously, these regions of interest are synchronized and marked in the real-time point cloud information, which is designated as the defect point cloud information. Edge features of traffic defects are extracted from both the defect image information and the real-time point cloud information. The type of defect characteristic for the weak signal road section is determined based on whether these edge features match. The type of defect characteristic is the road section defect. If the edge features of the point cloud image match the edge features of the image, the defect characteristic type for the road section defect in both the defect point cloud information and the defect image information is 3D defect information. 3D defect information refers to road infrastructure such as water barriers, crash barriers, vehicle parts, or accidentally dropped objects. If the edge features of the point cloud image differ from those of the image, the road section defect in the defect point cloud image and image information is a planar defect. Planar defects refer to road surface defects such as cracks and potholes. After completing all of the above steps, different methods are used to determine the specific location of the defect within the weak signal section, i.e., the road section defect location information. Finally, the unmanned inspection vehicle is controlled to transmit the acquired defect feature type and its corresponding road section defect location information to the terminal system, alerting relevant personnel to promptly correct the road and eliminate the defect, facilitating the travel of subsequent vehicles.

[0084] After obtaining the location information of all road defects on weak signal sections, the type of defect characteristics, the time when the defect was discovered, and the location information of the road defect will be uploaded to the cloud storage via the 4G / 5G network to provide data support for subsequent maintenance management. After the disease data is uploaded, the relevant sections will be marked to prevent repeated uploads and waste of resources. After uploading to the cloud, the maintenance platform will classify the road defects according to the time of discovery, type of defect characteristics, and severity of the defect. According to the classification results, the maintenance platform will automatically dispatch orders. Road maintenance personnel can receive work orders through the mobile terminal program. After completing the road maintenance task, they will take a photo of the current road condition and upload the road section information of the current road section through the mobile terminal program. At the same time, the maintenance platform will also update the road section information of the road section in a timely manner to realize the business closed loop. This can effectively improve the efficiency of business processing and supervision, timely maintain roads with road section diseases, reduce unnecessary economic losses, and avoid the occurrence of traffic accidents.

[0085] In one embodiment, before the unmanned inspection vehicle enters a weak signal section, the laser radar module is used to continuously collect real-time point cloud information and simultaneously obtain the real-time driving information of the unmanned inspection vehicle. The edge computing module is used to preliminarily construct a road simulation model and a driving state model based on the real-time point cloud information and real-time driving information. When the unmanned inspection vehicle enters the weak signal section, the road simulation model and the driving state model are updated in real time, including the following steps:

[0086] Before entering a weak-signal road section, the LiDAR module is used to continuously scan the weak-signal road section to obtain real-time point cloud information of the weak-signal road section.

[0087] Use the edge computing module to preliminarily build a road simulation model based on the real-time point cloud information obtained before entering the weak signal section;

[0088] Obtain real-time driving information of the inspection unmanned vehicle, and use the satellite positioning module installed on the road inspection unmanned vehicle to obtain the satellite positioning information of the road inspection unmanned vehicle in real time;

[0089] Preliminary construction of driving status model based on real-time driving information and satellite positioning information;

[0090] After entering the weak signal section, the road simulation model is updated in real time based on the real-time point cloud information obtained after entering the weak signal section;

[0091] The driving state model is updated in real time based on the real-time driving information obtained after entering the weak signal section.

[0092] In this embodiment, before entering a weak-signal section, the laser radar inside the laser radar module emits laser pulses in all directions, and real-time point cloud information of the weak-signal section is generated based on the reflection conditions of the laser pulses. Since there is still some distance from the weak-signal section at this time, the real-time point cloud information obtained at this time is not comprehensive, and the road simulation model constructed based on the real-time point cloud information obtained at this time is also not accurate. Therefore, after entering the weak-signal section, the real-time point cloud of the weak-signal section is continuously collected to continuously update the road simulation model. While collecting the real-time point cloud information, the inertial measurement module and satellite positioning module of the inspection unmanned vehicle are also controlled to collect the real-time driving information and satellite positioning information of the inspection unmanned vehicle in real time. The inertial measurement module, which collects real-time driving information, includes devices such as accelerometers, gyroscopes, and magnetometers. It measures the vehicle's angular velocity, acceleration, speed, and vehicle yaw direction during travel. For example, when the vehicle turns, the inertial measurement module measures the direction and angular velocity of the turn, thereby calculating the vehicle's driving state at that moment. The satellite positioning module receives satellite signals from the Global Positioning System (GPS) or Beidou Satellite Navigation System (BDS) to calculate the vehicle's latitude and longitude coordinates, thereby obtaining the vehicle's precise location information, known as satellite positioning information. Because the satellite positioning module is less effective after entering a weak signal road section, a preliminary driving state model must be constructed based on real-time driving information and satellite positioning information before the vehicle enters the weak signal road section. Since the vehicle's driving state will also change after entering the weak signal road section, the driving state model must be updated in real time based on the real-time driving information collected after entering the weak signal road section. The resulting road simulation model and driving state model are highly accurate and timely.

[0093] In one embodiment, using an edge computing module to preliminarily construct a road simulation model based on real-time point cloud information acquired before entering a weak signal road section includes the following steps:

[0094] Preprocess the real-time point cloud information;

[0095] Segment the real-time point cloud information to obtain road point cloud information;

[0096] Fit the road point cloud information to obtain the road feature information of the weak signal section;

[0097] The edge computing module is used to preliminarily build a road simulation model based on road feature information.

[0098] In this embodiment, before entering a weak signal section, once the laser radar module collects real-time point cloud information, it will immediately analyze and process the real-time point cloud information. The real-time point cloud information is first pre-processed. The main function of the pre-processing step is to denoise the real-time point cloud information. In the process of collecting real-time point cloud information, it will be affected by random errors and will inevitably contain noise, which will affect the accuracy of the road simulation model constructed subsequently. Since most noise points have a high elevation, the height difference between the noise point and the adjacent points on the scan line will be much higher than that of other points. Therefore, the noise can be removed by determining the average height difference between the selected point and its adjacent points. When the average height difference is greater than the preset height difference average threshold, it indicates that the point is a noise point and should be removed. After traversing all points, all noise points are removed to obtain the denoised real-time point cloud information.

[0099] The denoised real-time point cloud image is segmented. Because the real-time point cloud image also contains road point cloud information and non-road point cloud information, such as trees and guardrails around the road, this non-road point cloud information can occupy too many computational units when constructing the road simulation model, affecting computational efficiency. Therefore, the real-time point cloud image must be segmented to obtain road point cloud information. The main principle of road segmentation is to filter ground point clouds from the real-time point cloud image. Methods for filtering ground point clouds include 2.5D grid methods, plane fitting methods, and linear extraction methods. For example, the 2.5D grid method maps 3D point cloud data to a 2D grid. Each grid cell contains the average height or other statistical features of the points within the area. Ground point clouds are filtered based on a preset average height threshold to obtain road point cloud information. Image processing techniques are used to extract features from the road point cloud image information. Road boundaries are extracted and fitted to obtain road feature information for weak signal sections. This extracted road feature information is then used to generate the road simulation model. The road simulation model can reflect the width of the current weak signal section, road smoothness and other information.

[0100] In one embodiment, updating the driving state model in real time based on real-time driving information acquired after entering a weak signal road section includes the following steps:

[0101] Build a location prediction model for unmanned road inspection vehicles based on recurrent neural networks;

[0102] The real-time driving information and satellite positioning information before entering the weak signal section are used as training sets to train the location prediction model;

[0103] After entering a weak signal section, the driving state model obtained after entering the weak signal section is input into the trained position prediction model to obtain the predicted position of the road inspection unmanned vehicle;

[0104] The driving state model is continuously updated based on the predicted position.

[0105] In this embodiment, a position prediction model for a road inspection unmanned vehicle is constructed based on a recursive neural network. For example, a position prediction model is constructed based on a long short-term memory network (LSTM). The position prediction model can predict the position information of the inspection unmanned vehicle based on the real-time driving information of the inspection unmanned vehicle. The real-time driving information and satellite positioning information before entering the weak signal section are used as a training set to train the position prediction model. The data in the training set is first cleaned to remove noise, outliers, and missing values in the data to ensure the quality and accuracy of the data. After completing the data cleaning of the training set, the structure of the LSTM model is determined, including the number of nodes in the input layer and the hidden layer, and the connection weights and bias items in the network are initialized. The training set is input into the position prediction model for forward propagation, the output of each layer is calculated, and then backpropagation is performed to calculate the error between the forward propagation output result and the backpropagation output result. The connection weights and bias items are updated according to the error until the error is reduced. The forward propagation and backpropagation are repeated many times until the preset maximum number of iterations is reached, and the position prediction model training is completed.

[0106] Since the inspection unmanned vehicle cannot be positioned by satellite signals after entering a weak-signal section, a position prediction model is constructed to predict its position. The driving status model obtained after entering the weak-signal section is input into the trained position prediction model to obtain the predicted position of the road inspection unmanned vehicle. The driving status model is updated according to the predicted position, so that the driving status model can accurately display the position, speed, acceleration and other information of the road inspection unmanned vehicle at different times.

[0107] In one embodiment, using image recognition technology to perform real-time image recognition on real-time environmental image information and real-time point cloud information, and combining the image recognition results with a road driving model to obtain the location information of road defects on weak signal sections includes the following steps:

[0108] Registering the collected real-time environmental image information and real-time point cloud information;

[0109] Use image enhancement algorithm to enhance the real-time environment image information after registration;

[0110] Using image recognition algorithms, the region of interest containing road section disease characteristics is extracted from real-time environmental image information, and the region of interest is represented as disease image information;

[0111] Map the area of interest from the real-time environmental image information to the real-time point cloud information at the same time node to obtain the disease point cloud information;

[0112] Compare the edge features of the defect point cloud information with the defect image information, and obtain the defect feature type of the weak signal section based on the edge feature comparison results;

[0113] The location information of road defects on weak signal sections is obtained by combining the types of defects characteristics and the road driving model.

[0114] In this embodiment, the real-time environmental image information and real-time point cloud information collected by the unmanned inspection vehicle after entering a weak signal section are registered. This is because the LiDAR module and the image acquisition module are located slightly differently on top of the unmanned inspection vehicle, resulting in different acquisition angles for the real-time point cloud information and the real-time environmental image information. Using the real-time environmental image information as a reference image, the real-time point cloud information is geometrically corrected through translation, rotation, and affine transformation until the angle and position of the road section defects in the real-time point cloud information and the real-time environmental image information are consistent in the image, completing the registration.

[0115] Image enhancement algorithms are used to enhance the registered real-time environmental image information. Algorithms that can be used for image enhancement include histogram equalization, grayscale world algorithm, and automatic white balance. Taking histogram equalization as an example, a transformation function is used to correct the original image's histogram to a uniformly distributed histogram. The histogram is used to adjust the image contrast, making the image appear clearer and facilitating subsequent image recognition. Image recognition technology is used to extract regions of interest containing road section defect characteristics from the real-time environmental image information. The regions of interest extracted from the real-time environmental image information are the defect image information. The real-time point cloud information mapped to the same time node is selected from the real-time environmental image information. That is, the same region of interest is extracted from the same location in the real-time point cloud information acquired at the same time as the real-time environmental image information. The regions of interest extracted from the real-time point cloud information are the defect point cloud information. This is because the real-time environmental image information and the real-time point cloud information have been registered. Theoretically, at the same time node, the road section defects in the real-time point cloud information can overlap with the road section defect locations in the real-time environmental image information. Therefore, only by performing image recognition on the real-time environmental image information, the locations of the road section defects in the image and point cloud can be preliminarily marked, effectively reducing the computational complexity of image recognition and improving image recognition efficiency.

[0116] The edge features of the defect point cloud and defect image information are extracted separately. If the edge features of the two are the same, the defect features of the road section defects in the defect point cloud and defect image information are three-dimensional defect information. Three-dimensional defect information refers to traffic facilities such as water barriers, crash barriers, and vehicle parts, or accidentally dropped objects on the road. Since these traffic facilities and accidentally dropped objects can affect road traffic and even cause traffic accidents, they are called road section defects. Because these road section defects are three-dimensional and have geometric characteristics, they are called three-dimensional defect information. If the edge features of the two are different, the defect features of the road section defects in the defect point cloud and defect image information are called planar defect information. Planar defect information refers to road defects such as cracks and potholes on the road surface. These road defects increase wear and tear on passing vehicles and increase gasoline consumption, thereby increasing transportation costs and causing economic losses. Therefore, they are also road section defects. Since these road section defects are planar, they are called planar defect information. Since the geometric features of three-dimensional disease information can be identified through lidar scanning and edge features can be extracted in the point cloud map, while planar disease information is difficult to identify through lidar scanning, it is difficult to extract edge features of planar disease information in the point cloud map, or the extracted edge features are incomplete. Therefore, the type of disease feature of the road section disease can be determined by comparing the edge features.

[0117] In one embodiment, referring to Figure 2 , performing edge feature comparison between the defect point cloud information and the defect image information, and obtaining the defect feature type of the weak signal section based on the edge feature comparison results includes the following steps:

[0118] S201, performing denoising processing on the disease point cloud image information;

[0119] S202, extracting the edge points of the diseased point cloud image after denoising based on an adaptive multi-feature fusion method;

[0120] S203, clustering the extracted disease edge points and performing fitting to obtain the edge features of the disease point cloud information;

[0121] S204, grayscale processing is performed on the disease image information to obtain grayscale disease image information;

[0122] S205, using an edge enhancement operator to extract all edge points of the grayscale disease image information, and integrating all edge points to obtain image edge features of the grayscale disease image information;

[0123] S206, performing edge matching on the edge features of the point cloud image and the edge features of the image, and determining whether the edge features of the point cloud image and the edge features of the image are the same according to the edge matching result;

[0124] S207: If the edge features of the point cloud image are the same as the edge features of the image, then the defect point cloud image information of the weak signal section is determined to be three-dimensional defect information;

[0125] S208: If the edge features of the point cloud image are different from the edge features of the image, it is determined that the defect image information of the weak signal road section is plane defect information.

[0126] In this embodiment, the radius filtering method is used to perform preliminary denoising on the defect point cloud information, and the points with lower density in the defect point cloud information are removed as noise points. It is first assumed that each point in the defect point cloud information contains at least a certain number of neighboring points within the specified radius neighborhood. The number of neighboring points of each point within the preset search radius is calculated. If the number of neighborhood points is less than the preset neighborhood point number threshold, the point is regarded as a noise point and removed. After traversing each point in the defect point cloud information, the defect point cloud information after preliminary denoising is obtained, and then the fitting plane method is used to perform local denoising on the defect point cloud information after preliminary denoising. The least squares method is used to perform plane fitting on the defect point cloud information, and the distance between each point and the plane is calculated. The points exceeding the preset plane distance threshold are extracted to finally obtain the denoised defect point cloud information.

[0127] After denoising, the denoised defect point cloud information is extracted for defect edge points based on the adaptive multi-feature fusion method, and the neighboring point height difference, smoothness, and neighboring point angle of each point in the defect point cloud information are calculated. The neighboring point height difference, smoothness, and neighboring point angle are fused to form a multi-feature fusion strategy to determine whether a point is a defect edge point. Defect edge points with a neighboring point height difference less than a preset height difference threshold, a smoothness greater than a preset smoothness threshold, and a neighboring point angle less than a preset neighboring point angle threshold are extracted. All extracted defect edge points are aggregated using a clustering algorithm, such as the DBSCAN (density-based clustering method with noise) algorithm, which aggregates defect edge points with a point cloud density greater than a preset threshold. The aggregated defect edge points are then curve-fitted using the least squares method to obtain the point cloud edge features of the defect point cloud information. The point cloud edge features can reflect the geometric shape characteristics of the road section defect.

[0128] The damage image information is grayscaled to make the road damage outline in the damage image information clearer. After obtaining the grayscale damage image information, the edge points are extracted using the edge enhancement operator. Commonly used edge enhancement operators include the Sobel operator, the Laplace operator, and the Roberts operator. Taking the Sobel operator as an example, the Sobel operator contains two sets of 3x3 matrices. The horizontal and vertical matrices are planar convolved with the grayscale reservoir water image information to obtain the horizontal and vertical brightness difference approximations respectively. The horizontal and vertical brightness difference approximations are squared, added, and then squared to obtain the image gradient of the grayscale reservoir water image information. Then set two thresholds, high and low (for example, the high threshold can be set to 70 and the low threshold can be set to 40), and traverse the entire image gradient matrix. If the image gradient of a point is higher than the high threshold, then set it to 1 in the result. If the image gradient of the point is lower than the low threshold, then set it to 0 in the result. If the image gradient of the point is between the high and low thresholds, the following judgment needs to be made: check the 8 neighboring points of the point (regarding it as the center point) to see if there is a point with an image gradient higher than the high threshold. If so, it means that the center point is connected to the determined edge point, so set it to 1 in the result, otherwise set it to 0. All points set to 1 are the edge points of the grayscale disease image information. All edge points are integrated to obtain the image edge features of the grayscale disease image information.

[0129] Edge matching is performed on the point cloud edge features and the image edge features to compare whether the two edge features overlap. If they do, the point cloud edge features are identical to the image edge features. If they do not, the point cloud edge features are different from the image edge features. If the edge features are identical, the road section defects in the defect point cloud information and the defect image information are three-dimensional defect information. Three-dimensional defect information refers to traffic facilities such as water barriers, crash barriers, vehicle parts, or accidentally dropped objects on the road. Because these traffic facilities and accidentally dropped objects can affect road traffic and even cause traffic accidents, they are called road section defects. Because these road section defects are three-dimensional and have geometric characteristics, they are called three-dimensional defect information. If the edge features of the two images differ, the road segment defects in the defect point cloud and image are planar defects. Planar defects refer to road surface defects such as cracks and potholes. These defects increase wear and tear on passing vehicles and fuel consumption, thereby increasing transportation costs and causing economic losses. These defects are also considered planar defects. Because these defects are planar, they are named planar defects. Since the geometric features of three-dimensional defects can be identified through LiDAR scanning, edge features can be extracted from the point cloud. However, planar defects are difficult to identify through LiDAR scanning, and therefore, edge features are difficult to extract from the point cloud, or the extracted edge features are incomplete. Therefore, the type of road segment defect feature can be determined by comparing the edge features. By comparing the edge features of the point cloud and the image to determine whether they are identical, the type of road segment defect feature can be determined. Using different methods to obtain road segment defect location information based on the type of defect feature can reduce computational complexity and improve the efficiency of road segment defect location.

[0130] In one embodiment, denoising the disease point cloud image information includes the following steps:

[0131] Calculate the point cloud density of each point in the disease point cloud map information;

[0132] Remove points with a point cloud density less than a preset point cloud density threshold to obtain the defect point cloud map information after preliminary denoising;

[0133] Perform plane fitting on the defect point cloud image after preliminary denoising to obtain the defect point cloud plane;

[0134] Calculate the plane distance between each point in the disease point cloud information and the point cloud plane;

[0135] The points whose plane distance is greater than the preset point cloud distance are eliminated to obtain the denoised defect point cloud information.

[0136] In this embodiment, the radius filtering method is used to perform preliminary denoising on the defect point cloud information, and the points with lower density in the defect point cloud information are removed as noise points. It is first assumed that each point in the defect point cloud information contains at least a certain number of neighboring points within the specified radius neighborhood. The number of neighboring points of each point within the preset search radius is calculated. If the number of neighborhood points is less than the preset neighborhood point number threshold, the point is regarded as a noise point and removed. After traversing each point in the defect point cloud information, the defect point cloud information after preliminary denoising is obtained, and then the fitting plane method is used to perform local denoising on the defect point cloud information after preliminary denoising. The least squares method is used to perform plane fitting on the defect point cloud information, and the distance between each point and the plane is calculated. The points exceeding the preset plane distance threshold are extracted to finally obtain the denoised defect point cloud information.

[0137] In one embodiment, obtaining the road section defect location information of the weak signal road section by combining the defect feature type and the road driving model includes the following steps:

[0138] If the defect point cloud information of the weak signal section is three-dimensional defect information, the center point of the three-dimensional defect information is selected as the three-dimensional defect point;

[0139] Select the center point of the inspection unmanned vehicle as the disease inspection point;

[0140] The Euclidean distance formula is used to calculate the spatial distance between the three-dimensional disease point and the disease inspection point in the road driving model;

[0141] Obtain the location information of the unmanned inspection vehicle on the weak signal road section based on the road driving model;

[0142] Combining the spatial distance and the location information of the unmanned vehicle, the three-dimensional defect information is obtained in the road section with weak signal.

[0143] In this embodiment, if the defect point cloud information of the weak signal section is three-dimensional defect information, the center point of the three-dimensional defect information is selected as the three-dimensional defect point, and the center point of the inspection unmanned vehicle is selected as the defect inspection point. The purpose of selecting the center point is to facilitate the subsequent calculation of the distance between the three-dimensional defect information and the inspection unmanned vehicle. The spatial distance between the three-dimensional defect point and the defect inspection point is calculated using the Euclidean distance formula. The Euclidean distance formula is: Among them, (x1, y1, z1) is the coordinate of the disease inspection point in space, and (x2, y2, z2) is the coordinate of the three-dimensional disease point in space.

[0144] Based on the road driving model, the inspection vehicle's position information on weak signal sections is obtained. Combined with the spatial distance and the vehicle's position information, the three-dimensional defect information is used to determine the location of road defects on weak signal sections. Because the road driving model updates the longitude and latitude of the inspection vehicle in real time, if the spatial distance between the three-dimensional defect information and the inspection vehicle is known, the longitude and latitude of the inspection vehicle can be used as a reference to determine the three-dimensional defect information, i.e., the location of the road defect on the weak signal section.

[0145] In one embodiment, the method further comprises:

[0146] If the defect image information of the weak signal section is plane defect information, the position information of the plane defect information in the road driving model is determined according to the position information of the unmanned vehicle;

[0147] Obtain device parameters of the image acquisition device in the real-time image acquisition module;

[0148] The location information is corrected according to the equipment parameters to obtain the section defect location information of the plane defect information in the weak signal section.

[0149] In this embodiment, if the defect point cloud information for a weak-signal road section is planar defect information, the location information of the planar defect information within the road model is determined based on the unmanned vehicle's position information. The position information of the unmanned inspection vehicle at the time the image acquisition module captured the planar defect information is obtained. Using the unmanned inspection vehicle as a reference, a preliminary estimate of the distance between the planar defect information and the unmanned inspection vehicle is made to obtain the location information of the planar defect information within the road model. The internal and external parameters of the image acquisition device in the image acquisition module are obtained. For example, the focal length, pixel size, and distortion coefficient of the HD camera are obtained. The position information is then corrected using the internal and external parameters of the image acquisition device to obtain the section defect location information of the planar defect information within the weak-signal road section. Furthermore, compared to the section defect location information of the three-dimensional defect information within the weak-signal road section, the section defect location information of the planar defect information within the weak-signal road section has a certain error, but this error can be ignored. However, the latter requires less computation and is more efficient.

[0150] This application also discloses a road inspection system based on data fusion and edge computing, including:

[0151] a memory configured to store instructions; and

[0152] The processor is configured to call instructions from the memory and implement the above-mentioned road inspection method based on data fusion and edge computing when executing the instructions.

[0153] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0154] Among them, the memory can be an internal storage unit of a computer device, such as a hard disk or memory of a computer device, or an external storage device of a computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) or flash memory card (FC) equipped on the computer device. In addition, the memory can also be a combination of an internal storage unit and an external storage device of a computer device. The memory is used to store computer programs and other programs and data required by the computer device. The memory can also be used to temporarily store data that has been output or is to be output. This application does not impose any restrictions on this.

[0155] An embodiment of the present application also provides a machine-readable storage medium, which stores instructions for enabling a machine to execute the above-mentioned road inspection method based on data fusion and edge computing.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of the processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0158] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0160] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0161] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0162] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transit media), such as modulated data signals and carrier waves.

[0163] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0164] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A road inspection method based on data fusion and edge computing, characterized in that: Applied to an unmanned inspection vehicle, the unmanned inspection vehicle includes an edge computing module, a laser radar module, and an image acquisition module. The edge computing module is arranged inside the unmanned inspection vehicle, and the laser radar module and the image acquisition module are arranged side by side on the top of the unmanned inspection vehicle. The method includes the following steps: Controlling the unmanned inspection vehicle to perform road inspection tasks along a preset inspection route according to a preset road inspection plan, wherein the inspection route includes pre-marked weak signal sections; Before the unmanned inspection vehicle enters the weak signal section, the laser radar module is used to continuously collect real-time point cloud information and simultaneously obtain the real-time driving information of the unmanned inspection vehicle. The edge computing module is used to preliminarily construct a road simulation model and a driving state model based on the real-time point cloud information and the real-time driving information. After the unmanned inspection vehicle enters the weak signal section, the road simulation model and the driving state model are updated in real time. The road simulation model and the driving state model that are updated in real time are fused in real time to obtain a road driving model; After entering the weak-signal road section, the unmanned inspection vehicle is controlled to continuously collect road section environment information of the weak-signal road section using the image acquisition module to obtain real-time environment image information; The real-time environmental image information and the real-time point cloud map information are subjected to real-time image recognition using image recognition technology, and the section defect location information of the weak signal section is obtained by combining the image recognition result with the road driving model.

2. The method according to claim 1, characterized in that Before the unmanned inspection vehicle enters the weak signal section, the laser radar module is used to continuously collect real-time point cloud information and simultaneously obtain real-time driving information of the unmanned inspection vehicle. The edge computing module is used to preliminarily construct a road simulation model and a driving state model based on the real-time point cloud information and the real-time driving information. After the unmanned inspection vehicle enters the weak signal section, the road simulation model and the driving state model are updated in real time, including the following steps: Before entering the weak signal section, continuously scan the environment of the weak signal section using the laser radar module to obtain real-time point cloud information of the weak signal section; Using the edge computing module to preliminarily construct a road simulation model based on the real-time point cloud information obtained before entering the weak signal road section; Acquire real-time driving information of the unmanned inspection vehicle, and use the satellite positioning module installed on the unmanned road inspection vehicle to acquire the satellite positioning information of the unmanned road inspection vehicle in real time; Preliminarily constructing a driving state model based on the real-time driving information and the satellite positioning information; After entering the weak signal section, the road simulation model is updated in real time based on the real-time point cloud information acquired after entering the weak signal section; The driving state model is updated in real time based on the real-time driving information acquired after entering the weak signal road section.

3. The method according to claim 2, characterized in that The use of the edge computing module to preliminarily construct a road simulation model based on the real-time point cloud information obtained before entering the weak signal road section includes the following steps: Preprocessing the real-time point cloud information; Segmenting the real-time point cloud information to obtain road point cloud information; Fitting the road point cloud information to obtain road feature information of the weak signal section; The edge computing module is used to preliminarily construct a road simulation model based on the road feature information.

4. The method according to claim 2, characterized in that The real-time updating of the driving state model based on the real-time driving information obtained after entering the weak signal road section comprises the following steps: Constructing a position prediction model for the road inspection unmanned vehicle based on a recursive neural network; Training the position prediction model using the real-time driving information and the satellite positioning information before entering the weak signal road section as a training set; After entering the weak signal section, the driving state model obtained after entering the weak signal section is input into the trained position prediction model to obtain the predicted position of the road inspection unmanned vehicle; The driving state model is continuously updated based on the predicted position.

5. The method according to claim 1, wherein The method of performing real-time image recognition on the real-time environmental image information and the real-time point cloud image information using image recognition technology, and obtaining the road section defect location information of the weak signal section by combining the image recognition result with the road driving model comprises the following steps: Registering the collected real-time environmental image information and the real-time point cloud image information; Performing image enhancement on the registered real-time environment image information using an image enhancement algorithm; Extracting a region of interest containing road section damage features from the real-time environmental image information using an image recognition algorithm, wherein the region of interest is represented as damage image information; Mapping the region of interest from the real-time environmental image information to the real-time point cloud image information at the same time node to obtain disease point cloud image information; Performing edge feature comparison between the defect point cloud information and the defect image information, and obtaining the defect feature type of the weak signal road section according to the edge feature comparison result; The road section defect location information of the weak signal road section is obtained by combining the defect feature type and the road driving model.

6. The method according to claim 5, characterized in that The step of performing edge feature comparison between the defect point cloud information and the defect image information and obtaining the defect feature type of the weak signal section according to the edge feature comparison result comprises the following steps: Performing denoising processing on the disease point cloud image information; Extracting the edge points of the diseased point cloud image after denoising based on the adaptive multi-feature fusion method; Clustering the extracted disease edge points and performing fitting to obtain the point cloud edge features of the disease point cloud information; grayscale the disease image information to obtain grayscale disease image information; Extracting all edge points of the grayscale defect image information using an edge enhancement operator, and integrating all the edge points to obtain image edge features of the grayscale defect image information; Performing edge matching on the edge features of the point cloud image and the edge features of the image, and determining whether the edge features of the point cloud image and the edge features of the image are the same according to the edge matching result; If the edge feature of the point cloud image is the same as the edge feature of the image, determining that the defect point cloud image information of the weak signal road section is three-dimensional defect information; If the edge feature of the point cloud image is different from the edge feature of the image, it is determined that the defect image information of the weak signal section is plane defect information.

7. The method according to claim 6, characterized in that The denoising process of the disease point cloud image information comprises the following steps: Calculating the point cloud density of each point in the disease point cloud image information; Removing points whose point cloud density is less than a preset point cloud density threshold to obtain the defect point cloud image information after preliminary denoising; Performing plane fitting on the defect point cloud image information after preliminary denoising to obtain a defect point cloud plane; Calculating the plane distance between each point in the disease point cloud image information and the point cloud plane; The points whose plane distance is greater than the preset point cloud distance are eliminated to obtain the denoised point cloud image information of the defect.

8. The method according to claim 6, characterized in that The step of obtaining the road section defect location information of the weak signal road section by combining the defect characteristic type and the road driving model comprises the following steps: If the defect point cloud image information of the weak signal section is three-dimensional defect information, the center point of the three-dimensional defect information is selected as the three-dimensional defect point; Select the center point of the inspection unmanned vehicle as the disease inspection point; Calculating the spatial distance between the three-dimensional defect point and the defect inspection point in the road driving model using the Euclidean distance formula; Acquire the unmanned vehicle position information of the unmanned inspection vehicle on the weak signal road section based on the road driving model; The spatial distance and the position information of the unmanned vehicle are combined to obtain the section defect position information of the three-dimensional defect information in the weak signal section.

9. The method according to claim 8, characterized in that The method further comprises: If the defect image information of the weak signal road section is plane defect information, determining the position information of the plane defect information in the road driving model according to the position information of the unmanned vehicle; Obtaining device parameters of the image acquisition device in the real-time image acquisition module; The position information is corrected according to the equipment parameters to obtain the section defect position information of the plane defect information in the weak signal section.

10. A road inspection system based on data fusion and edge computing, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and to implement the road inspection method based on data fusion and edge computing according to any one of claims 1 to 9 when executing the instructions.

Images