A road damage detection method and system based on multimodal data

Abstract

The present invention relates to the field of intelligent traffic safety detection technology, and provides a road damage detection method and system based on multimodal data. The road damage detection method based on multimodal data includes respectively encoding the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; respectively calculating the similarity between the image features and the text label features, and the similarity between the time series features and the text label features, and fusing and generating labels of a conditional generative adversarial network; inputting a random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels into the conditional generative adversarial network for training to generate multimodal data; and predicting the multimodal data to obtain damage classification information. The present invention realizes the automation of road damage detection and maintenance management.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traffic safety detection technology, and in particular to a road damage detection method and system based on multimodal data. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, researchers have realized automated road damage detection based on artificial intelligence methods and equipment. However, the road damage detection task under abnormal weather conditions is often affected by factors such as illumination, occlusion, and differences in the target itself, which undermines the accuracy and robustness of the detection equipment, making it impossible for the detection equipment to accurately distinguish between damage on the road and other confusing backgrounds. Summary of the invention

[0004] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a road damage detection method and system based on multimodal data. On the one hand, the present invention trains a multimodal road damage detection model by using multiple data such as time series data collected by image and vibration sensors, thereby realizing comprehensive detection of road conditions by edge detection equipment and improving the detection performance of the model. On the other hand, by transmitting and recording the acquired real-time road information and detected road damage information to the cloud service module, it is convenient for the staff to carry out subsequent road repairs.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A first aspect of the present invention provides a road damage detection method based on multimodal data.

[0007] A road damage detection method based on multimodal data, comprising:

[0008] The acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label are encoded respectively to obtain image features, time series features, and text label features; the similarity between the image features and the text label features, and the similarity between the time series features and the text label features are calculated respectively, and the labels of the generated conditional generative adversarial network are fused;

[0009] Obtain road image data, road surface vibration time series data during vehicle driving, and geographic location data of road damage;

[0010] A random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels are input into the conditional generative adversarial network for training to generate fused multimodal data suitable for graph convolutional neural network input;

[0011] The multimodal data is predicted and combined with the geographical location data of road damage to obtain the road damage location and damage classification information.

[0012] Furthermore, the method of encoding the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label respectively to obtain image features, time series features, and text label features includes:

[0013] Based on the historical road image data, a visual encoder is used to encode and obtain image features;

[0014] Based on the historical vibration time series data, a time series encoder is used to encode and obtain time series features;

[0015] Based on the road damage label, a text encoder is used to encode and obtain text label features.

[0016] Furthermore, the method of respectively calculating the similarity between the image feature and the text label feature, and the similarity between the time series feature and the text label feature, and fusing the generated condition to generate the label of the adversarial network includes:

[0017] The text label feature is regarded as a node of the entire graph structure. The similarity between the text label feature node and the image subgraph is calculated, and the similarity between the text label feature node and the vibration timing subgraph is calculated to connect the text label feature with the image subgraph and the vibration timing subgraph to obtain a multimodal graph structure to fuse the labels of the generative conditional generative adversarial network.

[0018] Furthermore, the process of fusing the labels of the generative conditional adversarial network adopts the following formula:

[0019]

[0020] In the formula, is the encoded label text feature, is the input matrix of time series features, is the input matrix of image features; The function is used to calculate the correlation between nodes; is the input matrix containing label text features and time series features, is the input matrix containing label text features and image features, for The adjacency matrix of for The adjacency matrix of .

[0021] Furthermore, the method of inputting a random noise vector, road image data, vibration time series data and a conditional generative adversarial network label into a conditional generative adversarial network for training to generate fused multimodal data suitable for graph convolutional neural network input includes:

[0022] A random noise vector, road image data and vibration time series data are input into the generator, and converted into multimodal data through deconvolution layers and fully connected layers; the multimodal data generated by the generator and the conditional generative adversarial network label are input into the discriminator, and feature extraction is performed through convolution layers and fully connected layers to obtain probability values; based on the probability values, the adversarial loss function is used to update the parameters of the generator and the discriminator.

[0023] Furthermore, in the training phase, after obtaining the multimodal data, it also includes:

[0024] The multimodal data is input into the graph convolutional neural network for training, and the output results of the graph convolutional neural network are calculated through forward propagation; the detection category loss and detection box loss of the output results are calculated using the cross entropy loss function and the IoU loss function respectively to update the parameters of the graph convolutional neural network; the training weights are obtained, and the training weights are loaded into the edge detection device to obtain road damage detection results in real time.

[0025] A second aspect of the present invention provides a road damage detection system based on multimodal data.

[0026] A road damage detection system based on multimodal data, comprising:

[0027] The cloud service module is configured to: respectively encode the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; respectively calculate the similarity between the image features and the text label features, and the similarity between the time series features and the text label features, and fuse the generated conditional generative adversarial network labels;

[0028] A data acquisition module, which is configured to: acquire road image data, vibration time series data of the road surface during vehicle driving, and geographical location data of road damage;

[0029] The cloud service module is further configured to: input a random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels into the conditional generative adversarial network for training, and generate fused multimodal data adapted to the graph convolutional neural network input;

[0030] The edge detection module is configured to: predict the multimodal data, combine the geographical location data of the road damage, and obtain the road damage location and damage classification information.

[0031] A third aspect of the present invention provides a road damage detection system based on multimodal data.

[0032] A road damage detection system based on multimodal data, comprising: a camera installed on a vehicle, a vibration sensor installed on a vehicle chassis, a GPS locator installed on the vehicle, and a cloud service module;

[0033] The cloud service module encodes the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; respectively calculates the similarity between the image features and the text label features, and the similarity between the time series features and the text label features, and fuses the generated conditional generative adversarial network labels;

[0034] The camera is used to obtain road image data, the vibration sensor is used to obtain the vibration time series data of the road surface during vehicle driving, and the GPS locator is used to obtain the geographical location data of road damage;

[0035] The cloud service module is also used to input a random noise vector, road image data, vibration time series data and conditional generative adversarial network labels into the conditional generative adversarial network for training, to generate fused multimodal data suitable for graph convolutional neural network input; to predict the multimodal data, and to obtain the road damage location and damage classification information by combining the geographical location data of the road damage.

[0036] Furthermore, the system also includes an edge detection device, which sends a communication request to the cloud service module by using a network connection, and after the cloud service module receives the instruction, transmits the road damage location and damage classification information to the cloud service module.

[0037] Furthermore, a database is used to store the detected road damage location and damage category information, and the InnoDB engine is used as its storage engine.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention constructs a real-time road damage detection system with a cloud-edge architecture, realizes the automation of road damage detection and maintenance management, and improves the system's central control and real-time processing capabilities under abnormal weather conditions.

[0040] In the process of model construction, the present invention adopts a road damage detection method that combines data fusion of the cGAN network and prediction of the graph convolutional neural network. By combining road images and time series data to train the model, the detection accuracy of the model in complex environments is further improved.

[0041] The present invention analyzes the actual computing power conditions of the cloud service module and the edge detection device, performs accurate but complex label generation in the cloud service module, and performs fast data fusion and reasoning in the edge detection device.

[0042] The present invention establishes a cloud database to store results including the detected road damage category and the road damage location located by GPS, so as to facilitate the staff to carry out subsequent road repair tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0044] Figure 1 is a flow chart of a road damage detection method based on multimodal data shown in the present invention;

[0045] Figure 2 It is a structural diagram of a multimodal road damage detection method based on a graph convolutional neural network shown in the present invention;

[0046] Figure 3 It is a framework diagram of a road damage detection system based on multimodal data shown in the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] Embodiment 1

[0051] like Figure 1 As shown, this embodiment provides a road damage detection method based on multimodal data, including:

[0052] The acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label are encoded respectively to obtain image features, time series features, and text label features; the similarity between the image features and the text label features, and the similarity between the time series features and the text label features are calculated respectively, and the labels of the generated conditional generative adversarial network are fused;

[0053] Obtain road image data, road surface vibration time series data during vehicle driving, and geographic location data of road damage;

[0054] A random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels are input into the conditional generative adversarial network for training to generate fused multimodal data suitable for graph convolutional neural network input;

[0055] The multimodal data is predicted and combined with the geographical location data of road damage to obtain the road damage location and damage classification information.

[0056] The present invention uses a multimodal convolutional neural network to generate multimodal data fusion labels to train the cGAN network so that it can directly learn multimodal data fusion rules, eliminating the encoding and alignment process. This design enables the edge detection module to obtain multimodal fusion data that is suitable for the input of the prediction model after one forward network inference, greatly accelerating the speed of edge detection.

[0057] The present embodiment is described in detail below:

[0058] Capture road images in real time and capture the status of the road by continuously taking photos and recording videos.

[0059] Image scaling, image cropping, and data augmentation are performed after acquisition. Image scaling is achieved by resizing the image to a fixed size using the nearest neighbor interpolation method. Random cropping methods are used to remove irrelevant parts or adjust the composition of the image. Image datasets are augmented by using random transformations such as rotation, translation, and flipping.

[0060] Real-time monitoring of the vibration time series data of the road surface during vehicle driving, including vibration frequency and vibration amplitude.

[0061] The vibration time series data is filtered to remove high-frequency noise and low-frequency drift, retaining the effective vibration signal; then, it is amplified to enhance the signal strength for more accurate analysis; finally, the analog signal is converted into a digital signal for further processing and analysis.

[0062] Collect satellite signals to determine the geographic location of equipment or road damage. The collected location data includes latitude, longitude, altitude, timestamp, etc.

[0063] The present invention uses a conditional generative adversarial network to fuse image data and vibration time series data, and uses a graph convolutional neural network for training and prediction. The road damage image is used as the main detection object, and the time series data of the vehicle vibration sensor is used to correct the detection objects that are easily confused and misjudged.

[0064] The cGAN network consists of two parts: the generator and the discriminator.

[0065] The model training process is as follows: First, the generator receives a random noise vector and three conditional vectors of image, vibration time series and text as input, and converts the noise and conditional information into multimodal fusion data suitable for graph convolutional neural network input through deconvolution layers and fully connected layers. Next, the discriminator receives the data and labels generated by the generator (the labels are matrices that conform to the input of the graph convolutional neural network after fusion, and the cGAN network model is trained by using vectors in this format as labels) as input, extracts features through a series of convolutional layers and fully connected layers, and outputs a probability value indicating the probability that the input data is real data. Finally, the generator and discriminator parameters are updated according to the set adversarial loss function. Repeat the training until the generator and discriminator converge or reach the preset number of training rounds. The model adversarial training process is as follows:

[0066] (1)

[0067] In the formula, is random noise, is the conditional vector, For labels; is random noise The data distribution, Yes Tags Data distribution; is the discriminator, is the generator. Among them, the most critical label It is obtained by constructing a multimodal graph.

[0068] The labels of the conditional generative adversarial network are obtained by constructing a multimodal graph. During the model construction process, the three types of data, image, time series, and label text, are encoded using corresponding encoders to obtain an input format that adapts to the graph structure; then, the sub-graphs are fused; finally, the fused data is obtained as the label of the conditional generative adversarial network. The specific process is as follows:

[0069] (1) The model first encodes and reconstructs the input data according to the structure of the multimodal graph. The encoder constructs a of Matrix. At the same time, the association between the node and other nodes is constructed to obtain a Adjacency Matrix .matrix and matrix As the input of the time series and visual subgraphs. The text label is passed through the text encoder to obtain the text label feature .

[0070] (2) Introducing text labels into the model to further achieve the fusion of the two sub-images of image and vibration time series. The fusion process is as follows: The text label feature is regarded as a node of the entire graph structure, and the text label feature is realized by calculating the similarity between the text label feature node and the image subgraph and the vibration time series subgraph. With the timing subgraph and image subgraph By connecting to form a whole multimodal graph, the real label of multimodal data is obtained. The specific generation formula is:

[0071] (2)

[0072] (3)

[0073] In the formula, is the encoded text feature, is the input matrix of time series features, is the input matrix of image features; The function is used to calculate the correlation between nodes. Here, the cosine similarity function is used; is the input matrix containing label text features and time series features, is the input matrix containing label text features and image features, for The adjacency matrix of for The adjacency matrix of .

[0074] The fused multimodal data is sent to the graph convolutional neural network for training to obtain the training weights, and the network training weights are loaded into the edge detection device to obtain the road damage detection results in real time. The training process of the network is as follows: First, the data is input into the graph convolutional neural network, and the output of the graph convolutional neural network is calculated through forward propagation. The cross entropy loss function and the IoU loss function are used to calculate the detection category loss and the detection box loss of the output result respectively; then, the loss contribution of each parameter is calculated layer by layer through the back propagation algorithm, and the parameters of the graph convolutional neural network are updated using the stochastic gradient descent algorithm to gradually reduce the loss; then, the previous steps are repeated until the specified number of training rounds is reached or the conditions for stopping training are met; finally, the performance and generalization ability of the trained graph convolutional neural network are evaluated using the test set.

[0075] Embodiment 2

[0076] like Figure 3 As shown, this embodiment provides a road damage detection system based on multimodal data, including:

[0077] The cloud service module is configured to: respectively encode the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; respectively calculate the similarity between the image features and the text label features, and the similarity between the time series features and the text label features, and fuse the generated conditional generative adversarial network labels;

[0078] A data acquisition module, which is configured to: acquire road image data, vibration time series data of the road surface during vehicle driving, and geographical location data of road damage;

[0079] The cloud service module is further configured to: input a random noise vector, road image data, vibration time series data, and a conditional generative adversarial network label into the conditional generative adversarial network for training, and generate fused multimodal data adapted to the graph convolutional neural network input;

[0080] The edge detection module is configured to: predict the multimodal data, combine the geographical location data of the road damage, and obtain the road damage location and damage classification information.

[0081] In some embodiments, the cloud service module is further configured to: encode based on historical road image data using a visual encoder to obtain image features; encode based on historical vibration time series data using a time series encoder to obtain time series features; and encode based on road damage labels using a text encoder to obtain text label features.

[0082] In some embodiments, the cloud service module is further configured to: regard the text label feature as a node of the entire graph structure, calculate the similarity between the text label feature node and the image subgraph, and calculate the similarity between the text label feature node and the vibration timing subgraph, so as to connect the text label feature with the image subgraph and the vibration timing subgraph to obtain a multimodal graph, so as to fuse and generate multimodal data.

[0083] In some embodiments, the cloud service module is further configured to: the process of fusing and generating multi-conditional generative adversarial network labels adopts the following formula:

[0084]

[0085] In the formula, is the encoded label text feature, is the input matrix of time series features, is the input matrix of image features; The function is used to calculate the correlation between nodes; is the input matrix containing label text features and time series features, is the input matrix containing label text features and image features, for The adjacency matrix of for The adjacency matrix of .

[0086] In some embodiments, a random noise vector, road image data, and vibration time series data are input into a generator, and converted into multimodal data through a deconvolution layer and a fully connected layer; the multimodal data generated by the generator and the conditional generative adversarial network label are input into a discriminator, and feature extraction is performed through a convolution layer and a fully connected layer to obtain a probability value; based on the probability value, an adversarial loss function is used to update the parameters of the generator and the parameters of the discriminator.

[0087] Embodiment 3

[0088] like Figure 3 As shown, this embodiment provides a road damage detection system based on multimodal data, including: a camera installed on a vehicle, a vibration sensor installed on a vehicle chassis, a GPS locator installed on the vehicle, and a cloud service module;

[0089] The cloud service module encodes the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; respectively calculates the similarity between the image features and the text label features, and the similarity between the time series features and the text label features, and fuses the generated conditional generative adversarial network labels;

[0090] The camera is used to obtain road image data, the vibration sensor is used to obtain the vibration time series data of the road surface during vehicle driving, and the GPS locator is used to obtain the geographical location data of road damage;

[0091] The cloud service module is also used to input a random noise vector, road image data, vibration time series data and conditional generative adversarial network labels into the conditional generative adversarial network for training, to generate fused multimodal data suitable for graph convolutional neural network input; to predict the multimodal data, and to obtain the road damage location and damage classification information by combining the geographical location data of the road damage.

[0092] like Figure 2 As shown in the figure, the system includes three parts: data acquisition module, edge detection module and cloud service module. The data acquisition module collects information about the road, including road images captured by the camera, vehicle vibration signals collected by the vibration sensor and real-time GPS positioning information of road damage; the edge detection module performs real-time road damage detection and sends the detection results to the cloud service module; the cloud service module mainly performs two parts of work. One part is to fuse and train the model based on the data collected by the data acquisition module, and send the trained model weights to the edge detection module for road damage detection. The other part is to store the road damage type (loss types include cracks, potholes, wear, subsidence, blistering and cracking) and location information in the detection results into the database, so that the staff can carry out subsequent road repair tasks.

[0093] The core task of the system is to achieve the fusion of multimodal road damage data. The present invention uses a graph structure to generate multimodal data fusion labels in the cloud service module to train the cGAN network so that it can directly learn the multimodal data fusion rules, eliminating the encoding and alignment process. This design enables the edge detection module to obtain multimodal fusion data that is adapted to the prediction model input after one forward network inference, greatly speeding up the edge detection speed.

[0094] 1. Data acquisition module

[0095] The data acquisition module is mainly responsible for sensor data acquisition and data preprocessing, and the data acquisition work included in it is divided into the following three parts.

[0096] 1.1 Image data acquisition

[0097] Cameras are installed at the front and rear of the vehicle to capture road images in real time, capturing the status of the road by continuously taking photos and recording videos.

[0098] The images captured by the camera are scaled, cropped, and augmented in sequence. Image scaling is achieved by resizing the image to a fixed size using the nearest neighbor interpolation method. Random cropping methods are used to remove irrelevant parts or adjust the composition of the image. Image datasets are augmented by using random transformations such as rotation, translation, and flipping.

[0099] 1.2 Vibration timing signal acquisition

[0100] The vibration sensor is installed on the chassis of the inspection vehicle to monitor the vibration of the road surface in real time during the vehicle's driving, including the vibration frequency and vibration amplitude.

[0101] The analog signal collected by the vibration sensor is filtered to remove high-frequency noise and low-frequency drift, retaining the effective vibration signal; then, it is amplified to enhance the signal strength for more accurate analysis; finally, the analog signal is converted into a digital signal for further processing and analysis.

[0102] 1.3 Location Information

[0103] GPS locators are used in road inspections to determine the geographic location of equipment or road damage through satellite signals. The collected location data includes latitude, longitude, altitude, timestamp and other information. The collected location data can be stored in real time in a local storage device or transmitted to edge detection devices and cloud service modules through a 4G wireless communication module.

[0104] 2. Cloud service module

[0105] The cloud service module performs the following two tasks: 2.1 Model construction; 2.2 Road damage information storage.

[0106] 2.1 Model Construction

[0107] The present invention uses a conditional generative adversarial network to fuse image data and vibration time series data, and uses a graph convolutional neural network for training and prediction. The road damage image is used as the main detection object, and the time series data of the vehicle vibration sensor is used to correct the detection objects that are easily confused and misjudged.

[0108] 2.1.1 cGAN network training

[0109] The cGAN network consists of two parts: the generator and the discriminator.

[0110] The model training process is as follows: First, the generator receives a random noise vector and three conditional vectors of image, vibration time series and text as input, and converts the noise and conditional information into multimodal fusion data suitable for graph convolutional neural network input through deconvolution layers and fully connected layers. Next, the discriminator receives the data and labels generated by the generator as input, extracts features through a series of convolution layers and fully connected layers, and outputs a probability value indicating the probability that the input data is real data. Finally, the generator and discriminator parameters are updated according to the set adversarial loss function. Repeat the training until the generator and discriminator converge or reach the preset number of training rounds. The model adversarial training process is as follows:

[0111] (1)

[0112] In the formula, is random noise, is the conditional vector, For labels; is random noise The data distribution, Yes Tags Data distribution; is the discriminator, is the generator. Among them, the most critical label It is obtained by constructing a multimodal graph.

[0113] 2.1.2 Multimodal Fusion Data Label Acquisition

[0114] The labels of the conditional generative adversarial network are obtained by constructing a multimodal graph. During the model construction process, the three types of data, image, time series, and label text, are encoded using corresponding encoders to obtain the input format of the adaptation graph; then, the sub-graphs are fused; finally, the fused data is obtained as the label of the conditional generative adversarial network. The specific process is as follows:

[0115] (1) The model first encodes and reconstructs the input data according to the structure of the multimodal graph. The encoder constructs a of Matrix. At the same time, the association between the node and other nodes is constructed to obtain a Adjacency Matrix .matrix and matrix As the input of the time series and visual subgraphs. The text label is passed through the text encoder to obtain the text label feature .

[0116] (2) Introducing text labels into the model to further achieve the fusion of the two sub-images of image and vibration time series. The fusion process is as follows: The text label feature is regarded as a node of the entire graph structure, and the text label feature is realized by calculating the similarity between the text label feature node and the image subgraph and the vibration time series subgraph. With the timing subgraph and image subgraph By connecting to form a whole multimodal graph, the real label of multimodal data is obtained. The specific generation formula is:

[0117] (2)

[0118] (3)

[0119] In the formula, is the encoded text feature, is the input matrix of time series features, is the input matrix of image features; The function is used to calculate the correlation between nodes. Here, the cosine similarity function is used; is the input matrix containing label text features and time series features, is the input matrix containing label text features and image features, for The adjacency matrix of for The adjacency matrix of .

[0120] 2.1.3 Graph Convolutional Neural Network Training

[0121] The fused multimodal data is sent to the graph convolutional neural network for training to obtain the training weights, and the network training weights are loaded into the edge detection device to obtain the road damage detection results in real time. The training process of the network is as follows: First, the data is input into the graph convolutional neural network, and the output of the graph convolutional neural network is calculated through forward propagation. The cross entropy loss function and the IoU loss function are used to calculate the detection category loss and the detection box loss of the output result respectively; then, the loss contribution of each parameter is calculated layer by layer through the back propagation algorithm, and the parameters of the graph convolutional neural network are updated using the stochastic gradient descent algorithm to gradually reduce the loss; then, the previous steps are repeated until the specified number of training rounds is reached or the conditions for stopping training are met; finally, the performance and generalization ability of the trained graph convolutional neural network are evaluated using the test set.

[0122] 2.2 Road damage information storage

[0123] MySQL is used as the database to store the road damage location information and damage category information that occurs during the road damage system detection process, and the InnoDB engine is used as its storage engine. The specific process of storing road damage information in the cloud service module is as follows:

[0124] (1) The edge detection device sends a communication request to the cloud service module by using a network connection. After receiving the command, the cloud service module transmits the road damage location information and damage category information to the cloud service module.

[0125] (2) The cloud service module receives the data transmitted by the edge detection device, parses and verifies it, and stores the received data in the MySQL database.

[0126] (3) Manage and analyze the data stored in the cloud service module database to obtain useful information to facilitate future road repair work.

[0127] 3. Edge detection module

[0128] The edge detection module detects road damage by using multimodal data collected by cameras and vibration sensors, and accurately locates the road damage location using the GPS locator. The detection results and location information are uploaded to the cloud service module. The specific workflow is as follows:

[0129] (1) The edge detection module fuses the real-time multimodal road data that has been preprocessed by the data acquisition module.

[0130] (2) Obtain the trained weights of the cloud service module model, and send the fused road data into the multimodal graph convolutional neural network with loaded weights for detection. The detection results are combined with the location information of the GPS locator to form the road damage detection results.

[0131] (3) Analyze the road conditions based on the test results and upload the test results to the cloud service module for storage.

[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A road damage detection method based on multimodal data, characterized in that: include: The acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label are encoded respectively to obtain image features, time series features, and text label features; Calculate the similarity between image features and text label features, as well as the similarity between temporal features and text label features, and fuse the generated conditions to generate labels for the adversarial network; Obtain road image data, road surface vibration time series data during vehicle driving, and geographic location data of road damage; A random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels are input into the conditional generative adversarial network for training to generate fused multimodal data suitable for graph convolutional neural network input; The multimodal data is predicted and combined with the geographical location data of road damage to obtain the road damage location and damage classification information.

2. The road damage detection method based on multimodal data according to claim 1, characterized in that: The method comprises encoding the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label respectively to obtain image features, time series features, and text label features. The method comprises: Based on the historical road image data, a visual encoder is used to encode and obtain image features; Based on the historical vibration time series data, a time series encoder is used to encode and obtain time series features; Based on the road damage label, a text encoder is used to encode and obtain text label features.

3. The road damage detection method based on multimodal data according to claim 1, characterized in that: The method of respectively calculating the similarity between the image feature and the text label feature, and the similarity between the time series feature and the text label feature, and fusing and generating multimodal data includes: The text label feature is regarded as a node of the entire graph structure. The similarity between the text label feature node and the image subgraph is calculated, and the similarity between the text label feature node and the vibration timing subgraph is calculated to connect the text label feature with the image subgraph and the vibration timing subgraph to obtain a multimodal graph, so as to fuse the labels of the generative conditional adversarial network.

4. The road damage detection method based on multimodal data according to claim 3, characterized in that: The process of fusing the labels of the generative adversarial network is as follows: In the formula, is the encoded label text feature, is the input matrix of time series features, is the input matrix of image features; The function is used to calculate the correlation between nodes; is the input matrix containing label text features and time series features, is the input matrix containing label text features and image features, for The adjacency matrix of for The adjacency matrix of .

5. The road damage detection method based on multimodal data according to claim 1, characterized in that: The method includes inputting a random noise vector, road image data, vibration time series data and a conditional generative adversarial network label into a conditional generative adversarial network for training to generate fused multimodal data suitable for graph convolutional neural network input. The method includes: A random noise vector, road image data and vibration time series data are input into the generator, and converted into multimodal data through deconvolution layers and fully connected layers; the multimodal data generated by the generator and the conditional generative adversarial network label are input into the discriminator, and feature extraction is performed through convolution layers and fully connected layers to obtain probability values; based on the probability values, the adversarial loss function is used to update the parameters of the generator and the discriminator.

6. The road damage detection method based on multimodal data according to claim 1, characterized in that: In the training phase, after obtaining multimodal data, it also includes: The multimodal data is input into the graph convolutional neural network for training, and the output results of the graph convolutional neural network are calculated through forward propagation; the detection category loss and detection box loss of the output results are calculated using the cross entropy loss function and the IoU loss function respectively to update the parameters of the graph convolutional neural network; the training weights are obtained, and the training weights are loaded into the edge detection device to obtain road damage detection results in real time.

7. A road damage detection system based on multimodal data, characterized in that: include: A cloud service module is configured to: respectively encode the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; Calculate the similarity between image features and text label features, as well as the similarity between temporal features and text label features, and fuse the generated conditions to generate labels for the adversarial network; A data acquisition module, which is configured to: acquire road image data, vibration time series data of the road surface during vehicle driving, and geographical location data of road damage; The cloud service module is further configured to: input a random noise vector, road image data, vibration time series data, and conditional generative adversarial network labels into the conditional generative adversarial network for training, and generate fused multimodal data adapted to the graph convolutional neural network input; The edge detection module is configured to: predict the multimodal data, combine the geographical location data of the road damage, and obtain the road damage location and damage classification information.

8. A road damage detection system based on multimodal data, characterized in that: include: A camera installed on the vehicle, a vibration sensor installed on the vehicle chassis, a GPS locator installed on the vehicle, and a cloud service module; The cloud service module encodes the acquired historical road image data, the historical vibration time series data of the road surface during vehicle driving, and the road damage label to obtain image features, time series features, and text label features; Calculate the similarity between image features and text label features, as well as the similarity between temporal features and text label features, and fuse the generated conditions to generate labels for the adversarial network; The camera is used to obtain road image data, the vibration sensor is used to obtain the vibration time series data of the road surface during vehicle driving, and the GPS locator is used to obtain the geographical location data of road damage; The cloud service module is also used to input a random noise vector, road image data, vibration time series data and conditional generative adversarial network labels into the conditional generative adversarial network for training, to generate fused multimodal data suitable for graph convolutional neural network input; to predict the multimodal data, and to obtain the road damage location and damage classification information by combining the geographical location data of the road damage.

9. The road damage detection system based on multimodal data according to claim 8, characterized in that: The system also includes an edge detection device, which sends a communication request to the cloud service module by using a network connection, and after the cloud service module receives the instruction, transmits the road damage location and damage classification information to the cloud service module.

10. The road damage detection system based on multimodal data according to claim 8, characterized in that: The database is used to store the detected road damage location and damage category information, and the InnoDB engine is used as its storage engine.

Images