Road crack detection method and system based on multimodal fusion

Abstract

The present invention relates to a road crack detection method and system based on multimodal fusion, comprising the steps of: collecting crack image data sets, forming training sets, verification sets and test sets after manual annotation; training the training set using the constructed improved YOLOv4 model, and then performing sparse training and channel pruning on the trained improved YOLOv4 model; inputting the test set into the improved YOLOv4 model and outputting the test results; collecting the original vibration signal through a nine-axis gyroscope, and extracting the features of the original vibration signal using HHT; judging whether it is a crack according to the change of the instantaneous frequency value obtained after HHT analysis; fusing the test results output by the improved YOLOv4 model and the crack results judged according to the original vibration signal and uploading them to a server; the server sends the crack position to the user terminal through GPS positioning, and alarms. The present invention integrates the target detection technology with the vibration signal processing technology, thereby realizing intelligent, real-time, efficient and convenient comprehensive detection of road cracks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and signal processing, and in particular to a road crack detection method and system based on multi-modal fusion. Background Art

[0002] In recent years, my country's road construction has made remarkable achievements. According to official statistics from the Ministry of Transport, my country's road maintenance mileage in 2020 is 5.144 million kilometers, accounting for 99% of the total road mileage. Up to now, road crack detection is mainly done by manual methods, which not only consumes a lot of manpower and material resources, but also takes a long time, has low detection efficiency and high cost, and is difficult to meet the development needs of my country's road transportation industry. With the development of deep learning technology, cracks can be well identified by target detection methods. However, affected by external environmental factors, target detection methods may have false detections or missed detections under dark lighting conditions. Summary of the invention

[0003] The purpose of the present invention is to integrate target detection technology with vibration signal processing technology, so as to realize intelligent, real-time, efficient and convenient comprehensive detection of road cracks, and provide a road crack detection method and system based on multimodal fusion.

[0004] In order to achieve the above-mentioned object of the invention, the embodiment of the present invention provides the following technical solutions:

[0005] The road crack detection method based on multimodal fusion includes the following steps:

[0006] Step S1: Collect crack image data sets, and after manual annotation, form training sets, validation sets, and test sets; use the constructed improved YOLOv4 model to train the training set, and then perform sparse training and channel pruning on the trained improved YOLOv4 model; input the test set into the improved YOLOv4 model, and output the test results;

[0007] Step S2: The original vibration signal is collected by the nine-axis gyroscope, and the feature of the original vibration signal is extracted by HHT; whether it is a crack is determined according to the change of the instantaneous frequency value obtained after HHT analysis;

[0008] Step S3: The test results output by the improved YOLOv4 model and the crack results determined based on the original vibration signal are integrated and uploaded to the server; the server sends the crack location to the user terminal through GPS positioning and issues an alarm.

[0009] In the above scheme, the YOLOv4 model is improved into a lightweight model, which achieves the purpose of reducing the number of parameters and improving the computing speed.

[0010] Furthermore, the step of collecting the crack image data set and forming a training set, a validation set, and a test set after manual annotation includes:

[0011] The collected crack image dataset includes transverse cracks, longitudinal cracks, mesh cracks and irregular cracks. The labeling software LabelImg is used to manually label the crack images, and a training set is formed after labeling. At the same time, an XML file containing the key points of road cracks is generated.

[0012] Create Main_trainval_txt.py and boc_label.py files in the root directory of the generated xml file to form the validation set and test set respectively.

[0013] Furthermore, the structure of the improved YOLOv4 model includes:

[0014] Input: Input the crack image to be detected;

[0015] Backbone: MobileNetV3, used to reduce the amount of computation and parameters;

[0016] Neck: SPP+PANet, SPP is used to enhance the network’s receptive field, and PANet is used to solve the multi-scale problem in target detection and improve the detection capability of objects of different sizes;

[0017] Head: Inherits the three detection heads of the YOLOv3 model, predicts image features, generates bounding boxes and predicts their categories.

[0018] Furthermore, the step of training the training set using the constructed improved YOLOv4 model includes:

[0019] The labeled crack image dataset is input into the backbone network MobileNetV3 for feature extraction. First, 1×1 convolution is used to increase the dimension. Then 3×3 depth-separable convolution is performed. The attention mechanism is used to adjust the weight of each channel. Finally, the h-swish activation function is used.

[0020] The extracted feature information is passed to the spatial pyramid pooling module to enhance the receptive field of the network. The maximum pooling of scales 5×5, 9×9, and 13×13 is performed respectively. After pooling, the outputs of the three pooling layers are spliced ​​through concat. After splicing, the feature dimension is reduced through a 1×1 convolution layer, and then the special diagnosis images from different kernel sizes are connected as output;

[0021] The maximum pooling results of three different scales are transmitted to PANet to achieve the effect of fusing shallow features with deep features. The output features of shallow features and deep features are made the same through 3×3 and 1×1 convolutions. Finally, the feature map of the target is predicted by the detection head to obtain the crack detection result.

[0022] Furthermore, the step of performing sparse training and channel pruning on the trained improved YOLOv4 model includes:

[0023] A scaling factor is introduced into each channel of the improved YOLOv4 model and trained together with the weights. During the training process, the channels with scaling factors that tend to 0 are cut off to obtain a compact network.

[0024] The loss function formula in the sparse training process of the improved YOLOv4 model is as follows:

[0025]

[0026] Among them, loss YOLO represents the balance of two losses, λ represents the hyperparameter, γ represents the scaling factor, and g(γ)=|γ| represents L1-regularization;

[0027] To improve the YOLOv4 model, a batch normalization layer BN is required, and then an activation function is used to increase the convergence speed of the YOLOv4 model. The calculation formula of the batch normalization layer BN is as follows:

[0028]

[0029] Among them, x represents the input of the batch normalization layer BN, y represents the output of the batch normalization layer BN, γ represents the scaling factor, β represents the translation parameter, μ b represents the mean of the input data, represents the variance of the input data, and ε represents a small integer added to avoid the variance being zero.

[0030] Furthermore, the step of collecting the original vibration signal by the nine-axis gyroscope, extracting the features of the original vibration signal by HHT, and judging whether it is a crack according to the change of the instantaneous frequency value obtained after HHT analysis includes:

[0031] Fix the nine-axis gyroscope on the vehicle-mounted equipment, collect the original vibration signal, and divide the original vibration signal into n equal parts; perform EMD decomposition on each segment of the original vibration signal to obtain IMF components and spectrum diagrams; add the IMF components with kurtosis coefficients greater than 3 to obtain IMFre, and obtain the IMFre marginal spectrum and instantaneous frequency fEmax through Hilbert transform;

[0032] Compare the road scene video with the video when the nine-axis gyroscope captures the crack, classify the flatness corresponding to the instantaneous frequency fEmax, and verify whether it meets the inspection conditions of the independent sample T; according to the waveform change of the instantaneous frequency fEmax and combined with the video comparison, determine whether the original vibration signal detected by the nine-axis gyroscope is a crack.

[0033] The step of performing EMD decomposition on each segment of the original vibration signal to obtain the IMF component and the spectrum diagram respectively comprises:

[0034] Find all extreme points of each segment of the original vibration signal x(t), fit all the maximum points to the upper envelope through the cubic spline function; fit all the minimum points to the lower envelope through the cubic spline function, and calculate the mean envelope m(t);

[0035] The intermediate signal c(t) is obtained by subtracting the mean envelope m(t) from the original vibration signal x(t); the intermediate signal c(t) is assumed to be the first high-frequency component decomposed and is recorded as IMF1; the mean envelope m(t) is taken as the original vibration signal to continue to decompose the next high-frequency component until the residual component r(t) is a monotonic function or a constant, at which time the decomposition is stopped; thereby obtaining multiple IMF components.

[0036] Furthermore, the formulas for EMD decomposition and Hilbert transform are as follows:

[0037] EMD decomposition:

[0038] Among them, x(t) is the original vibration signal of the tth segment, c i (t) is the nth-order IMF component, r n (t) is the nth-order residual component;

[0039] Hilbert Transform:

[0040] where H(x(t)) is x(t) and convolution.

[0041] The road crack detection system based on multimodal fusion includes:

[0042] The target detection module is used to input the collected crack image data set into the improved YOLOv4 model for detection after manual annotation, so as to obtain the crack detection results;

[0043] The vibration signal detection module is used to extract the features of the original vibration signal collected by the nine-axis gyroscope using HHT, and obtain the instantaneous frequency value, and obtain the crack detection result according to the instantaneous frequency value;

[0044] The server is used to fuse the crack detection results of the target detection module and the vibration signal detection module, send the crack location to the user terminal through GPS positioning, and issue an alarm.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] (1) The present invention adopts a lightweight improved YOLOv4 model to reduce the number of parameters and improve the computing speed. By pruning the improved YOLOv4 model, the number of parameters is reduced, and the number of inference parameters is reduced. Finally, the improved YOLOv4 model is deployed on mobile terminals and edge devices to form a road crack detection system, so as to perform real-time detection of road conditions.

[0047] (2) The present invention uses a nine-axis gyroscope to collect vibration signals and utilizes HHT decomposition technology to perform feature extraction research on the vibration signals.

[0048] (3) The present invention integrates the lightweight improved YOLOv4 model with the nine-axis gyroscope, captures the radio frequency in real time and collects the vibration signal and uploads it to the server, locates the crack position through GPS, and sends alarm information to the user terminal in real time.

[0049] (4) The device of the present invention has low cost, small size, and is easy to carry. The lightweight improved YOLOv4 model combined with the HHT decomposition technology acquires and processes data in a more efficient and quick manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 This is a flow chart of the crack detection method of the present invention;

[0052] Figure 2 This is a structural diagram of an improved YOLOv4 model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0054] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0055] The present invention is achieved through the following technical solutions: Figure 1 As shown, the road crack detection method based on multimodal fusion includes the following steps:

[0056] Step S1: Collect crack image data sets, and after manual annotation, form training sets, validation sets, and test sets; use the constructed improved YOLOv4 model to train the training set, and then perform sparse training and channel pruning on the trained improved YOLOv4 model; input the test set into the improved YOLOv4 model and output the test results.

[0057] First, a road crack image dataset is collected. The crack image dataset comes from crack images taken manually and crack images in papers, including horizontal cracks, longitudinal cracks, mesh cracks and irregular cracks. Then the labeling software LabelImg is used to manually label the crack images, and a training set is formed after labeling. At the same time, an XML file containing the key points of road cracks is generated. The XML file records: 1) folder name; 2) image name; 3) file path; 4) image size and depth; 5) target category name and border coordinates, in preparation for the training dataset.

[0058] Create Main_trainval_txt.py and boc_label.py files in the root directory of the generated xml file to form the validation set and test set respectively.

[0059] Improved YOLOv4 model: Replace the backbone network CSPDarknet53 of the ordinary YOLOv4 model with the lightweight MobileNetV3. The structure of the improved YOLOv4 model includes:

[0060] (1) Input: Input the crack image to be detected;

[0061] (2) Backbone: MobileNetV3, used to reduce the amount of computation and parameters;

[0062] (3) Neck: SPP+PANet, SPP is used to enhance the network’s receptive field, and PANet is used to solve the multi-scale problem in target detection and improve the detection capability of objects of different sizes;

[0063] (4) Head: Inherits the three detection heads of the YOLOv3 model, predicts image features, generates bounding boxes and predicts their categories.

[0064] See also Figure 2 When the constructed improved YOLOv4 model is used to train the training set, the labeled crack image dataset is input into the backbone network MobileNetV3 for feature extraction. First, 1×1 convolution is used for dimension increase; then 3×3 depth-separable convolution is performed; the attention mechanism is used to adjust the weight of each channel; finally, the h-swish activation function is used to reduce the amount of calculation and improve performance.

[0065] The extracted feature information is passed into the spatial pyramid pooling module to enhance the receptive field of the network, and maximum pooling of scales 5×5, 9×9, and 13×13 is performed respectively. After pooling, the outputs of the three pooling layers are concatenated through concat, and then the feature dimension is reduced through a 1×1 convolutional layer. Then, the special diagnosis images from different kernel sizes are connected as output.

[0066] The maximum pooling results of three different scales are transmitted to PANet to achieve the effect of fusing shallow features with deep features. The output features of shallow features and deep features are made the same through 3×3 and 1×1 convolutions. Finally, the feature map of the target is predicted by the detection head to obtain the crack detection result.

[0067] Figure 2 In the figure, CBh: consists of Conv (convolutional layer), BN (batch normalization layer) and h-swish (activation function); CBL: consists of Conv (convolutional layer), BN (batch normalization layer) and Leaky-relu (activation function); Bneck: is the bottleneck layer, a residual structure.

[0068] After replacing the backbone network with MobileNetV3, the thrust speed of the improved YOLOv4 model is improved. In order to further reduce the parameters of the improved YOLOv4 model and improve its reasoning speed, the improved YOLOv4 model is sparsely trained and channel pruned while ensuring the detection accuracy to improve the practicality of the improved YOLOv4 model. Finally, the improved YOLOv4 model is tested and the test results are output.

[0069] The process of improving YOLOv4 model sparse training and channel pruning includes:

[0070] Introduce scaling factors into the channels of the improved YOLOv4 model and train them together with the weights. During the training process, cut off the channels of the scaling factors whose regularization tends to 0, thus obtaining a compact network.

[0071] The loss function formula in the sparse training process of the improved YOLOv4 model is as follows:

[0072]

[0073] Among them, loss YOLO represents the balance of two losses, λ represents the hyperparameter, γ represents the scaling factor, and g(γ)=|γ| represents L1-regularization;

[0074] Since the improvement of the YOLOv4 model requires a batch normalization layer BN (Batch Normalization), and then an activation function to improve the convergence speed of the YOLOv4 model, the calculation formula of the batch normalization layer BN is as follows:

[0075]

[0076] Among them, x represents the input of the batch normalization layer BN, y represents the output of the batch normalization layer BN, γ represents the scaling factor, and the γ of the BN layer can be directly used as the scaling factor for sparsification, and β represents the translation parameter; μ b represents the mean of the input data, represents the variance of the input data, and ε represents a small integer added to avoid the variance being 0, such as 0.001.

[0077] The model size (Size), parameter (Parameter), inference time (Inference Speed), precision (P), recall (R), average precision (mAP), and F1 score of the improved YOLOv4 model before pruning (Orginal) and after pruning (Pruned) are shown in Table 1:

[0078] Table 1

[0079]

[0080] In Table 1, the mean average precision (mAP) is used as the evaluation index of detection accuracy; the F1 score is also called the balanced F score, which is defined as the harmonic mean of precision and recall. As can be seen from Table 1, the model size after pruning is reduced by 87.13%, the parameters are reduced by 88.39%, the inference time is increased by 80.95%, and the accuracy after pruning reaches 98%.

[0081] Step S2: Collect the original vibration signal through the nine-axis gyroscope, and use HHT to extract the features of the original vibration signal; determine whether it is a crack based on the change of the instantaneous frequency value obtained after HHT analysis.

[0082] The nine-axis gyroscope collects the original vibration signal, and the HHT decomposition is used to perform feature extraction on the original vibration signal. First, the nine-axis gyroscope is fixed on the vehicle-mounted equipment to collect the original vibration signal, and the original vibration signal is divided into n equal parts; EMD decomposition is performed on each segment of the original vibration signal to obtain the IMF component and spectrum diagram respectively.

[0083] IMFre is obtained by adding the IMF components with kurtosis coefficient greater than 3. The kurtosis coefficient indicates the probability of occurrence of large-amplitude pulses caused by faults. The characteristic of the kurtosis coefficient is that the vibration signal probability density of bearings that generally run smoothly and have no obvious periodic interference is close to the normal distribution, and the kurtosis coefficient of the normal distribution is 3.

[0084] Then, the IMFre marginal spectrum and instantaneous frequency fEmax are obtained through Hilbert transform. Finally, the road scene video is extracted from the road video collected by the camera fixed on the car, and the video of the crack collected by the nine-axis gyroscope is compared with the two videos.

[0085] The flatness corresponding to the instantaneous frequency fEmax is classified to verify whether it meets the test conditions of independent sample T. Independent sample T is used to test the difference in data obtained from two groups of unrelated sample subjects. The conditions that need to be met are: 1) Random sampling, all observations should be randomly drawn from the target population; 2) Normal distribution, the population from which each sample comes must satisfy the normal distribution; 3) Homogeneity of variance, when comparing means, the variances of the two populations must be equal.

[0086] According to the waveform change of the instantaneous frequency fEmax, an impact is obtained when passing through the crack, and the instantaneous frequency fEmax increases. Combined with the crack video taken during video comparison, it can be determined whether the vibration signal detected by the nine-axis gyroscope is a crack.

[0087] In detail, the HHT decomposition process includes two parts: empirical mode decomposition (EMD decomposition) and Hilbert transform (HT), where:

[0088] EMD decomposition: find all extreme points of each segment of the original vibration signal x(t), fit all the maximum points to the upper envelope through the cubic spline function; fit all the minimum points to the lower envelope through the cubic spline function, and calculate the mean envelope m(t); use the segment of the original vibration signal x(t) to subtract the mean envelope m(t) to obtain the intermediate signal c(t); let the intermediate signal c(t) be the first high-frequency component decomposed, and record it as IMF1; treat the mean envelope m(t) as the original vibration signal and continue to decompose it to obtain the next high-frequency component until the residual component r(t) is a monotonic function or a constant, then stop decomposing; thereby obtaining multiple IMF components.

[0089] Hilbert transform: Add the IMF components with kurtosis coefficient greater than 3 to obtain IMFre, and then obtain the IMFre marginal spectrum and instantaneous frequency fEmax through Hilbert transform.

[0090] The formulas for EMD decomposition and Hilbert transform are as follows:

[0091] EMD decomposition:

[0092] Among them, x(t) is the original vibration signal of the tth segment, c i (t) is the nth-order IMF component, r n (t) is the nth-order residual component;

[0093] Hilbert Transform:

[0094] where H(x(t)) is x(t) and convolution.

[0095] Step S3: The test results output by the improved YOLOv4 model and the crack results determined based on the original vibration signal are integrated and uploaded to the server; the server sends the crack location to the user terminal through GPS positioning and issues an alarm.

[0096] The weights of the improved YOLOv4 model and the nine-axis gyroscope during detection are divided according to the light intensity. When the light is strong, the improved YOLOv4 model is used to detect road cracks. When the light is dark, the nine-axis gyroscope is used to detect road cracks, or another detection method is used as a reference.

[0097] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A road crack detection method based on multimodal fusion, characterized by: The following steps are involved: Step S1: Collect crack image data sets, and after manual annotation, form training sets, validation sets, and test sets; use the constructed improved YOLOv4 model to train the training set, and then perform sparse training and channel pruning on the trained improved YOLOv4 model; input the test set into the improved YOLOv4 model, and output the test results; The structure of the improved YOLOv4 model includes: (1) Input: Input the crack image to be detected; (2) Backbone: MobileNetV3, used to reduce the amount of computation and parameters; (3) Neck: SPP+PANet, SPP is used to enhance the network’s receptive field, and PANet is used to solve the multi-scale problem in target detection and improve the detection capability of objects of different sizes; Head: inherits the three detection heads of the YOLOv3 model, predicts image features, generates bounding boxes and predicts their categories; Step S2: The original vibration signal is collected by the nine-axis gyroscope, and the feature of the original vibration signal is extracted by HHT; whether it is a crack is determined according to the change of the instantaneous frequency value obtained after HHT analysis; The steps of collecting the original vibration signal by the nine-axis gyroscope, extracting the features of the original vibration signal by using HHT, and judging whether it is a crack according to the change of the instantaneous frequency value obtained after HHT analysis include: Fix the nine-axis gyroscope on the vehicle-mounted equipment, collect the original vibration signal, and divide the original vibration signal into n equal parts; perform EMD decomposition on each segment of the original vibration signal to obtain IMF components and spectrum diagrams; add the IMF components with kurtosis coefficients greater than 3 to obtain IMFre, and obtain the IMFre marginal spectrum and instantaneous frequency fEmax through Hilbert transform; Compare the road scene video with the video of the cracks collected by the nine-axis gyroscope, classify the flatness corresponding to the instantaneous frequency fEmax, and verify whether it meets the inspection conditions of the independent sample T; according to the waveform change of the instantaneous frequency fEmax, combined with the video comparison, determine whether the original vibration signal detected by the nine-axis gyroscope is a crack; Step S3: The test results output by the improved YOLOv4 model and the crack results determined based on the original vibration signal are integrated and uploaded to the server; the server sends the crack location to the user terminal through GPS positioning and issues an alarm.

2. The road crack detection method based on multimodal fusion according to claim 1 is characterized in that: The step of collecting the crack image data set and manually annotating it to form a training set, a verification set, and a test set includes: The collected crack image dataset includes transverse cracks, longitudinal cracks, mesh cracks and irregular cracks. The labeling software LabelImg is used to manually label the crack images, and a training set is formed after labeling. At the same time, an XML file containing the key points of road cracks is generated. Create Main_trainval_txt.py and boc_label.py files in the root directory of the generated xml file to form the validation set and test set respectively.

3. The road crack detection method based on multimodal fusion according to claim 1, characterized in that: The step of training the training set using the constructed improved YOLOv4 model includes: The labeled crack image dataset is input into the backbone network MobileNetV3 for feature extraction. First, 1×1 convolution is used to increase the dimension. Then 3×3 depth-separable convolution is performed. The attention mechanism is used to adjust the weight of each channel. Finally, the h-swish activation function is used. The extracted feature information is passed to the spatial pyramid pooling module to enhance the receptive field of the network. The maximum pooling of scales 5×5, 9×9, and 13×13 is performed respectively. After pooling, the outputs of the three pooling layers are spliced ​​through concat. After splicing, the feature dimension is reduced through a 1×1 convolution layer, and then the special diagnosis images from different kernel sizes are connected as output; The maximum pooling results of three different scales are transmitted to PANet to achieve the effect of fusing shallow features with deep features. The output features of shallow features and deep features are made the same through 3×3 and 1×1 convolutions. Finally, the feature map of the target is predicted by the detection head to obtain the crack detection result.

4. The road crack detection method based on multimodal fusion according to claim 1, characterized in that: The step of performing sparse training and channel pruning on the trained improved YOLOv4 model comprises: A scaling factor is introduced into each channel of the improved YOLOv4 model and trained together with the weights. During the training process, the channels with scaling factors that tend to 0 are cut off to obtain a compact network. The loss function formula in the sparse training process of the improved YOLOv4 model is as follows: Among them, loss YOLO represents the balance of two losses, λ represents the hyperparameter, γ represents the scaling factor, and g(γ)=|γ| represents L1-regularization; To improve the YOLOv4 model, a batch normalization layer BN is required, and then an activation function is used to increase the convergence speed of the YOLOv4 model. The calculation formula of the batch normalization layer BN is as follows: Among them, x represents the input of the batch normalization layer BN, y represents the output of the batch normalization layer BN, γ represents the scaling factor, β represents the translation parameter, and μ b represents the mean of the input data, represents the variance of the input data, and ε represents a small integer added to avoid the variance being zero.

5. The road crack detection method based on multimodal fusion according to claim 1, characterized in that: The step of performing EMD decomposition on each segment of the original vibration signal to obtain the IMF component and the spectrum diagram respectively comprises: Find all extreme points of each segment of the original vibration signal x(t), fit all the maximum points to the upper envelope through the cubic spline function; fit all the minimum points to the lower envelope through the cubic spline function, and calculate the mean envelope m(t); The intermediate signal c(t) is obtained by subtracting the mean envelope m(t) from the original vibration signal x(t); the intermediate signal c(t) is assumed to be the first high-frequency component decomposed and is recorded as IMF1; the mean envelope m(t) is taken as the original vibration signal to continue to decompose the next high-frequency component until the residual component r(t) is a monotonic function or a constant, at which time the decomposition is stopped; thereby obtaining multiple IMF components.

6. The road crack detection method based on multimodal fusion according to claim 1 or 5, characterized in that: The formulas for EMD decomposition and Hilbert transform are as follows: EMD decomposition: Among them, x(t) is the original vibration signal of the tth segment, c i (t) is the nth-order IMF component, r n (t) is the nth-order residual component; Hilbert Transform: where H(x(t)) is x(t) and convolution.

7. A road crack detection system based on multimodal fusion, used to implement the method according to any one of claims 1 to 6, characterized in that: include: The target detection module is used to input the collected crack image data set into the improved YOLOv4 model for detection after manual annotation, so as to obtain the crack detection results; The vibration signal detection module is used to extract the features of the original vibration signal collected by the nine-axis gyroscope using HHT, and obtain the instantaneous frequency value, and obtain the crack detection result according to the instantaneous frequency value; The server is used to fuse the crack detection results of the target detection module and the vibration signal detection module, send the crack location to the user terminal through GPS positioning, and issue an alarm.

Images