Road disease detection method and system based on space-frequency dual-domain fusion

Abstract

The invention provides a road disease detection method and system based on space-frequency dual-domain fusion. The method belongs to the field of road disease detection and comprises the following steps: acquiring a road disease image; extracting frequency domain features and spatial domain features of different scales from the road disease image; performing feature fusion on the frequency domain features and the spatial domain features of the corresponding scales by adopting a dynamic parameter weighting method to obtain a multi-scale feature map, and inputting the fused multi-scale feature map into a feature pyramid network for feature enhancement; and inputting the enhanced multi-scale feature map into a detection head for detection to obtain a detection result, and outputting the detection result. By extracting, fusing and enhancing the spatial domain features and the frequency domain features of the images, the accuracy of road disease detection and prediction is improved, and efficient, accurate and stable road disease recognition is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of road disease detection, in particular to a road disease detection method and system based on space-frequency dual-domain fusion. BACKGROUND

[0002] Early detection and timely treatment of road diseases are of great significance for preventing road damage and improving road service life. Traditional road disease detection methods rely on manual inspection, which is not only inefficient and prone to omissions, but also poses certain risks to traffic safety.

[0003] In the prior art, some methods use traditional image processing methods to simply convert images, and then use machine learning methods such as decision trees and clustering algorithms for detection. Such methods are not only inefficient, but also have high requirements for the shooting angle and resolution of the picture, and are often helpless for complex environments or not obvious road cracks. However, with the rapid development of deep learning, various algorithms based on convolutional neural networks have greatly improved the detection accuracy for such problems, such as the common YOLO detection algorithm series and the R-CNN model. However, the detection algorithm based on convolutional neural network is mainly for the detection of common targets in life, while the shape of road disease is often a long and narrow crack with a large difference in horizontal and vertical scales. Therefore, the detection accuracy of road disease using such algorithms is not high. SUMMARY

[0004] The purpose of the present application is to provide a road disease detection method and system based on space-frequency dual-domain fusion, which can improve the detection accuracy of road diseases.

[0005] The present application is implemented as follows:

[0006] In a first aspect, the present application provides a road disease detection method based on space-frequency dual-domain fusion, comprising the following steps:

[0007] S101: acquiring a road disease image;

[0008] S102: extracting frequency domain features and spatial domain features of different scales from the road disease image;

[0009] S103: using a dynamic parameter weighting method to fuse the frequency domain features and spatial domain features corresponding to the scales to obtain a multi-scale feature map, and the formula of the dynamic parameter weighting method is represented as:

[0010] Output=Add(x1,x2);

[0011] x1=X×v rgb ×T rgb ;

[0012] x2=X feq ×v feq ×T feq ;

[0013] wherein, Output represents the finally generated multi-scale feature map, Add(x1,x2) represents the fusion of two different features, X and X feq are the spatial domain features and frequency domain features corresponding to the scale, v rgb and v feq are two different learnable dynamic parameters, v rgb is used to adjust the weight and transformation of the spatial domain feature, v feq is used to adjust the weight and transformation of the frequency domain feature, T feq and T rgb are the frequency domain and spatial domain features corresponding to the channel after splicing and convolution;

[0014] S104: input the fused multi-scale feature map to the feature pyramid network for feature enhancement;

[0015] S105: input the enhanced multi-scale feature map to the detection head for detection to obtain a detection result and output.

[0016] Based on the first aspect, the specific steps of extracting different scale spatial domain features from the road disease image include:

[0017] input the road disease image to an image spatial domain feature extraction model, the image spatial domain feature extraction model including a STEM layer, a GR-BneckV2 and a dynamic snake convolution DSConv; wherein the STEM layer performs down-sampling and feature enrichment on the image feature, which is expressed by the formula:

[0018]

[0019] wherein, F r 0 represents the image feature processed by the STEM layer, represents the operation of the STEM layer, and I represents the input image feature;

[0020] After the down-sampled feature is enhanced by the GR-BneckV2, the image feature is divided into two groups and input to the multi-layer GR-BneckV2 structure and the single-layer dynamic snake convolution DSConv, respectively, the former can ensure the model to obtain more detailed information, and the latter can capture crack-like features while transmitting low-dimensional feature gradient information;

[0021] first layer processing: T r i =GRBneckv2(F r i-1 );

[0022] Multi-layer GR-BneckV2 structure processing: M r i = GRBneckv2 m (T r i );

[0023] Single-layer DSConv structure processing: N r i = DSConv(T r i );

[0024] Feature splicing and final processing: F r i = GRBneckv2(Concat[M r i , N r i ]);

[0025] wherein the first layer processing is down-sampling feature F r i-1 is enhanced by the GR-BneckV2 module to obtain the enhanced feature T r i ; GRBneckv2 is a specific convolutional layer for extracting and enhancing features; the multi-layer GR-BneckV2 structure processing is to enhance the feature T r i is input into the m-layer GRBneckv2 m structure, and after multiple GRBneckv2 processing, the further enhanced feature M r i is obtained; the single-layer DSConv structure is also input into the dynamic snake convolution DSConv module for the enhanced feature T r i ; DSConv can capture the dynamic changes of the features to generate new features N r i ; the feature splicing and final processing is to splice the multi-layer GRBneckv2 structure processed feature M r i and the DSConv processed feature N r i , and the spliced feature is integrated again by the GRBneckv2 module to obtain the final spatial feature F r i ;

[0026] Wherein, the GRBneckv2 follows the original network structure design when the step is 1; when the step is 2, the dynamic snake convolution is added as the enhancement way of convolution; represented by the formula as:

[0027] T = Ghost_Module(In)·DFC(In);

[0028]

[0029] Wherein, In and out are the input and output of the GRBneckv2 structure respectively; when the step s = 1, the GRBneckv2 structure depends on the Ghost Module to generate additional feature maps; T represents the difference or enhancement part between the feature maps obtained by processing the input In through the Ghost Module and the feature maps obtained by processing In through the dimension reduction convolution layer DFC; the final output Out is the combination of In and T, which represents that on the basis of keeping the original input features, the enhanced features obtained by processing through the Ghost Module and DFC are added;

[0030] When the step s = 2, the GRBneckv2 structure introduces the dynamic snake convolution DSCConv as an additional enhancement way on the basis of the Ghost Module and DFC; the output Out is the combination of In and T processed by DSCConv; which represents that when the step is 2, the DSCConv is introduced to enhance the feature extraction ability to adapt to larger spatial downsampling.

[0031] Further, the specific steps of extracting different scale frequency domain features from the road disease image include:

[0032] The road disease image is input into the image frequency domain feature extraction model, and the image frequency domain feature extraction model includes PatchFFT, FS-SSBlock and Patch merging;

[0033] The road disease image is input into the image frequency domain feature extraction model, and the image frequency domain feature extraction model includes PatchFFT, FS-SSBlock and Patch merging; The road disease image is input into the image frequency domain feature extraction model, and the image frequency domain feature extraction model includes PatchFFT, FS-SSBlock and Patch merging; After being processed by the PatchFFT, the road disease image respectively obtains the features represented by the formula as:

[0034]

[0035] Wherein, Represent the feature representation of the whole image and the block Fourier transform respectively; PatchFFT(In) means using the PatchFFT method to process the input image In and obtain the corresponding Fourier transform result;

[0036] The inference result obtained by FS-SSBlock processing is expressed as follows:

[0037] Output=Linear(Input)+Linear(SS2D(DWConv(Linear(Input))));

[0038] Among them, Linear(Input) means that the input data is first linearly transformed to prepare for subsequent processing, DWConv() means that depthwise separable convolution is then applied to reduce the amount of calculation and the number of parameters, SS2D() means that space-spectrum two-dimensional conversion is then applied to extract frequency domain features, and Linear() means that the SS2D converted data is finally linearly transformed again to further process the frequency domain features;

[0039] After completing the inverse fast Fourier transform IFFT, the scale is aligned using 3×3 convolution to obtain the frequency domain features {F f i |i=0,1,2}, where F f i |i represents the i-th frequency domain eigenvector. i is an index used to distinguish different frequency domain eigenvectors, and its value range is 0 to 2.

[0040] Furthermore, the obtaining of road damage images includes:

[0041] According to the computing power conditions, the requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model, adjust the image resolution to ensure that the input image meets the input requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model and adapts to different computing power conditions.

[0042] Furthermore, the steps of inputting the fused multi-scale feature map into the feature pyramid network for feature enhancement include:

[0043] The FS-SSM module is used to obtain inference information at different depths to achieve attention and feature enhancement of frequency and spatial strip images.

[0044] Furthermore, the specific steps of inputting the enhanced multi-scale feature map into the detection head for detection to obtain the detection result and output include:

[0045] The detection results are filtered out through non-maximum suppression to filter out redundant features, obtain the detection target information with the highest confidence, and output the results.

[0046] In a second aspect, the present application provides a road damage detection system based on space-frequency dual-domain fusion, comprising:

[0047] Image acquisition module: used to obtain road damage images;

[0048] Feature extraction module: used to extract frequency domain features and spatial domain features of different scales from road damage images;

[0049] Feature fusion module: The frequency domain features and spatial domain features of the corresponding scales are fused using the dynamic parameter weighting method to obtain a multi-scale feature map. The formula used in the dynamic parameter weighting method is expressed as follows:

[0050] Output=Add(x1,x2);

[0051] x1=X×v rgb ×T rgb ;

[0052] x2=X feq ×v feq ×T feq ;

[0053] Among them, Output represents the final generated multi-scale feature map, Add(x1,x2) represents the fusion of two different features, X represents the spatial domain feature of the corresponding scale, and v rgb and T rgb are two different learnable dynamic parameters that are related to color information and are used to adjust the weight and transformation of spatial features; X feq Represents the frequency domain characteristics of the corresponding scale, v feq and T feq are two other learnable dynamic parameters that are related to frequency information and are used to adjust the weight and transformation of frequency domain features;

[0054] Feature enhancement module: used to input the fused multi-scale feature map into the feature pyramid network for feature enhancement;

[0055] Detection and output module: used to input the enhanced multi-scale feature map into the detection head for detection to obtain the detection results and output them.

[0056] In a third aspect, the present application provides an electronic device, comprising:

[0057] a memory for storing one or more programs;

[0058] processor;

[0059] When the one or more programs are executed by the processor, the above method is implemented.

[0060] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method described above.

[0061] Compared with the prior art, the present application has at least the following advantages or beneficial effects:

[0062] The present application provides a road disease detection method and system based on space-frequency dual-domain fusion. Different scale frequency domain features and spatial domain features are extracted from road disease images. The corresponding scale frequency domain features and spatial domain features are fused by using a dynamic parameter weighting method to obtain a multi-scale feature map. The fused multi-scale feature map is input into a feature pyramid network for feature enhancement. The spatial domain features and frequency domain features of the image are extracted, fused and enhanced in sequence, which improves the accuracy of road disease detection and prediction, realizes efficient, accurate and stable road disease identification, provides more reliable automated detection support for traffic management departments, and helps traffic management departments to better detect and supervise road quality, maintain daily maintenance, and improve the service life of the road. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0064] Figure 1 The flowchart of an embodiment of the road disease detection method based on space-frequency dual-domain fusion of the present application;

[0065] Figure 2 The structural diagram of the image spatial domain feature extraction model in the road disease detection method based on space-frequency dual-domain fusion of the present application;

[0066] Figure 3 The structural diagram of the Ghost Module component of the image spatial domain feature extraction model in the road disease detection method based on space-frequency dual-domain fusion of the present application;

[0067] Figure 4 The structural diagram of the GRBneckV2 component of the image spatial domain feature extraction model in the road disease detection method based on space-frequency dual-domain fusion of the present application when the step length is 1;

[0068] Figure 5 The structural diagram of the GRBneckV2 component of the image spatial domain feature extraction model in the road disease detection method based on space-frequency dual-domain fusion of the present application when the step length is 2;

[0069] Figure 6 FIG. 1 is a structural schematic diagram of an image frequency domain feature extraction model in a road disease detection method based on space-frequency dual domain fusion according to an embodiment of the present application.

[0070] Figure 7 FIG. 2 is a schematic diagram of a method for feature fusion by using dynamic parameter weighting of frequency domain features and space domain features in a road disease detection method based on space-frequency dual domain fusion according to an embodiment of the present application.

[0071] Figure 8 FIG. 3 is a structural schematic diagram of a road disease detection system based on space-frequency dual domain fusion according to an embodiment of the present application.

[0072] Figure 9 FIG. 4 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0073] Icon:

[0074] 1, image acquisition module; 2, feature extraction module; 3, feature fusion module; 4, feature enhancement module; 5, detection and output module; 6, processor; 7, memory; 8, communication interface. DETAILED DESCRIPTION

[0075] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0076] Some embodiments of the present application will be described in detail below with reference to the drawings. In the case of no conflict, each of the following embodiments and each feature in the embodiments can be combined with each other.

[0077] EMBODIMENTS

[0078] The embodiments of the present application provide a road disease detection method and system based on space-frequency dual domain fusion, which can improve the detection accuracy of road diseases.

[0079] Please refer to Figure 1 The present application provides a road disease detection method based on space-frequency dual domain fusion, comprising the following steps:

[0080] S101: acquiring road disease images;

[0081] Specifically, the data acquisition device such as a high-definition camera is used to shoot the road to obtain image information containing potential diseases.

[0082] S102: Extracting different scale frequency domain features and space domain features from the road disease image;

[0083] Specifically, the embodiment sets an image space domain feature extraction model and an image frequency domain feature extraction model to extract frequency domain features and space domain features respectively. The space domain feature extraction model is described in detail in the image space domain feature extraction model Figure 2 , and the space domain feature extraction method includes inputting the road disease image into the image space domain feature extraction model, the image domain feature extraction model including a STEM layer, a GRBneckV2 and a dynamic snake convolution DSConv; the GRBneckV2STEM layer performs down-sampling and feature enrichment on the image features, which is expressed by the formula:

[0084]

[0085] wherein, F r 0 represents the image features processed by the STEM layer, represents the operation of the STEM layer, and I represents the input image features;

[0086] After the down-sampled features are enhanced by the GR-BneckV2, the image features are divided into two groups and input into a multi-layer GR-BneckV2 structure and a single-layer dynamic snake convolution DSConv, respectively. The former can ensure that the model obtains more detailed information, and the latter can capture crack-like features while transmitting low-dimensional feature gradient information;

[0087] First layer processing: T r i = GRBneckv2(F r i-1 );

[0088] Multi-layer GR-BneckV2 structure processing: M r i = GRBneckv2 m (T r i );

[0089] Single-layer DSConv structure processing: N r i = DSConv(T r i );

[0090] Feature splicing and final processing: F r i = GRBneckv2(Concat[M r i , N r i ]);

[0091] The first layer processing is down-sampling feature F r i-1 The enhanced feature T is obtained through the enhancement processing of the GRBneckv2 module r i GRBneckv2 is a specific convolutional layer used for feature extraction and enhancement; the second layer processing is the enhanced feature T r i is input to the multi-layer GRBneckv2 m structure, and the further enhanced feature M is obtained through multiple GRBneckv2 processing r i The third layer is the enhanced feature T r i is also input to the dynamic snake convolution DSConv module; DSConv can capture the dynamic changes of the feature to generate a new feature N r i The feature splicing and final processing are to splice the features M and N processed by the multi-layer GRBneckv2 structure and the DSConv r i respectively r i After the splicing, the spliced feature is processed by the GRBneckv2 module again to obtain the final spatial feature F r i ;

[0092] Wherein, GRBneckv2 follows the original when the step is 1. It can be seen that, different from other spatial feature extraction methods, the spatial feature extraction model adopted by the present application adopts a double-channel feature extraction method for feature extraction. The input down-sampling feature is enhanced by GRBneckV2 and input to the multi-layer GRBneckV2 structure and dynamic snake convolution DSConv. Network structure design; when the step is 2, dynamic snake convolution is added as an enhanced way of convolution; please refer to Figure 4 , Figure 4 is the structure diagram of GRBneckv2 when the step is 1, and dynamic snake convolution is added as an enhanced way of convolution; please refer to Figure 5 , Figure 5 is the structure diagram of GRBneckv2 when the step is 2, which is expressed by the formula as follows:

[0093] T = Ghost_Module(In) · DFC(In)

[0094]

[0095] wherein, In and out are the input and output of the GRBneckv2 structure respectively; when the step s = 1, the GRBneckv2 structure relies on the Ghost Module to generate additional feature maps; T represents the difference or enhancement part between the feature maps obtained by processing the input In through the Ghost Module and the feature maps obtained by processing In through the dimension reduction convolution layer DFC; the final output Out is the combination of the input In and T, which represents that on the basis of maintaining the original input features, the enhanced features obtained by processing through the Ghost Module and the DFC are added;

[0096] When the step s = 2, the GRBneckv2 structure introduces the dynamic snake convolution DSCConv as an additional enhancement method on the basis of the Ghost Module and the DFC; the output Out is the combination of the input In and T processed through the DSCConv; which represents that when the step is 2, the ability of DSCConv to enhance feature extraction is introduced to adapt to larger spatial downsampling.

[0097] It should be noted that the Ghost_Module structure and the DFC structure follow the original Ghostnet model design. Figure 3 The design is shown in FIG. 1, Figure 3 which is a structural schematic diagram of the image spatial domain feature extraction model component Ghost Module; wherein, CBR represents the collective term of 2D convolution, normalization and Relu activation function.

[0098] Through this improvement, the image spatial domain feature extraction model can better capture the crack-like structural features of the damage structure, and effectively reduce the complexity of the backbone model compared to the method of using dynamic snake convolution structure entirely. The frequency domain feature extraction model is described in detail in Figure 6 , the frequency domain feature extraction method includes:

[0099] The road disease image is input into the image frequency domain feature extraction model; the image frequency domain feature extraction model includes PatchFFT, FS-SSBlock and Patch merging;

[0100] After the road disease image is processed by PatchFFT, the features are obtained respectively. After being processed by the FS-SSBlock, the inference result is obtained through 3x3 convolution with a step of 2, and then the Patch merging method is used to change the features into imaginary numbers again, so as to be converted by IFFT in the next step; the purpose of IFFT conversion is to match the positions of the frequency domain features and the spatial domain features, so as to infer and determine the position of the damage target.

[0101] Wherein, the road disease image is obtained after the PatchFFT processing respectively By formula:

[0102]

[0103] Wherein, Respectively represent the feature representation after the full image and the block Fourier transform; PatchFFT(In) represents that the input image In is processed using the PatchFFT method, and the corresponding Fourier transform result is obtained;

[0104] The inference result obtained by FS-SSBlock processing is represented by formula:

[0105] Output=Linear(Input)+Linear(SS2D(DWConv(Linear(Input))));

[0106] Wherein, Linear(Input) represents that the input data is first linearly transformed to prepare for subsequent processing, DWConv() represents that then the deep separable convolution is applied to reduce the amount of calculation and the number of parameters, SS2D() represents that then the spatial-spectral two-dimensional conversion is applied to extract the frequency domain features, Linear() represents that finally the data after SS2D conversion is linearly transformed again, and the frequency domain features are further processed;

[0107] After completing the inverse fast Fourier transform IFFT conversion, scale alignment is performed by using 3*3 convolution to obtain the frequency domain feature {F f i |i=0,1,2}, wherein, F f i | represents the i-th frequency domain feature vector, i is an index, used to distinguish different frequency domain feature vectors, and its value range is 0 to 2.

[0108] S103: The frequency domain features and the spatial domain features corresponding to the scale are fused by using a dynamic parameter weighting method to obtain a multi-scale feature map, and the formula used by the dynamic parameter weighting method is represented as:

[0109] Output=Add(x1,x2);

[0110] x1=Xxv rgb xT rgb ;

[0111] x2=X feq xv feq xT feq ;

[0112] wherein Output represents the finally generated multi-scale feature map, Add(x1, x2) represents the fusion of two different features, X and X feq are the spatial domain features and frequency domain features corresponding to the scale, respectively, v rgb and v feq are two different learnable dynamic parameters, v rgb is used to adjust the weight and transformation of the spatial domain feature, v feq is used to adjust the weight and transformation of the frequency domain feature, T rgb and T feq are the frequency domain and spatial domain features corresponding to the channel after splicing and convolution;

[0113] Specifically, please refer to Figure 7 , Figure 7 is a schematic diagram of using a dynamic parameter weighting method for feature fusion of the frequency domain feature and the spatial domain feature, Figure 7 wherein X and X feq are the spatial domain feature and the frequency domain feature corresponding to the scale, respectively, and the spatial domain feature X and the frequency domain feature X feq are merged to form a more rich feature representation. The merged feature is processed by a convolution layer CONV to extract a deeper level feature, and the output of the convolution layer CONV is transformed Transformation for subsequent feature fusion;v rgb and v feq are two different learnable dynamic parameters, v rgb is used to adjust the weight and transformation of the spatial domain feature, v feq is used to adjust the weight and transformation of the frequency domain feature. By dynamically adjusting the weight of the spatial domain feature and the frequency domain feature, the two are weighted and fused. This method can dynamically adjust the contribution of each feature according to the feature requirement of different scales, so as to obtain a more robust feature representation. Finally, the weighted spatial domain feature and the frequency domain feature are added to obtain the fused multi-scale feature map Feature. Further, the spatial domain feature and the frequency domain feature have three parameters, which are input two by two during feature fusion, for example, the spatial domain feature is {1, 2, 3} and the frequency domain is {4, 5, 6}, then the input is {1, 4} {2, 5} {3, 6}, by integrating this information, the finally fused multi-scale feature map is {7, 8, 9}, 7 is the fusion of 1 and 4, 8 is the fusion of 2 and 5, and 9 is the combination of 3 and 6.

[0114] S104: input the fused multi-scale feature map to a feature pyramid network for feature enhancement;

[0115] Specifically, after obtaining the multi-scale feature maps, it is necessary to use the feature pyramid model to enhance the features of the input multi-scale feature maps. In order to better obtain image feature information, the FS-SSM module is used for feature enhancement, and the FS-SSM module is mainly obtained by using the module designed in the frequency domain feature extraction, and a convolution branch is designed to obtain inference information of different depths. The model structure in the FS-SSM module is expressed by a formula as follows:

[0116] T=CBR(Input);

[0117] OUTPUT=FS-SSBlock^3(T)+CBR(t);

[0118] Wherein, the input data Input is subjected to convolution Convolution, batch normalization Batch Normalization and ReLU activation operation, and this series of operations is simply referred to as CBR, and the feature map T is obtained after CBR processing, and this step is the preliminary stage of feature extraction, which provides basic features for subsequent operations. Then, the feature map T is input into the FS-SSBlock module, and the module is stacked and used three times (FS-SSBlock^3). Each time through the FS-SSBlock module, the feature map is further enhanced and processed to extract deeper feature information. Three times of stacking operation means that the feature map has undergone three times of feature enhancement of different levels, so as to capture more diversified features. At the same time, the input data also undergoes an additional CBR operation to obtain the feature map t. This feature map t can be regarded as a transformed version of the input data, which is used to provide inference information of different depths. Finally, the output result of the FS-SSBlock module stacked three times is added to the feature map t to obtain the final output OUTPUT. The OUTPUT is the output of the FS-SSM module, which contains the feature map after multiple feature enhancement and different depth inference information stacking. Such design enables the FS-SSM module to effectively enhance the feature information of the input data, improve the feature extraction effect, and thus help subsequent image analysis and processing tasks.

[0119] S105: input the enhanced multi-scale feature map into the detection head to obtain a detection result and output.

[0120] Specifically, the detection result is screened by non-maximum suppression to filter out redundant features, and the detection target information with the highest confidence is obtained, and the result is output. From the candidate box list output by the detector, all candidate boxes are sorted in descending order according to the confidence (such as classification score), to ensure that the detection result with the highest confidence is processed first. The candidate box with the highest confidence is selected as the reference box, which is added to the final output list and removed from the candidate list. Through such setting, the road disease detection accuracy is improved.

[0121] Further, the road disease image acquisition includes:

[0122] According to the computing power condition, the image frequency domain feature extraction model and the image spatial domain feature extraction model, the image resolution is adjusted to ensure that the input picture meets the input requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model and adapts to different computing power conditions.

[0123] Specifically, the image frequency domain feature extraction model and the image spatial domain feature extraction model are used by default in this embodiment. If they are not trained, pre-training is required. At this time, the road disease image processing is divided into two stages: in the first stage, a part of the road disease image is preprocessed and divided. The division is into training data and test data, so as to evaluate the image frequency domain feature extraction model and the image spatial domain feature extraction model. In the second stage, the road disease image is preprocessed and directly input into the model for inference. The preprocessing in this stage is the adjustment of the image resolution. Through such setting, it is ensured that the road disease image meets the input requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model.

[0124] Please refer to Figure 8 The application also provides a road disease detection system based on space-frequency dual-domain fusion, comprising:

[0125] An image acquisition module 1 is used to acquire road disease images.

[0126] A feature extraction module 2 is used to extract frequency domain features and spatial domain features of different scales from the road disease images.

[0127] A feature fusion module 3 is used to fuse the frequency domain features and the spatial domain features of corresponding scales by using a dynamic parameter weighting method to obtain a multi-scale feature map. The formula of the dynamic parameter weighting method is as follows:

[0128] Output = Add (x1, x2);

[0129] x1 = X x v rgb x T rgb ;

[0130] x2 = X feqXv feq XT feq ;

[0131] wherein, Output represents the final generated multi-scale feature map, Add(x1, x2) represents the fusion of two different features, X represents the spatial domain feature corresponding to the scale, v rgb and T rgb are two different learnable dynamic parameters related to color information, used to adjust the weight and transformation of the spatial domain feature; X feq represents the frequency domain feature corresponding to the scale, v feq and T feq are another two learnable dynamic parameters related to frequency information, used to adjust the weight and transformation of the frequency domain feature;

[0132] Feature enhancement module 4: for inputting the fused multi-scale feature map into the feature pyramid network for feature enhancement;

[0133] Detection and output module 5: for inputting the enhanced multi-scale feature map into the detection head for detection to obtain the detection result and output.

[0134] Specifically, the specific implementation of the road disease detection system based on spatial and frequency dual domain fusion can refer to the specific implementation of the road disease detection method based on spatial and frequency dual domain fusion, which will not be described here.

[0135] For reference Figure 9 , the present application also provides an electronic device, comprising:

[0136] Memory 7, for storing one or more programs;

[0137] Processor 6; the processor 6 and the memory 7 are connected through the communication interface 8;

[0138] When the one or more programs are executed by the processor 6, all or part of the above methods are realized.

[0139] In the fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor 6 to realize all or part of the above methods.

[0140] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics thereof. The embodiments should be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference herein to any prior art is to be taken as an admission that the application is not entitled to antedate such prior art by virtue of prior application. Any reference to the use of a term in the singular herein shall be understood in the context to describe a particular example or embodiment of the application and should not be construed as limiting the scope of the application to that particular example or embodiment. Any reference to use of terms in the plural herein shall be understood as describing a particular example or embodiment of the application and should not be construed as limiting the scope of the application to that particular example or embodiment.

Claims

1. A road damage detection method based on space-frequency dual-domain fusion, characterized in that: The following steps are involved: S101: Acquire road damage images; S102: Extract frequency domain features and spatial domain features of different scales from road damage images; S103: The frequency domain features and spatial domain features of the corresponding scales are fused using a dynamic parameter weighting method to obtain a multi-scale feature map. The formula used in the dynamic parameter weighting method is expressed as: Output=Add(x1,x2); x1=X×v rgb ×T rgb ; x2=X feq ×v feq ×T feq ; Among them, Output represents the final generated multi-scale feature map, Add(x1,x2) represents the fusion of two different features, X and X feq They are the spatial domain features and frequency domain features of the corresponding scales, v rgb and v feq are two different learnable dynamic parameters, v rgb Used to adjust the weight and transformation of spatial features, v feq Used to adjust the weight and transformation of frequency domain features, T feq and T rgb It is the frequency domain and spatial domain characteristics of the corresponding channel after splicing and convolution; S104: Input the fused multi-scale feature map into the feature pyramid network for feature enhancement; S105: Input the enhanced multi-scale feature map into the detection head for detection to obtain the detection result and output it.

2. The road damage detection method based on space-frequency dual-domain fusion according to claim 1 is characterized in that: The specific steps of extracting spatial features of different scales from road damage images include: The road damage image is input into the image spatial domain feature extraction model, which includes the STEM layer, GR-BneckV2, and dynamic snake convolution DSConv. The STEM layer downsamples and enriches the image features, which can be expressed as follows: Among them, F r 0 represents the image features after STEM layer processing, Represents the operation of the STEM layer, and I represents the input image feature; After the downsampled features are enhanced by GR-BneckV2, the image features are divided into two groups and input into the multi-layer GR-BneckV2 structure and the single-layer dynamic snake convolution DSConv respectively. The multi-layer GR-BneckV2 structure ensures that the model obtains richer detailed information, while the single-layer dynamic snake convolution DSConv captures crack-like features while transmitting low-dimensional feature gradient information. First layer processing: T r i =GRBneckv2(F r i-1 ); Multi-layer GR-BneckV2 structure processing: M r i =GRBneckv2 m (T r i ); Single-layer DSConv structure processing: N r i =DSConv(T r i ); Feature stitching and final processing: F r i =GRBneckv2(Concat[M r i ,N r i ]); Among them, the first layer is processed as the down-sampled feature F r i-1 After enhanced processing by GR-BneckV2 module, the enhanced feature T is obtained. r i ;GRBneckv2 is used to extract and enhance features; multi-layer GR-BneckV2 structure is processed into enhanced features T r i is input to the m-layer GRBneckv2 m In the structure, after multiple GRBneckv2 processing, the further enhanced feature M is obtained. r i ; The single-layer DSConv structure is: the enhanced feature T r i It is also input into the dynamic snake convolution DSConv module; DSConv can capture the dynamic changes of features and generate new features N r i ; Feature splicing and final processing are: the feature M after multi-layer GRBneckv2 structure processing r i and the feature N after DSConv processing r i Concat is performed, and the concatenated features are integrated again through the GRBneckv2 module to obtain the final spatial feature F r i ; Among them, GRBneckv2 follows the design of the original network structure when the step size is 1; when the step size is 2, dynamic snake convolution is added as a convolution enhancement method; it is expressed by the formula: T=Ghost_Module(In)·DFC(In); Among them, In and out are the input and output of the GRBneckv2 structure respectively; when the step size s = 1, the GRBneckv2 structure relies on the feature map generated by the Ghost Module; T represents the difference or enhanced part between the feature map obtained after the input In is processed by the Ghost Module and the feature map obtained after the dimensionality reduction convolution layer DFC is processed. The final output Out is the combination of the input In and T, which means that the enhanced features obtained by the Ghost Module and DFC processing are added on the basis of maintaining the original input features; When the step size s = 2, the GRBneckv2 structure introduces dynamic snake convolution DSCConv as an additional enhancement method based on the Ghost Module and DFC; the output Out is the combination of the input In and T processed by DSCConv; this means that when the step size is 2, the introduction of DSCConv enhances the feature extraction capability to adapt to larger spatial downsampling.

3. The road damage detection method based on space-frequency dual-domain fusion according to claim 2 is characterized in that: The specific steps of extracting frequency domain features of different scales from road damage images include: Input the road damage image into the image frequency domain feature extraction model, which includes PatchFFT, FS-SSBlock and Patch merging; The road damage image is processed by PatchFFT to obtain the features feature After 3×3 convolution with a step size of 2 and FS-SSBlock processing, the inference results are obtained. Then, the Patch merging method is used to convert the features into imaginary form for the next IFFT conversion. Among them, the road disease image obtains features after PatchFFT processing. It is expressed by the formula: in, Represent the feature representation of the whole image and the block Fourier transform respectively; PatchFFT(In) means using the PatchFFT method to process the input image In and obtain the corresponding Fourier transform result; The inference result obtained by FS-SSBlock processing is expressed as follows: Output=Linear(Input)+Linear(SS2D(DWConv(Linear(Input)))); Among them, Linear(Input) means that the input data is first linearly transformed to prepare for subsequent processing, DWConv() means that depthwise separable convolution is then applied to reduce the amount of calculation and the number of parameters, SS2D() means that space-spectrum two-dimensional conversion is then applied to extract frequency domain features, and Linear() means that the SS2D converted data is finally linearly transformed again to further process the frequency domain features; After completing the inverse fast Fourier transform IFFT, the scale is aligned using 3×3 convolution to obtain the frequency domain features {F f i |i=0,1,2}, where F f i |i represents the i-th frequency domain eigenvector. i is an index used to distinguish different frequency domain eigenvectors, and its value range is 0 to 2.

4. The road damage detection method based on space-frequency dual-domain fusion according to claim 3 is characterized in that: The obtaining of road damage images comprises: According to the computing power conditions, the requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model, adjust the image resolution to ensure that the input image meets the input requirements of the image frequency domain feature extraction model and the image spatial domain feature extraction model and adapts to different computing power conditions.

5. The road damage detection method based on space-frequency dual-domain fusion according to claim 3 is characterized in that: The steps of inputting the fused multi-scale feature map into the feature pyramid network for feature enhancement include: The FS-SSM module is used to obtain inference information at different depths to achieve attention and feature enhancement of frequency and spatial strip images.

6. The road damage detection method based on space-frequency dual-domain fusion according to claim 3 is characterized in that: The specific steps of inputting the enhanced multi-scale feature map into the detection head for detection to obtain the detection results and output them include: The detection results are filtered out through non-maximum suppression to filter out redundant features, obtain the detection target information with the highest confidence, and output the results.

7. A road damage detection system based on space-frequency dual-domain fusion, characterized by: include: Image acquisition module: used to obtain road damage images; Feature extraction module: used to extract frequency domain features and spatial domain features of different scales from road damage images; Feature fusion module: The frequency domain features and spatial domain features of the corresponding scales are fused using the dynamic parameter weighting method to obtain a multi-scale feature map. The formula used in the dynamic parameter weighting method is expressed as follows: Output=Add(x1,x2); x1=X×v rgb ×T rgb ; x2=X feq ×v feq ×T feq ; Among them, Output represents the final multi-scale feature map, Add(x1,x2) represents the fusion of two different features, X represents the spatial domain feature of the corresponding scale, and v rgb and T rgb are two different learnable dynamic parameters used to adjust the weight and transformation of spatial features; X feq Represents the frequency domain characteristics of the corresponding scale, v feq and T feq are two other learnable dynamic parameters used to adjust the weight and transformation of frequency domain features; Feature enhancement module: used to input the fused multi-scale feature map into the feature pyramid network for feature enhancement; Detection and output module: used to input the enhanced multi-scale feature map into the detection head for detection to obtain the detection results and output them.

8. An electronic device, characterized in that: include: a memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images