Super-resolution processing method and system for road surface image shot by unmanned aerial vehicle

Abstract

The invention relates to a super-resolution processing method and system for a pavement image shot by an unmanned aerial vehicle, and the method comprises the following steps: carrying out the aerial photographing of a pavement target through the unmanned aerial vehicle, and obtaining a pavement image; meanwhile, a high-pixel camera is used for shooting the same target as a reference image, and the reference images of all the targets form a reference image set; the ESR-KAN network model comprises a KAN fusion module, a reconstruction module and M serial KAN-channel attention, the reconstruction module is used for recovering the spatial shape of an image, and the M serial KAN-channel attention is used for obtaining M serial KAN-channel attention; and performing residual connection on a result processed by the reconstruction module and a result processed by the up-sampling processing of the reference image to obtain a final high-resolution image Y. In the super-resolution processing of the road surface image shot by the unmanned aerial vehicle, the image quality is improved, the availability and the analysis accuracy of the image are enhanced, and a new solution is provided for image processing of the unmanned aerial vehicle in various application scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution processing, and in particular to a super-resolution processing method and system for road surface images taken by an unmanned aerial vehicle. Background Art

[0002] When drones perform aerial reconnaissance, terrain mapping, traffic monitoring and other tasks, the image quality, especially the resolution, taken by the cameras they carry is crucial to obtaining accurate information. In the process of drone inspections, the image resolution determines the ability of the drone camera to capture road details. High-resolution images can provide more detailed information, which helps to accurately identify, track and analyze road defects.

[0003] In order to improve the resolution of road images taken by drones, existing methods mainly use high-resolution cameras and multi-frame image synthesis and other technologies. For example, some studies have suggested using a high-resolution SLR camera with more than 20 million pixels and selecting the maximum resolution mode when shooting to reduce noise and blur in road images. However, directly using a high-resolution camera requires a significant increase in equipment costs. Technologies such as multi-frame image synthesis require an increase in the number of photos taken, and synthesis takes a lot of time, is cumbersome to operate, and may introduce errors.

[0004] In recent years, image super-resolution technology has been widely used in the field of improving image resolution. Through deep learning models, high-resolution images are reconstructed from low-resolution images to restore more detailed features (wide-angle images, macroscopic cracks visible to the naked eye on the road surface, shooting methods). However, these methods have some defects and shortcomings. Traditional image processing methods, such as filtering and sharpening, may introduce artifacts or cause the loss of important feature information. In addition, due to the low resolution and small targets of road images taken by drones, the convolutional neural network structure is affected by the loss of fine-grained feature information and feature blurring, and the performance is significantly reduced. Some super-resolution methods for microscopic images are not accurate enough for the recognition of such wide-angle road images taken by drones.

[0005] With the rapid development of artificial intelligence technology, image super-resolution technology has become one of the key areas for improving image quality. This technology can effectively reconstruct high-resolution images from low-resolution images through deep learning models, thereby restoring rich detail features. Image super-resolution processing technology provides a new solution to the limitations of traditional image processing methods. Compared with traditional methods, these advanced image processing technologies can improve the resolution of drone images with higher efficiency and better quality, and gradually become a new trend to replace traditional image processing methods. Through these innovative technologies, drones will be able to obtain clearer and more accurate image data when performing road shooting tasks, greatly enhancing the usability of images and the accuracy of analysis. Summary of the invention

[0006] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a super-resolution method and system for photographing road images using an unmanned aerial vehicle, with the aim of solving the above-mentioned obvious defects in the prior art.

[0007] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0008] In a first aspect, the present invention provides a method for super-resolution processing of road images taken by a drone, the method comprising the following contents:

[0009] The road target is photographed by drone to obtain a road image. At the same time, a high-pixel camera is used to photograph the same target as a reference image. The reference images of all targets constitute a reference image set. The shooting angles of the reference image and the road image can be different.

[0010] Construct an efficient KAN reference image super-resolution ESR-KAN network model:

[0011] The ESR-KAN network model includes a KAN fusion module, a reconstruction module, and M serial KAN-channel attention. The KAN fusion module includes a first layer composed of two KANs in parallel and a second layer composed of one KAN. The three KANs have the same structure. The two KANs in the first layer are used to analyze the input image X and the reference image X respectively. R Acquire feature map information; after the two KAN processing results of the first layer are spliced, they enter the KAN of the second layer to obtain the output feature map O0;

[0012] Each KAN-channel attention consists of a serial structure consisting of a global average pooling GAP and two consecutive KANs, as well as a multiplication operation and residual connection to combine the inputs; in the KAN-channel attention, the input feature map O iGlobal average pooling GAP is applied to compress channel features. Two consecutive KANs are used to learn the importance weight of each channel. A new vector is generated from the importance weight of each channel. The new vector is used to transform the input feature map O i The channels are weighted to obtain the channel weight vector; the channel weight vector is combined with the input feature map O i Multiply, and then add to the input feature map O i The addition operation is performed to obtain the output O of the KAN-channel attention i+1 , and used as the input of the next KAN-channel attention, and so on to get the output O of the last KAN-channel attention M , i=0,1,…,M-1;

[0013] The reconstruction module is used to restore the spatial shape of the image. The result processed by the reconstruction module is residually connected with the result of the reference image after upsampling to obtain the final high-resolution image Y.

[0014] Furthermore, the reconstruction module includes a 3×3 convolutional layer, a ReLU activation function and a 3×3 convolutional layer connected in sequence.

[0015] Further, M=4-6.

[0016] Furthermore, the objective loss function l during the training of the ESR-KAN network model is total For: total = l pred +l sparse

[0017] l pred =|YX|1

[0018]

[0019] Among them, l pred is the prediction loss function; l sparse is the KAN spline sparse loss function; λ is the parameter that controls the overall regularization amplitude, μ1 and μ2 are relative calculation parameters, both set to 1; φ l represents the activation function of the lth layer in the last KAN of the last KAN-channel attention, |φ l |1 represents the L1 norm of the activation function, S(φ l ) is the entropy regularization term, L is the number of KAN layers; X represents the original input image, and Y represents the high-resolution image predicted by the network structure.

[0020] In a second aspect, the present invention provides a super-resolution processing system for road images taken by a drone, the system comprising:

[0021] UAVs are used to obtain road surface images of road targets through aerial photography;

[0022] A high-pixel camera is used to obtain a high-resolution reference image of the target at a certain viewing angle;

[0023] An image preprocessing module is used to perform data enhancement processing on road surface images and reference images;

[0024] Database, used to store images and parameters;

[0025] The ESR-KAN network model is used to perform super-resolution reconstruction on road surface images.

[0026] Furthermore, the system also includes a network interface, a display device, and an image processing unit, and the display device is used to view the processing results and image quality assessment results in real time.

[0027] Furthermore, the system also includes a road defect semantic segmentation model, and the super-resolution image reconstructed by the ESR-KAN network model is used for road defect recognition.

[0028] Furthermore, the super-resolution images are used to monitor forest cover changes, urban expansion and agricultural land changes, road maintenance and road inspection.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The efficient KAN mentioned in this invention has a reference image super-resolution processing technology ESR-KAN (Efficient Super-Resolution with Reference Image via Kolmogorov-Arnold) which shows significant advantages in many aspects:

[0031] The three core components of the reference image super-resolution ESR-KAN network model of the present invention, namely, the KAN fusion module, the KAN-channel attention and the reconstruction module, work together to enable the model to significantly improve the detail clarity and overall quality of the image when processing low-resolution images of aerial road targets.

[0032] The KAN fusion module in the present invention is the primary link of the model. The KAN fusion module cleverly utilizes multiple KAN structures with a triangular distribution, adopts a parallel processing mechanism, and simultaneously performs deep feature extraction on the input low-resolution road image and its corresponding high-resolution reference image. This process not only enhances the model's ability to capture image features, but also provides more abundant and comprehensive information for subsequent processing procedures through the merging of feature maps. The number of reference images in the present invention is relatively small, and it is sufficient to ensure that each target has a corresponding reference image, and there is no requirement for the shooting angle, etc.

[0033] KAN-channel attention uses global average pooling and continuous KAN layers to accurately learn the importance weight of each channel, achieving fine-grained control of feature maps. It not only enhances the model's discrimination ability, but also provides a solid foundation for the final image reconstruction. The fine-grained feature control capability enables the model to highlight important features and suppress unimportant information, further improving the quality of the reconstructed image.

[0034] The reconstruction module undertakes the important task of converting the optimized feature map into a high-resolution image. It uses a combination of 3×3 convolutional layers and ReLU activation functions to accurately restore the spatial details of the image. At the same time, the residual connection mechanism ensures the consistency between the reconstructed image and the original high-resolution reference image. This process not only improves the accuracy of the reconstructed image, but also enhances the clarity and realism of the image, meeting the requirements of the super-resolution task for high-quality output.

[0035] Combining these three modules, the ESR-KAN network model has demonstrated excellent performance in the field of image super-resolution. This model can not only effectively process low-resolution images, but also make full use of the information in high-resolution reference images to achieve high-precision image reconstruction. In addition, computational efficiency and parameter optimization are taken into account to reduce the scale of the number of computational parameters in the network structure, so that it can run efficiently on resource-constrained devices and has broad application prospects. This innovative network structure can accurately capture and reconstruct image details while keeping the model lightweight, providing a new solution for the field of image super-resolution.

[0036] In summary, the ESR-KAN network model of the present invention not only improves the image quality, enhances the image availability and the accuracy of analysis in the super-resolution processing of road images taken by drones, but also provides a new solution for image processing of drones in various application scenarios. It also plays an important role in cost control and improving the accuracy of subsequent computer vision tasks, and will promote the further development and application of drone technology in various fields. The present invention significantly improves the resolution and quality of drone aerial road images, and provides more accurate and clear image support for drone vision application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a structural diagram of the ESR-KAN network model in the present invention.

[0038] Figure 2 Flowchart of road surface super-resolution for practical application of ESR-KAN. DETAILED DESCRIPTION

[0039] In order to more clearly describe the technical problems, technical solutions and advantages of the present invention, the following will be described in detail with reference to the drawings and embodiments. It should be noted that these embodiments are only used to illustrate the principles and application scope of the present invention and should not be regarded as limiting the present invention.

[0040] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0041] The super-resolution processing method of the road surface image taken by the unmanned aerial vehicle of the present invention uses an efficient KAN reference image super-resolution ESR-KAN (Efficient Super-Resolution with Reference Image via Kolmogorov-Arnold) network model, and includes the following steps:

[0042] Step 1: Get the dataset

[0043] The road target is photographed by drone aerial photography to obtain a road image, and the same target is photographed by a high-pixel camera as a reference image. The reference images of all targets constitute a reference image set, and the road image set and the reference image set obtained by drone aerial photography of the road are stored in corresponding files respectively. The reference image can be an image of the same target and the road image at different shooting angles. Multiple road images of the same target can correspond to one reference image.

[0044] In order to further improve the generalization ability and robustness of the model, data augmentation technology is applied. The road images of drone aerial road targets in the training set are rotated, scaled, flipped, brightness and contrast adjusted, and random noise is added to generate additional variants, expand the diversity of the training data set, enhance the generalization ability of the model, and ensure that the model can better handle various actual situations.

[0045] Step 2: Build an efficient KAN reference image super-resolution ESR-KAN network model

[0046] The efficient KAN has a reference image super-resolution ESR-KAN network model, and its structure diagram is as follows Figure 1 As shown in the figure, it mainly consists of three modules: KAN fusion module, reconstruction module and M serial KAN-channel attention modules.

[0047] The KAN fusion module includes a first layer composed of two KANs in parallel and a second layer composed of one KAN. The three KANs have the same structure and are distributed like a triangle. The two KANs in the first layer are set in parallel to perform a fusion operation on the input image X and the reference image X. R To obtain feature map information, where X, (H, W represent the height and width of the image, and C represents the number of channels of the image). In this process, the processing of two parallel KANs can be expressed as:

[0048]

[0049] Afterwards, the obtained X0 and After splicing, the two layers are input into the second layer of KAN for processing. O0 represents the output result of the KAN fusion module, which can be expressed using the following formula:

[0050]

[0051] Wherein, D is a hyperparameter representation, which refers to the channel dimension that the feature map O0 should maintain. In this embodiment, D is set to 256.

[0052] The KAN-channel attention includes a serial structure consisting of a global average pooling GAP and two consecutive KANs, as well as a multiplication operation and residual connection to combine the inputs. i Global average pooling GAP is applied to compress channel features. Two consecutive KANs are used to learn the importance weight of each channel. A new vector is generated from the importance weight of each channel. The new vector is used to transform the input feature map O i The channels are weighted to obtain the channel weight vector; the channel weight vector is combined with the input feature map O i Multiply, and then add to the input feature map O i The addition operation is performed to obtain the output O of the KAN-channel attention i+1 , and used as the input of the next KAN-channel attention, and so on to get the output O of the last KAN-channel attention M , i=0,1,…,M-1.

[0053] The structure of the reconstruction module can be described as: a 3×3 convolution layer, a ReLU activation function, and a 3×3 convolution layer at the end. These three components are serially connected to form a reconstruction module. The reconstruction module is used to restore the spatial shape of the image. The result processed by the reconstruction module and the result of the upsampling of the reference image are added to obtain the final output high-resolution image Y. The whole process can be expressed as the following formula:

[0054] Y = Conv(ReLU(Conv(O M )))+UP(X R )

[0055] The overall process of the ESR-KAN network model can be summarized as follows: the input is the training set image X and the reference image X of the same type. R First, the input image X is passed through a KAN in the first layer of the KAN fusion module to obtain feature map information; at the same time, the reference image X R Perform an upsampling to get UP(X R ), and then UP(X R ) is input into a KAN of the first layer of the KAN fusion module to obtain feature map information. After the two KAN operation processes of the first layer are completed, the two feature map information obtained are spliced, and then processed by the KAN of the second layer of the KAN fusion module to output the feature map O0.

[0056] Next, M identical KAN-channel attention serial processing flows are performed, and feature map O0 is used as the input of the first KAN-channel attention. First, feature map O0 uses global average pooling GAP (GlobalAverage Pooling) to compress each channel feature of input feature map O0, and then two layers of continuous KAN are used to learn the importance weight of each channel. Then, a weight vector is generated from the learned weights, and the channels of the input feature map are weighted by the weight vector to obtain the channel weight vector.

[0057] Finally, the channel weight vector is multiplied by the input feature map O0, and the result is added to the input feature map O0 to obtain the input O1 of the next KAN-channel attention. Through the same process, after M consecutive series of KAN-channel attention, the final output feature map O M .

[0058] Finally, the feature map O M As input, the image is reconstructed through a reconstruction module consisting of a 3×3 convolutional layer, an activation function ReLU and a 3×3 convolutional layer to obtain the final high-resolution image Y.

[0059] KAN (Kolmogorov-Arnold Networks) in the present invention is a new type of deep learning model, whose design is inspired by the Kolmogorov-Arnold representation theorem. Its core feature is that it places activation functions on the edge of the network (on weights), and these activation functions are learnable and are usually parameterized using B-splines. KAN includes SiLU activation functions and linear layers, as well as a residual structure based on B-spline functions.

[0060] The processing of KAN can be defined as:

[0061]

[0062] Among them, ω1 and ω2 represent weights; c i represents the basis function coefficient; G represents the grid parameter, the default value is 5; k is the specified parameter set, the default value is 3; specifically, G is the number of intervals of the original grid, and k is the order of the B-spline, indicating how many node intervals each B-spline function is non-zero.

[0063] The B-spline function is composed of multiple piecewise polynomials. Each piecewise polynomial is defined by a control point (grid point). According to the characteristics of the B-spline function, a specified domain [t0, t G ], when a one-dimensional function is approximated using a k-order B-spline function, a coarse grid with G intervals is expanded to a fine grid with G+k intervals. In the process of expanding to G+k, the piecewise polynomial Given a recursive definition:

[0064]

[0065] For k>0:

[0066]

[0067] The objective loss function of the network model of the present invention combines the prediction loss function l pred and KAN spline sparse loss function l sparse , which provides a more refined adjustment capability for the model and the objective loss function l total is defined as follows:

[0068] l pred =|YX|1

[0069] l total = l pred +l sparse

[0070] Among them, X represents the original input image, and Y represents the predicted high-resolution image;

[0071] In the present invention, the KAN spline sparse loss function refers to the spline sparse loss of the last KAN of the last KAN-channel attention. The KAN spline sparse loss function l sparse Defined as:

[0072]

[0073] Among them, λ is a parameter that controls the overall regularization amplitude, μ1 and μ2 are relative calculation parameters, both set to 1. l represents the activation function of the lth layer in the last KAN, L is the number of layers in the last KAN, |φ l |1 represents the L1 norm of the activation function, S(φ l ) is the entropy regularization term.

[0074] The present invention can reduce the number of parameters while maintaining high accuracy, improve the interpretability and sparsity of the network model, and demonstrate higher model performance.

[0075] Step 3: Train the Efficient KAN reference image super-resolution ESR-KAN network model

[0076] The training process of the ESR-KAN network model with reference image super-resolution is:

[0077] When starting training, first complete the parameter setting for network initialization: the epoch of the training network is set to 500, the optimizer uses the Adam optimizer, and the initial learning rate of Adam is set to 5e-5 and 1e-6.

[0078] The training set comes from road images obtained by drone aerial photography and reference images of the same target taken with a high-pixel camera. These images are processed by data augmentation techniques, including rotation, scaling, flipping, brightness and contrast adjustment, and adding random noise to enhance the generalization ability and robustness of the model.

[0079] During the training process, the input images of the training set and the reference images are input into the ESR-KAN network model, and these images are read according to the training set storage path;

[0080] When the overall network objective loss function l total When there is no longer a significant decrease (error ±1e-5), the network training is considered to be stable and the network training process is completed.

[0081] The ESR-KAN network model has achieved significant cost reduction and efficiency improvement in the field of UAV aerial road image processing. It provides a more economical solution for scientific research and industrial applications by improving processing efficiency and reducing processing costs, and greatly promotes the popularization and application of UAV aerial road image processing technology.

[0082] Example 1

[0083] The super-resolution processing method of the road surface image taken by the UAV of the present invention uses the Efficient Super-Resolution with Reference Image via Kolmogorov-Arnold (ESR-KAN) network model to achieve the super-resolution processing of the road surface image taken by the UAV through the following steps:

[0084] 1. Data collection and preparation stage

[0085] 1.1 Data Collection

[0086] Equipment: Use drones to collect target aerial road images. High-pixel camera equipment captures high-resolution (4000*3000 pixels) reference images.

[0087] Operation: Take a set of aerial road images for each sample, and use a high-pixel camera to take reference images to ensure that different viewing angles and multiple sets of target images are covered.

[0088] For example, in this embodiment, the target of the acquisition is the area where the road surface is damaged. The target area is locked, and aerial photography of the target area is performed at different angles. At the same time, a high-pixel camera is used to take reference images at any angle. This target area is a road surface defect that is visible to the naked eye, and the drone obtains its wide-angle image through aerial photography.

[0089] A total of 4,800 road images of road targets taken by drones during the network model training process of the present invention are obtained. These road images are divided into a training set and a test set in a ratio of 7:3; the number of reference image sets is 200, and reference images of the same type of targets are used for training.

[0090] 1.2 Data processing

[0091] Dataset acquisition: The collected image data is preliminarily processed, including removing obvious noise and outliers in the image, and cropping and resizing the image to ensure the consistency and standardization of the image dataset. The collected images are divided into training set and test set in a 7:3 ratio.

[0092] Data augmentation: Perform data augmentation operations such as rotation, scaling, flipping, adjusting brightness and contrast on the training set to improve the robustness and generalization ability of the model.

[0093] 2. Network model training phase

[0094] 2.1 Model construction

[0095] Network structure: Construct the ESR-KAN network model, including the KAN fusion module, 5 serial KAN-channel attention and reconstruction modules. M = 5.

[0096] The KAN fusion module consists of two KANs in parallel in the first layer and one KAN in the second layer. The three KANs have the same structure. The two KANs in the first layer are used to fusion the input image X and the reference image X respectively. R Acquire feature map information; after the two KAN processing results of the first layer are spliced, they enter the KAN of the second layer to obtain the output feature map O0;

[0097] The processing of each KAN-channel attention can be divided into two stages. In the first stage, the input feature map O i (i=0,1,…,M-1;O i Represents the input of the i-th KAN-channel attention) applies global average pooling GAP for channel feature compression, which means that each channel in the feature map will get a value to represent its global importance. In the second stage, two consecutive KANs are used to learn the importance weights of each channel, and then these weights are used to generate a new vector that is used to compress the input feature map O i Finally, the channel weight vector is combined with the input feature map O i Multiply, adjust the global synthesis ratio of local pixels with high precision, and then add it to the input feature map O i The addition operation is performed to obtain the output O of the KAN-channel attention i+1 , and used as the input of the next KAN-channel attention, and so on to get the output O of the last KAN-channel attention M .

[0098] The process of obtaining the final high-resolution image Y after the reconstruction module is expressed as:

[0099] Y=Conv(ReLU(Conv(O5)))+UP(X R )

[0100] Among them, Conv represents a 3×3 convolutional layer; UP represents upsampling; ReLU represents the ReLU activation function.

[0101] The network first extracts feature map information through the KAN fusion module, and then strengthens the feature map information through the serial processing of KAN-channel attention. Each KAN-channel attention improves the importance of features through global average pooling and weight learning. Finally, the reconstruction module restores the spatial details of the image through a 3×3 convolutional layer and ReLU activation function, generating the desired high-resolution output image Y.

[0102] 2.3ESR-KAN network model training

[0103] Training settings: The network model is trained using a deep learning framework. The Adam optimizer is used during training, and the initial learning rate is set to 5e-5 and 1e-6. The learning rate and other hyperparameters are adjusted according to the needs of model training.

[0104] Loss function: Combined with the prediction loss function l pred and KAN spline sparse loss function l sparse , calculate the target loss function l total , optimize the model parameters to improve image quality.

[0105] When the overall network objective loss function l total When there is no longer a significant decrease (error ±1e-5), the network training is considered to be stable and the network training process is completed. When testing the network, the prepared test set images are input, and no additional reference images are required. The network weights trained with the training set are imported. For the road images taken by drones, when the peak signal-to-noise ratio PSNR and structural similarity SSIM of the road images taken by drones reach 44.89 (error ±0.03) and 0.991 (error ±0.02) respectively, the entire network training is completed.

[0106] 3. Super-resolution processing and analysis stage

[0107] 3.1 Model Deployment

[0108] Deployment: The trained ESR-KAN network model is deployed to the road image processing system for real-time processing of drone aerial road images.

[0109] Processing: For the input image, the trained ESR-KAN network model is used to generate the corresponding high-resolution image for detail restoration and enhancement.

[0110] 3.2 Results Analysis

[0111] Quality Assessment: The quality of the super-resolution image is evaluated by calculating the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index) of the image.

[0112] 4. Application and optimization stage

[0113] 4.1 Image Application

[0114] Scientific research: Processed high-resolution images are used for scientific research to improve the accuracy of target observation.

[0115] Practical application: Improve the clarity and details of drone aerial road images, and support more accurate target observation and analysis.

[0116] 4.2 System Optimization

[0117] Performance improvement: Based on feedback from actual applications, optimize the network model and processing algorithm to improve the system's processing efficiency and image quality.

[0118] User Feedback: Collect user feedback on image processing effects to further adjust and improve system settings.

[0119] Hardware configuration:

[0120] Processor: Responsible for model training and image processing, it must have high-performance computing capabilities and be suitable for deep learning tasks.

[0121] Memory: used to store training data sets, model weights, and processed images, requiring high speed and high reliability.

[0122] Graphics Processing Unit (GPU): Accelerates the training and reasoning process of deep learning models and improves computing efficiency.

[0123] Network interface: used for data transmission and model update, ensuring high speed and stability.

[0124] Display device: used to view processing results and image quality assessment results in real time.

[0125] On the same target drone aerial road image dataset, the ESR-KAN network model of the present invention is used to compare with the existing methods. The comparison results show that compared with the existing excellent super-resolution methods such as MASA-SR (Masa-sr: Matching acceleration and spatial adaptation for reference-based image super-resolution), RRSGAN (RRSGAN: Reference-based super-resolution for remote sensing image), and DATSR (Reference-based image super-resolution with deformable attention transformer), the super-resolution method implemented by the present invention has lower computational complexity and better performance. The comparison results of different network models used for super-resolution processing of drone aerial road images after training are shown in Table 1.

[0126] Table 1 Comparison of indicators of different network models

[0127]

[0128] The results shown in Table 1 show that in the super-resolution processing of drone aerial road images, the peak signal-to-noise ratio and structural similarity of the ESR-KAN network model are higher than those of the existing MASA-SR, RRSGAN and DATSR, which have relatively good performance. Compared with the above-mentioned similar super-resolution methods, the difference is significant, which proves that the reference image super-resolution processing technology of the present invention has superior application performance.

[0129] In order to more objectively demonstrate the superior performance of the model application of the present invention, the result images after super-resolution processing by the ESR-KAN network model are saved as sample data sets and applied to the semantic segmentation task of road defects. Using several typical network structures, the high-resolution images obtained by the present invention show more superior performance when used. The results are shown in Table 2, where mAP represents mean pixel accuracy, AP represents pixel accuracy, and mIoU represents mean intersection-over-union ratio.

[0130] Table 2 Performance of image processing results using ESR-KAN on different models

[0131]

[0132] Example 2

[0133] Another application example of the present invention is to use the efficient KAN reference image super-resolution ESR-KAN network model to improve the resolution of road images taken by drone groups to support more accurate road information system analysis and road environment monitoring. The workflow is as follows: Figure 2 The following are the specific implementation steps of this application example:

[0134] 1. Data collection and preparation stage

[0135] 1.1 Data Collection

[0136] Equipment: Obtain low-resolution images of the target road area from a swarm of drones and reference images from high-resolution aerial photography.

[0137] Operation: Ensure that the collected road surface images cover different states of the target area to obtain comprehensive surface feature information.

[0138] 1.2 Data processing

[0139] Dataset acquisition: The collected images are divided into training set and test set in a ratio of 7:3.

[0140] Data preprocessing: remove cloud and haze from images, and correct radiation and geometric distortion to ensure the quality of the dataset.

[0141] Data augmentation: Perform augmentation operations on the training set images, such as adjusting exposure and simulating different lighting conditions, to improve the adaptability of the model.

[0142] 2. Network model training phase

[0143] 2.1 Model construction

[0144] Network structure: Construct the ESR-KAN network model, including the KAN fusion module, KAN-channel attention and reconstruction module, which is specially optimized for the characteristics of UAV aerial road images.

[0145] 2.2 Model Training

[0146] Training settings: The network model is trained using a deep learning framework, using the Adam optimizer. The initial learning rate of Adam is set to 5e-5 and 1e-6, and the learning rate and other hyperparameters are adjusted as needed.

[0147] Loss function: Combined with the prediction loss function l pred and KAN spline sparse loss function l sparse , calculate the target loss function l total , optimize the model parameters to improve image quality.

[0148] 3. Super-resolution processing and analysis stage

[0149] 3.1 Model Deployment

[0150] Deployment: The trained ESR-KAN network model is integrated into the drone aerial road image processing process to process the actual drone swarm aerial road image data.

[0151] Processing: For the input low-resolution drone aerial road images, the ESR-KAN network model generates corresponding high-resolution images to enhance the recognizability of road surface features.

[0152] 3.2 Results Analysis

[0153] Quality assessment: Evaluate the accuracy and reliability of super-resolution images by comparing them with actual road survey data.

[0154] 4. Application and optimization stage

[0155] 4.1 Image Application

[0156] Environmental monitoring: Processed high-resolution images are used to monitor environmental indicators such as forest cover changes, urban expansion, and changes in agricultural land use.

[0157] Analysis: Provided to pavement information system analysts for more accurate pavement maintenance and pavement inspection.

[0158] 4.2 System Optimization

[0159] Performance improvement: Based on feedback from analysts and road environment monitoring teams, the network model and processing algorithms are optimized to improve the system's processing efficiency and image quality.

[0160] User Feedback: Collect user feedback on image processing effects to further adjust and improve system settings.

[0161] Hardware configuration:

[0162] Processor: High-performance computing capabilities are required to process large-scale drone-photographed road image datasets.

[0163] Memory: used to store training data sets, model weights, and processed images, requiring large capacity and high speed.

[0164] Graphics Processing Unit (GPU): Accelerates the training and reasoning process of deep learning models and improves computing efficiency.

[0165] Network interface: used for data transmission and model update, ensuring high speed and stability.

[0166] Display device: used for real-time viewing of processing results and image quality assessment, requiring high resolution and color accuracy.

[0167] Through these steps, the system of the present invention can effectively improve the resolution of the road surface images taken by drones, and provide more accurate image data support for road maintenance and road environment monitoring. In addition, the system allows for further optimization and expansion to adapt to future technological developments and new application requirements.

[0168] Compared with traditional methods, the ESR-KAN network model of the present invention not only achieves significant improvement in image quality in super-resolution processing of road images taken by drones, but also shows great potential in cost optimization and accuracy improvement of subsequent computer vision tasks.

[0169] First, in terms of cost optimization, traditional high-resolution cameras are often expensive, and multi-frame image synthesis technology requires a lot of shooting and post-processing time, which will greatly increase the cost of the project. In contrast, the ESR-KAN network model can reconstruct high-resolution images from low-resolution images without increasing hardware costs through deep learning technology, thereby recovering more detailed features. This method not only reduces the dependence on high-cost camera equipment, but also avoids the cumbersome operation and time consumption of multi-frame synthesis technology, achieving effective cost control. Secondly, in terms of improving the accuracy of subsequent computer vision tasks, the ESR-KAN network model can significantly improve the detail clarity and overall quality of the image through its innovative network structure. This is of great significance for subsequent computer vision tasks such as road defect detection, target tracking and analysis. High-resolution images can provide more detailed information, which significantly improves the accuracy and reliability of these tasks. For example, by using images processed by the ESR-KAN network model, targets on the road can be more accurately identified and tracked, improving the accuracy and robustness of target tracking. In addition, the high efficiency of the ESR-KAN network model means that it can process images in real time while the drone is in flight, which is particularly important for application scenarios that require rapid response. Real-time high-resolution image analysis can provide decision makers with instant and clear image data, allowing them to make decisions quickly and improve the efficiency of emergency response.

[0170] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A super-resolution processing method for road images taken by an unmanned aerial vehicle, characterized in that: The method includes the following: The road targets are photographed by drones to obtain road images; at the same time, the same targets are photographed by high-pixel cameras as reference images, and the reference images of all targets constitute a reference image set; Construct an efficient KAN reference image super-resolution ESR-KAN network model: The ESR-KAN network model includes a KAN fusion module, a reconstruction module, and M serial KAN-channel attention. The KAN fusion module includes a first layer composed of two KANs in parallel and a second layer composed of one KAN. The three KANs have the same structure. The two KANs in the first layer are used to analyze the input image X and the reference image X respectively. R Acquire feature map information; after the two KAN processing results of the first layer are spliced, they enter the KAN of the second layer to obtain the output feature map O0; Each KAN-channel attention consists of a serial structure consisting of a global average pooling GAP and two consecutive KANs, as well as a multiplication operation and residual connection to combine the inputs; in the KAN-channel attention, the input feature map O i Global average pooling GAP is applied to compress channel features. Two consecutive KANs are used to learn the importance weight of each channel. A new vector is generated from the importance weight of each channel. The new vector is used to transform the input feature map O i The channels are weighted to obtain the channel weight vector; the channel weight vector is combined with the input feature map O i Multiply, and then add to the input feature map O i The addition operation is performed to obtain the output O of the KAN-channel attention i+1 , and used as the input of the next KAN-channel attention, and so on to get the output O of the last KAN-channel attention M , i=0,1,…,M-1; The reconstruction module is used to restore the spatial shape of the image. The result processed by the reconstruction module is residually connected with the result of the reference image after upsampling to obtain the final high-resolution image Y.

2. The method according to claim 1, characterized in that: The reconstruction module includes a 3×3 convolutional layer, a ReLU activation function and a 3×3 convolutional layer connected in sequence.

3. The method according to claim 1, characterized in that M＝4-6。 4. The method according to claim 1, characterized in that The objective loss function of the ESR-KAN network model during training is l total For: total = l pred +l sparse Among them, l pred is the prediction loss function; l sparse is the KAN spline sparse loss function; λ is the parameter that controls the overall regularization amplitude, μ1 and μ2 are relative calculation parameters, both set to 1; φ l represents the activation function of the lth layer in the last KAN of the last KAN-channel attention, |φ l |1 represents the L1 norm of the activation function, S(φ l ) is the entropy regularization term, L is the number of KAN layers; X represents the original input image, and Y represents the high-resolution image predicted by the network structure.

5. The method according to claim 1, characterized in that The training process of the ESR-KAN network model is: Initialize the network parameter settings: the epoch of the training network is set to 500, the optimizer uses the Adam optimizer, and the initial learning rate of Adam is set to 5e-5~1e-6; The training set comes from road images obtained by drone aerial photography and reference images of the same target taken by a high-pixel camera. The input images of the training set and the reference images are input into the ESR-KAN network model, and these images are read according to the storage path. When the overall objective loss function l total When there is no longer a significant decrease (error ±1e-5), the network model training is considered to be stable and the network training process is completed; The test set images also come from road images obtained through drone aerial photography. When conducting network testing, the test set images are input, and no additional reference images are needed. The network weights trained with the training set are imported. When the peak signal-to-noise ratio PSNR and structural similarity SSIM of the drone aerial road images reach 44.89±0.03 and 0.991±0.02 respectively, the entire network training is completed.

6. A super-resolution processing system for road images taken by drones, characterized in that: The system comprises: UAVs are used to obtain road surface images of road targets through aerial photography; A high-pixel camera is used to obtain a high-resolution reference image of the target at a certain viewing angle; An image preprocessing module is used to perform data enhancement processing on road surface images and reference images; Database, used to store images and parameters; The ESR-KAN network model described in claims 1-5 is used for super-resolution reconstruction of road surface images.

7. The system according to claim 6, characterized in that The system also includes a network interface, a display device, and an image processing unit. The display device is used to view the processing results and image quality evaluation results in real time.

8. The system according to claim 6, characterized in that The system also includes a road defect semantic segmentation model, and the super-resolution image reconstructed by the ESR-KAN network model is used for road defect recognition.

9. The system according to claim 6, characterized in that The super-resolution images are used to monitor forest cover changes, urban expansion and agricultural land changes, road maintenance and road inspection.

Images