Road manhole cover identification method based on thermal infrared image and deep learning

Abstract

The invention relates to the technical field of surveying and mapping science and remote sensing science, in particular to a road manhole cover recognition method based on a thermal infrared image and deep learning, and the method comprises the steps: obtaining a target thermal infrared image of a measurement region through the photogrammetry of an unmanned aerial vehicle, enhancing the features of the infrared image through a multiplexing adaptive histogram equalization algorithm, and obtaining a recognition result. A residual network structure is utilized to improve a YOLOv8 deep neural network to extract road manhole cover features in an image, and geometric parameters of a manhole cover are identified through edge extraction and circle fitting. According to the method, based on deep learning, high-precision identification and parameter extraction of the road manhole cover in the thermal infrared image are realized, and the method is of great significance to urban water supply and drainage information management and road safety risk identification.

Description

Technical Field

[0001] The present invention relates to the fields of surveying and mapping science and remote sensing science and technology, and in particular to a road manhole cover recognition method based on thermal infrared images and deep learning. Background Art

[0002] Traditional visible light cameras rely on natural or artificial light for imaging and monitoring, which makes it difficult to meet the all-weather information needs of smart cities. Thermal infrared imaging devices, on the other hand, rely on the infrared heat energy radiated by the object itself to produce clear images. Unlike color RGB optical cameras, thermal infrared imaging cameras can still work normally at night and in poor lighting conditions, and are not affected by natural or artificial light environments, achieving all-weather target detection. However, infrared thermal imaging cameras have problems such as low image quality and blurred target boundaries, which limits the accuracy of specific target recognition.

[0003] The maintenance of urban road water supply and drainage facilities is crucial to the long-term operation of the city, especially in the summer when flood disasters are more frequent. Road manhole covers are the basis of the water supply and drainage system, but traditional search and maintenance methods are inefficient, and visible light cameras are limited by natural light and obstructions, which makes daily inspection and maintenance difficult. Damaged and missing manhole covers not only pose a traffic safety hazard, but may also cause serious accidents and endanger the safety of pedestrians, cyclists and vehicles. At night or in bad weather conditions, these problems are more difficult to detect in time, increasing safety risks. Therefore, timely and effective manhole cover monitoring and maintenance are particularly important.

[0004] Manhole cover recognition technology has broad application prospects in intelligent assisted driving and unmanned driving, which can help vehicles perceive and avoid potential dangers in advance, improve driving safety and traffic efficiency. In the construction of smart cities, accurate identification and management of manhole covers can not only realize intelligent maintenance of infrastructure, but also reduce maintenance costs, improve the level of refined management, and provide residents with a safer and more convenient living environment.

[0005] To address the above problems, this application proposes a road manhole cover recognition method based on thermal infrared images and deep learning. Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings existing in the prior art and propose a road manhole cover recognition method based on thermal infrared images and deep learning. This method uses a drone equipped with an infrared camera to quickly obtain the infrared grayscale image of the road manhole cover in the target area, and quickly extracts and identifies it, which is of great significance to improving the maintenance efficiency of the water supply and drainage system and promoting the construction of smart cities.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A road manhole cover recognition method based on thermal infrared images and deep learning includes the following steps:

[0009] Step S1: Acquire an infrared image of the target in the survey area, and use a drone equipped with an infrared camera to complete the thermal infrared image acquisition of the survey area;

[0010] Step S2: Multiplex sample enhancement to produce a data set of infrared thermal imaging images of the survey area; the original infrared image is rotated, cropped and randomly reorganized into a new grayscale image copy A; the copy A is enhanced into copy B using the adaptive histogram equalization algorithm; the original infrared image is directly enhanced into copy C using the adaptive histogram equalization algorithm; the features of the training data set are enhanced; on the basis of the original infrared image and a series of generated copies, the road manhole covers are manually marked, and the image label coordinate samples are saved in .txt format;

[0011] Step S3: Construct a manhole cover detection algorithm model and introduce a residual network structure to improve YOLOv8; YOLOv8 contains a backbone network (Backbone), a neck (Neck) and a head (Head), in which the backbone network is responsible for extracting image features; the neck is responsible for fusing feature maps of different scales to improve detection accuracy; the head is responsible for predicting target detection and classification; a residual network is introduced into the backbone of YOLOv8, and the feature map parameters are transferred and optimized between the layers of the backbone network; it is suitable for learning the few grayscale features of small targets and enhancing feature learning capabilities;

[0012] Step S4: deep neural network model training and verification, using the deep neural network model to perform model training and verification on the infrared image data set;

[0013] Step S5: Use the trained deep neural network model to realize the recognition and extraction of manhole covers in the infrared thermal images of drones, and extract the manhole cover parameters; incrementally construct the data set; and improve the accuracy and generalization of the model.

[0014] Preferably, in step S1, the specific implementation steps of the drone obtaining the survey area image are:

[0015] Step S11: Use a drone equipped with infrared thermal imaging to take thermal infrared images of the target area to obtain basic data for implementing the following steps.

[0016] Preferably, in step S2, the specific implementation steps of preparing the measurement area thermal imaging image data set are:

[0017] Step S21: First, compare the thermal infrared image of the survey area taken by the drone with the color RGB image to confirm that the road manhole cover is not missing in the infrared thermal imaging image, and ensure the correctness of the infrared thermal image target recognition by the later model. Then, manually mark and outline the road manhole cover in the original infrared image, and save its sample label coordinate file in .txt text format to obtain the sample preliminary feature label;

[0018] Step S22: Since the sample annotation only occupies a very small part of the entire infrared image, a series of copies created by a multiplexed adaptive histogram equalization algorithm are introduced to enhance the infrared image, increase the diversity of samples, and improve the generalizability of the deep neural network model;

[0019] Step S23: Introduce a multiplexed adaptive histogram equalization algorithm to create a series of copies. The original infrared image is rotated, cropped, and randomly reorganized into a series of copies A. The adaptive histogram equalization algorithm is used to enhance the grayscale features of copy A to form a series of copies B. The original infrared image is directly enhanced with an adaptive histogram equalization algorithm to form a series of copies C. Based on the original infrared image and the series of copies, the road manhole covers are manually marked, and the image label coordinate samples are saved in .txt format to complete the data set production.

[0020] Preferably, in step S3, a manhole cover detection algorithm model is constructed, and a residual network structure is introduced to improve YOLOv8, and the specific implementation steps are:

[0021] Step S31: The data set is a grayscale image, which is different from the traditional color RGB image. The target learning feature is the road manhole cover facility, whose number of pixels accounts for a very small proportion of the entire image, and YOLOv8 recognition is poor. The residual network structure is introduced into its backbone network to improve the difficulty of grayscale target recognition of the model;

[0022] Step S32: Introduce a residual network structure, the model is YOLOv8s; construct a branch residual module between each convolution module and the connection feature map module in its backbone network;

[0023] Step S33: the residual network contains 4 residual modules, and the number of convolution kernels is 64, 128, 256, and 512 respectively; the residual network feature parameters are first increased in dimension and added to the feature pyramid pooling module with the backbone network feature parameters, and the maximum pooling operation is performed for 3 consecutive times to gradually downsample and then the feature map is spliced ​​to output the learning features;

[0024] Feature(Y)=BackboneF(Y)+ResidualF(Y)

[0025] Among them, Feature(Y) represents the output feature, BackboneF(Y) represents the backbone network feature, and ResidualF(Y) represents the residual network feature;

[0026] Step S34: Each residual module contains 2 convolutional layers and 1 SiLU function activation layer; the convolution kernel size of the convolutional layer is 3×3, and the step size is 1; the SiLU function activation layer is used to filter the convolution features between the two convolutional layers; the SiLU function is:

[0027] SiLU(X)=x×Sigmod(X)

[0028]

[0029] The SiLU (Sigmod Linear Unit) function is obtained by optimizing the standard Sigmod function by adding the linear function y=x. x is the convolution learning feature before the input residual module SiLU function. SiLU (X) is the output convolution learning feature after x participates in the SiLU function screening.

[0030] The SiLU function activates the output to the next convolution layer. After the feature parameters are extracted again, the feature map of the next backbone network convolution module is added to the next branch residual module.

[0031] Step S35: The initial residual module is used as the beginning of the residual network, and the feature map input is the 64-channel 160×160 feature map of the backbone network convolution module; the convolution module for establishing the branch by initially inputting the feature map is C1, the initial residual module is R1, and the feature maps of i and j are F by analogy. The residual network feature map relationship ResidualF(Y) satisfies:

[0032]

[0033]

[0034]

[0035] ReidualF is the residual network feature map, which includes F R 、F C Two parts; F R is the feature map provided by the residual module in the residual network, F R The subscript i represents the order number of residual modules in the residual network, for example, F R When the subscript i = 1, it represents the feature map provided by the first residual module in the residual network, and so on; F C is the feature map provided by the convolutional module participating in the residual network in the backbone network, F C The subscript j represents the number of convolutional modules in the backbone network that participate in the residual network, for example, F C When the subscript j = 1, it represents the feature map provided by the first convolutional module participating in the residual network in the backbone network. Similarly, since the initial residual module feature map in the residual network originates from the convolutional module in the backbone network, when the second residual module feature map is convolutionally screened, when the subscript i = 1 and j = 1 exists, F R =F c; There are 4 backbone network convolution modules and 4 residual network residual modules in the residual network. i and j are positive integers. ReidualF(Y) is F R 、F C The mathematical relationship between

[0036] The residual module has a convolution layer and a SiLU function activation layer, which are used to reduce the number of grayscale feature operation parameters and efficiently extract and transfer deep features. Each residual module outputs a feature map that is convolved and activated and then accumulated and transferred to the feature pyramid pooling module, which performs pooling extraction together with the backbone network feature map to correct the grayscale prediction target feature deviation.

[0037] The sample image feature map f in the residual module exists:

[0038] f=w×h′

[0039]

[0040] Where w is the image width, h is the image height, f is the number of convolution kernels, p is padding, and s is the convolution kernel step size; w' is the image width of feature map f', h' is the image height of feature map f'; f' is the feature map;

[0041] Step S36: Implement the algorithm based on the Pytorch framework, and introduce the above residual network structure into YOLOv8s to improve the network's ability to recognize manhole cover objects in infrared images.

[0042] Preferably, in step S4, the specific implementation steps of deep neural network model training and verification are:

[0043] Step S41: Use a deep neural network implemented based on the Pytorch framework to train the training data set, and set appropriate batch processing parameters, learning rate and number of processes;

[0044] Step S42: Adjust the network parameters according to the learning rate and recall rate curves, update the .pt network weight file, and evaluate the training results.

[0045] Preferably, in step S5, the trained deep neural network model is used to realize the recognition and extraction of manhole covers by infrared thermal imaging of drones; the specific implementation steps of incrementally constructing the data set are:

[0046] Step S51: After the training is completed, the training results are evaluated. In order to improve the generalization ability of the network, a road manhole cover grayscale feature dataset is incrementally constructed, and the pre-trained model is used to identify the grayscale features of the road manhole covers. After manual correction, the network is added to continue further training, and the trained model is used to extract the grayscale features of the road manhole covers in drone photography thermal imaging.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention utilizes a drone equipped with an infrared camera to quickly obtain infrared grayscale images of road manhole covers in the target area, and quickly extracts and identifies them, which is of great significance for improving the maintenance efficiency of water supply and drainage systems and promoting the construction of smart cities. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the present invention;

[0050] Figure 2 A block diagram of multiplexed sample enhancement in the present invention;

[0051] Figure 3 This is the improved Yolov8 residual network structure diagram in the present invention;

[0052] Figure 4 This is a sample diagram of the data set in the present invention;

[0053] Figure 5 The adaptive histogram equalization effect and a series of curve graphs in the present invention;

[0054] Figure 6 This is the prediction graph of the network model in the present invention. DETAILED DESCRIPTION

[0055] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the advantages and features of the present invention, thereby making a clearer definition of the protection scope of the present invention. The embodiments described in the present invention are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0056] like Figure 1 As shown, a road manhole cover recognition method based on thermal infrared images and deep learning includes the following steps:

[0057] Step S1: Acquire an infrared image of the target in the survey area, and use a drone equipped with an infrared camera to complete the thermal infrared image acquisition of the survey area;

[0058] Step S2: Multiplex sample enhancement to produce a data set of infrared thermal imaging images of the survey area; the original infrared image is rotated, cropped and randomly reorganized into a new grayscale image copy A; the copy A is enhanced into copy B using the adaptive histogram equalization algorithm; the original infrared image is directly enhanced into copy C using the adaptive histogram equalization algorithm; the features of the training data set are enhanced; on the basis of the original infrared image and a series of generated copies, the road manhole covers are manually marked, and the image label coordinate samples are saved in .txt format;

[0059] Step S3: Construct a manhole cover detection algorithm model and introduce a residual network structure to improve YOLOv8; YOLOv8 contains a backbone network (Backbone), a neck (Neck) and a head (Head), in which the backbone network is responsible for extracting image features; the neck is responsible for fusing feature maps of different scales to improve detection accuracy; the head is responsible for predicting target detection and classification; a residual network is introduced into the backbone of YOLOv8, and the feature map parameters are transferred and optimized between the layers of the backbone network; it is suitable for learning the few grayscale features of small targets and enhancing feature learning capabilities;

[0060] Step S4: deep neural network model training and verification, using the deep neural network model to perform model training and verification on the infrared image data set;

[0061] Step S5: Use the trained deep neural network model to realize the recognition and extraction of manhole covers in the infrared thermal images of drones, and extract the manhole cover parameters; incrementally construct the data set; and improve the accuracy and generalization of the model.

[0062] Specifically, in step S1, the specific implementation steps of obtaining the infrared image of the target in the measurement area are:

[0063] Step S11: Use a drone equipped with infrared thermal imaging (DJI Mavic 3 Thermal Enterprise) to shoot thermal infrared images of the target area to obtain an infrared grayscale image;

[0064] Step S12: The flight altitude of the drone survey route is 70 meters, and the pixel size of the infrared camera image is 640*512 to ensure the accuracy of the image ground resolution.

[0065] Specifically, in step S2, the specific implementation steps of multiplexing sample enhancement and preparing the thermal infrared image data set of the measurement area are as follows:

[0066] Step S21: manually label the acquired infrared thermal imaging of the measurement area to draw the coordinates of the road manhole cover label, save it in .txt text format, and obtain the preliminary feature label of the sample;

[0067] Step S22: cropping and rotating the grayscale images of the thermal imaging with the marked part and the unmarked part respectively, and saving the images in .jpg format with a pixel size of 640*640;

[0068] Step S23: introducing a multiplexed adaptive histogram equalization algorithm to create a series of enhanced copies, the original infrared image is rotated, cropped and randomly reorganized into a series of copies A;

[0069] Step S24: using an adaptive histogram equalization algorithm to enhance the grayscale features of copy A to obtain a series of copies B;

[0070] Step S25: the original infrared image is directly enhanced by an adaptive histogram equalization algorithm to obtain a series of copies C;

[0071] Step S26: Enhance the features of the training data set; manually mark the road manhole covers based on the original infrared image and the series of copies, and save the image label coordinate samples in .txt format;

[0072] Step S27: The images and labels in the data set are divided into a training set, a validation set, and a test set (7:2:1) according to an appropriate ratio of quantity. The file names in the training set, the validation set, and the test set are extracted and saved as text formats of train.txt, val.txt, and test.txt respectively; Figure 4 shown.

[0073] Specifically, in step S3, the manhole cover detection algorithm model is constructed, and the specific implementation steps of introducing the residual network structure to improve YOLOv8 are as follows:

[0074] Step S31: introduce the residual network structure, the model is YOLOv8s;

[0075] Step S32: The data set is a grayscale image, which is different from the traditional color RGB image. The learning feature target is the road manhole cover facility, whose number of pixels accounts for a very small proportion of the entire image, and YOLOv8 recognition is poor. The residual network structure is introduced into its backbone network to improve the difficulty of grayscale target recognition in the model.

[0076] Step S33: Introduce a residual network structure, the model is YOLOv8s; construct a branch residual module between each convolution module and the connection feature map module in its backbone network;

[0077] Step S34: The residual network contains 4 residual modules, and the number of convolution kernels is 64, 128, 256, and 512 respectively; the residual network feature parameters are first increased in dimension and added to the feature pyramid pooling module with the backbone network feature parameters, and the maximum pooling operation is performed for 3 consecutive times to gradually downsample and then the feature map is spliced ​​to output the learning feature map:

[0078] Feature(Y)=BackboneF(Y)+ResidualF(r)

[0079] Among them, Feature(Y) represents the output feature, BackboneF(Y) represents the backbone network feature, and Residualf(Y) represents the residual network feature.

[0080] Step S35: Each residual module contains 2 convolutional layers and 1 SiLU function activation layer; the convolution kernel size of the convolutional layer is 3×3, and the step size is 1; the SiLU function activation layer is used between the two convolutional layers to filter the convolution features; the SiLU function is:

[0081] SiLU(X)=x×Sigmod(X)

[0082]

[0083] The SiLU (Sigmod Linear Unit) function is obtained by optimizing the standard Sigmod function by adding the linear function y=x. x is the convolution learning feature before the input residual module SiLU function. SiLU (X) is the output convolution learning feature after x participates in the SiLU function screening.

[0084] The SiLU function activates the output to the next convolution layer. After the feature parameters are extracted again, the feature map of the next backbone network convolution module is added to the next branch residual module.

[0085] Step S36: The initial residual module is used as the beginning of the residual network, and the feature map input is the 64-channel 160×160 feature map of the backbone network convolution module; the convolution module for establishing the branch by initially inputting the feature map is C1, the initial residual module is R1, and the feature maps of i and j are F by analogy. The residual network feature map relationship ResidualF(Y) exists:

[0086]

[0087]

[0088]

[0089] ReidualF is the residual network feature map, which consists of two parts: FR and FC; FR is the feature map provided by the residual module in the residual network, and the FR subscript i represents the sequential number of residual modules in the residual network. For example, when the FR subscript i=1, it represents the feature map provided by the first residual module in the residual network, and so on; FC is the feature map provided by the convolution module participating in the residual network in the backbone network, and the FC subscript j represents the sequential number of convolution modules participating in the residual network in the backbone network. For example, when the FC subscript j=1, it represents the feature map provided by the first convolution module participating in the residual network in the backbone network, and so on; since the initial residual module feature map in the residual network originates from the convolution module in the backbone network, when the second residual module feature map is convolutionally screened, when the subscript i=1 and j=1 exist, FR=FC; there are 4 backbone network convolution modules and 4 residual network residual modules in the residual network, and ReidualF(Y) is the mathematical relationship between FR and FC;

[0090] The residual module has a convolution layer and a SiLU function activation layer, which are used to reduce the number of grayscale feature operation parameters and efficiently extract and transfer deep features. Each residual module outputs a feature map that is convolved and activated and then accumulated and transferred to the feature pyramid pooling module, which performs pooling extraction together with the backbone network feature map to correct the grayscale prediction target feature deviation.

[0091] The sample image feature map f in the residual module exists:

[0092] f=w×h′

[0093]

[0094] Where w is the image width, h is the image height, f is the number of convolution kernels, p is the padding, s is the convolution kernel step size, and w ’ is the feature map f ’ The image width, h ’ is the feature map f ’ The image height; f ’ is the feature map; the convolution structure parameters are as follows Figure 3 Show.

[0095] Step S37: Use the Pytorch framework to introduce the above residual network structure into YOLOv8s; Figure 3 shown.

[0096] Specifically, in step S4, the specific implementation steps of deep neural network model training and verification are:

[0097] Step S41: training the training data set using a deep neural network implemented based on Pytorch;

[0098] Step S42: Network initialization parameter settings: the number of object recognition classes number_class is 1; the initial learning rate lr is set to 0.01; the confidence conf is 0.5; the intersection over union ratio IoU is 0.7; the loss rate loss is 0.01; the input image size img_size is 640; the training batch batch_size is 4;

[0099] Step S43: Use GeForce RTX 3090 24GB to call and build a deep neural network based on the PyTorch framework (based on CUDA11.8 version) to perform feature training on the training set; process 4 images per batch and train 400 rounds;

[0100] Step S44: Check the learning rate and recall rate curves; update the .pt network weight file and evaluate the training results, such as Figure 5 shown.

[0101] Specifically, in step S5, the trained deep neural network model is used to realize the recognition and extraction of manhole covers by infrared thermal imaging of drones. The specific implementation steps of incrementally constructing the data set are:

[0102] Step S51: After the training is completed, the .pt file (Pytorch weight format) is updated in real time, and the trained model is used to extract grayscale features of road manhole covers from drone photography thermal imaging;

[0103] Step S52: Use the trained deep neural network to extract the line features of the drone photogrammetry, detect the test set and the original infrared image, and label it as a gray wireframe ManholeCover to represent the road manhole cover extraction map of the test set and the original infrared image, as shown in Figure 6 shown.

[0104] In summary, the present invention realizes high-precision recognition and parameter extraction of road manhole covers in thermal infrared images based on deep learning, which is of great significance for urban water supply and drainage information management and road safety risk identification.

[0105] The description and practice disclosed in the present invention are easy to think and understand for ordinary technicians in the technical field, and several improvements and modifications can be made without departing from the principles of the present invention. Therefore, modifications or improvements made without departing from the spirit of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A road manhole cover recognition method based on thermal infrared images and deep learning, characterized in that: The steps include: Step S1: Acquire an infrared image of the target in the survey area, and use a drone equipped with an infrared camera to complete the thermal infrared image acquisition of the survey area; Step S2: Multiplex sample enhancement to produce a data set of infrared thermal imaging images of the survey area; the original infrared image is rotated, cropped and randomly reorganized into a new grayscale image copy A; the copy A is enhanced into copy B using the adaptive histogram equalization algorithm; the original infrared image is directly enhanced into copy C using the adaptive histogram equalization algorithm; the features of the training data set are enhanced; on the basis of the original infrared image and a series of generated copies, the road manhole covers are manually marked, and the image label coordinate samples are saved in .txt format; Step S3: construct a manhole cover detection algorithm model and introduce a residual network structure to improve YOLOv8; YOLOv8 contains a backbone network, a neck, and a head, where the backbone network is responsible for extracting image features; The neck is responsible for fusing feature maps of different scales to improve detection accuracy; the head is responsible for detecting and classifying predicted targets; the residual network is introduced into the YOLOv8 backbone, and the feature map parameters are optimized and enhanced during transmission between the layers of the backbone network; Suitable for learning very few grayscale features of small targets and enhancing feature learning capabilities; Step S4: deep neural network model training and verification, using the deep neural network model to perform model training and verification on the infrared image data set; Step S5: Use the trained deep neural network model to realize the recognition and extraction of manhole covers in the infrared thermal images of drones, and extract the manhole cover parameters; incrementally construct the data set; and improve the accuracy and generalization of the model.

2. The road manhole cover recognition method based on thermal infrared image and deep learning according to claim 1 is characterized in that: In step S1, the specific implementation steps for the drone to obtain the survey area image are: Step S11: Use a drone equipped with infrared thermal imaging to take thermal infrared images of the target area to obtain basic data for implementing the following steps.

3. The road manhole cover recognition method based on thermal infrared image and deep learning according to claim 1 is characterized in that: In step S2, the specific implementation steps of making the thermal imaging image data set of the measurement area are: Step S21: First, compare the thermal infrared image of the survey area taken by the drone with the color RGB image to confirm that the road manhole cover is not missing in the infrared thermal imaging image, and ensure the correctness of the infrared thermal image target recognition by the later model. Then, manually mark and outline the road manhole cover in the original infrared image, and save its sample label coordinate file in .txt text format to obtain the sample preliminary feature label; Step S22: Since the sample annotation only occupies a part of the size of the entire infrared image, a series of copies created by a multiplexed adaptive histogram equalization algorithm are introduced to enhance the infrared image, increase the diversity of samples, and improve the generalizability of the deep neural network model; Step S23: The original infrared image is rotated, cropped and randomly reorganized into a series of copies A; the grayscale features of copy A are enhanced using an adaptive histogram equalization algorithm to form a series of copies B; the original infrared image is directly enhanced using an adaptive histogram equalization algorithm to form a series of copies C; based on the original infrared image and the series of copies, the road manhole covers are manually marked, and the image label coordinate samples are saved in .txt format to complete the data set production.

4. The road manhole cover recognition method based on thermal infrared image and deep learning according to claim 1 is characterized in that: In step S3, a manhole cover detection algorithm model is constructed, and the residual network structure is introduced to improve YOLOv8. The specific implementation steps are as follows: Step S31: The data set is a grayscale image, which is different from the traditional color RGB image. The learning feature target is the road manhole cover facility. The number of pixels of the manhole cover object accounts for a small proportion of the entire image. When YOLOv8 is used directly for target recognition, the recognition result is not ideal. The residual structure module is introduced into the backbone network of YOLOv8 to improve the problem of inaccurate grayscale target recognition of the model. Step S32: Introduce a residual network structure, the model is YOLOv8s; construct a branch residual module between each convolution module and the connection feature map module in its backbone network; Step S33: the residual network contains 4 residual modules, and the number of convolution kernels is 64, 128, 256, and 512 respectively; the residual network feature parameters are first increased in dimension and added to the feature pyramid pooling module with the backbone network feature parameters, and the maximum pooling operation is performed for 3 consecutive times to gradually downsample and then the feature map is spliced ​​to output the learning features; Feature(Y)=BackboneF(Y)+ResidualF(Y) Among them, Feature(Y) represents the output feature, BackboneF(Y) represents the backbone network feature, and ResidualF(Y) represents the residual network feature; Step S34: Each residual module contains 2 convolutional layers and 1 SiLU function activation layer; the convolution kernel size of the convolutional layer is 3×3, and the step size is 1; the SiLU function activation layer is used to filter the convolution features between the two convolutional layers; the SiLU function is: SiLU(X)=x×Sigmod(X) The SiLU function is obtained by optimizing the standard Sigmod function by adding the linear function y=x, where x is the convolutional learning feature map before the input residual module SiLU function, and SiLU(X) is the output convolutional learning feature map after x participates in the SiLU function screening; The SiLU function activates the output to the next convolution layer. After the feature parameters are extracted again, the feature map of the next backbone network convolution module is added to the next branch residual module. Step S35: The initial residual module is used as the beginning of the residual network, and the feature map input is the 64-channel 160×160 feature map of the backbone network convolution module; the convolution module for establishing the branch by initially inputting the feature map is C1, the initial residual module is R1, and so on for i and j; the feature map is F, and the residual network feature map relationship ResidualF(Y) satisfies: ReidualF is the residual network feature map, which includes F R 、F C Two parts; F R is the feature map provided by the residual module in the residual network, F R The subscript i represents the order number of residual modules in the residual network, for example, F R When the subscript i = 1, it represents the feature map provided by the first residual module in the residual network, and so on; F C is the feature map provided by the convolutional module participating in the residual network in the backbone network, F C The subscript j represents the number of convolutional modules in the backbone network that participate in the residual network, for example, F C When the subscript j = 1, it represents the feature map provided by the first convolutional module in the backbone network participating in the residual network, and so on; since the initial residual module feature map in the residual network originates from the convolutional module in the backbone network, when the second residual module feature map is convolutionally screened, when the subscript i = 1 and j = 1 exists, F R =F C ; There are 4 backbone network convolution modules and 4 residual network residual modules in the residual network. i and j are positive integers. ReidualF(Y) is F R 、F C The mathematical relationship between The residual module has a convolution layer and a SiLU function activation layer, which are used to reduce the number of grayscale feature operation parameters and efficiently extract and transfer deep features. Each residual module outputs a feature map, which is convolved and activated, and then accumulated and transferred to the feature pyramid pooling module, which performs pooling extraction together with the backbone network feature map to correct the grayscale prediction target feature deviation. The sample image feature map f' in the residual module exists: f′=w′×h′ Where w is the image width, h is the image height, f is the number of convolution kernels, p is the number of padded pixels, s is the convolution kernel step size, w' is the image width of feature map f', h' is the image height of feature map f'; f' is the feature map; Step S36: Implement the algorithm based on the Pytorch framework, and introduce the above residual network structure into YOLOv8s to improve the network's ability to recognize manhole cover objects in infrared images.

5. The road manhole cover recognition method based on thermal infrared image and deep learning according to claim 4 is characterized in that: In step S4, the specific implementation steps of deep neural network model training and verification are: Step S41: Use a deep neural network implemented based on the Pytorch framework to train the training data set, set batch processing parameters, learning rate and number of processes; Step S42: Adjust the network parameters according to the learning rate and recall rate curves, update the .pt network weight file, and evaluate the training results.

6. The road manhole cover recognition method based on thermal infrared image and deep learning according to claim 1 is characterized in that: In step S5, the trained deep neural network model is used to realize the recognition and extraction of manhole covers in the infrared thermal imaging of drones; the specific implementation steps of incrementally constructing the data set are: Step S51: After the training is completed, the training results are evaluated. In order to improve the generalization ability of the network, a road manhole cover grayscale feature dataset is incrementally constructed, and the pre-trained model is used to identify the grayscale features of the road manhole covers. After manual correction, the network is added to continue further training, and the trained model is used to extract the grayscale features of the road manhole covers in drone photography thermal imaging.

Images