A maintenance abnormal event recognition method based on waterway UAV images

Abstract

This invention discloses a method for identifying maintenance anomalies based on unmanned aerial vehicle (UAV) images of waterways. First, images of waterway maintenance events are collected using UAVs, and the collected image data is standardized. Based on this, a preliminary screening is performed to identify all maintenance anomalies in a waterway segment. Finally, specific maintenance anomalies are further identified to distinguish detailed anomaly information, thus completing the identification of maintenance anomalies. This solution utilizes integrated aerospace information and artificial intelligence technology to achieve accurate identification of common river maintenance anomalies. It also performs further detection on certain specific anomalies. Building upon existing waterway information infrastructure resources, it provides a comprehensive, error-free, and high-precision intelligent maintenance detection method for waterway operation status monitoring, effectively promoting the transformation of waterway maintenance inspection business models, reducing labor costs, and improving the quality of the waterway traffic operation environment.

Description

Technical Field

[0001] The present invention belongs to the field of inland waterway inspection, and in particular relates to a method for identifying abnormal maintenance events based on waterway drone images. Background Art

[0002] Inland waterway inspection is an important means for traffic law enforcement departments to maintain navigation rights and improve the quality of navigation. Traditional inland waterway management usually adopts motorboat patrols, with each boat carrying at least 4 staff members. During patrols, technicians are sometimes unable to board the remediation buildings due to factors such as ship draft, underwater reefs, and inconvenience for docking, and can only take photos from a distance; even if they can board the remediation buildings, technicians are required to conduct on-site surveys separately. Each technician needs to take a large number of photos and video materials to record the survey of the remediation buildings on site, which often takes a lot of time. With the continuous dredging and expansion of inland waterways, the pressure of manual inspections has increased sharply. The method of observation by the naked eye is inefficient and labor-intensive, and problems cannot be discovered in a timely manner, especially for river sections with relatively poor geographical conditions, where effective management is difficult to implement. In addition, commonly used waterway inspection methods include navigation law enforcement vehicles and road patrols by law enforcement personnel. Because most waterways are formed by natural rivers, there are few roads along the route that can be used for patrols. Traditional waterway patrol methods have the disadvantages of insufficient coverage, low efficiency, inconvenient patrols due to restrictions on waters and bridges, hidden and difficult to detect illegal acts such as bank damage, incomplete archived materials, and high patrol costs. Therefore, the use of new-generation information technology, the integration of multiple detection methods to improve the level of intelligent waterway patrols, the study of air-ground integrated inland waterway intelligent patrol technology, and the improvement of patrol efficiency and reduction of patrol costs are issues that need to be studied and resolved at present.

[0003] At present, the waterway maintenance business model is still relatively traditional, with daily maintenance observations mainly based on regular manual inspections. Maintenance personnel visually observe the revetments, navigation marks and other waterway facilities within the waterway on a monthly basis to determine the damage to the facilities and develop maintenance plans. This traditional maintenance method is not timely and cannot effectively guarantee the navigation rate of the increasingly busy waterway transportation. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a method for identifying abnormal maintenance events based on waterway drone images, which can accurately detect abnormal maintenance events on the waterway, facilitate staff to formulate maintenance plans, and thus ensure the navigation rate of waterway transportation.

[0005] The specific technical solution for achieving the purpose of the present invention is:

[0006] A maintenance abnormal event recognition method based on waterway UAV images includes the following steps:

[0007] Step 1: Collect images of waterway maintenance events based on drones;

[0008] Step 2: standardize the collected image data;

[0009] Step 3: Perform preliminary screening on the standardized images to identify all maintenance abnormalities in the flight segment;

[0010] Step 4: Further identify specific abnormal maintenance events, distinguish detailed abnormal information, and complete the identification of abnormal maintenance events.

[0011] Compared with the prior art, the present invention has the following beneficial effects:

[0012] (1) This plan can promote the development and construction of smart waterways and improve the level of waterway intelligence: the application of aerospace information + artificial intelligence integrated inspection technology can significantly improve the level of intelligence in maintenance and inspection in the port and shipping industry. Based on the existing waterway information infrastructure resources, it provides a full-coverage, no-missing, high-precision intelligent maintenance and detection method for waterway operation status monitoring, effectively promoting the transformation of waterway maintenance and inspection business models, reducing labor costs, and improving the quality of the waterway traffic operation environment.

[0013] (2) This solution can effectively improve the utilization level of waterway information infrastructure: it can realize multi-scenario and complex applications of waterway drones, video surveillance and other sensing equipment, and deeply integrate business and technology. It is an important way to realize the complex application of waterway information infrastructure, an important method to improve the investment efficiency of waterway information infrastructure, and expand the investment efficiency of infrastructure.

[0014] The present invention is further described below in conjunction with specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flow chart of the maintenance abnormal event identification method based on waterway UAV images of the present invention.

[0016] Figure 2 This is a schematic diagram of the wavelet threshold denoising principle of the present invention.

[0017] Figure 3 It is a schematic diagram of the bank damage event identification process of the present invention.

[0018] Figure 4 It is a schematic diagram of the vegetation damage event identification process of the present invention.

[0019] Figure 5 This is a schematic diagram of the improved deeplabV3+ model structure of the present invention.

[0020] Figure 6 It is a schematic diagram of the riverside construction event identification process of the present invention. DETAILED DESCRIPTION

[0021] Example

[0022] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an kind" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present application. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to the actual proportional relationship. The technology, method and equipment known to those of ordinary skill in the relevant field may not be discussed in detail, but in appropriate cases, the technology, method and equipment should be considered as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so that once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0025] Combination Figure 1 , a maintenance abnormal event recognition method based on waterway UAV images, comprising the following steps:

[0026] Step 1: Collect images of waterway maintenance events based on drones;

[0027] Step 2: Standardize the collected image data, including image denoising, image adaptive segmentation and image expansion;

[0028] Among them, the image denoising is to perform wavelet denoising on the images collected by the drone, the image adaptive segmentation is to use DeepLabv3+ as the encoder module, and add a decoder module to perform semantic segmentation on the denoised image, and the image expansion is to use a generative adversarial neural network to expand the number of images.

[0029] More specifically, combined with Figure 2 , the process of wavelet denoising in standardization is:

[0030] For the input image to be denoised, determine whether the image is a grayscale image. If not, perform grayscale processing to convert it into a grayscale image;

[0031] Perform wavelet decomposition on the grayscale image and decompose it into wavelet coefficients of different scales;

[0032] Threshold processing is applied to the wavelet coefficients of each scale, and the coefficients smaller than a certain threshold are set to zero, while the coefficients larger than the threshold are retained;

[0033] The wavelet coefficients after threshold processing are inversely transformed to reconstruct the image after preliminary denoising. Subsequently, the high-frequency part of the reconstructed image (the horizontal, vertical and diagonal detail components obtained after wavelet decomposition) is thresholded again to further suppress the residual noise while retaining the detail information in the image as much as possible to obtain the final denoised image.

[0034] After decomposing the noisy wavelet coefficients, it is necessary to calculate the wavelet coefficient value that is infinitely close to the pure image wavelet coefficient. After the noisy image is processed by wavelet coefficient decomposition, the wavelet coefficient consists of two parts, in which the pure wavelet coefficient is abbreviated as k. i,j Indicates that the noise wavelet coefficient p i,j , the sum of the two coefficients is the wavelet coefficient x of the noisy image decomposition i,j , the processing of noisy images using the threshold value, that is, the threshold function, is to convert the wavelet coefficient y after the threshold value i,j , so that y i,j -k i,j ≈0. This can maximize the removal of noise and obtain a purer wavelet coefficient reconstructed image.

[0035] Among them, the coefficient decomposition formula is:

[0036]

[0037] The reconstruction formula is:

[0038]

[0039] Among them, f represents the input signal (or image) function, represents the target to be decomposed, represents the wavelet basis function, represents the scale function (also called the parent wavelet), j is the scale (decomposition level), k is the translation index, represents the inner product, which represents the input signal f and the wavelet basis function The projection coefficient of f, which represents the component of f at a specific scale and position; pk-2j Represents the filter coefficient, usually from the wavelet transform filter, such as the Daubechies filter coefficient. It is used to decompose the details or approximate information of the signal. -1 / 2 represents the scaling factor used to normalize the wavelet decomposition.

[0040] Wavelet threshold denoising is widely used in wavelet transform denoising methods. The average time consumption of this algorithm is very short in practical applications. The target matching denoising algorithm based on wavelet threshold denoising is not only used to detect this common noise caused by weather changes, but also has a good suppression effect on other uniform noise in the channel environment.

[0041] Combination Figure 3 , the process of image adaptive segmentation in standardization is:

[0042] DeepLabv3+ is used as the encoder module, and a decoder module is added to perform semantic segmentation on the denoised image. Depthwise separable convolution is applied to convolve the space of each channel separately. This method can greatly reduce the computational workload while ensuring the performance remains unchanged. The dilated convolution formula is shown in formula (3).

[0043] y[i]=∑ k x[i+r×k]w[k]#(3)

[0044] Where x is the input feature map, i represents the output position, w is the convolution kernel, k is the size of the convolution kernel, r is the convolution expansion rate, and r = 1 represents a special case of standard convolution.

[0045] In this embodiment, dilated convolution is used to extract features, and Xception is used and improved. Xception uses depthwise separable convolution to decompose the standard convolution into depthwise convolution and pointwise convolution (i.e., 1×1 convolution), which can greatly reduce the number of network parameters and computation. DeepLabv3+ uses a deeper Xception structure and uses depthwise separable convolution to replace all maximum pooling operations to extract feature maps for images of any resolution.

[0046] At the same time, a BN layer and a ReLU layer are added after the 3×3 deep convolution layer. At the same time, DeepLabv3+ proposes a simple and effective decoder module. In the original DeepLabv3, the predicted feature map is directly upsampled 16 times to the original size using the bilinear interpolation method. Such a simple decoder module cannot successfully restore the segmentation details;

[0047] DeepLabv3+ first passes the output of the encoder module through a 1×1 convolution to reduce the number of output channels, and then upsamples it four times using bilinear interpolation and combines it with the corresponding low-level features of the same resolution;

[0048] Finally, 3×3 convolution is used to refine the features, and bilinear interpolation is used to upsample four times to obtain the final prediction result.

[0049] In order to fuse multi-scale information, DeeplabV3+ introduces the Encoder-Decoder structure commonly used in semantic segmentation. In this architecture, the resolution of the encoder to extract features can be arbitrarily controlled, and through the dilated convolution, each convolution output contains a larger range of information without losing information, thus balancing the segmentation accuracy and time consumption.

[0050] The image expansion in the standardization processing is to solve the problem of insufficient sample data of ROI video of waterway revetment. It is proposed to generate sample data based on deep residual generative adversarial network technology, and use residual network and convolutional network to construct generation model and discriminant model respectively. The learning optimization strategy of GAN is trained by combining positive and negative sample fusion training of various ROI video data of waterway revetment, so as to enhance the robustness of adversarial network and alleviate the instability of adversarial network training and the occurrence of mode collapse.

[0051] The image expansion adopts a generative adversarial neural network to expand the number of images; the steps of generating images based on the deep residual generative adversarial network are as follows:

[0052] (1) Input randomly generated Gaussian noise into the generative model;

[0053] (2) After the noise is deconvolved through five neural network layers, the dimension of the input noise tensor is expanded from 100 to 512. Each subsequent deconvolution layer reduces the dimension by half, and the output dimension of the last layer is 3, corresponding to the three channels of the RGB image.

[0054] (3) Finally, the generated image is output.

[0055] The steps to generate an adversarial network based on deep residual to discriminate images are as follows:

[0056] (1) Input a 3-channel RGB image and extract image features through 5 convolutional layers;

[0057] (2) Each convolutional layer uses a different number of 4×4 convolution kernels to obtain tensors of different dimensions, and does not add a pooling layer, so that the convolutional layer can obtain a larger receptive field;

[0058] (3) The dimension change of the input tensor of each layer is opposite to that of the generative model. All layers use LeakyReLU as the activation function. Finally, the sigmoid function is used to convert the tensor into a scalar output with a size between (0, 1), which represents the probability that the image is a real image.

[0059] Step 3: Perform preliminary screening on the standardized images to identify all maintenance abnormalities in the flight segment;

[0060] Perform image registration on the standardized images and the images in the database, and make the points at the same position in space correspond to those in the images taken at different times and in different environments;

[0061] Abnormal event detection is then achieved through abnormal information comparison, and the abnormal events are classified according to the results of abnormal event detection to generate maintenance abnormal event identification and classification results.

[0062] In this embodiment, after the abnormal event recognition and classification results are generated, a list of abnormal events can be generated based on the abnormal event list. The maintenance workers confirm the abnormal events according to the list of abnormal events and finally generate a report.

[0063] Generally speaking, abnormal maintenance events in the field of waterway maintenance cover construction, revetment, signs and signs, ship violations, etc., usually including:

[0064] Abnormal events related to riverside construction: including riverside construction machinery, riverside construction site clearance, riverside construction soil and material piles, etc.;

[0065] Abnormal events related to river construction: including river construction piling ships, river construction dredging ships, etc.;

[0066] Abnormal events of concrete structures: including the fall-off of concrete capping or vertical capping, local holes in concrete, vertical local holes, etc.;

[0067] Abnormal events of revetment structures: including collapse of vertical revetment and slope revetment;

[0068] Abnormal events related to ships: including ships entering light beacon areas, fishing nets blocking rivers, abnormal ship congestion, illegal ship docking, etc.

[0069] Other types of abnormal events: including fallen trees occupying the road, fallen signs and signboards, blocked signs and signboards, and dumped construction waste.

[0070] The image registration is implemented by grayscale and template-based registration or image feature matching-based registration.

[0071] The abnormal information comparison is implemented by using a comparison method based on pixel comparison, a comparison method based on a histogram, a comparison method based on feature extraction and matching, or a comparison method based on a deep learning model.

[0072] The pixel comparison method is one of the simplest and most direct image comparison methods. It calculates the mean square error (MSE) or structural similarity index (SSIM) by comparing the RGB values ​​and grayscale values ​​of two images pixel by pixel to determine the abnormal information comparison result.

[0073] The histogram-based comparison method calculates the histograms of two images and compares the similarity of the histograms (such as chi-square distance, cosine similarity, etc.) to determine the abnormal information comparison result; histogram comparison is a comparison method based on image color distribution. In the histogram comparison method, a color histogram is usually used to represent the color distribution of the image. The color histogram counts the color value of each pixel in the image as the frequency of different color intervals, thereby obtaining a histogram describing the color distribution of the image. Commonly used color spaces include RGB, HSV, and Lab.

[0074] The feature extraction and matching-based comparison method uses an image processing library (such as OpenCV) to extract features of the image (such as SIFT, SURF, ORB, etc.), performs feature matching, and determines the abnormal information comparison result based on the result of feature matching; commonly used feature extraction models include VGG16, ResNet, Inception, etc. These models can extract high-level semantic information of the image, thereby more accurately measuring the similarity between images.

[0075] The deep learning model-based comparison method extracts features from images using a deep learning model (such as a convolutional neural network (CNN)), and determines the abnormal information comparison result by calculating the similarity of the features.

[0076] Step 4: Further identify specific abnormal maintenance events, distinguish detailed abnormal information, and complete the identification of abnormal maintenance events.

[0077] In this step, when further identifying the abnormal maintenance events initially screened, it is necessary to determine whether there are bank damage events, vegetation damage events, and riverside construction events;

[0078] Channel revetment damage is one of the important tasks of channel maintenance events. Severe revetment collapse will directly affect the navigation safety of the canal. This project intends to compare the advantages and disadvantages of traditional machine learning methods and deep learning algorithms, conduct in-depth analysis of the identified abnormal positions, and use a variety of technical information such as target recognition, combined with drone real-time positioning, image preprocessing, etc., and propose the most suitable algorithm for identifying revetment damage at this stage based on experimental comprehensive optimization.

[0079] Bank collapse takes various forms, including exposed rocks peeling off the bank breast wall / wave-breaking wall, large-scale damage and fracture, scattered surface layer, sliding, block fracture or scouring loss, etc. In addition, the colors of the damaged parts are different, and it is difficult to accurately obtain the specific characteristics of the bank collapse event through the feature recognition algorithm, which puts high demands on the target detection system. In this embodiment, in view of these characteristics, when judging whether there is a bank damage event, a recognition model is constructed based on the YOLO convolutional neural network, and the recognition model is trained using samples of the collected bank collapse photo data set. The data set of bank damage events includes a large number of bank damage events under different lighting conditions and weather conditions. Several images in the data set are reserved as a test set. The end-to-end characteristics of the YOLO convolutional neural network are used to use the model to detect the input image. If scattered targets are detected on the bank, it is determined that there is bank damage. Figure 4 As shown;

[0080] The bank collapse target recognition algorithm based on the YOLO convolutional neural network can achieve a bank collapse event recognition accuracy of no less than 85%. It supports algorithm self-learning and algorithm expansion, and the accuracy can gradually increase with the increase of sample size, and the missed detection rate does not exceed the technical indicator requirement of 15%.

[0081] Channel revetment vegetation plays an important role in maintaining channel safety and ecological environment. The roots of vegetation can penetrate deep into the soil, enhance soil stability, prevent soil erosion, prevent water flow from eroding the bank slope, and maintain the stability of the channel. At the same time, channel revetment vegetation can also absorb the mechanical noise emitted by ships and reduce the impact of noise on the surrounding environment.

[0082] In this embodiment, when judging whether there is a vegetation damage event, extraction is performed in the revetment vegetation area. On the basis of identifying vegetation damage events by pattern matching, in-depth research and analysis are conducted on the shape characteristics, color characteristics, distribution characteristics and time-varying characteristics of vegetation in complex environments. In addition, the image semantic segmentation technology based on deep learning is innovatively applied to construct an automatic detection model for revetment vegetation damage events. That is, a revetment state feature recognition model is constructed based on the deeplabV3+ semantic segmentation model. The GAN generative adversarial network in the deep learning algorithm is used to train real samples to obtain a large number of generated sample sets, and the revetment state feature recognition model is trained. The input image is detected based on the trained model to realize the recognition of vegetation damage events.

[0083] Combination Figure 5The revetment status feature recognition model in this embodiment is implemented based on the deeplabV3+ semantic segmentation model, that is, the CNN network is used to extract features for classification. In order to further improve the processing speed of multi-channel video streams, the deeplabV3+ backbone network is improved. The deeplabV3 backbone network is replaced with MobileNetV2, and the main body of its upsampling module Decoder is DCNN with dilated convolution. A commonly used classification network such as ResNet is used, and a spatial pyramid pooling module ASPP with dilated convolution is used;

[0084] In specific practice, the video image features of bank protection vegetation are generally manifested as follows: the image features of vegetation damage events include the region of interest and event features;

[0085] (1) Area of ​​interest: Damage to bank vegetation generally occurs when natural disasters such as floods, hurricanes, landslides, etc. cause serious damage to bank vegetation. Deforestation, inappropriate human activities, and pests and diseases cause large-scale damage to bank vegetation.

[0086] (2) Event characteristics: The color difference of the damaged part of the vegetation is very obvious (such as brown-yellow soil, white, black, and colorful garbage, etc.). This part is not within the green connected domain, resulting in a disconnection of the connected domain.

[0087] In the embodiment of the present invention, since there are few examples of damaged revetment vegetation, in order to ensure the number of model training sets, the GAN generative adversarial network in the deep learning algorithm is first used to train real samples to obtain a large number of generated sample sets, which can effectively make up for the lack of diversity in the data sample set. Among them, images with a size of 1440x1080 are used for model training, and the total number of positive and negative samples containing damaged revetment vegetation is not less than 3,000, and 20% of the images are reserved as test sets. Then the fitted sample set is used as the input of the DeepLabv3+ model, and the revetment features are extracted by the downsampling module Encoder. At the same time, the upsampling module Decoder is responsible for restoring the feature map size through interpolation, transposed convolution and other methods. The interrupted part of the connected domain is judged as a vegetation damage event. The detection scheme can achieve the technical indicator requirements of the recognition accuracy of vegetation damage events not less than 85% and the missed detection rate not exceeding 15%.

[0088] Riverside construction may cause serious damage to the revetment, so it is necessary to monitor temporary construction work in real time, automatically identify riverside construction work of waterway revetment, and complete the report, so that the management personnel can determine whether there is any illegal construction. The present invention adopts the target recognition method to identify ships sailing on the waterway, and if a construction ship is found, it is determined to be a riverside construction event.

[0089] In this solution, when judging whether there is a riverside construction event, the darknet framework is used to build a YOLO target detection model, and the trained YOLO target detection model is used to detect the input image to realize the identification of riverside construction events. Figure 6 As shown;

[0090] The darknet framework is used in the present invention to implement YOLO target detection. The YOLO neural network has 53 convolutional layers, as well as a shortcut layer, a routing layer, and a detection layer.

[0091] Direct connection layer: The current network layer in the neural network is integrated with a previous network layer, and then the result is input into the next network layer. This layer effectively solves the problem of gradient divergence and difficulty in training caused by the increase in depth in the neural network, ensuring the detection effect of the model in the deep network.

[0092] Routing layer: The routing layer is similar to the shortcut layer. This layer merges several previous network layers and essentially copies the operation of the network layer.

[0093] Adaboost is an iterative algorithm. Its core idea is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier). By analyzing the image features and using different features to complete the classifier, a binary classifier can be obtained. The classifiers are then cascaded using a screening method to form a cascade classifier to improve the accuracy of target detection. Among them, images with a size of 1440x1080 are used for model training. The total number of positive and negative samples containing typical construction is not less than 4,000, and 20% of the images are reserved as test sets.

[0094] The above-mentioned riverside construction event recognition adopts the riverside construction event recognition based on YOLO neural network, which can achieve an accuracy rate of riverside construction event recognition of not less than 80%, support algorithm self-learning and algorithm expansion, and the accuracy can gradually increase with the increase of sample size, and the missed detection rate does not exceed 15% of the technical requirements.

[0095] The solution of the present invention is based on the combination of drones and artificial intelligence. It provides a full-coverage, no-missing, high-precision intelligent maintenance and detection method for waterway operation status monitoring based on the existing waterway information infrastructure resources, which effectively promotes the transformation of waterway maintenance and inspection business models, reduces labor costs, improves the quality of waterway traffic operation environment, and can significantly improve the level of waterway intelligence.

[0096] In addition, the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0097] Step 1: Collect images of waterway maintenance events based on drones;

[0098] Step 2: standardize the collected image data;

[0099] Step 3: Perform preliminary screening on the standardized images to identify all maintenance abnormalities in the flight segment;

[0100] Step 4: Further identify specific abnormal maintenance events, distinguish detailed abnormal information, and complete the identification of abnormal maintenance events.

[0101] The present invention also provides a computer storable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0102] Step 1: Collect images of waterway maintenance events based on drones;

[0103] Step 2: standardize the collected image data;

[0104] Step 3: Perform preliminary screening on the standardized images to identify all maintenance abnormalities in the flight segment;

[0105] Step 4: Further identify specific abnormal maintenance events, distinguish detailed abnormal information, and complete the identification of abnormal maintenance events.

[0106] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A maintenance abnormal event recognition method based on waterway drone images, characterized in that: The following steps are involved: Step 1: Collect images of waterway maintenance events based on drones; Step 2: standardize the collected image data; Step 3: Perform preliminary screening on the standardized images to identify all maintenance abnormalities in the flight segment; Step 4: Further identify specific abnormal maintenance events, distinguish detailed abnormal information, and complete the identification of abnormal maintenance events.

2. The maintenance abnormal event recognition method based on waterway drone images according to claim 1 is characterized in that: The image standardization processing in step 2 includes image denoising, image adaptive segmentation and image expansion; Among them, the image denoising is to perform wavelet denoising on the images collected by the drone, the image adaptive segmentation is to use DeepLabv3+ as the encoder module, and add a decoder module to perform semantic segmentation on the denoised image, and the image expansion is to use a generative adversarial neural network to expand the number of images.

3. The maintenance abnormal event recognition method based on waterway drone images according to claim 2 is characterized in that: The process of performing wavelet denoising on the image collected by the drone is as follows: For the input image to be denoised, determine whether the image is a grayscale image. If not, perform grayscale processing to convert it into a grayscale image; Perform wavelet decomposition on the grayscale image and decompose it into wavelet coefficients of different scales; Threshold processing is applied to the wavelet coefficients of each scale, and the coefficients smaller than a certain threshold are set to zero, while the coefficients larger than the threshold are retained; The wavelet coefficients after threshold processing are inversely transformed to reconstruct the image after preliminary denoising. Subsequently, the high-frequency part of the reconstructed image is thresholded again to further suppress the residual noise while retaining the detail information in the image as much as possible to obtain the final denoised image.

4. The maintenance abnormal event recognition method based on waterway drone images according to claim 1 is characterized in that: The preliminary screening and maintenance of abnormal events in step 3 are specifically as follows: Perform image registration on the standardized images and the images in the database, and make the points at the same position in space correspond to those in the images taken at different times and in different environments; Abnormal event detection is then achieved through abnormal information comparison, and the abnormal events are classified according to the results of abnormal event detection to generate maintenance abnormal event identification and classification results.

5. The maintenance abnormal event recognition method based on waterway drone images according to claim 4 is characterized in that: The image registration is implemented by using a registration method based on grayscale and template or a registration method based on image feature matching. The abnormal information comparison is implemented by using a comparison method based on pixel comparison, a comparison method based on a histogram, a comparison method based on feature extraction and matching, or a comparison method based on a deep learning model.

6. The maintenance abnormal event recognition method based on waterway drone images according to claim 5 is characterized in that: The pixel comparison method calculates the mean square error value or the structural similarity index value by comparing the RGB values ​​and grayscale values ​​of the two images pixel by pixel to determine the abnormal information comparison result; The histogram-based comparison method calculates the histograms of the two images and compares the similarity of the histograms to determine the abnormal information comparison result; The comparison method based on feature extraction and matching uses an image processing library to extract features of the image, performs feature matching, and determines the abnormal information comparison result based on the result of the feature matching; The deep learning model-based comparison method extracts features from images using a deep learning model, and determines the abnormal information comparison result by calculating the similarity of the features.

7. The maintenance abnormal event recognition method based on waterway drone images according to claim 4 is characterized in that: The abnormal maintenance events include: Abnormal events related to riverside construction: including riverside construction machinery, riverside construction site clearance, and riverside construction soil and material piles; Abnormal events related to river construction: including river construction piling ships and river construction dredging ships; Abnormal events of concrete structures: including the fall-off of concrete capping or vertical capping, local holes in concrete, and vertical local holes; Abnormal events of revetment structures: including collapse of vertical revetment and collapse of slope revetment; Abnormal events related to ships: including ships entering light beacon areas, fishing nets blocking rivers, abnormal ship congestion, and ships docking illegally; Other types of abnormal events: including fallen trees occupying the road, fallen signs and signboards, blocked signs and signboards, and dumped construction waste.

8. The maintenance abnormal event recognition method based on waterway UAV images according to claim 1 is characterized in that: The specific abnormal maintenance event is further identified in step 4, specifically: Further identification is conducted among the maintenance abnormal events initially screened to determine whether there are bank damage events, vegetation damage events, and riverside construction events; When judging whether there is a revetment damage event, a recognition model is built based on the YOLO convolutional neural network, and the recognition model is trained using the collected revetment collapse photo dataset samples. The revetment damage event dataset includes a large number of revetment damage events under different lighting conditions and weather conditions. Several images in the dataset are reserved as a test set. The input image is detected using the model using the end-to-end characteristics of the YOLO convolutional neural network. If scattered targets are detected on the revetment, it is determined that there is revetment damage. When judging whether there is a vegetation damage event, a revetment status feature recognition model is built based on the deeplabV3+ semantic segmentation model. The GAN generative adversarial network in the deep learning algorithm is used to train real samples to obtain a large number of generated sample sets, and the revetment status feature recognition model is trained. The input image is detected based on the trained model to realize the recognition of vegetation damage events. When judging whether there is a riverside construction event, the darknet framework is used to build a YOLO target detection model, and the trained YOLO target detection model is used to detect the input image to realize the identification of riverside construction events.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer storable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images