Channel foreign matter image generation and channel foreign matter detection method based on generative adversarial network

Abstract

The invention relates to a channel foreign matter image generation and channel foreign matter detection method based on a generative adversarial network, and the method comprises the steps: image collection: employing an unmanned plane for inspection, and collecting a channel image; image noise reduction: carrying out noise reduction processing on the acquired image by adopting a median filtering technology; model improvement: based on the GAN model, introducing an SE attention mechanism, and improving a loss function to obtain an improved GAN model; model training: training the improved GAN model through a training data set; image generation: generating a channel foreign matter image by using the trained foreign matter image generation model, and forming an extended sample set; foreign matter recognition: training a target detection network in combination with the original sample set and the extended sample set, and detecting and recognizing the channel foreign matter based on the trained target detection network; and foreign matter positioning: according to an identification result, based on a GIS system, determining a spatial position of a foreign matter. The method is beneficial to generating images similar to real channel foreign matter features, and the channel foreign matter detection precision is improved.

Description

Technical Field

[0001] This invention relates to the field of foreign object monitoring in waterways, specifically to a method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks. Background Technology

[0002] Waterways serve multiple functions, including flood control, water diversion, navigation, and environmental improvement, and are among the most widely distributed and directly beneficial water conservancy projects in my country. However, with socio-economic development and urban expansion, the exploitation of waterway shorelines has increased significantly. Unauthorized construction, dumping, and occupation of waterways have become rampant, leading to encroachment on waterways and a continuous narrowing of riverbanks. Furthermore, the discharge of large amounts of wastewater and the proliferation of floating aquatic plants obscuring the water surface have significantly weakened the river's ecological function, posing incalculable risks to the ecological environment and navigation. Therefore, strengthening the supervision and protection of waterways, especially the removal of unauthorized construction, dumping, and other illegal activities and debris, is urgently needed.

[0003] Deep learning object detection algorithms have been applied in the field of foreign object identification in waterways. By learning from a large number of samples, they can achieve high-precision detection and identification. However, there are currently few publicly available datasets of foreign objects in waterways, and manually constructing datasets through on-site photography is time-consuming and costly. Therefore, deep learning object detection algorithms still face the problem of insufficient sample sets in the field of foreign object identification in waterways. Summary of the Invention

[0004] The purpose of this invention is to provide a method for generating and detecting foreign objects in waterways based on generative adversarial networks. This method is beneficial for generating images that are similar to the features of real foreign objects in waterways, thereby improving the accuracy of foreign object detection in waterways.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks, comprising:

[0006] Image acquisition: Using drones for inspection to collect images of the waterway;

[0007] Image noise reduction: Median filtering is used to reduce noise in the acquired images;

[0008] Model Improvement: Based on the GAN model, the SE attention mechanism is introduced, and the loss function of the model is improved to obtain the improved GAN model;

[0009] Model training: The improved GAN model is trained using the training dataset to obtain a trained foreign object image generation model;

[0010] Image generation: Use the trained foreign object image generation model to generate images of foreign objects in the waterway, forming an expanded sample set;

[0011] Foreign object identification: The target detection network is trained by combining the original sample set and the extended sample set, and the foreign objects in the waterway are detected and identified based on the trained target detection network;

[0012] Foreign object location: Based on the identification results, the spatial location of the foreign object is determined using a GIS system.

[0013] Furthermore, the method for implementing the image acquisition is as follows:

[0014] First, the power system, communication link, and optical payload status of the UAV are checked. Then, the flight path and shooting angle of the UAV are set. The UAV is then made to patrol the target channel and both banks at different time periods and at different flight altitudes according to the set flight path, and take images of the channel during the patrol process.

[0015] Furthermore, the method for implementing image noise reduction is as follows:

[0016] For a pixel in an image, the pixel values ​​in the neighborhood around the pixel are sorted with the pixel as the center, and the pixel value in the middle of the sort is taken as the pixel value of the current pixel.

[0017] The above operations are performed on all pixels in the image to achieve noise reduction.

[0018] Furthermore, the method for implementing the model improvement is as follows:

[0019] The GAN model is used as the basic model for generating foreign object images;

[0020] By introducing the SE attention mechanism, the SE layer is added to the generator and discriminator of the GAN model. Image features are fused in a weighted manner through the SE layer, thereby improving network performance at a relatively low cost.

[0021] A composite loss function is formed by using the mean absolute value error function, the mean square error function, and the CGAN loss function, and is used as the loss function of the model.

[0022] Furthermore, the SE layer includes the following processing steps:

[0023] Compression: Dimensionality reduction is performed on each feature map using global average pooling, for the input tensor X∈R. H×W×C Each channel is compressed to obtain a 1×1×C tensor z, that is:

[0024] Where H represents the height of the image, W represents the width of the image, C represents the depth of the image, and X(i,j) represents the element in the i-th row and j-th column of the input tensor;

[0025] This allows us to obtain the compression information for each channel and generate the corresponding channel descriptor.

[0026] Activation: The C-dimensional vector z obtained in the previous step is input into a fully connected layer, compressing its dimension to C / r, where r is a set hyperparameter. Then, the ReLU function is used to activate z, followed by another fully connected layer to restore the dimension of z to C. Finally, the Sigmoid function is used to activate z, and z is multiplied by the input tensor X to obtain the feature U'. The SE layer uses a fully connected layer and activation function to reduce the computational cost of the model and enable the model to learn the weight coefficients of different channels. The activation process is shown in the following formula:

[0027] U'=σ([W1·Vec(z)] + )

[0028] Where Vec(·) represents transforming a tensor into a column vector, [·] + σ(·) represents the ReLU function activation operation, σ(·) represents the Sigmoid function activation operation, and W1 is the weight of the fully connected layer.

[0029] Through the above steps, the SE layer can adaptively learn the importance of each channel and assign different weights to different channels, thereby improving the model's ability to extract and express feature maps.

[0030] Furthermore, the composite loss function combines the mean absolute value error function, the mean square error function, and the CGAN loss function, wherein the mean absolute value error function L L1 As shown in formula (1), the mean square error function L L2 As shown in formula (2), the CGAN loss function L CGAN As shown in formula (3), the loss function L, which combines the mean absolute value error function and the mean square error function, is shown in formula (4). The composite loss function G*, which is formed by combining the loss function L and the CGAN loss function, is shown in formula (5).

[0031] L L1 =Ε c,x (xG(c)1) (1)

[0032] L L2 =Ε c,x ((xG(c) 2 (2)

[0033] L CGAN =Ε c,x (logD(c,x))+Ε c (log(1-D(c,G(c)))) (3)

[0034] G *=argmin G max D L CGAN (G,D)+λL(G) (4)

[0035]

[0036] Where c is the input random noise, x is the input image, D(·) represents the output of the discriminator D, G(·) represents the output of the generator G, and E c (·) represents the mathematical expectation related to the input noise after training, E c,x (·) represents the mathematical expectation related to the input noise and the input image after training, and λ is the gradient penalty coefficient.

[0037] Furthermore, the method for implementing the model training is as follows:

[0038] Set the training parameters for the improved GAN model as follows: number of iterations epoch = 50000, batch_size = 2, number of CPU threads used during data loading num_woker = 0, gradient penalty coefficient λ = 10;

[0039] The improved GAN model is trained using the training dataset to obtain a well-trained foreign object image generation model.

[0040] Furthermore, the method for generating the image is as follows:

[0041] The image to be detected is input into the trained foreign object image generation model to generate an expanded sample set.

[0042] The foreign object image generation model mainly consists of two parts: a generator G and a discriminator D. The generator receives a random noise c and is then trained to fit the distribution of the original sample, making it increasingly closer to the real sample. The discriminator is used to distinguish the difference between the generated sample and the real sample. The two compete against each other and continuously optimize their respective network parameters, eventually making it difficult for the discriminator to determine whether the received sample comes from the real sample or the generated sample, that is, reaching Nash equilibrium. At this point, it can be considered that the generator has learned the data distribution of the original image.

[0043] Furthermore, the method for implementing foreign object identification is as follows:

[0044] Acquire images of foreign objects in the waterway to be detected, the images of foreign objects in the waterway including an original sample set and an extended sample set;

[0045] Foreign objects in the waterway in the image are divided into five categories: random piles, illegal constructions, illegal occupation, sewage outlets, and water hyacinths. Labeling software is used to label the five categories of target foreign objects in the noise-reduced image.

[0046] The image of foreign objects in the waterway is input into a pre-trained target detection network to obtain the target detection result of the image of foreign objects in the waterway.

[0047] The target detection network is trained using an original sample set and an extended sample set with labeled information. The extended sample set is obtained by combining the target slice images from the original sample set with the channel foreign object images generated by the foreign object image generation model. Each channel foreign object image in the extended sample set is obtained by inputting different random noises into a pre-trained foreign object image generation model based on a generative adversarial network.

[0048] Furthermore, the method for locating the foreign object is as follows:

[0049] Extract Pos data from the image, i.e., the GIS coordinate information of the exterior orientation elements in UAV oblique photogrammetry. By constructing the relationship between latitude / longitude, Earth ellipsoid parameters, and the first eccentricity parameter, convert the degree-minute-second format to a degree-format latitude / longitude coordinate system, and then convert it to a geodetic coordinate system. The geodetic coordinate system describes the spatial location of a point using latitude B and longitude L. The specific formula is as follows:

[0050]

[0051] Where X is the horizontal rectangular coordinate, Y is the vertical rectangular coordinate, B is the latitude of the GIS coordinate of the foreign object, L is the longitude of the GIS coordinate of the foreign object, L0 is the longitude of the coordinate origin, L0 = 0, e is the first eccentricity, and K is the Earth's oblateness.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) The method for generating images of foreign objects in waterways proposed in this invention can be repeatedly used to generate images of foreign objects in waterways by constructing a generative adversarial network once. It is convenient to operate and has good feasibility.

[0054] (2) This invention proposes a target detection method for foreign object images in waterways under limited training sample conditions. This method eliminates the need for expensive data acquisition and allows for easy expansion of training samples using image processing and other methods. By utilizing both the original and expanded sample sets for training, the training accuracy of the target detection network can be improved, avoiding overfitting due to insufficient training samples. This enhances the target detection accuracy of the trained network for foreign object images in unknown waterways. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the method implementation of an embodiment of the present invention;

[0056] Figure 2This is a schematic diagram illustrating the working principle of the GAN model in this embodiment of the invention;

[0057] Figure 3 This is a network structure diagram of the improved GAN model generator in this embodiment of the invention;

[0058] Figure 4 This is a network structure diagram of the improved GAN model discriminator in this embodiment of the invention. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0060] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0061] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0062] like Figure 1 As shown, this embodiment provides a method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks, including:

[0063] S1. Image Acquisition: Use drones for inspection and collect images of the waterway.

[0064] In this embodiment, the image acquisition method is as follows:

[0065] First, the power system, communication link, and optical payload status of the UAV are checked. Then, the flight path and shooting angle of the UAV are set. The UAV is then made to patrol the target channel and both banks at different time periods and at different flight altitudes according to the set flight path, and take images of the channel during the patrol process.

[0066] The filming targets include five categories: three types of waterway irregularities that affect the function and ecology of the waterway (illegal dumping, illegal construction, and illegal occupation) and two common types of waterway debris (illegal sewage outlets and water hyacinths on the river surface).

[0067] S2. Image Denoising: Median filtering technology is used to denoise the acquired images.

[0068] In this embodiment, the image noise reduction method is as follows:

[0069] For a pixel in an image, the pixel values ​​in the surrounding neighborhood of the pixel are sorted with the pixel as the center, and the pixel value in the middle of the sorted position is taken as the pixel value of the current pixel.

[0070] Perform the same operation on all pixels in the image to achieve noise reduction.

[0071] In this embodiment, the neighborhood of the median filter is set to a size of 7*7 to process the waterway image.

[0072] S3. Model Improvement: Based on the GAN model, the SE attention mechanism is introduced, and the loss function of the model is improved to obtain the improved GAN model.

[0073] In this embodiment, the method for implementing the model improvement is as follows:

[0074] A Generative Adversarial Network (GAN) model is used as the basic model for generating foreign object images. GAN is a generative model mainly composed of a generator G and a discriminator D. Its working principle is as follows: Figure 2 As shown, the generator receives random noise c and is trained to fit the distribution of the original samples, making it increasingly closer to the real samples; the discriminator is used to distinguish the differences between generated samples and real samples. Ideally, the two compete against each other and continuously optimize their respective network parameters, eventually making it difficult for the discriminator to determine whether the received samples come from real samples or generated samples, i.e., reaching Nash equilibrium. At this point, it can be considered that the generator has learned the data distribution of the original image.

[0075] In this embodiment, the GAN model is improved, specifically including the following:

[0076] 1. Introducing the Squeeze-and-Excitation (SE) attention mechanism: Specifically, an SE layer is added to the GAN model network. Image features are fused in a weighted manner through the SE layer, improving network performance at a relatively low cost. The SE layer includes the following processing steps:

[0077] 1) Squeeze compression: Dimensionality reduction is performed on each feature map using global average pooling, for the input tensor X∈R. H×W×C Each channel is compressed to obtain a 1×1×C tensor z, that is:

[0078] Where H represents the height of the image, W represents the width of the image, C represents the depth of the image, and X(i,j) represents the element in the i-th row and j-th column of the input tensor.

[0079] This allows us to obtain the compression information for each channel and generate the corresponding channel descriptor.

[0080] 2) Excitation (Excitation): The C-dimensional vector z obtained in the previous step is first input into a fully connected layer to compress its dimension to C / r, where r is a set hyperparameter. Then, the ReLU function is used to activate z, followed by another fully connected layer to restore the dimension of z to C. Finally, the Sigmoid function is used to activate z, and z is multiplied by the input tensor X to obtain the feature U'. The SE layer uses a fully connected layer and activation function, which reduces the computational cost of the model and allows the model to learn the weight coefficients of different channels. The excitation process is shown in the following formula:

[0081] U'=σ([W1·Vec(z)] + )

[0082] Where Vec(·) represents transforming a tensor into a column vector, [·] + σ(·) represents the ReLU function activation operation, σ(·) represents the Sigmoid function activation operation, and W1 is the weight of the fully connected layer.

[0083] Through the above steps, the SE attention layer can adaptively learn the importance of each channel and assign different weights to different channels, thereby improving the model's ability to extract and express feature maps.

[0084] 2. Improve the loss function, specifically by abandoning the cross-entropy loss function in the original GAN ​​model and instead using a composite loss function formed by the mean absolute value error function, the mean squared error function, and the CGAN loss function as the model's loss function. Among these, the mean absolute value error function L... L1 As shown in formula (1), the mean square error function L L2 As shown in formula (2), the CGAN loss function L CGAN As shown in formula (3), the loss function L, which combines the mean absolute value error function and the mean square error function, is shown in formula (4). The composite loss function G*, which is formed by combining the loss function L and the CGAN loss function, is shown in formula (5).

[0085] L L1 =Ε c,x (xG(c)1) (1)

[0086] L L2 =Ε c,x ((xG(c) 2 (2)

[0087] L CGAN =Ε c,x (logD(c,x))+Ε c (log(1-D(c,G(c)))) (3)

[0088] G * =argmin G max D L CGAN (G,D)+λL(G) (4)

[0089]

[0090] Where c is the input random noise, x is the input image, D(·) represents the output of the discriminator D, G(·) represents the output of the generator G, and E c (·) represents the mathematical expectation related to the input noise after training, E c,x (·) represents the mathematical expectation related to the input noise and the input image after training, and λ is the gradient penalty coefficient.

[0091] Composite loss functions are beneficial for reducing gradient changes in the early stages of model training, maintaining training stability, and accelerating convergence in the later stages of training.

[0092] The generator network structure and discriminator network structure in the improved GAN model are as follows: Figure 3 , Figure 4 As shown.

[0093] Among them, the Dense layer is a fully connected layer used to transform the dimension of the input data; the Deconv layer is a transposed convolutional layer used to perform transposed convolution operations on the input data; the Conv layer is a convolutional layer used to perform convolution operations on the input data; the BN layer is a batch normalization layer used to regularize the input data; and GLU, Tanh, and ReLU are activation functions.

[0094] S4. Model Training: The improved GAN model is trained using the training dataset to obtain a trained foreign object image generation model.

[0095] In this embodiment, the model training is implemented as follows:

[0096] Set the training parameters for the improved GAN model as follows: number of iterations epoch = 50000, batch_size = 2, number of CPU threads used during data loading num_woker = 0, and gradient penalty coefficient λ = 10.

[0097] The improved GAN model is trained using the training dataset to obtain a well-trained foreign object image generation model.

[0098] S5. Image Generation: Generate images of foreign objects in the waterway using a trained foreign object image generation model to form an expanded sample set.

[0099] In this embodiment, the image generation method is as follows:

[0100] The image to be detected is input into the trained foreign object image generation model to generate an expanded sample set.

[0101] The foreign object image generation model mainly consists of two parts: a generator G and a discriminator D. The generator receives a random noise c and is then trained to fit the distribution of the original sample, making it increasingly closer to the real sample. The discriminator is used to distinguish the differences between the generated sample and the real sample. Ideally, the two compete against each other and continuously optimize their respective network parameters, eventually making it difficult for the discriminator to determine whether the received sample comes from the real sample or the generated sample, that is, reaching Nash equilibrium. At this point, it can be considered that the generator has learned the data distribution of the original image.

[0102] S6. Foreign Object Identification: The target detection network is trained by combining the original sample set and the extended sample set, and the foreign objects in the waterway are detected and identified based on the trained target detection network.

[0103] In this embodiment, the method for foreign object identification is as follows:

[0104] Acquire images of foreign objects in the waterway to be detected, the images of foreign objects in the waterway including an original sample set and an extended sample set.

[0105] Foreign objects in the waterway in the image are divided into five categories: haphazardly piled up, haphazardly constructed, haphazardly occupied, sewage outlets, and water hyacinths. Labeling software is used to label the five types of target foreign objects in the noise-reduced image.

[0106] The image of the foreign object in the waterway is input into a pre-trained target detection network to obtain the target detection result of the image of the foreign object in the waterway.

[0107] The main task of object detection is to find regions of interest from input scene images and determine the object category. Embodiments of this invention can employ any existing neural network for object detection. For example, the object detection network may include: R-CNN, Faster R-CNN, SSD, YOLO, AlexNet, VGGNet, ResNet, MobileNet, etc. For a detailed understanding of the specific detection process of the object detection network, please refer to relevant prior art; it will not be described in detail here.

[0108] The target detection results of the foreign object image in the navigation channel can include the category of each target in the image and the location of the target in the image. The target detection results of the foreign object image in the navigation channel can be output as separate text, such as outputting a single string representing the target detection result. Alternatively, a common approach is to directly mark the target detection results on the foreign object image in the navigation channel. For example, the location of the target in the foreign object image can be marked with a rectangle containing the target, and the pixel coordinates of one vertex of the rectangle, as well as the width and height of the rectangle, can be indicated next to the rectangle. In a simpler approach, only the pixel coordinates of one vertex of the rectangle can be indicated.

[0109] The target category can be distinguished and identified using different characters. In one optional implementation, a confidence level can be added to identify the category. The confidence level represents the probability that the target belongs to a category, and its value ranges from 0 to 1. A higher confidence level indicates a greater likelihood that the target belongs to that category. Alternatively, in a single-category target recognition scenario, such as determining whether a target in a waterway foreign object image is a water hyacinth, the category can be represented by "water hyacinth" and its corresponding confidence level. The level of confidence determines whether the target is a water hyacinth.

[0110] The target detection network is trained using an original sample set and an extended sample set with labeled information. The extended sample set is obtained by combining the target slice images from the original sample set with the channel foreign object images generated by the foreign object image generation model. Each channel foreign object image in the extended sample set is obtained by inputting different random noises into a pre-trained foreign object image generation model based on a generative adversarial network.

[0111] S7. Foreign Object Location: Based on the identification results and using a GIS system, determine the spatial location of the foreign object.

[0112] In this embodiment, the method for locating the foreign object is as follows:

[0113] Extract Pos data from the image, i.e., the GIS coordinate information of the exterior orientation elements in UAV oblique photogrammetry. By constructing the relationship between latitude / longitude, Earth ellipsoid parameters, and the first eccentricity parameter, convert the degree-minute-second format to a degree-format latitude / longitude coordinate system, and then convert it to a geodetic coordinate system. The geodetic coordinate system describes the spatial location of a point using latitude B and longitude L. The specific formula is as follows:

[0114]

[0115] Where X is the horizontal rectangular coordinate, Y is the vertical rectangular coordinate, B is the latitude of the GIS coordinate of the foreign object, L is the longitude of the GIS coordinate of the foreign object, L0 is the longitude of the coordinate origin, L0 = 0, e is the first eccentricity, and K is the Earth's oblateness.

[0116] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0118] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0119] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0120] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks, characterized in that, include: Image acquisition: Using drones for inspection to collect images of the waterway; Image noise reduction: Median filtering is used to reduce noise in the acquired images; Model Improvement: Based on the GAN model, the SE attention mechanism is introduced, and the loss function of the model is improved to obtain the improved GAN model; Model training: The improved GAN model is trained using the training dataset to obtain a trained foreign object image generation model; Image generation: Use the trained foreign object image generation model to generate images of foreign objects in the waterway, forming an expanded sample set; Foreign object identification: The target detection network is trained by combining the original sample set and the extended sample set, and the foreign objects in the waterway are detected and identified based on the trained target detection network; Foreign object location: Based on the identification results, the spatial location of the foreign object is determined using a GIS system.

2. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 1, characterized in that, The method for image acquisition is as follows: First, the power system, communication link, and optical payload status of the UAV are checked. Then, the flight path and shooting angle of the UAV are set. The UAV is then made to patrol the target channel and both banks at different time periods and at different flight altitudes according to the set flight path, and take images of the channel during the patrol process.

3. The method for generating and detecting foreign objects in waterways based on generative adversarial networks according to claim 1, characterized in that, The method for image noise reduction is as follows: For a pixel in an image, the pixel values ​​in the neighborhood around the pixel are sorted with the pixel as the center, and the pixel value in the middle of the sort is taken as the pixel value of the current pixel. The above operations are performed on all pixels in the image to achieve noise reduction.

4. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 1, characterized in that, The method for implementing the model improvement is as follows: The GAN model is used as the basic model for generating foreign object images; By introducing the SE attention mechanism, the SE layer is added to the generator and discriminator of the GAN model. Image features are fused in a weighted manner through the SE layer, thereby improving network performance at a relatively low cost. A composite loss function is formed by using the mean absolute value error function, the mean square error function, and the CGAN loss function, and is used as the loss function of the model.

5. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 4, characterized in that, The SE layer includes the following processing steps: Compression: Dimensionality reduction is performed on each feature map using global average pooling, for the input tensor X∈R. H×W×C Each channel is compressed to obtain a 1×1×C tensor z, that is: Where H represents the height of the image, W represents the width of the image, C represents the depth of the image, and X(i,j) represents the element in the i-th row and j-th column of the input tensor; This allows us to obtain the compression information for each channel and generate the corresponding channel descriptor. Activation: The C-dimensional vector z obtained in the previous step is input into a fully connected layer, compressing its dimension to C / r, where r is a set hyperparameter. Then, the ReLU function is used to activate z, followed by another fully connected layer to restore the dimension of z to C. Finally, the Sigmoid function is used to activate z, and z is multiplied by the input tensor X to obtain the feature U'. The SE layer uses a fully connected layer and activation function to reduce the computational cost of the model and enable the model to learn the weight coefficients of different channels. The activation process is shown in the following formula: U'=σ([W1·Vec(z)] + ) Where Vec(·) represents transforming a tensor into a column vector, [·] + σ(·) represents the ReLU function activation operation, σ(·) represents the Sigmoid function activation operation, and W1 is the weight of the fully connected layer. Through the above steps, the SE layer can adaptively learn the importance of each channel and assign different weights to different channels, thereby improving the model's ability to extract and express feature maps.

6. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 4, characterized in that, The composite loss function combines the mean absolute value error function, the mean square error function, and the CGAN loss function, where the mean absolute value error function L L1 As shown in formula (1), the mean square error function L L2 As shown in formula (2), the CGAN loss function L CGAN As shown in formula (3), the loss function L, which combines the mean absolute value error function and the mean square error function, is shown in formula (4). The composite loss function G*, which is formed by combining the loss function L and the CGAN loss function, is shown in formula (5). L L1 =E c,x (xG(c)1) (1) L L2 =E c,x ((xG(c) 2 )) (2) L CGAN =E c,x (logD(c,x))+E c (log(1-D(c,G(c)))) (3) G * =argmin G max D L CGAN (G,D)+λL(G) (4) Where c is the input random noise, x is the input image, D(·) represents the output of the discriminator D, G(·) represents the output of the generator G, and E c (·) represents the mathematical expectation related to the input noise after training, E c,x (·) represents the mathematical expectation related to the input noise and the input image after training, and λ is the gradient penalty coefficient.

7. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 1, characterized in that, The model training is implemented as follows: Set the training parameters for the improved GAN model as follows: number of iterations epoch = 50000, batch_size = 2, number of CPU threads used during data loading num_woker = 0, gradient penalty coefficient λ = 10; The improved GAN model is trained using the training dataset to obtain a well-trained foreign object image generation model.

8. The method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks according to claim 1, characterized in that, The method for generating the image is as follows: The image to be detected is input into the trained foreign object image generation model to generate an expanded sample set. The foreign object image generation model mainly consists of two parts: a generator G and a discriminator D. The generator receives a random noise c and then trains it to fit the distribution of the original sample, making it closer and closer to the real sample. The discriminator is used to distinguish the differences between generated samples and real samples; the two compete with each other and continuously optimize their respective network parameters, eventually making it difficult for the discriminator to determine whether the received sample comes from a real sample or a generated sample, that is, reaching Nash equilibrium. At this point, it can be considered that the generator has learned the data distribution of the original image.

9. The method for generating and detecting foreign objects in a waterway based on generative adversarial networks according to claim 1, characterized in that, The method for foreign object identification is as follows: Acquire images of foreign objects in the waterway to be detected, the images of foreign objects in the waterway including an original sample set and an extended sample set; Foreign objects in the waterway in the image are divided into five categories: random piles, illegal constructions, illegal occupation, sewage outlets, and water hyacinths. Labeling software is used to label the five categories of target foreign objects in the noise-reduced image. The image of foreign objects in the waterway is input into a pre-trained target detection network to obtain the target detection result of the image of foreign objects in the waterway. The target detection network is trained using an original sample set and an extended sample set with labeled information; the extended sample set is obtained by combining the target slice images from the original sample set with the channel foreign object images generated by the foreign object image generation model. Each foreign object image in the extended sample set is obtained by inputting different random noises into a pre-trained foreign object image generation model based on a generative adversarial network.

10. The method for generating images of foreign objects in waterways and detecting foreign objects in waterways based on generative adversarial networks according to claim 1, characterized in that, The method for locating the foreign object is as follows: Extract Pos data from the image, i.e., the GIS coordinate information of the exterior orientation elements in UAV oblique photogrammetry. By constructing the relationship between latitude / longitude, Earth ellipsoid parameters, and the first eccentricity parameter, convert the degree-minute-second format to a degree-format latitude / longitude coordinate system, and then convert it to a geodetic coordinate system. The geodetic coordinate system describes the spatial location of a point using latitude B and longitude L. The specific formula is as follows: Where X is the horizontal rectangular coordinate, Y is the vertical rectangular coordinate, B is the latitude of the GIS coordinate of the foreign object, L is the longitude of the GIS coordinate of the foreign object, L0 is the longitude of the coordinate origin, L0 = 0, e is the first eccentricity, and K is the Earth's oblateness.

Images