Inland waterway inspection method and system and storage medium

Abstract

The present application relates to inland waterway inspection method, system and storage medium, including navigation mark detection and water level detection, navigation mark detection includes: step 11. Data acquisition;Step 12. Navigation mark identification;Step 13. Navigation mark positioning;Step 14. The real coordinates of navigation mark are inquired in the preset navigation mark data table, if the real coordinates of navigation mark have corresponding navigation mark in the preset navigation mark data table, then the current position information of the navigation mark is recorded;Step 15. Corresponding detection report is generated;Water level detection includes: step 21. Image acquisition;Step 22. Obtain the water gauge position image;Step 23. The water gauge position image is filtered, smoothed and denoised;Step 24. The water gauge scale and the image information with similar RGB values are extracted by setting threshold, and the scale is extracted by character extraction method, and the final water level information is obtained.The present application can automatically judge whether the navigation mark is damaged, the position is deviated, whether there is missing condition and water level.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, specifically to inland waterway inspection methods, systems, and storage media. Background Technology

[0002] Traditional inland waterway inspection methods typically rely on staff traveling by boat to conduct thorough inspections of navigational aids. This method not only consumes significant resources and manpower but is also prone to omissions and misdiagnoses due to eye fatigue from prolonged work.

[0003] In recent years, with the rapid advancement of drone technology, waterway inspection has seen the emergence of two new methods. The first method involves drone patrols followed by detailed manual analysis of the data collected by staff. The second method involves live-streaming the drone patrols to staff, who then assess for any anomalies in real time. However, despite the emergence of drone inspection methods, traditional manual inspection remains the mainstream approach for waterway inspection. This is primarily because these two new drone inspection methods still require human intervention in the anomaly assessment process, thus limiting their convenience and widespread application.

[0004] Therefore, it is necessary to develop a new method, system, and storage medium for inland waterway inspection. Summary of the Invention

[0005] The purpose of this invention is to provide an inland waterway inspection method, system and storage medium that can determine whether navigation marks are damaged, whether their positions are off, whether they are missing, and the water level, thus providing strong protection for the safety of inland waterway traffic, without requiring human intervention in the anomaly judgment process.

[0006] In a first aspect, the inland waterway inspection method of the present invention includes navigation mark detection and water level detection, wherein:

[0007] The steps for the navigation mark detection include:

[0008] Step 11. Collect data:

[0009] During the daytime, drones are used to collect images of navigational aids and record flight-related data, including latitude, longitude, altitude, and yaw angle.

[0010] If it is nighttime, use drones to collect video and images of the beacon flashing at the same location, and record flight-related data;

[0011] Step 12. Beacon Identification:

[0012] During the daytime, the navigation mark image is input into the improved YOLOv7-tiny-s network model to identify the navigation mark and determine whether it is damaged;

[0013] If it is nighttime, all lights in the inland waterway are detected frame by frame, and HSV color threshold space is used for filtering to retain only red and green lights; abnormal boxes are removed, and boxes with high repetition rate are merged to obtain a light array; moving lights and constantly lit lights are removed; the flashing frequency of the lights is calculated, and only navigation lights with flashing frequency within the preset flashing frequency range are retained, thus completing the navigation mark identification.

[0014] Step 13. Beacon positioning:

[0015] The true coordinates of the navigation beacon are estimated based on its position and size in the navigation beacon image, combined with the flight data of the UAV.

[0016] Step 14. Search the preset navigation mark data table for the actual coordinates of the navigation mark. If there is a corresponding navigation mark in the preset navigation mark data table, record the current position information of the navigation mark.

[0017] Step 15. Calculate the difference between the current position information of the navigation beacon and the preset position, determine whether the difference exceeds the preset difference, and generate a corresponding detection report. The detection report includes whether the navigation beacon is damaged, whether the position is offset, and whether there are any missing items.

[0018] The YOLOv7-tiny-s network model includes a backbone network and a neck network. A small target feature extraction module is set in the neck network, while the medium target feature extraction module and the large target feature extraction module are removed. The SimAM attention mechanism is added to the backbone network to optimize features, and a Gaussian NWD loss function suitable for small target detection is adopted to obtain the improved YOLOv7-tiny-s network model.

[0019] The water level detection steps include:

[0020] Step 21. Capture images using a drone;

[0021] Step 22. By scanning and matching, the position most similar to the preset water level gauge image template is found in the entire acquired image to obtain the water level gauge position image;

[0022] Step 23. Filter, smooth, and denoise the water level indicator image;

[0023] Step 24. Extract the water level gauge scale and image information with similar RGB values ​​by setting a threshold, and extract the scale using a character extraction method to obtain the final water level information; and display the water level information in the detection report.

[0024] Optionally, during water level detection, if the scale of the water gauge is obstructed or corroded, the following steps are performed before identifying the water gauge reading:

[0025] Watermark Generation: An improved version of the DCGAN model in Generative Adversarial Networks (GANs) is used to generate a complete watermark dataset. The improvements include: increasing the number of layers in the generator and discriminator to generate higher-quality images; incorporating self-attention mechanisms in both the generator and discriminator to enhance details and make the generated watermark images more realistic; introducing perceptual loss into the original DCGAN loss function and using VGG19 as the feature extractor, which significantly improves the quality of the generated images by comparing feature differences; and using diversity feature regularization to address the mode collapse problem and increase the diversity of the generated images.

[0026] Water erosion simulation: Noise is added to specific locations in the water level gauge image to simulate water erosion. The specific locations are the lower half of the water level gauge. Different areas at different heights are selected each time to add noise of different intensities to simulate the process of the water level gauge being covered by sludge, thus obtaining a water erosion dataset.

[0027] Water erosion restoration: An improved cycle GAN network is used to train the image restoration task. The water erosion dataset is used as the training input and the complete dataset is used as the training output to train the model and obtain a model that can convert water erosion images into complete images.

[0028] The improved cycle GAN network is obtained by improving cycle GAN, and the improvement methods include:

[0029] A self-attention mechanism is incorporated into both the generator and the discriminator; the self-attention mechanism can enhance some details of the image, making the restored image closer to the real image;

[0030] Adding residual blocks to the generator can improve training speed and enhance the stability of network training.

[0031] Adding a fully connected layer at the end of the discriminator allows the network to adapt to more complex tasks.

[0032] Optionally, the backbone network of the improved YOLOv7-tiny-s network model includes a first module to a ninth module connected in sequence; wherein: the first module and the second module are CBS modules; the third module and the fifth module are ELAN-simAM modules; the fourth module, the sixth module and the eighth module are MP modules, and the seventh module and the ninth module are ELAN modules.

[0033] Optionally, the neck network of the improved YOLOv7-tiny-s network model includes modules 10 to 26 connected in sequence, wherein module 13 is also connected to module 7 of the backbone network through module 27, module 17 is also connected to module 5 of the backbone network through module 28; module 21 is also connected to module 3 of the backbone network through module 29, and module 22 is also connected to module 30.

[0034] The tenth module is the SPPCSPC module;

[0035] Modules 11, 15, 19, 23, 26, 27, 28, and 30 are all CBS modules;

[0036] The twelfth, sixteenth, and twentieth modules are all UPSample modules;

[0037] The thirteenth, seventeenth, twenty-first, and twenty-fourth modules are all Cat modules;

[0038] Modules fourteen, eighteen, twenty-two, and twenty-five are all ELAN modules;

[0039] Modules eighteen through twenty-four, twenty-nine, and thirtieth constitute the small target feature extraction module. The improved YOLOv7-tiny-s network model is more accurate and effective in detecting small target navigation marks in complex inland waterway environments, achieving a recognition accuracy of 97%, an 8% improvement compared to the original YOLOv7-tiny-s network model. Simultaneously, the number of model parameters is halved, and the detection rate per image reaches 21.3 ms.

[0040] Optionally, the generator in the DCGAN model of the improved generative adversarial network comprises layers one through nine connected sequentially, wherein:

[0041] The first, second, third, fifth, seventh, and eighth layers all include Deconv units, BN units, and ReLU units; wherein, the Deconv unit represents activation and deconvolution operations; the BN unit is the batch normalization operation; and the ReLU unit is the ReLU activation function.

[0042] Both the fourth and sixth layers are self_Attn units; where the self_Attn unit is a self-attention mechanism;

[0043] The ninth layer includes Deconv units and Tanh units; wherein, the Tanh unit is a tanh activation function.

[0044] Optionally, the discriminator in the improved generative adversarial network's DCGAN model comprises layers 10 through 18 connected sequentially, wherein:

[0045] The tenth, eleventh, twelfth, fourteenth, sixteenth, and seventeenth layers all include conv units, BN units, and Leaky ReLU units; where conv units are convolution operations and Leaky ReLU units are Leaky ReLU activation functions.

[0046] Both the thirteenth and fifteenth layers are self_Attn units;

[0047] The eighteenth layer includes conv units and sigmoid units; the sigmoid unit is activated by the sigmoid activation function. The improved DCGAN model in the generative adversarial network can accurately and quickly achieve water erosion restoration.

[0048] Optionally, the Gaussian NWD loss function is:

[0049] ;

[0050] in: This represents the normalized loss value; C is a constant, and the value of C depends on the dataset. It is the Gaussian Wasserstein distance between the ground truth bounding box and the predicted bounding box;

[0051] ;

[0052] in:( ) represents the center coordinates of the true bounding box, ( () represents the center coordinates of the target box, and w and h represent its width and height, respectively; It is the width of the actual frame; The height of the actual bounding box; It is the width of the target bounding box; It represents the height of the bounding box; T represents the matrix transpose.

[0053] Small targets often perform worse than large targets due to their smaller size and fewer feature information. Compared to the original IoU loss function, the Gaussian NWD loss function can more accurately measure the distance between the predicted bounding box and the ground truth bounding box, which can encourage the model to pay more attention to these small targets, thereby improving the detection accuracy of small targets.

[0054] Optionally, The formula for the perceived loss is:

[0055] ;

[0056] in: To perceive loss; It is the input image to be generated; It is a real image of the target; and They represent their respective positions in the feature extraction network. Layer feature representation; It is the number of feature layers;

[0057] We use the ImageNet pre-trained model VGG19 as the feature extractor, and its first 18 layers of features are used as input to the perceptual loss. The perceptual loss compares the generated image with the real image. Compared to general image comparison, the perceptual loss performs image comparison at the feature level, resulting in more accurate comparisons. Furthermore, using the perceptual loss to supervise the generator makes the generated image closer to the real image.

[0058] Secondly, the inland waterway inspection system of the present invention includes:

[0059] Drones are used to collect image and video information;

[0060] A data processing server includes a memory and a controller. The memory stores a computer-readable program that, when invoked by the controller, can execute the steps of the inland waterway inspection method as described in this invention.

[0061] Thirdly, the present invention provides a storage medium storing a computer-readable program, which, when invoked, can execute the steps of the inland waterway inspection method described in the present invention.

[0062] The beneficial effects of this invention: This invention combines drone and artificial intelligence technologies to realize a fully automated method and system for inspecting navigation aids. During the day, this method and system primarily inspect navigation aids, determining whether they are damaged, misaligned, or missing. At night, the system focuses on detecting the color and flashing frequency of the navigation aid's lights to ensure its normal operation. This inspection method ensures the safety and reliability of navigation aids, providing strong protection for the safety of inland waterway transportation, and eliminates the need for manual intervention in the anomaly detection process. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the inland waterway inspection system in the embodiments of this application;

[0064] Figure 2 This is a schematic diagram of the daytime navigation mark detection structure in an embodiment of this application;

[0065] Figure 3 This is a schematic diagram of the improved YOLOv7-tiny-s network model in the embodiments of this application;

[0066] Figure 4 This is a flowchart of daytime navigation mark detection in an embodiment of this application;

[0067] Figure 5 This is a schematic diagram of the structure for detecting navigation marks at night, as shown in the embodiments of this application.

[0068] Figure 6 This is a schematic diagram illustrating the nighttime detection principle of navigation light in this embodiment of the application.

[0069] Figure 7 This is a flowchart illustrating nighttime navigation mark detection in an embodiment of this application;

[0070] Figure 8 This is a schematic diagram of the water level detection structure in an embodiment of this application;

[0071] Figure 9 This is a schematic diagram of the principle of water erosion gauge reading in the embodiments of this application;

[0072] Figure 10 This is a schematic diagram of the improved DCGAN model structure in the embodiments of this application;

[0073] Figure 11 This is a schematic diagram of the cycle GAN model structure in the embodiments of this application;

[0074] Figure 12 This is a schematic diagram of the water erosion reduction effect in the embodiments of this application;

[0075] Figure 13 This is a schematic diagram of the complete water level detection system in the embodiments of this application;

[0076] Figure 14 This is a schematic diagram of the inland waterway inspection system described in the embodiments of this application. Detailed Implementation

[0077] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0078] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0079] In this embodiment of the application, an inland waterway inspection method includes navigation mark detection and water level detection.

[0080] In this embodiment, for navigational aid detection, during the daytime, the main detection items include: determining whether the navigational aid is damaged, whether its position has shifted, and whether it is missing. At night, the main detection items include: the color and flashing frequency of the navigational aid's lights to ensure its normal operation. This detection method ensures the safety and reliability of the navigational aids, providing strong protection for the safety of inland waterway transportation.

[0081] In one possible implementation, daytime navigational aid detection primarily focuses on verifying whether the navigational aid's position has deviated and whether it is damaged. Images of the navigational aid are acquired by taking aerial photos using a drone, and the data is transmitted to an image workstation. At the image workstation, machine vision object recognition algorithms are used to identify the navigational aid and determine its damage. Subsequently, coordinate transformation is performed using the image information to accurately map the image coordinates to the actual location of the navigational aid. The core of this design is to accurately determine whether the navigational aid's position has shifted or is damaged through comprehensive analysis of image data.

[0082] like Figure 2 As shown below, the steps for daytime navigation mark inspection are explained in detail:

[0083] Step 11. Collect data:

[0084] During the day, drones are used to collect images of navigational aids and record flight-related data, including latitude, longitude, altitude, and yaw angle.

[0085] Step 12. Beacon Identification:

[0086] Beacon recognition encompasses both the identification of beacons and the detection of their damage. To ensure their safety, drones must fly at high altitudes, requiring high-resolution images to capture clear images of beacons. Beacons are relatively small within the overall image, posing a small target recognition problem within the field of target identification. Current target recognition algorithms have limitations in detecting small targets, making it difficult to achieve the required accuracy for beacon recognition systems.

[0087] To address this issue, this application proposes an improved YOLOv7-tiny-s network model (i.e., an improvement upon the original YOLOv7-tiny network model). The YOLOv7-tiny-s network model comprises a backbone network and a neck network. The neck network introduces a small target feature extraction module specifically designed for small targets, while removing the medium-sized target feature extraction modules and the large-sized target feature extraction modules. Furthermore, the backbone network incorporates a SimAM attention mechanism to optimize features, significantly improving the model's ability to recognize small targets. In addition, a Gaussian NWD loss function, more suitable for small target detection, is adopted, further enhancing detection efficiency and accuracy. These improvements enable the model to detect small target navigation marks more accurately and effectively in complex inland waterway environments. The improved model achieves a recognition accuracy of 97%, an 8% improvement over the original model, while reducing the number of parameters by half and achieving a detection rate of 21.3 ms per image.

[0088] like Figure 3 As shown in the embodiment of this application, the improved YOLOv7-tiny-s network model includes a backbone network comprising a first module to a ninth module connected in sequence; wherein: the first module and the second module are CBS modules; the third module and the fifth module are ELAN-simAM modules; the fourth module, the sixth module and the eighth module are MP modules, and the seventh module and the ninth module are ELAN modules. The neck network comprises modules 10 through 26 connected sequentially. Module 13 is connected to module 7 of the backbone network via module 27, and module 17 is connected to module 5 of the backbone network via module 28. Module 21 is connected to module 3 of the backbone network via module 29, and module 22 is connected to module 30. Module 10 is an SPPCSPC module; modules 11, 15, 19, 23, 26, 27, 28, and 30 are all CBS modules; modules 12, 16, and 20 are all UPSample modules; modules 13, 17, 21, and 24 are all Cat modules; and modules 14, 18, 22, and 25 are all ELAN modules. Modules 18 through 24, 29, and 30 constitute the small target feature extraction module.

[0089] The original YOLOv7-tiny model processes input images using a max pooling (MP) module to extract features. However, the MP module loses a significant amount of image information, which is particularly detrimental to small targets with limited detail, thus significantly impacting their detection accuracy. To address this issue, this embodiment adds a small target feature extraction module (see...). Figure 3 Meanwhile, the small target feature extraction module is directly connected to the third module of the backbone network. That is, the input image goes directly into the small target feature extraction module after passing through the first, second and third modules. In other words, a branch is drawn out before passing through the MP module for feature fusion, thereby improving the small target detection accuracy.

[0090] In this embodiment, during automatic navigation mark inspection, since the detection of medium and large targets is not required, the output features P4 and P5 (where P4 and P5 represent medium and large-scale feature maps, respectively) are removed from the original YOLOv7-tiny-s network model. If the output features P4 and P5 are not removed, it would not only be detrimental to the final detection result but would also increase the burden on the IDetect module in the YOLOv7-tiny-s network model, leading to more false positives. The IDetect module is used to process the input image to produce the final target detection result. This modification not only allows the model to focus on the detection of small targets but also significantly reduces the number of model parameters from 6M to 3.5M, significantly improving the model's processing speed.

[0091] In this embodiment, the CBS module is the basic convolution in the YOLOv7-tiny network, which includes convolutional layers, Batch Normalization (BN), and the Leaky ReLU activation function. The convolutional layers are primarily responsible for feature extraction and are one of the most important operations in the model. BN ensures that the output of each layer is as consistent as possible with the input data distribution of the next layer, making the model more stable during training. This suppresses internal covariate shifts, reducing the network model's sensitivity to initial parameter values ​​and effectively improving convergence speed. Leaky ReLU, as the model's activation function, is more lightweight than the SiLU activation function, which helps improve the model's efficiency and performance.

[0092] In this embodiment, the ELAN-simAM module is a deep learning network layer that combines ELAN (Efficient Aggregation Network) and simAM (simple attention module). ELAN aims to enhance the learning ability of the network by efficiently aggregating network structures, while simAM is a lightweight attention module used to improve the expressive power of feature maps. In the ELAN-simAM layer, features are first efficiently aggregated using the ELAN structure, and then the simAM attention module is used to weight the aggregated features, enabling the model to focus on more important feature information. This combination allows the network to further improve performance while maintaining computational efficiency.

[0093] In this embodiment, the MP module in deep learning typically refers to the Max Pooling layer. It is an important component in convolutional neural networks, used to reduce the spatial dimensions (height and width) of the input while preserving important feature information. Max pooling works by selecting the maximum value from each local region of the input feature map, which helps reduce the computational complexity of the model, decrease overfitting, and improve the robustness of the features.

[0094] In this embodiment, the ELAN module is responsible for aggregating features at different scales or levels.

[0095] In this embodiment, the SPPCSPC module obtains different receptive fields through max pooling to detect targets of different sizes in the image to be detected.

[0096] In this embodiment, the Upsample module is an upsampling layer, an important component in convolutional neural networks, used to increase the spatial dimensions (such as height and width) of the input data. This layer is typically used for upsampling operations, transforming low-resolution or small-sized feature maps into higher-resolution or larger-sized outputs.

[0097] In this embodiment, the cat layer is used to fuse the extracted image feature information from different levels.

[0098] Step 13. Beacon positioning:

[0099] For beacon positioning, while the UAV is acquiring images of the beacon, the current yaw angle, latitude, longitude, and altitude are retained. The original image coordinates are transformed to the real-world WSG84 coordinate system using coordinate transformation formulas, thereby achieving beacon positioning. The transformation formulas are shown in formulas (1) and (2). Where H is the altitude of the UAV flight, α is the camera gimbal yaw angle, n and e are the coordinates of the image center in the WSG84 coordinate system, x and y are the real coordinates, and xi and yi are the pixel coordinates.

[0100] (1)

[0101] (2)

[0102] Step 14. Search the preset navigation mark data table for the actual coordinates of the navigation mark. If there is a corresponding navigation mark in the preset navigation mark data table, record the current position information of the navigation mark.

[0103] Step 15. Calculate the difference between the current position information of the navigation beacon and the preset position, determine whether the difference exceeds the preset difference, and generate a corresponding detection report. The detection report includes whether the navigation beacon is damaged, whether the position is offset, and whether there are any missing items.

[0104] like Figure 4 As shown, in one possible embodiment, daytime navigation mark detection provides two detection modes: one is single-image detection for user viewing, and the other is batch processing.

[0105] In single-image detection mode, users do not need to import any preset information. They only need to select the image to be detected and click the detection button. The system automatically imports the image, reads the image and its information, and determines whether there is a navigational aid vessel. If there is a navigational aid (usually a navigational aid vessel or a canister marker), the system calculates the position of the navigational aid vessel (i.e., the navigational aid position) through coordinate transformation. The system then compares the navigational aid position with each preset coordinate in the preset navigational aid data table to determine if there is a corresponding coordinate. If so, the system fills in the current position information of the corresponding navigational aid.

[0106] In batch detection mode, users need to first import preset information, then select the storage path of the folder, and detect each image in the folder (similar to single image detection). The system compares the images with each preset coordinate in the preset navigation mark data table to determine if a corresponding coordinate exists. If so, the current position information of the corresponding navigation mark is entered.

[0107] Based on the inspection results, an inspection conclusion report is generated, including whether the navigation marks are damaged, whether their positions have shifted, and whether there are any missing items. Users can view these results through the interface or choose to export them as a Word file for further review. Finally, the software interface and its effects are demonstrated.

[0108] In one possible embodiment, the inspection conclusion report is as follows:

[0109] Conclusion of UAV Inspection of xxx Waterway Operation and Maintenance Status

[0110] xxx - xxx conducted a drone inspection of the xx waterway, and the inspection conclusions are as follows:

[0111] The water level at the 1.xxx bridge is 3.34m.

[0112] 2. Daytime inspection results of navigation marks indicate that:

[0113] There are no navigational aids whose position deviation exceeds the design value.

[0114] The missing navigation marks are: Longchuanchong White Can, Longchuanchong Red Can 1, Wanda Plaza Red Boat 1, Wanda Plaza White Boat, Wanda Plaza Red Boat 2, White Can upstream of Xijiang Bridge, Red Boat upstream of Xijiang Bridge, White Boat upstream of Xijiang Bridge, and Red Boat upstream of Wuzhou Ring Expressway. It is recommended to conduct on-site inspection to check whether the navigation mark positions are correct.

[0115] 3. Nighttime inspection results of navigation marks indicate that:

[0116] There are no navigation marks whose position deviation exceeds the design value.

[0117] The following navigation marks are missing or malfunctioning: Longchuanchong Red Can, Longchuanchong White Can, Longchuanchong Red Can 1, White Boat downstream of Xijiang Bridge, Red Boat downstream of Xijiang Bridge, Red Boat 1 at Wanda Plaza, White Boat at Wanda Plaza, Red Boat 2 at Wanda Plaza, White Can upstream of Xijiang Bridge, Red Boat upstream of Xijiang Bridge, White Boat upstream of Xijiang Bridge, Red Boat upstream of Xijiang Bridge, and Red Boat upstream of Wuzhou Ring Expressway. It is recommended to check the location of the navigation marks on-site to ensure they are correct.

[0118] like Figure 5 As shown in this embodiment, the focus of nighttime navigational aid detection is on identifying the lights of nighttime navigational aids and measuring their flashing frequency. To achieve this, a drone is used to capture a video of the navigational aid flashing at a fixed point, and at the end of the video, another image of the navigational aid is captured at the same location. This image is used to extract the longitude, latitude, altitude, and yaw angle information of the current scene. The detection of the navigational aid's location is similar to that during the day, but at night, special attention needs to be paid to the presence of lights. To identify the flashing frequency of the navigational aid's lights, an image processing algorithm is used to process each frame of the video to obtain the accurate flashing frequency of the navigational aid.

[0119] like Figure 6As shown, in one possible embodiment, the overall algorithm structure for navigational light recognition is as follows: First, the algorithm detects all lights in the inland waterway frame by frame and uses HSV color threshold space filtering to retain only red and green lights. Next, outlier boxes are removed and boxes with high repetition rates are merged. Then, the retained lights are stored in a light array, and the light array within 240 frames is further detected. Based on field investigation, it was found that navigation lights at night are usually constantly on, and ships are always in motion and do not stay in the waterway. Therefore, the total number of luminous frames at the same location will not exceed 5 frames, and those that do not meet the luminous frame count requirement for navigational lights will be removed. After the first round of looping, all navigational lights are collected into a dedicated navigational light array. Subsequently, in the second round of looping, the navigational light array is analyzed to identify the flashing cycle and convert the frame count into seconds. Therefore, it is necessary to ensure that the video recording duration exceeds two flashing cycles. In addition, the algorithm will continuously detect the light cycle and determine the final navigational light flashing cycle by averaging multiple cycles.

[0120] In one possible implementation, nighttime navigational aid detection also offers two detection modes: one is single-image detection for user viewing, and the other is batch processing.

[0121] like Figure 7 As shown, in single-image detection mode, users do not need to import any preset information. They only need to select the image to be detected and click the detection button. The system automatically performs the following steps: First, it identifies nighttime lights, counts frames, and determines if there are navigational beacons. If so, it adds the red and green lights to the light array and checks if the time exceeds 240 frames. If not, it returns to check if there are navigational beacons. If so, it checks if the light array contains navigational beacon lights. If it does, it adds them to the navigational beacon light array and detects the flashing cycle over one flashing period. In other words, it obtains the current navigational beacon light color and flashing cycle.

[0122] like Figure 7 As shown, in batch detection mode, users need to first import preset information, then select a folder for batch processing, with videos and images placed in the same path. Each nighttime light video in the folder is processed, and the location coordinates of the navigation beacon lights are calculated using the corresponding image information. These coordinates are then compared with each preset coordinate in the preset navigation beacon data table to determine if a corresponding coordinate exists. If so, the current location information, light color, and flashing cycle of the corresponding navigation beacon are entered.

[0123] An inspection conclusion report is generated based on the test results.

[0124] In this embodiment, water level detection employs image processing to acquire water level gauge images and identify the water level. This requires identifying the water level gauge and obtaining its reading from a large image captured by a drone. Furthermore, due to prolonged water erosion, some water level gauges exhibit water erosion. Eroded water level gauges need to be restored before the water level can be read; therefore, water level detection is divided into intact water level detection and water erosion water level detection. For the overall structure of the water level detection system, see [link to relevant documentation]. Figure 8 .

[0125] In real-world scenarios, water gauges may experience water erosion. This erosion can obscure or corrode the gauge's markings, making direct extraction of marking information impossible. To address this issue, this system proposes a water gauge reconstruction method based on a generative adversarial network (GAN). The GAN is trained to reconstruct the eroded water gauge before reading it. Previously, GANs required a substantial dataset for training to achieve the desired results, but water gauge images were difficult to collect. Therefore, it was necessary to first supplement the water gauge dataset to obtain a sufficient amount of data. Thus, the reading of the eroded water gauge is divided into three parts: water gauge image generation, water erosion simulation, and water gauge reconstruction. (See [link to relevant documentation]). Figure 9 .

[0126] In this embodiment of the application, for a complete water gauge, the water level identification steps include:

[0127] Step 21. Capture images using a drone;

[0128] Step 22. Since the image captured by the drone is very large, while the water level gauge only occupies a very small part of the entire image, the water level gauge image must be extracted first before processing it. Template matching can be used to find the position in the entire large image that is most similar to the template of the water level gauge image through scanning matching. By obtaining the position of the water level gauge, the water level and its surrounding image can be extracted, thus transforming the processing of large images into the processing of small images.

[0129] Step 23. Due to noise and other issues affecting the extracted water level gauge image, resulting in low image clarity, image enhancement is needed to improve image clarity and better identify the water level gauge markings. Image enhancement is mainly achieved through filtering, smoothing, and noise reduction.

[0130] Step 24. Thresholding segmentation can remove most of the background information of the image, reducing interference caused by the background. By setting a threshold, image information with similar RGB values ​​of the water level gauge is extracted. The scale is then extracted using methods such as character extraction. The final water level information is obtained through calculation and displayed in the detection report.

[0131] In this embodiment of the application, before conducting water level detection, the following steps are performed on the water erosion gauge:

[0132] Water level gauge generation: An improvement to the DCGAN (Deep Convolutional Generative Adversarial Network) model within generative adversarial networks is used to complete the water level gauge dataset generation task. Improvement methods include:

[0133] 1. Improve the network structure of the original DCGAN model by increasing the number of network layers in the generator and discriminator to generate higher quality images.

[0134] 2. Adding a self-attention mechanism to the generator and discriminator can enhance some details of the generated water level gauge image, making the generated water level gauge image closer to the real image.

[0135] 3. To further improve the realism of the generated images, a perceptual loss was introduced into the original DCGAN loss function, and VGG19 was used as the feature extractor. By comparing the differences in features between the images, the quality of the generated images can be greatly improved.

[0136] 4. To optimize the pattern collapse problem caused by imbalanced training in generative adversarial networks, diversity feature regularization is used to optimize the pattern collapse problem and improve the diversity of generated images.

[0137] like Figure 10 As shown, in one possible embodiment, the generator in the DCGAN model of the improved generative adversarial network comprises layers one through nine connected sequentially, wherein: the first, second, third, fifth, seventh, and eighth layers each include a Deconv unit, a BN unit, and a ReLU unit. The Deconv unit represents activation and deconvolution operations; the BN unit is a batch normalization operation; and the ReLU unit is the ReLU activation function. The fourth and sixth layers are both self_Attn units; where the self_Attn unit is a self-attention mechanism. The ninth layer includes a Deconv unit and a Tanh unit; where the Tanh unit is the tanh activation function.

[0138] like Figure 10 As shown, in one possible embodiment, the discriminator in the improved DCGAN model of the generative adversarial network comprises layers 10 through 18 connected sequentially. Layers 10, 11, 12, 14, 16, and 17 each include a conv unit, a BN unit, and a Leaky ReLU unit; where the conv unit is a convolution operation and the Leaky ReLU unit is a Leaky ReLU activation function. Layers 13 and 15 both consist of self_Attn units. Layer 18 includes a conv unit and a sigmoid unit; where the sigmoid unit is a sigmoid activation function.

[0139] In one possible embodiment, the Gaussian NWD loss function is:

[0140] ;

[0141] in: This represents the normalized loss value; C is a constant, and the value of C depends on the dataset. It is the Gaussian Wasserstein distance between the ground truth bounding box and the predicted bounding box;

[0142] ;

[0143] in:( ) represents the center coordinates of the true bounding box, ( () represents the center coordinates of the target box, and w and h represent its width and height, respectively; It is the width of the actual frame; The height of the actual bounding box; It is the width of the target bounding box; It represents the height of the bounding box; T represents the matrix transpose.

[0144] Small targets often perform worse than large targets due to their smaller size and fewer feature information. Compared to the original IoU loss function, the Gaussian NWD loss function can more accurately measure the distance between the predicted bounding box and the ground truth bounding box, which can encourage the model to pay more attention to these small targets, thereby improving the detection accuracy of small targets.

[0145] In one possible embodiment, the formula for the perceptual loss is:

[0146] ;

[0147] in: To perceive loss; It is the input image to be generated; It is a real image of the target; and They represent their respective positions in the feature extraction network. Layer feature representation; It is the number of feature layers;

[0148] We use the ImageNet pre-trained model VGG19 as the feature extractor. The first 18 layers of the VGG19 feature extractor consist of 16 convolutional layers and 2 fully connected layers. The first 16 layers are all convolutional layers, with a max-pooling operation following layers 2, 4, 8, 12, and 16 to reduce dimensionality. Layers 17 and 18 are fully connected layers. The features from the first 18 layers of the VGG19 feature extractor are used as input to the perceptual loss. The perceptual loss compares the generated image with the real image. Compared to general image comparison, the perceptual loss performs image comparison at the feature level, resulting in more accurate comparisons. Furthermore, using the perceptual loss to supervise the generator makes the generated image closer to the real image.

[0149] Water Erosion Simulation: This section uses algorithms to simulate real-world water erosion problems. Essentially, water erosion is the prolonged scouring of a water level gauge by water, causing the lower part of the gauge to become obscured by mud or other contaminants. Since the height of the water scouring and the area obscured by the mud vary randomly each time, the simulation algorithm needs to reflect this randomness. Water erosion is simulated by adding noise at specific locations in the water level gauge image. Because water erosion occurs in areas close to the water body in the image, the location must be chosen in the lower half of the gauge. Different heights and intensities of noise are added to different areas each time to simulate the process of the water level gauge being obscured by mud.

[0150] Water Erosion Restoration: The image restoration task is trained using the two datasets mentioned earlier: the complete dataset and the water erosion dataset. Generative Adversarial Networks (GANs) are also required. The Cycle GAN network is chosen for image restoration; it's a very popular image-to-image network in recent years and is quite effective for image restoration. The model is trained using the water erosion dataset as input and the complete dataset as output, resulting in a model that can convert water-eroded images into complete images. Similarly, the Cycle GAN needs to be improved, specifically as follows:

[0151] 1. Add a self-attention mechanism to the generator and discriminator. The self-attention mechanism can enhance some details of the image, making the restored image closer to the real image.

[0152] 2. Adding residual blocks to the generator can improve training speed and enhance the stability of network training.

[0153] 3. Adding a fully connected layer at the end of the discriminator can make the network adaptable to more complex tasks.

[0154] like Figure 11As shown, in one possible embodiment, a self-attention mechanism is added to both the generator (including generator 1 and generator 2) and discriminator of the Cycle GAN model structure. The self-attention mechanism can capture long-range dependencies in the image, helping the model focus on key parts of the image and generate more refined details. In the generator, the self-attention mechanism can enhance the key features of the generated image, making the generated image more realistic. In the discriminator, the self-attention mechanism can help the model better identify key parts in the image, thus providing more accurate feedback to guide the generator's training. Residual blocks are added to the generators (including generator 1 and generator 2): Residual blocks can solve the gradient vanishing problem in deep neural networks, thereby improving training speed and enhancing the stability of network training. By adding residual blocks to the generator, the model can more easily learn high-level features of the image and generate higher-quality images. Residual blocks can also increase the depth of the network, further improving the model's performance. A fully connected layer is added at the end of the discriminator: The fully connected layer can map the features extracted by the convolutional layers to the final classification output, allowing the network to adapt to more complex tasks. Adding fully connected layers to the discriminator increases the model's complexity and improves its ability to discriminate complex images. At the same time, fully connected layers can help the model learn more global information, further improving the discriminator's performance.

[0155] like Figure 12 The image shows the effect of water erosion reduction.

[0156] like Figure 13 As shown, in one embodiment, water level detection provides two detection modes: one is single-image detection for user viewing, and the other is batch processing.

[0157] In single-image detection mode, users do not need to import any preset information. They only need to select the image to be detected and click the detection button. The system automatically uses the water level gauge image for template matching. If the matching fails, it continues matching. If the matching succeeds, it returns the position of the water level gauge in the image and extracts the water level gauge. The extracted water level gauge is preprocessed to extract the white parts of the image, calculate the total length of the scale and the length of each scale, and obtain the water level height.

[0158] In batch detection mode, users first need to import preset information, then select a folder (the folder containing water level gauge images) for batch processing. Each image in the folder is processed, the relationship between the image and preset positions is calculated, the average of the results is taken, and the processing results are directly recorded in the results information. Users can view these results through the interface or choose to export them as a Word file for review.

[0159] like Figure 1 and Figure 14As shown in the embodiments of this application, an inland waterway inspection system includes a drone and a data processing server. The drone is used to collect image and video information. The data processing server includes a memory and a controller. The memory stores a computer-readable program, which, when invoked by the controller, can execute the steps of the inland waterway inspection method as described in the embodiments of this application.

[0160] In this embodiment of the application, a storage medium stores a computer-readable program, which, when invoked, can execute the steps of the inland waterway inspection method described in this embodiment of the application.

[0161] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for inspecting inland waterways, characterized in that: This includes navigational aid inspection and water level inspection, among which: The steps for the navigation mark detection include: Step 11. Collect data: During the daytime, drones are used to collect images of navigational aids and record flight-related data, including latitude, longitude, altitude, and yaw angle. If it is nighttime, use drones to collect video and images of the beacon flashing at the same location, and record flight-related data; Step 12. Beacon Identification: During the daytime, the navigation mark image is input into the improved YOLOv7-tiny-s network model to identify the navigation mark and determine whether it is damaged; If it is nighttime, all lights in the inland waterway are detected frame by frame, and HSV color threshold space is used for filtering to retain only red and green lights; abnormal boxes are removed, and boxes with high repetition rate are merged to obtain a light array; moving lights and constantly lit lights are removed; the flashing frequency of the lights is calculated, and only navigation lights with flashing frequency within the preset flashing frequency range are retained, thus completing the navigation mark identification. Step 13. Beacon positioning: The true coordinates of the navigation beacon are estimated based on its position and size in the navigation beacon image, combined with the flight data of the UAV. Step 14. Search the preset navigation mark data table for the actual coordinates of the navigation mark. If there is a corresponding navigation mark in the preset navigation mark data table, record the current position information of the navigation mark. Step 15. Calculate the difference between the current position information of the navigation beacon and the preset position, determine whether the difference exceeds the preset difference, and generate a corresponding detection report. The detection report includes whether the navigation beacon is damaged, whether the position is offset, and whether there are any missing items. The YOLOv7-tiny-s network model includes a backbone network and a neck network. A small target feature extraction module is set in the neck network, while the medium target feature extraction module and the large target feature extraction module are removed. The SimAM attention mechanism is added to the backbone network to optimize features, and a Gaussian NWD loss function suitable for small target detection is adopted. The features are then used to obtain the improved YOLOv7-tiny-s network model. The water level detection steps include: Step 21. Capture images using a drone; Step 22. By scanning and matching, the position most similar to the preset water level gauge image template is found in the entire acquired image to obtain the water level gauge position image; Step 23. Filter, smooth, and denoise the water level indicator image; Step 24. Extract the water level gauge scale and image information with similar RGB values ​​by setting a threshold, and extract the scale using a character extraction method to obtain the final water level information; and display the water level information in the detection report.

2. The inland waterway inspection method according to claim 1, characterized in that: In water level monitoring, if the scale on the water gauge is obstructed or corroded, the following steps must be performed before reading the water gauge: Watermark Generation: This project improves upon the DCGAN model in Generative Adversarial Networks (GANs) to generate a complete watermark dataset. The improvements include: increasing the number of network layers in the generator and discriminator; incorporating self-attention mechanisms in both the generator and discriminator; introducing perceptual loss into the original DCGAN loss function and using VGG19 as the feature extractor; and using diversity feature regularization to address the mode collapse problem. Water erosion simulation: Noise is added to specific locations in the water level gauge image to simulate water erosion. The specific locations are the lower half of the water level gauge. Different areas at different heights are selected each time to add noise of different intensities to simulate the process of the water level gauge being covered by sludge, thus obtaining a water erosion dataset. Water erosion restoration: An improved cycle GAN network is used to train the image restoration task. The water erosion dataset is used as the training input and the complete dataset is used as the training output to train the model and obtain a model that can convert water erosion images into complete images. The improved cycle GAN network is obtained by modifying the cycle GAN, and the improvement methods include: A self-attention mechanism is incorporated into both the generator and the discriminator; Add residual blocks to the generator; A fully connected layer is added at the end of the discriminator.

3. The inland waterway inspection method according to claim 1, characterized in that: The backbone network of the improved YOLOv7-tiny-s network model includes modules one through nine connected in sequence; wherein: the first and second modules are CBS modules; the third and fifth modules are ELAN-simAM modules; the fourth, sixth, and eighth modules are MP modules; and the seventh and ninth modules are ELAN modules.

4. The inland waterway inspection method according to claim 3, characterized in that: The neck network of the improved YOLOv7-tiny-s network model includes modules 10 to 26 connected in sequence. Module 13 is also connected to module 7 of the backbone network through module 27, and module 17 is also connected to module 5 of the backbone network through module 28. Module 21 is also connected to module 3 of the backbone network through module 29, and module 22 is also connected to module 30. The tenth module is the SPPCSPC module; Modules 11, 15, 19, 23, 26, 27, 28, and 30 are all CBS modules; The twelfth, sixteenth, and twentieth modules are all UPSample modules; The thirteenth, seventeenth, twenty-first, and twenty-fourth modules are all Cat modules; Modules fourteen, eighteen, twenty-two, and twenty-five are all ELAN modules; Among them, modules eighteen to twenty-four, module twenty-nine, and module thirtieth constitute the small target feature extraction module.

5. The inland waterway inspection method according to claim 2, characterized in that: The generator in the improved generative adversarial network DCGAN model comprises layers one through nine connected sequentially, wherein: The first, second, third, fifth, seventh, and eighth layers all include Deconv units, BN units, and ReLU units; wherein, the Deconv unit represents activation and deconvolution operations; the BN unit is the batch normalization operation; and the ReLU unit is the ReLU activation function. Both the fourth and sixth layers are self_Attn units; where the self_Attn unit is a self-attention mechanism; The ninth layer includes Deconv units and Tanh units; wherein, the Tanh unit is a tanh activation function.

6. The inland waterway inspection method according to claim 5, characterized in that: The discriminator in the improved generative adversarial network DCGAN model comprises layers 10 through 18 connected sequentially, wherein: The tenth, eleventh, twelfth, fourteenth, sixteenth, and seventeenth layers all include conv units, BN units, and Leaky ReLU units; where conv units are convolution operations and Leaky ReLU units are Leaky ReLU activation functions. Both the thirteenth and fifteenth layers are self_Attn units; The eighteenth layer includes conv units and sigmoid units; wherein, the sigmoid unit is a sigmoid activation function.

7. The inland waterway inspection method according to claim 1, characterized in that: The Gaussian NWD loss function is: ； in: This represents the normalized loss value; C is a constant, and the value of C depends on the dataset. It is the Gaussian Wasserstein distance between the ground truth bounding box and the predicted bounding box; ； in:( ) represents the center coordinates of the true bounding box, ( () represents the center coordinates of the target box, and w and h represent its width and height, respectively; It is the width of the actual frame; The height of the actual bounding box; It is the width of the target bounding box; It represents the height of the bounding box; T represents the matrix transpose.

8. The inland waterway inspection method according to claim 2, characterized in that: The formula for the perceived loss is: ； in: To perceive loss; It is the input image to be generated; It is a real image of the target; and They represent their respective positions in the feature extraction network. Layer feature representation; It is the number of feature layers; The ImageNet pre-trained model VGG19 is used as the feature extractor, and its first 18 layers of features are used as input to the perceptual loss.

9. An inland waterway inspection system, characterized in that, include: Drones are used to collect image and video information; A data processing server includes a memory and a controller, wherein the memory stores a computer-readable program that, when invoked by the controller, can perform the steps of the inland waterway inspection method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: It contains a computer-readable program that, when invoked, can perform the steps of the inland waterway inspection method as described in any one of claims 1 to 8.

Images