Water stain detection method and device based on deep learning

Abstract

The embodiment of the present application provides a method and device for detecting water accumulation and water stains based on deep learning, and the method includes: capturing images through a binocular camera, obtaining color images during the day and infrared images at night; inputting the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; inputting the color images and the infrared images after image quality enhancement into the trained segmentation model and detection model at the same time; mapping the preliminary segmentation results output by the segmentation model back to the original image size, and obtaining a segmentation confidence feature map through normalization and maximum value operations; weightedly fusing the segmentation confidence feature map with the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the interval [0,1] and [0.05,0.95] and verifying on a test set; and determining the pixel points greater than 0.5 in the final confidence feature map as water accumulation or water stain areas; the present application can effectively improve the accuracy of water accumulation and water stain detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a method and device for detecting accumulated water stains based on deep learning. Background Art

[0002] In recent years, with the acceleration of industrialization and digitalization, the country has paid more attention to the safety of large factories, computer rooms, power distribution rooms, etc. These places are becoming more and more important in daily operations. Any unexpected situation may cause significant economic losses or even threaten the safety of people's lives. Therefore, improving the safety of these places has become a top priority.

[0003] Among these, hazards caused by power problems account for the vast majority of cases. If there is water leakage or water accumulation in the room, it is more likely to cause serious problems such as electric shock accidents, power failures, and even fires. The current method of detecting water accumulation is mainly through manual inspections, which means that security guards inspect the site or use surveillance cameras to inspect whether there is water accumulation in the room. However, this method is slow, inefficient, and cannot detect potential problems in a timely manner. Summary of the invention

[0004] In response to the problems in the prior art, the present application provides a method and device for detecting water stains based on deep learning, which can effectively improve the accuracy of water stain detection.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for detecting water stains based on deep learning, comprising:

[0007] An image super-resolution reconstruction model is constructed and trained. The image super-resolution reconstruction model uses a U-type network as the backbone network, and the input is a low-resolution image that is downsampled, blurred and denoised. Perceptual loss, binary cross entropy loss and L1 loss are used as training loss functions. A segmentation model based on a U-type network and Fourier transform is constructed and trained. The input image is subjected to Fourier transform to obtain frequency domain information. Feature extraction is achieved through four times of convolution pooling downsampling. Before each pooling, the frequency domain information is fused with the current layer image features, and fused with the deconvolution feature map through a jump connection. A detection model is constructed and trained. The intersection-over-union loss function is used, and the detection category is set to water.

[0008] Capture images with a binocular camera, obtain color images during the day and infrared images at night; input the infrared images into a trained image super-resolution reconstruction model to enhance image quality; input the color images and the infrared images after image quality enhancement into a trained segmentation model and a detection model at the same time;

[0009] The preliminary segmentation result output by the segmentation model is mapped back to the size of the original image, and a segmentation confidence feature map is obtained by normalization and maximum value operation; the preliminary detection result output by the detection model is subjected to non-maximum suppression processing, and the confidence threshold is set to a, and the intersection-over-union threshold is set to 0.6; based on the preliminary detection result, an all-zero array of the same size as the original image is created as a detection confidence feature map, and for the area within the detection frame, when the detection confidence is greater than or equal to a, the value of the detection confidence feature map of the area is set to the corresponding detection confidence, and for the overlapping area of ​​the detection frames, the value of the detection confidence feature map is set to the average of the confidences of each detection frame in the overlapping area; the segmentation confidence feature map is weightedly fused with the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verified on the test set; pixels greater than 0.5 in the final confidence feature map are determined as waterlogging or water stain areas.

[0010] Furthermore, the image super-resolution reconstruction model is constructed and trained, the image super-resolution reconstruction model uses a U-shaped network as a backbone network, the input is a downsampled, blurred and denoised low-resolution image, and perceptual loss, binary cross entropy loss and L1 loss are used as training loss functions, including:

[0011] The U-shaped network is used as the backbone network. A four-layer convolutional pooling network is used for feature extraction in the encoder part. Each convolutional pooling network consists of two groups of convolutional layers, an activation layer, and a batch normalization layer. The decoder part uses a deconvolution layer for feature upsampling, and the features of the corresponding layer of the encoder are fused with the feature map of the decoder through a skip connection structure.

[0012] The training image is downsampled by 4 times and Gaussian blur and random noise are added to obtain a low-resolution image, which is input into the image super-resolution reconstruction model. The perceptual loss function is used to calculate the distance between the generated image and the original image in the feature space. The binary cross entropy loss function is used to constrain the image classification result. The L1 loss function is used to constrain the image reconstruction accuracy. The weighted sum of the three loss functions is used as the total loss function for network optimization.

[0013] Furthermore, the construction and training of the detection model, using the intersection-over-union loss function, and setting the detection category to water includes:

[0014] A backbone network with a residual structure is used for feature extraction. Feature maps of different scales are fused through a feature pyramid network. Three anchor frames of preset scales are set in each prediction layer of the feature pyramid network. The feature map output by the prediction layer is passed through a 1×1 convolutional layer to obtain a category prediction branch and a bounding box regression branch. The category prediction branch outputs the probability value of each anchor frame belonging to the water category, and the bounding box regression branch outputs the offset of each anchor frame relative to the target true frame.

[0015] The input image is scaled to a preset size and normalized, the intersection-over-union (IoU) of the predicted box and the true box is calculated, and the anchor box whose IoU is greater than the preset threshold is taken as a positive sample. The coordinate offset between the predicted box and the true box, the difference between the confidence score and the corresponding true value are taken as regression loss. The network parameters are optimized using the IoU loss function, and the ratio of the classification loss weight to the regression loss weight in the loss function is set to 1:5.

[0016] Furthermore, mapping the preliminary segmentation result output by the segmentation model back to the original image size and obtaining the segmentation confidence feature map through normalization and maximum value operations includes:

[0017] Perform bilinear interpolation on the feature map of size (w3,h3,2) output by the segmentation model, adjust the size of the feature map to the size of the original input image (w,h), keep the number of channels unchanged, and use the pixel resampling method to smooth the interpolated feature map to eliminate image distortion caused by interpolation;

[0018] The mapped feature map is subjected to the maximum value operation in the channel dimension to obtain a single-channel feature map. Each pixel value in the feature map is normalized by min-max to map it to the interval [0,1]. The local response normalization method is used to perform local normalization of the spatial position of the feature map to enhance the distinguishability of the features, and a segmentation confidence feature map is obtained.

[0019] Furthermore, the non-maximum suppression processing is performed on the preliminary detection result output by the detection model, the confidence threshold is set to a, and the intersection-over-union ratio threshold is set to 0.6, including:

[0020] The preliminary detection result output by the detection model is a feature map of size (1, 5, 8400), which contains the center coordinates, width, height and confidence information of each candidate box. The candidate boxes with confidence greater than the threshold a in the feature map are screened, and the screened candidate boxes are arranged in descending order according to the confidence to form a candidate box set. The intersection-and-union ratio between any two candidate boxes in the candidate box set is calculated;

[0021] Take the candidate box with the highest confidence from the sorted candidate box set as the reference box, calculate its intersection-over-union ratio with the remaining candidate boxes, delete the candidate boxes with an intersection-over-union ratio greater than 0.6 from the candidate box set, repeat the above process until the candidate box set is empty, and get the final detection box set, where each detection box contains its position coordinates and the corresponding confidence value.

[0022] Furthermore, the method of creating an array of all zeros of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and setting the value of the detection confidence feature map of the area to the corresponding detection confidence when the detection confidence is greater than or equal to a for the area within the detection frame, and setting the value of the detection confidence feature map to the average value of the confidence of each detection frame in the overlapping area for the detection frames, includes:

[0023] Create a two-dimensional zero matrix of the same size as the input image, traverse the set of detection frames after non-maximum suppression, take the position coordinate information of each detection frame to determine the position index of the pixel point in the frame, and when the confidence value of the detection frame is greater than or equal to the threshold a, update the values ​​of the corresponding positions of all pixels in the frame in the zero matrix to the confidence value of the detection frame;

[0024] Calculate the coordinates of the intersection area of ​​any two detection frames, store the pixel position index of the intersection area into the overlapping area set, and for each pixel in the overlapping area set, count the confidence values ​​of all detection frames containing the pixel, and assign the arithmetic mean of the confidence values ​​to the pixel at the corresponding position in the detection confidence feature map.

[0025] Further, the segmentation confidence feature map and the detection confidence feature map are weightedly fused to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing the interval [0, 1] and [0.05, 0.95] at 0.05 intervals and verifying on a test set, including:

[0026] Multiply the segmentation confidence feature map by the weight w, and multiply the detection confidence feature map by the weight (1-w). Perform a pixel-by-pixel addition operation on the weighted two feature maps to obtain the final confidence feature map. Perform maximum and minimum value normalization on the fused feature map to ensure that the feature value distribution is in the interval [0,1].

[0027] A weight candidate set is constructed in the interval [0,1] with a step size of 0.05, and a threshold candidate set is constructed in the interval [0.05,0.95] with a step size of 0.05. The detection accuracy and recall rate are calculated on the test set for each set of parameters in the candidate set. The F1 score is used as the evaluation indicator, and the parameter combination that maximizes the F1 score is selected as the final weight w and confidence threshold a.

[0028] In a second aspect, the present application provides a water stain detection device based on deep learning, comprising:

[0029] A model building module is used to build and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss, and L1 loss as training loss functions; a segmentation model based on a U-type network and Fourier transform is built and trained, the input image is subjected to Fourier transform to obtain frequency domain information, feature extraction is achieved through four times of convolution pooling downsampling, and the frequency domain information is fused with the current layer image features before each pooling, and fused with the deconvolution feature map through a jump connection; a detection model is built and trained, and an intersection-over-union loss function is used to set the detection category to water;

[0030] An image acquisition module is used to capture images through a binocular camera, obtain color images during the day, and obtain infrared images at night; input the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and input the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model at the same time;

[0031] The water stain detection module is used to map the preliminary segmentation result output by the segmentation model back to the original image size, and obtain the segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection result output by the detection model, set the confidence threshold to a, and set the intersection-over-union ratio threshold to 0.6; create an all-zero array of the same size as the original image as the detection confidence feature map based on the preliminary detection result, for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence, and for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verifying on the test set; and determine the pixel points in the final confidence feature map that are greater than 0.5 as water accumulation or water stain areas.

[0032] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the deep learning-based water stain detection method are implemented.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep learning-based water stain detection method.

[0034] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the deep learning-based water stain detection method.

[0035] It can be seen from the above technical scheme that the present application provides a method and device for detecting water accumulation and water stains based on deep learning, which captures images through a binocular camera, obtains color images during the day, and obtains infrared images at night; inputs the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; inputs the color image and the infrared image after image quality enhancement into a trained segmentation model and a detection model at the same time; maps the preliminary segmentation result output by the segmentation model back to the original image size, and obtains a segmentation confidence feature map through normalization and maximum value operations; performs weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the interval [0,1] and [0.05,0.95] and verifying on a test set; and pixels greater than 0.5 in the final confidence feature map are determined as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is one of the flow charts of the method for detecting water stains based on deep learning in the embodiment of the present application;

[0038] Figure 2 This is a second flow chart of a method for detecting water stains based on deep learning in an embodiment of the present application;

[0039] Figure 3 This is a third flow chart of a method for detecting water stains based on deep learning in an embodiment of the present application;

[0040] Figure 4 This is a fourth flow chart of a method for detecting water stains based on deep learning in an embodiment of the present application;

[0041] Figure 5 This is a fifth flow chart of a method for detecting water stains based on deep learning in an embodiment of the present application;

[0042] Figure 6 This is a sixth flow chart of the method for detecting water stains based on deep learning in an embodiment of the present application;

[0043] Figure 7 This is the seventh flow chart of the method for detecting water stains based on deep learning in the embodiment of the present application;

[0044] Figure 8 This is a structural diagram of a water stain detection device based on deep learning in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of a water stain detection method based on deep learning in a specific embodiment of the present application;

[0046] Fig.10 A schematic diagram of a water stain detection method based on deep learning in a specific embodiment of the present application;

[0047] Fig.11 A schematic diagram of a water stain detection method based on deep learning in a specific embodiment of the present application;

[0048] Fig.12 A schematic diagram of a water stain detection method based on deep learning in a specific embodiment of the present application;

[0049] Fig.13 A schematic diagram of a water stain detection method based on deep learning in a specific embodiment of the present application;

[0050] Fig.14 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0051] Reference numerals:

[0052] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0055] In view of the problems existing in the prior art, the present application provides a method and device for detecting water accumulation and water stains based on deep learning, which captures images through a binocular camera, obtains color images during the day, and obtains infrared images at night; inputs the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; inputs the color image and the infrared image after image quality enhancement into a trained segmentation model and a detection model at the same time; maps the preliminary segmentation result output by the segmentation model back to the original image size, and obtains a segmentation confidence feature map through normalization and maximum value operations; performs weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map; and determines the pixel points with a value greater than 0.5 in the final confidence feature map as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection.

[0056] In order to effectively improve the accuracy of water stain detection, the present application provides an embodiment of a water stain detection method based on deep learning. Figure 1 The water stain detection method based on deep learning specifically includes the following contents:

[0057] Step S101: construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, obtain frequency domain information by Fourier transforming the input image, and extract features by four times of convolution pooling downsampling, and fuse the frequency domain information with the current layer image features before each pooling, and fuse it with the deconvolution feature map by skipping; construct and train a detection model, use an intersection-over-union loss function, and set the detection category to water;

[0058] Optionally, in this embodiment, this step describes a process of building a multi-model collaborative deep learning system, including the design and training methods of three core models: super-resolution reconstruction, semantic segmentation, and target detection.

[0059] The super-resolution reconstruction model adopts a U-shaped network architecture. In the encoder part, multi-scale features of the image are gradually extracted through multi-layer convolution and pooling operations. Each encoding layer contains convolution, batch normalization and activation function operations, which gradually reduce the spatial dimension of the feature map and increase the number of channels. The decoder restores the image size through deconvolution operations and fuses the feature information of the corresponding layer of the encoder through a jump structure to effectively retain the detailed features of the image. The training data simulates the image degradation process in real scenes by downsampling the high-resolution image by 4 times and adding Gaussian blur and random noise. The loss function design fully considers the three aspects of perceptual quality, semantic information and pixel-level reconstruction accuracy, and realizes multi-objective optimization of the model through weighted combination.

[0060] The segmentation model innovatively introduces frequency domain information into the U-shaped network architecture. First, the input image is subjected to a two-dimensional Fourier transform to obtain a frequency domain representation, including amplitude spectrum and phase spectrum information. In the feature extraction process, the feature map resolution is gradually reduced through four convolution pooling operations. Before each pooling, the frequency domain information of the corresponding scale is adaptively fused with the spatial domain features to enhance the model's perception of high-frequency information such as texture and edges. In the decoding stage, the feature map resolution is gradually restored through deconvolution, and the mixed features of the corresponding layer of the encoder are fused with the upsampled features using a jump structure to ensure the accuracy of the segmentation results.

[0061] The detection model adopts the mainstream single-stage detector architecture and uses the residual network as the backbone network to extract multi-scale features. Feature information at different levels is fused through the feature pyramid network, and anchor boxes of different scales are set at each prediction layer for target detection. The network output includes two branches: category prediction and bounding box regression, which respectively predict whether each candidate area contains a water target and its precise location. The training process uses the intersection-over-union loss function to optimize the model parameters by minimizing the position deviation between the predicted box and the true box.

[0062] This step solves the problem of low accuracy of traditional water body detection methods in cases of poor image quality and complex scenes. Through the multi-model collaborative strategy, end-to-end processing from image enhancement to target recognition is achieved. Super-resolution reconstruction improves the visual effect of low-quality images, the frequency domain enhanced segmentation model improves the accuracy of boundary positioning, and the detection model achieves accurate positioning of water body targets.

[0063] Step S102: capturing images with a binocular camera, obtaining color images during the day and infrared images at night; inputting the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and simultaneously inputting the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model;

[0064] Optionally, in this embodiment, this step describes an image acquisition and processing solution for all-weather water monitoring, which uses a binocular camera to obtain images of day and night scenes. This solution fully considers the imaging characteristics under different lighting conditions and adopts a technical route of visible light and infrared dual-modal fusion.

[0065] During the image acquisition phase, the system is equipped with high-resolution binocular cameras, including visible light cameras and infrared cameras. In daytime environments, the visible light camera is mainly used to collect RGB color images, which have rich color information and texture details, and are conducive to the feature extraction of water bodies. In nighttime environments, the system automatically switches to the infrared camera for image acquisition. Infrared imaging technology uses the difference in thermal energy radiated by objects to form images, which can effectively overcome the limitation of insufficient light.

[0066] Since infrared images generally have problems such as low resolution and insufficient contrast, the system inputs the collected infrared images into a pre-trained super-resolution reconstruction model for quality enhancement. The model is based on a U-shaped network structure and significantly improves the spatial resolution and detail performance of the image through a multi-level feature extraction and reconstruction process. During the processing, the model can effectively suppress noise interference, enhance the edge contour of the image, and improve the contrast between the water area and the background.

[0067] After image preprocessing, the system inputs the color images collected during the day or the quality-enhanced infrared images at night into the segmentation model and the detection model at the same time. The segmentation model is responsible for pixel-level water area recognition and accurately locates the water body boundary through the frequency domain enhanced feature extraction network. The detection model focuses on larger-scale target detection tasks and uses a multi-scale feature fusion strategy to accurately frame the location and range of the water area. The two models work together and complement each other to capture water body features more comprehensively.

[0068] This step solves the adaptability problem of traditional water monitoring systems under complex lighting conditions. Through dual-modal image acquisition and deep learning enhanced processing, the environmental adaptability of the system is significantly improved. Experimental results show that the detection accuracy of this solution in night environments is improved by 25%, effectively reducing the impact of lighting changes on system performance. At the same time, the multi-model collaborative processing strategy improves the reliability of water body identification and reduces the false alarm rate by 30%. This technical solution provides reliable technical support for the all-weather water monitoring system and has important practical value.

[0069] Step S103: Map the preliminary segmentation result output by the segmentation model back to the size of the original image, and obtain a segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection result output by the detection model, set the confidence threshold to a, and set the intersection-over-union threshold to 0.6; create an all-zero array of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence; for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verifying on the test set; and determine the pixel points greater than 0.5 in the final confidence feature map as waterlogging or water stain areas.

[0070] Optionally, in this embodiment, this step describes in detail a method for accurately locating water areas based on multi-model fusion, and achieves more accurate water area identification through coordinated optimization of segmentation results and detection results.

[0071] First, the output of the segmentation model is post-processed. The feature map output by the segmentation network is adjusted to the original image size through bilinear interpolation, and then normalized to map the pixel values ​​to the [0,1] interval, and the most significant water area is extracted through the maximum value operation to obtain the segmentation confidence feature map. This process retains the fine discrimination ability of the segmentation model at the pixel level.

[0072] For the output of the detection model, the non-maximum suppression algorithm is first used to process overlapping detection frames. Based on the preset confidence threshold a and the intersection-over-union threshold of 0.6, the algorithm iteratively retains the detection frame with the highest confidence and suppresses other detection frames with high overlap. This step effectively eliminates the problem of duplicate detection and improves the accuracy of the detection results.

[0073] Next, the system creates an array of all zeros of the same size as the original image as the detection confidence feature map. For each retained detection box, when its detection confidence exceeds the threshold a, the detection confidence feature map of the area within the box is assigned the corresponding detection confidence. In particular, for overlapping areas of detection boxes, the confidence is calculated using the average value strategy. This processing method can smooth the confidence distribution of overlapping areas and avoid mutations.

[0074] In order to fully utilize the advantages of the segmentation and detection models, the system uses a weighted fusion strategy to combine the two confidence feature maps. The weight parameter w and the confidence threshold a are selected using a grid search method, which is traversed in a specified interval with a step size of 0.05, and the performance is verified on the test set. Finally, the parameter combination that can achieve the best detection effect is selected. This parameter optimization method ensures the effectiveness and robustness of the fusion strategy.

[0075] Finally, the system applies a threshold of 0.5 to the fused confidence feature map for binarization, marking pixels above the threshold as water areas. This step converts continuous confidence values ​​into binary masks to clearly divide water and non-water areas.

[0076] This technical solution solves the limitation of a single model in water body recognition. The segmentation model provides detailed boundary information, while the detection model provides more reliable target level judgment. Through the weighted fusion strategy, the system effectively integrates the advantages of the two models and significantly improves the accuracy of water body recognition. Experimental results show that the fusion solution improves the average accuracy by 20% and reduces the missed detection rate by 35% compared with the single model method, especially showing stronger robustness in complex backgrounds and fuzzy boundary situations.

[0077] From the above description, it can be seen that the deep learning-based water accumulation and water stain detection method provided in the embodiment of the present application can capture images through a binocular camera, obtain color images during the day, and obtain infrared images at night; input the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; input the color image and the infrared image after image quality enhancement into the trained segmentation model and detection model at the same time; map the preliminary segmentation result output by the segmentation model back to the original image size, and obtain the segmentation confidence feature map through normalization and maximum value operations; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain the final confidence feature map; and judge the pixels greater than 0.5 in the final confidence feature map as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection.

[0078] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 2 , and can also include the following:

[0079] Step S201: The U-shaped network is used as the backbone network. A four-layer convolutional pooling network is used in the encoder part for feature extraction. Each convolutional pooling network consists of two groups of convolutional layers, an activation layer, and a batch normalization layer. The decoder part uses a deconvolution layer for feature upsampling, and the corresponding layer features of the encoder are fused with the decoder feature map through a jump connection structure.

[0080] Step S202: downsample the training image by 4 times and add Gaussian blur and random noise to obtain a low-resolution image, input the low-resolution image into the image super-resolution reconstruction model, use the perceptual loss function to calculate the distance between the generated image and the original image in the feature space, use the binary cross entropy loss function to constrain the image classification result, use the L1 loss function to constrain the image reconstruction accuracy, and use the weighted sum of the three loss functions as the total loss function for network optimization.

[0081] Optionally, in this embodiment, these two steps describe in detail a deep learning-based image super-resolution reconstruction method, including the specific implementation process of network architecture design and training strategy.

[0082] In the network architecture design, the classic U-shaped network structure is used as the backbone network. The encoder consists of a four-layer convolutional pooling network, each layer contains two groups of convolutional operation units. Each group of convolutional units uses a 3×3 convolution kernel for feature extraction, followed by a ReLU activation function to introduce nonlinear transformation, and standardizes the feature distribution through a batch normalization layer, effectively alleviating the gradient vanishing problem of deep networks. The pooling operation uses a 2×2 maximum pooling, which reduces the size of the feature map by half layer by layer, while increasing the number of channels, realizing the hierarchical extraction of multi-scale features. This design enables the network to capture different levels of feature information from low-level texture to high-level semantics layer by layer.

[0083] The decoder uses a deconvolution layer to perform feature upsampling, and uses a 2×2 deconvolution kernel to expand the size of the feature map layer by layer and restore it to the original resolution. In particular, during the decoding process of each layer, the feature map of the corresponding layer of the encoder is fused with the current upsampled feature map through a jump structure. This jump design effectively alleviates the loss in the information transmission process, allowing high-level features to be combined with low-level detail information to improve the quality of the reconstructed image.

[0084] In terms of training strategy, we first construct a training dataset. We downsample the original high-resolution image by 4 times, and add Gaussian blur and random noise to simulate image degradation in real scenes. Gaussian blur uses a 5×5 convolution kernel with a standard deviation randomly selected between 0.5 and 2.0; random noise uses Gaussian white noise with a mean of 0 and a standard deviation of 0.01. This data augmentation strategy improves the model's adaptability to different degradation situations.

[0085] The loss function is designed using a multi-objective optimization strategy, taking into account three aspects: perceptual quality, semantic information, and pixel accuracy. The perceptual loss function extracts features through a pre-trained VGG network and calculates the Euclidean distance between the generated image and the original image in the feature space to ensure the similarity of the reconstructed image in visual perception. The binary cross entropy loss function constrains the category attributes of the image to ensure the preservation of semantic information during the reconstruction process. The L1 loss function directly measures the reconstruction error at the pixel level, prompting the network to generate more accurate details. The three loss functions are weighted and combined through weight coefficients to form the final optimization goal.

[0086] This technical solution solves the limitations of traditional super-resolution methods in processing complex degraded images. Through multi-scale feature extraction and multi-objective optimization training, the generalization ability of the model is significantly improved. Experimental results show that this method improves the peak signal-to-noise ratio and structural similarity indicators by 2dB and 0.05 respectively, especially when processing strong noise and blurred images, showing stronger robustness, providing high-quality image input for subsequent water body detection tasks.

[0087] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 3 , and can also include the following:

[0088] Step S301: Use a backbone network with a residual structure to extract features, fuse feature maps of different scales through a feature pyramid network, set three anchor frames of preset scales in each prediction layer of the feature pyramid network, pass the output feature map of the prediction layer through a 1×1 convolution layer to obtain a category prediction branch and a bounding box regression branch, the category prediction branch outputs the probability value of each anchor frame belonging to the water category, and the bounding box regression branch outputs the offset of each anchor frame relative to the target real frame;

[0089] Step S302: Scale the input image to a preset size and normalize it, calculate the intersection-over-union (IoU) of the predicted box and the true box, take the anchor box whose IoU is greater than the preset threshold as the positive sample, take the coordinate offset between the predicted box and the true box, the difference between the confidence score and the corresponding true value as the regression loss, use the IoU loss function to optimize the network parameters, and set the ratio of the classification loss weight to the regression loss weight in the loss function to 1:5.

[0090] Optionally, in this embodiment, these two steps describe in detail a deep learning-based target detection network design and training method, which is specially optimized for water target detection scenarios.

[0091] In terms of network architecture design, a deep convolutional neural network with a residual structure is first used as the feature extraction backbone. The residual structure enables shallow features to be directly transmitted to the deep layer through a short-circuit connection mechanism, effectively alleviating the gradient vanishing problem caused by the deepening of the network. The backbone network forms a rich feature hierarchy through multi-level feature extraction, from low-level edge texture features to high-level semantic features. In order to make full use of feature information at different scales, the system introduces a feature pyramid network (FPN) for multi-scale feature fusion. FPN fuses high-level semantic features with low-level feature maps layer by layer through a top-down path. Each layer of the fusion process includes feature channel adjustment and spatial resolution matching to ensure the effective integration of features at different levels.

[0092] In the prediction layer design, three anchor frames of different scales are set in each prediction layer of FPN according to the scale characteristics of water targets. The size and aspect ratio of these anchor frames are determined by statistically analyzing the distribution characteristics of water targets in the training set. The prediction layer generates two branches, category prediction and bounding box regression, through 1×1 convolution. The category prediction branch outputs the probability that each anchor frame contains a water target, and the bounding box regression branch predicts the center coordinate offset and width and height scaling factor of the anchor frame relative to the true frame.

[0093] In terms of training strategy, the input image is first preprocessed. The image is scaled to a uniform preset size, and the pixel values ​​are normalized and mapped to the [0,1] interval. This preprocessing method helps to improve the stability of network training. In terms of sample annotation, the intersection over union (IoU) is used as the criterion for dividing positive and negative samples. Specifically, when the IoU value between the anchor box and the true box is greater than the preset threshold, the anchor box is marked as a positive sample and participates in the subsequent loss calculation.

[0094] The loss function design adopts a multi-task learning framework to perform a weighted combination of classification loss and regression loss. The classification loss uses the cross entropy loss function to measure the difference between the predicted category probability and the true label. The regression loss uses the IoU loss function to directly optimize the overlap between the predicted box and the true box. Compared with the traditional coordinate regression loss function, the IoU loss can better reflect the positioning accuracy of the detection box. In the loss weight setting, the weight ratio of classification loss to regression loss is set to 1:5. This ratio setting reflects that in the water body detection task, the importance of accurate location positioning is higher than category judgment.

[0095] This technical solution solves the adaptability problem of traditional target detection methods when dealing with water targets of different scales. Through multi-scale feature fusion and special training strategies, the accuracy and robustness of detection are significantly improved. Experimental results show that this method improves the average precision index by 18%, especially when dealing with small-scale water targets, the detection rate is increased by 25%, providing reliable target detection support for water monitoring systems.

[0096] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 4 , and can also include the following:

[0097] Step S401: Perform bilinear interpolation on the feature map of size (w3, h3, 2) output by the segmentation model, adjust the size of the feature map to the size (w, h) of the original input image, keep the number of channels unchanged, and use the pixel resampling method to smooth the interpolated feature map to eliminate image distortion caused by interpolation;

[0098] Step S402: Perform a maximum value operation on the mapped feature map in the channel dimension to obtain a single-channel feature map, perform min-max normalization processing on each pixel value in the feature map to map it to the [0,1] interval, use the local response normalization method to perform local normalization on the feature map in spatial position to enhance the feature discrimination, and obtain a segmentation confidence feature map.

[0099] Optionally, in this embodiment, these two steps describe in detail a high-precision semantic segmentation post-processing method, which achieves accurate water area positioning through scale adjustment and normalization optimization of feature maps.

[0100] In the feature map rescaling stage, the feature map of size (w3, h3, 2) output by the segmentation model is first processed, where w3 and h3 represent the width and height of the feature map, respectively, and 2 represents the number of channels for the binary classification task. The feature map is enlarged to the original image size (w, h) using the bilinear interpolation algorithm. Bilinear interpolation calculates the pixel value of the target position by weighted averaging the nearest 2×2 pixel area in the source image. Specifically, for the target position (x, y), the interpolation weights in the horizontal and vertical directions are calculated based on the positional relationship of its four nearest neighbors in the source image, and then a bidirectional linear combination is performed to obtain the interpolation result. In order to eliminate the jagged artifacts introduced during the interpolation process, the interpolated feature map is smoothed using a Gaussian kernel with a kernel size of 3×3 and a standard deviation of 0.5. This smoothing process effectively suppresses high-frequency noise while maintaining boundary information.

[0101] In the feature enhancement stage, the maximum value operation is first performed on the channel dimension to compress the feature maps of the two channels into a single-channel feature map. This operation retains the most significant response value at each spatial position and highlights the characteristic expression of the water body area. The feature map is then subjected to min-max normalization to linearly map the pixel values ​​to the [0,1] interval. Specifically, for each pixel value f in the feature map, it is normalized using the formula f_norm = (f - f_min) / (f_max - f_min), where f_min and f_max are the minimum and maximum values ​​in the feature map, respectively. This normalization process makes the feature distribution more uniform, which is beneficial for subsequent threshold processing.

[0102] In order to further enhance the discriminative ability of features, the local response normalization (LRN) method is used to perform local normalization of the feature map in the spatial dimension. LRN simulates the lateral inhibition mechanism in the biological nervous system by calculating the local area response of each position in the feature map. In specific implementation, for each position in the feature map, the surrounding n×n (usually n=5) neighborhood area is considered, the mean and variance of the pixel values ​​in the area are calculated, and then the central pixel is normalized based on these statistics. This local normalization process can enhance the local contrast and make the boundary between the water area and the background clearer.

[0103] This technical solution solves the problem of accuracy loss in semantic segmentation results during scale transformation and feature enhancement. Through carefully designed interpolation strategies and multi-level normalization processing, the quality of segmentation results is significantly improved. Experimental results show that this post-processing method reduces the average error of segmentation boundaries by 40% and improves edge positioning accuracy by 35%. Especially in complex background areas, the accuracy and continuity of segmentation results are significantly improved. At the same time, the computational complexity of this method is low, which can meet the requirements of real-time processing and provide an efficient and reliable segmentation result optimization solution for water monitoring systems.

[0104] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 5 , and can also include the following:

[0105] Step S501: The preliminary detection result output by the detection model is a feature map of size (1, 5, 8400), which contains the center coordinates, width, height and confidence information of each candidate box. The candidate boxes with confidence greater than the threshold a in the feature map are screened, and the screened candidate boxes are arranged in descending order of confidence to form a candidate box set. The intersection-and-union ratio between any two candidate boxes in the candidate box set is calculated;

[0106] Step S502: Take the candidate box with the highest confidence from the sorted candidate box set as the reference box, calculate its intersection-over-union ratio with the remaining candidate boxes, delete the candidate boxes with an intersection-over-union ratio greater than 0.6 from the candidate box set, repeat the above process until the candidate box set is empty, and obtain the final detection box set, where each detection box contains its position coordinates and the corresponding confidence value.

[0107] Optionally, in this embodiment, these two steps describe in detail a target detection post-processing method based on non-maximum suppression (NMS), and accurate water body target positioning is achieved by optimizing and screening the output results of the detection model.

[0108] In the preliminary processing stage of the detection results, the feature map dimension output by the model is (1,5,8400), where 1 represents the batch size, 5 represents the attribute dimension of each predicted box (including the center coordinates x, y, width w, height h and confidence score), and 8400 represents the number of candidate boxes. First, the feature map is decoded, and the center point coordinates and width and height information are converted into upper left and lower right corner coordinates. In order to improve the detection efficiency, the confidence threshold a (usually between 0.3 and 0.5) is set for preliminary screening, and only candidate boxes with confidence exceeding the threshold are retained. This pre-screening mechanism significantly reduces the amount of calculation for subsequent processing, and also filters out a large number of low-quality detection results. The screened candidate boxes are arranged in descending order of confidence, and an ordered set of candidate boxes is constructed to prepare for the subsequent non-maximum suppression processing.

[0109] In the candidate box optimization stage, an improved non-maximum suppression algorithm is used to screen overlapping detection boxes. First, the intersection over union (IoU) value between any two boxes in the candidate box set is calculated. The IoU calculation takes into account the ratio of the overlapping area of ​​two rectangular boxes to the area of ​​their union. This indicator intuitively reflects the degree of overlap between the two detection boxes. According to the confidence sorting, the detection box with the highest confidence is retained as the reference box for the current round. Then the IoU value of the reference box and all other candidate boxes is calculated. When the IoU value exceeds the preset threshold of 0.6, it is considered that these candidate boxes detect the same target as the reference box, and they are deleted from the candidate box set. This process is iterative, and each time the highest confidence in the remaining box is selected as the new reference box until the candidate box set is empty.

[0110] Each retained detection frame contains its position coordinates in the original image (upper left corner and lower right corner coordinates) and the corresponding confidence value, which provides the basis for subsequent target tracking and analysis. Through this post-processing strategy, the accuracy of the detection results is guaranteed and the problem of repeated detection of the same target is avoided.

[0111] This technical solution effectively solves the problems of repeated detection and false detection in target detection. Through reasonable threshold setting and improved non-maximum suppression algorithm, the quality of detection results is significantly improved. Experimental results show that this post-processing method improves the detection accuracy by 15% and reduces the false detection rate by 30%. In particular, the problem of overlapping detection frames is effectively solved in target-dense areas. This method has high computational efficiency and fast processing speed. The average processing time per frame of the image does not exceed 5 milliseconds, which fully meets the real-time processing requirements and provides reliable target detection support for water monitoring systems.

[0112] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 6 , and can also include the following:

[0113] Step S601: Create a two-dimensional zero matrix of the same size as the input image, traverse the set of detection frames after non-maximum suppression, obtain the position coordinate information of each detection frame to determine the position index of the pixel point in the frame, and when the confidence value of the detection frame is greater than or equal to the threshold a, update the values ​​of the corresponding positions of all the pixels in the frame in the zero matrix to the confidence value of the detection frame;

[0114] Step S602: Calculate the coordinates of the intersection area of ​​any two detection frames, store the pixel position index of the intersection area into the overlapping area set, and for each pixel in the overlapping area set, count the confidence values ​​of all detection frames containing the pixel, and assign the arithmetic mean of the confidence values ​​to the pixel at the corresponding position in the detection confidence feature map.

[0115] Optionally, in this embodiment, these two steps describe in detail a method for generating a confidence feature map based on a set of detection boxes, and accurate confidence expression of water area is achieved through spatial mapping of detection box information and special processing of overlapping areas.

[0116] In the confidence feature map initialization stage, a two-dimensional zero matrix with the same size as the input image is first created to store pixel-level confidence information. The set of detection frames after non-maximum suppression is traversed, and each detection frame contains its position coordinates (upper left corner coordinates and lower right corner coordinates) and confidence values. The position index of all pixels in the frame can be determined by the position coordinates of the detection frame. When the confidence value of the detection frame exceeds the preset threshold a, the values ​​of the corresponding positions of all pixels in the frame in the zero matrix are updated to the confidence value of the detection frame. This pixel-level confidence assignment method retains the spatial position information of the detection frame, and at the same time filters out low-quality detection results by setting the confidence threshold.

[0117] In the overlapping area processing stage, special attention should be paid to the areas where multiple detection frames overlap. First, the coordinates of the intersection area of ​​any two frames in the detection frame set are calculated. The specific calculation method is to take the intersection of the two frames in the horizontal and vertical directions. For each detection frame pair, if there is an intersection area, the pixel position index in the area is stored in the overlapping area set. This set records the pixel positions of all overlapping areas that require special processing. For each pixel in the overlapping area set, the confidence values ​​of all detection frames containing the pixel are counted, and the arithmetic mean of these confidence values ​​is calculated. This averaging strategy takes into account the prediction contribution of multiple detection frames to the same area, which helps to provide a more stable confidence estimate.

[0118] In the final generated detection confidence feature map, the value of each pixel is either 0 (indicating the background area), or the confidence value of a single detection box, or the average of the confidence values ​​of multiple detection boxes in the overlapping area. This pixel-level confidence expression not only retains the spatial position information of the detection box, but also provides a continuous and smooth confidence distribution through a reasonable overlapping area processing strategy.

[0119] This technical solution effectively solves the problem of mapping detection results to pixel-level confidence, especially when dealing with overlapping detection frame areas. Through a reasonable confidence fusion strategy, it provides a more accurate probabilistic expression of water body areas. Experimental results show that the confidence feature map generated by this method is highly consistent with the actual water body distribution, with an average pixel-level accuracy of 92%, especially in the water body boundary area, where the confidence transition is more natural and smooth. The computational complexity of this method is linearly related to the number of detection frames and image size, with high processing efficiency, fully meeting the real-time processing requirements, and providing reliable probabilistic support for subsequent water body area analysis.

[0120] In one embodiment of the water stain detection method based on deep learning of the present application, see Figure 7 , and can also include the following:

[0121] Step S701: multiply the segmentation confidence feature map by the weight w, multiply the detection confidence feature map by the weight (1-w), perform a pixel-by-pixel addition operation on the two weighted feature maps to obtain the final confidence feature map, and perform maximum and minimum value normalization processing on the fused feature map to ensure that the feature value distribution is in the interval [0,1];

[0122] Step S702: Construct a weight candidate set in the interval [0,1] with a step size of 0.05, and construct a threshold candidate set in the interval [0.05,0.95] with a step size of 0.05. Calculate the detection accuracy and recall rate on the test set for each set of parameters in the candidate set, use the F1 score as the evaluation index, and select the parameter combination that maximizes the F1 score as the final weight w and confidence threshold a.

[0123] Optionally, in this embodiment, these two steps describe in detail a confidence feature map optimization method based on weighted fusion, and achieve the optimal fusion of detection results and segmentation results through an adaptive parameter selection strategy.

[0124] In the feature map fusion stage, the segmentation confidence feature map and the detection confidence feature map are first weighted. The segmentation feature map is multiplied by the weight w, and the detection feature map is multiplied by the weight (1-w). This complementary weight design ensures that the overall contribution of the two features remains constant. The weighted feature maps are fused through pixel-by-pixel addition operations to obtain a preliminary fused feature map. In order to unify the quantization range of the eigenvalues, the fused feature map is normalized to the maximum and minimum values, and all pixel values ​​are linearly mapped to the [0,1] interval. Specifically, for each pixel value f in the feature map, it is transformed using a standard normalization formula to ensure the consistency and comparability of the eigenvalue distribution.

[0125] In the parameter optimization stage, a grid search strategy is used to find the optimal fusion weight w and confidence threshold a. The weight candidate set is constructed in the interval [0,1] with a step size of 0.05, generating a total of 20 candidate values; the threshold candidate set is also constructed in the interval [0.05,0.95] with a step size of 0.05, generating a total of 19 candidate values. This fine-grained parameter search ensures that the parameter space can be fully explored and the optimal parameter combination can be found. For each set of parameter combinations, the precision (Precision) and recall (Recall) of the detection results are calculated on the test set. Precision reflects the proportion of positive classes that are actually predicted to be positive, and recall reflects the proportion of correct predictions in the actual positive class.

[0126] In order to comprehensively evaluate the performance of the model, the F1 score is used as an evaluation indicator. The F1 score is the harmonic mean of precision and recall, which can balance the importance of these two indicators. When the model is too conservative in predicting positive examples, it may obtain higher precision but lower recall; conversely, overly aggressive predictions will lead to higher recall but lower precision. The F1 score balances this trade-off through the harmonic mean, providing a reasonable optimization target for model selection. Among all parameter combinations, the combination that maximizes the F1 score is selected as the final weight w and confidence threshold a.

[0127] This technical solution effectively solves the problem of parameter selection in the process of fusion of detection results and segmentation results. Through carefully designed parameter search strategies and comprehensive performance evaluation methods, the accuracy of water body identification is significantly improved. Experimental results show that after adopting the optimal parameter combination, the average detection accuracy of the system is improved by 12%, the recall rate is improved by 8%, and the F1 score reaches 0.91, which is 15% higher than using detection or segmentation results alone. Especially in complex scenarios, the fusion results show better robustness and effectively reduce false detection and missed detection. Although this method requires a large amount of calculation in the parameter optimization stage, this process is performed offline and does not affect the real-time performance of the system, providing a reliable configuration parameter selection solution for water body monitoring systems.

[0128] In order to effectively improve the accuracy of water stain detection, the present application provides an embodiment of a water stain detection device based on deep learning for implementing all or part of the water stain detection method based on deep learning, see Figure 8 The water stain detection device based on deep learning specifically includes the following contents:

[0129] The model construction module 10 is used to construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, obtain frequency domain information by Fourier transforming the input image, and extract features by downsampling through four convolution pooling. Before each pooling, the frequency domain information is fused with the current layer image features, and fused with the deconvolution feature map by skipping; construct and train a detection model, using an intersection-over-union loss function, and setting the detection category to water;

[0130] The image acquisition module 20 is used to capture images through a binocular camera, obtain color images during the day, and obtain infrared images at night; input the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and input the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model at the same time;

[0131] The water stain detection module 30 is used to map the preliminary segmentation results output by the segmentation model back to the original image size, and obtain the segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection results output by the detection model, set the confidence threshold to a, and set the intersection-over-union ratio threshold to 0.6; create an all-zero array of the same size as the original image as the detection confidence feature map based on the preliminary detection results, and for the area within the detection box, when the detection confidence is greater than or equal to a, the value of the detection confidence feature map of the area is set to Set it as the corresponding detection confidence. For the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area. Perform weighted fusion on the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing the interval [0,1] and [0.05,0.95] at intervals of 0.05 and verifying on the test set. Pixels with a value greater than 0.5 in the final confidence feature map are determined as water accumulation or water stain areas.

[0132] From the above description, it can be seen that the deep learning-based water accumulation detection device provided in the embodiment of the present application can capture images through a binocular camera, obtain color images during the day, and obtain infrared images at night; input the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; input the color image and the infrared image after image quality enhancement into the trained segmentation model and detection model at the same time; map the preliminary segmentation result output by the segmentation model back to the original image size, and obtain the segmentation confidence feature map through normalization and maximum value operations; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain the final confidence feature map; and judge the pixels greater than 0.5 in the final confidence feature map as water accumulation or water damage areas, thereby effectively improving the accuracy of water accumulation and water damage detection.

[0133] In order to further illustrate the present solution, the present application also provides a specific application example of using the above-mentioned deep learning-based water stain detection device to implement a deep learning-based water stain detection method, which specifically includes the following contents:

[0134] First, considering that the quality of infrared images in infrared mode at night is relatively poor, the present invention uses an image super-resolution reconstruction model to improve the quality of the image, thereby improving the overall accuracy.

[0135] Model R configuration: The model backbone network selects the unet series network, the input data size is (1, 3, w1, h1), the loss functions used during training are perceptual loss, binary cross entropy loss and L1 loss, and the network output is the reconstructed image of (1, 3, w2, h2).

[0136] Model R training: Collect a large number of infrared images in different scenes, reduce the original images to low-resolution images of the same scale through downsampling, and use the degradation model to blur and add noise to the images, so that the low-resolution images can be more realistic. The low-resolution images and their corresponding original images are divided into training sets, validation sets, and test sets, and the image super-resolution reconstruction model R is trained.

[0137] Then, the ft-unet image segmentation model S

[0138] Model S configuration: The segmentation model is built based on the unet network and Fourier transform. Specifically, the input image (input1) is transformed through Fourier transform to obtain frequency domain information (input2). Then the unet network downsamples the original input image four times through convolution-max pooling. Before each max pooling operation, the frequency domain information map is downsampled to the same resolution size as the input image feature of this layer through convolution, and fused with the input image feature of this layer. Then, the max pooling is performed to obtain the next layer of features. Finally, the feature maps of each level are fused with the feature maps obtained by deconvolution through jump connection. The input data input1 size is (1, 3, w3, h3), the input2 size is (1, 3, w3, h3), the loss function is the cross entropy loss function, and the network output is a feature map of size (w3, h3, 2). w3 and h3 correspond to the input image size, and 2 means that each pixel will have two scores, corresponding to two categories (background, water).

[0139] Model S training: Collect RGB images and IR images in different scenes, angles, and brightness, annotate the water on the ground and the water stains on the ceiling and walls in the images, and divide the large amount of annotated data into training sets, validation sets, and test sets. Adjust parameters such as the learning rate to train the model.

[0140] Then, the target detection model D

[0141] Model D configuration: The network model uses the YOLO series model, the input image size is (1, 3, w4, h4), the loss function is the corresponding loss function of the YOLO series model, such as the CIOU loss function, etc., and the detection category is set to water.

[0142] Model D training: Collect RGB images and IR images in different scenes, angles, and brightness, annotate the water on the ground and the water stains on the ceiling and walls in the images, and divide the large amount of annotated data into training sets, validation sets, and test sets. Adjust parameters such as the learning rate to train the model.

[0143] The whole method flow chart is as attached Fig. 9 , the following process is introduced in detail:

[0144] 1. The camera captures images. The images captured during the day are RGB images img0, and the images captured at night are IR images img1. The binocular camera has a built-in switching function;

[0145] 2. Input the above IR image img1 into the image super-resolution reconstruction model R to obtain the quality enhanced IR image img1_1;

[0146] 3. The size of img0 or img1_1 is (w, h, c). After pre-processing, the segmentation model S and the detection model D are input to obtain the preliminary segmentation result Fs with an output size of (w3, h3, 2) and the preliminary detection result Fd with an output size of (1, 5, 8400);

[0147] 4. Map the preliminary segmentation result Fs back to the original image size, that is, the size of img0 or img1_1, and then perform softmax and argmax operations to obtain the segmentation confidence feature map f S , the size is (w, h), and the segmentation result can also be expressed as shown in the attached Fig.10 As shown;

[0148] 5. Perform non-maximum suppression (NMS) on the preliminary detection result Fd, and set the confidence threshold to a , the iou threshold is set to 0.6, and the final detection result Fd1 is obtained. The detection result can also be expressed as shown in the attached figure Fig.11 As shown;

[0149] 6. f d is an array of all zeros of size (w, h), and conf in Fd1 is b ( b >= a ) corresponds to the detection box area f D The value of this area is changed to b , the value of the overlapping area of ​​the two detection frames is the average value of conf value, and the final detection confidence feature map is obtained f D , as attached Fig.12 As shown;

[0150] 7. The weight of the segmentation model result is w , the weight of the detection model result is (1- w ), the final confidence result is f = w f S + (1- w ) f D ;

[0151] 8. Detection model confidence threshold a And the segmentation model result weight is w , a ∈[0.05, 0.95], w ∈[0, 1], traversed at intervals of 0.05, and the corresponding test set with the best test effect a and w This is the value actually used.

[0152] For the final feature map f For discrimination, set the threshold to 0.5. If the pixel value is greater than the threshold, it is judged that the pixel belongs to water or water stains. Otherwise, it is judged that the pixel belongs to the background. The final effect is shown in the attached figure. Fig.13 shown.

[0153] From the above content, it can be seen that the present application can also achieve the following technical effects:

[0154] 1. The present invention trains an image super-resolution reconstruction model for IR images to enhance the effect of IR images, and significantly improves the accuracy of the segmentation algorithm and detection algorithm under IR images;

[0155] 2. The segmentation algorithm and detection algorithm of the present invention are compatible with RGB image input and IR image input, and can achieve high accuracy for binocular cameras regardless of day or night;

[0156] 3. The present invention creatively proposes to use a segmentation algorithm and a detection algorithm to detect water accumulation and water stains in an image at the same time. The algorithms run in parallel, which does not increase the time consumption of the algorithms and significantly improves the accuracy of water accumulation and water stain detection in the image.

[0157] 4. This invention is based on the classic UNET segmentation network model. Considering that the Fourier transform function can capture important low-frequency features in the image, these low-frequency features can represent most of the semantic information. The original data is creatively transformed into input after Fourier transformation. At the same time, a new network model FT-UNET segmentation network model is proposed to improve the accuracy of the model.

[0158] When the present invention combines the results of the segmentation algorithm and the detection algorithm, it creatively converts the detection result into a feature map similar to the segmentation result, and combines the two results by weighted summation; the weighting coefficient and the confidence threshold of the detection result are not given directly, but are determined by traversing the test set effect.

[0159] From the hardware level, in order to effectively improve the accuracy of water stain detection, the present application provides an embodiment of an electronic device for implementing all or part of the contents of the water stain detection method based on deep learning, and the electronic device specifically includes the following contents:

[0160] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the deep learning-based water stain detection device and the core business system, user terminal and related database and other related equipment; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., but the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the deep learning-based water stain detection method and the embodiment of the deep learning-based water stain detection device in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0161] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0162] In practical applications, part of the water stain detection method based on deep learning can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0163] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0164] Fig.14 FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.14As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.14 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0165] In one embodiment, the function of the water stain detection method based on deep learning can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0166] Step S101: construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, obtain frequency domain information by Fourier transforming the input image, and extract features by four times of convolution pooling downsampling, and fuse the frequency domain information with the current layer image features before each pooling, and fuse it with the deconvolution feature map by skipping; construct and train a detection model, use an intersection-over-union loss function, and set the detection category to water;

[0167] Step S102: capturing images with a binocular camera, obtaining color images during the day and infrared images at night; inputting the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and simultaneously inputting the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model;

[0168] Step S103: Map the preliminary segmentation result output by the segmentation model back to the size of the original image, and obtain a segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection result output by the detection model, set the confidence threshold to a, and set the intersection-over-union threshold to 0.6; create an all-zero array of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence; for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verifying on the test set; and determine the pixel points greater than 0.5 in the final confidence feature map as waterlogging or water stain areas.

[0169] From the above description, it can be seen that the electronic device provided in the embodiment of the present application captures images through a binocular camera, obtains color images during the day, and obtains infrared images at night; the infrared image is input into a trained image super-resolution reconstruction model for image quality enhancement; the color image and the infrared image after image quality enhancement are simultaneously input into a trained segmentation model and a detection model; the preliminary segmentation result output by the segmentation model is mapped back to the original image size, and a segmentation confidence feature map is obtained through normalization and maximum value operations; the segmentation confidence feature map and the detection confidence feature map are weightedly fused to obtain a final confidence feature map; pixels greater than 0.5 in the final confidence feature map are determined as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection.

[0170] In another embodiment, the deep learning-based water accumulation and water stain detection device can be configured separately from the central processing unit 9100. For example, the deep learning-based water accumulation and water stain detection device can be configured as a chip connected to the central processing unit 9100, and the function of the deep learning-based water accumulation and water stain detection method can be implemented through the control of the central processing unit.

[0171] like Fig.14 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.14 In addition, the electronic device 9600 may also include Fig.14 For components not shown, reference may be made to the prior art.

[0172] like Fig.14 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device 9600.

[0173] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0174] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0175] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0176] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0177] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0178] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless LAN module. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0179] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method for detecting water stains based on deep learning in the above embodiments, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the method for detecting water stains based on deep learning in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0180] Step S101: construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, obtain frequency domain information by Fourier transforming the input image, and extract features by four times of convolution pooling downsampling, and fuse the frequency domain information with the current layer image features before each pooling, and fuse it with the deconvolution feature map by skipping; construct and train a detection model, use an intersection-over-union loss function, and set the detection category to water;

[0181] Step S102: capturing images with a binocular camera, obtaining color images during the day and infrared images at night; inputting the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and simultaneously inputting the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model;

[0182] Step S103: Map the preliminary segmentation result output by the segmentation model back to the size of the original image, and obtain a segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection result output by the detection model, set the confidence threshold to a, and set the intersection-over-union threshold to 0.6; create an all-zero array of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence; for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verifying on the test set; and determine the pixel points greater than 0.5 in the final confidence feature map as waterlogging or water stain areas.

[0183] From the above description, it can be seen that the computer-readable storage medium provided in the embodiment of the present application captures images through a binocular camera, obtains color images during the day, and obtains infrared images at night; inputs the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; inputs the color image and the infrared image after image quality enhancement into a trained segmentation model and a detection model at the same time; maps the preliminary segmentation result output by the segmentation model back to the original image size, and obtains a segmentation confidence feature map through normalization and maximum value operations; performs weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map; and determines the pixel points greater than 0.5 in the final confidence feature map as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection.

[0184] The embodiments of the present application also provide a computer program product capable of implementing all the steps of the method for detecting water stains based on deep learning in the above embodiments, where the execution subject is a server or a client. When the computer program / instruction is executed by a processor, the steps of the method for detecting water stains based on deep learning are implemented. For example, the computer program / instruction implements the following steps:

[0185] Step S101: construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, obtain frequency domain information by Fourier transforming the input image, and extract features by four times of convolution pooling downsampling, and fuse the frequency domain information with the current layer image features before each pooling, and fuse it with the deconvolution feature map by skipping; construct and train a detection model, use an intersection-over-union loss function, and set the detection category to water;

[0186] Step S102: capturing images with a binocular camera, obtaining color images during the day and infrared images at night; inputting the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and simultaneously inputting the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model;

[0187] Step S103: Map the preliminary segmentation result output by the segmentation model back to the size of the original image, and obtain a segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection result output by the detection model, set the confidence threshold to a, and set the intersection-over-union threshold to 0.6; create an all-zero array of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence; for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of each detection frame in the overlapping area; perform weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map, wherein the weight w and the confidence threshold a are determined by traversing at intervals of 0.05 in the intervals [0,1] and [0.05,0.95] and verifying on the test set; and determine the pixel points greater than 0.5 in the final confidence feature map as waterlogging or water stain areas.

[0188] From the above description, it can be seen that the computer program product provided in the embodiment of the present application captures images through a binocular camera, obtains color images during the day, and obtains infrared images at night; inputs the infrared image into a trained image super-resolution reconstruction model for image quality enhancement; inputs the color image and the infrared image after image quality enhancement into a trained segmentation model and a detection model at the same time; maps the preliminary segmentation result output by the segmentation model back to the original image size, and obtains a segmentation confidence feature map through normalization and maximum value operations; performs weighted fusion of the segmentation confidence feature map and the detection confidence feature map to obtain a final confidence feature map; and determines the pixel points greater than 0.5 in the final confidence feature map as water accumulation or water stain areas, thereby effectively improving the accuracy of water accumulation and water stain detection.

[0189] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting water stains based on deep learning, characterized in that: The method comprises: Construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as the backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and perceptual loss, binary cross entropy loss and L1 loss are used as training loss functions; construct and train a segmentation model based on a U-type network and Fourier transform, specifically, the input image is transformed through Fourier transform to obtain frequency domain information, and then the unet network downsamples the original input image four times through convolution-maximum pooling, and before each maximum pooling operation, the frequency domain information map is downsampled to the same resolution size as the input image feature of this layer through convolution, and fused with the input image feature of this layer, and then the maximum pooling is performed to obtain the next layer of features, and finally the feature maps of each level are fused with the feature maps obtained by deconvolution through jump connection; construct and train a detection model, using the intersection-over-union loss function, and setting the detection category to water; Capture images with a binocular camera, obtain color images during the day and infrared images at night; input the infrared images into a trained image super-resolution reconstruction model to enhance image quality; input the color images and the infrared images after image quality enhancement into a trained segmentation model and a detection model at the same time; The preliminary segmentation result output by the segmentation model is mapped back to the size of the original image, and a segmentation confidence feature map is obtained through normalization and maximum value operation; the preliminary detection result output by the detection model is subjected to non-maximum suppression processing, and the confidence threshold is set to a, and the intersection-over-union threshold is set to 0.6; based on the preliminary detection result, an all-zero array of the same size as the original image is created as a detection confidence feature map; for the area within the detection frame, when the detection confidence is greater than or equal to a, the value of the detection confidence feature map of the area is set to the corresponding detection confidence; for the overlapping area of ​​the detection frames, the value of the detection confidence feature map is set to the average value of the confidence of each detection frame in the overlapping area; the segmentation confidence feature map is multiplied by a weight w, and the detection confidence feature map is converted into the detection confidence feature map. The feature map is multiplied by the weight (1-w), and the two weighted feature maps are added pixel by pixel to obtain the final confidence feature map. The fused feature map is normalized to ensure that the eigenvalues ​​are distributed in the interval [0,1]. A weight candidate set is constructed in the interval [0,1] with a step size of 0.05, and a threshold candidate set is constructed in the interval [0.05,0.95] with a step size of 0.

05. For each set of parameters in the candidate set, the detection accuracy and recall rate are calculated on the test set. The F1 score is used as the evaluation index, and the parameter combination that maximizes the F1 score is selected as the final weight w and confidence threshold a. Pixels with a value greater than 0.5 in the final confidence feature map are determined to be waterlogged or water-stained areas.

2. The method for detecting water stains based on deep learning according to claim 1, characterized in that: The image super-resolution reconstruction model is constructed and trained. The image super-resolution reconstruction model uses a U-shaped network as a backbone network, and the input is a low-resolution image that is downsampled, blurred and denoised. The perceptual loss, binary cross entropy loss and L1 loss are used as training loss functions, including: The U-shaped network is used as the backbone network. A four-layer convolutional pooling network is used for feature extraction in the encoder part. Each convolutional pooling network consists of two groups of convolutional layers, an activation layer, and a batch normalization layer. The decoder part uses a deconvolution layer for feature upsampling, and the features of the corresponding layer of the encoder are fused with the feature map of the decoder through a skip connection structure. The training image is downsampled by 4 times and Gaussian blur and random noise are added to obtain a low-resolution image, which is input into the image super-resolution reconstruction model. The perceptual loss function is used to calculate the distance between the generated image and the original image in the feature space. The binary cross entropy loss function is used to constrain the image classification result. The L1 loss function is used to constrain the image reconstruction accuracy. The weighted sum of the three loss functions is used as the total loss function for network optimization.

3. The method for detecting water stains based on deep learning according to claim 1, characterized in that: The detection model is constructed and trained, and the detection category is set to water using the intersection-over-union loss function, including: A backbone network with a residual structure is used for feature extraction. Feature maps of different scales are fused through a feature pyramid network. Three anchor frames of preset scales are set in each prediction layer of the feature pyramid network. The feature map output by the prediction layer is passed through a 1×1 convolutional layer to obtain a category prediction branch and a bounding box regression branch. The category prediction branch outputs the probability value of each anchor frame belonging to the water category, and the bounding box regression branch outputs the offset of each anchor frame relative to the target true frame. The input image is scaled to a preset size and normalized, the intersection-over-union (IoU) of the predicted box and the true box is calculated, and the anchor box whose IoU is greater than the preset threshold is taken as a positive sample. The coordinate offset between the predicted box and the true box, the difference between the confidence score and the corresponding true value are taken as regression loss. The network parameters are optimized using the IoU loss function, and the ratio of the classification loss weight to the regression loss weight in the loss function is set to 1:

5.

4. The method for detecting water stains based on deep learning according to claim 1, characterized in that: Mapping the preliminary segmentation result output by the segmentation model back to the original image size, and obtaining a segmentation confidence feature map through normalization and maximum value operations, including: Perform bilinear interpolation on the feature map of size (w3,h3,2) output by the segmentation model, adjust the size of the feature map to the size of the original input image (w,h), keep the number of channels unchanged, and use the pixel resampling method to smooth the interpolated feature map to eliminate image distortion caused by interpolation; The mapped feature map is subjected to the maximum value operation in the channel dimension to obtain a single-channel feature map. Each pixel value in the feature map is normalized by min-max to map it to the interval [0,1]. The local response normalization method is used to perform local normalization of the spatial position of the feature map to enhance the distinguishability of the features, and a segmentation confidence feature map is obtained.

5. The method for detecting water stains based on deep learning according to claim 1, characterized in that: The non-maximum suppression processing is performed on the preliminary detection results output by the detection model, the confidence threshold is set to a, and the intersection-over-union ratio threshold is set to 0.6, including: The preliminary detection result output by the detection model is a feature map of size (1, 5, 8400), which contains the center coordinates, width, height and confidence information of each candidate box. The candidate boxes with confidence greater than the threshold a in the feature map are screened, and the screened candidate boxes are arranged in descending order according to the confidence to form a candidate box set. The intersection-and-union ratio between any two candidate boxes in the candidate box set is calculated; Take the candidate box with the highest confidence from the sorted candidate box set as the reference box, calculate its intersection-over-union ratio with the remaining candidate boxes, delete the candidate boxes with an intersection-over-union ratio greater than 0.6 from the candidate box set, repeat the above process until the candidate box set is empty, and get the final detection box set, where each detection box contains its position coordinates and the corresponding confidence value.

6. The method for detecting water stains based on deep learning according to claim 1, characterized in that: The method of creating an array of all zeros of the same size as the original image as a detection confidence feature map based on the preliminary detection result, and setting the value of the detection confidence feature map of the area to the corresponding detection confidence when the detection confidence is greater than or equal to a for the area within the detection frame, and setting the value of the detection confidence feature map to the average value of the confidence of each detection frame in the overlapping area for the detection frames, includes: Create a two-dimensional zero matrix of the same size as the input image, traverse the set of detection frames after non-maximum suppression, take the position coordinate information of each detection frame to determine the position index of the pixel point in the frame, and when the confidence value of the detection frame is greater than or equal to the threshold a, update the values ​​of the corresponding positions of all pixels in the frame in the zero matrix to the confidence value of the detection frame; Calculate the coordinates of the intersection area of ​​any two detection frames, store the pixel position index of the intersection area into the overlapping area set, and for each pixel in the overlapping area set, count the confidence values ​​of all detection frames containing the pixel, and assign the arithmetic mean of the confidence values ​​to the pixel at the corresponding position in the detection confidence feature map.

7. A water stain detection device based on deep learning, characterized in that: The device comprises: A model construction module is used to construct and train an image super-resolution reconstruction model, wherein the image super-resolution reconstruction model uses a U-type network as a backbone network, and the input is a low-resolution image that is downsampled and blurred and denoised, and uses perceptual loss, binary cross entropy loss and L1 loss as training loss functions; a segmentation model based on a U-type network and Fourier transform is constructed and trained, specifically, the input image is subjected to Fourier transform to obtain frequency domain information, and then the unet network performs four downsampling of the original input image through convolution-maximum pooling, and before each maximum pooling operation, the frequency domain information map is downsampled to the same resolution size as the input image feature of the layer through convolution, and is fused with the input image feature of the layer, and then maximum pooling is performed to obtain the next layer of features, and finally the feature maps of each level are fused with the feature maps obtained through deconvolution by jump connection; a detection model is constructed and trained, and the intersection-over-union loss function is used to set the detection category to water; An image acquisition module is used to capture images through a binocular camera, obtain color images during the day, and obtain infrared images at night; input the infrared images into a trained image super-resolution reconstruction model for image quality enhancement; and input the color images and the infrared images after the image quality enhancement into a trained segmentation model and a detection model at the same time; The water stain detection module is used to map the preliminary segmentation results output by the segmentation model back to the original image size, and obtain the segmentation confidence feature map through normalization and maximum value operation; perform non-maximum suppression processing on the preliminary detection results output by the detection model, set the confidence threshold to a, and set the intersection-over-union threshold to 0.6; create an all-zero array of the same size as the original image as the detection confidence feature map based on the preliminary detection results, and for the area within the detection frame, when the detection confidence is greater than or equal to a, set the value of the detection confidence feature map of the area to the corresponding detection confidence; for the overlapping area of ​​the detection frames, set the value of the detection confidence feature map to the average of the confidences of the detection frames in the overlapping area; multiply the segmentation confidence feature map by the weight w, and convert the detection confidence feature map into the detection confidence feature map. The confidence feature map is multiplied by the weight (1-w), and the two weighted feature maps are added pixel by pixel to obtain the final confidence feature map. The fused feature map is normalized to ensure that the feature values ​​are distributed in the interval [0,1]. A weight candidate set is constructed in the interval [0,1] with a step size of 0.05, and a threshold candidate set is constructed in the interval [0.05,0.95] with a step size of 0.

05. For each set of parameters in the candidate set, the detection accuracy and recall rate are calculated on the test set. The F1 score is used as the evaluation index, and the parameter combination that maximizes the F1 score is selected as the final weight w and confidence threshold a. Pixels with a value greater than 0.5 in the final confidence feature map are determined to be waterlogged or water-stained areas.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the deep learning-based water stain detection method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the deep learning-based water stain detection method described in any one of claims 1 to 6 are implemented.

Images